JSON over HTTP: Choosing Gzip, Brotli, and Streaming

1. Scope: three layers, three different jobs

Whitespace removal changes a text representation. Escaping makes text safe inside another string and often adds characters. Gzip and Brotli are negotiated as HTTP Content-Encoding and are normally decoded before application code reads a response. Identify whether the issue is transfer size, an oversized response shape, or consumption before choosing a remedy.

2. Minimal reproduction: response headers describe the representation

A compressed response needs an accurate Content-Type, Content-Encoding, and Vary header. In browsers, fetch exposes the decoded response body; a normal fetch result should not be manually decompressed as if it were raw Brotli bytes.

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Encoding: br
Vary: Accept-Encoding

{"items":[{"id":"a1","name":"Ada"}]}

3. Root cause: Accept-Encoding is negotiation, not JSON data

A client advertises encodings it accepts, and a server or CDN chooses a representation and labels it in the response. Algorithm settings, content repetition, caches, and proxies affect results, so fixed ratios or timings cannot be promised. Vary prevents a shared cache from serving the wrong representation to another client.

4. Recommended choice: reduce data before changing transport

Use normal JSON with HTTP compression for ordinary responses. For a collection that is too large, use cursor pagination or field selection first. Use application/x-ndjson only when the producer can emit independent records and the consumer can process them independently; it is not a line-splitting shortcut for a JSON array.

• Small complete object: JSON plus Content-Encoding.
• Split collection: cursor pagination or requested fields.
• Independent continuous records: NDJSON, one complete JSON value per line.

5. Complete browser example: reading NDJSON safely

This browser-side function checks the response, retains a partial line across chunks, and flushes the decoder at the end. The callback consumes records without making assumptions about UI rendering.

async function readNdjson(url: string, onItem: (item: unknown) => void): Promise<void> {
  const response = await fetch(url, { headers: { Accept: "application/x-ndjson" } });
  if (!response.ok) throw new Error("HTTP " + response.status);
  if (!response.body) throw new Error("ReadableStream is unavailable");
  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let pending = "";
  const consume = (line: string) => { if (line.trim() !== "") onItem(JSON.parse(line)); };
  while (true) {
    const chunk = await reader.read();
    if (chunk.done) break;
    pending += decoder.decode(chunk.value, { stream: true });
    const lines = pending.split("\n");
    pending = lines.pop() ?? "";
    for (const line of lines) consume(line.endsWith("\r") ? line.slice(0, -1) : line);
  }
  pending += decoder.decode();
  consume(pending.endsWith("\r") ? pending.slice(0, -1) : pending);
}

6. Incorrect approaches

Stringifying JSON again adds escaping and a second parse unless a protocol explicitly requires embedded text. Base64-wrapping compressed bytes often expands text and bypasses normal HTTP negotiation. Splitting arbitrary JSON on newlines is invalid; only NDJSON guarantees one complete record per line.

7. Boundaries: caches, error responses, and stream tails

An error response may be HTML or plain text, so check status and Content-Type first. The last NDJSON line may lack a newline, and TextDecoder may hold an incomplete multibyte character; flush it with decoder.decode(). Decide explicitly whether a malformed record stops the stream or is represented by the protocol.

8. Verification

Inspect Accept-Encoding, Content-Encoding, and Vary in the Network panel. Compare headers for the same resource rather than borrowing someone else's compression numbers. Test the reader with split lines, blank lines, CRLF, and a final line without a newline, then assert callback order and values.

9. FAQ

Is Brotli always better than Gzip? No; server, client, cache, and content conditions matter.

Can compression replace pagination? No; it does not change how much data the client ultimately handles.

Can NDJSON contain nested data? Yes, if each top-level record is a complete JSON value on one line.

10. Summary

Minification, escaping, and Content-Encoding are separate layers. Use negotiated Gzip or Brotli for ordinary JSON, reduce unnecessary data before changing the transport, and choose NDJSON only for independently consumable records. Verify headers and stream-tail behavior with your own endpoint.