Large JSON: Web Workers and Virtual Lists

1. Identify the four bottlenecks

The raw ArrayBuffer, decoded JavaScript string, parsed object, flattened row index, and DOM can coexist. Sending the complete object from a worker to the main thread also invokes structured cloning.

Virtualization reduces DOM size, not parsed-object memory. A worker prevents main-thread parsing but does not make JSON.parse streaming or reduce total memory by itself.

• File reading loads bytes into memory.
• TextDecoder creates a JavaScript string.
• JSON.parse synchronously constructs the full value.
• Recursive rendering can create an excessive number of DOM nodes.

2. Architecture: retain full data in the worker

Transfer the file ArrayBuffer to the worker. ArrayBuffer is transferable; a normal JavaScript object is not zero-copy transferable in the same way. The worker owns the parsed data and row index, returns only the total count, and serves small start/end slices as the viewport changes.

export type MainToWorker =
  | { type: 'load'; buffer: ArrayBuffer }
  | { type: 'range'; start: number; end: number };

export type JsonRow = {
  id: number;
  depth: number;
  key: string;
  preview: string;
  valueType: string;
};

3. Worker implementation

This example flattens a tree into lightweight rows for a fixed-height viewer. It still stores the parsed value and all rows, so it is not an unlimited solution. Extremely deep trees, huge string fields, or millions of nodes need lazy indexing, incremental parsing, or server preprocessing.

// json-viewer.worker.ts
type JsonRow = {
  id: number;
  depth: number;
  key: string;
  preview: string;
  valueType: string;
};

let rows: JsonRow[] = [];
let nextId = 0;

function preview(value: unknown): string {
  if (typeof value === 'string') {
    return JSON.stringify(value.length > 200 ? `${value.slice(0, 200)}…` : value);
  }
  if (value === null) return 'null';
  if (typeof value === 'object') {
    return Array.isArray(value) ? `Array(${value.length})` : 'Object';
  }
  return String(value);
}

function flatten(value: unknown, key = '$', depth = 0): void {
  rows.push({
    id: nextId++,
    depth,
    key,
    preview: preview(value),
    valueType: value === null ? 'null' : Array.isArray(value) ? 'array' : typeof value,
  });
  if (Array.isArray(value)) {
    value.forEach((child, index) => flatten(child, `[${index}]`, depth + 1));
  } else if (value && typeof value === 'object') {
    for (const [childKey, childValue] of Object.entries(value)) {
      flatten(childValue, childKey, depth + 1);
    }
  }
}

self.onmessage = (event: MessageEvent) => {
  try {
    if (event.data.type === 'load') {
      rows = [];
      nextId = 0;
      const text = new TextDecoder('utf-8', { fatal: true }).decode(event.data.buffer);
      flatten(JSON.parse(text));
      self.postMessage({ type: 'ready', total: rows.length });
    } else if (event.data.type === 'range') {
      const start = Math.max(0, event.data.start);
      const end = Math.min(rows.length, event.data.end);
      self.postMessage({ type: 'rows', start, rows: rows.slice(start, end) });
    }
  } catch (error) {
    self.postMessage({ type: 'error', message: (error as Error).message });
  }
};

4. Transfer the input buffer from the client

After postMessage transfers the buffer, the main-thread buffer is detached. In Nuxt 3, create the worker only on the client, for example inside onMounted, because Worker, File, and window are not available during SSR.

import { onBeforeUnmount, onMounted } from 'vue';

let worker: Worker | undefined;

onMounted(() => {
  worker = new Worker(
    new URL('./json-viewer.worker.ts', globalThis._importMeta_.url),
    { type: 'module' },
  );
});

async function loadFile(file: File): Promise<void> {
  if (!worker) throw new Error('Worker is not ready');
  const buffer = await file.arrayBuffer();
  worker.postMessage({ type: 'load', buffer }, [buffer]);
}

onBeforeUnmount(() => worker?.terminate());

5. Fixed-height virtualization math

Create a spacer with total * rowHeight, calculate the visible range from scrollTop, and add overscan. Disable wrapping and truncate previews; variable-height rows require a different measurement and caching strategy.

const rowHeight = 28;
const overscan = 8;

function calculateRange(scrollTop: number, height: number, total: number) {
  const first = Math.floor(scrollTop / rowHeight);
  const count = Math.ceil(height / rowHeight);
  const start = Math.max(0, first - overscan);
  const end = Math.min(total, first + count + overscan);
  return {
    start,
    end,
    offsetY: start * rowHeight,
    totalHeight: total * rowHeight,
  };
}

6. Vue viewport structure

Request a new worker slice only when start/end changes, ideally once per animation frame. The spacer provides the scrollbar, and the visible row container is translated to the correct offset.

<template>
  <div class="viewport" @scroll="scheduleRange">
    <div :style="{ height: `${totalHeight}px` }" />
    <div class="rows" :style="{ transform: `translateY(${offsetY}px)` }">
      <div
        v-for="row in visibleRows"
        :key="row.id"
        class="row"
        :style="{ height: `${rowHeight}px`, paddingLeft: `${row.depth * 16}px` }"
      >
        <strong>{{ row.key }}</strong>: {{ row.preview }}
      </div>
    </div>
  </div>
</template>

7. Memory boundaries and alternatives

JSON.parse requires complete text. Peak memory can be much larger than the file because bytes, text, objects, and indexes coexist. Do not promise a universal file-size threshold.

Prefer server pagination, NDJSON, chunked APIs, a streaming parser, or a desktop process when the file exceeds the device budget, the top-level array is naturally pageable, or the user needs only a small subset of fields.

• Do not postMessage the complete parsed object to the UI thread.
• Do not keep raw, pretty-printed, parsed, and fully rendered copies indefinitely.
• Highlight only visible rows.
• Do not create a new worker for every scroll event.

8. Verification, FAQ, and conclusion

Measure decode, parse, and flatten separately in the worker; inspect main-thread long tasks and peak memory; confirm that the DOM node count tracks the viewport rather than total rows. Include large arrays, deep objects, long strings, and invalid UTF-8 in tests.

Does a worker make JSON.parse faster? Not necessarily; it mainly keeps the UI responsive.

Can a parsed object be transferred? Normal objects are structured-cloned; specific types such as ArrayBuffer are transferable.

Why is memory still high after virtualization? The full text, parsed object, and row index still exist. The robust design transfers input bytes, retains data in the worker, and sends viewport slices only. See /articles/json-unexpected-token and /articles/json-number-precision for separate parsing concerns.