Unicode: UTF-8, UTF-16 and Surrogate Pairs

1. Problem: confusing Character Sets with Transfer Encodings

Many developers confuse Unicode with UTF-8. Unicode is a dictionary mapping characters to unique numerical Code Points (U+0000 sim U+10FFFF). UTF-8 and UTF-16 are encoding mechanisms that map those Code Points to binary byte streams. Confusing the two makes it difficult to reason about byte sizes and string pointer offsets.

2. Minimal reproduction: binary representations across encodings

This comparison illustrates the binary differences between Chinese character '中' and Emoji '🚀':

/* 1. Chinese character '中' (Unicode Code Point U+4E2D): */
// UTF-8 Storage: 3 bytes -> 0xE4 0xB8 0xAD (11100100 10111000 10101101)
// UTF-16 Storage: 2 bytes (1 Code Unit) -> 0x4E2D

/* 2. Emoji '🚀' (Unicode Code Point U+1F680): */
// UTF-8 Storage: 4 bytes -> 0xF0 0x9F 0x9A 0x80
// UTF-16 Storage: 4 bytes (2 Code Units / Surrogate Pair) -> High: 0xD83D, Low: 0xDE80

3. Root cause: Surrogate Pair math and UTF-8 bitwise structural rules

1. Basic Multilingual Plane (BMP): characters in U+0000 sim U+FFFF fit within a single 16-bit UTF-16 Code Unit. 2. Supplementary Planes: characters in U+10000 sim U+10FFFF exceed 16-bit bounds. Unicode reserved the range U+D800 sim U+DFFF exclusively for Surrogate Pairs. 3. Surrogate Pair Formula: subtract 0x10000 from the Code Point to yield a 20-bit offset N. The upper 10 bits plus 0xD800 form the High Surrogate (0xD800 sim 0xDBFF); the lower 10 bits plus 0xDC00 form the Low Surrogate (0xDC00 sim 0xDFFF). 4. UTF-8 Prefix Rules: uses 1 to 4 bytes based on Code Point value, marking continuation bytes with 10xxxxxx.

4. Recommendation: bitwise formulas and native Code Point methods

1. Bitwise Bitshifting: master the upper 10-bit shift (N >> 10) + 0xD800 and lower 10-bit mask (N & 0x3FF) + 0xDC00. 2. Native Modern APIs: use String.fromCodePoint() and str.codePointAt() in JavaScript instead of legacy fromCharCode() and charCodeAt().

5. Complete code: Code Point to UTF-16 Surrogate Pair calculation helper

This TypeScript snippet demonstrates manually deriving High and Low UTF-16 Surrogate Pairs for supplementary plane Unicode code points.

function codePointToSurrogatePair(codePoint: number): { high: number; low: number } | null {
  if (codePoint <= 0xffff) {
    return null; // BMP characters do not require surrogate pairs
  }

  const N = codePoint - 0x10000;
  const high = (N >> 10) + 0xd800;
  const low = (N & 0x3ff) + 0xdc00;

  return { high, low };
}

const cp = 0x1f680; // U+1F680 ('🚀')
const pair = codePointToSurrogatePair(cp);
console.log("🚀 High Surrogate (Hex):", pair?.high.toString(16).toUpperCase()); // D83D
console.log("🚀 Low Surrogate (Hex):", pair?.low.toString(16).toUpperCase());  // DE80

6. Incorrect approaches

Confusing charCodeAt(0) with codePointAt(0) (charCodeAt returns only half a surrogate pair for supplementary characters). Assuming all Chinese characters in UTF-8 consume 2 bytes (most consume 3 bytes). Mixing Big-Endian (BE) and Little-Endian (LE) UTF-16 byte orders without BOM header checks.

7. Boundaries: Lone Surrogates and malformed byte handling

An un-paired High Surrogate missing its Low Surrogate partner forms an invalid 'Lone Surrogate'. Converting Lone Surrogates to UTF-8 is invalid; modern APIs (like TextEncoder) replace malformed surrogates with the U+FFFD replacement character.

8. Verification

Inspect Byte Order Marks (BOM) like FE FF (UTF-16 BE) or FF FE (UTF-16 LE). Enforce strict decoding validation using new TextDecoder('utf-8', { fatal: true }) to catch malformed byte sequences.

9. FAQ

Which is more space-efficient: UTF-8 or UTF-16? For ASCII characters, UTF-8 (1 byte) saves 50% space over UTF-16 (2 bytes). For Chinese text, UTF-16 (2 bytes) is more compact than UTF-8 (3 bytes).

Why did Unicode reserve 0xD800 sim 0xDFFF? Reserving this range ensures no single valid character Code Point falls in this block, allowing parsers to unambiguously detect surrogate pairs.

How does JavaScript represent strings internally? The ECMAScript spec defines String as a sequence of 16-bit UTF-16 Code Units.

10. Summary

Character sets are abstract dictionaries; transfer encodings are physical binary implementations. Mastering UTF-16 Surrogate Pair math and UTF-8 variable byte structures is crucial for low-level text parsing.