Fix btoa Errors: UTF-8, Chinese Text and Emoji

1. Problem: the Latin-1 character set restriction of btoa

Browser-native window.btoa() expects input code points within the U+0000 to U+00FF range (Latin-1). When inputs contain Chinese characters (e.g., U+4E2D) or Emojis (e.g., U+1F600), btoa cannot map multi-byte sequences directly and throws an InvalidCharacterError.

2. Minimal reproduction: calling btoa on Unicode text throws an exception

The JavaScript snippet below demonstrates the crash caused by calling btoa directly on non-Latin1 text:

try {
  /* Contains Chinese characters exceeding Latin-1 (0-255) */
  window.btoa("Hello 世界!");
} catch (e) {
  console.error(e);
  // DOMException: Failed to execute 'btoa': The string contains characters outside Latin1 range.
}

3. Root cause: missing byte stream conversion and character mapping

Base64 maps binary byte sequences to 6-bit printed units. btoa assumes each JS char code represents a single byte. However, modern JS uses UTF-16, and Chinese characters consume 3 bytes in UTF-8. Passing raw UTF-16 code points to btoa misaligns code points with bytes. Base64 is a reversible encoding scheme, not encryption.

4. Recommendation: use TextEncoder and TextDecoder for byte bridging

1. Encoding flow: convert Unicode strings into UTF-8 Uint8Array byte arrays using TextEncoder, map bytes to Latin-1 single-byte characters, and pass them to btoa. 2. Decoding flow: obtain binary strings via atob, convert them to a Uint8Array, and restore Unicode text via TextDecoder. 3. Node.js: use Buffer.from(str, 'utf-8').toString('base64').

5. Complete code: cross-platform Unicode Base64 helper functions

This TypeScript snippet demonstrates safely encoding and decoding Unicode strings containing Chinese, Japanese, and Emojis.

function unicodeToBase64(str: string): string {
  const bytes = new TextEncoder().encode(str);
  let binString = "";
  for (let i = 0; i < bytes.length; i++) {
    binString += String.fromCharCode(bytes[i]);
  }
  return btoa(binString);
}

function base64ToUnicode(base64: string): string {
  const binString = atob(base64);
  const bytes = Uint8Array.from(binString, function (char) {
    return char.charCodeAt(0);
  });
  return new TextDecoder().decode(bytes);
}

const originalText = "Hello 世界! 👋";
const encoded = unicodeToBase64(originalText);
console.log(encoded); // "SGVsbG8g5LiW55WMISDwn5iA"
console.log(base64ToUnicode(encoded)); // "Hello 世界! 👋"

6. Incorrect approaches

Relying on deprecated encodeURIComponent + unescape hacks that fail on UTF-16 surrogate pairs. Attempting to use Base64 to conceal passwords instead of AES/RSA encryption.

7. Boundaries: surrogate pairs and 4-byte Emojis

Complex Emojis (family or flag icons) require TextEncoder to produce standard UTF-8 byte streams; never split strings with split('').

8. Verification

Assert base64ToUnicode(unicodeToBase64(text)) === text. Cover empty input, CJK, emoji, combining characters, unpaired surrogates, and large values, and define the expected failure for malformed Base64.

9. FAQ

Why was btoa designed to support only Latin-1? btoa (Binary to ASCII) was originally created to handle 8-bit binary data streams, not multi-byte Unicode text.

Can Base64 protect against web scrapers or store passwords? No! Base64 uses no secret keys; anyone can run atob to recover plain text. Use Argon2id or bcrypt for passwords.

What if legacy environments lack TextEncoder? All modern browsers and Node.js 11+ support it natively; legacy apps can include fast-text-encoding Polyfills.

10. Summary

Encoding Chinese and Emojis to Base64 follows the path: Unicode -> UTF-8 Bytes -> Base64 Chars. TextEncoder and TextDecoder provide standard, reliable handling in modern JS.