Garbled Text: Charset Diagnosis and Recovery Limits

1. Problem: bytes, escapes, and replacement loss are different failures

Common cases include literal \u4e2d\u6587 text that crossed an extra escaping layer, UTF-8 bytes mapped one-to-one through a single-byte encoding, GBK bytes rejected by a strict UTF-8 decoder and replaced with �, and multi-stage re-encoding that has already overwritten the source information. Each case requires different evidence and recovery bounds.

2. Minimal reproduction: typical garbled text generation paths

This comparison demonstrates common garbled text generation patterns:

/* 1. UTF-8 bytes mis-decoded as ISO-8859-1 (Latin1): */
// Original Text: "中文" (UTF-8 bytes: E4 B8 AD E6 96 87)
// ISO-8859-1 result has code units E4 B8 AD E6 96 87, including controls.

/* 2. GBK bytes mis-decoded as UTF-8: */
// Original Text: "中文" (GBK bytes: D6 D0 CE C4)
// Decoded Result: "���" (invalid sequences produce U+FFFD replacement characters)

3. Root cause: escaping layers, charset mismatches, and lost bytes

1. Escaping layer: an API may return a string value containing literal \u4e2d; verify that the complete JSON document is parsed exactly once rather than blindly parsing an arbitrary user string twice. 2. Charset mismatch: missing or incorrect HTTP charset data can interact with HTML sniffing, legacy defaults, and client behavior; ISO-8859-1 is not a universal browser fallback. 3. Database mismatch: table, column, connection, and driver encodings can disagree during reads or writes. 4. Irreversible loss: once malformed input is replaced with � and that string overwrites the source, the original bytes cannot be uniquely recovered.

4. Recommendation: preserve bytes first, then determine reversibility

Capture original response or file bytes together with headers and connection settings. If stored text already contains �, recover from the source, packet capture, logs, or backups rather than claiming deterministic repair. Only reverse a Latin1 mapping when evidence shows each original byte survived one-to-one as a code unit. Standardize new paths on UTF-8/utf8mb4 and verify byte-level vectors containing Chinese, emoji, and combining characters.

5. Complete code: Latin1 mis-decoded UTF-8 string recovery helper function

This TypeScript snippet demonstrates diagnosing garbled text patterns and recovering UTF-8 strings incorrectly decoded as ISO-8859-1 (Latin1).

function repairLatin1GarbledUtf8(garbledStr: string): string {
  try {
    const bytes = new Uint8Array(garbledStr.length);
    for (let i = 0; i < garbledStr.length; i++) {
      bytes[i] = garbledStr.charCodeAt(i) & 0xff;
    }

    const decoder = new TextDecoder("utf-8", { fatal: true });
    return decoder.decode(bytes);
  } catch (e) {
    console.error("Recovery failed: bytes suffered irreversible replacement damage or are not UTF-8");
    return garbledStr;
  }
}

const garbled = String.fromCharCode(0xe4, 0xb8, 0xad, 0xe6, 0x96, 0x87);
console.log("Recovered String:", repairLatin1GarbledUtf8(garbled)); // "中文"

6. Incorrect approaches

Assuming multiple encodeURIComponent calls automatically fix garbled text. Storing binary streams directly in VARCHAR database columns without charset awareness. Omitting charset directives in HTTP response headers.

7. Boundaries: legacy MySQL pseudo-Latin1 storage

Some legacy systems stored UTF-8 byte sequences in columns declared as latin1. Changing connection or column declarations without a byte-preserving migration can reinterpret or transcode that data again. Back up first, inspect HEX(column), reproduce the current read/write path, and migrate verified byte ranges in a staging copy; editing dump declarations alone is not a universal repair recipe.

8. Verification

Preserve raw bytes and headers, classify literal \u, single-byte mis-decoding, and � replacement separately, then audit file encoding, HTTP charset, driver connection settings, and column definitions at each hop. Replay a known byte vector to locate the first transformation that changes it.

9. FAQ

Why does a page show literal \u4e2d? Fix the serialization boundary so a complete JSON document is parsed once; string concatenation into JSON.parse or regex unescaping arbitrary input mishandles quotes, backslashes, and surrogate pairs.

Can text containing � be recovered? Not uniquely if U+FFFD already replaced and overwrote the source; use original files, backups, logs, or captured bytes.

How do I prevent mojibake across services? Specify and verify UTF-8 at protocol, file, driver, and database boundaries, and test exact bytes rather than only visual output.

10. Summary

Diagnosing garbled text requires identifying where physical byte streams decoupled from decoding tables. Distinguish between reversible mis-decoding and irreversible replacement damage, and enforce UTF-8 across the entire stack.