MD5 Hash Mismatches: UTF-8 vs GBK Encoding

1. Problem: character encoding mismatches causing hash calculation drift

Developers often wrongly assume MD5 operates on string text. In reality, cryptographic hash functions accept raw byte streams. For example, the string '中文' occupies 6 bytes in UTF-8 (e4 b8 ad e6 96 87) but only 4 bytes in GBK (d6 d0 ...). Different byte inputs guarantee completely different MD5 output hashes.

2. Minimal reproduction: identical text yielding different MD5 under UTF-8 and GBK

This example illustrates physical byte differences and MD5 outputs for identical Chinese text converted to UTF-8 vs GBK bytes:

/* Text string: "中文" */
const text = "中文";

// 1. Front-end / modern backend (UTF-8, 6 bytes: E4 B8 AD E6 96 87)
// MD5("中文" in UTF-8) -> "a7bac2239fcdcb3a067903d8077c73a5"

// 2. Legacy Java / C# backend (GBK, 4 bytes: D6 D0 CE C4)
// MD5("中文" in GBK)   -> "b2ea9170eedece3f0e0c8b323671a742"

/* Conclusion: identical string literals with different character encodings produce mismatched MD5 hashes! */

3. Root cause: byte-level hash nature and implicit default character sets

1. Byte-stream input: MD5 (RFC 1321) processes 512-bit (64-byte) blocks of raw binary bits, ignorant of Unicode concepts. 2. Implicit defaults: legacy Java string.getBytes() uses OS defaults (GBK or Windows-1252), whereas modern JavaScript TextEncoder strictly defaults to UTF-8. 3. MD5 security role: MD5 suffers collision flaws; use it solely for legacy checksums, never for passwords.

4. Recommendation: explicit UTF-8 byte stream contracts and encoding layers

1. Unified protocol contract: enforce UTF-8 character encoding in all API signature specifications. 2. Explicit encoding parameters: disallow parameterless getBytes() in backend code—always use getBytes(StandardCharsets.UTF_8). 3. Legacy compatibility: when interfacing with legacy GBK systems, convert strings to GBK bytes before hashing.

5. Complete code: explicit UTF-8 bytes and diagnostics

This TypeScript snippet uses the browser's native UTF-8 encoder and prints the exact input bytes, so both systems can align bytes before comparing a digest.

// Native browser TextEncoder emits UTF-8 only.
function encodeUtf8(text: string): Uint8Array {
  return new TextEncoder().encode(text);
}

function bytesToHex(bytes: Uint8Array): string {
  return Array.from(bytes)
    .map((b) => b.toString(16).padStart(2, "0"))
    .join(" ");
}

const inputStr = "开源工具";
const utf8Bytes = encodeUtf8(inputStr);

console.log("Input string:", inputStr);
console.log("UTF-8 Raw Bytes:", bytesToHex(utf8Bytes));
// Passing utf8Bytes into MD5 guarantees matching front-end and back-end hashes.

6. Incorrect approaches

Performing bitwise truncations directly on Unicode JS strings. Assuming all platform defaults are UTF-8. Using MD5 to hash user passwords or security tokens.

7. Boundaries: UTF-8 BOM, Emoji, and surrogate pairs

A UTF-8 BOM (EF BB BF) adds three leading bytes. The 👋 emoji is U+1F44B and is represented by a UTF-16 surrogate pair inside JavaScript. Both sides must agree on Unicode normalization and character encoding before hashing.

8. Verification

Compare hexadecimal input bytes before comparing the final MD5. Maintain fixed test vectors for empty input, Chinese text, 👋, combining characters, BOMs, and line endings, and assert both bytes and digests on every platform.

9. FAQ

Why do English alphanumeric strings usually yield identical MD5 across encodings? ASCII-range characters have the same byte values in UTF-8, GBK, and many single-byte encodings; non-ASCII text exposes the difference.

Can browser JavaScript encode text as GBK before hashing? Yes, but native encoders do not support GBK. Choose a trusted codec that explicitly supports it and verify fixed byte vectors; a UTF-8 polyfill is not a GBK encoder.

Is MD5 still secure? No. Practical collision attacks make it unsuitable for passwords, signatures, or adversarial integrity checks; migrate legacy checksums to SHA-256 where possible.

10. Summary

Hashes compute raw byte streams, not abstract text. Enforcing explicit UTF-8 byte encoding across front-end and back-end eliminates hash verification failures.