UTF-8 UTF-8 (8-bit Unicode Transformation Format) Unicode Flagship

What it is

UTF-8 is the variable-width encoding that represents every Unicode code point using 1 to 4 bytes. It's backward-compatible with 7-bit ASCII — any valid ASCII file is already valid UTF-8, byte-for-byte — which is a big part of why it won over competing encodings during the 1990s–2000s transition to Unicode. As of 2026, UTF-8 accounts for the overwhelming majority of web content, and it's the default or required encoding in most modern languages, databases, and file formats.

If you're debugging garbled text anywhere in a modern stack, there's a good chance UTF-8 is involved on at least one side of the mismatch — either as the correct encoding being misread as something else, or as the target encoding a legacy system failed to produce correctly.

Byte structure

UTF-8 encodes each Unicode code point using a pattern based on how large the code point is:

Every continuation byte starts with 10, which is what lets a decoder resync mid-stream even if it starts reading from a random offset — a property plain ASCII-extended encodings don't have. Basic Latin characters (English letters, digits, punctuation) stay 1 byte. Most European accented characters, Greek, Cyrillic, and Hebrew land in the 2-byte range. Common CJK (Chinese/Japanese/Korean) characters take 3 bytes. Emoji and rarer historic/symbol scripts — anything outside the Basic Multilingual Plane — need the full 4 bytes.

This variable width is exactly why "character count" and "byte length" diverge in UTF-8: a string with 10 emoji might be 10 characters but 40 bytes, which trips up any code that assumes 1 character = 1 byte.

Why your text is garbled (symptom → cause)

Compatibility & gotchas

• MySQL's utf8 is not full UTF-8. It's capped at 3 bytes per character, which silently mangles or errors on emoji and other 4-byte code points. Always use utf8mb4 for new schemas — full breakdown on the utf8 vs utf8mb4 page.
• Python 3 strings are Unicode internally, and str.encode() defaults to UTF-8 — but reading a file with open() uses the platform's default locale encoding unless you explicitly pass encoding='utf-8'. This is a frequent source of "works on my Mac, breaks on Windows" bugs.
• JavaScript strings are UTF-16 internally, not UTF-8 — see the UTF-16 page for how that affects .length and string indexing on characters outside the BMP (like most emoji).
• HTTP responses should declare UTF-8 explicitly via Content-Type: text/html; charset=utf-8 — without it, browsers fall back to sniffing or locale defaults, which is inconsistent across clients.
• The UTF-8 BOM is optional and often unwanted. Unlike UTF-16, UTF-8 doesn't need a byte order mark since there's no byte-order ambiguity — but some Windows tools (notably Excel and older Notepad) still write one, which can break shebang lines (#!/usr/bin/env), JSON parsers, and CSV import tools that don't expect leading bytes.
• Locale-dependent behavior in C/C++ and older Windows APIs — many legacy Windows APIs still default to the system's active code page rather than UTF-8, so cross-platform code needs explicit -utf-8 compiler flags or SetConsoleOutputCP(CP_UTF8) calls to behave consistently.

Fixing it

• Database columns/tables: ALTER TABLE your_table CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; — do this per-table, and check the connection charset too (SET NAMES utf8mb4), since a UTF-8 column can still get corrupted by a connection negotiating a different charset.
• Command-line conversion: iconv -f WINDOWS-1252 -t UTF-8 input.txt -o output.txt converts a misencoded file once you know its actual source encoding (often discoverable by trial and error, or with file -i filename on Linux/macOS).
• Python file reads: always pass encoding='utf-8' explicitly — open('file.txt', encoding='utf-8') — rather than relying on the platform default.
• Stripping a UTF-8 BOM: most languages have a one-liner — in Python, content.lstrip('\ufeff'); on the command line, sed -i '1s/^\xef\xbb\xbf//' file.txt.
• HTTP/HTML: declare charset explicitly and early — <meta charset="utf-8"> should be the first thing inside <head>, and the HTTP header should match.

UTF-8 vs. commonly confused encodings

FAQ

Why does my UTF-8 text show up as weird symbols like é or ’?

This is the classic "double-decode" mojibake pattern: your UTF-8 bytes were read by something expecting Latin-1 or Windows-1252, and each multi-byte UTF-8 character got reinterpreted as two or three separate Latin-1 characters. The fix is to make sure every layer of your stack — file encoding, database connection, HTTP headers — consistently declares and uses UTF-8.

Is UTF-8 the same as Unicode?

No. Unicode is the character set — the master list assigning a unique code point to every character. UTF-8 is one of several encodings that turn those code points into actual bytes for storage or transmission (UTF-16 and UTF-32 are the other major ones). Unicode defines what a character is; UTF-8 defines how it's represented as bytes.

Why do emoji sometimes turn into question marks in my database?

This almost always means your MySQL column or connection is using the legacy utf8 charset (3-byte max) instead of utf8mb4 (true UTF-8, up to 4 bytes) — most emoji live outside the Basic Multilingual Plane and require 4 bytes. See the dedicated utf8 vs utf8mb4 page for the full fix.

Should I use a BOM with UTF-8?

Generally no, for web and cross-platform code — UTF-8 doesn't need a BOM to resolve byte order (unlike UTF-16), and a stray BOM can break parsers that don't expect leading bytes (shebang lines, some JSON/CSV parsers). It's occasionally useful as an explicit "this file is UTF-8" signal in Windows-only tooling, but it causes more problems than it solves in most modern contexts.

Fun fact

UTF-8 was reportedly designed on a placemat. Ken Thompson and Rob Pike sketched out the encoding scheme in a New Jersey diner in September 1992, and implemented it in Plan 9 within a day — the elegant self-synchronizing continuation-byte design (every non-leading byte starts with 10) came out of that single sitting.