UTF-8 BOM Explained — Why It Breaks JSON, CSV, Shebangs

What it is

The UTF-8 Byte Order Mark (BOM) is an optional 3-byte sequence (EF BB BF in hex) that some tools prepend to the very start of a UTF-8-encoded file, as an explicit "this file is UTF-8" signal. It's borrowed conceptually from UTF-16/UTF-32, where a BOM genuinely serves a functional purpose — indicating which byte order (big-endian or little-endian) the file uses, since those encodings have real byte-order ambiguity. UTF-8, being a byte-oriented (not word-oriented) encoding, has no such ambiguity to resolve — which means the UTF-8 BOM is purely an optional identification signal, not a functional necessity, and its presence causes more practical problems than it solves in most modern contexts.

Byte structure

Bytes (hex)	Meaning
`EF BB BF`	The UTF-8 BOM, appearing as the first 3 bytes of a file

When decoded incorrectly as a different encoding (like Latin-1), these three bytes render as the recognizable garbage sequence `` (three specific mojibake-style characters) — a useful visual signal that a file starts with a BOM that wasn't handled correctly.

Why your file is broken (symptom → cause)

Symptom	Likely cause
A script's shebang line (`#!/usr/bin/env python3`) fails to execute, with an obscure error	The BOM's 3 bytes precede the `#!`, so the operating system doesn't recognize the shebang syntax at the true start of the file — a classic, hard-to-diagnose BOM gotcha
JSON parsing fails with a syntax error at position 0 or "unexpected token"	Strict JSON parsers (per the JSON spec) don't expect a BOM and treat it as invalid content before the opening `{` or `[`
CSV import shows garbage characters or an extra empty-looking column in the first header cell	The BOM bytes get absorbed into the first column's header name, corrupting it invisibly for tools that don't strip it
File "looks the same" but a checksum/diff shows unexpected differences from a supposedly identical file	One version has a BOM, the other doesn't — an easy, invisible source of "identical-looking but not byte-identical" file mismatches
Excel displays a file correctly, but other tools show mojibake at the very start	Excel specifically adds a BOM when saving "UTF-8" CSVs (its way of signaling UTF-8 vs. its default ANSI/Windows-1252 export) — a deliberate choice that other tools don't always expect

Compatibility & gotchas

Not required for UTF-8 to function correctly — UTF-8 has no byte-order ambiguity, so the BOM serves purely as an optional identification signal, unlike its genuinely functional role in UTF-16/UTF-32.
Excel deliberately adds a BOM when using its "CSV UTF-8" export option specifically, as a way to distinguish UTF-8 output from its default locale-based (often Windows-1252) CSV export — this is Excel-specific behavior that surprises developers expecting plain, BOM-free UTF-8.
Breaks shebang lines in scripts, since the interpreter-detection mechanism relies on #! being the literal first two bytes of the file — a BOM preceding it silently breaks script execution on Unix-like systems.
Not universally handled by JSON/CSV parsers — strict specification-compliant parsers may reject BOM-prefixed content outright, while more lenient ones silently strip it; behavior varies enough across tools that BOM presence is a genuine source of cross-tool compatibility bugs.
Some Windows tools (older Notepad, some legacy software) add a BOM by default when saving as "UTF-8", while most Unix/web-focused tooling defaults to BOM-free UTF-8 — this platform-driven inconsistency is the root cause of most real-world BOM-related bugs.

Fixing it

Stripping a BOM in Python: content = content.lstrip('\ufeff') after decoding, or read with encoding='utf-8-sig' instead of 'utf-8', which automatically detects and strips a leading BOM if present.
Command-line stripping: sed -i '1s/^\xef\xbb\xbf//' file.txt removes a leading BOM in place on Unix-like systems.
Avoiding BOM issues at the source: when saving files intended for cross-platform/tooling compatibility (scripts, JSON, config files), prefer editors/tools with an explicit "UTF-8 without BOM" save option over generic "UTF-8" if the option is offered, since some tools default to adding one.
Detecting BOM presence programmatically: check whether a file's first three bytes are exactly EF BB BF before deciding whether to skip/strip them during parsing.

Comparison

Context	BOM behavior
UTF-8	Optional, purely an identification signal — not needed to resolve any actual ambiguity
UTF-16 / UTF-32	Functionally meaningful — indicates actual byte order (BE vs. LE), which those encodings genuinely need to be interpreted correctly

FAQ

Should I include a BOM in my UTF-8 files?

Generally no, for most modern web/cross-platform development — it's not needed for UTF-8's correctness and causes more compatibility problems (broken shebangs, strict JSON parser errors) than it solves. It's mainly relevant as an explicit signal in specific Windows-centric tooling contexts.

Why does my Python/Node script's shebang line fail with a BOM present?

The BOM's 3 bytes sit before the #! characters, so the operating system's interpreter-detection mechanism doesn't see #! as the literal first bytes of the file and fails to recognize it as a valid shebang line — a subtle, hard-to-diagnose bug specifically caused by BOM presence.

How do I know if my file has a BOM?

Check whether the first three bytes are EF BB BF in hex — most hex editors, or a quick xxd file.txt | head -1 on the command line, will reveal this immediately.

Fun fact

The UTF-8 BOM's mojibake signature when misdecoded as Latin-1 () is common enough in real-world data that it has its own informal nickname among some developers — sometimes called the "BOM ghost" or similar, since it's a recognizable, recurring artifact specifically tied to this one very particular encoding mismatch rather than a generic garbling pattern.

UTF-8 BOM UTF-8 Byte Order Mark Encoding Schemes