Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Grapheme Clusters: Why "Character" Is a Trap

When you read the word character, you think of a thing on a page. A letter. A digit. A punctuation mark. One slot in a monospace font. That mental model served you fine in ASCII. In Unicode, it quietly stops working.

This chapter names what you were actually thinking of — a grapheme cluster — and shows you why a grapheme cluster is not the same as a code point, and why mistaking one for the other is the single most common source of Unicode bugs.

What a user means by "character"

The informal definition that matters: a character is whatever a human reader would point to and call a character. It is the unit the cursor moves over when you press the right arrow key. It is what the backspace key deletes.

Unicode has a technical name for that thing: an extended grapheme cluster, defined by Unicode Annex #29 (UAX #29). We will shorten it to grapheme cluster — sometimes just grapheme — and use it precisely from now on.

A grapheme cluster is a sequence of one or more code points that is treated as a single unit by a reader. Most of the time it is exactly one code point. Sometimes it is more.

Combining marks

Consider the letter é — a lowercase Latin e with an acute accent. Unicode can represent this in two different ways:

  • Precomposed: a single code point, U+00E9 (LATIN SMALL LETTER E WITH ACUTE).
  • Decomposed: two code points, U+0065 (LATIN SMALL LETTER E) followed by U+0301 (COMBINING ACUTE ACCENT).

Both render as é on your screen. Both are legitimate Unicode. They are not the same sequence of code points, but they are the same grapheme cluster: one visual character, one cursor slot, one thing to backspace over.

>>> a = "\u00e9"           # precomposed
>>> b = "\u0065\u0301"     # decomposed
>>> a == b                  # byte-for-byte comparison
False
>>> len(a), len(b)
(1, 2)
>>> print(a, b)
é é

The code-point counts disagree. Visually they look identical. Neither is wrong.

U+0301 is a combining mark — a code point that, on its own, doesn't render as a standalone character. It attaches to the preceding base character. Unicode contains hundreds of combining marks for accents, tone marks, stacking diacritics, and so on. You can stack them:

e + ́ + ̂ + ̈  →  é̂̈        # e with acute, circumflex, and diaeresis

The grapheme cluster here is still one cluster; it just contains four code points.

People use combining marks intentionally to produce creations like "Zalgo text," which stacks many combining marks on a single base character. Zalgo text is mostly silly, but it is a useful reminder that a grapheme cluster can be arbitrarily many code points long.

Why both forms exist

Why not just pick one? Historical reasons, of course. Precomposed forms for common accented letters (é, ü, ñ, and so on) existed in legacy encodings like Latin-1, and Unicode preserved them for round-tripping. Combining marks exist because they generalize: you cannot precompose every letter–accent combination that a linguist might need, but you can combine them.

The result is that for many characters, there are multiple legitimate code-point sequences that render identically. In Chapter 5 we will cover normalization, the official way to turn any of these equivalent sequences into a canonical form, so that string comparison can be made reliable.

Emoji and their modifiers

Modern emoji turn the grapheme-cluster problem from an edge case into a headline feature.

Skin tone modifiers (Fitzpatrick)

Consider the waving hand emoji 👋. On its own, it is U+1F44B — one code point, one grapheme cluster.

Now consider 👋🏽 — a waving hand with medium skin tone. This is two code points:

U+1F44B WAVING HAND SIGN
U+1F3FD EMOJI MODIFIER FITZPATRICK TYPE-4

But it is one grapheme cluster. Your cursor moves over it as one character. Your backspace deletes both code points at once.

Unicode defines five skin tone modifiers (U+1F3FB through U+1F3FF), corresponding to the Fitzpatrick scale's types 1–2, 3, 4, 5, and 6. A supported base emoji followed by a modifier renders as a single tinted emoji, a single grapheme cluster, two code points, and — in UTF-8 — eight bytes.

Variation selectors

Some symbols can be rendered as either a "text" glyph (monochrome, aligned to the baseline) or an "emoji" glyph (colored, possibly animated). The default differs by symbol and platform. To force one or the other, Unicode provides two variation selectors:

  • U+FE0E (VARIATION SELECTOR-15) forces text presentation.
  • U+FE0F (VARIATION SELECTOR-16) forces emoji presentation.

Consider the heart symbol:

  • ♥ alone: U+2665. Renders as text or emoji depending on context.
  • ♥︎ = U+2665 U+FE0E. Forced text presentation.
  • ♥️ = U+2665 U+FE0F. Forced emoji presentation.

All three are one grapheme cluster. The second and third are two code points each.

ZWJ sequences

The zero-width joiner (U+200D, or ZWJ) tells a renderer "treat the characters on either side of me as a single ligature if you can." It is the mechanism behind the most elaborate emoji sequences in Unicode.

Consider the family emoji 👨‍👩‍👧‍👦 — man, woman, girl, boy. As a sequence of code points:

U+1F468 MAN
U+200D  ZERO WIDTH JOINER
U+1F469 WOMAN
U+200D  ZERO WIDTH JOINER
U+1F467 GIRL
U+200D  ZERO WIDTH JOINER
U+1F466 BOY

Seven code points. One grapheme cluster. In UTF-8, that is 25 bytes. In UTF-16, 11 code units (four supplementary code points × 2 + three ZWJs × 1). The renderer — if its font supports this particular family configuration — draws a single glyph.

Not all ZWJ sequences have defined glyphs. An unrecognized ZWJ sequence typically falls back to rendering the individual emoji side by side, possibly with the ZWJ as a visible gap. This is why family emoji look different across platforms, and why composing arbitrary ZWJ combinations may or may not produce meaningful pictures.

A worked example

Let us take the string "Hi 👨‍👩‍👧‍👦!" and count it four different ways.

>>> s = "Hi 👨\u200d👩\u200d👧\u200d👦!"
>>> len(s)                              # code points
11
>>> len(s.encode("utf-8"))              # UTF-8 bytes
29
>>> len(s.encode("utf-16-le")) // 2     # UTF-16 code units
15

Grapheme clusters: 5 (H, i, , 👨‍👩‍👧‍👦, !). Python's standard library does not count these directly; you need the regex module (not re) or the grapheme package:

>>> import regex
>>> len(regex.findall(r"\X", s))        # \X matches one grapheme cluster
5

Five is the answer a user would give if you asked "how many characters is this?" None of the other counts match it.

How to count grapheme clusters correctly, by language

This is the operation most languages make awkward. Here is where to reach for in each:

  • Python: the third-party regex module, regex.findall(r"\X", s), or the grapheme package.
  • JavaScript: Intl.Segmenter (ECMA-402, available in all modern browsers and Node ≥ 16). [...new Intl.Segmenter().segment(s)].length.
  • Java: java.text.BreakIterator.getCharacterInstance().
  • Go: the golang.org/x/text/unicode/norm and rivo/uniseg modules.
  • Rust: the unicode-segmentation crate. s.graphemes(true).count().
  • Swift: s.count. This is the default behavior of String.count — it counts grapheme clusters, not code points. Swift is the one major exception.
  • C/C++: the ICU library. icu::BreakIterator::createCharacterInstance.

If you are writing user-facing code — rendering a text field, truncating a tweet, counting remaining characters in a form — you want grapheme clusters. Counting code points will miscount any string with combining marks, emoji with modifiers, or ZWJ sequences. Counting code units is even more wrong. Counting bytes is most wrong of all.

Grapheme cluster boundaries: the rough rules

UAX #29 defines grapheme cluster breaks by a complex table of code-point categories. At a high level, a break is allowed between two code points unless the code points are in one of several non-breaking combinations:

  • No break after a CR followed by LF. (Treats \r\n as one cluster.)
  • No break between a base character and a combining mark.
  • No break between emoji components connected by ZWJs.
  • No break within a regional indicator pair (flag emoji).
  • No break between an emoji and a Fitzpatrick modifier.
  • No break between Hangul L, V, T, LV, and LVT syllable components.

The rules have exceptions and are revised with each Unicode version; if you need them in production, use an implementation, not your own code. But knowing the shape of the rules lets you predict what will happen to a weird string.

Regional indicators and flags

Flag emoji have a particularly elegant encoding: each country flag is a pair of regional indicator code points (U+1F1E6 through U+1F1FF), where each regional indicator corresponds to a Latin letter A–Z. Two regional indicators form an ISO 3166-1 alpha-2 country code.

🇺 + 🇸  =  🇺🇸          U+1F1FA U+1F1F8 ("US")
🇯 + 🇵  =  🇯🇵          U+1F1EF U+1F1F5 ("JP")
🇬 + 🇧  =  🇬🇧          U+1F1EC U+1F1E7 ("GB")

A pair of regional indicators is one grapheme cluster. An odd number of consecutive regional indicators is ambiguous and typically breaks after every pair.

Subnational flags (Scotland, Wales, England, Texas) use a different mechanism involving tag characters — U+E0000 through U+E007F — which we will revisit in Chapter 14. For now, observe the elegance: Unicode did not assign a code point per country flag. It defined a combinatorial rule, and the world's ~250 country flags fall out of it automatically.

The key takeaway

When you write length(s) or s.length or len(s), you have to ask two questions:

  1. What unit is this counting? Code points, code units, bytes, or grapheme clusters?
  2. Is that the unit I actually want?

For user-facing purposes — truncation, cursor movement, character count — the answer is almost always grapheme clusters. For wire-format purposes — buffer size, database column width — the answer is usually bytes (or code units, if your storage is 16-bit). For internal string manipulation where you need to reason about abstract text identity, code points are the right unit.

In the next chapter, we will tackle the follow-up problem: when the same cluster can be written two different ways (precomposed vs. decomposed), how do we reliably compare them?