Romanisation, script conversion and kana
Youzhi
Yu
University of
Chicago
Source: vignettes/transliteration.Rmd
transliteration.RmdFour conversions, four different levels of trust
This vignette covers the transforms that change what characters are there rather than how wide they are. Three of them share an implementation, ICU’s transliterators via stringi (Unicode Consortium 2024; Gagolewski 2022), and the jamo pair applies the Unicode arithmetic directly. They should not be trusted equally.
| Conversion | Reversible? | Uses context? | Trust |
|---|---|---|---|
cjk_jamo() / cjk_compose_jamo()
|
yes, exactly | no | complete |
to_hiragana() / to_katakana()
|
within one syllabary | no | high |
cjk_simplify() |
no | yes | good |
cjk_traditionalize() |
no | yes | good on characters, blind to regional vocabulary |
cjk_romanize() |
no | no | a key for Chinese; wrong for Japanese kanji |
The rest of this vignette is why.
Hangul jamo: exact
A modern Hangul syllable is built from a leading consonant, a vowel
and an optional trailing consonant, and Unicode encodes all 11,172
combinations precomposed. The relationship is arithmetic,
SIndex = (LIndex * 21 + VIndex) * 28 + TIndex, so the
decomposition is exact rather than tabulated: no table can be out of
date and no syllable is missed.
cjk_jamo("한글")
#> [[1]]
#> [1] "ᄒ" "ᅡ" "ᆫ" "ᄀ" "ᅳ" "ᆯ"cjk_jamo() returns a list, one vector of jamo per
string, so join each element before composing it back:
back <- vapply(cjk_jamo("한글"), paste, character(1), collapse = "")
cjk_compose_jamo(back)
#> [1] "한글"The round trip is exact, because both verbs apply the Hangul arithmetic and nothing else. Text that is not Hangul passes through untouched, which is the difference from Unicode normalisation: NFD would split the accented letter below, and NFD and NFC alike would replace the compatibility ideograph with its unified form, which in a Korean column means rewriting Hanja.
cjk_jamo("café")
#> [[1]]
#> [1] "c" "a" "f" "é"
cjk_compose_jamo("豈") == "豈"
#> [1] TRUEJamo are the right unit for questions the syllable hides: which
initial consonants a corpus favours, or whether two spellings differ
only in a final consonant. The same syllable counts three different
ways, each right for a different question. 한 is one
character to nchar(), two terminal columns to
cjk_width(), and three jamo here:
Kana: safe, with two things to know
Hiragana and katakana encode the same sounds, so the mapping never depends on the surrounding words.
to_hiragana("カタカナ")
#> [1] "かたかな"
to_katakana("ひらがな")
#> [1] "ヒラガナ"
to_hiragana(to_katakana("ひらがな"))
#> [1] "ひらがな"That round trip is exact because the input was in one syllabary. On mixed text it is not, and cannot be: converting to a single syllabary erases the distinction between the two, and that distinction carries meaning, because katakana marks loanwords, onomatopoeia and emphasis.
x <- "アか" # one katakana, one hiragana
to_hiragana(x) # both become hiragana
#> [1] "あか"
to_katakana(to_hiragana(x)) # and cannot be told apart again
#> [1] "アカ"The second thing is that ICU makes a few choices of its own. The
small katakana ヵ and ヶ become the full-size
hiragana か and け, and the digraph
ヿ (koto) is spelled out as two kana, so even text
that was all katakana does not always survive the trip:
to_hiragana("ヵヶ")
#> [1] "かけ"
to_katakana("ヿ")
#> [1] "コト"So the conversion is safe in the sense that matters, which is that it never guesses from context, but run it in one direction, as a normalisation, rather than expecting to undo it.
Kanji, Latin and punctuation are untouched, so it is safe on a mixed column:
to_katakana("日本語のtext です")
#> [1] "日本語ノtext デス"This is the normalisation you want before grouping Japanese text, where the same word is often written either way for emphasis.
Halfwidth katakana is handled too, and composed on the way through.
ｶﾞ is two code points, the kana and the halfwidth voiced
mark U+FF9E, and comes back as the single character ガ:
to_hiragana("ｶﾀｶﾅ")
#> [1] "かたかな"
to_katakana("ｶﾞ")
#> [1] "ガ"
nchar(to_katakana("ｶﾞ"))
#> [1] 1Simplified and traditional Han: right on characters, blind to vocabulary
cjk_simplify("漢字")
#> [1] "汉字"
cjk_traditionalize("汉字")
#> [1] "漢字"It is better than it looks. Simplified to traditional is genuinely
one-to-many (simplified 发 is 發, “to send”,
or 髮, “hair”, and 干 is 乾,
幹 or 干), and ICU resolves these from
context, not character by character:
cjk_traditionalize(c("头发", "发送")) # hair, send
#> [1] "頭髮" "發送"
cjk_traditionalize(c("干净", "树干")) # clean, tree trunk
#> [1] "乾淨" "樹幹"
cjk_traditionalize(c("后天", "皇后")) # after, empress: 后 splits both ways
#> [1] "後天" "皇后"All six are right, and so were the fourteen other words checked for
this release, including 面条/面对 and
里面/公里.
What ICU does not do is substitute regional vocabulary. The two standards differ in the words they use, not only in glyph shape, and that is lexical:
cjk_traditionalize(c("软件", "鼠标", "网络"))
#> [1] "軟件" "鼠標" "網絡"Those are the correct characters and the wrong words in Taiwan, where
the terms are 軟體, 滑鼠 and
網路. No character mapping can reach that, and it is what
OpenCC (Kuo 2024) and its regional configurations exist
for. Use these functions for script conversion; reach for OpenCC when
you need Taiwanese or Hong Kong idiom.
Romanisation: depends entirely on the language
cjk_romanize("中文")
#> [1] "zhōng wén"
cjk_romanize("中文", ascii = TRUE)
#> [1] "zhong wen"For Chinese this is pinyin, with tone marks unless you ask for ASCII.
ascii = TRUE gives a sortable, greppable key for a CJK
column, which is often the real reason to romanise at all.
cjk_romanize(c("こんにちは", "안녕하세요"))
#> [1] "kon'nichiha" "annyeonghaseyo"Three caveats, in increasing order of severity.
Han readings are per character. A character with several readings gets ICU’s preferred one regardless of the word it appears in, so common words come back wrong:
cjk_romanize(c("银行", "音乐")) # bank, music: yín háng and yīn yuè
#> [1] "yín xíng" "yīn lè"That is fine for a sort key and wrong for pinyin you show a reader.
No boundary between scripts. ICU spaces Han syllables but does not insert a break where the script changes:
cjk_romanize("漢字カナ")
#> [1] "hàn zìkana"Segment first with cjk_segment() if you need word
boundaries in the output.
Japanese kanji come back as Chinese. This is the
severe one. ICU routes every Han character through pinyin regardless of
the surrounding language, so 日本語 romanises to
rì běn yǔ rather than nihongo, and
私 to sī rather than watashi:
cjk_romanize("私は日本語を話します")
#> [1] "sīha rì běn yǔwo huàshimasu"That is not an approximation of the Japanese reading; it is the
Chinese one. The kana in the same string romanise correctly, which makes
the output look plausible at a glance and is exactly what makes it
dangerous. Note too that the particle は gives
ha, which is how it is written rather than how it is
said.
For Japanese, use a morphological analyser that carries readings:
gibasa binds MeCab and returns them (Kudo et al. 2004; Kato 2025).
cjk_romanize() is for Chinese, for kana, and for producing
an ASCII sort key.
Width forms are a separate concern
to_halfwidth() and to_fullwidth() change
width and nothing else. The usual advice for fullwidth text is an
NFKC pass, which does fix width, and also rewrites
ligatures, Roman numerals and circled numbers, none of which a change of
width asked for. When that wider fold is what you want, it is
cjk_normalize() in the next section rather than a side
effect here:
as.numeric(to_halfwidth("１２３"))
#> [1] 123
to_halfwidth("½ Ⅸ ①") # NFKC rewrites all three; this leaves them
#> [1] "½ Ⅸ ①"The one composition it does perform is voiced halfwidth katakana, where a single syllable is written as two code points. Composing them is the difference between text that matches a literal and text that silently does not:
x <- "ｶﾞ"
nchar(x)
#> [1] 2
nchar(to_halfwidth(x))
#> [1] 1
to_halfwidth(x) == "ガ"
#> [1] TRUENormalisation is the fourth axis
Width, script and syllabary are three axes; the code points
themselves are the fourth. cjk_normalize() applies the
Unicode normalisation forms (Whistler
2026), which is what decides whether two strings that render
identically compare equal:
both <- c("が", "が") # composed, then KA + voiced mark
nchar(both)
#> [1] 1 2
nchar(cjk_normalize(both))
#> [1] 1 1"nfkc" is the blunt instrument the width verbs exist to
avoid, and it is the right tool when matching rather than display is the
goal:
cjk_normalize("ＡＢ ①", form = "nfkc")
#> [1] "AB 1"
cjk_normalize("ＡＢ ①", form = "nfc")
#> [1] "ＡＢ ①"Canonical normalisation is not a no-op on Han.
Compatibility ideographs have singleton canonical mappings, so
"nfc" rewrites them just as "nfkc" does. This
is the part that surprises people who reach for NFC because it is
supposed to be the safe one:
block <- vapply(c(0xF900:0xFA6D, 0xFA70:0xFAD9), intToUtf8, character(1))
sum(cjk_normalize(block, "nfc") != block) # folded away by plain NFC
#> [1] 460
sum(cjk_normalize(block, "nfc") == block) # and the survivors
#> [1] 12The twelve survivors are unified ideographs that were encoded in the compatibility block by accident and have no mapping to apply. So the block is neither preserved nor uniformly folded: if the distinction between a compatibility ideograph and its unified form means something in your data, as it can in Korean and Japanese name records, keep the original column.
Variation selectors are the other thing that defeats an exact match, and four of the five forms leave them alone:
ivs <- "辻\U000E0100"
nchar(cjk_normalize(ivs))
#> [1] 2
nchar(cjk_normalize(ivs, drop_variation_selectors = TRUE))
#> [1] 1They are dropped before the form is applied, not after: a selector has combining class zero and blocks canonical composition across itself, so stripping one afterwards can leave text that is no longer in the form you just asked for.