Overview
Almost every text tool in R assumes that a word is a run of
characters between two spaces. That assumption is invisible until it
fails, and it fails completely on Chinese, Japanese and Korean, because
Chinese and Japanese do not put spaces between words at all. Hand
我今天很開心 to a whitespace tokeniser and it returns one
token: the whole sentence. Split it per character instead and you get
six, which cuts 開心 (“happy”) in half and destroys the
very signal you were trying to measure.
The problem is not only segmentation. A CJK character occupies
two terminal columns but counts as one
character, so nchar() and the width of a printed table
disagree, and every console table containing CJK text comes out ragged.
Fullwidth digits １２３ look like digits and do not parse
as a number. A Japanese place name written only in Han characters cannot
be told from Chinese by script alone, and most libraries guess
anyway.
tidycjk addresses these as separate, honest problems, each built on the Unicode standard (Unicode Consortium 2026a, 2026b, 2026c) and on ICU, the Unicode Consortium’s own library, rather than on hand-written rules:
| Task | Function(s) |
|---|---|
| Detect |
has_cjk(), cjk_script(),
cjk_detect_language(), cjk_ratio()
|
| Summarise a column |
cjk_summary(), cjk_char_counts()
|
| Segment |
cjk_segment(), cjk_tokens(),
cjk_segmenters(),
register_cjk_segmenter()
|
| Sentences, n-grams, cleaning |
cjk_sentences(), cjk_ngrams(),
cjk_strip_punct()
|
| Order |
cjk_sort(), cjk_order()
|
| Measure width | cjk_width() |
| Lay out |
cjk_pad(), cjk_truncate(),
cjk_wrap()
|
| Normalise |
to_halfwidth(), to_fullwidth() (width),
cjk_normalize() (Unicode forms) |
| Romanise | cjk_romanize() |
| Convert Han |
cjk_simplify(), cjk_traditionalize()
|
| Convert kana |
to_hiragana(), to_katakana()
|
| Hangul jamo |
cjk_jamo(), cjk_compose_jamo()
|
| Reference data | cjk_blocks() |
Two layers, one contract
Most verbs work on a plain character vector, in the manner of
stringr. Three of them, cjk_summary(),
cjk_char_counts() and cjk_tokens(), take
(data, column) instead and return a tibble, so they drop
into a dplyr pipeline without an adaptor (Wickham 2014). Those three want a data frame;
they do not accept a bare character vector.
The contract is the same throughout, and it is worth stating once
because the rest of this vignette relies on it. Every vector-layer verb
is vectorised: NA in gives NA out, zero-length
input gives zero-length output, and nothing silently recycles a ragged
pair: a mismatch is an error rather than a warning and a short
answer.
The tidy verbs answer about the column rather than about each row, so
they summarise rather than map. cjk_char_counts() and
cjk_tokens() return no rows for an empty column;
cjk_summary() always returns exactly one, with
NA where there was nothing to average.
posts <- data.frame(
id = 1:5,
text = c(
"我今天很開心", # Chinese
"こんにちは、元気ですか", # Japanese, with kana
"안녕하세요", # Korean
"東京都", # Japanese written without kana
"no CJK here at all"
)
)
cjk_summary(posts, text)
#> # A tibble: 1 × 4
#> n_docs n_with_cjk prop_with_cjk mean_ratio
#> <int> <int> <dbl> <dbl>
#> 1 5 4 0.8 0.8prop_with_cjk answers “how many of these rows contain
any CJK at all”; mean_ratio answers “how CJK are they”.
They come apart exactly where you would want them to: a corpus of
English with one stray ideograph per row scores near 1 on the first and
near 0 on the second.
cjk_char_counts(posts, text)
#> # A tibble: 25 × 5
#> char codepoint script block n
#> <chr> <int> <chr> <chr> <int>
#> 1 我 25105 han CJK Unified Ideographs 1
#> 2 今 20170 han CJK Unified Ideographs 1
#> 3 天 22825 han CJK Unified Ideographs 1
#> 4 很 24456 han CJK Unified Ideographs 1
#> 5 開 38283 han CJK Unified Ideographs 1
#> 6 心 24515 han CJK Unified Ideographs 1
#> 7 こ 12371 hiragana Hiragana 1
#> 8 ん 12435 hiragana Hiragana 1
#> 9 に 12395 hiragana Hiragana 1
#> 10 ち 12385 hiragana Hiragana 1
#> # ℹ 15 more rowsDetection, and the value of refusing to guess
cjk_script() reports the dominant script;
cjk_detect_language() reports the language.
cjk_script(posts$text)
#> [1] "han" "hiragana" "hangul" "han" NA
cjk_detect_language(posts$text)
#> [1] NA "japanese" "korean" NA NARows one, four and five come back NA. Row five is
uninteresting: it has no CJK in it at all. Rows one and four are the
point. 東京都 is Tokyo Metropolis, unambiguously Japanese,
but it is written entirely in Han characters, and nothing in the
script distinguishes it from Chinese. Japanese written without
kana and Chinese are the same character set. Any library that answers
"chinese" there is guessing, and it will be wrong on
Japanese input in a way you cannot detect downstream, because the output
looks exactly like a confident correct answer.
tidycjk returns NA. If your corpus is known
to be Chinese, ask for the guess explicitly, so that the assumption
lives in your script where a reader can see it rather than in a library
default where they cannot:
cjk_detect_language(posts$text, han_only = "chinese")
#> [1] "chinese" "japanese" "korean" "chinese" NASegmentation
Where a word ends is a fact about a language, not about Unicode, so
it needs a dictionary. tidycjk dispatches on an engine name
rather than committing to one, and two engines ship with the
package.
"icu" is a real word segmenter, using the
dictionary-based break iterators that ICU carries (Unicode Consortium 2024) and
stringi exposes (Gagolewski
2022):
cjk_segment("我今天很開心", engine = "icu")
#> [[1]]
#> [1] "我" "今天" "很" "開心"我 / 今天 / 很 / 開心: words, not characters.
"character" is the dictionary-free baseline, and the
contrast shows what a segmenter is for:
cjk_segment("我今天很開心", engine = "character")
#> [[1]]
#> [1] "我" "今" "天" "很" "開" "心"In a pipeline, cjk_tokens() is the CJK-aware counterpart
of tidytext’s unnest_tokens() (Silge and Robinson 2016):
cjk_tokens(posts[1:2, ], text, engine = "icu")
#> # A tibble: 8 × 3
#> id text token
#> <int> <chr> <chr>
#> 1 1 我今天很開心 我
#> 2 1 我今天很開心 今天
#> 3 1 我今天很開心 很
#> 4 1 我今天很開心 開心
#> 5 2 こんにちは、元気ですか こんにちは
#> 6 2 こんにちは、元気ですか 元気
#> 7 2 こんにちは、元気ですか です
#> 8 2 こんにちは、元気ですか かvignette("segmentation", package = "tidycjk") covers the
engines, their limits, and how to register gibasa (Kato 2025) or jiebaR.
Sentences and n-grams
Two preprocessing steps that generic tooling gets wrong. A regular
expression on [.!?] finds no sentence boundary at all in
Chinese, because the terminator is 。:
cjk_sentences("我今天很開心。你呢？天氣很好！")
#> [[1]]
#> [1] "我今天很開心。" "你呢？" "天氣很好！"Character n-grams are the dictionary-free baseline for Chinese retrieval and classification. Most words are one or two characters, so bigrams capture the majority of them with no model at all, and they hold up on the names and coinages where a segmenter is least reliable:
cjk_ngrams("中文很好")
#> [[1]]
#> [1] "中文" "文很" "很好"No gram is formed across whitespace, so nothing straddles two words
of a Latin run. Punctuation is a different matter, and
cjk_strip_punct() is the step that deals with it. It
replaces each mark with a space rather than deleting it, so a bigram
cannot span the full stop between two sentences:
cjk_strip_punct("他說（今天）。真好")
#> [1] "他說 今天 真好"
cjk_ngrams(cjk_strip_punct("好。天")) # nothing spans the stop
#> [[1]]
#> character(0)
cjk_ngrams(cjk_strip_punct("好。天", "")) # deleting it invents 好天
#> [[1]]
#> [1] "好天"gsub("[[:punct:]]", "", x) is the spelling to avoid: R’s
default engine resolves the class through the C library, so it removes
nothing under LC_ALL=C, and perl = TRUE
removes nothing from CJK in any locale.
Sorting
sort() reads LC_COLLATE, so the order it
gives Han text is a property of the session rather than of the data.
These three surnames come out in code point order under C
and en_US.UTF-8, in pinyin order under
zh_CN.utf8, and in a third order under
ja_JP.utf8. Each is defensible; none is reproducible, and
that is the trap: the same script sorts a name column differently on a
colleague’s machine and says nothing about it.
The fallback order is worth knowing, because it is not arbitrary. Within the base block the unified ideographs are arranged in KangXi radical-stroke order, so code point order really is radical order. It is simply not phonetic, and it stops being radical order across blocks, since every Extension A character sorts before every base-block one, leaving a rare character nowhere near its radical-mates.
Naming the collation is what makes the answer the same everywhere:
Width, and why your tables are ragged
nchar() counts characters. A terminal counts columns,
and Unicode Annex #11 assigns CJK characters two of them (Unicode Consortium 2026a).
Pad to a width rather than to a character count and the column lines up:
Truncation and wrapping follow the same rule. Wrapping additionally respects the Unicode line breaking algorithm (Unicode Consortium 2026b), so it will not put a break before a closing bracket or a full stop:
vignette("width-and-layout", package = "tidycjk") goes
further.
Normalisation and transliteration
to_halfwidth() narrows fullwidth forms and does
only that, unlike an NFKC pass, which also
rewrites ligatures, Roman numerals and circled numbers:
as.numeric(to_halfwidth("１２３"))
#> [1] 123
to_halfwidth("½ Ⅸ ①") # NFKC would rewrite all three
#> [1] "½ Ⅸ ①"When the blunt instrument is the right one, for matching
rather than display, cjk_normalize() is it, and it also
answers the case a reader may not expect: plain "nfc"
rewrites most of the CJK compatibility ideographs, because they have
singleton canonical mappings.
ga <- c("\u304c", "\u304b\u3099") # composed, then KA + voiced mark
nchar(ga)
#> [1] 1 2
nchar(cjk_normalize(ga))
#> [1] 1 1
cjk_normalize("ＡＢ ①", form = "nfkc")
#> [1] "AB 1"Romanisation, Han script conversion, kana conversion and Hangul jamo decomposition are all available on the same contract:
cjk_romanize("中文")
#> [1] "zhōng wén"
cjk_simplify("漢字")
#> [1] "汉字"
to_hiragana("カタカナ")
#> [1] "かたかな"
cjk_jamo("한")
#> [[1]]
#> [1] "ᄒ" "ᅡ" "ᆫ"Each has limits worth knowing before you trust it. Romanisation reads
each Han character on its own, so a word like 银行 (“bank”)
gets the wrong reading; simplified/traditional conversion resolves
characters from context but does not substitute regional vocabulary.
vignette("transliteration", package = "tidycjk") covers
each.
What tidycjk is built on
cjk_blocks() exports the block table the whole package
rests on, so the definition of “CJK” can be read rather than guessed
at:
head(cjk_blocks(), 8)
#> # A tibble: 8 × 5
#> block script start end n_codepoints
#> <chr> <chr> <int> <int> <int>
#> 1 Hangul Jamo hangul 4352 4607 256
#> 2 CJK Symbols and Punctuation punctuation 12288 12351 64
#> 3 Hiragana hiragana 12352 12447 96
#> 4 Katakana katakana 12448 12543 96
#> 5 Bopomofo bopomofo 12544 12591 48
#> 6 Hangul Compatibility Jamo hangul 12592 12687 96
#> 7 Kanbun kanbun 12688 12703 16
#> 8 Bopomofo Extended bopomofo 12704 12735 32
nrow(cjk_blocks())
#> [1] 35Nothing here makes a network request, and the package needs no compiled code of its own.