tidycjk: Tidy Tools for Chinese, Japanese and Korean Text
Source:R/tidycjk-package.R
tidycjk-package.RdA tidy toolkit for text that is written in Chinese, Japanese or Korean. Most text tooling in R assumes that words are separated by whitespace, which CJK writing does not use, so ordinary summaries of a text column either treat a sentence as one undifferentiated blob or split it into isolated characters. Word segmentation is therefore a pluggable engine that the caller names explicitly rather than a bundled dictionary, because where a word ends is a fact about a language and not about Unicode. 'tidycjk' classifies characters by Unicode block, reports which script and which language a text is written in, measures how much of a text is CJK, and turns those measurements into tibbles that slot straight into a 'tidyverse' workflow. It also measures display width in terminal columns, pads and truncates to a width rather than to a character count, and normalises fullwidth and halfwidth forms surgically – including composing halfwidth katakana voiced marks into single code points – without the collateral damage of a full 'NFKC' pass. Language detection deliberately returns NA rather than guessing when a text is written in Han characters only, because Japanese written without kana cannot be distinguished from Chinese by script alone. Everything is derived from the Unicode specification; the package makes no network requests and needs no compiled code of its own.
Output and naming contract
The package has two layers. The vector layer takes an atomic character
vector and returns an atomic vector of the same length, in the manner of
stringr: has_cjk(), cjk_script(), cjk_detect_language(),
cjk_width(), cjk_pad(), cjk_truncate(), cjk_ratio(),
to_halfwidth() and to_fullwidth(). cjk_segment() belongs to the same
layer but returns a list, because the number of tokens per string varies.
The tidy layer takes verb(data, col, ...) with the column unquoted and
returns a tibble: cjk_summary(), cjk_char_counts() and cjk_tokens().
Every vector-layer function is vectorised, propagates NA element-wise,
and returns a zero-length vector of the right type for zero-length input.
What counts as CJK
A character is CJK when its code point falls in one of the Unicode blocks
listed by cjk_blocks(). That set is deliberately wide: it includes the
ideographs and the three phonetic scripts, but also CJK punctuation and the
halfwidth and fullwidth forms, because a text column that has been through a
CJK input method carries those too. cjk_script() tells you which of them
you actually have, so the wide definition never hides the detail.
Related packages
tidycjk deliberately stops where another package already does the job:
Chinese word segmentation – jiebaR, which
cjk_segmenters()shows how to register as an engine. It was archived from CRAN on 2025-05-01, which is why it is not a dependency.Romanisation – pinyin and hanyupinyin.
Traditional/simplified conversion – tmcn at the character level. Character-level conversion is context-blind and often wrong, so this package ships none; OpenCC is the phrase-level answer outside R.
Tokenising whitespace-delimited text – tidytext, whose
unnest_tokens()cjk_tokens()mirrors for text that has no spaces between words.
Relationship to stringi
tidycjk does not re-implement Unicode. Display width comes from
stringi::stri_width() and padding from stringi::stri_pad() –
stringi – both of which
read the live Unicode tables in ICU, the
Unicode Consortium's C library. tidycjk adds the CJK-specific
layer on top and keeps the naming consistent with the rest of the package.
If all you need is the width of a string, call stringi directly.