Package index • tidycjk

Summarise a text column

Verbs taking (data, column) and returning a tibble, for a dplyr pipeline.

cjk_summary(): Summarise CJK content in a text column
cjk_char_counts(): Count the CJK characters in a text column
cjk_tokens(): One row per token

Segment

Splitting CJK text into words. Where a word ends is a fact about a language rather than about Unicode, so the engine is a required, pluggable choice. Dictionary-based segmentation for Chinese and Japanese ships via ICU; further engines can be registered.

cjk_segment(): Split CJK text into words
cjk_segmenters() register_cjk_segmenter(): Segmentation engines

Sentences and n-grams

Preprocessing that generic tooling gets wrong on CJK: sentence terminators are U+3002, U+FF01 and U+FF1F rather than “.!?”, and character n-grams are the dictionary-free baseline for Chinese.

cjk_sentences(): Split text into sentences
cjk_ngrams(): Character n-grams
cjk_strip_punct(): Remove punctuation from CJK text

Order

ICU collation. R’s sort() reads LC_COLLATE, so the order it gives Han depends on the machine it runs on; naming a collation makes it not.

cjk_sort() cjk_order(): Sort CJK text by pronunciation or stroke

Detect & classify

has_cjk(): Does the text contain CJK characters?
cjk_script(): Which CJK script dominates the text?
cjk_detect_language(): Which language is the text written in?
cjk_ratio(): What share of the text is CJK?

Display width

Terminal columns rather than characters, so CJK tables line up.

cjk_width(): Display width in terminal columns
cjk_pad(): Pad text to a display width
cjk_truncate(): Truncate text to a display width
cjk_wrap(): Wrap text to a display width

Normalise

Making strings that look the same compare the same. The width verbs move text along one axis and touch nothing else; cjk_normalize() is the Unicode forms, including the fact that plain NFC rewrites most of the CJK compatibility ideographs.

to_halfwidth() to_fullwidth(): Normalise fullwidth and halfwidth forms
cjk_normalize(): Apply Unicode normalisation

Transliterate

Romanisation, Han script conversion, the kana syllabaries and Hangul jamo. Each help page states how far it can be trusted: kana conversion is a normalisation, the jamo round trip is exact, Han conversion resolves ambiguity from context but not regional vocabulary, and romanisation reads each Han character alone, and as Chinese.

cjk_romanize(): Romanise CJK text
cjk_simplify() cjk_traditionalize(): Convert between simplified and traditional Han
to_hiragana() to_katakana(): Convert between hiragana and katakana
cjk_jamo() cjk_compose_jamo(): Decompose and recompose Hangul syllables

Reference data

cjk_blocks(): The Unicode blocks 'tidycjk' recognises

Package

tidycjk tidycjk-package: tidycjk: Tidy Tools for Chinese, Japanese and Korean Text