What are the emoji in this text costing a tokeniser?

emoji_token_cost() measures the size of the emoji in each row: bytes, code points, grapheme clusters, and an estimate of the tokens they will cost a byte-level tokeniser. Emoji are several times more expensive than their visual weight suggests – a single ZWJ family emoji can run to well over a dozen tokens – which makes them a real line item in a prompt budget.

Usage

emoji_token_cost(data, text, tokenizer = NULL)

Arguments

data

A data frame or tibble containing a text column. Grouped data frames are accepted. The verbs that work a row at a time (adding columns, or keeping and expanding rows) carry the grouping through to their result, as dplyr::mutate() and dplyr::filter() do. The verbs that pool across rows – the counts, the co-occurrence edge lists, the time series – warn that they ignore the grouping and return one corpus-wide answer.

text

The text column to scan, supplied unquoted. Any atomic column is accepted and read as character, so a factor works and a numeric, Date or logical one simply contains no emoji. A list column – or a data-frame column – is refused rather than coerced, because coercing one deparses it and the emoji found would be in the code rather than in your data. What counts as an emoji is the same in every verb; see the Detection section of tidyEmoji for the one case that surprises people, code points that are emoji only when they carry U+FE0F.

tokenizer

Optional function taking a character vector and returning either token counts (a numeric vector of the same length) or a list of token vectors. It is called on the row's emoji, concatenated. NULL (default) uses the byte heuristic.

What it returns is checked, because a wrong answer here is silent otherwise. A count is rounded up, and it has to be finite, not negative and within integer range; NA is accepted and passed through, for a tokeniser that cannot answer for a row. A data frame is refused rather than read, since lengths() on one counts its columns rather than its tokens.

Value

data, as a tibble, with added columns .emoji_n, .emoji_bytes, .emoji_codepoints, .emoji_graphemes and .emoji_token_estimate.

Details

Bytes, code points and graphemes are exact and tidyEmoji can be authoritative about them. The token count cannot be: it depends on the tokeniser. Without tokenizer, .emoji_token_estimate is a deliberately crude heuristic of roughly two UTF-8 bytes per token, which is in the right range for byte-level BPE vocabularies but is an estimate and should never be quoted as a bill. Pass your real tokeniser through tokenizer when the number matters.

.emoji_graphemes is the number of emoji occurrences, since the package's detection is grapheme-aware: a skin-toned family emoji is one grapheme and many code points, which is precisely the gap that makes emoji expensive.

See also

emoji_sanitize() for acting on the answer; emoji_ratio() for the share of the text that is emoji; vignette("reversible-preprocessing", package = "tidyEmoji") for the before-and-after token budget on a real corpus.

Examples

family <- paste0("\U0001F468\u200d\U0001F469\u200d",
                  "\U0001F467\u200d\U0001F466")
df <- data.frame(text = c("hi \U0001f600", family, "plain"))
emoji_token_cost(df, text)
#> # A tibble: 3 × 6
#>   text  .emoji_n .emoji_bytes .emoji_codepoints .emoji_graphemes
#>   <chr>    <int>        <int>             <int>            <int>
#> 1 hi 😀        1            4                 1                1
#> 2 👨‍👩‍👧‍👦           1           25                 7                1
#> 3 plain        0            0                 0                0
#> # ℹ 1 more variable: .emoji_token_estimate <int>