Skip to contents

tidyEmoji: Discover, Count, Categorise, Score, Translate and Relate Emoji in Text

Source: R/tidyEmoji.R
tidyEmoji-package.Rd

A tidy toolkit for working with the emoji in any text column, such as social-media posts, product reviews, chat logs or survey responses. Unicode is awkward to handle and not every code point is an emoji, which makes emoji statistics fiddly to obtain. 'tidyEmoji' extracts, counts, categorises, sentiment-scores and emotion-scores emoji, converts them to and from text (for accessibility and NLP preprocessing), searches the emoji catalogue, maps emoji co-occurrence and sequences (graph-ready edge lists and n-grams), measures where and how densely emoji are used, and builds document-by-emoji feature tables for machine learning, with grapheme-aware detection (so skin-tone and multi-person sequences stay intact), returning tidy data frames that slot straight into a 'tidyverse' workflow. It also quantifies how much annotators disagreed about an emoji (interpretation risk), extracts the words around each emoji, tracks emoji use over time, measures text-emoji sentiment mismatch, and applies explicit emoji-preprocessing policies for language-model pipelines. The bundled emoji sentiment lexicon is from the Emoji Sentiment Ranking of Kralj Novak et al. (2015) doi:10.1371/journal.pone.0144296 , released under CC BY-SA 4.0; the emotion lexicon is from EmoTag1200 of Shoeb & de Melo (2020) https://aclanthology.org/2020.emnlp-main.720/, released under the MIT licence.

Output and naming contract

Every verb follows verb(data, text, ...), takes the text column unquoted, and returns a tibble. Output column names come in three shapes, and which one you get tells you what the column is:

  • .emoji_* – a measurement of your text, added to your data (.emoji, .emoji_name, .emoji_category, .emoji_sentiment, .emoji_n, ...). Dotted so it will not collide with your own columns.

  • .row_number, .position, .period, .period_prev, .period_label – structural indices saying where a row came from rather than what was measured: the position of the entry in data (emoji_extract_unnest(), emoji_context(), emoji_ngrams(), emoji_dfm()), where in that entry something sits, or the time bucket (emoji_trend(), emoji_turnover(), emoji_seasonality()). Dotted for the same reason, and reserved on the same terms. That is the whole list.

    .position is the one of the five whose unit depends on the verb, so it is worth reading before you index with it. In emoji_context() it is a code-point offset into the text, the unit substr() takes, so substr(text, .position, .position + nchar(.emoji) - 1) returns the glyph. In emoji_ngrams() it is the index within the row's emoji sequence, so the first n-gram of a row is 1 whatever the text looks like. Both pages say which, and the two are not interchangeable.

  • bare names – the columns of a new summary tibble, which is not your data with something added (emoji_frequency()'s emoji, name, n; emoji_ambiguity()'s ambiguity, rank). emoji_dfm() is the one verb whose column names are data: one per emoji, named with the glyph itself.

Every dotted name is reserved: a verb overwrites any column of its own output name that is already there, without warning. That is what makes verbs chainable and re-runnable – emoji_sentiment() then emoji_position() both write .emoji_n, and both mean the same thing – but it also means a column of your own called .emoji_n will be replaced, and that includes the text column itself if you named it .emoji_n. Rename it first if you need to keep it.

Two of the shared dotted names do not mean the same thing in every verb that writes them, so chaining those verbs replaces a number with a different one rather than with the same one:

So emoji_sentiment() followed by emoji_emotion() leaves a .emoji_n_scored describing the emotion lexicon beside a .emoji_sentiment that does not. Rename the first result's column before adding the second, or keep the two tables apart.

group always refers to the Unicode top-level category (the term used by the underlying emoji::emojis table). Every glyph-to-metadata join is normalised through a codepoint key that strips the U+FE0F variation selector, so qualified and unqualified emoji forms resolve identically in every verb.

Detection

Detection is grapheme-aware: a skin-tone modifier or a zero-width-joiner sequence (a family, a couple, a profession) stays intact as one emoji, and every verb asks the same question, so counts agree across the package.

There is one systematic exclusion, and it is worth knowing before you read a count. Some code points are emoji only in their emoji-presentation form, that is only when the variation selector U+FE0F is present. The best-known is the heart: U+2764 U+FE0F is detected, the bare U+2764 is not, and several keyboards emit the bare form. Across the reference catalogue 1252 emoji carry U+FE0F, and 216 of those become undetectable if it is dropped – in the bundled sentiment lexicon, 57 of the scorable glyphs. Counted the other way round, 212 of the catalogue's 5042 rows are spellings that are themselves undetectable; the two figures measure different things and both are right.

The selector does not always go at the end. For 200 of those 212 it does, so appending U+FE0F is what makes them detectable. The exceptions are the 12 keycap sequences – #, * and 0 to 9 followed by the enclosing keycap mark U+20E3 – where the selector belongs between the two: U+0031 U+FE0F U+20E3 is detected and U+0031 U+20E3 U+FE0F is not. Inserting U+FE0F after the first code point is the rule that repairs all 212.

The default does not match the bare forms, and that is deliberate rather than an oversight: the same set contains U+00A9, U+00AE and U+2122, so matching them unqualified would count the copyright sign in a legal footer as emoji use. Detection is the only thing affected – the join is not. Every glyph-to-metadata lookup strips U+FE0F first, so if you hand a bare U+2764 to as_emoji_name(), emoji_sentiment()'s lexicon or emoji_ambiguity(), it resolves exactly like the qualified form.

Joined sequences are unaffected either way. Unicode lists several spellings of a zero-width-joiner sequence – fully qualified, and shorter forms with the selectors omitted – and a shorter one can leave an undetectable component in the middle. Detection repairs those: every canonical spelling in the reference table, and all but two of the shorter ones, is read as exactly one emoji, so U+2764 U+200D U+1F525 is "heart on fire" rather than "fire" even with its selectors stripped. The two exceptions are spellings in which no component at all is detectable, and both have a canonical form that is found.

Everything above is about what detection misses. It also admits two things that are well formed but not emoji, and both flow through every verb, so a corpus statistic can be inflated by them:

  • An invalid regional-indicator pair. Any two regional indicators form one grapheme cluster, so U+1F1FD U+1F1FD is read as a single emoji even though no country has that code. It appears in emoji_frequency() with name = NA, gets a column in emoji_dfm() and a node in emoji_pairs(). Only 262 of the pairs are real: subgroup is "country-flag" for 259 rows of the reference table and "subdivision-flag" for 3, so you can filter against that set – emoji_frequency() carries group, and emoji_provenance() reports which catalogue you have.

  • An orphan skin-tone modifier or hair component. A modifier applied to a base that cannot take one, as in U+1F600 U+1F3FB, leaves the swatch standing alone – and because the Component group is in the reference table it comes back named, as "light skin tone" in group "Component", not as NA. as_emoji_type() labels these "component", which is the way to find and drop them: subset(emoji_frequency(df, text), as_emoji_type(emoji) != "component").

Both are defensible as raw detection and misleading as a corpus statistic, which is why they are named here rather than silently filtered: dropping them inside the verbs would make the emoji counts disagree with the text.

Which spelling comes back

Two spellings of one emoji, differing only by U+FE0F, are one emoji to every lookup: the name, the score, the category and the type all resolve through a key that strips the selector. They are not always one row. A verb that reports a glyph either hands back the spelling it found or collapses both onto the catalogue's, and which it does follows from what the verb is for:

A corpus holding both spellings shows the difference in one line: emoji_frequency() returns two rows of n = 1 where emoji_cooccurrence() returns one node of n = 2. Nothing is lost either way, and the two sides line up on the name, which is the same for both spellings: emoji_frequency() already carries it, and as_emoji_name() supplies it for a glyph column of your own.

This is only ever about spellings of the same emoji. Skin tones, genders and the members of a ZWJ sequence are different emoji and stay apart in every verb.

Grouped data frames

Grouping is respected where it can be, and reported where it cannot. The verbs that work a row at a time – the ones that add .emoji_* columns, and the ones that keep or expand rows – carry the input's grouping through to their result, exactly as dplyr::mutate() and dplyr::filter() do, so a group_by() upstream still means something to a summarise() downstream. The verbs that pool across rows – emoji_frequency(), emoji_dfm(), emoji_pairs(), the time series, and the other corpus-level summaries – cannot honour groups yet: they warn and return a single corpus-wide answer. Splitting the data yourself, or passing a doc_id where the verb offers one, is the way to get per-group results today.

See also

Useful links:

Author

Maintainer: Youzhi Yu yuyouzhi666@icloud.com