“Heap, Heap, Array!”

Allonsay’s language classification algorithm

How you can detect the language of texts, without using the “ML” word.

“You can’t name this thing Allonsay and not include support for French”

macOS comes with a say command that reads texts out loud using the system’s default voice. Unfortunately, voices are monolingual so if you often consume content in different languages you’re out of luck. Allonsay is a tiny command-line application that detects which language (and thus voice) you need. How does it work?

In an ideal sitation there’s no need to guess which language a text is written in. Web pages (like this one) often include metadata that tell you which language is used on the page. Some websites will also tell you when they use foreign words, so your screen reader knows how to pronounce them correctly.

Such information is rarely available in practice though, so we need a way to detect what language a text is written in.

Sometimes we’re lucky, and we only need to identify a single language that has its own unique writing systems (e.g. Thai and Hebrew). Text written in these languages will use Unicode code points that you won’t find in any other language. In all other cases we need a more sophisticated solution…

The easiest and most intuitive solution by far is to simply look up words of a text in dictionaries for various languages until you’ve found enough matches in one of the dictionaries. But this solution doesn’t scale very well. Any tool that needs to identify languages now needs to ship with dictionaries for every supported language. This can be expensive in terms of disk or bandwidth usage and (possibly) licensing costs, so we need to look for a solution that works without dictionaries!

Letter frequencies

Fortunately we don’t have to look very far, because the characters that occur in a text can also tell us a lot about its language!

This might sound strange at first, because for western languages most of these characters will be in the a–z range. However, the frequency at which characters occur still differs between languages.

The distributions are largely similar – both are part of the West Germanic language group after all – but there are some clear differences too. For example, the character e occurs a lot more in Dutch than in English, while y is basically non-existent in Dutch.

We can use this to guess the language of a text. The overall idea is pretty simple. First, we determine the distribution of characters in a text and then check which language has the most similar distribution.

Example

Let’s say we have the following input text:

People assume that time is a strict progression of cause to effect, but, actually, from a non-linear, non-subjective viewpoint, it’s more like a big ball of wibbly-wobbly… timey-wimey… stuff

This text has the following character distribution:

We can compare the distribution of the input text with the overall distributions in the English and Dutch languages:

The last row shows the sum of the percentage point differences between the input text and the two languages. Our input text is more similar to English (36.782) than to Dutch (63.604), so we can assume that it’s written in English!

Summary