Wolfram Cloud Document

Comparing translated texts to quantify language efficiency
by Wolfram Emerging Leaders Program

Abstract

The aim of this project was to create methods to quantify the information density of text, as well as to analyze what aspects of specific texts and languages affect information density.

In this project, we created three unique algorithms to measure the information density of text, which we refer to as “ShannonDensity,” “TranslationRandForest,” and “TranslationWordVectors.” If you’d like to skip ahead to seeing our results, check out the cloud deployment at https ://www .wolframcloud .com /obj /henryoliver2005 /language-efficiency-deploy, where you can experiment with these algorithms yourself.

Introduction

Despite decades of innovation to increase information transmission rates, language remains as one of the primary vectors through which information is relayed into our consciousness. As a result, the rate at which information is relayed to us is set not by gigabyte per second internet bandwidth speeds but by the contents of texts and the speed with which we can process them. By analyzing and comparing data across different languages and genres, we aimed to develop models to approximate the "efficiency" of a text, or the "information density."

We defined information density broadly as the average amount of "meaning" per unit of text, such as per character or per syllable. It is a necessarily vague definition, because no perfect algorithm for calculating information density exists, and the preferred methodology may differ depending on the algorithm's intended use.

While the topic of information efficiency is interesting from a theoretical standpoint, there are also many practical reasons to want to be able to approximate the information density of text. Some such situations could include, but are not limited to:

◼

determining whether information is lost when a book is translated into other language

◼

computationally assessing the reading level of books or articles

◼

analyzing how writing style changes across different genres, languages, and time periods

◼

arriving at a better understanding of how we communicate

There already exists a considerable amount of research into using computational methods to approximate the information density of text [1] [2]. With that in mind, we wanted to avoid creating an algorithm that would be simply a less-sophisticated version of existing ideas. Our solution was that, rather than using an existing method to directly calculate information density, we would do so indirectly, using tools provided by the Wolfram Data and Function Repositories. Using this method, we were able to create accurate and effective algorithms for approximating information density.

In brief, our approach was to take a large body of sentences that had been translated into multiple languages. Ideally, each sentence’s translations would retain the same “ideas,” or semantic content. Thus, after accounting for certain differences between languages, we would be able to see if a sentence was particularly “efficient,” or information-dense, by verifying wether its translations into other languages contained more syllables. In this way, we were able to assign an approximate information-density score to a large body of sentences, which we used as the training data for two different machine learning models. To test the accuracy of our two algorithms, we also created a third algorithm, which relied upon more traditional metrics for evaluating information density.

Data Sources

In order to ensure the consistency of semantic content, we needed a standardized dataset of human-translated text. Since each translation conveyed the same amount of content, any statistically significant differences between the texts could be reasonably attributed to the nature of the language.

Europarl

Our primary dataset contained transcripts of the European Parliament compiled by Phillipp Koehn from the University of Edinburgh. Mainly used to train machine translation models, the Europarl dataset contains spoken, non-fiction content translated into 21 languages by humans. We preferred to use this data set in particular because compared to other potential data sources, the translations replicated the grammatical level of speech much less. The translations conveyed the same meaning without simply translating word-for-word. This was important for us because we wanted to allow factors such as word choice and sentence structure to influence information density, rather than it being solely determined by the lengths of specific words.

Since the translation units were stored as pair-wise TMX files, we needed to extract all the sentences for each language and line them up such that each row contained the translation for the same amount of information content. Using English as the reference point, we merged the TMX files containing Spanish, Italian and French translations from English.

Farkin Translations

For our fiction dataset, we used translations of out-of-copyright literary works, including Alice in Wonderland, House of Usher, Jane Eyre, and Robinson Crusoe, compiled by Andras Farkas . Each book was aligned to create parallel texts where each row corresponded to the same sentence in the different languages.

First, we split the text file of each book by line to separate the list of sentences. To exclude chapter headings and author information, we deleted any lines with fewer than 100 characters.

rows=Select[Drop[StringSplit[data,"\n"],3],StringLength[#]>100&]

Spiting each row by tabs, we got a list of the sentences translated into different languages. Some lines didn't have translations in all the languages so we deleted them.

sentences=Select[StringSplit[rows,"\t"],Length[#]==9&]

Counting Syllables

WordData

While designing our algorithms, we decided that it generally made the most sense to measure efficiency in terms of the amount of information contained in each syllable. To do that, we needed to actually be able to count the number of syllables in a word. For English, Wolfram Language already has a function that will split a word into syllables.

Wolfram Language's WordData function splitting word by syllables

In[]:=

WordData["computational","Hyphenation"]

Out[]=

{com,pu,ta,tion,al}

Using Length to get number of syllables:

Length[%]

Out[]=

There are some words that WordData does not recognize and cannot split into syllables. For those words, we were able to convert the word into its phonetic form. The phonetic form of a word is a sequence of sounds that constitute the word. Each sound is from the International Phonetic Alphabet (IPA). We could then simply count the vowel sounds in that phonetic form.

Getting the phonetic form of a word with WordData when hyphenation is not available

WordData["adjourned","Hyphenation"]WordData["adjourned","PhoneticForm"]

Out[]=

Missing[NotAvailable]

Out[]=

ədʒˈɝnd

With a list of the IPA vowels in English, we were able to count the number of vowel sounds to get an estimate of the syllable count.

ipaVowels = "ɛ", "ɪ", "æ", "ʌ", "ɒ", "ʊ", "ə", "ˈi", "ˈɒ", "ˈɔ", "ˈɝ", "ˈu", "ˈeɪ", "ˈoʊ", "ˈaʊ", "ˈɪ", "ˈɛ", "ˈaɪ", "ˈʊ";StringCount[WordData["adjourned", "PhoneticForm"], ipaVowels]

Out[]=

Our Own Function

By combining these two methods, we created a pretty accurate function to count the number of syllables in a word in English. However, WordData does not work with any other languages, so for our final product, we needed to create a function to count syllables ourselves. It was important to us that the function work fairly similarly between different languages, so that any errors (such as consistently over- or under-counting) would not affect each language differently. For our function, we counted each of the following things:

◼

The number of individual vowels in the word

◼

The number of diphthongs (combinations of two vowels to make one sound)

◼

Depending on the language, we also counted triphthongs (three vowels making one sound) if they contained two diphthongs

◼

If a triphthong contained only one diphthong, we ignored it because it was already counted correctly

◼

If the word had an ‘e’ at the end (if the word was in a language that has silent ‘e’s)

Finally, we just returned Vowels - Diphthongs - Triphthongs - silentE as the number of syllables. Each language has different vowel characters, different diphthongs, etc, so we created a function that took a language as a parameter. One problem we encountered was that while an 'e' at the end of a word is usually silent in languages like English, in words with no other vowels, it is not (ex. "we", "be", "the", etc). To fix this issue, we simply checked if a word ending in 'e' would be given a value of zero syllables, and if so, increased it to 1.

Our function to manually count syllables

In[]:=

CountSyllables[word_String,language_String]:=Module[{DiphthongVowels,TriphthongVowels,LanguageVowels,Vowels,Diphthongs,Triphthongs,silentE},DiphthongVowels=Which[language=="Spanish",{"ai","au","ay","ue","ey","oa","oe","oi","oy","ua","ui","uy"},language=="English",{"ai","au","ay","ea","ee","ei","eu","ey","ie","io","oa","oe","oi","oo","ou","oy","ua","ui","uy"},language=="French",{"ai","ay","au","eil","eu","ie","ee","ille","oi","ou","oo","ui","ue","io"},language=="Italian",{}];LanguageVowels=Which[language=="Spanish",{"a","e","i","o","u","y","á","é","í","ó","ú"},language=="English",{"a","e","i","o","u","y"},language=="French",{"a","e","i","o","u","y","à","â","æ","é","è","ê","ë","î","ï","ô","œ","ù","û","ü","ÿ"},language=="Italian",{"a","e","i","o","u","y","á","é","í","ó","ú","è"}];TriphthongVowels=Which[language=="French",{"eau","oeu","uoi"},language=="Italian",{"iai","iei","uoi"},language=="English",{},language=="Spanish",{}];Vowels=Total[Table[StringCount[word,vowel],{vowel,LanguageVowels}]];Diphthongs=Total[Table[StringCount[word,dip],{dip,DiphthongVowels}]];Triphthongs=Total[Table[StringCount[word,trip],{trip,TriphthongVowels}]];silentE=If[language=="French"||language=="English",Boole[(StringTake[word,{StringLength[word]}]"e")&&(StringLength[word]>2&&!MemberQ[word,StringTake[word,{StringLength[word]-1}]])],0];Vowels-Diphthongs-Triphthongs-silentE]

We compared our function with WordData to make sure that it was sufficiently accurate.

Testing our function against WordData on 100 random words

In[]:=

Length[WordData[#,"Hyphenation"]]-CountSyllables[#,"English"]&/@RandomWord[100]

Out[]=

{0,-3,0,0,0,1,0,0,0,-1,0,1,0,0,0,0,-3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,-2,0,0,0,0,0,0,-1,0,0,0,-2,0,0,0,0,0,0,0,-3,0,-1,-2,0,1,-2,-1,0,0,0,0,0,0,0,1,0,-1,0,1,0,-1,0,-2,0,0,0,0,0,0,0,0,-4,0,-1,0,0,-1,0,0,0,0,-1,0,0,0,0,0}

While it was occasionally off by a syllable, as shown above, we considered it accurate enough. Most importantly, our function is as consistent as possible across languages.

Relative Overall Efficiency Rates of Languages

Before we moved on to measuring the efficiency of text, we realized we could use our data to provide an estimate of which languages are most efficient overall. Since we had a few data sources with the same information written in four languages, we could use our CountSyllables function to count how many syllables each language takes to convey the same meaning. The ratio of the values gave us an estimate of which languages are generally more and less efficient, in terms of information per syllable.

Separating our data from the European Parliament into its languages

englishData=#[[1]]&/@europarlData;

frenchData=#[[2]]&/@europarlData;

spanishData=#[[3]]&/@europarlData;

italianData=#[[4]]&/@europarlData;

Counting the syllables in each language

CountSyllables[englishData,"English"]CountSyllables[spanishData,"Spanish"]CountSyllables[italianData,"Italian"]CountSyllables[frenchData,"French"]

Out[]=

3463738

Out[]=

4610070

Out[]=

4842070

Out[]=

3963251

Normalizing our data to understand it better (lower numbers correspond to lower syllable counts, meaning more efficient)

N/@Normalize[{3463738,4610070,4842070,3963251}]

Out[]=

{0.407067,0.541787,0.569052,0.465771}

We found that our languages in order of efficiency are English, French, Spanish, then Italian. This roughly matched data that we found in other sources. It was interesting to compare these numbers to metrics like the number of characters.

N/@Normalize[StringLength/@{englishData,spanishData,italianData,frenchData}]

Out[]=

{0.469634,0.505195,0.505492,0.518362}

While English and French were the most efficient in terms of syllables, they both increased their normalized value when we measured in terms of characters.

Measuring in terms of sentences

N/@Normalize[Length[TextSentences[#]]&/@{englishData,spanishData,italianData,frenchData}]

Interestingly, these values were all incredibly similar, likely because the translation was mostly done one sentence at a time. We decided that, for the rest of the project, we would measure efficiency in terms of the number of syllables because it seemed to be a good balance and was the closest to what would be most efficient to say.

Translation Method- Random Forest

Introduction

Once we were satisfied with our data sets and syllable counting function, we were ready to implement a machine learning-based algorithm to determine the information density of a given text. First, we needed to create a system whereby a model would input a text and output a number corresponding to its efficiency. In order to do this, we devised a function to create our training data, that would determine the "efficiency score" for each line from our source texts in terms of the line's translations, by comparing the number of syllables used in a line to its translations lines while accounting for the average number of syllables used per line in each language. We then tested different machine learning methods on our data, and settled on Random Forest as we found it to be the best suited for this task.

Training Data Creation

Then, the data was standardized in order to ensure that a sufficient range of data was available for the model to train on.

Random Forest Model

A random forest model consists of multiple decision trees which will all reach an output in a randomized manner. The most common output from all the decision trees will be the output of the model. Each decision tree's classification method is determined by bagging -- in which each tree will randomly select different parts of the data without replacement (for example, if a data set consisted of the numbers {1, 2, 3, 4}, randomly selected subsets include {1, 1, 2} or {1, 2, 3, 3}), and feature randomness -- in which each tree randomly selects features used to determine an output. The numerous random decision trees in a random forest model minimize the possibility for bias towards an individual feature. We found that the random forest model was able to perform better than the other in-built Wolfram machine learning methods, which is why we chose to use it for this specific task.

The random forest model was originally supposed to solely input an English, Spanish, French, or Italian line, then output the efficiency of that line based on its translations to the other languages. However, the model was not very accurate. We realized that random forest works best with numerical inputs, so to fix this, we inputted both the line and the number of syllables in that line (using the syllable counter) into the model, and it worked much better.

Training our model

The higher the score that the model outputs, the less efficient the sentence, while the lower the score, the more efficient the sentence is. Let's test it on some phrases:

If we test it on a less efficient sentence with the same number of syllables, we get a larger number:

Now, let's see how well the models for the other languages work. Theoretically, "ciao" (hello in Italian) should be more efficient than "hola" (hello in Spanish), "bonjour" (hello in French), and "hello" as they all have two syllables, whereas "ciao" only has one:

As we expected, the efficiency is negative.

Translation Method - Word Vectors

For our second translation-based method, we extracted features known as “word vectors” in order to improve the accuracy of our model. Word vectors are a list of floating point numbers where each element represents a dimension of the word’s semantic meaning [3]. So, the difference between the word vectors “car” and “truck” should be small because both words have similar meaning.

We started by removing any diacritics or numbers that don't have corresponding word vectors and then separated each sentence by word.

Inputting these words into the "FeatureExtraction" function, we obtain a list of vectors containing 50 numbers for each word.

Using our example from before, let's compare the word vectors of "car" and "truck".

Pretty close, especially when compared to the difference between the word vectors of two completely unrelated words.

Using the "Predict" function, we trained a model mapping the word vectors of each sentence to its corresponding score.

Repeating this process for each language, we get the following function.

Now, let's run our function on a couple of test cases to make sure the numbers match our intuition.

As we expected, adding superfluous adjectives that don't add any meaning will lower the sentence's efficiency score.

Shannon Density

Introduction

After implementing our idea for approximating the density of text by using translations, we wanted to have a way to evaluate how well our translation-based algorithms would work. To do so, we decided to program a separate algorithm that uses the information-theory definition of "information density." If the results predicted by this algorithm would turn out to be similar to those predicted by our translations-based algorithm, we'd know we were on the right track.

In information theory, the information content of a linguistic event, also known as "surprisal" or "Shannon information," can be thought of as the unlikelihood of the event occurring given the context [4].

For example, in the sentence, "After staying up all night, Sally was tired," the word "tired" carries little additional information, because we expect it to appear. By contrast, in the sentence, "After staying up all night, Sally was enormous," the word "enormous" carries much more information, because it is unexpected.

More rigorously, given an event x with probability P, the information content of x is given by:
I(x) = - log(P).

The base of the logarithm determines the units; if it were base 2, we would say the information is measured in "bits," or "Shannons." For the purposes of this project, we used the natural logarithm, for which the unit is called "nats."

Implementation

Broadly, our aim was to calculate the information density of a text by summing up the information content I(x) for each letter within the text. Then, we divided the total by the number of characters in the text to find the information density.

To calculate the probability of a letter occurring in the text given all the preceding letters, we used a built-in Wolfram function called SequencePredict, applying it to English, French, and Spanish. The function works similarly to the text generation on your phone.

Set up the function for English:

Predict the next letter at the end of a text:

Get the probability of a certain letter appearing at the end of the text:

Finally, the construction of our function for Shannon density:

While the above ShannonDensity function is what we used for the majority of our analysis, it does have the downside of being very slow. The impatient reader may prefer our FasterShannonDensity function, which returns a less-precise result by only considering every other character within the inputted text.

As a final note, we point out that although the previous two algorithms evaluated information content on a syllable-by-syllable basis, ShannonDensity goes by the character. This decision was made based on convenience considering the way the built-in SequencePredict functions operate. We did have the option to use SequencePredict to train a predictor function on the sentences within our database rather than using the built-in functions, which would have made it possible to determine the information content of larger chunks of text, including of syllables. However, when we attempted this method our scores were much less accurate, likely because we did not have enough data.

Testing and Exploration

Does it work? We can start by doing a few sanity checks. For one, we would hope that inputting a text such as "eeeeeeeeeeeeee," where most of the letters are very easy to predict, would give us a low score, which for this function indicates low information density.

On the other hand, a random series of characters should give us a high score, corresponding to high information density, because we can't ever be certain about what letter will come next.

What about actual English text? Hopefully something in-between.

Results/Data Exploration

Charts and Visualizations

Over the course of the project, we spent a lot of time experimenting with the data generated by our functions. Some of the questions we wanted to answer were:

◼

How similar are our functions?

◼

How do our functions treat different languages?

◼

What factors influence the density scores determined by our functions? (eg. sentence length, number of words in sentence, style of writing, etc.)

We will now attempt to briefly explore some of these questions.

When comparing between our three functions, we found that since each function outputted a different range of raw scores, it made more sense to look at percentile scores. We will go more into the implementation of percentile scores under “cloud deployment,” but the idea is basically that we look at what percentage of sentences in our database are less information-dense than the inputted sentence -- so the higher the percentile score, the more dense the sentence. This allows us to view a set of outputted scores that have been standardized for all three functions.

So how do our functions compare? To start, we can compare the plots of the density-percentile score over sentence length for each sentence in Adam Smith's "The Wealth of Nations."

The distributions are fairly different, but there are some interesting similarities to note. For one, all three functions show a tendency to assign higher scores to shorter sentences. We feel that this makes some intuitive sense, because shorter sentences have less room for filler or unnecessary words.

At the same time, though, sentences of the same length could differ widely in percentile score. Interestingly, the range of scores for the Translation-RandomForest function was much tighter than for the others.

We were further curious to see whether the average length of words in a sentence would affect its score. We were somewhat unsure of what we’d find -- on the one hand, we thought that sentences with lots of short words are more likely to be written in a simple, direct style. On the other hand, longer words often have much more specific meanings, so we thought they might contribute more to the information content of sentences.

Luckily, we can test it out, again using the book “The Wealth of Nations:”

So far we’ve neglected to look at different types of writing. We were interested to see whether the scores for different writing styles would be significantly different. To test this, we can look at three different bodies of text: transcriptions from the European Parliament, Lewis Carroll’s “Alice in Wonderland,” and Adam Smith’s “Wealth of Nations” again.

The European Parliament (Europarl) transcripts were used as the basis for our percentile scoring, so the average score for sentences from that database should be 50% for all three functions. But what about the other texts? “Wealth of Nations” is quite wordy and written in a fairly archaic style, so although it’s a heavier read, we expected it to register as less dense on average than Europarl. By contrast, sentences in “Alice in Wonderland” are much shorter, simpler, and more direct, leading us to believe that it would score the highest of the texts.

Here are our results:

The outcome is a bit of a mixed bag. While both translation methods returned results consistent with our predictions, ShannonDensity did not. The average sentence score that ShannonDensity returned for “Alice in Wonderland” was also strangely high, almost double that of Europarl.

Lastly, we didn’t program our functions to work with French, Spanish, and Italian for nothing. We were curious to see how the percentile scores compared for different languages.

For simplicity, let’s look just at the Translation-RandomForest function. We can compare the data for the English, Spanish, French, and Italian translations of “Alice in Wonderland”:

Pretty similar distributions. Is there any difference between the average percentile scores for the different languages?

It would appear there is a noticeable difference. To be able to explain the significance of this result with a degree of confidence, we would probably need to see how it compares with many more translated texts. However, it’s still interesting to think about; “Alice in Wonderland” was originally written in English, which has the highest density score. Could it be that translators of the text were more likely to leave out information than add new details? Or is it just a product of unrelated factors, like our training data?

Our conclusion is that the functions we came up with are a far cry from exact (if it’s even possible to provide an exact answer to this sort of problem), but reflect our intuitions closely enough to be interesting.

Cloud Deployment

Introduction

We also decided to create a web app where a user could enter their own text and test our methods of measuring efficiency to show them how efficient their text is at conveying information. They should be able to enter:

◼

Text

◼

The language of the text

◼

The genre of the text so that we can use specific training data

◼

Which of three methods should be used to calculate efficiency

Then we wanted to be able to show the user:

◼

How many syllables are in the text

◼

An efficiency score

◼

A percentile score to give more context, indicating what percent of sentences in our data sets are less efficient than the inputted sentence

◼

The most efficient and least efficient sentence in their text

◼

Some of the graphs we created before to help visualize the data

Calculating Percentile

We wanted to be able to show the user a comparison of the efficiency of the text they entered and the efficiency of other text. To do this, we found the position of the model’s predicted score in the position of the sorted scores of the lines in the Europarl and the combined data set of the books to give a percentile for each line in terms of what its efficiency would be in fiction and non-fiction. Here is a list of the efficiency values we got for our data, sorted in order of least to most efficient:

A function to give a percentile from 1-100 based on where one value falls in a list

We then passed in our data and the score of a specific sentence to this function. For example, here are a few tests using our random forest method. A higher percentile indicates that the sentence is more efficient.

Most and Least Efficient Sentences

We also wanted to give examples of the most and least efficient sentences in the inputted text. It was useful for us to see the most extreme results to get an idea of how our function is working and to gain insight about what makes sentences most efficient. To do this, we simply took the maximum and minimum scores and used their indices in the list to access the corresponding sentence.

A function to get the most efficient sentence

A function to get the least efficient sentence

Graphs

To help visualize the information more, we created two simple charts to show the user.

◼

A histogram, showing the distribution of density scores.

◼

A plot of the lengths of sentences and their scores.

Our function for the graph took the sentences, their scores and the function used as input.

Graphing the length of sentences against their score

For our histogram, we simply passed the density scores into Wolfram Language's Histogram function.

Putting it all Together

Once we had all the elements we wanted to display on the cloud-deployed page, we used FormPage to allow the user to enter text, select its language, and select a function.

Next, we created a Module in order to reuse a few expressions. For example, we created a variable and gave it the value of the list of efficiency scores for every sentence in the inputted text.

Our function to calculate all efficiency scores

We could then use Column and list every piece of information we wanted to output, using the functions we created earlier. We also gave the page a title and description.

Using CloudDeploy to make our form publically accessible. (The above form is not formatted well)

Future Works

If given more time, two straightforward ways to improve the accuracy of our algorithms would be to use more data and more languages. Thus, our next steps could include extending the syllable counting function to work with many more languages and working on diversifying our data set. While experimenting, we found that English, a Germanic language, was more efficient in terms of syllables used compared to Romance languages such as French, Italian, and Spanish; with data from other language families, we could draw conclusions about the relative efficiencies of each language group, and the efficiency of a text compared to all of human language.

There are also numerous extensions to this project we would be interested in pursuing, such as analyzing how efficiency changes between different genres of text and whether the patterns are similar for different languages, comparing the average efficiencies of different authors and songwriters, and exploring differences between how different individuals express themselves in everyday, conversational speech.

References

[1] https://www.researchgate.net/publication/235971274_A _cross-Language_Perspective _on _Speech _Information _Rate
[2] https://www.science.org/doi/10.1126/sciadv.aaw2594
[3] https://towardsdatascience.com/from-words-to-vectors-e24f0977193e
[4] https://en.wikipedia.org/wiki/Information_content
[5] https://opus.nlpl.eu/Europarl.php
Data Source: https://farkastranslations.com/bilingual_books.php