Wolfram Language has a catalogue of languages however it does not have alphabet and word data for many of these languages. The WordList function can produce common and known word lists for many common languages but not for less commonly spoken languages. The Universität Leipzig has word and sentence lists in 257 languages that are curated from Wikipedia. Those lists were scraped so that a word list as well as the occurrences of each word are produced. This helped when trying to determine word frequency data for these languages. For languages that did not already have an alphabet in Wolfram Language the uni-code block helped to develop this so that the word lists could be cleaned. The lists that were created do not take into consideration variations of the same word (ie. running, ran, run) so there is a lot to improve in terms of creating a criteria for words that can be included in the word list.
Telugu Word List
Telugu Word List
This file contains a list of sentences from Wikipedia, curated by the Universität Leipzig. The word lists on the Universität Leipzig corpora has statistical data about their word lists, however the Word Counts function in Wolfram Language used with the sentences list will provide a more accurate list because the words are being used as they usually are rather than just each word standing alone.
In[]:=
teluguWikiText=Import["/Users/mnamuduri/Library/Mobile Documents/com~apple~CloudDocs/Downloads/tel_wikipedia_2021_300K/tel_wikipedia_2021_300K-sentences.txt"];
The file is formatted like a table and to scrape the information the formatting needs to be removed and each sentence becomes its own element.
In[]:=
teluguSentences=StringCases[teluguWikiText,"\t"~~Shortest[x__]~~"\n"->x]
The sentences need to be combined into one string so that they can be broken up into individual words and so that when the Word Counts function is applied later the occurrences of each word reflects how many times they show up over the entire text rather than just for each sentence.
In[]:=
teluguText=StringJoin@teluguSentences
Since the project is primarily centered around finding common word lists, using the Word Counts function provides useful information on the occurrences of each words. Using this function also separates the words of the text without including repeats.
In[]:=
teluguAssociation=WordCounts[teluguText]
The first step in cleaning the word list so that there are no numbers and words in other languages is to take the words that the association has as keys and forming a list out of them
In[]:=
teluguWordList=Keys[teluguAssociation]
To get rid of the The uni-block code for a language can be used to create an alphabet for a language. For the language Telugu, along with having basic vowels and consonants, the language also has consonant conjuncts and consonant-vowel ligatures. This is where consonants and/or vowels will combine in order to form a different sound. The uni-block code shows the letters as the separate parts a word or letter will be recognized as when it is inputted into the code.
In[]:=
teluguAlphabet="అ","ఆ","ఇ","ఈ","ఉ","ఊ","ఋ","ౠ","ఌ","ౡ","ఎ","ఏ","ఐ","ఒ","ఓ","ఔ","ఁ","ం","ః","క","ఖ","గ","ఘ","ఙ","చ","ఛ","జ","ఝ","ఞ","ట","ఠ","డ","ఢ","ణ","త","థ","ద","ధ","న","ప","ఫ","బ","భ","మ","య","ర","ల","వ","శ","ష","స","హ","ళ","్","ఱ","ా","ి","ీ","ు","ూ","ృ","ౄ","ౢ","ౣ","ె","ే","ై","ొ","ో","ౌ","ఀ","ఽ","ౕ","ౖ","ౘ","ౙ","ౚ";
This makes the Telugu alphabet a file.
In[]:=
Export["teluguAlphabet.wxf",teluguAlphabet]
In[]:=
Import["teluguAlphabet.wxf"]
When this function is run it breaks each word into characters which are then compared to the characters of the alphabet that has been assigned to the language so any characters like numbers or characters in other languages will be removed.
In[]:=
teluguCommonWordList=Union[Select[teluguWordList,ContainsOnly[Characters[#],teluguAlphabet]==True&]];
This makes the word list a file.
In[]:=
Export["teluguCommonWordList.wxf",teluguCommonWordList]
In[]:=
Import["teluguCommonWordList.wxf"]
The word occurrences use the information that the Word Counts function outputted for the associations and applies it to the cleaned up word list.
In[]:=
teluguWordOccurrences=Association[Map[#->teluguAssociation[#]&,teluguCommonWordList]]
Finding the total number of word occurrences will provide a way to measure the how frequently a word is used
In[]:=
Total@teluguWordOccurrences
The common word lists are generated by the words that are used the most and then alphabetizing them. The word occurrence lists are already alphabetized but they still need to be capped so that they are only including the words that are most commonly used so this function ensures that a word is only included in the list if they occur a certain number of times
In[]:=
Select[teluguWordOccurrences,#>=10&]
The word frequency refers to how each word’s occurrences compare to the total number of word occurrences.
In[]:=
teluguWordFrequency=teluguWordOccurrences/2184831//N
This makes the word frequency data a file.
In[]:=
Export["teluguWordFrequency.wxf",teluguWordFrequency]
In[]:=
Import["teluguWordFrequency.wxf"]
In[]:=
WordCloud@teluguWordFrequency
Czech Word List
Czech Word List
The Czech word list was curated using a similar method to the Telugu word list with the major differences being the number of sentences included in the data set and the alphabet already being available in Wolfram language.
In[]:=
czechWikiText=Import["/Users/mnamuduri/Downloads/ces_wikipedia_2021_30K/ces_wikipedia_2021_30K-sentences.txt"];
In[]:=
czechSentences=StringCases[czechWikiText,"\t"~~Shortest[x__]~~"\n"->x]
In[]:=
czechText=StringJoin@czechSentences
In[]:=
czechAssociation=WordCounts[czechText]
In[]:=
czechWordList=Keys[czechAssociation]
In[]:=
Alphabet
In[]:=
czechCommonWordList=UnionSelectczechWordList,ContainsOnlyCharacters[#],Alphabet==True&;
In[]:=
Export["czechCommonWordList.wxf",czechCommonWordList]
In[]:=
Import["czechCommonWordList.wxf"]
In[]:=
czechWordOccurrences=Association[Map[#->czechAssociation[#]&,czechCommonWordList]]
In[]:=
Total@czechWordOccurrences
In[]:=
Select[czechWordOccurrences,#>=5&]
In[]:=
czechWordFrequency=czechWordOccurrences/337283//N
In[]:=
Export["czechWordFrequency.wxf",czechWordFrequency]
In[]:=
Import["czechWordFrequency.wxf"]
In[]:=
WordCloud@czechWordFrequency
Concluding Remarks
Concluding Remarks
This project uses data scraping on collections of sentences and words in order to curate common word lists and word frequency data for languages that Wolfram Language does not already have word lists for as well as curating an alphabet for a language that did not have an alphabet in Wolfram language. A good continuation of this project would be to figure out the criteria that should be considered when forming word lists like these. The number of words in dictionaries in the same language will often greatly differ based on the perimeters being used. The criteria for what a word is can change greatly based on the dictionary and creating a set of rules for what Wolfram Language should include in its word lists would be very useful.
Acknowledgments
Acknowledgments
Thank you to Stephen Wolfram for helping me find a project that involves my interest in the humanities and to my mentor Jofre Espigule for helping me through every phase in my project and helping me develop my computational skills.
References
References