This project aims to dive into the challenge of counting the total number of words in the English language, which is a complex task given the ambiguities of the definition of a “word”. It attempts to get an estimate by employing the LLM features of Wolfram Language to filter out real words, reach to a final count and visualize their extent based on their frequencies of use.
Introduction: what counts as a “word”?
Introduction: what counts as a “word”?
This may seem like a straightforward question, but in reality it does not have a clear answer. While we often rely on spaces and punctuation marks to clearly separate words, we must also consider other factors such as morphology, function and meaning. Morphological variations of a word, such as the addition of prefixes and suffixes (recognize vs. unrecognizable), and inflections (sing, sang, sung, singing…) challenge our notion of a word as a discrete orthographical unit. Additionally, determining whether we should include acronyms, slang, proper nouns, and compound words (“air vent” for instance) complicates matters even further.
For the purpose of this project, we define a word as an orthographic unit (unigram) with meaning, excluding proper nouns, acronyms and abbreviations. This approach considers any collection of characters as an individual word and compound words like “air vent” as two, regardless of whether they are morphological variations, proper English words or slang. However, it includes the aforementioned proper nouns and abbreviations, as well as nonsensical combinations, which we eliminate with Wolfram Language’s LLM functionalities as explained later.
For the purpose of this project, we define a word as an orthographic unit (unigram) with meaning, excluding proper nouns, acronyms and abbreviations. This approach considers any collection of characters as an individual word and compound words like “air vent” as two, regardless of whether they are morphological variations, proper English words or slang. However, it includes the aforementioned proper nouns and abbreviations, as well as nonsensical combinations, which we eliminate with Wolfram Language’s LLM functionalities as explained later.
Finding and filtering words
Finding and filtering words
The dataset
The dataset
Firstly, we download a word list from Kaggle containing the 333,333 most common English words in the Google Web Trillion Word Corpus. It lists the unigrams as well as their respective frequencies, ranked from highest to lowest. Because it includes unigrams that do not fit my definition, and nonsensical entries (“eptve”, “ffv”, “golgw”, etc.), our next task is to eliminate them from my count. We will also use the built-in Wolfram Language word list as a reference.
Preparing the dataset for filtering
Preparing the dataset for filtering
Firstly, we import our dataset as kaggleList with the first entry dropped, since it is the header of the list.
kaggleList=Drop[Import["unigram_freq.csv"],1];
Then, we load the reference word list commonWords and obtain the words that are in kaggleList but not in commonWords.
In[]:=
commonWords=ToLowerCase[WordList[]]
In[]:=
wordsNotInCommon=Complement[kaggleList[[All,1]],commonWords]
Out[]=
While a lot of the unigrams in wordsNotInCommon are nonsense, there are still some real words that we wish to count. So, we define a word finder with LLMExampleFunction using GPT-3.5-Turbo as our model.
In[]:=
SeedRandom[1234];inputExample1=RandomSample[wordsNotInCommon,50];
In[]:=
outputExample1={"dilations","pretends","hexahedral","cherishes","retinopathy","framesdirect","culpa","capuchin","resorcinol","bouvier"};
SeedRandom[2345];inputExample2=RandomSample[wordsNotInCommon,50];
In[]:=
outputExample2={"wallon","rudds","driveby","netcraft","saloons","rerelease","stockpiles","peacemakers"};
SeedRandom[5678];inputExample3=RandomSample[wordsNotInCommon,50];
outputExample3={"neurodegeneration","decades","bromoform","chapbook","arborist"};
SeedRandom[1111];inputExample4=RandomSample[wordsNotInCommon,50];
outputExample4={"wheezes","brucellosis","taliesin","culi","seismographic","warns","infoview","disentangling","ascetics","lauter","soucy","condenses","botas","placentation","hulled"};
SeedRandom[9000];inputExample5=RandomSample[wordsNotInCommon,50];
outputExample5={"subsisting","payments","shimmies","cyanuric","specifc","gemology","lupins","principals","unblocking","impinger"};
SeedRandom[2222];inputExample6=RandomSample[wordsNotInCommon,50];
outputExample6={"brewsters","furbearers","tailhook","eccrine","microenterprise","farewells","concerns"};
SeedRandom[3333];inputExample7=RandomSample[wordsNotInCommon,50];
outputExample7={"parian","waterslide","quiddity","underpinning","viroid","saddlers","monetization","globalism","incentivized","workbooks","ladies","leachate","participated","bruant"};
In[]:=
wordFinderNC=LLMExampleFunction[{"Select only real English words from the following list and put them in a list. Do not include any nonsensical strings (like wgq, libsilc, and yronwode). Do not select proper nouns, acronyms and abbreviations. If a word should start with uppercase, rewrite it as such.",{inputExample1->ToString[outputExample1,InputForm],inputExample2->ToString[outputExample2,InputForm],inputExample3->ToString[outputExample3,InputForm],inputExample4->ToString[outputExample4,InputForm],inputExample5->ToString[outputExample5,InputForm],inputExample6->ToString[outputExample6,InputForm],inputExample7->ToString[outputExample7,InputForm]}},"Expression"]
We obtain the input-output prompts by generating random samples from wordsNotInCommon (input) and picking the unigrams that are real English words (output). Then, we feed them into the model to eliminate non-words from any list of unigrams we input into it, yielding a list of only real words.
Eliminating non-English words
Eliminating non-English words
One issue we quickly encounter is that wordsNotInCommon is too big to be input directly into our word finder. So, we split it into partitions of 300 unigrams.
In[]:=
unigramsOrderedByFrequency=Select[Drop[kaggleList[[All,1]],1],MemberQ[commonWords,#]==False&]
Out[]=
In[]:=
partitions=Partition[unigramsOrderedByFrequency,300]
Out[]=
which gives us a total of 993 partitions. We then define another function that runs the word finder for the n-th partition. We write it in a way that it runs up to 10 times if it returns a failure.
In[]:=
filterWordsByPartitionNumber[n_Integer]:=Module[{words=partitions[[n]]},Catch[Do[With[{res=wordFinderNC[ToString[words]]},If[ListQ[res],Throw[res]]],10];$Failed]]
Afterwards, we run it in three instances, each in increments of 331. This yields 3 filtered partitions containing only real words.
In[]:=
results1=Table[filterWordsByPartitionNumber[k],{k,331}]
Out[]=
In[]:=
results2=Table[filterWordsByPartitionNumber[k],{k,332,662}]
Out[]=
In[]:=
results3=Table[filterWordsByPartitionNumber[k],{k,663,993}]
Out[]=
We originally thought that ordering the unigrams by frequency would make evaluating the filter easier as the frequencies decreased, since there would be fewer real words to be detected. However, the filter ran without much issue, so it was not a relevant step.
Computing total word count
Computing total word count
To count the words we combine commonWords and the resulting partitions in a single giant list.
In[]:=
combinedWords=Flatten[Join[commonWords,results1,results2,results3]]
Out[]=
In[]:=
totalWords=DeleteDuplicates[combinedWords]
Out[]=
where totalWords is the duplicate-free version ready to be measured:
Length[totalWords]
168313
This is comparable with the count of 171,476 words in current use that are in the Second Edition of Oxford Dictionary, although it is off by approximately 3,000.
Frequencies
Frequencies
The Kaggle word list also includes the frequencies of use of each unigram. This should give us a way to visualize the extent of the unigrams compared to the real words. We can easily extract them as follows.
We can then plot the following histogram.
What this histogram tells us is that at low frequencies most of the unigrams are not associated to any real English word. As it increases, they start to overlap until at very high frequencies every unigram becomes a real word. It is also interesting to note that the majority of the unigrams in the list are not even considered “words” according to our analysis.
Concluding remarks
Concluding remarks
The word count we obtained is remarkably close to the one from Oxford Dictionary we must take this result with a grain of salt. Firstly, our model is far from perfect, as it may have eliminated unigrams that are real words and mistakenly counted non-words as words like “spsn” and “ooogla”. The real words that were eliminated contributed to our word count being a few thousand words off. We also determined from the histogram the trend of higher frequencies correlating to the unigrams being actual words.
One possible future improvement is to implement another LLM function that inputs words from our curated list of words to output some data about them, such as their types (noun, verb, adjective, etc.) and meanings. It would allow us to determine which words do not have any meaning, are proper nouns or loanwords, etc. This could be used together with the word finder to eliminate more nonsensical unigrams and others we do not count as words, further refining our list. Providing more or better examples of input-output to the word finder would also mitigate this issue.
Furthermore, since dictionaries like Oxford and Merriam-Webster also include obsolete words and other words we did not count, their numbers are much larger (~600,000 and ~470,000 respectively). Languages constantly change as new words appear in everyday vocabulary, which makes reaching to an accurate word count a complex question with no definite answer. Ultimately, counting how many words there are in English depends on how we define a word which may yield results that vary massively.
Keywords
Keywords
◼
Word count
◼
Unigrams
◼
English language
◼
Word lists
Acknowledgments
Acknowledgments
Thanks to all the guidance and help from my mentor Jofre Espigule-Pons, especially with writing the code. Thanks to Stephen Wolfram who personally suggested me this topic.
References
References
◼
“How many words are there in the English language?” Oxford English Dictionary. https://web.archive.org/web/20160907151313/http://www.oxforddictionaries.com/words/how-many-words-are-there-in-the-english-language
◼
Merriam-Webster. https://www.merriam-webster.com/help/faq-how-many-english-words
◼
Oxford English Dictionary. https://public.oed.com/about/
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Investigating the extent of the English Language
by Thiago Han
Wolfram Community, STAFF PICKS, July 12, 2023
https://community.wolfram.com/groups/-/m/t/2958812
by Thiago Han
Wolfram Community, STAFF PICKS, July 12, 2023
https://community.wolfram.com/groups/-/m/t/2958812