Constructed languages, or conlangs, are languages that were artificially created instead of being naturally evolved over several generations. A crucial part of every conlang is the lexicon — the vocabulary. While most conlang lexicons are generated manually or through the help of random combination algorithms, I introduce several algorithms and metrics for computationally generating a quantitatively “average” lexicon, that is, generating new words that have similar features, spelling, or pronunciation from translations of the same word in several languages. In the end, I present μLang, a conlang lexicon using computationally generated “average” words.
Introduction
Introduction
A conlang’s lexicon can offer crucial information about the purpose and intended speakers of the conlang. The conlang Esperanto created in 1887, aiming to create a “universal” language that could be learned and spoken by anyone with relative ease. However, Esperanto’s lexicon was heavily biased towards Romance and Germanic languages, undermining its original purpose of being universal. This project uses genetic algorithms, neural networks, and other approaches to attempt to quantify universality and generate a lexicon that is the optimal “average” of the world languages.
Initial Setup
Initial Setup
In order to derive average words, I needed a large corpus of multilingual words. I used the built-in database of languages in the Wolfram Language. Because not all languages use the Latin alphabet, I transliterated all language into the Latin alphabet. Additionally, I deleted any duplicates that may have biased the training data.
Get the top 100 most spoken languages:
In[]:=
languages=EntityClass["Language",{"TotalSpeakers"->TakeLargest[100]}]//EntityList;Iconize[languages,"List of languages"]
Out[]=
Get raw translations of our keyWord to the determined languages:
In[]:=
keyWord="sun";rawTranslations=KeyValueMap[#1->First[#2]&,WordTranslation[keyWord,languages]];
Standardize all translated words — Transliterate to the Roman alphabet for easier string manipulation, ensure all words are lowercase, and remove translations that are not available.
In[]:=
translatedList=DeleteCases[DeleteDuplicates[ToLowerCase[Select[Transliterate[Values[rawTranslations]],StringMatchQ[#,CharacterRange["A","z"]..]&]]],"notavailable"]
Out[]=
{sun,surya,shams,sol,soleil,suryya,solnce,matahari,swrj,sonne,ri,jua,nuryudu,gunes,katiravan,taeyang,rana,srengenge,sole,phraxathity,suraja,slonce,suran,taiyang,sonce,panonpoe,anwu,ilanga,aftab,araw,soare,zon,qorrax,son,adlaw}
Most Common Letter
Most Common Letter
The first thing I thought of was to join the most frequent characters together. I counted the most frequent letter at every index, found the average length of the original words, and concatenated the most common letters together to form a new string equal to the length of the average length.
Split each word into a List of characters:
In[]:=
charLists=Characters/@translatedList;charLists//Short
Out[]//Short=
{{s,u,n},{s,u,r,y,a},{s,h,a,m,s},{s,o,l},27,{z,o,n},{q,o,r,r,a,x},{s,o,n},{a,d,l,a,w}}
Normalize the lengths of the words through padding and truncation using the average length:
In[]:=
avgLen=Round[Mean[Length/@charLists]];augmentedLists=Map[Function[chars,If[Length[chars]>avgLen,Take[chars,avgLen],PadRight[chars,avgLen,"-"]]],charLists];augmentedLists//Short
Out[]//Short=
{{s,u,n,-,-},{s,u,r,y,a},{s,h,a,m,s},{s,o,l,-,-},28,{q,o,r,r,a},{s,o,n,-,-},{a,d,l,a,w}}
I transpose the list of characters to align all the characters into their corresponding indices.
Find the most frequent character at each position and generate the average word:
In[]:=
transposed=Transpose[augmentedLists];mostFreqChar=If[DeleteCases[#,"-"]==={},"-",SortBy[Tally@DeleteCases[#,"-"],-Last[#]&]][[1,1]]&/@transposed;
In[]:=
mostFreqCharWord=StringJoin[DeleteCases[mostFreqChar,"-"]];"Average word generated with the most frequent character at each position: "<>mostFreqCharWord
Out[]=
Average word generated with the most frequent character at each position: sonaa
Generate an animation of the most common character at each index and the generation of the word:
animatedFreq=Table[Module[{partialTransposed,partialMostFreq,charCounts,sortedCounts,labels},partialTransposed=Take[transposed,i];partialMostFreq=Map[If[DeleteCases[#,"-"]==={},"-",First@First@SortBy[Tally@DeleteCases[#,"-"],-Last[#]&]]&,partialTransposed];charCounts=Tally@DeleteCases[partialTransposed[[-1]],"-"];sortedCounts=ReverseSortBy[charCounts,Last];labels=Placed[sortedCounts[[All,1]],Below];Column[{Style["Position: "<>ToString[i],Bold,16],Style[StringJoin[DeleteCases[partialMostFreq,"-"]],24,Red],BarChart[sortedCounts[[All,2]],ChartLabels->labels,ImageSize->Large,PlotLabel->"Character Frequency at Position "<>ToString[i]]}]],{i,1,Length[transposed]}];ListAnimateanimatedFreq,
Out[]=
While this method does work, it may not be optimally average. If there are two characters of equal frequency at a position, there is a chance the suboptimal character is selected. Additionally, this approach doesn’t produce very creative words. It always outputs one very predictable word and there is no variance. To generate words with more variety, I explored more methods.
Most Common Digram
Most Common Digram
My second approach was to split each word into digrams (pairs of consecutive letters) and find the most common digram in each position. Then, we can join the most common digrams together. For example, “su” is the first digram in “sun” and “un” is the second digram in “sun.” Joining the two digrams would output “suun.”
Calculate the average syllable count:
In[]:=
syllableCount=Round[Mean[Length/@(ResourceFunction["WordSyllables"][#]&/@translatedList)]]
Out[]=
2
Split each word into digrams and record the position of each digram:
In[]:=
positionedDigrams=Flatten[Table[If[StringLength[word]>=pos+1,{pos,StringTake[word,{pos,pos+1}]},Nothing],{word,translatedList},{pos,StringLength[word]-1}],1];positionedDigrams//Short
Out[]//Short=
{{1,su},{2,un},{1,su},{2,ur},{3,ry},{4,ya},{1,sh},{2,ha},{3,am},{4,ms},{1,so},{2,ol},{1,so},{2,ol},127,{4,re},{1,zo},{2,on},{1,qo},{2,or},{3,rr},{4,ra},{5,ax},{1,so},{2,on},{1,ad},{2,dl},{3,la},{4,aw}}
Count the frequency of each digram in each position:
In[]:=
groupedByPosition=GroupBy[positionedDigrams,First->Last];countsByPosition=AssociationMap[Counts[groupedByPosition[#]]&,Keys[groupedByPosition]];countsByPosition//First
Out[]=
su5,sh1,so8,ma1,sw1,ri1,ju1,nu1,gu1,ka1,ta2,ra1,sr1,ph1,sl1,pa1,an1,il1,af1,ar1,zo1,qo1,ad1
Get the most frequent digram in each position:
In[]:=
topDigramsByPosition=Table[Keys@TakeLargestBy[countsByPosition[pos],Identity,UpTo[1]],{pos,Sort@Keys[countsByPosition]}]
Out[]=
{{so},{ur},{ry},{ya},{ce},{ng},{ri},{an},{it},{ty}}
Generate new words by randomly combining the most frequent digrams:
In[]:=
generateWord[]:=Module[{numSyllables,selectedLists,syllables},numSyllables:=RandomInteger[{Max[1,syllableCount-2],Min[Length[topDigramsByPosition],syllableCount+2]}];selectedLists=Take[topDigramsByPosition,numSyllables];syllables=RandomChoice/@selectedLists;StringJoin[syllables]]
Suggest a few possible words using random combinations of the most frequency digrams:
This method isn’t ideal either. While it occasionally produces pronounceable words, it relies too heavily on common digrams and uses a very basic substitution approach. Since some digrams end with the same letter others begin with, the result is often awkward combinations, like doubled vowels or consonants, making the words harder to read and pronounce. Both of the previous methods suffer from the same issue: they depend too rigidly on character frequency.
Genetic Algorithm
Genetic Algorithm
Another approach is to treat the most average word as an optimal evolutionary trait. Imagine the original list of translated words as a starting population, all trying to evolve closer to this ideal average word. Using a genetic algorithm, we can simulate this evolution by selecting the most optimal words in each generation and creating new, unique words that are similar in morphology to the multilingual wordlist but still contains minor mutations that encourage further evolution
In each generation, I select the top 50% of the most fit words to be parents. These parents undergo a modified two-point crossover, which allows the child words to be more flexible with their length. Additionally, each word has a 20% chance to mutate, meaning a random character in the word is changed. This process repeats generation after generation, gradually evolving the population toward the optimal trait.
In each generation, I select the top 50% of the most fit words to be parents. These parents undergo a modified two-point crossover, which allows the child words to be more flexible with their length. Additionally, each word has a 20% chance to mutate, meaning a random character in the word is changed. This process repeats generation after generation, gradually evolving the population toward the optimal trait.
Define important variables. populationSize is the size of the population each generation and mutationRate:
To quantify how close a given word is to an “optimally average” word, I defined a fitness function where higher fitness scores reflect a word closer to the average. The measure of “closeness” is based on a weighted sum of two factors: the mean Levenshtein distance between the candidate word and all words in a multilingual wordlist, and the average phonetic distance between the candidate word and the same wordlist.
To measure the phonetic distance, I used the Soundex to convert the transliterated words into generalized phonetic markers. Word with similar phonetics will be considered more similar or even equal. For example, “back” and “pack” will be closer than “back” and “rack” because the /b/ and /p/ sounds have the same place of articulation.
To measure the phonetic distance, I used the Soundex to convert the transliterated words into generalized phonetic markers. Word with similar phonetics will be considered more similar or even equal. For example, “back” and “pack” will be closer than “back” and “rack” because the /b/ and /p/ sounds have the same place of articulation.
Define the fitness function:
Note the fitness of a word similar to possible keyWord translations is significantly greater than a completely unrelated string:
My initial population will be a list of randomly combined digrams. This allows the population to start off reasonably similar to the multilingual wordlist but also allows diversity for future recombination.
Create our original population by randomly combining digrams:
With everything set up, I can simulate evolution. I used a modified two-point crossover algorithm that allowed both parent words to vary in length. I choose two random points within the shorter parent as crossover points. Then, the head and tail of the first parent is combined with the center of the second parent.
Implementation of two-point crossover:
Implementation of string mutation:
I record the best/”most fit” word each generation:
The best performing strings in each generation:
Log the most fit string:
Plot the average fitness of every population of each generation over time:
Overall, there is a clear trend of increasing fitness throughout the entire evolutionary period. There is a particularly rapid increase during the first 15 generations as the population quickly adapts to the multilingual wordlist. This shows the entire population is evolving towards both morphological and phonetic traits in the word list.
Plot the fitness of the best candidate in each generation over time:
This shows a very weak increasing trend in the fitness of the most fit words in every generation. Both graphs above show clear improvement in the fitness of the population and best word, demonstrating that my genetic algorithm is effective.
Evolutionary String Optimization
Evolutionary String Optimization
Building the μLang lexicon can also be viewed as an optimization problem. Given a random string, I can iteratively change its characters to increase its fitness and, at some point, achieve an optimal average word. The goal is to evolve a string that minimizes the average edit distance to all multilingual words. In each generation, I identify the word with the highest edit distance from the current string and modify a single character in the string to match a character from that word. Over time, this process converges on an optimal average word.
Set up basic variables and helper functions to get differing indices between words and calculate positions for the words to occupy on the animation.
I assign a weight to each character based on its frequency. This way, I encourage evolving towards more frequent characters at any position.
Calculate character frequencies for each position:
Normalize frequencies to a decimal number between 0 and 1:
Calculate weighted character frequencies:
Decide the character to evolve into based on the frequency weightings:
Mutate a character at randomly 5% of the time:
Evolve towards the decided character:
Simulate hundreds of generations:
Generate animation of the string evolution and evolutionary targets over time:
The gray words in the circle are from the word list, while the red string represents the word currently evolving. In each generation, the red line points to the target word the red string is aiming to evolve into. As the red word changes step by step, it gets closer to the target, indicating improved fitness between the two.
Plot the average word distance between the evolving string and multilingual word list over generations:
Generation with Morphological Neural Net Features
Generation with Morphological Neural Net Features
A more modern approach may be to take advantage of neural net embeddings. A neural net/deep learning model encodes words in numeric representations called embeddings or feature vectors. For text, embeddings usually encode semantic meaning, but the . We can ask the neural net to generate new words with similar embedding values as the training set (the original word list), generating new but similar words.
Set the vocabulary of characters the neural net can train from and predict to:
Replace the default Input port with a NetEncoder that converts input strings to character-level encodings:
Train the neural network on the multilingual dataset:
Extract the prediction layer to act as a character-level generator:
Now that our model is trained, it can autoregressively generate new strings. At every timestep, the output of the previous timestep is used as the input, allowing the network to predict the most likely letter to follow.
Autoregressively generate new strings:
Concatenate all the possible generated average words from approaches so far:
Generation with a Neural Net trained on phonetic characters
Generation with a Neural Net trained on phonetic characters
I also apply the same neural net pipeline for multilingual words converted into the International Phonetic Alphabet (IPA).
Initialize the neural network:
Because this network can only accept Roman letters, numerals, and punctuation, I map the IPA symbols to numbers and punctuation symbols .
Convert each multilingual word to its IPA representation and replace the IPA symbols with the symbolMap:
Train the neural network on phonetic data:
Autoregressively generate new strings using phonetic symbols:
Convert generated strings back into IPA:
N-gram Evaluation
N-gram Evaluation
To evaluate the differences between possible μLang words and the original wordlist, I performed an N-gram analysis on both vocabularies.
Both the μLang wordlist and the multilingual wordlist have “so” as their most frequent bigram (2-gram). In total, the two wordlists share 26 bigrams and 15 trigrams.
I also calculated the Jaccard Similarity for the bigram and trigram sets. The Jaccard Similarity is defined as the size of the intersection divided by the size of the union of the two sets. In this case, it’s the number of shared n-grams divided by the total number of unique n-grams across both wordlists.
The Jaccard Similarity for bigrams is approximately 0.286, while for trigrams it is approximately 0.139. This means there is some similarity between both wordlists, though there is still significant deviation. This is expected as the evolutionary and neural network approaches are not deterministic and can have some random mutation every time.
I also calculated the Jaccard Similarity for the bigram and trigram sets. The Jaccard Similarity is defined as the size of the intersection divided by the size of the union of the two sets. In this case, it’s the number of shared n-grams divided by the total number of unique n-grams across both wordlists.
The Jaccard Similarity for bigrams is approximately 0.286, while for trigrams it is approximately 0.139. This means there is some similarity between both wordlists, though there is still significant deviation. This is expected as the evolutionary and neural network approaches are not deterministic and can have some random mutation every time.
Display the total frequency of each type of 2-gram:
Ease of pronunciation
Ease of pronunciation
Considering the application of this work in actual conlanging, I also need to pick pronounceable words that can be spoken regularly. I heavily penalize consonant clusters longer than 4 consonants and also encourage a similar number of vowels and consonants. A greater score means a more difficult to pronounce word.
Implement a penalty for difficult to pronounce words:
Sort generated average words by pronounceability:
Graphical Interface
Graphical Interface
To make my algorithms accessible to conlangers and linguists, I designed a GUI for users to experiment with. Given a word, it will display the outputs of all the algorithms discussed in this research. I hope that this GUI will be of use to aspiring conlangers to generate unique but realistic words. Due to computational power limitations, I was unable to directly host the GUI on this notebook. The code for the full GUI is below:
Here is a screenshot of what the GUI is supposed to look like:
I also hosted a condensed version of the GUI at this link: https://www.wolframcloud.com/obj/hendryyy2008/mulang-lexicon-generator-full. It does not have neural network generation due to cloud hosting limitations but does include the genetic, evolutionary, and frequency based algorithms. Note that the Wolfram Cloud form takes a few minutes to produce results.
Conclusion
Conclusion
This work introduces several methods to computationally generate “average” words that contain morphological and phonetic features of many different multilingual translations. I tried using the most frequent character, most frequent digram, a genetic algorithm, a string evolution algorithm, and two neural networks. Overall, the genetic and evolutionary algorithms consistently produce words that are easy to pronounce and very close to the optimal average based on my fitness metrics. On the other hand, the morphological and phonetic neural networks produce words that have more variance and use a wider range of sounds.
Future Directions
Future Directions
My evolutionary algorithm and neural network approaches show promise in generating speakable words. With more training data from outside of Wolfram’s language database, a more reliable universal word could be achieved. Additionally, limited computational power prevented me from pushing the iterative genetic and evolutionary algorithms to find a more optimal word. With access to greater computational resources, I could research more complex algorithms that may produce better results.
The lexicon is just a small part of a full-fledged conlang. To ensure that μLang can operate as a truly speakable language, I need to decide on grammar rules such as word order, verb tenses, morphosyntactic alignment, and much more. I plan to use my work to develop a fully speakable conlang over this summer. In spirit of the “average” language, I plan to aggregate grammatical data from a wide range of languages and compute average grammatical features.
The lexicon is just a small part of a full-fledged conlang. To ensure that μLang can operate as a truly speakable language, I need to decide on grammar rules such as word order, verb tenses, morphosyntactic alignment, and much more. I plan to use my work to develop a fully speakable conlang over this summer. In spirit of the “average” language, I plan to aggregate grammatical data from a wide range of languages and compute average grammatical features.
References
References
Kawasaki, H. (n.d.). An Acoustical Basis for Universal Constraints on Sound Sequences.
Soundex Algorithm—An overview | ScienceDirect Topics. (n.d.). Retrieved July 9, 2025, from https://www.sciencedirect.com/topics/computer-science/soundex-algorithm
WordPhoneticSyllabify | Wolfram Function Repository. (n.d.). Retrieved July 9, 2025, from https://resources.wolframcloud.com/FunctionRepository/resources/WordPhoneticSyllabify/
WordSyllables | Wolfram Function Repository. (n.d.). Retrieved July 9, 2025, from https://resources.wolframcloud.com/FunctionRepository/resources/WordSyllables/
Soundex Algorithm—An overview | ScienceDirect Topics. (n.d.). Retrieved July 9, 2025, from https://www.sciencedirect.com/topics/computer-science/soundex-algorithm
WordPhoneticSyllabify | Wolfram Function Repository. (n.d.). Retrieved July 9, 2025, from https://resources.wolframcloud.com/FunctionRepository/resources/WordPhoneticSyllabify/
WordSyllables | Wolfram Function Repository. (n.d.). Retrieved July 9, 2025, from https://resources.wolframcloud.com/FunctionRepository/resources/WordSyllables/
Acknowledgements
Acknowledgements
This project would not have been possible with the help of my mentor Mark Greenberg, my mentor group peers Adit Amit Kumar, Daniel Xu, and Viola Pande. I would also like to thank other mentors including Lyman Hurd, Anton Antonov, and Aayush Dubey for helping me debug and brainstorm. The directors of the Wolfram Summer Research Program, Rory Foulger, Eryn Gillam, and Megan Davis also played a key role in helping me formulate my ideas and publish my work.
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
μLang — computationally generating a morphologically average conlang lexicon
by Hendry Xu
Wolfram Community, STAFF PICKS, July 10, 2025
https://community.wolfram.com/groups/-/m/t/3501302
by Hendry Xu
Wolfram Community, STAFF PICKS, July 10, 2025
https://community.wolfram.com/groups/-/m/t/3501302