Introduction
Introduction
My project uses a Recurrent Neural Network (RNN) to train a program to generate “pseudowords” - phonetically correct, English-like words, based on the first syllable of an input word. Training data was generated by splitting random English words apart at random places, showing all the letters before the split and the letter after (e.g: mil -> k), and the RNN was trained with approximately 10000 pieces of training data. The changeable parameters in my function are the input word, and the length, in characters, of the output word, and the output is simply a pseudoword. I worked on this project as a part of the Wolfram High School Summer Camp 2020 (WSC2020) , where I worked closely with several mentors and Stephen Wolfram himself. My interest in pseudowords and generating them using RNNs was inspired by several resources I found online, such as the Wuggy algorithm, an application built to generate pseudowords based on input words, and a project written in TenseFlow to generate Shakespearean text. I chose to use an RNN for this project because I have long held a passion for programming AI, and this provided to be great practice and a good achievement.
Generating Training Data
Generating Training Data
My first step was to generate random English words, convert them to all lowercase, and make sure that they did not contain characters such as apostrophes and dashes using LetterQ. I did this because such characters are irregular occurrences in English, and could thus throw the network off if used to train it. Although 10000 words were used to train the RNN, only 10 random words have been generated in the example below. A set of test words were also generated for a validation set (not shown below).
Generating Random English Words:
In[]:=
trainingWords=Select[ToLowerCase/@RandomWord[10],LetterQ]
Out[]=
{comparably,keyboardist,fertilizer,hummingbird,assizes,cavort,poulterer,pharmacologist,irrupt,handkerchief}
Next, I mapped two lists together, one being the list of training words, and the other being a list of random integers between 1 and the length of the corresponding training word. I then put these two lists into an association to use to split the training words at random places. I chose to split the training words in random places because there are sometimes patterns in English that occur at certain places in words (e.g: the last three letters 'ing' in verbs), which could result in the RNN's results generating more real words than pseudowords. Once again, the same thing was done for the validation set.
Mapping Split Positions to Words:
In[]:=
randomTrainingParts=RandomInteger[{1,#-1}]&/@Flatten[StringLength[StringSplit[trainingWords]]]randomAssoc=AssociationThread[trainingWordsrandomTrainingParts]
Out[]=
{3,4,8,4,5,3,3,9,4,10}
Out[]=
comparably3,keyboardist4,fertilizer8,hummingbird4,assizes5,cavort3,poulterer3,pharmacologist9,irrupt4,handkerchief10
Finally, to finish formatting the training data, I printed all the characters of each training word up to the splitting point, and then the next character.
Generating Training Data:
In[]:=
trainingData=StringTake[#,StringLength[#]-randomAssoc[#]]StringPart[#,-randomAssoc[#]]&/@trainingWords
Out[]=
{comparab,keyboard,fer,hummingb,ass,cavo,poulter,pharma,irr,han}
Creating and Training the RNN
Creating and Training the RNN
My RNN used trainingData as a training set and testData as a validation set. Although building the RNNwith NetChain seems simple, tweaking it proved to be a long and tedious task. At the beginning, during my first trial tests, my validation set's graph kept diverging from the path of that of the training set and overfitting. However, introducing a Dropout Layer lessened this and made the net far more accurate - the path of the validation set stayed more or less consistent with that of the training set. Eventually, I found the optimal settings to have the Dropout probability at 0.425 and to train the network for 25 epochs.
An example of an overfitting RNN:
In[]:=
I generated only 100 training data and test data points, which is fewer than the 10000 that I recommend using for training the net:
trainingData=;testData=;
Building the RNN:
In[]:=
Building the GeneratePseudowords Function
Building the GeneratePseudowords Function
After training the RNN, all that was left was to tie everything up into a user-friendly, customizable function that took the input word and the length of the pseudoword to be generated. I did this by building a recursive function where charAdd characters were added to the first syllable of the input word, which was extracted using WordData. These characters were generated by using the RNN's results (trainedNet) to get the next character and appending it to the previously generated string. The With function around my Module verifies that the pseudoword generated is indeed a pseudoword, and not a real dictionary word. If the word turns out to be a real dictionary word, the function returns nothing.
Code for GeneratePseudowords:
GeneratePseudoword[inputWord_,wordLength_,firstTime_:True]:= With[ word=ToString[Module[{firstSyllable,appendChar,charAdd}, If[wordLength0,inputWord, If[firstTime, firstSyllable=WordData[inputWord,"Hyphenation"][[1]]; appendChar=StringJoin[firstSyllable,trainedNet[firstSyllable]];charAdd=wordLength-StringLength[firstSyllable], appendChar=StringJoin[inputWord,trainedNet[inputWord]]; charAdd=wordLength; ]; GeneratePseudoword[appendChar,charAdd-1,False] ]]],If[DictionaryWordQ[word]False,word] ]
Generating pseudowords was the satisfying and fun end to this project. Below are some of the most fun pseudowords I generated:
In[]:=
GeneratePseudoword["ghost",8]ghosteryGeneratePseudoword["sea",9]sealinglyGeneratePseudoword["sky",9]skyberessGeneratePseudoword["aurora",8]auterateGeneratePseudoword["spa",11]sparticallyGeneratePseudoword["football",11]footmentingGeneratePseudoword["computer",9]competizeGeneratePseudoword["science",10]scillingly
For more variety, I recommend retraining the RNN with different random words as training data.