Abstract
Abstract
Writing style is an integral part of literature. In this essay, the process of mimicking “style” will be explored, allowing pre-existing Natural Language Generation models to capture the most characteristic elements of a writer’s work. Additionally, an objective way of comparing the notion of style is also discussed as well as applied to assess model performance.
Introduction
Introduction
Recently, Natural Language Generation has been made increasingly more accessible through the publication of well known models such as OpenAI's GPT. GPT was trained on a very large dataset made up off book data, known as BookCorpus. Whilst this allows for somewhat believable text generation, training on such a variety of texts results in the text generated being devoid of "personality" or "style". This post shows how I attempted to use the pre-existing text generating capabilities of GPT, but inject the ability to mimic the style of a sample author or text.
GPT - Generative Pre-trained Transformer
GPT - Generative Pre-trained Transformer
Let's take a look at the GPT model available from the Wolfram Repository:
Out[]=
Before moving onto the guts of the model, let's consider the inputs and outputs. The input, as you can see, is a string. Coherent strings are made up of words. The input layer of GPT has a vector of size 40,478 which is effectively the model's "vocabulary" - every known word, from the model's perspective. The output is similarly a vector corresponding to certain words but there is no requirement that these two vectors be identical, though the vanilla GPT model does use identical input and output vectors.
We can see that there are three main components: embeddings, decoders, and the classifier. In rough terms, the embedding component's job is to take in a word and convert it into a numerical vector which attempts to capture the word's meaning. Additionally, the architecture as is has no notion of position, so a positional encoding is also computed for a given word as to allow the remaining components of the model to understand the order of the input text.
Next, we have the "decoder", which is really a stack of 12 individual decoders. This is really the "heart" of the model - and also the reason for "Transformer" being in the name. The precise technicalities of these decoder units are outside of the scope of this post, though the general idea is that they aim to express the notion of "attention", which really means "relative word importance". Each such decoder unit is built from a "Masked Self-Attention" process, as well as a standard feed forward neural network.
Finally, "classifier". The purpose of the classifier is to "interpret" the output signal from the stack of decoders and choose the appropriate word from the output vector. Classifier in the vanilla GPT model is simply a single connected layer, as you can see from the graphic. This is really the key piece that I focus on in this post. The intuition as to why is to think of word choice as the method of interpreting the output of the decoders implying that modifying this method, modifies word choice and so modifies style.
Just as a note, GPT2 and GPT are separate models with GPT being the older of the two. For this project, I chose to work with GPT.
We can see that there are three main components: embeddings, decoders, and the classifier. In rough terms, the embedding component's job is to take in a word and convert it into a numerical vector which attempts to capture the word's meaning. Additionally, the architecture as is has no notion of position, so a positional encoding is also computed for a given word as to allow the remaining components of the model to understand the order of the input text.
Next, we have the "decoder", which is really a stack of 12 individual decoders. This is really the "heart" of the model - and also the reason for "Transformer" being in the name. The precise technicalities of these decoder units are outside of the scope of this post, though the general idea is that they aim to express the notion of "attention", which really means "relative word importance". Each such decoder unit is built from a "Masked Self-Attention" process, as well as a standard feed forward neural network.
Finally, "classifier". The purpose of the classifier is to "interpret" the output signal from the stack of decoders and choose the appropriate word from the output vector. Classifier in the vanilla GPT model is simply a single connected layer, as you can see from the graphic. This is really the key piece that I focus on in this post. The intuition as to why is to think of word choice as the method of interpreting the output of the decoders implying that modifying this method, modifies word choice and so modifies style.
Just as a note, GPT2 and GPT are separate models with GPT being the older of the two. For this project, I chose to work with GPT.
Transfer Learning
Transfer Learning
Transfer Learning is taking a pre-trained model, trained on a specific task (in our case, general language generation), and then repurposing that model for a different yet related task (for us, not generally generating text but generating a specific style of text). There is more detail on this later, but a common approach is to retrain certain aspects of the model, whilst keeping other parts intact. The canonical example is in image classification. There exist sophisticated, pre-trained classifiers that can classify objects in images into thousands of different classes. If we instead want to vastly reduce the available output classes to two, for instance "computer" or "not computer", then we could apply transfer learning to the pre-trained model, retaining associations built up by the original model and just slightly repurposing them.
Pre-Processing
Pre-Processing
Before getting into the modifying the model architecture, it is important to familiarise ourselves with the data we will be handling. The raw data for this project is simply the pure text written by various authors. It is not, however, as simple as throwing this data into GPT and crossing our fingers… We want to retrain the later layers of GPT, so in order to do that we need to create some training and testing examples.
Train-Test Split
Train-Test Split
In most machine learning projects, of a given data set, we would segregate roughly 60-70% for training and the remaining 30-40% would be for testing or validation. Most of the time, the data is also shuffled at this stage as to not impact model performance. For this project, a train-test split was also needed, but due to the sequential nature of our task, there was no easy way of shuffling (one idea is to section text into sentences then shuffle sentence order, but what if you want to train on text length longer than a sentence?). An appropriate function was written as a result:
In[]:=
textTestTrainSplit[text_String,ratio_Real]:=Map[StringRiffle,ResourceFunction["TrainTestSplit"][StringSplit[text],"Shuffle"->False,"TrainingSetSize"->ratio]]
Note, however, that we need to be careful in using this. To illustrate why, imagine an academic paper. The style of the first ~80%, could be potentially very different from the last ~20% containing the conclusion. Luckily, when I was choosing the texts I had this in mind, so it was unlikely to pose a serious problem.
Generating Samples
Generating Samples
The training data our model is expecting is of the form: String → Word. An added constraint here is that the word or sub-word being mapped to must be within a “vocabulary” which is covered next. Note also that we can map to individual pieces of punctuation. Another interesting question to consider is the length of “String”. I solved this problem by simply defining a maximum length and taking random samples of length up to that maximum. When training, I typically used a max length of 20 although I occasionally upped it to 50 if it made contextual sense. Seeing the code should clear things up, I encourage you to take a look at the attached library file. Here's a sample from Stephen Wolfram's blogs:
"trainingText" is just a random Wolfram blog, "50" is the max length of a sample, "5004" means we want to randomly generate 5004 samples, "genNewWordClass" is explained in Vocabulary below.
trainingSet=generateSamples[trainingText,50,5004,genNewWordClass[blogs]];
In[]:=
trainingSet[[4]]
Out[]=
a business card and say : “if you ever want to chat more , send me mail” . and , yes , the ones i hear from are a very self - selected set . typically i’ll do an initial phone call tolearn
Vocabulary
Vocabulary
GPT works by taking in a text string and classifying the most likely next word from a large word vector. I refer to this vector as a model’s “vocabulary”. Because that last layer will be getting retrained regardless, it made sense to reconstruct the model’s vocabulary to better reflect that of the text’s. An important note is that I did not only use the words present in a given text as that could cause problems if the sample text was sufficiently small. As a result, I combined the text’s vocabulary with “Dale-Chall Easy Words” (3000 common words, more on this later!) as well as miscellaneous symbols like punctuation and numbers.
In[]:=
genNewWordClass[text_String]:=Module[{specialCharacters,replacementRules,textList,commonWords}, specialCharacters={".",",",":",";","?","(",")","[","]","\"","'s","'","!","_","-","/","\\","—","0","1","2","3","4","5","6","7","8","9"}; replacementRules=Map[#->" "<>#<>" "&,specialCharacters]; textList=ToLowerCase/@StringSplit[StringReplace[text,replacementRules]]; textList=Union[textList,textList]; commonWords=StringSplit[Import["daleChallEasyWords.txt"]]; textList=Union[textList,commonWords]; textList=Union[textList,specialCharacters]; textList=Map[#<>" "&,textList]; textList=Append[textList," "]];newDecoderFromClass[class_List]:=NetDecoder[{"Class",class}]
Armed with our training and testing sets, let’s move onto model construction.
Modifying GPT
Modifying GPT
Transfer learning typically is done by taking a pre-trained model and “re-initialising” components of that model that occur towards the model’s output (replacing “later” components). The general idea is that you keep the relations the model learnt about a data set intact, you just specify a slightly different task by allowing those final components, or layers in our case, to re-train.
reducedNet=NetTake[pretrainGPT,{1,"last"}];newClassifier=NetInitialize[LinearLayer["Input"768,"Output"4956]];newNet=NetChain[<|"gptCore"reducedNet,"newClassifier"newClassifier,"newSoftmax"SoftmaxLayer[]|>,"Output"decoder]
Out[]=
We also want to limit weight updates to only those last layers, we do not want to touch the earlier layers! This is specified as follows during training.
Specifically, consider the "LearningRateMultipliers" option.
trainedNet=NetTrain[newNet,trainingSet,ValidationSettestSet,TargetDevice"CPU",LearningRateMultipliers{"newClassifier"1,_0}]
An additional step that has proved to work in prior research is to, after training with the settings above, go for another full training session but this time increasing the number of layers that have their weights updated. For most of the models that I showcase, this approach was taken.
trainedShakespeare=NetTrain[trainedNet,trainingSet,ValidationSettestSet,TargetDevice"CPU",LearningRateMultipliers{"newClassifier"0.5,{"gptCore","decoder",12}1,{"gptCore","decoder",11}1,{"gptCore","decoder",10}0.5,_0}]
Here's an example training output:
Out[]=
Algorithmic Style Comparisons
Algorithmic Style Comparisons
The concept of a writer’s “style” is quite nebulous. I can imagine someone saying “oh, author x has a really distinct style!”, but they would surely be pressed when it comes to defining what they mean in an atomic, precise sense. My view of style can be described in three components: “meaning”, “level and purpose”, and “word usage”.
“Meaning” is by far the hardest to implement in code and, due to time constraints, I did not bother even attempting to tackle it (that does not mean there aren’t possible approaches, see “future work”).
“Level and purpose” should really be thought of as the intersection of the concept of “level” and the concept of “purpose”. Take, for instance, a purpose of: “to educate”. OK. That’s quite broad. “The Very Hungry Caterpillar” aims to educate, as does “A Classical Introduction to Galois Theory” but, I’m sure without having even read these two books, you can tell that their styles are very different… I think the easiest-to-quantify stylistic difference between them is their “level” (you may be unconvinced by this argument, but I encourage you to consider the style similarity between something like “... Intro to Galois Theory” and “An Elementary Introduction to Number Theory” as well as to remember that I was also looking for the indicator which had the highest ease of implementation to efficacy ratio).
Alright, so how did I actually go about writing an algorithm to find level and purpose? The main metric was utilising readability tests. There's a wide range of various formulas so I chose the Flesch-Kincaid readability test which looks primarily at syllable usage, as well as the Dale-Chall readability test which looks mainly at the frequency of non-standard words (remember the list of 3000 words mentioned earlier? Any word not on that list is "non-standard"). These two formulas are nice and well defined, here's the code:
“Meaning” is by far the hardest to implement in code and, due to time constraints, I did not bother even attempting to tackle it (that does not mean there aren’t possible approaches, see “future work”).
“Level and purpose” should really be thought of as the intersection of the concept of “level” and the concept of “purpose”. Take, for instance, a purpose of: “to educate”. OK. That’s quite broad. “The Very Hungry Caterpillar” aims to educate, as does “A Classical Introduction to Galois Theory” but, I’m sure without having even read these two books, you can tell that their styles are very different… I think the easiest-to-quantify stylistic difference between them is their “level” (you may be unconvinced by this argument, but I encourage you to consider the style similarity between something like “... Intro to Galois Theory” and “An Elementary Introduction to Number Theory” as well as to remember that I was also looking for the indicator which had the highest ease of implementation to efficacy ratio).
Alright, so how did I actually go about writing an algorithm to find level and purpose? The main metric was utilising readability tests. There's a wide range of various formulas so I chose the Flesch-Kincaid readability test which looks primarily at syllable usage, as well as the Dale-Chall readability test which looks mainly at the frequency of non-standard words (remember the list of 3000 words mentioned earlier? Any word not on that list is "non-standard"). These two formulas are nice and well defined, here's the code:
In[]:=
fleschKincaidScore[text_String]:=N[206.835-1.015(getWordCount[text]/getSentenceCount[text])-84.6(getSyllableCount[text]/getWordCount[text])]
In[]:=
daleChallScore[text_String]:=Module[{easyWords,numDiffWords}, easyWords=TextWords[Import["daleChallEasyWords.txt"]]; numDiffWords=Length[Select[ToLowerCase/@TextWords[text],!MemberQ[easyWords,#]&]]; N[0.1579((numDiffWords/getWordCount[text])*100)+0.0496(getWordCount[text]/getSentenceCount[text])]]
The last part of this section is what I refer to as (generically and slightly unhelpfully): saturation. What the really means is "descriptive saturation", or, even more precisely, "ratio of adjectives and adverbs to all words". As you can probably tell, this metric mainly deals with "purpose", a report is likely to have very low descriptive word count, but a novel is likely to have a higher count.
In[]:=
descriptiveSaturation[text_String]:=N[(Length[TextCases[text,"Adjective"]]+Length[TextCases[text,"Adverb"]])/getWordCount[text]]
Finally, we have "word usage" which is by a long way the easiest to program. Word usage aims to reflect a writer's idiosyncratic vocabulary - think of it as quantifying someone's "favourite" words. For this I created a simple word frequency chart, along with a visually pleasing albeit less concrete Word Cloud.
Text Generation
Text Generation
Now that we have all the boring stuff out of the way… let’s generate some text! I trained three overarching models, two of which have two sub-versions. The models shown here are the ones which proved most effective from my testing. Each one is tailored to either a specific author or a more general style. Let’s take a look.
Shakespeare's Sonnets
Shakespeare's Sonnets
Depending on the type of person you are, hearing or reading his name likely conjured up memories of boredom and thinking to yourself: "I literally can't understand a single word I am reading". Of course... that wasn't me... Moving on, the reason why I chose Shakespeare's sonnets was that, firstly, there was an easy and large data source (his sonnets are published online for free) and, secondly, it should be quite obvious if our modified model had "gotten" the style. Here are the results:
"Although in me each part " is the only text input given, 250 specifies number of words to generate, and 0.875 is the "temperature", it essentially controls the frequency of choosing less likely words - if the model always chooses it's top pick, it tends to get stuck in a loop.
Quite nifty I have to say that I impressed myself! If you squint you'll see a Shakespearean work. At least, that's how my untrained eye perceives this. Of course, it was unrealistic to expect it to be anywhere near perfect, but I was very surprised at the apparent effectiveness. Shakespeare's style is very different from the original material GPT was trained on. I was expecting this test to completely fail as a result of the deeper layers not being updated to accommodate the new style.
For reference, here's the original GPT's rendition:
For reference, here's the original GPT's rendition:
Stephen Wolfram's Blogs
Stephen Wolfram's Blogs
As an attendee of the Wolfram High School Summer Camp, I see it only fitting that I try mimic the style of Stephen Wolfram. Luckily, he has a large collection of writings, but I wanted to challenge the model and only try train on roughly 85 KB of data. Here are the results:
I think that, out of all the models, this one marginally least effective. This is quite strange because I think Wolfram's base style is most similar to that of the original BookCorpus data. The obvious conclusion is to point to the lack of data, but the Shakespeare model was trained on only slightly more (and was still significantly more challenging). My belief, although I did not get a chance to experiment with this, is that Stephen Wolfram's blogs are typically laser focused on very specific and sometimes niche topics. Jargon is thus prevalent and I believe this impacted the model's ability to comprehend his writing (since the actual input layer was not touched by the Transfer Learning process). One interesting observation, however, is that Stephen Wolfram has a tendency to start sentences with conjunctions (i.e. "and", "but", "yet", etc...). As you can see, the model does the same which hints at certain aspects of his writing being picked up.
And now GPT's...:
Legalese: The Federalist Papers
Legalese: The Federalist Papers
I could pretend to be erudite and say that I know about The Federalist Papers because of my casual legal reading, but the truth is that I only know about these documents thanks to the hit musical: Hamilton. Regardless, The Federalist Papers (co-authored by John Jay, James Madison, and Alexander Hamilton, published in 1788) are an elephantine 180,000 words, filled with somewhat archaic words and most certainly written in that old-timey, legalese-esque style. Let's see how our model does:
I definitely think this is the strongest word generator I built. It certainly is the most legible and, in my opinion, captures the very characteristic style talked about earlier. It did have a lot of things going for it, however. This model in particular had the largest amount of text to train on, roughly five times the other models! Additionally, in comparison with some of Stephen Wolfram's blogs, a lot of the legal jargon and tell-tale words they use are recognised by GPT's base vocabulary. Finally, the original text has a lot of innate repetition, much like Shakespeare's sonnets.
And finally, the great legal mind of GPT:
Style Comparisons; Man VS Machine
Style Comparisons; Man VS Machine
Time to put our style algorithms into action! The column on the left is the data associated with the original body of work that the models got to train on. Consequently, the column on the right is style data of the generated text. We are looking for similarity, an optimum outcome is the two columns being exactly the same.
Shakespeare
Shakespeare
Stephen Wolfram's Blogs
Stephen Wolfram's Blogs
Stephen Wolfram's Blogs
Legalese
Legalese
Legalese
So, as you can see, the results are rather sporadic. The first observation is that the "Average Estimated Grade" seems to be off. I was expecting it to be inaccurate (you can only really get so far with those formulae), but it suggests that all of the original texts could be comfortably read by someone in seventh grade. Another interesting result is the shape of the "Word Chart". The generated texts seem to roughly follow the curve of the original texts, and many key words show up in both charts. The most compelling indicator for similarity is the word cloud, which, as you can see, seems to indicate that word usage between generated and original work is related.
Another thing to consider when looking at these results is that only 250 words of generated text from one sentence suggestion are being used for comparison against the entirety of the original text. Here's the legal example but compared against only the paragraph that the sentence starter was taken from:
Another thing to consider when looking at these results is that only 250 words of generated text from one sentence suggestion are being used for comparison against the entirety of the original text. Here's the legal example but compared against only the paragraph that the sentence starter was taken from:
As we can see, the statistics are somewhat closer. This is especially true with the readability-related scores.
Conclusion, Future Work, and Acknowledgements
Conclusion, Future Work, and Acknowledgements
Future Work
Future Work
There are many possible extensions, let's start with the promised potential solution for capturing "meaning" in style algorithms. "Meaning" is a very abstract topic, but for a rudimentary overview, using something along the lines of GLoVe Word Vectors to capture it has proven successful in the past. I thought about implementing this but was worried about time constraints.
Another big area that I did not cover at all was an alternative approach to Transfer Learning specifically with the GPT model. Past research has indicated success in mimicking Tweet styles by collating a "history" (in the Twitter example, this just captured the "point" of the tweet), labelling such data accordingly and feeding it into the GPT model along with the beginning of the text you want to generate from, and tune the model from there.
The original GPT model was used for this work. GPT2 is, in many regards, a significant upgrade over GPT. Porting this project's process over to GPT2's architecture and potentially observing a corresponding increase in model performance could be interesting.
The final and probably most obvious extension is to just train the models on more data! During development, I experimented with a couple other texts that I decided not to publish. One such example is a model trained on a Noam Chomsky article which showed lackluster results. Another example is the model I trained on a movie script. Now, this sounds promising. Movies have distinct styles and it could have been interesting to see if it picked up on the nuances. The issue was that I trained it on the script to the 1990 Scorsese classic, Goodfellas... If you are reading this and have seen the film, you know exactly why I was hesitant to show off the mafioso-model (and if you haven't, all you need to know is that Goodfellas is currently ranked as the 16th highest movie in terms of f-word occurrences... a fact that really stands out when you don't have time to write a profanity filter).
Another big area that I did not cover at all was an alternative approach to Transfer Learning specifically with the GPT model. Past research has indicated success in mimicking Tweet styles by collating a "history" (in the Twitter example, this just captured the "point" of the tweet), labelling such data accordingly and feeding it into the GPT model along with the beginning of the text you want to generate from, and tune the model from there.
The original GPT model was used for this work. GPT2 is, in many regards, a significant upgrade over GPT. Porting this project's process over to GPT2's architecture and potentially observing a corresponding increase in model performance could be interesting.
The final and probably most obvious extension is to just train the models on more data! During development, I experimented with a couple other texts that I decided not to publish. One such example is a model trained on a Noam Chomsky article which showed lackluster results. Another example is the model I trained on a movie script. Now, this sounds promising. Movies have distinct styles and it could have been interesting to see if it picked up on the nuances. The issue was that I trained it on the script to the 1990 Scorsese classic, Goodfellas... If you are reading this and have seen the film, you know exactly why I was hesitant to show off the mafioso-model (and if you haven't, all you need to know is that Goodfellas is currently ranked as the 16th highest movie in terms of f-word occurrences... a fact that really stands out when you don't have time to write a profanity filter).
Conclusion
Conclusion
With this project I set out three key objectives. The first was to see if, in broad terms, writing style could imitated via Transfer Learning. The second was to test the extent to which the methodology I presented could operate successfully under increasingly challenging data sets. And finally, I wanted to create and apply a "style comparison" function. Of these, I would definitely contend that I have shown style being successfully mimicked; it's difficult to argue that the Shakespearean model did not capture any of Shakespeare's writing tendencies, for instance. The style comparison functions were less effective than I had imagined them being, but I still believe they offer at least a consistent pointer in the right direction, roughly indicating style. Finally, with various tests I was able to experiment with the limits of my methodology. Though I would have liked for the models to have been more robust, it is certainly interesting to look at the perceived barriers and limitations. All-in-all, I am very pleased with this project and I believe I met my key objectives to a high standard.
Acknowledgements
Acknowledgements
I would like to chiefly thank my mentor for this project: Harshal Gajjar, who helped enormously with various Wolfram Language issues and who always offered feedback when needed. I would also like to thank the mentor, TA, and admin team as a whole which, among helping out with various issues and offering guidance, also made the entire experience additionally enjoyable.
Additional Resources
Additional Resources
GPT Paper: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf
Transformers made simple: https://jalammar.github.io/illustrated-transformer/
GPT-2 made simple: https://jalammar.github.io/illustrated-gpt2/
Introduction to Transfer Learning: https://towardsdatascience.com/a-comprehensive-hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning-212bf3b2f27a
Readability Tests: https://en.wikipedia.org/wiki/List_of_readability_tests_and_formulas
Transfer Learning with GPT, Mentioned in future work: https://www.aclweb.org/anthology/P19-1608.pdf
Transfer Learning with GPT, Mentioned in future work: https://pdfs.semanticscholar.org/351c/e5a2ccf002f95e1d24068a97206115808c23.pdf
GLoVe Word Vectors: https://nlp.stanford.edu/projects/glove/
Transformers made simple: https://jalammar.github.io/illustrated-transformer/
GPT-2 made simple: https://jalammar.github.io/illustrated-gpt2/
Introduction to Transfer Learning: https://towardsdatascience.com/a-comprehensive-hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning-212bf3b2f27a
Readability Tests: https://en.wikipedia.org/wiki/List_of_readability_tests_and_formulas
Transfer Learning with GPT, Mentioned in future work: https://www.aclweb.org/anthology/P19-1608.pdf
Transfer Learning with GPT, Mentioned in future work: https://pdfs.semanticscholar.org/351c/e5a2ccf002f95e1d24068a97206115808c23.pdf
GLoVe Word Vectors: https://nlp.stanford.edu/projects/glove/
Keywords
Keywords
◼
GPT
◼
Natural Language Generation
◼
Transfer Learning
◼
Natural Language Processing
◼
Writing Style
◼
Linguistics