GPT-2 is a large language model (LLM) produced by OpenAI that predicts the next word or phrase given either a question or the start of a sentence. As artificial intelligence models become more commonly used, they grow more advanced, from being able to generate text to imitating others and outputting various artistic forms. This project seeks to investigate how altering specific parts of the model affects its response to gain insight into how GPT-2 and other text-generative AI models work. This can be done by altering the tokens, nodes, and temperature parameters of the LLM and studying various factors of the output.
Introduction
Introduction
With over 1.5 billion users around the world, the development of artificial intelligence chatbots, such as generative pre-trained transformers (GPT), has grown rapidly. Its many functions, such as the ability to produce accurate and relevant responses in a matter of seconds, make machine learning models more powerful and relevant in society today. The increasing popularity of such technology leaves many wondering how exactly LLMs can generate useful outputs in about 50% of the time it would take for a human to do the same.
To fully understand how GPT works, we can look into the "brain" of the model- that is, various factors of the model such as temperature or nodes that each play a role in delivering an output to the user. Specifically, we will alter GPT-2, which was released by OpenAI in 2019, to analyze the level of manipulation at which the output is affected. This allows us to delve into various aspects of language models, such as how the chatbots we interact with daily work and why phenomenons such as AI hallucinations (which is when LLMs generate incorrect or incomprehensible information) occur.
To fully understand how GPT works, we can look into the "brain" of the model- that is, various factors of the model such as temperature or nodes that each play a role in delivering an output to the user. Specifically, we will alter GPT-2, which was released by OpenAI in 2019, to analyze the level of manipulation at which the output is affected. This allows us to delve into various aspects of language models, such as how the chatbots we interact with daily work and why phenomenons such as AI hallucinations (which is when LLMs generate incorrect or incomprehensible information) occur.
Setting up the GPT-2 Model
Setting up the GPT-2 Model
Before starting, we need to import GPT-2 into the Wolfram Mathematica notebook and define it as a variable- "GPT2model"- so that the model can be easily referenced in later sections. Doing so is the basis of the program and allows us to view information such as number of nodes/layers of the model, types of layers, and the weights.
Imports the GPT-2 model into Wolfram Mathematica:
In[]:=
GPT2model=NetModel[{"GPT2 Transformer Trained on WebText Data","Task""LanguageModeling"}]
Out[]=
NetChain
In this case, GPT-2 is a neural net model with a transformer structure; models with a transformer structure take significantly time to train compared to larger LLMs, and it implements attention to specifically look at important pieces from the input. The net model also tells us that GPT-2 was also trained on a large amount of WebText data- 8 million websites, to be exact- in order to generate text as accurately as possible.
Defines the function "generateSample" that allows the model to generate text and finish a sentence based on an input:
In[]:=
generateSample[languagemodel_, input_String, numTokens_:10, temperature_:1]:= Nest[ Function[ StringJoin[ #, languagemodel[#, {"RandomSample", "Temperature"->temperature}] ] ], input, numTokens];
Text can now be generated by GPT-2 when the function above is called with a sample prompt from the user:
In[]:=
generateSample[GPT2model,"AI is cool because",30]
Out[]=
AI is cool because it offers people a way to opt out of the government but sometimes you want one single hope and that is in Africa. The way things are I'm
The following displays information of the GPT-2 mode: all the arrays/layers within the model, how many arrays there are, and the weights within a specific array, respectively. These variables will be referenced later to modify certain parts of the model. All arrays and weights are compressed because of the size of the model.
As shown below, there is a total of 196 arrays in GPT-2, each containing 700-800 weights (for example, array two alone has 768 weights). When the plus sign on NetGraph is clicked, the model expands to show all the layers/classifiers that the input processes through to reach an output:
In[]:=
displayArray=Normal[Information[GPT2model,"Arrays"]];Length[displayArray]specificArray=2;dataInArray=displayArray[[specificArray]][[2]]//Normal;NetFlatten@GPT2model
Out[]=
196
Out[]=
NetGraph
Below is a brief look at how array two's weights look when expanded:
In[]:=
Effect of Altering Tokens
Effect of Altering Tokens
GPT-2 models generate text token by token. Tokens are integers that represent the smallest units or elements of text either inputted by the user or generated by the model (for reference, 1 token = 4 characters or 3/4 of a word). Because of this, we can zero out certain tokens in the GPT-2 model and analyze how doing so affects the output generated.
When tokens #1-10 are zeroed out:
In[]:=
inputText="AI is cool because";
Storing the tokens generated by the model in a new variable, generatedTokens, and defining the start and end tokens to zero out:
In[]:=
generatedTokens=GPT2model["GenerateTokens"];positionStart=1;positionEnd=10;
Creating a function that sorts through the tokens and replaces the start and final tokens defined earlier with zero:
modifiedTokens=ReplacePart[generatedTokens,Table[i->0,{i,positionStart,positionEnd}]];
Using the generateSample function, calls the GPT-2 model to generate text after the specified tokens have been modified:
In[]:=
modifiedText=generateSample[GPT2model,inputText,20]
Out[]=
AI is cool because the downside is the wiki.com(ã).corPec21.com is called KENN
Two functions that can be used to analyze the outputs of GPT-2 and its coherency when altered are TextStructure, to locate the point when the outputs’s sentence structure begins to fail, and DictionaryWordQ, to find when the model begins to make up nonsensical words.
Analyzing the sentence structure of modifiedText:
In[]:=
TextStructure["AI is cool because the downside is the wiki.com(ã).corPec21.com is called KENN","PartsOfSpeech"]
Finding how many words the LLM produces that are not in the English dictionary to see if the output contains any nonexistent words:
In[]:=
dictwords=DictionaryWordQ/@StringSplit["AI is cool because the downside is the wiki.com(ã).corPec21.com is called KENN"];Count[dictwords,False]
Out[]=
2
According to the text structure and dictionary word analysis, the GPT-2 model has difficulties placing appropriate punctuation and writing grammatically correct sentences when the first ten tokens are zeroed out; not only does the sentence structure display that the model outputted three different forms of punctuation but the sentence itself is illogical and does not correlate to the prompt.
When tokens #50-60 are zeroed out:
In[]:=
inputText="AI is cool because";
Storing the tokens generated by the model in a new variable, generatedTokens, and defining the start and end tokens to zero out:
In[]:=
generatedTokens=GPT2model["GenerateTokens"];positionStart=50;positionEnd=60;
Function that sorts through the tokens and replaces the start and final tokens defined earlier with zero:
In[]:=
modifiedTokens=ReplacePart[generatedTokens,Table[i->0,{i,positionStart,positionEnd}]];
Using the generateSample function, calls the GPT-2 model to generate text after the specified tokens have been modified:
In[]:=
modifiedText=generateSample[GPT2model,inputText,70]
Out[]=
AI is cool because it has graphical designers already that know how to create macros. It is easy to use either on a Mac or PC. We can create an XML markup for a document that cannot be obtained natively from Javascript. TypeScript with Updater, thanks to HackenbugMonk, can quickly and easily translate word processors into corporate clients of higher institutions
Displays the grammatical structure of text generated when the modifiedText function was generated earlier:
In[]:=
TextStructure["AI is cool because it lets users experience the world in their personal pockets","PartsOfSpeech"]
Out[]=
| ||||||||||||||||||||||||||||
Sentence |
Number of words not found in the English dictionary:
In[]:=
dictwords=DictionaryWordQ/@StringSplit["AI is cool because it lets users experience the world in their personal pockets"];Count[dictwords,False]
Out[]=
0
The sentence generated when later tokens are zeroed out shows several drastic differences from the previous output. All of the words that the model produced could be found in the English dictionary, and the analysis of the sentence's grammatical structure proves that the model produced a complete thought with all appropriate parts of speech. Although not perfect (the response left out minor elements, such as punctuation at the end of the sentence), the GPT-2 model's output is significantly more relevant to the input than before. This suggests that altering tokens later in the LLM has less of an impact as the model is still able to produce understandable, human-like sentences even after tokens 50 through 60 were set to zero.
When all tokens are zeroed out:
Defines the input text for the model to use when developing an output:
In[]:=
inputText="AI is cool because";
Storing the tokens generated by the model in a new variable, generatedTokens, and setting all tokens in the model equal to zero (cancelling them out):
In[]:=
generatedTokens=GPT2model["GenerateTokens"];modifiedTokens=ConstantArray[0,Length[generatedTokens]];
Using the generateSample function, calls the GPT-2 model to generate text after the specified tokens have been modified:
Here, we have two examples of an output generated when all tokens are zeroed from the GPT-2 model. Let’s select one of these sentences to analyze:
Displays the grammatical structure when the modifiedText function was run earlier:
Number of words not found in the English dictionary:
In this situation, the modified AI produced three nonsensical words: "becauseouh," "InBev," and the sequence of brackets at the end of the sentence, which was counted as a word according to the DictionaryWordQ function. Furthermore, when analyzing the sentence for both logic and grammar, the model did not produce a complete thought or idea; the sentence did not flow or follow correct grammatical syntax, further displaying the generated sentence is illogical after all tokens were zeroed out. This is one example of artificial intelligence/LLM hallucinations as the model is producing text that does not logically make sense. Thus, this research proves that aside from limited training data, AI hallucinations can also be a result of any alterations of the model's tokens.
According to all the responses, when either all of the tokens or the first few tokens are zeroed out, the response is confusing or does not correlate to the prompt. In contrast, deleting tokens in the middle or end has a less significant change on the output generated by GPT-2. Now, let's take a look at the unchanged GPT-2 model’s response from the section earlier to compare text structures and words between models with unchanged vs. changed tokens.
Generating an output from the unchanged GPT-2 model:
Displays the grammatical structure of the unchanged model:
According to the output, GPT-2 had only a few minor errors regarding sentence structure, such as unfinished sentences and punctuation missing, but it overall generated a grammatically correct and relevant output. The sentences developed are, for the most part, related to the input given about AI and technology, and there are no phrases that are made-up or confusing to comprehend. This further builds on the idea that altering earlier tokens of the model have the most significant impact on the behavior of the model.
From this information, we can conclude that the GPT-2 model, and possibly other GPT models, produces outputs by looking at previous tokens first, meaning that the generation of tokens- and the text itself- essentially builds upon each other.
Zeroing Out Nodes
Zeroing Out Nodes
Furthermore, GPT-2 models implement nodes, or layers, that each play a specific role in how the model behaves. Neural nets essentially process the input through all the layers of the model, and each layer has a unique weight- or parameter- assigned (typically, these weights are determined by the data the model trained from during the machine learning process). These weights take the input data and generate a response, and its size correlates to how much that layer affects the output.
In order to test the effect that changing nodes have on the output of the GPT-2 model, we can zero out all the weights within the nodes and see if the new output is different from the original.
In order to test the effect that changing nodes have on the output of the GPT-2 model, we can zero out all the weights within the nodes and see if the new output is different from the original.
A new variable, list, is defined as the values of the arrays of GPT-2. Each element of the list is set to be an array of zeros with the dimensions of both the element and the zeroed-out array match:
Information about the arrays is assigned to the parameters variable, which is then used in an association called listNew:
The list with altered data is assigned to a new GPT-2 model to produce an output:
As another metric to analyze the behavior of GPT-2 when the nodes are zeroed out, let's also take a quick look at how many words that are not in the English dictionary were produced.
Number of words not found in the English dictionary:
As we can see above, the response after zeroing out all nodes is incoherent and irrelevant to the prompt given, and the model outputs random characters with words spelled wrong. With 13 words in the output that cannot be found in the English dictionary, according to the DictionaryWordQ function, this analysis proves that altering the weights within the nodes of the model does have a significant impact on GPT-2's behavior. Compared to when we altered the tokens earlier, changing weights within the nodes have a much more significant impact on how the model reacts. Thus, the weights of the arrays are essentially the base of the LLM that greatly impact the model's comprehensibility when changed.
One can think of the nodes (arrays) as parts of the human brain. Similar to how the brain contains over 20 structures that each play a significant role in human function, GPT-2 also contains 196 nodes that affect the model's behavior. When certain parts of the human brain are altered, that corresponding function is affected and may appear as changes in the person's senses or how they act. The image below from MIT News further describes this by highlighting the specific parts of the brain that affect a person's eyesight and how altering those areas can have an impact on their vision:
One can think of the nodes (arrays) as parts of the human brain. Similar to how the brain contains over 20 structures that each play a significant role in human function, GPT-2 also contains 196 nodes that affect the model's behavior. When certain parts of the human brain are altered, that corresponding function is affected and may appear as changes in the person's senses or how they act. The image below from MIT News further describes this by highlighting the specific parts of the brain that affect a person's eyesight and how altering those areas can have an impact on their vision:
Likewise, GPT-2 and other LLMs have arrays that each contain a set of weights (the parameters that allow the output to be generated from the input). Zeroing out these weights within the arrays essentially equates to turning off the parts of the model's brain, resulting in a jumbled-up mess of characters and random words.
Temperature Parameters
Temperature Parameters
Another part of the GPT-2 model that we can alter to change the output are its temperature parameters. The temperature ranges from 0 to 2- although GPT-2 and other LLMs can technically go up to infinity- and affects the randomness of the output. Higher temperatures make the model more creative and random, whereas low temperatures closer to 0 make the model output more predictable/common words.
Temperature is already included in the generateSample function we created earlier, so the desired temperature can simply be added at the end when calling the function.
Temperature is already included in the generateSample function we created earlier, so the desired temperature can simply be added at the end when calling the function.
GPT-2 model with a temperature of 0:
Aside from sentence structure and use of dictionary words, we can look at repetitiveness to analyze the model's intelligence for varying temperatures. With a temperature of 0, the model uses the most predictable words (according to the WebText dataset developed by OpenAI used to train GPT-2). After the first sentence, however, it continuously repeats itself without outputting any other variations of words.
The same case occurs for a temperature of 0.5, as shown below:
The same case occurs for a temperature of 0.5, as shown below:
GPT-2 model with a temperature of 0.8:
A temperature of 0.8 is the "happy medium" of the GPT-2 model; we find that temperatures near that number are ideal as the model produces the most relevant and grammatically-correct outputs to the prompt. On the other hand, temperatures above 0.8 produce outputs that are no longer repetitive but are instead more creative; however, the word choice can also become more irrelevant to the prompt asked. As the temperature increases past the ideal, the model begins to grow more and more “insane.”
GPT-2 model with a temperature of 1:
GPT-2 model with a temperature of 1.2:
From the output, we find that a temperature of 1.2 is the GPT-2 model’s breaking point when it begins to, in its simplest form, go crazy and generate extreme, nonsensical outputs. Past a temperature of 1.2 up until 2 (or technically infinity), the model will begin to react the same way and may even output text that is even more incomprehensible, similar to that of zeroing out all the nodes of the model from the previous section. Thus, it can be concluded that the temperature is another significant structure of the model's "brain" that impacts its behavior toward the user when altered.
Conclusion and Future Extensions
Conclusion and Future Extensions
In this project, the connection between the human brain and the GPT-2 model's structure was studied. The structures within the model itself were also expanded upon, particularly analyzing the effect that modifying nodes, temperature parameters, and tokens has on the behavior of GPT-2. We analyzed how changing the temperature can make the model either repetitive or creative, sometimes generating incomprehensible outputs at certain temperatures. We also took a closer look at how zeroing out nodes and tokens can affect several aspects of the output- grammatical structure, logic, and word choice- and at what point these factors are affected.
One aspect of this project that I hope to expand on in the future is studying what purpose each array in the entire GPT-2 model serves. Because my aim is to study the similarities between the human brain and the minds of LLMs, I would like to try zeroing out individual nodes, similar to turning off certain parts of the brain; this extension would allow me to understand how certain nodes affect the overall behavior of the model and what each node does. I also hope to analyze the structures of other similar LLMs, such as ChatGPT and the newly developed GPT-4, and compare those findings to the research I have conducted with GPT-2. From this, I could analyze if the structures and nodes of other AI chatbots are different from GPT-2, which is an older model, and what those differences are.
Acknowledgements
Acknowledgements
I would like to thank CJ Applebaum, my mentor, for her continuous support throughout the development of this project. I also wish to thank Nicolò Monti, who helped me troubleshoot parts of my code and provided valuable insight.
References
References
1
.Wolfram, S. (2023, February 14). What Is ChatGPT Doing … and Why Does It Work?. Stephen Wolfram Writings RSS. https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/
2
.Trafton, A. (2011, March 1). Parts of brain can switch functions. MIT News | Massachusetts Institute of Technology. https://news.mit.edu/2011/brain-language-0301
3
.What are tokens and how to count them?. OpenAI. (n.d.). https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Analyzing GPT-2's brain development
by Shriya Ramanan
Wolfram Community, STAFF PICKS, July 13, 2023
https://community.wolfram.com/groups/-/m/t/2963742
by Shriya Ramanan
Wolfram Community, STAFF PICKS, July 13, 2023
https://community.wolfram.com/groups/-/m/t/2963742