A neural network is a collection of artificial neurons intended to simulate the cognitive function of the human brain, albeit in a more simplified and abstract manner. Due to the wide range of data neural networks are able to take in, the guide for neural networks found on the Wolfram Documentation emphasizes how they are “a central component in many areas, like image and audio processing, natural language processing, robotics, automotive control, medical systems and more” [1]. In the case of transformers, such as GPT-2, they are pre-trained on large observational datasets, such as the examples given prior. Through this training, the neural network attempts to model a function that would best map this data, while also recognizing patterns. However, instead of further training GPT-2, this project will focus on altering the transformer and analyzing how these changes affect the outputs of the model.
Introduction
Introduction
The GPT-2 Transformer, which is accessible via the Wolfram Neural Network Repository, was trained on a vast amount of textual data, specifically the WebText dataset that contains around 40 gigabytes (GB) of data [2]. However, the architecture of this GPT-2 model is built by stacking decoder-only transformer blocks on top of each other. This architecture is typically seen in neural networks tailored towards text generation [3]. During this project, I manipulated this architecture by experimenting with various parts and parameters of the neural network and observing how it affects the output. The intention of this project is to fully understand how neural nets are pieced together as well as what a “broken” Machine Learning (ML) model looks like. Although the anatomy of neural networks in the Wolfram Language is not what is conventionally encountered, this language’s ability to visualize the components of neural networks in a unique and clean manner will facilitate this process.
To begin the process of manipulating a Machine Learning (ML) model, specifically a GPT-2 Transformer (or simply GPT-2) neural network, it is necessary to import the neural network as a NetModel into a Wolfram|One Workbook.
In[]:=
gpt2Model = NetModel[{"GPT2 Transformer Trained on WebText Data","Task"->"LanguageModeling"}]
Out[]=
NetChain
The model above can also be expanded by hitting the plus symbol and clicking on various layers, NetChains, NetGraphs, etc. This can be done on various dynamic outputs below as well.
We can also output the visuals of the neural network in a much more abstract, but aesthetically pleasing, diagram.
In[]:=
Information[gpt2Model,"SummaryGraphic"]
Out[]=
However, the SummaryGraphic is not as accessible as a NetModel of GPT-2.
Tokenization is extremely important when talking about GPTs. Tokens are a measurement that Natural Language Processors (NLPs) utilize to break down inputs; these can be based on words, sub words, or characters. Below, there are some examples of tokenization [4].
◼
“Let us learn tokenization.”
◼
Words : [“Let”, “us”, “learn”, “tokenization.”]
◼
Sub words : [“Let”, “us”, “learn”, “token”, “ization.”]
◼
Characters : [“L”, “e”, “t”, “u”, “s”, “l”, “e”, “a”, “r”, “n”, “t”, “o”, “k”, “e”, “n”, “i”, “z”, “a”, “t”, “i”, “o”, “n”, “.”]
These different types of tokenization each have their pros and cons. The general consensus is that the more broken down tokens are, the more the semantic meaning of the words is lost. For example, tokenization by character will look at each individual letter, instead of the message these letters are providing at a broader scope. The GPT-2 Transformer tokenizes in the form of sub words.
To actually utilize this neural network and tokens, the function generateSample has to be constructed. In this case, the function has tokens set to 10 and a temperature of 1. [5]
In[]:=
generateSample[languagemodel_,input_String,numTokens_:10,temperature_:1]:= Nest[ Function[ StringJoin[#,languagemodel[#,{"RandomSample","Temperature"->temperature}]]],input,numTokens];
Neural Network Structure
Neural Network Structure
Neural networks in the Wolfram Language are structured differently than standard neural networks. Specifically, traditional neural networks are composed of nodes, which are the basic computational units. Comparing this to the brain, each node can be thought of as the cell body of a neuron. These nodes are arranged in layers—input, hidden, and output layers—where each layer is a stack of nodes. The input layer receives raw data, much like how dendrites receive signals in biological neurons. These signals are then passed through connections known as edges, which are analogous to axons. Each connection has an associated weight, determining the importance or strength of the signal. The signal reaches the next node, comparable to an axon terminal delivering neurotransmitters. In the hidden layers, each node applies a weighted sum of the inputs, adds a bias, and passes the result through an activation function. These layers are responsible for transforming and extracting deeper features from the data. The input layer, however, does not perform computations; it simply forwards the data into the network.In Wolfram, however, layers are the smallest unit of neural nets. There are many different types of layers, some with biases, weights, etc. which we will delve into in greater detail later. The largest units are NetGraphs, which are “used to combine multiple inputs or outputs for [various] operations” and thus, string together various components and form a robust system. Below this, there are NetChains, which are a special case of NetGraphs, but they are simply a pipeline of layers. Everything mentioned here can be depicted as nodes, as seen above, and they can also be viewed in a list, especially layers and ports. Ports are not exactly like edges, but they receive input and output information. Another key component of ports is that the output port of one layer must match the input port of another.
Intricacies and Structure of a Neuron [6]
Intricacies and Structure of a Neuron [6]
Wolfram Neural Networks [7]
Wolfram Neural Networks [7]
GPT-2 Transformer’s model is similar to the representation of ColorNet. However, it is more dense and complicated. The above image depicts an abstract version ColorNet of a Wolfram neural network, in the form of a NetChain, which is useful for sequential layers, and NetGraphs, more used in complex structures. In this case, a NetChain can contain both NetGraphs and more NetChains, as well as various other layers.
Layers are the simplest building block of Wolfram neural nets. They, like nodes in standard neural networks, represent groups of artificial neurons and nodes and perform certain operations on tensors, which are representations of vectors and arrays as lists.
Every component discussed above (NetGraphs, NetChains, layers) can be represented through nodes. By selecting a node, you are able to access and view all of its properties. Oftentimes, there may be more NetGraphs and NetChains within outer NetGraphs and NetChains, but at the very core, there will be various types of layers that I will delve into in greater detail later.
Layers are the simplest building block of Wolfram neural nets. They, like nodes in standard neural networks, represent groups of artificial neurons and nodes and perform certain operations on tensors, which are representations of vectors and arrays as lists.
Every component discussed above (NetGraphs, NetChains, layers) can be represented through nodes. By selecting a node, you are able to access and view all of its properties. Oftentimes, there may be more NetGraphs and NetChains within outer NetGraphs and NetChains, but at the very core, there will be various types of layers that I will delve into in greater detail later.
To get a better grasp of the size of items in a neural net, we can use the Length function, as seen below.
This line outputs the length of the first index in the GPT-2 model, which is the number of nodes for NetGraphs responsible for embedding.
In[]:=
Length[gpt2Model[[1]]]
Out[]=
4
Similarly, these are the number of nodes related to NetChains responsible for decoding, as they are in the second index of the model’s list.
In[]:=
Length[gpt2Model[[2]]]
Out[]=
13
Below, these three lines output the length of the number of nodes related to layers; however, layers do not have nodes so it will output zero.
In[]:=
Length[gpt2Model[[3]]]Length[gpt2Model[[4]]]Length[gpt2Model[[5]]]
Out[]=
0
Out[]=
0
Out[]=
0
In addition to seeing length, it is also possible to view the size of the neural net through Information. Here, I examined the size of the 13 NetChains, labeled “decoder” in the GPT-2 model, and the four NetGraphs, labeled “embedding” in the GPT-2 model, using Information. Holistically, there is a greater number of megabytes for the thirteen NetChains, almost two times more than the number of megabytes in the four NetGraphs—which is logical due to there being almost three times more NetChains than NetGraphs.
In the output for the NetGraphs, you can see that the number of nodes is found under the “Layers Count”. In this case, the “embedding” region has four layers (one inside each node), which was observed above, while the “decoder” region has many layers in each of the eleven NetChains, plus a NormalizationLayer, which I will manipulate later.
In[]:=
Information[gpt2Model[[1]]]
Out[]=
However, if we look at individual nodes, it is noticeable that the first node of the NetGraphs is much larger than the first node of the NetChains. In fact, it comprises almost 99% of the “embedding” region, indicating that, despite having solely one layer (like the other 3 nodes) it is still highly important to GPT-2’s function. The rest of the nodes have very few megabytes in comparison, especially the second and fourth node, which both have zero bytes. Meanwhile, the first node of the “decoder” region is equivalent to ten other NetChains. As I mentioned earlier, there is a NormalizationLayer which is, understandably, much smaller than the NetChains.
Accessing the first node inside the “embedding” region.
Accessing the third node inside the “embedding” region.
Accessing the first node inside the “decoder” region.
Accessing the twelfth node inside the “decoder” region.
Recalling using Length to look at the number of nodes inside the “embedding” section, it is also possible to visualize these nodes through indexing instead. Below, you can see that there are 4 nodes shown (each represents a NetGraph) as this reflects the number of NetGraphs that GPT-2 has.
By using the appropriate indexes to get specific parts of the neural network, it is possible to get all of the components in “embedding,” which is located in the first part of the neural net.
Here, we can look more closely at each of the four nodes inside “embedding.”
Using certain indices, I can look closely at specific layers within various NetChains and NetGraphs. In this case, I am displaying an AttentionLayer from the “attention” node inside the first NetGraph, which is inside the first of 12 NetChains, while also defining it as a variable.
Deleting NetGraph, NetChain, and Layers
Deleting NetGraph, NetChain, and Layers
GPT-2 only allows a select few layers and nodes to be deleted. For example, deleting SequenceLastLayer or LinearLayer will produce errors. In some cases, even if these layers or nodes are successfully deleted, they will later cause a problem when running the generateSample function, which is seen for deleting all 4 nodes of the NetGraphs or a SoftMaxLayer. Deletions that cause an error only when running a function are all likely due to their importance in processing input prompts and outputting a response, while ensuring that the inputs and outputs are the appropriate sizes for the next layer to process.
It is feasible to delete all 13 NetChains (the decoder) and generate an output where more repetition occurs in the output, so long as the temperature is exactly 1. Temperature is a parameter that controls the randomness and creativity of the generated text [8].
It is feasible to delete all 13 NetChains (the decoder) and generate an output where more repetition occurs in the output, so long as the temperature is exactly 1. Temperature is a parameter that controls the randomness and creativity of the generated text [8].
Deleting the “decoder” region, which is in the second index of GPT-2, including all 13 of its nodes.
To see how the deletion affects the transformer, we can run both deletedDecoder and the original with the generateSample function.
Though there is gibberish in this output (as it has not been trained during the project) repetition is not prominent.
Moreover, you can delete a “cluster” of nodes, as in all of the NetChains (which are composed of 13 nodes) but you cannot access one of the 13 nodes and delete it, even through indexing.
Moreover, you can delete a “cluster” of nodes, as in all of the NetChains (which are composed of 13 nodes) but you cannot access one of the 13 nodes and delete it, even through indexing.
Below, although the two lines of code are different from each other, both output the same exact result due to Delete failing to support removing a single node from a “cluster.”
Finding Patterns
Finding Patterns
Continuing from the previous section, after observing that deletedDecoder created more repetition, I created a function that showed how many times a certain string occurred more than five times (which is seen as significant). This function allows one to confidently grasp whether or not there are any noticeable anomalies when generating text with the GPT-2 model that has its “decoder” region removed. Thus, it contributes to the goal of determining whether manipulating certain aspects of a neural network causes the output to vary from the original.
The repeatingString function counts how many times a single string appears within a larger string. Then, it checks if the repetition of this string occurs more than five times, as that is what I determined to be significant.
Generating Samples with the Altered Model and Observing Significance
Generating Samples with the Altered Model and Observing Significance
To better observe significance, I looked at a larger set of samples, and then applied the repeatingString function for both the unaltered and altered models.
The repeatingString and Table functions were used to generate 100 different lists and check for a repetition of a string occurring at least five times in every list.
Below, you can see how many times a certain string repeated, and what exactly the repeating string is. It is quite obvious that deletedDecoder has significantly more instances of repetition than the original. It is also noticeable that in addition to more cases of repetition, the deletedDecoder contains actual words instead of the repeated spaces or symbols seen in the original model.
After defining the values of the list of repeating strings, the empty values—which previously held strings that repeated fewer than five times, were deleted.
While these lists are helpful, it is also possible to visualize these results in the form of a bar chart and compare them from a more visual perspective.
Accessing Weights and Biases
Accessing Weights and Biases
While delving deeper into neural networks in the Wolfram Language, I frequently used the function NetExtract. It is especially useful when replacing layers, as using indexing may fail or reduce readability. For example, the association below has the key of “embeddingtokens” with a value of an EmbeddingLayer.
Here, NetExtract is breaking down what exactly is in “embedding” and it formats each part in an association.
Below is a possible way to get the weights for “embeddingtokens” in “embedding,” using Normal (which displays everything as a list) and NetExtract. However, this line of code is only for 1 of 50257 sections of weights.
However, this list of weights is quite exhaustive. If I only wanted to focus on a very limited set of weights, I could use indices to get a set number of weights.
With indices, I am accessing the first 50 weights in “embeddingtokens.”
Manipulating Layers
Manipulating Layers
Manually Manipulating and Replacing LinearLayer (or “classifier”) in GPT-2
Manually Manipulating and Replacing LinearLayer (or “classifier”) in GPT-2
A LinearLayer is a type of basic layer in Wolfram, one that utilizes both weights and biases. Specifically, they multiply the weights, which can be represented with “w,” with whatever data is inputted into the neural network. This input can be referred to as “x.” Then, they add the biases (“b”) to the product of the weights and input, similar to a linear equation.
Therefore, the equation the LinearLayer uses looks a linear equation: wx +b
Manipulating the input with weights and biases is vital as when information is first inputted, there are no parameters used to manipulate them. Thus, the “embedding” LinearLayer is the first section of the neural net that will process the input, before sending the information off to other sections.
Therefore, the equation the LinearLayer uses looks a linear equation: wx +b
Manipulating the input with weights and biases is vital as when information is first inputted, there are no parameters used to manipulate them. Thus, the “embedding” LinearLayer is the first section of the neural net that will process the input, before sending the information off to other sections.
Once again, I am using NetExtract to look at the classifier of the GPT-2 Model (which is a LinearLayer) from GPT-2 to observe all of its properties, while also defining it as a variable to avoid indexing in the future.
Next, I am extracting the variable classifier, that I defined above. It stores the various pieces of information listed below to an association.
I can also use this method to solely look at the Weights inside “classifier.”
Before replacing any nodes or layers in GPT-2, I reconstructed the LinearLayer based on the information provided by the NetExtract, and I was able to successfully do so.
Now, to manipulate the LinearLayer, because the default value of Biases is set to “None” it is impossible to perform algebraic operations on Biases.
Instead, to manipulate the biases, it is possible to do so by completely replacing them with values I determined myself.
However, as weights already exist in this layer, I was able to simply multiply every weight by an integer.
Similarly to what was done above to biasedClassifier, here, the original weights are being replaced by the manipulated ones.
Generating a sample using the altered biases and weights, respectively,
Manipulating this layer’s biases made it more susceptible to outputting symbols (such as $, %, #, etc.) and numbers. However, manipulating this layer’s weights outputted instances of whole sentences repeating and more cases of “\n” (new line).
Manipulating and Replacing LinearLayer (or “classifier”) in GPT-2 using changeNeuralNet
Manipulating and Replacing LinearLayer (or “classifier”) in GPT-2 using changeNeuralNet
As seen above, it is possible to manipulate the biases and weights for a layer in the Wolfram Language. Thus, taking inspiration from the previous section, I decided to create a function that would manipulate weights and biases in a LinearLayer, and then replace the layer with the manipulated parameters in the GPT-2 model, while still allowing the original model to remain intact.
changeNeuralNet first gets the biases and weights from a LinearLayer in the original neural network and manipulates them. Afterwards, it reconstructs a LinearLayer that is identical to the one originally found, except with the appropriate modifications to the layer that the user entered.
These lines of code use changeNeuralNet to view how manipulating certain parts of the neural network affected GPT-2.
Here, I multiplied four to the weights, and then created new biases and multiplied four to that as well. The output displays how classifier has changed inside the neural net.
Finally, I used the generateSample function and compared the outputs produced by the classifierModel and the gpt2Model.
Comparing the repetition of strings from the altered classifier with the original model.
After defining the values of the list of repeating strings, the empty values—which previously held strings that repeated fewer than five times—were deleted.
Graph visualizing the repetition of certain strings from the classifierModel and the original model.
Thus, when combining both the manipulation of weights and biases in a LinearLayer, the prevalence of repeating strings and characters was quite obvious.
Manipulating and Replacing the Weights of EmbeddingLayer (or “embedding”) using changeNeuralNet
Manipulating and Replacing the Weights of EmbeddingLayer (or “embedding”) using changeNeuralNet
Another common layer found in GPT-2 is the EmbeddingLayer, a type of Sequence-Handling Layer. This layer has no bias parameters, but it does contain weights. EmbeddingLayer operates on one integer, which produces a vector or a rank-1 tensor, a certain rank of integers which produces a rank-k+1 array. Moreover, this layer simply implements components into arrays; thus it does not need biases.
As changeNeuralNet is a function that will only take in a LinearLayer, it must be tailored to EmbeddingLayer.
changeNeuralNet is now an overloaded function: it will take in inputs of various types and appropriately act based on the input provided.
Using the changeNeuralNet function and accessing the right parts of the GPT-2 neural net, I was able to successfully replace the previous EmbeddingLayer with one that has its weights 4 times greater than the original.
Then, I generated a sample with the new EmbeddingLayer model (embeddingModel).
To compare, I generated a sample with the original GPT-2.
Quickly glancing at the outputs, I noticed that repetition in the form of specific strings was not prevalent. However, there were significantly more “\n” characters (new lines) in the output generated by the embeddingModel, and tracked the number of “\n” characters present in samples generated by both models.
I used StringCount on each element in the list.
I then used Select to show that there were five or more characters of “\n” (as that is what I determined to be significant).
As seen below, the number of "\n" (new lines) occurring is more prominent in outputs generated with an altered EmbeddingLayer than the original GPT-2. Thus, manipulating this layer' s weights seems to increase the number of “\n” (new lines) in GPT-2's outputs when compared to the original GPT-2 Model. However, the pattern is not as strong when compared to new line repetition in other edited layers.
Manipulating and Replacing the Biases of NormalizationLayer (found in “decoder”) using changeNeuralNet
Manipulating and Replacing the Biases of NormalizationLayer (found in “decoder”) using changeNeuralNet
The NormalizationLayer, classified as a Training Optimization Layer, has the parameters bias and scaling, not weights. Its purpose is to apply bias to input data across many dimensions in order to normalize it. Manipulating this layer’s biases and scaling help train the net layer.
As the NormalizationLayer is found in the decoder, I used NetExtract, without indexing, to store the decoder.
Then, I used NetExtract on the thirteenth layer of decoder, as that is where the NormalizationLayer is located.
Finally, I continued to overload changeNeuralNet as that would allow users to manipulate NormalizationLayer. Moreover, it allows access to specific areas in the decoder and also manipulates the biases of this function by simply multiplying the biases by four, as they are already present in the layer.
I then used the function and encountered an unexpected situation, the layer was uninitialized, meaning its learnable parameters had not been yet initialized. Thus, if I run generateSample on a neural network with an uninitialized layer, it will output an error.
However, using the function NetInitalize quickly fixed this issue.
Then, I ran various generateSamples with the normModel, varying my tokens and prompts.
After running generateSample various times, using various tokenization sizes and prompts, it was clear that altering the biases of NormalizationLayer in the decoder region results in GPT-2 repeatedly generating “the.” On some occasions, it will also put a comma after “the.” However, as of now I am unsure as to why exactly this occurs, as there is no clear cause for this behavior.
Manipulating and Replacing the Dropout Rate of Attention Biases in an AttentionLayer (inside “decoder”) using changeNeuralNet
Manipulating and Replacing the Dropout Rate of Attention Biases in an AttentionLayer (inside “decoder”) using changeNeuralNet
The AttentionLayer is a type of Sequence-Handling Layer. This layer has no weights, as it implicitly weights components with attention weights (an array of weights with certain dimensions) and no biases, as NormalizationLayer usually handles this aspect (and the NormalizationLayer is also located after the AttentionLayers). A parameter that AttentionLayer does have is dropouts, which represents the chance the layer will drop an attention weight. Dropouts are between the value of zero and one.
First, I defined every essential part needed to replace attentionLayer with a version that has manipulated dropout weights.
NetExtract allows us to view the unchanged AttentionLayer.
I continued overloading the changeneuralNet function, expanding it to accept AttentionLayer.
This version of changeNeuralNet did not have a multiplier and instead used addition, as multiplying anything by zero (the initial value of the dropout rate) would simply result in zero.
To test out my function, I ran the code below, which increased the dropout rate to 90% from 0%.
Similar to the normModel, the attentionModel was also uninitialized, a quick and easy fix.
After initializing the sample, I generated various lists for the attentionModel as well.
However, after running the StringCase test multiple times, a strong pattern could not be established.
A visual diagram using PairedBarChart further demonstrated the lack of a pattern.
ElementwiseLayer - Manipulating Two Types of ElementwiseLayers and Replacing them using changeNeuralNet
ElementwiseLayer - Manipulating Two Types of ElementwiseLayers and Replacing them using changeNeuralNet
The First Method of Manipulating an ElementwiseLayer
The final layer I manipulated was the ElementwiseLayer. This layer applies a unary function, a function with only one operand, to every element of the input array. There are no learnable weights or biases in this layer, as the function is fixed. However, it is still possible to change the function itself.
For both ways of manipulating an ElementwiseLayer, it is necessary to first use NetExtract multiple times to get the ElementwiseLayer from the decoder.
Then, using the layers that were utilized to access the AttentionLayer, with just a couple of tweaks, we can successfully use the changeNeuralNet function to make the appropriate changes to the ElementwiseLayer in the first index of the first NetChain inside the decoder.
I then ran generateSample on elementModel.
The Second Method of Manipulating an ElementwiseLayer
The second way to manipulate the ElementwiseLayer is similar to the first. However, it focuses on replacing the type of function, instead of manipulating the specific values the function uses.
This function will take in a specific function type, such as Rectified Linear Unit (ReLU) and replaces the original function, in this case, Gaussian Error Linear Unit (GELU).
Here is a visual representation of GELU:
In addition, here is a visual representation of ReLU:
There are various types of functions that can be used, and this is usually dependent on the type of neural network. These functions are highly useful as they are based on the “action potential” cycle, a process that neurons undergo when firing. Below is a picture that illustrates the process [5].
Accessing the NetGraph that holds the “gelu” function.
Now, to manipulate the function, this line is actually putting the function in use by looking into the decoder, getting the first NetChain, the second set of NetGraphs, and locating the GELU graph, then replacing it with a ReLU graph.
Although the name of the node remains “gelu,” the graph has shifted its structure to reflect that of a ReLU graph.
I then tested out elementFunctionModel by generating a few samples.
Significance for Both Methods
Between the two models that manipulate the ElementwiseLayer, repetition of words or certain strings was not prevalent, but there were some instances of “\n” being repeated. I decided to use StringCount and compare the significance between the original model and both elementModel and elementFunctionModel.
Using StringCount to identify how many new line characters are found in the sample generated using ElementwiseLayer.
Selecting and displaying the significant values (more than five new lines) from the number of spaces.
There was not a strong case of significance shown for each example.
Even visualizing the results with PairedBarChart better depicts how this is not significant.
Conclusion
Conclusion
In this project, I was able to not only delete and replace key aspects of the GPT-2 neural network, but I was also able to analyze significant patterns that arose while utilizing these altered models. While some of my alterations, such as deleting nodes, were quite drastic, others seemed more subtle to me, such as replacing specific layers. However, it was shocking to see the ripple effects it had on the neural network, namely when I manipulated a single NormalizationLayer, and it essentially reduced the transformer to become completely useless.
Future Work
Future Work
As I continue to manipulate neural networks in Wolfram, I plan to further understand ports, as it could enable me to have a neural network that contains various portions inserted and deleted, but can still produce output. Another important aspect of neural networks in the Wolfram Language that I would like to explore are encoders and decoders. Although I manipulated a decoder, I do not have a concrete understanding of decoders. Moreover, I am ecstatic to explore and modify various parameters that I have overlooked and layers I did not have time to address, namely layers related to image processing. In addition, although I shifted the activation functions of the ElementwiseLayer, which essentially effects the way the neuron is set up and fired, I hope I can use this information in a way that exposes the scientific and mathematical differences between various types of brains. Penultimately, I would love to not only replace one layer at a time in a model, like I was doing in my project, but I am enthralled by the idea of introducing various manipulated NetGraphs, NetChains, and layers. However, although these ideas are all areas I will strive to achieve, my most important goal would be figuring out a way to delete a single node, instead of clusters of nodes, and observing what trends emerge from this. Finally, in terms of broader and much larger goals, I hope to better the representation of artificial neurons in the Wolfram Language and beyond, contributing to the fields of cognitive science and psychology in some way, shape, or form.
Acknowledgements
Acknowledgements
Thank you so much to my mentor, Sophie Chen, as she truly helped me grasp concepts behind neural networks in Wolfram, which is no easy feat. She also constantly proposed various additions to my project, ones that led to fascinating discoveries and bolstered my perspective of Machine Learning. Moreover, I would like to give a big thank you to my advocate, Zoya Brahimzadeh. Her passion for neuroscience is truly motivating, as well as her support during the very beginning, when I had a weaker foundation of the Wolfram Language.
Another thank you to the Lead Teaching Assistants (TAs), Anne Shuai and Bertie Bennett, for constantly reminding me to take a break and ensuring that I would not burnout. In addition, I would like to thank all the TAs for their support and constant check-ins, namely Alisa Zaitseva, Anoushka Muchhal, Averyl Xu, Damala Temur, and Gloria Chan.
I would also like to thank Aayush Dubey for his informative lecture on what exactly standard neural networks are, and how they function. The knowledge I gained from this lecture allowed me to better connect the dots and grasp the intricacies of neural networks in the Wolfram Language. A special thank you to Jenny Wang for her support throughout the program as well.
I must express my gratitude to Shriya Ramanan for providing me inspiration for my project. It is wonderful to see that the project that I wrote about in my application is same one that I build off of in the program.
A huge thanks to the camp directors Eryn Gillam, Rory Foulger, and Megan Davis for making WSRP a memorable and enjoyable summer for everyone and I, as well as Carol Cronin for giving me the chance to attend.
A big shout out to Stephen Wolfram himself for giving us this powerful computing language, as well as allocating a unique and novel project idea for me to embark on. I hope it did it justice!
Finally, I cannot overlook the support received from many friends I have made along the way. Thank you all so much for being there for me, and I am glad some of you felt the same.
Another thank you to the Lead Teaching Assistants (TAs), Anne Shuai and Bertie Bennett, for constantly reminding me to take a break and ensuring that I would not burnout. In addition, I would like to thank all the TAs for their support and constant check-ins, namely Alisa Zaitseva, Anoushka Muchhal, Averyl Xu, Damala Temur, and Gloria Chan.
I would also like to thank Aayush Dubey for his informative lecture on what exactly standard neural networks are, and how they function. The knowledge I gained from this lecture allowed me to better connect the dots and grasp the intricacies of neural networks in the Wolfram Language. A special thank you to Jenny Wang for her support throughout the program as well.
I must express my gratitude to Shriya Ramanan for providing me inspiration for my project. It is wonderful to see that the project that I wrote about in my application is same one that I build off of in the program.
A huge thanks to the camp directors Eryn Gillam, Rory Foulger, and Megan Davis for making WSRP a memorable and enjoyable summer for everyone and I, as well as Carol Cronin for giving me the chance to attend.
A big shout out to Stephen Wolfram himself for giving us this powerful computing language, as well as allocating a unique and novel project idea for me to embark on. I hope it did it justice!
Finally, I cannot overlook the support received from many friends I have made along the way. Thank you all so much for being there for me, and I am glad some of you felt the same.
References
References
1
.Wolfram. (n.d.). Neural networks. Wolfram Language Documentation. Retrieved July 13, 2025, from https://reference.wolfram.com/language/guide/NeuralNetworks.html
2
.Wolfram. (2019, July 17). GPT-2 transformer trained on WebText data. Wolfram Neural Net Repository. Retrieved July 13, 2025, from https://resources.wolframcloud.com/NeuralNetRepository/resources/GPT-2-Transformer-Trained-on-WebText-Data
Based on Wolfram Research’s adaptation of the original Language Models are Unsupervised Multitask Learners GitHub implementation by Radford, Wu, Child, Luan, Amodei, and Sutskever (2019).
Based on Wolfram Research’s adaptation of the original Language Models are Unsupervised Multitask Learners GitHub implementation by Radford, Wu, Child, Luan, Amodei, and Sutskever (2019).
3
.Wu, H. (2024, September 15). GPT-2 detailed model architecture. Medium. Retrieved July 13, 2025, from https://medium.com/@hsinhungw/gpt-2-detailed-model-architecture-6b1aad33d16b
4
.Khanna, C. (2021, July 1). Word, subword, and character-based tokenization: Know the difference. Towards Data Science. Retrieved July 13, 2025, from https://towardsdatascience.com/word-subword-and-character-based-tokenization-know-the-difference-ea0976b64e17/
5
.Ramanan, S. (2023, July 13). Analyzing GPT-2’s brain development. Wolfram Community. Retrieved July 13, 2025, from https://community.wolfram.com/groups/-/m/t/3502609?p_p_auth=PyLL1DUv.
6
.Guy-Evans, O. (2025). An easy guide to neuron anatomy with diagrams (S. McLeod, Ed.). SimplyPsychology. Retrieved July 13, 2025, from https://www.simplypsychology.org/neuron.html
7
.Wolfram. (2017, October 9). ColorNet image colorization trained on ImageNet competition data. Wolfram Neural Net Repository. Retrieved July 13, 2025, from https://resources.wolframcloud.com/NeuralNetRepository/resources/ColorNet-Image-Colorization-Trained-on-ImageNet-Competition-Data/
Based on Wolfram Research’s adaptation of the original Let There Be Color!: Joint End-to-End Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification GitHub implementation by Iizuka, Simo-Serra, and Ishikawa (2016).
Based on Wolfram Research’s adaptation of the original Let There Be Color!: Joint End-to-End Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification GitHub implementation by Iizuka, Simo-Serra, and Ishikawa (2016).
8
.Noble, J. (2024, December 16). What is LLM temperature? IBM. Retrieved July 13, 2025, from https://www.ibm.com/think/topics/llm-temperature
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Analyzing the outputs of an altered GPT-2 transformer model trained on webtext data
By Ananya Premkumar
Wolfram Community, STAFF PICKS, July 10, 2025
https:// community.wolfram.com/groups/-/m/t/3502609
By Ananya Premkumar
Wolfram Community, STAFF PICKS, July 10, 2025
https:// community.wolfram.com/groups/-/m/t/3502609