A Hopfield network stores memories in the weights between neurons, which lets me treat the deletions of those weights as a simulated neurodegenerative disorder, such as the synaptic loss seen in Alzheimer’s disease. By independently altering three quantities, namely the fraction of connections destroyed (synaptic loss), the number of patterns stored, and the distortion of the input memory, to be able to argue that the memory in a Hopfield network degrades in a silent manner up until a certain point, where it then collapses catastrophically. This pattern of silently degrading and then reaching a tipping point mirrors the way the human brain can often go years without symptomatic damage, and the Hopfield model offers a computational intuition for why Alzheimer’s tends to go unnoticed for a long period of time, which is then followed by a rapid decline.
Introduction
Introduction
This paper builds up to a model of memory failure in stages.
I start with a multilayer perceptron trained to classify MNIST digits. This shows how a neural net stores info in its weights, but it also highlights a problem, a perceptron just maps inputs to labels. It can’t take a corrupted piece of an image and rebuild the whole thing. Human memory is content-addressable, meaning a tiny fragment can bring back the entire memory. A standard classifier just doesn’t do that.
To get to that behavior, I implement the classical Hopfield network. Instead of predicting labels, it stores patterns directly in a weight matrix using a Hebbian rule, recalling them by updating neurons until the network settles. Every stored memory is basically a valley in an energy landscape, so recalling a memory just means the network rolls downhill into the closest valley. I implement this, test that it can actually fix a noisy digit, and plot the energy landscape to show that these memories really are stable. After that, I swap in a modern Hopfield network, which uses a softmax rule to hold way more memories while keeping the same core idea.
Once I have a working associative memory, I move on to the main point of the project. Because a Hopfield network smears each memory across the whole weight matrix instead of keeping it in one specific spot, I can go in, delete random connections, and watch what happens to the network’s ability to remember. In some sense, my project is just a computer simulation of Alzheimer’s disease and involves a physical loss of synapses over time. If I treat a network weight like a synapse, deleting weights lets me model brain damage in a way you obviously can’t do with a real patient.
In the last section, I tweak three things independently: how many weights get deleted (synaptic loss), how many patterns are stored (memory load), and how badly the input cue is messed up (degraded input). I measure how the recall fails using loss curves, and visually check the reconstructed digits to see what’s going on. The point is to watch the memory break down as the network gets damaged, and see if that breakdown tells us anything interesting about how Alzheimer’s works.
The Multi-layer Perceptron Problem
The Multi-layer Perceptron Problem
One of the most famous beginner projects for someone getting into machine learning is the MNIST digit recognition. It consists of 60,000 grayscale images of handwritten digits, and the goal is to create a multilayer perceptron that is able to classify which digit is being shown to it. The basis of this model is to find the probability of the input image being each of the digits it has been trained on and classify it as the one with the highest probability.
Pull the MNIST dataset and extract training and testing data:
In[]:=
resource=ResourceObject["MNIST"];trainingData=ResourceData[resource,"TrainingData"];testData=ResourceData[resource,"TestData"];
Flatten the dataset:
In[]:=
flatTrD1=MapAt[Flatten[ImageData[#]]&,trainingData,{All,1}];flatTeD1=MapAt[Flatten[ImageData[#]]&,testData,{All,1}];
To train the perceptron model, the dataset labels need to be turned into vectors of zeros and a single one at the position of the correct digit. This is called “one-hot” encoding.
NetEncoder turns a value into vectors of zeros and a single one at the position of the correct digit:
In[]:=
encoder=NetEncoder[{"Class",Range[0,9],"UnitVector"}];trKey=Keys[flatTrD1];trVal=Values[flatTrD1];teKey=Keys[flatTeD1];teVal=Values[flatTeD1];trEncoded=encoder/@trVal;teEncoded=encoder/@teVal;trainf=Transpose[{trKey,trEncoded}];testf=Transpose[{teKey,teEncoded}];
Next, the structure of the network is defined, and initialized with random weights
I use the following definitions for the network size:
inputSize is of size 784 as a result of flattening the 28x28 size images from the MNIST dataset
hiddenLayers is instantiated with size of 128 because that is the industry norm
outputSize is of size 10 because there are 10 digits in the number system
inputSize is of size 784 as a result of flattening the 28x28 size images from the MNIST dataset
hiddenLayers is instantiated with size of 128 because that is the industry norm
outputSize is of size 10 because there are 10 digits in the number system
In[]:=
inputSize=784;hiddenLayers=128;outputSize=10;
Assign random weights and start the biases at 0.01:
In[]:=
w1=RandomVariate[NormalDistribution[0,2.0/inputSize],{hiddenLayers,inputSize}];b1=ConstantArray[0.01,hiddenLayers];w2=RandomVariate[NormalDistribution[0,2.0/hiddenLayers],{outputSize,hiddenLayers}];b2=ConstantArray[0.01,outputSize];
Implement the softmax function which is a function that converts a vector of real numbers into a probability distribution where each value is between 0 and 1:
In[]:=
softmax[x_List]:=Exp[x]/ConstantArray[Total[Exp[x]],Length[x]];
Training a neural network has two main parts: the forwards pass to find how the current weights and biases process an input and the backwards pass which help adjust the weights to the optimal values
Forwards pass: generates predictions for the backpropagation algorithm by computing layer 1 (Ramp[] or “ReLU”) and layer 2 (Softmax) states:
forwardsPass[xBatch_]:=Module[{z1,z2,a1,a2},z1=Dot[w1,Transpose[xBatch]]+b1;a1=Ramp[z1];z2=Dot[w2,a1]+b2;a2=softmax[z2];{z1,a1,z2,a2}]
Backwards pass: computes the amount of change that I have to apply to each of the weights and biases based on the inputs of the forwards pass in a process known as gradient descent:
In[]:=
backwardsPass[xBatch_,yBatch_,a1_,a2_,z1_]:=Module[{dz1,dz2,dw1,dw2,db1,db2,reluder,dim},dim=Dimensions[xBatch][[1]];dz2=a2-Transpose[yBatch];dw2=1.0/(dim)*(Dot[Transpose[{dz2}],{a1}]);db2=1.0/(dim)*(Total[dz2,{2}]);reluder=UnitStep[z1];dz1=Dot[Transpose[w2],dz2]*reluder;dw1=1.0/(dim)*Dot[Transpose[{dz1}],{xBatch}];db1=1.0/(dim)*Total[dz1,{2}];{dw1,db1,dw2,db2}]
To set up the training loop in functional programming, split up the training into smaller steps and use recursive functions such as Nest
Set up hyperparameters and save the weights globally:
In[]:=
learningRate=0.1;bestWeights={w1,b1,w2,b2};
Run one round of forwards pass, backwards pass, and gradient descent on a single input image:
In[]:=
imgCounter=0;totalImgs=Length[trainf];currEpoch=0;runOneImage[inptlist_]:=Module[{z1,a1,z2,a2,dw1,db1,dw2,db2,img,target},{img,target}=inptlist;{z1,a1,z2,a2}=forwardsPass[img];{dw1,db1,dw2,db2}=backwardsPass[img,target,a1,a2,z1];w1=w1-(learningRate*dw1);b1=b1-(learningRate*db1);w2=w2-(learningRate*dw2);b2=b2-(learningRate*db2);imgCounter+=1;]
Run all the images of the dataset as one epoch:
In[]:=
epoch[inpt_List]:=Module[{shuffled},{w1,b1,w2,b2}=inpt;shuffled=RandomSample[trainf];imgCounter=0;runOneImage/@shuffled;currEpoch+=1;{w1,b1,w2,b2}]
Run 10 epochs:
currEpoch=0;bestWeights=Monitor[Nest[epoch,{w1,b1,w2,b2},10],{currEpoch,imgCounter}];
Executor code, takes a given input image and finds the highest probability of which digit it is:
In[]:=
classifyDigit[img_]:=Module[{layer1,layer2,imageData},imageData=Flatten[ImageData[img]];layer1=Dot[w1,imageData]+b1;layer2=Dot[w2,layer1]+b2;(Position[layer2,Max[layer2]]-1)[[1,1]]]
Test to see if code correctly classifies the digit:
In[]:=
classifyDigit
Out[]=
4
As seen above, this model is probabilistic and has a strict one way flow to map inputs to static outputs. This model fails to capture how the brain actually process information in an interconnected, dynamic way where memories are associative (the ability to recall information by thinking of related concepts and experiences together), allowing the brain to reconstruct the overall experience from just a fraction or heavily distorted past memory. To replicate this kind of dynamical system, John J. Hopfield developed an architecture where neurons constantly interact with each other through recurrent feedback loops. At the time, this was the closest replication of the dense synaptic connections in the cerebral cortex and later went on to be further expanded into one of the most advanced and biologically accurate model of the brain’s associative memory.
Classical Hopfield Networks (John J. Hopfield, 1982)
Classical Hopfield Networks (John J. Hopfield, 1982)
The famous saying “Neurons that fire together, wire together” inspired one of the earliest models of associative memory for neural networks. Specifically in 1982, John J. Hopfield (who later went on to win a Nobel Prize in 2024 for this work) was able to develop a network based on this principle. Hopfield Networks are a type of recurrent neural network that are built to store and recall patterns. They are specifically useful for their ability to recall a pattern given a partial, noisy, or distorted memory. They were built as an attempt to replicate the associative memory of the human brain. The model was inspired by physics where a physical system naturally falls to the lowest energy state and the network has ‘valleys’ in which memories are stored. This project will further explore the insides of these networks! For the simple scope of this project, the MNIST dataset will be used.
How it works:
How it works:
Consider a hilly surface with bumps and valleys. When you drop a marble on this surface, it naturally rolls into a valley. This is the core principle of a Hopfield Network. Every valley in our surface represents a stored memory of an experience and the marble represents the, usually distorted, memory. As the experience (ball) rolls down into its most stable state at a valley, the model is able to recall that stored memory, effectively recalling an experience from a corrupted or incomplete memory.
The algorithm can be split into 4 main parts:
The algorithm can be split into 4 main parts:
1
.The neurons: The network is made of neurons that are capable of taking on the values of either on (1) or off (-1). Every experience is just a specific pattern of these on and off nodes.
2
.The connections: One of the most important factors of the Hopfield Network is that every neuron is connected to every other neuron except itself, replicating the principle of Hebbian learning (“Neurons that fire together, wire together” ). The weights of their connections are stored and the model learns which neurons like to fire together.
3
.The energy: The model has a hidden property known as “energy.” A stable state has low energy while a chaotic input state has high energy. This is important because it serves as a mathematical guarantee that the model will converge to a state of stability. For classical Hopfield networks the formula is: E =
-
1
2
N
∑
i=1
N
∑
j=1
w
ij
s
i
s
j
4
.The recollection: When the model is fed a distorted image, it is considered as a high energy state. The network will change one neuron at a time in order to lower the energy until it hits the perfect, low-energy state which it will output as the recalled memory.
Main Implementation:
Main Implementation:
The first step is to collect the memories that the model should remember and set up the model hyperparameters.
Note: In ideal conditions, the original Hopfield Network is capable of storing ~0.138 * N patterns, where N is the number of neurons in the network. However, in this example a lot of the digits have similar makeups such as pixels in the center in a slanted manner. As a result, the model will only be shown two patterns to memorize.
Pull the first 2 images of the dataset and convert every pixel to either -1 or 1:
In[]:=
gathered=Keys[GatherBy[resource,Last][[All,1]]][[1;;2]]nonBinaryDS=N/@Flatten/@ImageData/@gathered;ds=Map[If[#>=0.5,1.,-1.]&,nonBinaryDS,{2}][[1;;2]];
Initialize the model with 784 neurons for the 28x28 = 784 pixel possibilities in the desired memories:
Compute the connection matrix values based off Hebbian Learning:
Take the sum of the product of every node, zero the diagonal, and normalize:
Implement the energy function to check for convergence
Take the the negative half of the dot product of the state and weight
Updates the network one neuron at a time:
Repeat onePass until the energy reaches a state of stability:
The initial memory has to be distorted to some extent to analyze how well the model performs.
Randomly choose a number of pixels proportionate to the input percent and randomly turn them black or white:
Display the effects of the distortion method:
The final step is to run a test image to run a distorted image through the whole pipeline
Run the code with an example of 0.7 distortion because that is a good challenge to the model while still being possible
Visualizing the model’s topography and how the memory converges towards a low-energy state is helpful in understanding this process in a simpler way.
Plot the energy states as a function of the overlap with each memory to see the minimum locations where the model reaches stability (code generated by Claude):
However, as stated above one of the key limitations of the classical Hopfield model is that it is very poor at storing highly similar patterns. For example, if the model was trained on the digits 0-5 there would be a lot of states that the model would combine leading to a phenomenon known as a “spurious state.”
Change the pipeline from processing 2 images to 5 images to show spurious states:
As the recalled memory shows, the model fails to get to an energy state that is truly a state that it was trained on.
Modern Hopfield Networks (Hubert Ramsauer, 2020)
Modern Hopfield Networks (Hubert Ramsauer, 2020)
As seen above the original Hopfield Network is rather poor at storing patterns that are too similar to each other or a large number of them and started to merge them together, leading to incorrect recalls. In 2020, researchers published a paper formalizing the idea of the Modern Hopfield Network (MHN) by changing the simple binary neuron updates to a continuous softmax architecture. This allows the model to retrieve memories more accurately, store exponentially more memories, and operate on high dimensionality data such as images and text. One of the most interesting things they found that the update rule was mathematically identical to that of the transformer model’s attention layers. In other words, models such as ChatGPT use a form of associative memory to find the most relevant patterns to the current input. With this newer, state-of-the-art formulation, the Modern Hopfield Network improves a lot of the shortcomings of Hopfield’s original structure.
How it works:
How it works:
Main Implementation:
Main Implementation:
Like before, the first step is to collect the desired memories to store.
Since the MHN has an exponentially large number of patterns it can store, all the digits can be stored as memories :
Initialize all the new hyperparameters of the model:
The learning stage of implementing the MHN most significantly differs from that of the classical Hopfield model. This is one of the most different steps in comparison to the original Hopfield Networks which rely on Hebbian Learning. Instead of holding correlation sums, the raw patterns are stored directly. This ‘learning’ step is just organizing the data into a matrix.
Store the raw patterns:
To perform inference, a similar tactic to the implementation of the classical Hopfield is employed here. First, the similarities between the current network state and all stored patterns is calculated. The softmax function is applied to this to amplify certain scores, to make sure they are more dominant during the next state update.
Calculates softmax weighted sum of all stored patterns based on their similarity to the current state:
Implement the new energy function:
Apply mhnUpdate until convergence:
Get a random digit, apply 0.6 distortion, and run the inference:
As seen, this model is able to return to a state of stability (6) from this seemingly formless state showing the capabilities of these models.
Simulating Synaptic Degradation and Alzheimer’s Disease
Simulating Synaptic Degradation and Alzheimer’s Disease
One of the biggest benefits of Hopfield networks are that it structurally mimics the brain’s associative memory, so it can be used to model a lot of different biological phenomena. For example, I can tweak the insides of our network to better simulate neurodegenerative diseases such as Alzheimer’s Disease. A healthy human brain can process a distorted memory and trigger the retrieval of the entire experience. However, people with AD experience the symptoms of synaptic loss, where the connections between neurons break down. A Hopfield model can replicate the damage caused by AD by zeroing out nodes a random percentage in the matrix w.
The classical Hopfield models are a better fit to build this analysis upon due to their simple architecture and easy manipulability.
Pull the MNIST dataset and find all possible positions in the matrix
Since the functionality of the following code is going to be reused multiple times, it is simpler to define each of them as modules and use them as needed
Sets the weights (same implementation as classical Hopfield networks)
Randomly samples a given number of positions in the connections matrix to zero out
Runs the pipeline by making the weights, distorting, and inferring the memory state:
Since we are training on the image of the 0, we can code the loss to be the difference of the pixel values of the state and the 0 given in the dataset:
Graphing the association between various metrics and loss shows the quiet collapse of the model architecture as it manages to continue returning to the state of lowest energy seemingly asymptomatic until a collapse that is too hard for it to recover from.
To create a plot to compare the amount of model ‘loss’ in comparison to the number of nodes zeroed out. More points are focused towards the end as that’s where the fine detail of the model collapse is important:
Create a plot to compare the amount of model ‘loss’ in comparison to the amount of distortion:
The graph stays low while the distorted state remains inside the target’s basin of attraction, then rises once distortion pushes the input across the basin boundary (around distortion ~ 0.9), where it settles into a spurious or inverted state.
To create a plot to compare the amount of model ‘loss’ in comparison to the number of memories stored:
The memory load makes the loss rise as more patterns are stored, because the MNIST dataset’s digits are very closely correlated and group up near one of the valleys in the weight space producing interference well before the theoretical 0.138xN limit.
Create the columns for which there will actually be perceptible change to the images as the amount of metrics change
Progressive zeros on rows and progressive distortion on columns
The recollection characteristics survives across most of the grid, then degrades in the corner where both stressors are high.
Progressively more memories being stored on rows and progressive zeroing on columns
The matrix of images shows the two stressors compounding and the amount damage the network can absorb slowly shrinks as more memories are packed in.
Progressively more memories on rows and progressive distortion on columns
As both memory load and distortion increase the matrix shows how a heavily loaded network loses its tolerance for corrupted seeds and they all merge into one highly confident but wrong spurious state.
Combining the findings from the two, it is clear that there is some evidence that that memory in an associative memory system does not degrade proportionally to the damage done to it. From the list line plots, the error stays flat while a huge chunk of synapses are destroyed, then climbs steeply after hitting a threshold. The grids show what that failure actually looks like. After the threshold, reconstructions often don’t just fade into static, but lock onto the wrong stored digit, outputting a confident but false memory. The two-variable image grids take this a step further by showing that structural damage and memory load compound. The network’s breaking point is a moving target, based on the current number of patterns it holds. The more the number of patterns, the less synaptic destruction it can handle before breaking. In the context of Alzheimer’s, this provides a computational explanation for how a system can silently tolerate massive connection loss, fail abruptly once its reserve is spent, and produce blended or mistaken memories rather than just not knowing the proper answer. The main takeaway is that simply deleting weights in the most basic associative model natively reproduces all of these dynamics.
Combining the findings from the two, it is clear that there is some evidence that that memory in an associative memory system does not degrade proportionally to the damage done to it. From the list line plots, the error stays flat while a huge chunk of synapses are destroyed, then climbs steeply after hitting a threshold. The grids show what that failure actually looks like. After the threshold, reconstructions often don’t just fade into static, but lock onto the wrong stored digit, outputting a confident but false memory. The two-variable image grids take this a step further by showing that structural damage and memory load compound. The network’s breaking point is a moving target, based on the current number of patterns it holds. The more the number of patterns, the less synaptic destruction it can handle before breaking. In the context of Alzheimer’s, this provides a computational explanation for how a system can silently tolerate massive connection loss, fail abruptly once its reserve is spent, and produce blended or mistaken memories rather than just not knowing the proper answer. The main takeaway is that simply deleting weights in the most basic associative model natively reproduces all of these dynamics.
Conclusion
Conclusion
The results indicate that Hopfield networks exhibit a sharp, sudden phase transition under damage rather than linear degradation. Across all of the three tested memory inhibitors, the model’s ability to recall remains near-perfect states within a specific range before dropping abruptly. The network tolerates massive synaptic ablation, handles heavy initial corruption, but due to the nature of the MNIST dataset, the number of memories stored reaches a spurious state quickly.
Framing this in the context of neurodegeneration and Alzheimer’s, there are three core takeaways:
1) Distributed resilience: Because representations are distributed across the weight matrix, isolated synaptic deletions don’t erase specific memories. This is very similar how the brain’s cognitive reserve is theorized to mask early-stage neurodegeneration.
2) Non-linear failure: The degradation is a tipping point, not a gradual slope. This offers a computational intuition for why cognitive ability declines rapidly after the first symptoms occur.
3) Limit-dependency: A network operating close to maximum capacity is more prone to collapse, showing that identical inputs can result in very differnt outcomes depending on the system’s mental stress.
Perhaps the most important conclusion to draw from this is that even though classical Hopfield models are some of the most basic models of the brain’s associative memory, there are still some intrinsic properties shared with the actual biological systems that govern how we think!
Framing this in the context of neurodegeneration and Alzheimer’s, there are three core takeaways:
1) Distributed resilience: Because representations are distributed across the weight matrix, isolated synaptic deletions don’t erase specific memories. This is very similar how the brain’s cognitive reserve is theorized to mask early-stage neurodegeneration.
2) Non-linear failure: The degradation is a tipping point, not a gradual slope. This offers a computational intuition for why cognitive ability declines rapidly after the first symptoms occur.
3) Limit-dependency: A network operating close to maximum capacity is more prone to collapse, showing that identical inputs can result in very differnt outcomes depending on the system’s mental stress.
Perhaps the most important conclusion to draw from this is that even though classical Hopfield models are some of the most basic models of the brain’s associative memory, there are still some intrinsic properties shared with the actual biological systems that govern how we think!
Future Work
Future Work
1) My model for simulating Alzheimer’s Disease is based off the classical Hopfield architecture because it is easier to manipulate, but a more biologically accurate version would be to replicate/recreate the study with the Modern Hopfield network architecture.
2) This model is a computational analogy of what happens in the brain as it slowly degrades to Alzheimer’s Disease. Being able to predict when it would happen before it is too late would be a natural extension of this project.
2) This model is a computational analogy of what happens in the brain as it slowly degrades to Alzheimer’s Disease. Being able to predict when it would happen before it is too late would be a natural extension of this project.
References
References
Hopfield Networks Hopfield Network (Discrete) -A recurrent autoassociative network. Recurrent autoassociative network: Contrast with recurrent autoassociative network shown above. (n.d.). Retrieved July 9, 2026, from https://www.dna.caltech.edu/courses/cns187/problemsets/Hopfield_intro.pdf
Belyaev, M. A., & Velichko, A. A. (2020). Classification of handwritten digits using the Hopfield network. IOP Conference Series: Materials Science and Engineering, 862(5), 052048. https://doi.org/10.1088/1757-899x/862/5/052048
17.2 Hopfield Model | Neuronal Dynamics online book. (n.d.). Neuronaldynamics.Epfl.Ch. Retrieved July 9, 2026, from https://neuronaldynamics.epfl.ch/online/Ch17.S2.html
Serban Liviu. (2019, September 10). Hopfield nets and the brain - Serban Liviu - Medium. Medium. https://medium.com/@serbanliviu/hopfield-nets-and-the-brain-e5880070cdba
GeeksforGeeks. (2021, May 30). Hopfield Neural Network. GeeksforGeeks. https://www.geeksforgeeks.org/machine-learning/hopfield-neural-network/
Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G. K., Greiff, V., Kreil, D., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2021). Hopfield Networks is All You Need. arXiv:2008.02217 [Cs, Stat]. https://arxiv.org/abs/2008.02217
Caltech. Accessed July 9, 2026. https://www.dna.caltech.edu/courses/cs191/paperscs191/Hopfield82.pdf.
Anthropic. (2026). Claude (Opus 4.8) [Large language model, used for generating code] https://claude.ai
Acknowledgements
Acknowledgements
Thanks to Stephen Wolfram, Rory Foulger, Megan Davis, Cyrus Taylor, Eryn Gillam, (Directors), Peter (mentor), Dan (advocate), Hendry, Arnay, and Max (TAs), and all the friends I made along the way :)
Additional thanks to Stephen Wolfram for suggesting this project.
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Hopfield networks - a model of the brain's associative memory and progressive degradation
by Deepac Ganesan
Wolfram Community, STAFF PICKS, July 9, 2026
https://community.wolfram.com/groups/-/m/t/3753939
by Deepac Ganesan
Wolfram Community, STAFF PICKS, July 9, 2026
https://community.wolfram.com/groups/-/m/t/3753939