A simple pendulum is perfectly repeatable, while double pendulum may go to chaos. Neural network learns task A, then task B, then task A again as a cycle, acting similar to a pendulum but much more degrees of freedom. After sufficient learning cycles, there are three branches of outcomes, 1. the network oscillates between repeatable weights, 2. the network falls to constant weights, 3. the network still presents chaotic weights. The learning process of a character-level language model also shows some similar patterns that may explain catastrophic forgetting and failures in relearning.

Repeated learning process

Introduction

Neural networks are usually trained to preserve only the learned weights that yield the best performance in a high-dimensional parameter space. The question is how the learning process affects the final weights and the properties of the resulting neural network.
​
From the perspective of mechanics, loading causes changes, and these changes reveal physical properties and laws. For example, when a pendulum at position A is pushed toward position B, its motion reveals the laws of dynamics. Similarly, a neural network can be trained on task A and then retrained on task B, and the difference may reveal hidden laws of learning.
​
Analogue to cyclic loading in solid materials, we can also repeatedly train a neural network from task A to task B and then back to task A. A simple pendulum may repeat its trajectory perfectly between positions A and B, so the question is whether a neural network can similarly repeat its trajectory in high-dimensional parameter space of weights. Cyclic loading may cause material fatigue with irreversible microstructural damage, so whether a neural network can similarly experience a degradation.
​
In the following sections, we first study a neural network cyclically trained between two continuous and differentiable functions. We then analyze the failure modes using various discontinuous and nondifferentiable functions. Finally, the learning process of a language model is compared with previous observations from the simpler neural network.

Chaotic beginning

Two learning tasks are represented by the target functions
f
A
(x)
and
f
B
(x)
. A parameter s is used to move gradually between them :

​
f(x,s)=(1-s)
f
A
(x)+s
f
B
(x),
​
​
When
s=0
, the network learns task A. When
s=1
, it learns task B. Increasing s from 0 to 1 and then decreasing it from 1 to 0 forms one complete learning cycle.
Out[]=
​
s
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
The network contains linear and Ramp layers. ADAM is used in the main experiments, while stochastic gradient descent is also tested, which gives a similar results.
Out[]=
Each training takes 1000 rounds, we log all weights every 100 rounds. After each training, the target function A gently transits to target function B, s increases from 0 (100% target function A) to 0.1, 0.2, finally reaches B (100% target function 2).
During the first 3 training cycles, the weights change in an chaotic way. Many weight trajectories cross and switch order:
Out[]=
Then the orders are stabilized, while values are still developing.
Out[]=

Cyclic growing

20 trainings form one cycle (
f
A

f
B

f
A
). After 6 cycles, weights are in an equilibrium state. Now, weights change from function A to function B and return to function A, with almost identical values. At this stage NN seems aperiodic. After a few more cycles, current equilibrium state is gradually distorted and may form a new equilibrium state, but nothing lasts long. Weight trajectories:
Out[]=
After a 50 cycles, current equilibrium state has weights monotonically changing in a much more stable way.
Out[]=
The distance to the last network decreases, which shows networks are becoming more and more similar, while two branches show that two tasks have different weight distribution.
Out[]=
The average weight magnitude increases and seems will approach a limit.
Out[]=
The weight difference means the amount of changing from
f
A
to
f
B
in networks, which shows the amount of changing is becoming more and more similar, indicating a more stabilized periodic pattern.
Out[]=

Final equilibrium

At this stage, the network seems reached a final equilibrium, which forms a clear periodic pattern.
Out[]=
Graph of weight
Out[]=
Weights are changed also in a periodical way.
Out[]=

Additional case 1-10

The repeatability is checked by additional 10 experiments (different weight distributions for the same target functions).

Various task functions

The smooth functions show that repeated learning can approach a stable cycle. The following cases change the continuity and differentiability of the target functions and also change the activation function. These cases reveal several ways in which cyclic learning can fail.

Case 1. Equilibrium but trapped

When average weight magnitude gradually increases, networks are becoming more and more similar. After many cycles, the weights may all fall into constant, trapped a local minimal point for both tasks that simply changing the learning rate cannot get out of this trap.
Trapped and shown wrong figure:
Weight trajectories:

Case 2. Equilibrium but trapped

Adding a middle step cannot improve this situation.
Trapped and shown wrong figure:
Weight trajectories:

Case 3. Similar to case 2 but LogisticSigmoid function

Change the activation function from Ramp to LogisticSigmoid, the average weight magnitude greatly increases and seems have not approach a limit, even after a long training, so the final state is still undecided. The weights are more and more similar but the weight changes are not similar which still indicates a random developing.
Weight trajectories:

Case 4. Nondifferentiable to differentiable

Nondifferentiable function is more difficult to learn, unless the nondifferentiable function is similar to activation function, the pattern is noticeably more periodic.
Weight trajectories:

Case 5. Nondifferentiable (Activation function) to differentiable

Weight trajectories:

Language model

The final experiment tests if similar behavior appears in a more complex network. A character-level language model is trained. The two corpora of “Beowulf” share many characters, but their character frequencies are different. The task A can be seen as character distribution from “Modern English”, and task B is from “Old English.” They are two related but different tasks.
Dataset for training, some samples:
Net to be trained:

Results

Similarly, during the first 3 training cycles, the weights change in a chaotic way:
The average weight magnitude also steadily increases, while no clear limit at this point.
The networks are getting more and more similar.
Sanity check shows that the trained models still reflect similar character distribution, while low frequency characters are poorly learned.

Concluding Remarks

Neural networks “walk” in a high-dimensional parameter space through repeated learning. The weight trajectories start in chaos and end in order. In the final periodic stage, the trajectories repeat themselves under the identical training procedure. The first few rounds of training are the most chaotic, which determine the most of the weight distribution at the final periodic stage. We present 10 additional cases at the periodic stage, but none of their weight distributions are similar. After pure chaos and before stabilized periodicity, there is a long developing period of training during which the overall weight amplitude increases linearly. For the language model case, we observed the chaotic stage and this developing stage, while, due to time constraint, the periodic stage has not been fully studied. It is still unknown whether a language model can also have weight trajectories that repeat themselves. There are failure cases worth noting. A neural network works well for an individual task after each training in the developing stage but can suddenly fall into failure, such that it can neither perform the trained task nor be trained for the next task. Even changing the learning rate or adding new training samples does not help. Two failure cases fell into the “trap” after 200 and 600 training rounds without a clear pattern. One solution found is to replace the Ramp activation function with the Sigmoid function. It shows that the failure may be caused by the nature of the task itself, the type of activation function, or the training history, which may help explain catastrophic forgetting and failures in relearning as well.

Acknowledgements

I would like to thank Dr. Stephen Wolfram for his advice and guidance on this research, Faizon Zaman for mentoring me throughout the project, Alejandra Ortiz for her coding advice, Enrico for helpful discussions, and Stephanie for all of her great announcements and arrangements. This project is supported by the generous scholarship from Wolfram and the fellowship from Thornton Tomasetti.

References

1
.
Stephen Wolfram (2024), “What’s Really Going On in Machine Learning? Some Minimal Models,” Stephen Wolfram Writings. writings.stephenwolfram.com/2024/08/whats-really-going-on-in-machine-learning-some-minimal-models.
3
.
Jamie Simon, Daniel Kunin, et al.(2026), “There Will Be a Scientific Theory of Deep Learning,” arXiv preprint arXiv:2604.21691.https://arxiv.org/abs/2604.21691
4
.
Ian Goodfellow, Mehdi Mirza, et al. (2014),
“An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks,” International Conference on Learning Representations.
https://arxiv.org/abs/1312.6211

AI Disclosure

The following generative AI tool was used in this project: ChatGPT 5.6. for grammar check. All written content was reviewed, understood and approved by the author.

CITE THIS NOTEBOOK

Learning Mechanics of Neural Network: Chaotic Pendulum Metaphor​
by Jinming Zhang​
Wolfram Community, STAFF PICKS, July 16, 2026
​https://community.wolfram.com/groups/-/m/t/3762346