Introduction

In Stephen Wolfram’s What's Really Going On in Machine Learning? Some Minimal Models blog post, he tried using more simple systems with simple grids of updating discrete rules similar to cellular automata as a base to better understand machine learning. In one part of the post, he briefly mentions:
​
“And along these lines, one can consider all sorts of different computational systems as foundations for machine learning. Here we’ve been looking at cellular-automaton-like and tag-system-like examples. But for example our Physics Project has shown us the power and flexibility of systems based on hypergraph rewriting. And from what we’ve seen here, it seems very plausible that something like hypergraph rewriting can serve as a yet more powerful and flexible substrate for machine learning.”
​
So I decided I would try and jump ahead and go ahead and try implementing Wolfram's hypergraph rewriting system as a base for some kind of machine learning system. Perhaps having networks that can change shape would allow the system to work with more flexible kinds of data. Well, so far I’ve had some amount of success with some preliminary tests, though testing with more kinds of data is something to do in the future.

Code


What I Tried

So I started thinking about how Stephen’s simple systems in the blog post used something he called “rule arrays”, where, where a traditional neural network would have networks of weights that get updated, he simply had an array of rules that would be updated. Rules would say take two boolean inputs and output a new boolean value. He would “train” the rule array by essentially using an evolutionary algorithm to mutate the rule array, and then select ones that were closer to the target function one is trying to match. Below Wolfram is trying to train the network to match this function:
They ended up looking something like the following picture from the the post, where there are two possible rules in each pixel of the array, which are colored red or green below. Inputs are along the top, where a gate is turned on, and then the “path” hits the left on the bottom it’s a 0, when it hits the right it’s a 1, and thus attempts to match the given function:
How could this idea be extended to (hyper)graph rewriting? Well, it’s tricky for me to think about how to make a 2D array of graph rewriting rules. However, at one point in the blog post, Wolfram mentioned the idea of a “vertically layered rule array” where there’s a different global rule at each step of the computation. They looked more like this, where Wolfram is trying to train the VLRA to run for a certain finite number of steps:
I figured this sort of thing was perfectly doable with a list of graph rewriting rules instead of a list of lists of repeated boolean functions. So I picked two network rewrite rules to alternate between, to see if I could get the system to go to some target state. I’m going to call this one a “network rule list”, since network rules can apply in multiple places in the same state, so I don’t really need a whole array like Wolfram did with the array of boolean functions. I picked a rule that makes nice globular structures for one part of the rule list, and for the other part I simply picked its inverse, imagining that I’d have one rule that grows the network, and one that’s likely to match the graph but makes it shrink instead— a balance.
​
The growing rule:
In[]:=
RulePlot@ResourceFunction["WolframModel"][expandRule]
Out[]=
and the “shrinking rule”, being simply its inverse:
In[]:=
RulePlot@ResourceFunction["WolframModel"][Reverse[expandRule]]
Out[]=

Results

I wanted to try and see if I could do something like Wolfram did where he tried to train the network. So I decided I’d give it expected output integers for given input integers. But how would I turn numbers into networks and vice versa? Let’s start with how I I decided to determine how close an output network was to a target integer. I decided to pick something simple (probably something better could be found, such as maybe turning integers into networks and then finding some kind of distance between the output networks and the target network one has now, and to minimize that). The simple thing I picked was to just count the number of vertices in the output network. As a preliminary test, I just tried to see if I could take a simple starting network and try and adapt the rule list, using an evolutionary algorithm, to output some target integer (number of vertices). I seem to have good luck with doing this!
​
Here I set the target number of nodes to be 10.
In[]:=
$target=20;
In[]:=
SeedRandom[1];
In[]:=
adaptedRuleList=
[◼]
AdaptRuleArray
[​​Table[1,20],​​rules,​​100,​​Abs[$target-decoder@ruleList[wolframModelStep][#,{{0,0},{0,0}}]]&,​​"Orientation"->"Vertical"];
Here is my loss function:
In[]:=
ListStepPlot[Last/@adaptedRuleList,PlotRange->All]
Out[]=
And here is the system converging on 10 nodes. Red backgrounds mean the constricting rule is attempted to be applied, green, the expanding rule.
In[]:=
Multicolumn[myNetworkRuleListPlot[adaptedRuleList[[-1,1]],{{0,0},{0,0}}],Appearance->"Horizontal"]
Out[]=
​
​
​
​
Now though we want to do the full training on lists of expected outputs for given inputs. Now we have to find a way to turn given input integers into graphs. I could have just enumerated networks, and picked the nth one in the list, but this would be an issue because some networks don’t actually grow for many steps, even if one keeps using the growth rule. So I after enumerating networks I pre-selected ones that continued to grow after a few steps. My hope is that once they are large enough they will have more places to match the rules and grow, and experimentally it does seem to be the case that if they grow for a little while they’ll keep growing forever. I also added an offset of 10 to the numbers before converting them to graphs so that I could use the function on negative integers, and also possibly test values below the training range. Again, there could possibly be a better way to turn integers into networks, but this is what I’ve tried for now.
​
I attempted to train the network to match this function:
In[]:=
SeedRandom[2];
In[]:=
trained=
[◼]
AdaptRuleArray
[​​Table[1,20],​​rules,​​70,​​myRuleList|->​​Total[​​Parallelize[​​Abs[networkRuleListF[myRuleList,$offset][#1]-#2]&@@@​​trainings]],​​"Orientation"->"Vertical"];
Here is a typical loss function for this system:
In[]:=
ListStepPlot[Last/@trained,PlotRange->{0,All}]
Out[]=
Here it is converging towards the original (dashed) function. It does ok though there could be ways to make it do better.
And here is the final trained network rule array running on each initial condition, just so one can get some idea of what’s going on inside the system:

Future Directions

Some ideas for future directions:
​
Again, perhaps the encoding and deciding could be done better. Maybe a different set of graph rewriting rules will work better. Wolfram Institute fellow Richard Assar suggested evolving the encoder and decoder functions as well, which he says I think is related to something called “Reservoir computing”.
​
I’ve really only tried one or two rules as of this time. Trying other rules would be good to do. I could also try even using a genetic algorithm on the rules themselves at each step, and each step could have many rule parts. Richard Assar also suggested that I could use graph rewriting to rewrite the rewrite rules themselves and sent me a few links along those lines of using rewrite rules to rewrite rules themselves.
​
Or maybe the system will do better if adapted for more steps / using larger rule arrays. Also it might be informative to just do more experiments on Wolfram’s vertical layered rule arrays of boolean functions to see if they work well on training sets to get some intuition from simpler systems before coming back to this system, or to try evolving just a single hypergraph rule to fit a function (related to evolving a single CA to fit a function, mentioned at the end of this section). Or to evolve a list of cellular automaton rules where the data is passed through the whole list, and where each rule is modified independently to see how that works.
​
Another interesting thing to try would be to attempt using other kinds of data than just turning integers into integers. Possibly an advantage of using network systems could be that they could work for more flexible kinds of data than just integers that have the main property of just being a certain size.
​
Trying to fit other more complicated functions, like a sin curve is also thing to try, and also seeing if the function extrapolates outside the training set in a meaningful way at all, which is something Stephen Wolfram looked at with the simpler models on his post about machine learning mentioned above.
​
One other thing is to go back and look at simpler systems to get more intuition about these systems. Along those lines, I have also recently worked on a project where I tried to evolve cellular automata to fit functions.

CITE THIS NOTEBOOK

Using Wolfram’s graph rewriting model as a base for machine learning​
by Eric Parfitt​
Wolfram Community, STAFF PICKS, September 16, 2024
​https://community.wolfram.com/groups/-/m/t/3274891