Wolfram Cloud Document

TL;DR: I benchmark multiplication accuracy across digit-sizes for three models (GPT-4o-mini, GPT-4o, GPT-5), then show temperature-driven “phase transitions” in output diversity/entropy with a short mathematical theory (a transcritical bifurcation), and finally probe random number generation and the Ultimatum game. All code is below; you can reproduce every figure by running the sections in order. The first figure is the 10×10 multiplication grid; subsequent sections generate the temperature bifurcation and the Prisoner’s Dilemma curves.

Multiplication accuracy across digit size

Experiment design. For each pair of digit lengths

(i,j)∈

1,…,10

I sample $100$ random integer pairs

(a,b)

with exactly

and

digits, query the model with a strict “number-only” instruction, parse the numeric answer, and score correctness against

ab

.The result is a

1010

grid of accuracies (percent correct). I run the same protocol for GPT-4o-mini, GPT-4o, and GPT-5.

Implementation Details

◼

Exact digit lengths are enforced by sampling

a∈[

,i-1

-1]

(analogously for

For GPT-4o-mini

For gpt-4o-mini I run the following code. It is quite straight forward: For the box with coordinates i,j I generate two uniformly distributed random numbers with i and j digits respectively. I then compute their product and check whether the LLM gets the same result in which case I register it as a correct solution. Per box we calculate 100 repeats and estimate the percentage of correct solutions based on that.

Monitor[result=Table[oneProblem=Table[a=RandomInteger[{10^(k-1),10^k-1}];b=RandomInteger[{10^(j-1),10^j-1}];(ToExpression[StringDelete[StringSplit[#,"= "][[-1]],","]]&@LLMSynthesize["What is "<>ToString[a]<>"*"<>ToString[b]<>"? Answer with the NUMBER ONLY! Do not output anything but the number. DO NOT GIVE ME THE EQUATION - just the answer as a number.",LLMEvaluator->LLMConfiguration[<|"Model"->"gpt-4o-mini"|>]])==a*b,{i,100}];Count[oneProblem,True]/100.,{k,1,10},{j,1,10}],{k,j}]

The output I generated was:

{{1., 1., 0.99, 0.99, 0.97, 0.93, 0.79, 0.84, 0.81, 0.82}, {0.98, 0.96, 0.78, 0.48, 0.18, 0.29, 0.28, 0.16, 0.23, 0.21}, {1., 0.87, 0.34, 0.13, 0.02, 0.03, 0.02, 0.02, 0., 0.05}, {0.99, 0.61, 0.16, 0.06, 0., 0., 0., 0., 0.01, 0.01}, {0.93, 0.28, 0.04, 0.01, 0., 0., 0., 0., 0., 0.}, {0.91, 0.25, 0.05, 0.02, 0., 0., 0., 0., 0., 0.}, {0.91, 0.26, 0.07, 0., 0.01, 0., 0., 0., 0., 0.}, {0.84, 0.23, 0.03, 0., 0., 0., 0., 0., 0., 0.}, {0.85, 0.19, 0.03, 0.01, 0., 0., 0., 0., 0., 0.}, {0.8, 0.17, 0.03, 0.02, 0.01, 0., 0., 0., 0., 0.}}

To plot it I use:

Overlay[{MatrixPlot[2*result-1,ColorFunction->"RedGreenSplit",LabelStyle->Directive[Bold,22]],GraphicsGrid[Map[Style[#,15]&,100result,{2}],ImageSize->{400,310},AspectRatio->1.05]},Alignment->Center]

We see in this case that only products of relatively small numbers with one or two digits are computed correctly. Each box shows the number (or equivalently percentage) of correct results. Red indicates few correct results and green indicates many correct results. The red area covers most of the square.

For GPT-4o

For gpt-4o I run identical code, with the exception of the model which now is changed in the LLMConfiguration to gpt-4o.

Monitor[result4o=Table[oneProblem=Table[a=RandomInteger[{10^(k-1),10^k-1}];b=RandomInteger[{10^(j-1),10^j-1}];(ToExpression[StringDelete[StringSplit[#,"= "][[-1]],","]]&@LLMSynthesize["What is "<>ToString[a]<>"*"<>ToString[b]<>"? Answer with the NUMBER ONLY! Do not output anything but the number. DO NOT GIVE ME THE EQUATION - just the answer as a number.",LLMEvaluator->LLMConfiguration[<|"Model"->"gpt-4o"|>]])==a*b,{i,100}];Count[oneProblem,True]/100.,{k,1,10},{j,1,10}],{k,j}]

The output we generated was:

Out[]=

{{1.,1.,1.,0.99,0.98,0.99,0.99,0.97,0.98,0.99},{1.,1.,0.96,0.94,0.88,0.76,0.77,0.67,0.61,0.59},{1.,0.94,0.66,0.35,0.26,0.21,0.13,0.09,0.15,0.14},{0.99,0.87,0.34,0.12,0.01,0.03,0.01,0.03,0.,0.03},{1.,0.86,0.21,0.05,0.,0.,0.,0.,0.01,0.},{0.99,0.79,0.11,0.02,0.,0.01,0.,0.,0.,0.},{0.97,0.7,0.14,0.03,0.,0.,0.,0.,0.,0.},{0.98,0.62,0.14,0.01,0.,0.,0.,0.,0.,0.},{1.,0.59,0.14,0.01,0.,0.,0.,0.,0.,0.},{0.95,0.54,0.17,0.01,0.,0.,0.,0.,0.,0.}}

To plot it I use:

Overlay[{MatrixPlot[2*result4o-1,ColorFunction->"RedGreenSplit",LabelStyle->Directive[Bold,22]],GraphicsGrid[Map[Style[#,15]&,100result4o,{2}],ImageSize->{400,310},AspectRatio->1.05]},Alignment->Center]

The plot shows a slightly expanded “green region”, and as a consequence the red region is smaller. The model does slightly better than gpt-4o-mini as expected.

For GPT-5

For gpt-5 I again run the same code, but for gpt-5.

Monitor[result5=Table[oneProblem=Table[a=RandomInteger[{10^(k-1),10^k-1}];b=RandomInteger[{10^(j-1),10^j-1}];(ToExpression[StringDelete[StringSplit[#,"= "][[-1]],","]]&@LLMSynthesize["What is "<>ToString[a]<>"*"<>ToString[b]<>"? Answer with the NUMBER ONLY! Do not output anything but the number. DO NOT GIVE ME THE EQUATION - just the answer as a number.",LLMEvaluator->LLMConfiguration[<|"Model"->"gpt-5"|>]])==a*b,{i,100}];Count[oneProblem,True]/100.,{k,1,10},{j,1,10}],{k,j}]

The output I generated was (these are expensive numbers!):

Out[]=

{{1.,1.,1.,1.,1.,1.,1.,1.,1.,1.},{1.,1.,1.,1.,1.,1.,1.,1.,1.,1.},{1.,1.,1.,1.,1.,1.,1.,1.,1.,1.},{1.,1.,1.,1.,0.99,1.,1.,1.,1.,1.},{1.,1.,1.,1.,1.,1.,1.,1.,1.,1.},{1.,1.,1.,1.,1.,1.,1.,1.,1.,1.},{1.,1.,1.,1.,1.,1.,1.,0.99,1.,1.},{1.,1.,1.,1.,1.,1.,0.99,1.,1.,1.},{1.,1.,1.,1.,1.,1.,1.,1.,0.99,1.},{1.,1.,1.,1.,1.,1.,1.,0.99,0.98,0.99}}

In this case we observe a dramatic change. the thinking model does substantially better than the previous two “non-thinking” models. We should note that some of the errors could be from time outs of the model. The two gpt-4o model runs were cheap and relatively fast. The gpt-5 model (producing lots of reasoning tokens) ran for 40+ hours and cost about 85+ USD. It might not be the best idea to use it for mass multiplications.

What I observe (summary)

What I observe. In my runs, GPT-4o-mini is reliable for

–

digits and degrades rapidly with more digits; GPT-4o extends the “green” region several columns/rows; GPT-5 is essentially saturated at

≈100

% across the full

1010

grid, with isolated 98%–99% cells.Caveats. These are prompted, closed-book tests with finite sampling

n100

per cell. Accuracy depends on seeds, prompts,

, and model revisions.Uncertainty bands and error structure. For

correct out of

, a Wilson interval for the true proportion



Wilson





, halfwidth



1-





with



kn

and

z≈1.96

for

% CIs.

It seems that there is a huge increase in abilities in GPT-5. As far as I know none of the models is using external tools. The API documentation suggests that they do not use tools, and I do not add tools via the Wolfram Language. I am not sure whether this is emergent behaviour. It is important to understand that GPT-5 is a “thinking” model, i.e. it uses thinking tokens as opposed to the other two models. I might want to look into the thinking process, if not in GPT-5 then perhaps in other models that allow me to see the thinking process.

Temperature sweeps: from determinism to disorder

Next, I want to see how the temperature parameter affects results. I ask a simple question “Give me a colour . Do not answer with anything but the colour!”. I am interested in whether there is a preference for colours or whether it gives us many different colours. For this experiment I use gpt-4o. gpt-5 does not allow to change the temperature.

Protocol. I vary the decoding temperature

and measure output diversity/entropy. For simple prompts (“give me a colour” or “random integer in

1…100

”), diversity is near zero until a threshold

, then increases roughly linearly with

Monitor[results=Table[{k,Table[LLMSynthesize["Give me a colour. Do not answer with anything but the colour!",LLMEvaluator-><|"Temperature"->k|>],{20}]},{k,0.0,2,0.1}],k]

{#[[1]],Length[DeleteDuplicates[StringDelete[#,"."]&/@#[[2]]]]/20.}&/@({#[[1]],#[[2]]/._Failure->"error"}&/@results)

Out[]=

{{0.,0.05},{0.1,0.05},{0.2,0.05},{0.3,0.05},{0.4,0.05},{0.5,0.05},{0.6,0.1},{0.7,0.1},{0.8,0.05},{0.9,0.05},{1.,0.05},{1.1,0.15},{1.2,0.05},{1.3,0.05},{1.4,0.1},{1.5,0.2},{1.6,0.3},{1.7,0.25},{1.8,0.3},{1.9,0.4},{2.,0.5}}

In[]:=

ListLinePlot[%,AxesLabel->{"Temperature","Percentage of different results (in 5% steps)"},LabelStyle->Directive[Bold,15],ImageSize->800]

Out[]=

I only computed 20 requests for each temperature. So 5% is the lowest value I can get (and will for small temperatures correspond to one single colour. For small temperatures we observe mostly one colour (few times two and one time three). After about a temperature of 1.3 the number of different outputs increases. Note that for very high temperatures, the model might produce nonsensical outputs as well.

Next I try a very similar example, but this time I ask for a tool instead of a colour. Note that “priming” can have an important effect here. For example, if instead of the prompt “Give me tool . Do not answer with anything but the name of the tool!” I were to ask “You are a medieval blacksmith. Give me tool . Do not answer with anything but the name of the tool!” I might get different results.

The overall behaviour is similar. We start with one tool for small temperature and later the number of different results increases.

Theory (sketch)

This is a very short sketch/idea of why this might happen. The idea is based on the logits and the softmax temperature. (The softmax will determine which colour/tools to finally choose.)

Making Random Numbers

The next point I want to address is the question of “How human-like is the behaviour/decision making” of a Large Language Model.

Humans are notoriously bad at making random numbers. Have a look at this video from Veritasium on Youtube. Some numbers like 7 or 37 are much for frequent when humans are asked to give a random number (between 1 and 100). Have LLMs inherited this “flaw”? Let’s check. WE ask gpt-4o to generate a random number between 1 and 100. We ask 1000 times.

Here is the surprising result:

Next, I can change that a bit if we increase the temperature:

Now we have many more different numbers. There are still pronounced peaks at 57 and 42, but many more numbers appear.

How does that look if we change the temperature more systematically?

I think that we need to add a bit of jitter to it as the points are “on top of each other”:

So it appears that we get more variation. We are still far away from a uniform distribution though. Also some numbers like 57 and 42 are still very common. Let’s do one more visualisation of the same data:

So it appears that gpt-4o has a preference for certain numbers, but it is possible to make produce a wider spectrum of numbers if the temperature is increased. Here I only show results for a “non-thinking” model, but it is interesting to look at the thought process of reasoning models.

The Ultimatum Game

So gpt-4o has number preferences just as humans (even though different numbers). What about other features.
The Ultimatum Game is a classic experiment in behavioural economics and game theory involving two players: a proposer and a responder. The proposer is given a sum of money (or another divisible resource) and suggests how to split it between themselves and the responder. The responder then chooses to either accept or reject this offer.
If the responder accepts the proposal, the money is divided as proposed; if the responder rejects it, both players receive nothing. The game is typically played as a one-shot interaction, so there are no future consequences or negotiations—the responder’s decision is final.
From a purely rational, self-interested perspective (classical economics), the proposer should offer the smallest possible non-zero amount, and the responder should accept any non-zero offer, as something is better than nothing (Nash Equilibrium). However, experiments consistently show that proposers tend to offer between 30% and 50% of the available sum, and responders often reject low offers out of a sense of fairness, even at a cost to themselves. This demonstrates the important role of social preferences—such as fairness, reciprocity, and altruism—in human decision-making.
Let’s play that with GPT and see whether it accepts non-zero offers following the Nash Equilibrium or whether it punishes unfairness.

So there is a clear transition around 20 USD. The “fair” amount would be 50USD, but below 20USD the model “retaliates”: it does not accept any money because this is not a fair offer....

Conclusion

CITE THIS NOTEBOOK

From arithmetic to phase transitions: how multiplication, temperature, and game theory reveal LLMs
by Marco Thiel
Wolfram Community, STAFF PICKS, October 27, 2025
https://community.wolfram.com/groups/-/m/t/3565492