Introduction
Introduction
In this notebook we show how to:
◼
Parse and ingest Mathematica / WL benchmark results posted at Mathematica StackExchange (MSE), like the ones posted here
◼
Do rudimentary data analysis of the benchmark results
The “parsing” is based on the XML objects expressions extraction -- hence, it is, sort of, “universal.”
Assumptions:
◼
There are multiple answers with benchmarks posted
◼
The benchmark results have a certain clear "list of rules" form
The form is provided by:
Needs["Benchmarking`"]
BenchmarkReport[]
BenchmarkReport[]
Remark: This notebook was used for the answer of the MSE question [AA1].
Remark: The benchmark results are taken from the answers of the MSE question [TT1].
Ingestion
Ingestion
Get all answers in a XML object:
In[]:=
xmlObject=Import["https://mathematica.stackexchange.com/questions/234881/benchmarking-with-mathematica-v-12-for-up-to-date-comparision-of-mathematica-acr/235384#235384","XMLObject"];
Get code cells:
In[]:=
lsRes1=Cases[xmlObject,XMLElement["code",{},code_]code,∞];
Drop Benchmark computation commands and convert to WL code:
In[]:=
lsRes2=Map[(ToExpression[Map[If[StringQ[#]&&SyntaxQ[#],#,Nothing]&,#]]&)@StringReplace[#,{"LaunchKernels[]""","BenchmarkReport[]""","Benchmark[]"""}]&,lsRes1];Length[lsRes2]
Out[]=
69
Select lists of rules only:
In[]:=
lsRes3=Select[Flatten/@lsRes2,MatchQ[#,{_Rule..}]&&Length[#]>1&];Length[lsRes3]
Out[]=
31
For each post flatten the metadata and benchmark results into associations:
In[]:=
lsRes4=Map[Association@Cases[Flatten[#/.HoldPattern["Results"r_](Rule@@@r)],_Rule]&,lsRes3];Length[lsRes4]
Out[]=
31
Make sure all associations have the same keys:
In[]:=
lsRes5=Block[{lsAllKeys=Union[Flatten[Keys/@lsRes4]]},Map[Join[AssociationThread[lsAllKeys,"NA"],#]&,lsRes4]];Length[lsRes5]
Out[]=
31
(Otherwise, the obtained dataset might be hard to use in some computations.)
Make a comparison dataset:
In[]:=
Select[Union[Keys/@lsRes5]〚1〛,!StringQ[#]&]
Out[]=
{}
In[]:=
Dataset[lsRes5]
Out[]=
In[]:=
dsBenchmarks=Dataset[lsRes5];dsBenchmarks=dsBenchmarks[SortBy[#BenchmarkResult&]]
Out[]=
Basic analysis
Basic analysis
Summarize the dataset:
In[]:=
ResourceFunction["RecordsSummary"][dsBenchmarks]
Out[]=
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
1 BenchmarkName | ||||
|
2 BenchmarkResult | ||||||||||||
|
3 Data Fitting | ||||||||||||||
|
4 Date | ||||||||||||||
|
5 Digits of Pi | ||||||||||||||
|
6 Discrete Fourier Transform | ||||||||||||||
|
7 Eigenvalues of a Matrix | ||||||||||||||
|
8 Elementary Functions | ||||||||||||||
|
9 FullVersionNumber | ||||||||||||
|
10 Gamma Function | ||||||||||||||
|
11 Large Integer Multiplication | ||||||||||||||
|
12 MachineName | ||||||||||||||
|
13 Matrix Arithmetic | ||||||||||||||
|
14 Matrix Multiplication | ||||||||||||||
|
15 Matrix Transpose | ||||||||||||||
|
16 Numerical Integration | ||||||||||||||
|
17 Polynomial Expansion | ||||||||||||||
|
18 Random Number Sort | ||||||||||||||
|
19 Singular Value Decomposition | ||||||||||||||
|
20 Solving a Linear System | ||||||||||||||
|
21 System | ||||||||||||
|
22 TotalTime | ||||||||||||
|
(Bar) plot results:
Breakdown statistics
Breakdown statistics
Using a `Map`-version of the code above we can do statistical plots per system:
Partition benchmark results per system:
Separate the overall benchmark results from the benchmark component results:
Show the overall benchmark results:
Show distribution statistical plots per system:
(Note, that we have to keep an eye on the number of available benchmarks per system. Some of those numbers are too small here...)
References
References
[TT1] Tugrul Temel, "Benchmarking with Mathematica v.12 for up to date comparison across different machines", (2020), MathematicaStackExchange.
[AA1] Anton Antonov, "How can I ingest and analyze benchmark results posted at MSE?”, (2020), MathematicaStackExchange.