In this project we train a neural network on a dataset of grid raster images generated with the “Grid[]” function. The images are labeled with the style options (converted to strings) which produced them and the network must learn how to associate images to these style options. In this way, the task can be viewed as a mix between image captioning and a narrow-scoped code generation task. The final system utilizes a Transformer-based architecture to extract the style options, although other architectures such as Convolutional Neural Networks and Recurrent Neural Networks were experimented with for demonstrative purposes as this was my first time working with Neural Nets. Further work may include the use of text recognition to reproduce cell contents (including spanning entries), word-based embeddings, and expansion of the network’s utility on real-world images.

Background

The “Grid” Function

The function “Grid[]” takes a one argument, a two-dimensional list, and returns the elements in a grid arrangement.
The Grid Function Demonstrated on a 4x6 array of “x”s with default style options:
In[]:=
Grid[Table[x,4,6]]
Out[]=
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
Grid[] also takes a variety of options which customize the style of the grid, such as: Dividers, Background, Spacings, Alignment, and FrameStyle.
Adding Style Options:
In[]:=
Iconize[{DividersAll,FrameStyleDirective[Thickness[Large],
,Dashing[{Small,Small}]],Spacings{{{6}},{{1}}},Background
},"Options"];​​Grid[Table[x,4,6],
Options
]
Out[]=
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
The Wolfram Language and System Documentation Center has more information on the technicalities of the function.

Data Generation

The data was generated using the “Grid[]” function in Wolfram Language for which the neural networks will be producing style options. This made it easy to produce text labels corresponding to their respective raster images for training the network. However, the style options for Grid[] are very expressive, which makes it nontrivial to devise a simple and non-redundant scheme for generating grids with random options. (Below are some examples of this principle.)
Producing the same output with Frame and Dividers:
In[]:=
Grid[Table[x,4,6],FrameAll]​​Grid[Table[x,4,6],DividersAll]​​Grid[Table[x,4,6],Dividers{},FrameTrue,FrameStyleRed]​​Grid[Table[x,4,6],Dividers{{1Red,-1Red},{1Red,-1Red}},Frame{}]
Out[]=
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
Out[]=
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
Out[]=
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
Out[]=
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
Overwriting Each-other:
In[]:=
Grid[Table[x,4,6],DividersNone,FrameTrue,FrameStyleDirective[Thick,RandomColor[],Dashed]]​​Grid[Table[x,4,6],Dividers{},FrameTrue,FrameStyleDirective[Thick,RandomColor[],Dashed]]
Out[]=
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
Out[]=
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
More examples from the “Grid[]” documentation page:
In[]:=
Iconize[{{"name","a","b","c"},{"first",1,2,3},{"second",11,5,6},{"third",111,8,9},{"fourth",1111,10,11}},"data"];​​Iconize[Sequence[​​Background{None,{Lighter[Yellow,.9],{White,Lighter[Blend[{Blue,Green}],.8]}}},Dividers{{Darker[Gray,.6],{Lighter[Gray,.5]},Darker[Gray,.6]},{Darker[Gray,.6],Darker[Gray,.6],{False},Darker[Gray,.6]}},Alignment{{Left,Right,{Left}}},​​ItemSize{{10,5,3,3}},​​FrameDarker[Gray,.6],​​ItemStyle14,​​Spacings{Automatic,.8}],"dataOptions"];​​Grid[
data
,
dataOptions
]
Out[]=
name
a
b
c
first
1
2
3
second
11
5
6
third
111
8
9
fourth
1111
10
11
In[]:=
Iconize[{{"",Item["",BackgroundRed],SpanFromLeft},{"",SpanFromAbove,SpanFromBoth},{Item["",BackgroundBlue],"",""},{SpanFromAbove,SpanFromAbove,Item["",BackgroundYellow]}},"modernArt"];​​Iconize[{Dividers->{{2->Thickness[3],3->Thickness[3]},{2->Thickness[5],3->Thickness[3],4->Thickness[3]}},ItemSize->{{1->4,2->9,3->1},{1->6,2->6,3->2,4->2}},​​Frame->True},"modernArtOptions"];​​Grid[
modernArt
,
modernArtOptions
]
Out[]=
For the proof of concept, we simply limited the number of options to the four (enumerated below) which produced the highest degree of variability for little added complexity in the option specifications. ♢ Dividers​ ♢ Background​ ♢ Spacings​ ♢ FrameStyle
Functions to produce the Right Hand Side (RHS):

Demonstrative RHS: All RHSs follow this same structure
In[]:=
getDividers[4,2]​​getBackground[]​​getSpacings[4,2]​​getFrameStyle[]
Out[]=
{{False,True,False},{False,False,True,False,False}}
Out[]=
Magenta
Out[]=
{{3,0,3},{0,1,1,3,2}}
Out[]=
Directive[{Thickness[Medium],Dashing[{Small,Small}],LightOrange}]
Further variability was introduced by varying the grid dimensions and cell contents and colors were set using the predefined list of symbols rather than with RandomColor[] to prevent an unruly amount of numbers in the labels.  The full generation script is attached in the “Full Code” section.
Red does convert to RGBColor when evaluated, but we used HoldForm[] to save “Red” as a string in the label:
In[]:=
Red//FullForm
Out[]//FullForm=
RGBColor[1,0,0]
This raster image forms a rule with a string-ified and simplified version of the option specifications to make the labels lighter for the network to process. Since the options are always generated in the same order, we were able to remove the left hand side from the dataset and re-introduce this element after the network has predicted the option specifications.
The following is an example output pair and a rough reconstruction of the style options from the RHS:

Architectures

Methods

Architecture One: CNNs and LSTMs

Architecture Two: Transformers

◼
  • Quick Note: When producing the tokenized vocabulary using Byte-Pair-Encoding (BPE) for the transformer, the data was modified to remove whitespace from the RHS
    • Following the results of the CNN/LSTM network trained on 10,000 images, it was decided that a different architecture was necessary to produce the desired results in the limited time of the school . The network was reconstructed with a vision transformer encoder for image processing and a text transformer decoder to generate the textual outputs. We also expanded the dataset size to 100,000 pairs.

    Results

    Architecture One: CNNs and LSTMs

    Below is the NetTrainResultsObject for sixLSTM.wlnet trained on 10,000 images (although an earlier version of the dataset formatting as you will see):

    Surgery

    Here are some demo the network surgery process (function attached at the end):
    Here are some demos of the network’s textual outputs (Earlier Version of the Dataset).
    Comparing the two outputs:
    ​Prediction:
    {{Left, Null}, {{2, {}}, {2, {}}}, {{}, 2  True}, True, Directive[{Thickness[Small], Dashing[{0, Small}], RGBColor[{0.9, 0.55, 0.2}]}], RGBColor[{0.9, 0.65, 0.35}]}
    ​Ground Truth: ​
    {{Center, Null}, {{3, {}}, {3, {}}}, {2  True, 2  True}, None, Directive[{Thickness[Medium], Dashing[{Small, Small}], RGBColor[0.664398, 0.743061, 0.323964]}], RGBColor[0.888247, 0.995461, 0.964079]}
    So, although the the network error rate accuracy of 95.35%, put into context this means little because the 4.75% that the network is getting incorrect are arguably the most important for reproducing the style options (e.g. RGBColor[] values & divider style options). At most, it can be said that this architecture always predicts the occurrence of curly braces, commas, and spaces (in a sense, memorizing the syntax).

    Architecture Two: Transformers

    Below is the NetTrainResultsObject for transformer3.wlnet trained on a modified dataset of 100,000 {input  output} pairs.

    Surgery

    Extract Network Components from Teacher Forcing Architecture:

    Demo

    Sample image and text from the test set:
    Generate Options for a given image and display:
    Comparing the outputs:

    Full Code

    Reflection and Next Steps

    It works! The final model produces syntactically correct option specification code which represents those shown in the input image. However, there is still room for improvement, particularly in expanding the number of options, inclusion of a wider variety of colors, and increasing the accuracy on the dimensions of the reconstructed grids (this kind of error is shown above). Further steps on this project may include TextRecognize[] or similar methods to reproduce the internal contents of the Grid[]. This project should be expanded upon to increase the reach of its efficacy into Grids which are not generated with the local function.

    Acknowledgments

    Thank you to Maria Sargysan for guiding me in the development of the project, as well as Bob Nachbar and Phileas Dazeley-Gaist for their invaluable assistance with learning the Wolfram Language and the functional paradigm. Thank you Sotiris Michos for the idea of removing the LHS of the option specifications from the training data! I would further like to extend gratitude to Matteo Salvarezza for his assistance throughout the entirety of the project and for letting me use his remote machine to train the networks and generate data. My computer would not have survived the task with 8gb of RAM.
    Last but not least thank you Stephen Wolfram for providing the project idea, and for the late night career advice sessions - lots of fun.

    References

    1. Wolfram, S. (2019). An Elementary Introduction to the Wolfram Language. Wolfram Media.
    2. Wolfram Language & System Documentation Center. (n.d.). https://reference.wolfram.com/language/
    3 . Salvarezza, M. (2024). Neural Networks I. WSS24 Presentations.
    4. Sargysan, M. (2024). Neural Networks II. WSS24 Presentations.

    Cite This Notebook

    “Deduce Grid Styles from an Image of a Table using Neural Networks”
    by Lucas Greenwood​
    ​https://community.wolfram.com/groups/-/m/t/3211022