Super Mario Bros. is one of the top selling video games of all time. It is known for it's excellently designed platforming levels which has pioneered the platforming video game genre. The project will also use a Convolutional Neural Network which will help determine whether an array is a Mario Level and proceed to generate them using levels from Super Mario Bros. and its sequel Super Mario Bros. The Lost Levels.
Introduction
Introduction
Mario is known for it's well crafted level design so attempting to generate them will probably not create the same quality but will hopefully create something similar enough. Super Mario Bros. and Super Mario Bros. The Lost Levels have similar sprites which will help in processing the images.
Processing Images of Mario Levels
Processing Images of Mario Levels
I have collected images of levels from Super Mario Bros. and Super Mario Bros. The Lost Levels. They are all Overworld Levels as their level designs don't differ too much and will have a clearer pattern. I then divide the image into a grid of 16x16px and process each square to map the level into an array.
Loading the images
Loading the images
I load the images of the levels and crop any extra stuff whilst removing the alpha channel just to simplify it:
In[]:=
map11 = RemoveAlphaChannel[ImagePad[CloudImport[
https://www.wolframcloud.com/obj/7c0ee3b5-1a3d-4bd8-a092-7f1f
54e4e20d
],{{0,0},{-240,0}}]];map21 = RemoveAlphaChannel[ImagePad[CloudImport[https://www.wolframcloud.com/obj/ac4c0aef-6687-4f31-a240-4620
39f34d96
],{{0,0},{-240,-240}}]];map31 = RemoveAlphaChannel[ImagePad[CloudImport[https://www.wolframcloud.com/obj/9bc488a8-13ba-48e6-a60d-3f03
f80c9e8b
],{{0,0},{-240,-240}}]];map41 = RemoveAlphaChannel[ImagePad[CloudImport[https://www.wolframcloud.com/obj/6bfbbb3f-cd90-4df1-adc4-4646
cc14fb22
],{{0,0},{-240,0}}]];map51 = RemoveAlphaChannel[ImagePad[CloudImport[https://www.wolframcloud.com/obj/17d83563-90ef-4dc9-b97e-5188
3ffdaccc
],{{0,0},{-240,0}}]];map52 = RemoveAlphaChannel[ImagePad[CloudImport[https://www.wolframcloud.com/obj/916d095e-05ba-4759-a8ae-4917
55b9f94a
],{{0,0},{-240,-240}}]];map61 = RemoveAlphaChannel[CloudImport[https://www.wolframcloud.com/obj/2a8960ac-f087-4151-a73d-28f17ad0f96f
]];map71 = RemoveAlphaChannel[ImagePad[CloudImport[https://www.wolframcloud.com/obj/bb972b44-0a50-480b-90a5-042e
4487d72f
],{{0,0},{-240,0}}]];map81 = RemoveAlphaChannel[ImagePad[CloudImport[https://www.wolframcloud.com/obj/02a7838b-6ac5-4491-a550-6414
c3fdd40c
],{{0,0},{-240,0}}]];mapl11 = RemoveAlphaChannel[ImagePad[CloudImport[https://www.wolframcloud.com/obj/65664715-e560-432d-918c-b7b
e4509ecb0
],{{0,0},{-240,0}}]];mapl31 = RemoveAlphaChannel[ImagePad[CloudImport[https://www.wolframcloud.com/obj/9efa1c36-2e01-42f1-8537-ba4
6c5f4f81a
],{{0,0},{-240,-240}}]];mapl41 = RemoveAlphaChannel[ImagePad[CloudImport[https://www.wolframcloud.com/obj/a0861449-e825-42bd-a06d-e56
cdced80a8
],{{0,0},{-240,-240}}]];mapl42 = RemoveAlphaChannel[ImagePad[CloudImport[https://www.wolframcloud.com/obj/d4791678-fb0f-493b-8644-65a
3daa12c09
],{{0,0},{-240,0}}]];mapl61 = RemoveAlphaChannel[ImagePad[CloudImport[https://www.wolframcloud.com/obj/353633ec-e377-4e69-9e62-c67
06633d7cd
],{{0,0},{-240,0}}]];mapl71 = RemoveAlphaChannel[ImagePad[CloudImport[https://www.wolframcloud.com/obj/1993bd2f-d0b9-44ec-83ec-9e5
38491dc2a
],{{0,0},{-240,0}}]];Creating a function to process each block
Creating a function to process each block
Processing each block will help simplify the data whilst not losing any information as Super Mario Bros. was made on a gird.
I get the sprite of each type of block by partitioning the image and taking the position of the block:
In[]:=
ground = ImageData[ImagePartition[map11,16][[-1]][[1]]]|ImageData[ImagePartition[mapl11,16][[-1]][[1]]];passthrough = ImageData[ImagePartition[map31,16][[-6]][[85]]];hard = ImageData[ImagePartition[map11,16][[-3]][[199]]];lucky = ImageData[ImagePartition[map11,16][[-6]][[17]]]|ImageData[ImagePartition[map31,16][[-6]][[17]]];brick = ImageData[ImagePartition[map11,16][[-6]][[21]]]|ImageData[ImagePartition[mapl11,16][[6]][[20]]];tpipe = ImageData[ImagePartition[map11,16][[-6]][[47]]] | ImageData[ImagePartition[map11,16][[-6]][[48]]] | ImageData[ImagePartition[map51,16]〚-5〛[[46]]] | ImageData[ImagePartition[map51,16]〚-5〛[[45]]]|ImageData[ImagePartition[map31,16][[-4]][[69]]] |ImageData[ImagePartition[map31,16][[-4]][[68]]];pipe = ImageData[ImagePartition[map11,16][[-5]][[47]]] | ImageData[ImagePartition[map11,16][[-5]][[48]]] | ImageData[ImagePartition[map51,16]〚-4〛[[46]]] | ImageData[ImagePartition[map51,16]〚-4〛[[45]]]|ImageData[ImagePartition[map31,16][[-3]][[69]]]|ImageData[ImagePartition[map31,16][[-3]][[68]]];bullet = ImageData[ImagePartition[map71,16][[-3]][[20]]]|ImageData[ImagePartition[map71,16][[-3]][[29]]]|ImageData[ImagePartition[map71,16][[-3]][[47]]];
I then use these values to create a function to transfer each block in the image to a number:
In[]:=
ConvertBlocks[image_]:=Switch[ImageData[image],ground,1,hard,1,lucky,1,brick,1,pipe|tpipe,1,bullet,1,passthrough,1,Except["else"],0]
I can now convert all of these images into arrays and check the function working through creating an ArrayPlot:
Out[]=
,
,
,
,
,
,
,
,
,
,
,
,
,
Simplifying and Gathering the images
Simplifying and Gathering the images
Turning the images into a 15x15 array will give me more data to use and also simplify the whole process as it wouldn't need to generate whole levels but instead sections.
I then create every a 240x240px possible square of each level that doesn't intersect the grid boxes of the level and convert each of those squares into a 15x15 array:
In[]:=
maps={map11,map21,map31,map41,map51,map52,map61,map71,map81,mapl11,mapl31,mapl41,mapl42,mapl61,mapl71};imagedataset=Flatten[Table[Table[ImageTake[map,240,{i,i+240-1}],{i,1,ImageDimensions[map][[1]]-240+16,16}(*thxisabel*)],{map,maps}]];dataset=Map[ConvertBlocks,ImagePartition[#,16],{2}]&/@imagedataset;
This gives us 3747 squares as shown from the output below:
Out[]=
{3747,15,15}
Attempting to generate using a single Wolfram Function
Attempting to generate using a single Wolfram Function
I first attempted trying to generate levels using SequencePredict and LearnDistribution to create Mario Levels which gave an ok output but now a very good one. SequencePredict will take each row whilst LearnDistribution will take in the whole array.
Using Sequence Predict
Using Sequence Predict
I trained SequencePredict on each row of a square to see what it would generate:
In[]:=
sp1=SequencePredict[Table[entry[[1]],{entry,dataset}]];sp2=SequencePredict[Table[entry[[2]],{entry,dataset}]];sp3=SequencePredict[Table[entry[[3]],{entry,dataset}]];sp4=SequencePredict[Table[entry[[4]],{entry,dataset}]];sp5=SequencePredict[Table[entry[[5]],{entry,dataset}]];sp6=SequencePredict[Table[entry[[6]],{entry,dataset}]];sp7=SequencePredict[Table[entry[[7]],{entry,dataset}]];sp8=SequencePredict[Table[entry[[8]],{entry,dataset}]];sp9=SequencePredict[Table[entry[[9]],{entry,dataset}]];sp10=SequencePredict[Table[entry[[10]],{entry,dataset}]];sp11=SequencePredict[Table[entry[[11]],{entry,dataset}]];sp12=SequencePredict[Table[entry[[12]],{entry,dataset}]];sp13=SequencePredict[Table[entry[[13]],{entry,dataset}]];sp14=SequencePredict[Table[entry[[14]],{entry,dataset}]];sp15=SequencePredict[Table[entry[[15]],{entry,dataset}]];
Importing pretrained SequencePredicts:
I then created an ArrayPlot to visualize how this would look like:
Using Learn Distribution
Using Learn Distribution
I then attempted to use Learn Distribution to see what it would produce. I need to change the dataset a bit because Wolfram Language assumes that if there's only 0s then it's a NumericalVector instead of a BooleanVector meaning I'll have decimals so I'll simply change one value to a 1 in the first two rows.
I then visualized it the same way I did for SequencePredict:
Overall the results from SequencePredict look better but LearnDistribution did train faster.
Attempting to Generate using Learn Distribution
Attempting to Generate using Learn Distribution
I actually first attempted to generate levels using a GAN but it took too long and the results that came out weren't so good. But the discriminator was working quite well so I decided to replace the generator with LearnDistribution since it trains faster than SequencePredict so I can get results faster. The way I've structured it is that the Discriminator will first be trained on the original Mario Levels and random bad arrays and then LearnDistribution will create their predicted arrays based off learning from the Mario Levels and if the Discriminator thinks what LearnDistribution made was good then it would put it in the arrays for LearnDistribution to learn off from. Whether or not the LearnDistribution's sample was good or not, the discriminator will put it in it's dataset to learn off from as bad.
The Discriminator
The Discriminator
The discriminator is a neural network that determines how close an array is to a Mario Level. The closer the result from putting an array into the discriminator is to 1, the closer the discriminator thinks it's a Mario Level.
It's a simple Convolutional Neural Network:
We also need to re-format our dataset since the neural network can only take in a 1x15x15 array and not a 15x15:
We then need to just make a set of bad data as an example:
Now we need to turn these sets into a set of rules for the neural network to train on:
The LearnDistribution
The LearnDistribution
LearnDistribution will be used to generate new Mario Levels which, if good, will go back into the LearnDistribution.
We also need to give data to the LearnDistribution so we will first give it the Mario Levels and build on to it:
Putting it all together
Putting it all together
We now need to make a few variables to put these all together:
ldlist is every creation from LearnDistribution:
ldsetrules makes the list of rules to make whatever the LD generates bad:
datagen combines ldsetrules with datasetRules:
goodLD are the samples from LearnDistribution going back to LearnDistribution:
The variables distrained will now only be used for the trained discriminator and ld will now only be used for the trained LearnDistribution. The following function has three parts, the first is to format the data to put into the discriminator, the second to train the discriminator and LearnDistribution and the third to create new data.
The following are pre-trained models for use
Finding patterns to make the level more realistic
Finding patterns to make the level more realistic
I'm going to apply general patterns of Mario Levels in order to construct the Mario Level to become more realistic. There are many features in Super Mario Bros such as Bullet Bill Launchers, Pipes, Stairs, Bricks, Lucky Blocks, Clouds and Grass which all have a certain pattern to it. Before using the following functions in this section you have to declare layer1 to 15.
The First and Second Layers (Very Top Sky)
The First and Second Layers (Very Top Sky)
The first 2 layers are always empty:
The Fifteenth and Fourteenth Layers (Ground)
The Fifteenth and Fourteenth Layers (Ground)
The bottom 2 layers are always identical so whatever the bottom layer is the second bottom layer also is .
The following code will generate the bottom layer and the 2nd bottom layer will just be the same:
The Thirteenth, Twelfth and Eleventh Layer (1 Block above Ground)
The Thirteenth, Twelfth and Eleventh Layer (1 Block above Ground)
For the thirteenth layer the following rules are often applied :
If a block is above a gap, remove it:
If there are just two consecutive blocks with nothing else to the left or right of them then it has to be a pipe:
If there is just one block with nothing else to the left or right of them then it can either be a wall or a bullet bill launcher:
If there are more than two consecutive blocks it's most likely a staircase:
The twelfth an eleventh layer then follow the thirteenth layer with pipes varying from sizes and staircases continuing their pattern.
To continue from the pipes:
To continue from bullet bills:
Rest of the layers
Rest of the layers
The rest of the layers either follow a staircase or are bricks and lucky blocks which can be generated from LearnDistribution. Although we did train the LearnDistribution, we will have a certain threshold measured by the LearnDistribution to make sure what it creates is good.
I then apply the LearnDistribution to the layers:
The Stairs
The Stairs
We now need to continue building the stairs which we do after we generate because we want the stairs to have priority.
The Lucky Blocks
The Lucky Blocks
Lucky blocks are often just found on their own.
The following function finds a brick around air and changes it into a lucky block:
Extra Decorations
Extra Decorations
Decorations are not interactable but are still important in a Mario level as it helps making the Overworld environment unique.
I then created the following function to add cloud decorations:
I then created the following functions to add grass decorations:
Turning the array back into an image
Turning the array back into an image
I now want to turn the array back into an image so I've created this function which replaces values back to images
I then use ImageAssemble to stitch the image back together
Wahoo!
Complete Function
Complete Function
I then generate an image collage of a variety of generated levels:
Here's the code to produce just one:
Conclusion & Future Work
Conclusion & Future Work
I found that the overall result of my project was really good! From the discriminator to the generation process, everything went well enough. If I had more time and resources, I would try using a Deep Convolutional GAN just to see how well that would work in comparison. I would also want to try using ROM Hacking to create the level in Super Mario Bros. and use OpenAI Gym Retro to check its playability. I'd also be interested in doing other level styles such as underground, castle or underwater.
Acknowledgements
Acknowledgements
◼
Thank you to my mentor Isabel
Citations
Citations
◼
Super Mario Bros. and Super Mario Bros. The Lost Level Images. The Spriters Resource. (n.d.). Retrieved July 18, 2022, from https://www.spriters-resource.com/
◼
DuplicatesList Function Source. Wolfram Staff. Retrieved July 21 2022, from https://resources.wolframcloud.com/FunctionRepository/resources/DuplicatesList