Where’s Waldo?

Using ImageContents to solve the popular childhood game.
I recently learned about ImageContents, which is new in Mathematica 12. Basically, this function tries to identify entities within an image and returns a dataset with those identities. I thought it might be kind of fun to try it with Where’s Waldo.
We will start with an easy image:
In[]:=
ImageContents
,AcceptanceThreshold0.3
Out[]=
Image
Concept
BoundingBox
Probability
person
Rectangle[{439.672,10.8243},{523.99,177.277}]
0.768212
person
Rectangle[{217.452,120.94},{279.089,275.639}]
0.730109
person
Rectangle[{1047.86,0},{1126.28,158.496}]
0.635373
person
Rectangle[{769.321,101.277},{852.753,261.607}]
0.551654
person
Rectangle[{76.2981,0},{146.296,134.347}]
0.502686
There’s Waldo! He’s the fourth entity the function found. Now, let’s try one a little harder. Since the image is more challenging, I decreased the acceptance threshold. Let’s run ImageContents on this picture:
In[]:=
ImageContents
,AcceptanceThreshold0.01
Out[]=
Image
Concept
BoundingBox
Probability
person
Rectangle[{0,108.437},{2048,1390.81}]
0.670839
automobile
Rectangle[{1668.48,11.5261},{1960.95,158.651}]
0.254533
person
Rectangle[{1325.,29.0125},{1380.07,135.994}]
0.23517
person
Rectangle[{1820.84,220.849},{1869.12,312.377}]
0.211782
person
Rectangle[{1574.42,81.1218},{1634.42,198.616}]
0.198535
person
Rectangle[{192.765,213.648},{234.725,318.089}]
0.166833
person
Rectangle[{1344.61,272.605},{1403.92,356.275}]
0.159248
person
Rectangle[{882.754,505.079},{918.318,602.936}]
0.158373
person
Rectangle[{268.808,453.572},{300.65,534.925}]
0.128339
person
Rectangle[{496.389,0},{552.538,59.8946}]
0.114545
person
Rectangle[{1658.95,421.054},{1715.99,511.98}]
0.106996
person
Rectangle[{1976.81,46.8063},{2020.73,185.536}]
0.106902
person
Rectangle[{798.01,2.00477},{843.179,68.9674}]
0.105498
person
Rectangle[{681.872,6.74407},{728.186,63.0428}]
0.104977
truck
Rectangle[{1445.57,700.517},{1747.11,896.499}]
0.0979186
person
Rectangle[{350.732,244.696},{409.84,332.914}]
0.0876513
person
Rectangle[{1472.48,245.975},{1526.69,327.266}]
0.078849
person
Rectangle[{1146.02,92.9965},{1207.04,163.266}]
0.0708893
person
Rectangle[{1156.23,400.954},{1204.24,473.774}]
0.0606934
person
Rectangle[{694.652,265.238},{759.995,339.445}]
0.0605189
showing 1–20 of 48
There’s Waldo again! This time, he is the third entity the function found. Now, let’s try one even harder.
​
This time I took the Acceptance Threshold as far down as I could before the function stopped increasing the results (capped out at 88 possible entities). Here’s what we get:
In[]:=
ImageContents
,AcceptanceThreshold0.0001
Out[]=
Image
Concept
BoundingBox
Probability
person
Rectangle[{82.1895,12.0289},{950.69,765.009}]
0.669038
boat
Rectangle[{628.202,710.618},{752.4,739.891}]
0.279466
umbrella
Rectangle[{595.104,262.854},{710.265,354.812}]
0.224542
umbrella
Rectangle[{489.711,372.32},{559.879,410.075}]
0.205047
umbrella
Rectangle[{787.902,367.417},{844.744,413.116}]
0.168073
umbrella
Rectangle[{178.876,335.305},{224.062,414.108}]
0.106247
person
Rectangle[{720.577,269.241},{756.135,331.485}]
0.0896135
umbrella
Rectangle[{635.418,282.579},{694.988,332.812}]
0.0828768
boat
Rectangle[{318.229,705.626},{398.039,731.961}]
0.0698757
umbrella
Rectangle[{875.461,287.506},{925.094,323.698}]
0.0654379
person
Rectangle[{421.474,274.953},{446.102,335.786}]
0.0568868
person
Rectangle[{806.777,278.323},{836.511,331.438}]
0.0556303
umbrella
Rectangle[{725.077,361.867},{766.003,404.035}]
0.0549581
person
Rectangle[{799.268,425.647},{842.278,471.29}]
0.0531209
umbrella
Rectangle[{647.228,362.725},{696.282,400.716}]
0.0518062
umbrella
Rectangle[{460.976,327.406},{589.147,421.114}]
0.0527523
person
Rectangle[{345.289,357.534},{376.914,414.326}]
0.0511422
person
Rectangle[{651.557,433.437},{684.655,483.147}]
0.0498688
umbrella
Rectangle[{573.467,360.225},{615.928,403.141}]
0.0482395
person
Rectangle[{264.857,430.42},{289.676,481.674}]
0.0470184
showing 1–20 of 88
Unfortunately, no Waldo detected this time, even after scrutinizing all 88 entries.
I know this notebook isn’t going to necessarily change the game for Mathematica, but I had fun trying out this experiment! I’m looking forward to seeing what this function and the rest of Mathematica 12 will bring to the community.