Analyzing Apple’s NeuralHash surveillance system
Analyzing Apple’s NeuralHash surveillance system
Apple has chosen to deploy a neural network algorithm to scan photos on every Apple device for matches to known images stored in a non-auditable government database. So-called “NeuralHashes” of images on Apple computers and iphones and ipads will be compared to hashes of the database images. It is important to understand the shortcomings of this technology as it has a potential to impact the lives of many people around the world, including those living under oppressive regimes. Apple may be the first to adopt this technology in such a large-scale manner, but many other companies and governments may choose follow Apple’s lead. We must dive into how effective this methodology -based heavily on our fledgling AI systems- really is.
Loading the Neural Network Weights
Loading the Neural Network Weights
Thanks to AsuharietYgvar: https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX
Convert Model Data to ONNX
Convert Model Data to ONNX
Oh lovely, it is nice and easy to find on my iMac...
In[]:=
nnDataFiles=FileNames["neuralhash*","/System/Library/Frameworks/Vision.framework/Resources/",IgnoreCaseTrue]
Out[]=
{/System/Library/Frameworks/Vision.framework/Resources/neuralhash_128x96_seed1.dat,/System/Library/Frameworks/Vision.framework/Resources/NeuralHashv3b-current.espresso.net,/System/Library/Frameworks/Vision.framework/Resources/NeuralHashv3b-current.espresso.shape,/System/Library/Frameworks/Vision.framework/Resources/NeuralHashv3b-current.espresso.weights}
In[]:=
CreateDirectory["nnData"]
copied=(CopyFile[#,FileNameJoin[{"nnData",FileNameTake[#]}]]&/@nnDataFiles);
{nnData/neuralhash_128x96_seed1.dat,nnData/NeuralHashv3b-current.espresso.net,nnData/NeuralHashv3b-current.espresso.shape,nnData/NeuralHashv3b-current.espresso.weights}
In[]:=
/usr/local/bin/brew install lzfse
In[]:=
python3 -m pip install --user onnx coremltools pyzmq
export PATH=$PATH:/usr/local/bin
cd nnData
dd if=NeuralHashv3b-current.espresso.net bs=4 skip=7 | lzfse -decode -o model.espresso.net
dd if=NeuralHashv3b-current.espresso.shape bs=4 skip=7 | lzfse -decode -o model.espresso.shape
cp NeuralHashv3b-current.espresso.weights model.espresso.weights
cd ..
cd nnData
dd if=NeuralHashv3b-current.espresso.net bs=4 skip=7 | lzfse -decode -o model.espresso.net
dd if=NeuralHashv3b-current.espresso.shape bs=4 skip=7 | lzfse -decode -o model.espresso.shape
cp NeuralHashv3b-current.espresso.weights model.espresso.weights
cd ..
In[]:=
1>/dev/null python3 TNN/tools/onnx2tnn/onnx-coreml/coreml2onnx.py nnData
Load the ONNX Model
Load the ONNX Model
In[]:=
model=NetReplacePart[Import["nnData/model.onnx"],"image"->NetEncoder[{"Image",360}]]
Out[]=
NetGraph
In[]:=
seed1=BinaryReadList["nnData/neuralhash_128x96_seed1.dat","Real32"][[33;;]]//ArrayReshape[#,{128,96}]&;seed1//Dimensions
Out[]=
{128,96}
In[]:=
predict[image_]:=((If[Positive[#],1,0]&/@(Flatten[model[image]].seed1))//FromDigits[#,2]&//IntegerString[#,16]&)
Analyzing Network Performance
Analyzing Network Performance
Run the network on an image
Run the network on an image
In[]:=
ex=
Out[]=
In[]:=
neuralHashEx=predict[ex]
Out[]=
f6200dad1c2b6e29f4e73a6c
Image Effect Sensitivity
Image Effect Sensitivity
Let’s check robustness to basic image effects. We will show results in the form of a grid with: {input image, hash, percentage match to original}
In[]:=
hexToBin[x_]:=PadLeft[IntegerDigits[FromDigits[x,16],2],96];computeSimilarity[x_,y_]:=1-(Total[Abs[hexToBin[x]-hexToBin[y]]]//N)/Length[hexToBin[y]]
In[]:=
hexToBin[predict[ex]]//Length
Out[]=
96
In[]:=
computeSimilarity["fa531c6a1a10f0ae4e461e11",neuralHashEx]
Out[]=
0.520833
In[]:=
getPredictionTriplet[aug_]:=With[{pred=predict[aug]},{ImageResize[aug,{128}],pred,PercentForm[computeSimilarity[pred,neuralHashEx]]}]
In[]:=
With[{aug=ImageEffect[ex,#]},Join[getPredictionTriplet[aug],{#}]]&/@{"ColorBoosting","SaltPepperNoise",{"Noise",0.1},{"Noise",0.2},{"Noise",0.5},{"GaussianNoise",0.2},"Deuteranopia","Protanopia","Tritanopia",{"Posterization",32},{"Posterization",16},{"Posterization",8},{"Posterization",4},"Solarization",{"FadedFrame",Scaled[1/5],All,"Cosine"},{"TornFrame",Scaled[1/10],All,30},"Vignette","Decolorization","Sepia","Charcoal","Comics","EdgeStylization","Embossing","Jitter","MotionBlur","OilPainting","Stippling"}//Grid
Out[]=
f6200dbd1d2b6e29f4e73a6c | 97.92% | ColorBoosting | |
f6200dad1c2b6e29f4e73a6c | 100% | SaltPepperNoise | |
f6200dbd1c2b6e29f4e73a6c | 98.96% | {Noise,0.1} | |
f62005bd1c2b6e29f4e73a6c | 97.92% | {Noise,0.2} | |
d62005bd1d2b7e8974e73a6c | 91.67% | {Noise,0.5} | |
f62005bd1c2b7e29f4e73a6c | 96.88% | {GaussianNoise,0.2} | |
f6200dad1c0b6e29f4e73a6c | 98.96% | Deuteranopia | |
f62205bd152b3a29f0e73b6c | 89.58% | Protanopia | |
f6200dad1c2b6e29f4e73a6c | 100% | Tritanopia | |
f6220dad1c2b6e29f4e73a6c | 98.96% | {Posterization,32} | |
f6220dad1c2b6e29f4e73a6c | 98.96% | {Posterization,16} | |
f6220dad342b6e29f4e73a6c | 96.88% | {Posterization,8} | |
f6261ca435297b68d4e73b7c | 81.25% | {Posterization,4} | |
8b72126c02c4459523778c69 | 45.83% | Solarization | |
57c34191b62926a0f8473846 | 65.63% | FadedFrame,Scaled 1 5 | |
722a011517a96608f4463a6e | 77.08% | TornFrame,Scaled 1 10 | |
f62015ad140b6e29f4e73b7c | 93.75% | Vignette | |
f66019ad140b6ea8b4e72a6c | 90.63% | Decolorization | |
f6220dad1c2b6e29f4e73b6c | 97.92% | Sepia | |
1fa8ea6e30c5da550714eee8 | 44.79% | Charcoal | |
ab2c1c2434c76988ad4fae73 | 58.33% | Comics | |
72aa3d2f97ab0a8d34762b74 | 69.79% | EdgeStylization | |
3d28eb6a18054a716744eeac | 58.33% | Embossing | |
7202198d150b66a8b1633bf4 | 77.08% | Jitter | |
e2021dbd140b22a9f0e72a74 | 83.33% | MotionBlur | |
f602398d14096ea9f4e72a74 | 86.46% | OilPainting | |
fa208ca62d2bb901d0ec7b7c | 72.92% | Stippling |
Seems relatively robust to many image effects, but the boarder fade seems surprisingly effective at throwing off the neural network.
The model is surprisingly robust to edge stylization effect, jitter, and motion blur.
The model is surprisingly robust to edge stylization effect, jitter, and motion blur.
Try swapping color channels
Average 10% match accuracy decrease.
let’s try to resize:
Seems pretty robust to image resizing,.
Rotation Sensitivity
Rotation Sensitivity
Now, let’s check out rotations:
ok... So rotations do not work out so great.
Combining Effects and Rotations
Combining Effects and Rotations
Wow, a 0.54 match. This is spectacularly bad. Hopefully the feds know how to rotate images and Apple ‘neural-hashes’ rotated images as well as non-rotated. They should also introduce edge-blur into their training regimen.
So our rotated, boarder-faded image is just as much of a match as a completely new tiger image.
Let’s try an image of a completely different category
Interesting, similar images (tiger) seem to be grouped more closely together in the ‘neural-hash’ space than unrelated images (dog, horse). This could indicate an ability to reconstruct images or families of similar images from a “neuralhash”. Given the alleged nature of these database materials, the results could be distasteful if anyone chooses to realize them.
These similar-looking cats end up with fairly similar hashes. Not very promising for a one-in-one-trillion unique image detector.
Affine Image Transformations
Affine Image Transformations
Not good by any standard.
The issue is that when you are talking about the scale of deploying this algorithm onto everyone’s smart phone and iMac computer, either the threshold will be so high that the algorithm is useless against minor perturbations or there will be so many false positives which need to be manually analyzed by Apple’s employees that any Apple user could be reported and arrested. The preliminary results from analysis done in this work are not looking promising.
Robustness to adversarial category modifications using similar architectures such as MobileNet trained on ImageNet should also be analyzed. Others have shown the effectiveness of adversarial attacks crafted to cause ‘neuralhash’ collisions (making unrelated images look the same to the neural network after adding human-imperceptible noise).
The issue is that when you are talking about the scale of deploying this algorithm onto everyone’s smart phone and iMac computer, either the threshold will be so high that the algorithm is useless against minor perturbations or there will be so many false positives which need to be manually analyzed by Apple’s employees that any Apple user could be reported and arrested. The preliminary results from analysis done in this work are not looking promising.
Robustness to adversarial category modifications using similar architectures such as MobileNet trained on ImageNet should also be analyzed. Others have shown the effectiveness of adversarial attacks crafted to cause ‘neuralhash’ collisions (making unrelated images look the same to the neural network after adding human-imperceptible noise).