Modeling migraine aura in convolutional neural networks (CNN)
by Fizra Khan
NED University of Engineering and Technology
by Fizra Khan
NED University of Engineering and Technology
This project is an attempt to model visual hallucinations experienced by an individual during or before the onset of migraine or occipital epilepsy. These hallucinations exhibit in the form of flashing or flickering lights, changes in colour contrasts, blobs, scotoma, random lines or spots. We developed Convolutional Neural Network (CNN) with simple encoder-decoder based architecture for image reconstruction using Neural Network Framework and utilized a pre-trained model from NN repository in Wolfram Language (WL). The visual aura phenomenon was modeled using two approaches; adding sine waves of different frequencies and interfering with filter weights in the encoder.
Convolutional Neural Networks (CNNs):
Convolutional Neural Networks (CNNs):
Artificial Neural Network (ANN) is a computational approach to mimic the connectivity of neurons in the human brain while CNNs are inspired by the visual perception that occur in primary visual cortex of the brain. This deep learning algorithm is based on encoding visual data and process it using learnable filters, hidden weights and biases.
Encoders:
Encoders:
In typical encoders, sequential operations with multiple filters, like convolution, batch normalization, max-pooling are performed to identify feature maps of the input data. For using less computational power and time, following architecture was developed to build an encoder using Convolutional layers in WL having varying filter sizes to compress an RGB image:
Out[]=
NetChain
Decoders:
Decoders:
In a typical decoder, the up-sampling, convolution, batch normalization and regression operations are used to prepare the output by recovering the loss of spatial resolution that occurred in the encoder process. The decoder was developed by using simple Deconvolution and Ramp Layers available in WL.
Out[]=
NetChain
Training of CNN:
Training of CNN:
The training of our encoder-decoder based CNN was performed by using CIFAR-100 dataset available in Neural Network Repository in WL. For saving time and computational power, only 200 images of 32 pixels were used by NetTrain in WL which automatically attaches a loss function to the training.
model101=NetInitialize@NetChain[{ConvolutionLayer[64,3],Ramp,ConvolutionLayer[128,3],Ramp,ConvolutionLayer[256,3],Ramp,DeconvolutionLayer[256,5],Ramp,DeconvolutionLayer[128,3],Ramp,DeconvolutionLayer[32,3,PaddingSize->1],Ramp,DeconvolutionLayer[3,3,PaddingSize->1]},"Input"->NetEncoder[{"Image",ImageSize->32}],"Output"->NetDecoder["Image"]]
Out[]=
NetChain
In[]:=
data1 = ResourceData["CIFAR-100"];
In[]:=
trainingdata101=data1[[1;;200]];
In[]:=
trainedmodel101=NetTrain[model101,trainingdata101[[All,1]]->trainingdata101[[All,1]]];
The training of the above model took around 30 minutes with 10 rounds and 500 batches.
The trained model can be found here: https://github.com/fizrakhan/WSS22-project
Modeling Visual Disturbances
Modeling Visual Disturbances
In neurological disorders like migraine or epilepsy, cortical spreading depression (CSD) is the pathophysiology caused due to inadequate blood flow in the cortical primary visual cortex [1] . During examination, EEG abnormalities such as slow waves, high and spontaneous neural activity have also been detected [1] . This shows that during visual aura, neurons are firing with higher amplitude and with slow speed.
Modeling in Encoder-Decoder based CNN:
Modeling in Encoder-Decoder based CNN:
To model this, we have modified our trained model to add some disturbance by using ElementwiseLayer.
model01 = Import["C:\\Users\\HP\\Documents\\trainedmodel101.wlnet"]
Out[]=
NetChain
modified01 = NetInsert[model01,ElementwiseLayer[0.5Sin[0.5Pi#]&], 6] (*Adding sine waves after third convolution layer*)
Out[]=
NetChain
GraphicsGrid[Table[{model01[trainingdata101[[t,1]]],modified01[trainingdata101[[t,1]]]}, {t,7}]] (*Evaluating results of trained model and modified model*)
Out[]=
Modeling in Pre-trained Neural Network:
Modeling in Pre-trained Neural Network:
Since, Neural Network repository of WL is rich with different pre-trained networks, we have utilized this model “Enhanced Super-Resolution GAN Trained on DIV2K, Flickr2K and OST Data”.
net = NetModel["Enhanced Super-Resolution GAN Trained on DIV2K, Flickr2K and OST Data"]; scape = WebImageSearch[SearchQueryString["Landscape"],"Thumbnails", MaxItems -> 5]; (*Importing images from Websearch*)images = ImageResize[#,{50,50}]&/@scape; (*Resizing them according to net*)net01 = NetReplacePart[net,"Input"-> NetEncoder[{"Image", ImageSize -> 50}]]; (*replacing array input with image input in net*)
Now, applying different variations of sine waves:
In[]:=
ae= NetInsert[net01,"alteration"->ElementwiseLayer[Sin[0.7Pi#]&],"conv2"][images[[2]]];ab = NetInsert[net01,"alteration"->ElementwiseLayer[Sin[0.9Pi#]&],"conv2"][images[[2]]]; (*after convolutional layer 2*)ac= NetInsert[net01,"alteration"->ElementwiseLayer[2Sin[0.5Pi#]&],"conv2"][images[[2]]];ad= NetInsert[net01,"alteration"->ElementwiseLayer[3Sin[0.5Pi#]&],"conv2"][images[[2]]];af= NetInsert[net01,"alteration"->ElementwiseLayer[2Sin[0.9Pi#]&],"conv2"][images[[2]]];
Interfering with weights in pre-trained model:
Interfering with weights in pre-trained model:
Another approach taken to model visual disturbances is interfering with the weights of convolutional layers.
◼
Weights Interference:
Enhancing weights of filter number 33, 43 and 58 by a factor of 10 in first convolutional layer have appeared in scotoma like visuals.
While enhancing weights of filter number 12, 24 and 39 by a factor of 20 in the third convolution layer have appeared in a zig-zag pattern.
Conclusion
Conclusion
During the modeling in pre-trained model, we found that adding sine waves with higher amplitude and lower frequencies gave appearance of a blob while disrupting individual weights in initial convolutional layers provided zig-zag pattern and scotoma type visuals. Encoder and decoder architecture can be optimized by making them more robust and using large dataset for training the network. Mathematical model based on reaction-diffusion equations for cortical spreading and depression (CSD) can be employed to computationally model visual aura in Artificial Neural Network (ANN).
Keywords
Keywords
◼
Migraine
◼
Visual Aura
◼
Convolutional Neural Networks (CNN)
◼
Epilepsy
◼
Artificial Neural Networks
Acknowledgment
Acknowledgment
I would like to extend immense gratitude to my mentor, Jesse Galef, who has patiently guided me throughout this project with his valuable suggestions and help with respect to code, research and development. Moreover, I would like to thank Stephen Wolfram for suggesting us this project.
References
References
◼
[1]O’Hare, L.; Asher, J.M.; Hibbard, P.B. Migraine Visual Aura and Cortical Spreading Depression—Linking Mathematical Models to Empirical Evidence. Vision 2021, 5, 30. https://doi.org/10.3390/vision5020030
◼
[2]O’Hare, L., & Hibbard, P. B. (2016). Visual processing in migraine. Cephalalgia, 36(11), 1057–1076. https://doi:10.1177/0333102415618952