ORIGINAL ARTICLE: Grabovsky, A., Vanchurin, V., Bio-inspired machine learning: programmed death and replication, Neural Comput & Applic (2023). DOI: https://doi.org/10.1007/s00521-023-08806-4
We analyze algorithmic and computational aspects of biological phenomena, such as replication and programmed death, in the context of machine learning. We use two different measures of neuron efficiency to develop machine learning algorithms for adding neurons to the system (i.e., replication algorithm) and removing neurons from the system (i.e., programmed death algorithm). We argue that the programmed death algorithm can be used for compression of neural networks and the replication algorithm can be used for improving performance of the already trained neural networks. We also show that a combined algorithm of programmed death and replication can improve the learning efficiency of arbitrary machine learning systems. The computational advantages of the bio-inspired algorithms are demonstrated by training feedforward neural networks on the MNIST dataset of handwritten images.
Introduction
Introduction
This work deals with the structure of neural networks and aims to answer 2 questions. First, how to remove a neuron from the net with minimal reduction of the net's accuracy. Second, how to add a neuron to the net to maximally increase its performance. To this end we assign efficiency to each neuron and give two different definitions of this efficiency via weights and signal averages in the paper. This efficiency shows how important the neuron is, i . e . how much the other neurons change when the output of this neuron changes. To answer the second question we proposed several algorithms to initialize the weights of the new neuron and studied performance of the nets with the new neurons added in different ways on the feedforward neural net with the MNIST dataset. The main idea is to split the weights of the neuron with the highest efficiency between this neuron and the new one. To answer the first question, we write how to modify the weights and biases of the remaining neurons to minimize the effect if we remove a particular neuron. We proposed three different algorithms to remove the neuron with the smallest efficiency and tested them on the feedforward neural net with the MNIST dataset. Work of the first of them - algorithm A1 is demonstrated in the notebook. The first section of the notebook initializes a neural net to study the MNIST dataset and gives the function for the A1 algorithm. The second section of the notebook trains the net to~95 % accuracy and applies the A1 algorithm to cut the input neurons with efficiencies less than 5, 10, 20, 30 %% of the maximal efficiency. Next, accuracies of the pruned nets are calculated and the removed neurons are plotted in the 28 x28 field. Accuracies of the pruned and the initial nets are also given after a short period of retraining. The results of the calculation show that one may have a much smaller number of input neurons than 28 x28 = 784 to get good accuracy.
Initialization
Initialization
First, we initialize a net to work with the MNIST dataset as in the Mathematica help
In[]:=
trainingData=ResourceData["MNIST","TrainingData"];testData=ResourceData["MNIST","TestData"];trainAssoc=<|"Input"Keys[trainingData],"Target"Values[trainingData]+1|>;testAssoc=<|"Input"Keys[testData],"Target"Values[testData]+1|>;mnistEncoder=NetEncoder[{"Image",{28,28},"ColorSpace""Grayscale"}];mnistDecoder=NetDecoder[{"Class",Range[0,9]}];
In[]:=
layer2=ElementwiseLayer[Tanh];layer3[n_,n1_]:=LinearLayer[n1,"Input"->{n}];layer4[n_]:=LinearLayer[10,"Input"->{n}];layeroutSM=SoftmaxLayer["Input"{10},"Output"->mnistDecoder];squar=FunctionLayer[{,#}&];
2
#
In[]:=
chain31[pos_,n_]:=With[{l=Length[pos]},NetGraph[{layer3[l,n],layer2,layer3[n,10],layeroutSM,squar,LinearLayer[l,"Input"->mnistEncoder,"Biases"->None,"Weights"->pos,"LearningRateMultipliers"->0],squar},{6->1->2->3->4->NetPort["p Output"],2->5->NetPort["corr 1"],6->7->NetPort["corr 01"]}]]
In[]:=
Information[chain31[IdentityMatrix[784],20],"FullSummaryGraphic"]
Out[]=
Here level 1 in untrainable. Its weight matrix is identity matrix. It serves to switch off the input neurons removing 1 from the diagonal.
Two extra units 3 and 6 return the average neuron' s value and the average squared neuron' s value on levels 4 and 1.
Two extra units 3 and 6 return the average neuron' s value and the average squared neuron' s value on levels 4 and 1.
Pruning algorithm A1 from the paper applied to the input layer only:
In[]:=
reducenicpercent1[net_,threshold_]:=Block{newnetdressed,newnet,newnet1,newnet2,influence,neuronstochange,w33,w11,b11,b32,fp,xs,ys,w3,b3,w1,b1,corr1,corr0,accuracy,corrdelta1,corrdelta0,efficiency2,positions,accuracy1,w91},(*Measurethevariancesoftheinputneuron'svaluesandlevel4neuron'svalues*){ys,corr1,corr0,accuracy}=NetMeasurements[net,trainAssoc,{NetPort[{"lenet",1,"Output"}],NetPort[{"corr 1"}],NetPort[{"corr 01"}],"Accuracy"},LossFunction"Loss"];corrdelta0=(#[[1]]-)&[corr0];corrdelta1=(#[[1]]-)&[corr1]/.a_:>∞/;a<=0;(*Thisconditionremovesthehiddenneuronswhichdonotchange,i.e.whosevarianceisnegativeduetocalculationerrorsorverysmall.Iputitheresincethisexampledealswiththeinputlayeronly.Inpracticewefirstprunethehiddenlayerbythisalgorithmandremovealldeadhiddenneurons.Andthenprunetheinputlevel.Sothisproblemdoesnothappen.*)(*Calculateinputneuron'sefficienciesaccordingtoalgorithmA1fromthepaper*)xs=corr0[[2]];{w1,b1,w3,b3}=Normal@NetExtract[net,{{"lenet",1,"Weights"},{"lenet",1,"Biases"},{"lenet",3,"Weights"},{"lenet",3,"Biases"}}];fp=;influence=Outer[Times,fp,corrdelta0];efficiency2=Total@(influence);(*Sorttheinputneuronsaccordingtotheirefficienciesandselecttheneuronslessefficientthanthecutoff*)positions=Position[efficiency2,#]&/@DeleteDuplicates[Sort[efficiency2]];neuronstochange=Flatten@Take[Position[efficiency2,#]&/@Sort[Cases[DeleteDuplicates[efficiency2],_?(#<thresholdMax[efficiency2]&)]],UpTo[Floor[0.9Length[efficiency2]]]];(*RebuildthenetwithouttheneuronslessefficientthanthecutoffaccordingtoA1algorithmfromthepaper*)w91=Delete[IdentityMatrix[784],List/@neuronstochange];w11=Delete[w1,List/@neuronstochange];b11=b1+Plus@@(w1[[All,#]]xs[[#]]&/@neuronstochange);newnet=NetReplacePart[chain31[w91,Length[b11]],{{3,"Weights"}->w3,{3,"Biases"}->b3,{1,"Weights"}->w11,{1,"Biases"}->b11}];newnetdressed=NetGraph[<|"lenet"newnet,"loss1"CrossEntropyLossLayer["Index"]|>,{NetPort["Input"]"lenet",NetPort["lenet","p Output"]NetPort["loss1","Input"],NetPort["Target"]NetPort["loss1","Target"]}];accuracy1=NetMeasurements[newnetdressed,trainAssoc,"Accuracy",LossFunction"Loss"];<|"newNet"->newnetdressed,"newAccuracy"->accuracy1,"oldAccuracy"->accuracy,"neuronsNumber"->Length[w91],"Cut"->threshold,"Mask"->Total[w91]|>
2
#[[2]]
2
#[[2]]
2
1
2
Cosh[ys]
corrdelta1
2
w1
Example of pruning the initial neurons via the A1 algorithm from the paper
Example of pruning the initial neurons via the A1 algorithm from the paper
Initialize the net “chain31”, train it for 50 rounds, and measure its accuracy.
In[]:=
innet=NetGraph[<|"lenet"NetInitialize[chain31[IdentityMatrix[784],20]],"loss1"CrossEntropyLossLayer["Index"]|>,{NetPort["Input"]"lenet",NetPort["lenet","p Output"]NetPort["loss1","Input"],NetPort["Target"]NetPort["loss1","Target"]}]
Out[]=
NetGraph
In[]:=
trainednet=NetTrain[innet,trainAssoc,All,ValidationSettestAssoc,MaxTrainingRounds50,BatchSize600,LossFunction"Loss",TrainingProgressReporting->"Panel"];
In[]:=
net=trainednet[{"TrainedNet","TotalTrainingTime"}]
Out[]=
NetGraph,37.1282
In[]:=
NetMeasurements[net[[1]],trainAssoc,"Accuracy",LossFunction"Loss"]
Out[]=
0.958117
Prune the initial neurons with the efficiency less than 5 % (max efficiency)
In[]:=
res=reducenicpercent1[net[[1]],0.05]
Out[]=
newNetNetGraph,newAccuracy0.944817,oldAccuracy0.958117,neuronsNumber317,Cut0.05,Mask{0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0}
Show the active neurons in black and the removed in white
So we removed more than one half of the input neurons reducing accuracy for ~1-2%
One may retrain the pruned net for 10 epochs
One may retrain the pruned net for 10 epochs
And retrain the unpruned net to compare
We see that retraining the pruned net takes about one half of the time necessary to retrain the initial net without large accuracy reduction
One can try different efficiency cuts:
Retrain these 4 nets for 10 epochs
So imposing efficiency cut ~0.3, we may take only ~ 50-150 of 784 neurons to have ~78-88% accuracy with this net
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Bio-inspired machine learning: programmed death and replication
by Andrey Grabovsky and Vitaly Vanchurin
Wolfram Community, STAFF PICKS, August 2, 2023
https://community.wolfram.com/groups/-/m/t/2982004
by Andrey Grabovsky and Vitaly Vanchurin
Wolfram Community, STAFF PICKS, August 2, 2023
https://community.wolfram.com/groups/-/m/t/2982004