The Wolfram Emerging Leaders Program: High School (WELP:HS) is a remote, semester-long program designed for exceptional graduates from the Wolfram High School Summer Research Program to continue to develop their skills. Students work in a small group over the course of a semester to do either a research-based project or a product development–based project under the guidance of expert mentors. Projects are centered around a topic at the intersection of the group’s interests.
This essay was written by:
This essay was written by:
Recent advancements in the field of artificial intelligence (AI) and the escalating complexity of models have led to a significant increase in computational and environmental costs. This research paper delves into the implementation of various neural network compression techniques to develop smaller, faster, and equally or more accurate neural networks. Our study centers around the application of random one-shot pruning, magnitude-based pruning, and iterative magnitude-based pruning on the LeNet model. We evaluate and compare these techniques in terms of their impact on model accuracy, training speed, and evaluation speed, thereby offering insights into the efficiency of compressed neural networks. Furthermore, we examine and validate the “winning lottery ticket hypothesis,” which posits that neural network pruning can reduce parameter counts significantly without compromising accuracy. Our findings reveal that standard pruning techniques can unveil subnetworks with initial configurations conducive to effective training, termed as “winning tickets.” These subnetworks, when trained in isolation, demonstrate a test accuracy comparable to the original network in similar training iterations. The study provides a compelling case for the optimization of AI models in terms of efficiency, with a focus on reducing training time and computational resources while preserving or enhancing model performance. Moreover, the explosion of large-scale models that are computationally expensive, including large language models like ChatGPT, have left a significant environmental impact that can be decreased through neural network compression.
Introduction
Introduction
The advancement of AI models, marked by increasing complexity and size, has led to an exponential growth in the required resources and time for their training and deployment. This trend is exemplified by the substantial carbon footprint associated with training a singular large-scale model, such as GPT-3.5, which is estimated to be equivalent to over 600,000 pounds of CO2. In light of these challenges, our research aims to contribute to the critical field of AI model efficiency. We focus on the application of neural network compression techniques on the LeNet model, a foundational architecture in deep learning, to create smaller, more efficient networks without sacrificing accuracy.
Our approach involves implementing and comparing various pruning methods—random one-shot pruning, magnitude-based pruning, iterative magnitude-based pruning— techniques on the LeNet model. These methods are evaluated based on their impact on the model’s accuracy, as well as the training and evaluation speeds, thereby providing a comprehensive understanding of the trade-offs involved in neural network compression.
Central to our investigation is the exploration of the “winning lottery ticket hypothesis,” first proposed by Frankle and Carbin in 2019. This hypothesis suggests that neural networks can be pruned to significantly reduce their parameter count—by over 90%—thereby decreasing storage requirements and improving computational performance during inference, all without compromising accuracy. A key challenge in this domain has been the difficulty in training sparse architectures from the outset. Our research addresses this challenge by identifying subnetworks within the pruned networks that possess initial configurations, or “winning tickets,” conducive to effective training. These subnetworks demonstrate the ability to achieve test accuracy comparable to the original, unpruned network within a similar number of training iterations.
The driving force behind our study is the pressing need for more efficient AI models. As AI becomes increasingly integral in various sectors, the demand for sustainable, accessible, and efficient models grows. Through our research, we aim to make a significant contribution to the theoretical and practical aspects of AI efficiency, providing pathways to substantial reductions in training time and computational resources while maintaining or improving model performance. This endeavor is crucial not only from a technical standpoint but also in fostering a more sustainable and inclusive future for AI technology.
Our approach involves implementing and comparing various pruning methods—random one-shot pruning, magnitude-based pruning, iterative magnitude-based pruning— techniques on the LeNet model. These methods are evaluated based on their impact on the model’s accuracy, as well as the training and evaluation speeds, thereby providing a comprehensive understanding of the trade-offs involved in neural network compression.
Central to our investigation is the exploration of the “winning lottery ticket hypothesis,” first proposed by Frankle and Carbin in 2019. This hypothesis suggests that neural networks can be pruned to significantly reduce their parameter count—by over 90%—thereby decreasing storage requirements and improving computational performance during inference, all without compromising accuracy. A key challenge in this domain has been the difficulty in training sparse architectures from the outset. Our research addresses this challenge by identifying subnetworks within the pruned networks that possess initial configurations, or “winning tickets,” conducive to effective training. These subnetworks demonstrate the ability to achieve test accuracy comparable to the original, unpruned network within a similar number of training iterations.
The driving force behind our study is the pressing need for more efficient AI models. As AI becomes increasingly integral in various sectors, the demand for sustainable, accessible, and efficient models grows. Through our research, we aim to make a significant contribution to the theoretical and practical aspects of AI efficiency, providing pathways to substantial reductions in training time and computational resources while maintaining or improving model performance. This endeavor is crucial not only from a technical standpoint but also in fostering a more sustainable and inclusive future for AI technology.
Methods and Analysis
Methods and Analysis
Random Pruning
Random Pruning
Randomized one-shot pruning is a neural network compression technique that aims to reduce the size of a neural network by removing a subset of its weights, or connections, in a single pass or “one shot.” This method involves randomly selecting weights to prune based on a predetermined sparsity level—that is, the percentage of weights that are to be removed.
More formally, we can think of pruning as introducing sparsity into the network’s weight matrix. For a weight matrix with dimensions then pruning converts a number of elements in to zero. The sparsity, or pruning ratio, can be expressed as the zero-ed elements by the total number of elements .
W
MXN
W
S
Z
MN
In[]:=
Needs["NeuralNetworks`"]
Construction of custom LeNet model, what we will be using for the majority of compression techniques we utilize. The LeNet model is a convolutional neural network that is designed for handwriting recognition and image classification tasks, we will use a sequence of convolutional, pooling, and fully connected layers to construct it.
In[]:=
lenet=NetChain[{ConvolutionLayer[20,5],(*20×5x5filters*)Ramp,PoolingLayer[2,2],(*2x2maxpooling*)ConvolutionLayer[50,5],(*50×5x5filters*)Ramp,PoolingLayer[2,2],(*2x2maxpooling*)FlattenLayer[],500,(*Fullyconnectedlayerwith500neurons*)Ramp,10,(*Outputlayerwith10classesforMNIST*)SoftmaxLayer[](*Softmaxactivationforclassification*)},"Output"->NetDecoder[{"Class",Range[0,9]}]]
Out[]=
NetChain
The MNIST dataset is a collection of 70, 000 labeled images of handwritten digits used for training and testing image processing systems in machine learning . Each 28 x 28 pixel grayscale image is associated with a label that indicates the digit it represents (0 through 9) . This dataset is a standard benchmark in the field of computer vision. We can use the datasets for the training and validation of the neural network.
In[]:=
trainingData=ResourceData["MNIST","TrainingData"];
In[]:=
validationData=ResourceData["MNIST","TestData"];
In[]:=
RandomSample[trainingData,5]
Out[]=
7,
5,
5,
4,
7
In[]:=
RandomSample[validationData,5]
Out[]=
5,
4,
9,
1,
9
In[]:=
trainedNet=NetTrain[lenet,trainingData,MaxTrainingRounds->2]
Out[]=
NetChain
In[]:=
initializedNet=trainedNet;
The pruneNet function takes a neural network, in this case LeNet, and a specified fraction as a percentage of neurons as inputs to perform the network pruning. It first determines the total number of weights in each layer of the network and then flattens these weights for easy manipulation . The function randomly selects a subset of these weights based on the specified pruning fraction and sets them to zero, effectively removing them from the network . This pruning process is carried out across all layers of the network, and the resulting pruned network, with a reduced number of active weights, is then returned .
As we can see, it evaluates on all the available weights of the neural network .
In[]:=
NetExtractArrays[initializedNet]
Out[]=
{1,Weights}NumericArray,{1,Biases}NumericArray,{4,Weights}NumericArray,{4,Biases}NumericArray,{8,Weights}NumericArray
,{8,Biases}NumericArray,{10,Weights}NumericArray,{10,Biases}NumericArray
In[]:=
pruneNet[net_,fractionToPrune_]:=Module[{prunedNet,weightSize,weights,indicesToPrune},weightSize=Length[Flatten[Normal[Part[NetExtractArrays[net],#]]]]&/@Range[Length[NetExtractArrays[net]]];weights=Flatten[Normal[Part[NetExtractArrays[net],#]]]&/@Range[Length[NetExtractArrays[net]]];indicesToPrune=Sort[RandomSample[Range[Part[weightSize,#]],Ceiling[fractionToPrune*Part[weightSize,#]]]]&/@Range[Length[NetExtractArrays[net]]];prunedNet=NetReplacePart[net,Part[Keys[NetExtractArrays[net]],1]->NumericArray[Flatten[Partition[Partition[Partition[Partition[ReplacePart[Part[weights,1],Thread[Part[indicesToPrune,1]->0]],5],5],3],20],1],"Real32"]]]
When we take a look at the weights of the retrainedNet, which has had 30% of its neurons pruned, we can note that a large percentage of the weights are now zeroed out and effectively discounted.
In[]:=
retrainedNet=pruneNet[initializedNet,0.3]
Out[]=
NetChain
In[]:=
Flatten[Normal[Part[NetExtractArrays[retrainedNet],#]]]&/@Range[Length[NetExtractArrays[retrainedNet]]]
Out[]=
In comparison, a 90% randomly pruned network would be almost entirely zero-ed out.
In[]:=
Flatten[Normal[Part[NetExtractArrays[pruneNet[initializedNet,0.9]],#]]]&/@Range[Length[NetExtractArrays[pruneNet[initializedNet,0.9]]]]
Out[]=
Through the pruning, the neural network becomes much smaller, yet it is also possible that it loses accuracy, as we can note from the example below where it does not recognize a symbol that it previously knew.
We want to be able to easily visualize the accuracy as the percentage of pruned neurons increases to find a balance between the optimal amount to prune. Noticeably, in the future, we can also compare how the accuracy will differ from other pruning methods (iterative or magnitude-based).
The accuracy of the completely full net is 0.99 while the accuracy of one with 90% of neurons pruned is merely 58%.
We can create a plot that shows how accuracy is affected as more of the neurons are randomly pruned based off a sample of 10,000 values of test data.
As expected, we can see that the accuracy quickly deteriorates beyond the range of 30% pruned neurons, yet seems unaffected for lesser ones.
Similarly, we want to keep track of how the speed of the measurements are calculated as the neural network decreases in size by testing it on batches of 50,000 values of test data.
The function returns a number to 6 significant figures in seconds.
We can create a plot that shows how the runtime is affected as more of the neurons are randomly pruned.
With the exception of outliers, we can see that the runtime generally trends downwards, reducing by about a half a second from start to end. Although this is a minimal amount, the 5% decrease in runtime evaluation can make an enormous difference in various applications of AI that are time-sensitive.
Similarly, we can view the confusion matrix of the neural network as the number of pruned neurons increases. We can observe that, as the number of pruned neurons increases, the neural network increasingly mischaracterizes samples of different numbers into the same group.
As before, we can visualize this as a plot and take the highest point as the most efficient amount.
Magnitude-based Pruning
Magnitude-based Pruning
We will again use the same LeNet model we defined in section 1.
Load uninitialized LeNet
Load the MNIST dataset (the same data as section 1
Get the weights and biases of all the layers in the model
Get convolution layers
Get weights and biases of convolution layers
As we can see, there are 4 layers of neurons, thus we will apply layer-wise pruning.
Define pruning ratio (prune 30 % of the neurons)
We then define a function that prunes a layer by using a mask to set weights and biases to 0:
◼
Looks at a given layer
◼
If it's a convolution layer...
◼
It extracts the weight and biases matrices for that layer
◼
It then uses the pruning ratio to calculate a cutoff weight...
◼
If the total absolute sum of all of the weights in a given weight matrix is less than that threshold, the elements of that weight matrix are set to zero
◼
You then look through the bias matrices, and for any weight matrix which was set to 0, the corresponding bias matrix is set to 0
◼
The layer is then updated with the new weights and biases
◼
For any convolution layer, we want to set the weight and bias matrices to 0 if the total absolute sum of the weight matrix is above a certain value
Loop through the model to prune all convolution layers
Now, in a similar fashion, we quantify the differences made by changing the pruningRatio on accuracy and runtime.
As expected, we can see that the accuracy again deteriorates deteriorates as the number of pruned neurons increases. However, using magnitude-based pruning, we can see that the accuracy remains much higher further into the pruning, at even 50% of pruned neurons the accuracy remains above 90% whereas the respective average accuracy for a 50% randomly pruned neuron is around 75%.
To clearly see the differences, we can overlay the two on the same graph:
We can create similarly also plot that shows how the runtime is affected as more of the neurons are magnitude-pruned.
Similar to before, the runtimes (excluding outliers) decreases by about 0.5 seconds from 11 to 10.5 seconds.
We can view the confusion matrix of the neural network as the number of pruned neurons increases. We can observe that, as the number of pruned neurons increases, the neural network increasingly mischaracterizes samples of different numbers into the same group.
As before, we can visualize this as a plot and take the highest point as the most efficient amount.
In comparison to randomized pruning, the efficiency metrics remain much higher with a higher pruning ratio, yet the highest efficiency remains with the least pruning.
Suppose we wish to directly compare the two pruning mechanisms as the amount of neurons pruned changes (for example visualizing the difference between a magnitude-based pruning technique with a random pruning technique) we can create a graph that plots them directly on each other.
From the following graphs, we can see that the magnitude-based pruning almost consistently has a higher accuracy and a lower runtime.
Iterative Pruning and Retraining
Iterative Pruning and Retraining
One-Shot Pruning proved to be a viable method of decreasing the size of the neural network while also somewhat maintaining its accuracy. To further build upon the idea of Magnitude Pruning, we experimented with Iterative Magnitude Pruning, or IMP. The IMP algorithm is a method for finding sparse, trainable neural networks that can achieve comparable or better performance than the original network. The algorithm is formulated in the lottery ticket hypothesis, which states that dense, randomly-initialized neural networks contain subnetworks (winning tickets) that can be trained effectively in isolation. In iterative pruning, the process of pruning and retraining happens in cycles where each cycle prunes a certain percentage of weights and retrains the network to recover accuracy. Cycles repeat until a desired level of sparsity of when the accuracy degrades beyond a set limit.
To perform IMP, the LeNet model needs to be set to its uninitialized state.
The IMP algorithm consists of two phases: training and pruning. In the training phase, the algorithm trains a dense network until it reaches a lower validation error than a threshold. The trained model is then saved. In the pruning phase, the algorithm continues to train it until it reaches an error rate less than another threshold. Then, the algorithm prunes the model by removing weights according to their magnitude and replacing them with parameters from the previously saved model, except for the masked neurons. This process is repeated until the model converges or exceeds a maximum number of iterations.
The reason behind using two thresholds is to find winning tickets that have initial weights that make training particularly effective and can be rewound to initialization by pruning. The first threshold is used to identify subnetworks that have low validation errors but high training errors, and the second threshold is used to identify subnetworks that have high validation errors but low training errors. By using these thresholds, the algorithm can balance between finding sparse models and preserving their performance. With this train-prune-rewind cycle, the model can gradually reduce its complexity while maintaining its accuracy on both training and validation data.
The reason behind using two thresholds is to find winning tickets that have initial weights that make training particularly effective and can be rewound to initialization by pruning. The first threshold is used to identify subnetworks that have low validation errors but high training errors, and the second threshold is used to identify subnetworks that have high validation errors but low training errors. By using these thresholds, the algorithm can balance between finding sparse models and preserving their performance. With this train-prune-rewind cycle, the model can gradually reduce its complexity while maintaining its accuracy on both training and validation data.
We need a function to replace all the weights and biases within a model with parameters from another without altering the masked neurons.
The steps for the function replaceLayer are as follows:
The steps for the function replaceLayer are as follows:
◼
Check if the layer is a convolution layer
◼
Extract the weights and biases within the same convolution layer from the previous and current model.
◼
Loop through all the parameters from the current model, and check if their values are 0 (masked). If they are, set the combined weights and biases to 0. If not, set them to the values from the previous model.
◼
Insert the combined weights and biases into a combined layer. Return the new layer.
The steps for the function iterativePruning are as follows:
◼
Train the model until it achieves a validation error lower than threshold 1. If the model fails to reach this error threshold even after training for the full duration, terminate the function.
◼
Save the current model as prevModel.
◼
Continue training the prevModel until it attains an error rate below threshold 2. If the model cannot reach this threshold after training, end the function.
◼
Prune the model using the pruneLayer function to eliminate specified components based on the defined pruningRatio.
◼
Replace the current model's neurons with parameters from prevModel, excluding the masked neurons.
◼
Iterate through the above steps until either the model reaches convergence, meeting the predefined criteria, or exceeds the specified maximum limit of iterations.
◼
Return the pruned model.
Perform IMP on LeNet
After pruning 4 times, the accuracy is still around 99%.
After pruning 4 more times, the accuracy dramatically decreases on the 4th iteration, which is the 7th overall iteration.
Examining the neurons, we can see that they’ve all been pruned. This explains why the model accuracy is close to the theoretical probability of randomly guessing a number from 0 to 9 (10%).
So what does the 7th iteration look like? We first train until the 6th iteration.
So what does the 7th iteration look like? We first train until the 6th iteration.
Now we find the final lottery ticket configuration.
The final lottery ticket model achieved an accuracy of 98.7% with a singular neuron in the first convolution layer and 4 neurons in the second convolution layer, after undergoing extensive pruning, which verifies the Lottery Ticket Hypothesis.
The implementation could be improved by moving the pruning process to the start of each iteration. This way, we would get the trained model without another round of pruning, so we won’t need to retrain the final model. This algorithm could also be further improved to find the ultimate lottery ticket configuration, for example by decreasing the pruning ratio and training for more intervals.
Conclusion and Future Work
Conclusion and Future Work
Conclusion
Conclusion
In this paper, we systematically explored the impact of various neural network compression techniques on the efficiency of the LeNet model. By implementing random one-shot pruning, magnitude-based pruning, and iterative magnitude-based pruning, we have demonstrated that it is feasible to significantly reduce the size of a neural network without adversely affecting its accuracy. Our findings reinforce the “winning lottery ticket hypothesis,” which postulates that dense, randomly-initialized networks contain inherently efficient subnetworks that can be trained to achieve comparable accuracy to the original network in a fraction of the iterations. Our empirical results have shown that, across all pruning strategies, there exists a trade-off between the degree of pruning and the performance of the neural network. The efficiency of these pruned models was quantified not just in terms of accuracy but also in evaluation speed, leading to the proposal of an efficiency coefficient that combines these critical aspects. The research concluded that, while accuracy tends to decline with increased pruning, the reduction in runtime can offset this to some extent, leading to an optimal balance between the two that defines the most efficient network configuration.
Future Work
Future Work
Moving forward, several avenues present themselves for extending this work. Firstly, there is an opportunity to apply the proposed compression techniques to a wider range of neural network architectures beyond LeNet, including those used in large-scale applications such as natural language processing and complex image recognition tasks. Additionally, the efficiency coefficient introduced could be refined to include factors such as energy consumption and training time, providing a more comprehensive measure of neural network efficiency. Exploring alternative pruning strategies, such as structured pruning or the use of evolutionary algorithms to identify the most efficient network topologies, could yield further improvements in efficiency. Further investigation into the “winning lottery ticket hypothesis” might also involve a deeper analysis of initialization strategies to consistently find and exploit these “winning tickets” in various network architectures. Moreover, the incorporation of hardware considerations into the efficiency assessment—taking into account the specific constraints and capabilities of deployment targets, such as mobile devices or edge computing platforms—will be critical in making neural network efficiency more practicable and applicable in the future.
References
References
[1] Frankle, J., & Carbin, M. (2019, March 4). The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv.org. https://arxiv.org/abs/1803.03635
[2] Morcos, A. S., Yu, H., Paganini, M., & Tian, Y. (2019, October 27). One ticket to win them all: Generalizing lottery ticket initializations across datasets and Optimizers. arXiv.org. https://arxiv.org/abs/1906.02773
[3] Aksu, M. (2020, May). Identifying winning lottery tickets in small neural network architectures. http://arno.uvt.nl/show.cgi?fid=156269
[4] You, H., Li, C., & Xu, P. (n.d.). Drawing early bird tickets: Towards more Efficient Training of Deep Networks. https://openreview.net/pdf?id=BJxsrgStvr
[5] Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. ICML, 2018.
[6] Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. Proceedings of ICLR, 2018
[7] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135–1143, 2015.
[8] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
Acknowledgements
Acknowledgements
Thank you to everyone who made this research project possible! To Rory Foulger and Eryn Gillam, from the Wolfram High School Summer Research Program to now, finishing our project at the Wolfram Emerging Leaders Program, thank you for making all of this possible. To our Mentor, Arben Kalziqi, thank you for your enthusiasm, encouragement, and guidance. And to all our family, friends, and fellow WELPers, thank you for all the support along the way, you’re the best.
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
Neural Network Compression and Empirical Verification of the Winning Lottery Hypothesis
by Wolfram Emerging Leaders Program
Wolfram Community, STAFF PICKS, January 2, 2024
https://community.wolfram.com/groups/-/m/t/3094975
by Wolfram Emerging Leaders Program
Wolfram Community, STAFF PICKS, January 2, 2024
https://community.wolfram.com/groups/-/m/t/3094975