The classification of photographs by date of creation has been investigated very little so far. The ability to do so would enable us to quickly and automatically sort and label large numbers of images . This project uses neural networks to achieve this task. Our machine learning model is trained using a dataset of over 80,000 images, ranging from the years 1930 to 1999 and labeled respectively . The number of photographs for each year is the same to avoid any bias towards any specific year. Additionally, this project visualizes the features that led to the date-classification, thereby giving meaningful insights into the inference of the model.
Introduction
Introduction
This project uses various Neural Network tools available in Wolfram Mathematica to analyze, extract features and make predictions of the year of the image. Identifying the year of the image requires the model to be trained on an image dataset that has labels indicating the year of each image, hence this is a classification problem. Ideally a large dataset of images is required, since images can be expected to be general, with a variety of objects and backgrounds, hence increasing the variety of features the model needs to analyse.
Realistically, a number of factors can affect the prediction of the age of photo, such as an older camera being used or a setting that might appear very similar to one years before. It is reasonable to expect the predictions to be off by a few years.
A manual scan by the human eye, of the images within various years, reveals interesting trends and qualities of the images. Such as the phenomena in which photos of the 1930’s tend to develop a yellowish background due to aging, also the evolving photography techniques that results in more varying angles of shot over the years, probably due to cameras being more portable. These features can potentially be leveraged in determining the age of a photograph.
This is an ideal problem for a machine learning model, since a vast number of factors need to be considered and a data driven approach can help in identifying the key features.
Realistically, a number of factors can affect the prediction of the age of photo, such as an older camera being used or a setting that might appear very similar to one years before. It is reasonable to expect the predictions to be off by a few years.
A manual scan by the human eye, of the images within various years, reveals interesting trends and qualities of the images. Such as the phenomena in which photos of the 1930’s tend to develop a yellowish background due to aging, also the evolving photography techniques that results in more varying angles of shot over the years, probably due to cameras being more portable. These features can potentially be leveraged in determining the age of a photograph.
This is an ideal problem for a machine learning model, since a vast number of factors need to be considered and a data driven approach can help in identifying the key features.
Data Set
Data Set
The data set used consists of over a 100 thousand images that have been split into 80% and 20% for training and testing respectively. The dataset consists of 1500 images for each year from 1930 to 1999. A similar number of images are maintained for each year, to avoid any bias towards any specific year. Subsequent manual scans has revealed more interesting insights into the data set, such as each year having images of certain objects more than others. This creates a possibility of bias of an object towards a specific year. The data also contains duplicates, that are removed using the hash function.
The whole data set can be found in the link https://www.radar-service.eu/radar/en/dataset/tJzxrsYUkvPklBOw.
The whole data set can be found in the link https://www.radar-service.eu/radar/en/dataset/tJzxrsYUkvPklBOw.
Feature Extraction
Feature Extraction
"ResNet-50 Trained on ImageNet Competition Data" neural net model from the Wolfram repository was used for the feature extraction process.
nM = NetModel["ResNet-50 Trained on ImageNet Competition Data"]nT = NetTake[nM,"flatten_0"]
The model is based on convolution neural network. The flatten layer is used for the output of the features extracted by the model which is a vector size of 2048.
Since this model is build for object detection, this was identified as a suitable model for feature extraction.
Since this model is build for object detection, this was identified as a suitable model for feature extraction.
Model creation and Training
Model creation and Training
The approach used for this project is a transfer learning from a convolution neural network (“ResNet-50 Trained on ImageNet Competition Data”), joined with a fully connected neural network (developed in this project). A dropout layer is used to avoid overfitting, while a linear layer of vector size 70 is fed into a softmax layer, which returns the probability of the years classification. The feature set extracted from the earlier step is used in the fully connected neural network for training the model. The loss function and error rate of the training can be observed.
nM = NetModel["ResNet-50 Trained on ImageNet Competition Data"]nT = NetTake[nM,"flatten_0"]csvTrainData=Import["fileNameHere"];trainDataSet=Map[#[[1]]->#[[2]]&,csvTrainData[[1;;All]]]//Normal;featureSetofTrainData=Thread[nT[Map[Import,trainDataSet[[All,1]]],TargetDevice->"GPU"]->trainDataSet[[All,2]]];dim=Length[featureSetofTrainData〚1,1〛];nn2=NetChain[{LinearLayer[2048],BatchNormalizationLayer[],DropoutLayer[0.25],ElementwiseLayer[Ramp],LinearLayer[70],SoftmaxLayer[]},"Input"dim,"Output"NetDecoder[{"Class",Range[1930,1999]}]]dnn2=NetTrain[nn2,featureSetofTrainData,TargetDevice"GPU"]
The feature set of the train data is used for training the model.
Out[]=
Loss
,ErrorRate
| rounds |
loss |
| rounds |
error rate |
Testing The Model
Testing The Model
Testing the model reveals a 24% accuracy in predicting the exact year. The confusion matrix further reveal that a significant amount of predictions are made close to the actual year.
Out[]=
As shown in below graph, the exact correct predictions of the year is over 5000 and the predictions within five years is over 10,000; the rest of the predictions that are off by more than 5 years is over 10,000.
Identifying Features that lead to Predictions
Identifying Features that lead to Predictions
This project uses NetPortGradient function in Wolfram to visualize the features that led to the prediction of an image. The gradient returned by the function is masked over the original image.These features visualized can provide an intuitive inference as to what features of an image led to a certain year classification.Some example of the visualized features shown below.
,
Conclusion
Conclusion
The project shows that, it is possible to reasonably estimate images by year using neural networks as well as a human can. The features extracted to identify the cause of predicted value, may help in intuitively inferring meaningful insights?
Future Recommendations
Future Recommendations
A larger dataset with more variations of images with diverse background and objects can lead to a better accuracy in classification of year of an image. Proper screening of data to ensure that there is a reasonable balance of objects in each year, can also enhance accuracy. These points also need to be considered when developing a neural net model in the future, so as to work with real world applications.
Acknowledgement
Acknowledgement
I would greatly like to acknowledge and thank my mentor Markus van Almsick, from whom I had learned a lot during the course of the summer school,
Stephen Wolfram for giving me the opportunity to work on this exciting project, Mads Bahrami, Danielle R Mayer and Emily Carter for guiding me through the various steps involved. Thank you.
Stephen Wolfram for giving me the opportunity to work on this exciting project, Mads Bahrami, Danielle R Mayer and Emily Carter for guiding me through the various steps involved. Thank you.