Classifying laboratory glassware with machine learning
​by Arunkumar Sharma

Abstract

This post provides a supervised machine learning example to identify laboratory glassware. This project was implemented in an Introduction to Scientific Computing course for first-year students at Wagner College. The goal of the exercise was to present a typical machine learning task in the context of a chemistry laboratory to engage students with computing and its applications to scientific
projects. This is an end-to-end data science experience with students creating the dataset, training a neural network, and analyzing the performance of the trained network. The students collected
pictures of various glassware in a chemistry laboratory. Four pre-trained neural networks, Inception-V1, Inception-V3, ResNet-50, and ResNet-101 were trained to distinguish between the objects in
the pictures. The students received hands-on training in the Wolfram Language and an elementary introduction to image classification as an example of application in the machine learning domain. Students enjoyed the introduction to machine learning applications and the hands-on experience of building and testing an image classifier to identify laboratory equipment. The pictures used for the training are located here and the study was published in the Journal of Computational Science Education. This post only shows the setup and results from training of the ResNet-50 neural network.

Helper functions for image effects and splitting data

In[]:=
SetDirectory[NotebookDirectory[]]
In[]:=
SetOptions[ImageRotate,Background None];(* This ensures that there is no black background when we use ImageRotate*)
In[]:=
imageSetAugmentation[objectImages_List]:=Module[{detailEnhanced,blurredImages,noisyImages,lightDarkImages,reflectedImages,rotatedImages,augmentedImages},​​detailEnhanced=ImageEffect[#,"DetailEnhancing"]&/@objectImages;​​blurredImages=Blur[#,RandomInteger[{1,3}]]&/@objectImages;​​noisyImages=ImageEffect[#,"Noise"]&/@objectImages;​​lightDarkImages=Join[Lighter[#]&/@objectImages,Darker[#]&/@objectImages];​​reflectedImages=ImageReflect[#,Left]&/@Join[objectImages,detailEnhanced,blurredImages,noisyImages,lightDarkImages];​​rotatedImages=ImageRotate[#,RandomInteger[{-10,10}]Degree]&/@Join[objectImages,detailEnhanced,blurredImages,noisyImages,lightDarkImages,reflectedImages];​​augmentedImages=Join[objectImages,reflectedImages,rotatedImages];​​Return[augmentedImages];​​]
Function to split list into two parts 80-20.
In[]:=
trainTestDataSplitter[images_List]:=TakeDrop[RandomSample[images],Round[0.8*Length[images]]]

Import Images (open section to view code)


Data Generation

Picture Collection

Students used their smartphones to click pictures of various laboratory glassware routinely used in a Chemistry laboratory. They uploaded the pictures into a shared Google folder directly from their phones. This procedure was adopted to simplify the data collection process. Most of the pictures were taken with the goal of having one main object in the image. However, some variability was also introduced in each class by intentionally including some background clutter or including multiple items in the picture. A mixture of empty and filled glassware was used to mimic a typical Chemistry laboratory setting. For example, beakers of various volume capacities, 250 mL, 500 mL, etc were used. The chemical composition of the solutions was not important for this exercise. Our intention was to introduce colors into the beakers to increase the sample space of pictures.

Class labels

In[]:=
classLabels={"Beaker","Buchner Funnel","Burette","Burette Stand","Erlenmeyer Flask Chem","Flat Bottom Flask","Funnel","Grad. Cylinder","Pipette","Round Bottom Flask","Separatory Funnel","Std. Flask","Test Tubes","Test Tube","Test Tube Stand","Viscometer","Wash Bottle"};

Erlenmeyer Flask images

Erlenmeyer flask images collected by the students. Different colored solutions were used to fill up the flasks to various capacities. Some images included empty flasks and flasks filled with water.
In[]:=
ImageCollage[eFlaskImagesBase,Method"Rows"]
Out[]=

Representative image from each class.

In[]:=
ImageCollage[Flatten[RandomSample[#,1]&/@{washBottlesImagesBase,viscometerImagesBase,ttCollectionImagesBase,ttStandAloneImagesBase,ttStandsImagesBase,stdMeasuringFlaskImagesBase,sepFunnelImagesBase,rbFlaskImagesBase,pipetteImagesBase,gradCylinderImagesBase,funnelsImagesBase,fbFlaskImagesBase,eFlaskImagesBase,buretteImagesBase,buretteStandImagesBase,beakersImagesBase,buchnerImagesBase}],Method"Rows"]
Out[]=

ImageEffects for one type of image.

The Wolfram language function, ImageEffect, was used to carry out detail enhancing and adding random noise effects to each image. Images were blurred using the Blur function with a randomly chosen pixel radius over which the blur was to be applied. Images were modified to appear lighter or darker using the appropriately named functions. Next, all of these images were collected and reflected from left to right. The final operation was to rotate all of these images with a randomly chosen rotation amount between -10 to 10 degrees. The result of all of these operations on one image taken from the set of Erlenmeyer flask images is shown below. For every image in the raw dataset, 19 images were produced by the image augmentation procedure described above. The number of raw images in the dataset was 335 and after the implementation of the imageSetAugmentation module the number of images increased to 6,365.
In[]:=
ImageCollage[imageSetAugmentation[{RandomChoice[eFlaskImagesBase]}],ImagePaddingNone,BackgroundWhite]
Out[]=

Class compositions

In[]:=
TableForm[{classLabels,Length/@{beakersImagesBase,buchnerImagesBase,buretteImagesBase,buretteStandImagesBase,eFlaskImagesBase,fbFlaskImagesBase,funnelsImagesBase,gradCylinderImagesBase,pipetteImagesBase,rbFlaskImagesBase,sepFunnelImagesBase,stdMeasuringFlaskImagesBase,ttStandAloneImagesBase,ttCollectionImagesBase,ttStandsImagesBase,viscometerImagesBase,washBottlesImagesBase}},TableDirectionsRow,TableHeadings{{"Glassware","Images"},{}}]
Out[]//TableForm=
Glassware
Images
Beaker
30
Buchner Funnel
20
Burette
11
Burette Stand
5
Erlenmeyer Flask Chem
24
Flat Bottom Flask
18
Funnel
23
Grad. Cylinder
47
Pipette
16
Round Bottom Flask
24
Separatory Funnel
22
Std. Flask
36
Test Tubes
6
Test Tube
17
Test Tube Stand
14
Viscometer
14
Wash Bottle
8

Training and Testing Sets

Training the neural network

ResNet 50

Remove the linear layer from the pre-trained net. Remove the final classification layers of the trained net and we will replace them with the layers specific to this new task. Notice that the LinearLayer and SoftmaxLayer have been removed. Create a new net composed of the pre-trained net followed by a linear layer and a softmax layer. The LinearLayer takes an integer argument for the number of classes. Here we will perform training with MaxTrainingRounds set to 1.

Classification Performance

The low frequency of misclassification is clear from the confusion matrix. Graduated cylinder gets misclassified as standard flask only twice out of 179 samples tested. The high accuracy with very little computational cost is quite a revelation for students.

Accuracy as a function of training rounds.

Teaching ideas

The images of the dataset and sample notebooks used for training of networks and data analysis are available on Zenodo. Short student projects to investigate performance of classification for smaller number of classes may be constructed using the dataset. Different training times could be used to assign student groups to measure its impact on accuracy of classification. Another extension that could be implemented is to carry out image augmentation using different types of transformations and/or subsets of transformations from those that have been used in our implementation. The students could then investigate the effectiveness of those transformations towards the final classification performance. The exercise can also be extended by adding more glassware images and investigating classification performance. An interesting and possibly more advanced application would be to identify the text on glassware that annotates the volume, especially on beakers or Erlenmeyer flasks. Another application would be to identify the piece of glassware and to identify the hand-written or printed chemical species from the label attached to the glassware. This would integrate image classification and hand-writing recognition.

Student feedback

This exercise was carried out with a cohort of eight first-year students in the author’s “Introduction to Scientific Computing” course at Wagner College. The students expressed enthusiasm and interest toward more applications of machine learning following this exercise. Students pointed out that they enjoyed the project through informal feedback and one-on-one interviews. They specifically enjoyed bringing their computing knowledge into the wet laboratory. Students with interest in biological sciences started discussions on applications of machine-learning methods to images obtained from microscopes. A majority of comments indicated that the activity helped them feel less intimidated about approaching machine learning or artificial intelligence related literature. They also reported increased interested in exploring computation as a tool toward their scientific domains of interest. A significant outcome of the informal feedback process was the realization from students that machine learning and advanced approaches are not limited to computer science majors or large technology companies.

Conclusion

We developed and implemented an end-to-end data science exercise with an application of machine learning experience for STEM students using their laboratory surroundings and equipment as the source of the project. Classification of images based on supervised learning is a common example in machine learning domain and the students adapted that into the chemistry laboratory. First-year students collected pictures of various glassware in the chemistry laboratory and implemented the training and testing of classifiers based on four pre-trained neural networks. These neural networks were chosen due to their wide availability and well-known performance on image classification tasks. The glassware images were split into two categories of full color images and grayscale images. Each set of images was enlarged with an image augmentation routine that resulted in a 19-fold increase in the size of the dataset. The students then compared the performance of the classification of glassware among the four networks and for each of the four types of datasets. The performance of the classifiers on the augmented datasets seems to be the most reliable irrespective of using color images or grayscale images. Our analysis shows that ResNet-50 provides the best trade-off between accuracy and training time for the datasets considered in this activity. We believe that this activity provides students with an accessible and empowering introduction to advanced techniques in the data science domain through the use of typical glassware in a chemistry laboratory.

Acknowledgements

We would like to record our appreciation and gratitude to students in the Introduction to Scientific Computing course. We would also like to thank Dr. Tuseeta Banerjee and Dr. Joshua Schrier for their helpful suggestions.

Publications associated with this post

[1] Sharma, A. K. Laboratory Glassware Identification: Supervised Machine Learning Example for Science Students. J. Comput. Sci. Educ. 2021, 12 (1), 8–15. https://doi.org/10.22369/issn.2153-4136/12/1/2.
[2] Sharma, A. K. A Model Scientific Computing Course for Freshman Students at Liberal Arts Colleges. J. Comput. Sci. Educ. 2017, 8 (2), 2–9. https://doi.org/10.22369/issn.2153-4136/8/2/1.