What punctuation should be added to a string of text?​
​By Hozaifa Bhutta
This project attempts to automatically punctuate text by using a neural network that leverages the Bidirectional Encoder Representations from Transformers (BERT) architecture. BERT outputs contextual feature vectors of an input sequence, which are essential to capture the meaning of each word in a sequence of text. The center embedding outputted by BERT is passed through a fully connected neural network, which then produces a classification of the punctuation type (if there is any punctuation) that follows the center word of the input sequence. The model was trained on “A New Kind of Science” (written by Stephen Wolfram) over a 2-hour period and reached a relatively high accuracy of 88% on the testing data.

Setup


Introduction

About two decades ago, Machine Learning (ML) reached the dawn of its golden age, and every year since then, ML has continued to impress people in ways previously thought impossible. But ML models can only be as good as the data they are trained on. Corresponding with the rise of ML, there has also been an increasing problem of unformatted text databases plaguing the internet. One such issue is that text can occasionally be stripped of punctuation during the storing process. For instance, when a closed-captioning tool is used to transcribe audio, no punctuation is included in the transcription by default.
A saying in the Data Science world illustrates the problem with unformatted datasets: “Garbage in, garbage out.” In other words, low-quality input will always generate incorrect output. Hence, unpunctuated text data can pose a problem for others who wish to train their Natural Language Processing (NLP) models on vast text data sets.

Neural Networks

First created in 1944, Neural Networks have come a long way since their inception. Neural Nets are computing algorithms based on how nodes in the human brain work, representing a series of complex operations on an input to produce the desired output.
Out[]=
Above is a three layer Neural Net work that take in an input, perform certain operations, and then produces an output. Each circle in the graph is a node, each row is a layer, and each line connecting the nodes is a connection. Each connection between the input and output node represents a simple y = mx + b equation, where the "m" and "b" values are fine-tuned through training (more on that later).
By adding more nodes and layers, we expand the network and allow it to extract more information from the input.
We can use a neural network to perform certain classification tasks (such as determining whether an image contains a person or not, or what tone is associated with a sequence of words) by training the model through a method called Gradient Descent.
In short, Gradient Descent is an optimization algorithm that adjusts the connections in a network by stepping in the opposite direction of the gradient of the output to find the local minimum of the "loss function". The loss function represents the neural network's inaccuracy on a dataset. By creating a comprehensive dataset and training the appropriate neural network on it, you can use deep learning to solve almost any classification problem.
If you want to learn more about neural networks, there are many great resources online:
​https://news.mit.edu/2017/explained-neural-networks-deep-learning-0414

BERT

The Bidirectional Encoder Representations from Transformers (BERT) algorithm is a transformer based machine learning neural network that essentially functions as a feature extractor for a sequence of text.
Out[]=
BERT is made up of a sequence of encoders and decoders that output a sequence of vectors. Each output vector is associated with an input word and contains a contextual understanding of that word in the sequence.
MatrixPlot[Take[bert["This is a six word sentence"],{2,7}]]
Out[]=
Below are a few examples of how the same word used in different contexts is treated differently by BERT
In[]:=
animal1=bert["The bat is my favorite nocturnal animal"][[3]];​​animal2=bert["My favorite animal is the bat"][[7]];​​object1=bert["He hit the ball with the bat"][[8]];​​object2=bert["You play baseball with a bat"][[7]];
FeatureSpacePlot3D[{object1->"Object 1",object2->"Object 2",animal1->"Animal 1",animal2->"Animal 2"}]
Out[]=
Depending on where the word “bat” appears in a sentence, it is given a completely different position in the vector space by BERT.
MatrixPlot[ArrayReshape[animal1,{1,768}]]​​MatrixPlot[ArrayReshape[object1,{1,768}]]
Out[]=
Out[]=
Above is another depiction of how the vectors differ in value, despite representing the same words.

Initializing BERT

bert=NetModel["DistilBERT Trained on BookCorpus and English Wikipedia Data","Type"->"BaseUncased"];
BERT is initialized with the already-trained parameters provided by the Wolfram Repository. This process is called “Transfer Learning,” which is when data is taken from another trained neural network and is tuned to solve new problems the model was not originally designed to solve. Additionally, this project used a variation of the BERT algorithm called “DistilBERT,” which contains 40% fewer connections (speeding up the algorithm considerably) while maintaining sufficient accuracy.

Architecture

The main program is a multi-layered neural network. The network incorporates the BERT algorithm at the beginning to capture the context of each word in the input sequence, and then passes the center embedding though a short fully-connected neural network to output a probability vector.
network=NetChain[{bert,PartLayer[midIndex],DropoutLayer[0.2],LinearLayer[768],DropoutLayer[0.2],ElementwiseLayer[Ramp],LinearLayer[100],DropoutLayer[0.2],ElementwiseLayer[Ramp],LinearLayer[3],SoftmaxLayer[]}]
Note, that the algorithm only classifies one word at a time, specifically the center word of the 21 word input sequence (in order to maximize the contextual understanding).
To use the network to find the punctuation for all words in a sentence, you "slide" the network over the input and output a prediction for each "center word" in the sequences.

Data Pre-Processing

An essential but often overlooked part of any data science project is data pre-processing.
For this project, I used "A New Kind of Science" written by Stephen Wolfram as it was one of the more modern books on the Wolfram Repository that followed generally acceptable punctuation rules.
For the sake of simplicity, I focused solely on commas (",") and full-stop punctuation (".","?","!") to avoid a dilution of data. Everything else was classified as a "space".
However, due to the nature of English (and almost any other language), there is a much higher proportion of spaces than any punctuation, so I randomly removed the majority of data instances with a "space" classification to evenly distribute the dataset.
Before cleaning
After cleaning

Overfitting

Overfitting is a phenomenon that occurs when the network performs so well on the training data, that it can not make generalizations and extrapolate on data is has never encountered before.
To ensure that the model did not overfit, I split the dataset into training data (90%) and testing data (10%). When the network stopped improving on the testing data but kept improving on the training data, I immediately stopped the program as the network had reach its optimal parameters.

Training the network

Over the course of a week, I trained the network with different hyperparameters to maximize the accuracy of the model on the validation set. The model was trained on the NVIDIA GeForce RTX 3060 Ti
GPUs have proven to train networks faster than CPUs as they can perform simultaneous computations significantly faster than an average CPU. There is a downside that the GPU used contained less memory than an average CPU, which meant a smaller batch size had to be used.
At its best, the model obtained an 88.5% accuracy on the testing data and 95.25% on the training data. It is typical of a neural network to perform better on the training data, as it can never perfectly extrapolate to new data.

Examples

Try it out yourself!

IMPORTANT: MAKE SURE THAT THE CODE ABOVE WORKS BEFORE RUNNING ANY CELL BELOW. IF NOT, GO TO SETUP AND RUN ALL THE INITIALIZATION CELLS FROM TOP TO BOTTOM
Note that the "slider" mechanism does not work perfectly, and a few words may be omitted from the input. For more details, look under “Setup”

Conclusion

By the end of the two weeks, this project has indicated that it is possible to use deep learning to restore punctuation in text. By leveraging BERT for contextual understanding and then passing the context vector through a short neural network, the model was able to predict the correct punctuation with a respectable accuracy.

Room for improvement

Of course, the model is far from perfect. One noticeable problem is that the model's accuracy deteriorates when it encounters text outside of "A New Kind of Science", the book it was trained on.
These results indicates that the model is failing to adapt to different styles of writing. To fix this problem, the model should be trained over a more diverse range of texts for a longer period of time. A varied dataset will train the model to become familiar with different styles of texts and possibly allow it to adapt to authors it has not encountered before.

Future Directions

Whether it be transcribing audio using a closed captioning tool or an inaccurate OCR being used to convert physical documents into virtual ones, there is a multitude of reasons why unpunctuated datasets exist. I believe that the proposed network, if trained and tuned more thoroughly, can be used to address this issue effectively. Furthermore, if used correctly, this project can create more opportunities for groups who wish to utilize vast text data for their own Natural Language Processing (NLP) models.
Credits
GPU for Deep Learning. RunAI. (n.d.). Retrieved July 16, 2022, from https://www.run.ai/guides/gpu-deep-learning
​
Hardesty, L. (2017, April 4). Explained: Neural networks. MIT News | Massachusetts Institute of Technology. Retrieved July 12, 2022, from https://news.mit.edu/2017/explained-neural-networks-deep-learning-0414
​
Vaswani, A., Shazeer, N., & Parmar, N. (2012). Attention Is All You Need.

Thanks

I wish to express gratitude to my mentor, Kurt, who helped me every step of the way during these two weeks. This project could not have gone so far in such a short amount of time without him.
I also want to thank a fellow campmate, Nicoló Monti, who was willing to train the network on his GPU and gave me great advice throughout the project.