Handwritten Recognition and Analysis
Handwritten text recognition is the ability of a computer to recognize and transcribe human handwriting. In addition to recognition, handwritten text analysis is commonly used to identify a person' s handwriting, match two handwritten samples, or otherwise analyze to extract other information such as the person' s characteristics or even identify possible diseases. In this presentation, we will talk about the history and techniques used for handwritten recognition and analysis and demo some solutions using neural networks trained on handwritten datasets. We will also show real - life application examples in various domains such as signature validation, writer identification, and more.

History of OCR and Analysis

Motivated by the fact that in about 1870 people were trying to find a solution to allow the blind people to read, Emanuel Goldberg took read characters and converted them into telegraph code in 1914.
Out[]=
This was one of the main reasons why the first generation of text recognition was called optical character recognition (OCR).
With the growth of the technology the optical era switched by classical image processing era.
Out[]=
And finally progressed into deep learning based solutions nowadays.
Out[]=
Along with character recognition development various methods and techniques were introduced for analyzing the printed and handwritten images.
Out[]=
The handwritten text recognition started taking of along with the development of classical image processing era (1980s). The task was much more challenging than the regular OCR, since the lack of technology and the computational power. Only with the AI technology the task has become doable with decent accuracies.
In[]:=
$collage
Out[]=
Today text in a digital form surrounds us everywhere so much, that we already stopped paying attention to it. The ability to recognize and analyze the text has opened a wide range of application domains from historical to medical, from mathematical to criminal.
The recognition is a very important task for various domains, for example, every day millions of petabytes of PDFs (i.e. medical, insurance, banking, etc. ) are scanned and recognized on daily basis.

Text Recognition Techniques

Out[]=
The standard convolutional neural network (CNN) works as a “one to one” system (i.e. classification), while-as for OCR needs “one to many” system, since the number of symbols in an output usually varies. Moreover, in tasks like OCR, handwritten-analysis, etc. it’s nearly impossible to state with high assurance which segment of an image caries the information about a feature or a character. Though various types of OCR share common techniques, we will concentrate on the handwritten text recognition ones.
Out[]=

RNN

One of the solutions to create a “one to many” system is to use recurrent neural networks (RNNs).
The above might be unfolded into a row representation for a better perception of the inner process.
There are various types of RNN cell architectures.
The big disadvantage of RNNs is that the previous state is always required for computing the current one, which means that GPU requires to concentrate the majority of the resource on the segment of the task rather than spreading to the largest possible scale, thus the training is not well parallelizable, hence is slow.

Transformers

Another approach to achieve “one to many” system is to use transformers, introduced in 2017 by Facebook.
The transformers use very important concept called “attention”.
One of the main advantages over RNN is the training speed and inference.

Existing Architectures for Handwritten Text Recognition

The systems to recognize image with sequences: CTC loss

One of the architectures is a combination of CNN and RNN, and it can be trained with the loss developed by Alex Graves, called CTC.

Decoding one element at a time Approaches: Cross-Entropy loss

Yet another approach is to use so called an attention based sequence to sequence model.

Techniques to Analyze Handwritten Text

The most important task in the systems such as writer identification/verification, style analysis, etc. is to extract the right feature set. There are several approaches to solve the task.

Classification Networks

One approach is to train a classification network, dropping the last classifier layers and using the result as a feature. A big disadvantage of such an approach is that the whole network has to be re-trained for every new example/sample. Another disadvantage is that the resulting representation vector is a linear combination of the class probabilities, which can be problematic if the number of classes is very big.

Deep Metric Learning

Another approach is to use deep metric learning techniques. During the training the loss forces the representations of the same class images to be closer, and distant for the different class images, in Euclidean space.
This approach makes it easy to use nearest neighbor classifier on top the representation vectors and solves the scaling problem.
However the disadvantage of this approach is the difficulty of the training process, since the training can diverge and be slow.
An example of deep metric learning approached is a Triplet-Loss approach.

The Importance of the Loss Function

First of all we needed to define a generic representation that can be used to solve the handwritten analysis tasks, such as writer identification/verification.
As a first attempt we trained a classification network, in order to use the resulting network as a ground for deep metric learning approach. Since the classification network has it’s disadvantages, a triplet loss based training was performed, which unfortunately diverged (due to large intraclass variability of data) and this was the main point to rethink the loss function architecture.
To fix the divergence we had to come up with a new loss function.
Since the loss uses Bhattacharyya’s distance formula for Gaussian distributions, we decided to name the loss itself after him as well.
The Loss uses two Gaussian distribution histograms, which instead of creating lots of triplets calculates the interclass and intraclass distance histograms and stimulates Gaussians to be and as far as possible (have minimal overlap) from each other and to preserve the Gaussian shape.
Here is an example result from the training.
With overlap ~6 %. Then this feature extractor can be used further.

Building and Training a Toy Network

Take two system fonts, that look similar to handwriting ones.
Define functions to create images from words.
Generate a dataset labeled images of words in different fonts.
Split the dataset into a training set and a test set.
Create an encoder that pre-process the image and creates a proper tensor of data.
We need a few utilities to construct the RNN and CNN architectures.
Now finalize the architectures.
Test if everything works.
Finally train the model.
And finally, let’s see how the toy example works.

Applications: Handwritten Recognition

Take a sample of a handwritten text.
First step is to detect the lines where the text is located.
Next, we can define a lexicon to be more precise in the recognition result.
Combine the recognition and the minimal post-processing into a simple wrapper.
Recognize the text from our sample.
Now, let’s apply the lexicon on the result:
We could then go ahead and apply any language model to improve the recognition result if desired.
See how the generalized network trained by us compares to the state of the art.
Here is an example recognition of a final result.
Future Improvements thing that may improve the accuracies
◼
  • Variable length architecture, possible will bring gain in speed
    • ◼
  • Bigger and cleaner data
    • ◼
  • Improved synthetic generation (degradation mechanisms, GANs to synthesize or degrade synthetic images)
    • ◼
  • More NLP in the architecture

    • Applications: Handwritten Verification

    The problem we will try to solve here, is to verify if two handwritten texts belong to the same writer. This problem is also known as forensic analysis of handwritten text.
    The resulting network that we created, is a deep feature extractor that takes important information to be used not only to distinguish writers, but also characterize handwritten text images without prior information what the text is about. In other words, the network allows to verify if the handwriting belongs to a specific person independently what the context of the text is.
    An example usage would be to extract the feature vector from the input image, that contains a handwritten text.
    We will need a utility function to divide an image into patches:
    And another small wrapper that will load the pre-trained networks, and perform the verification.
    See how accurate the verification is overall.

    Applications: Handwritten Identification

    The problem we will try to solve here, is to identify the author of a handwriting.
    Create a small train data.
    And a test data.
    Simple classification using the train data.
    And with the tiny datasets, we already have 80% accuracy.

    Applications: Signature Verification/Identification

    The problem we will try to solve here, is to identify and verify the signature of a person.
    Take a list of signatures associated to the writer.
    The data has 2 types of images: the writers original signature and the one faked by someone else.
    First, we want to solve the identification part by classifying the data.
    The results are pretty impressive.
    The owner of a signature can now be identified with high accuracy.
    Now we need to solve the verification part of the problem. For that we make pairs of signatures, half of which contain pair of signatures that belong to the same person, and the other half contains pair of signatures from different people.
    Train a classifier using the newly grouped dataset.
    And again signature can be validated with high accuracy.
    The state of the art for this problem from 2017 was 4.63%, whereas we already do better.

    Init