American Sign Language (ASL) is the predominant language within Deaf and hard-of-hearing communities in the US and Canada. The ASL alphabet consists of 26 signs corresponding to 26 letters of the English alphabet, with most being static and two involving movement. This project aimed to map the positions of the finger and joints of these signs on a Cartesian coordinate plane. This includes a frame/frames of each sign and an animation to move from one sign to another.

Process

I started by looking through the 26 signs, and determining how I wanted to plot them. I made note of the signs that shared some overlap with other signs, and also noticed how the signs for J and Z are not static.
​
When I first approached this project, I originally intended to include a machine learning model that would provide the coordinates and finger/hand joint positions of each sign. However, we quickly reached a multitude of road blocks that prevented this plan from following through. Namely, none of the approaches we took could work accurately in Mathematica while also being completed in a reasonable time frame.

Coordinates

I switched approaches from the machine learning model and decided to go through every sign and manually plot the points on a coordinate plane. I decided to have the plot be of the right hand and with the palm facing up, when applicable. The main points included the first row of knuckles (shown in red, ex. genK1, genK2), the second row of knuckles (shown in yellow, ex. aK, bK, cK), the fingertips (shown in blue, ex. aF, bF, cF), and a point dictating the bottom of the hand, which was put with the list of the first row of knuckles.
​
The variable names genK, letterK, and letterF are just shortened versions of "general knuckles", "letter knuckle" and "letter fingertip" respectively.
Here is an example of what the coordinates for the sign for A are:
In[]:=
genK1={{1.5,0.8},{0.8,4.4},{0,4.5},{-1,3.8},{-1.8,3.3},{0,0}}; ​​aK={{2.5,3},{1.1,5.6},{0.1,5.3},{-0.9,4.9},{-1.7,4.6}};​​aF={{1.7,6.8},{1,2.6},{0.3,1.8},{-0.3,2},{-0.9,2}};

Coordinates


Signs

After finding the coordinates of the joints and fingertips that I needed, I started mapping out the signs. To put it simply, I needed to connect the points I had already found. Of course, there were some difficulties with the signs for J and Z, since they involve movement.
This function to puts all the points of a sign together, while being mindful of the special cases.
In[]:=
signPoints[letter_]:=Module[{pos =Flatten[Position[Alphabet[],ToLowerCase[ToString[letter]]]]}, ​​Which[ContainsNone[pos,{7,8,10,15,16,17,26}],​​Flatten[{{genK1},fingersK1[[pos]],fingersF1[[pos]]},1],​​pos=={7}||pos=={8},Flatten[{{genK2},fingersK2[[pos-6]],fingersF2[[pos-6]]},1],​​pos=={15},{genK3,oK,oF},​​pos=={16},{genK4,pK,pF},​​pos=={17},{genK5,qK,qF},​​pos=={10},{{genK1,jK,jF},{jK1Final,jKFinal,jFFinal}},​​pos=={26},{{zK1Pos1,zKPos1,zFPos1},{zK1Pos2,zKPos2,zFPos2},{zK1Pos3,zKPos3,zFPos3},{zK1Pos4,zKPos4,zFPos4}}]]
In[]:=
signPoints[A]
Out[]=
{{{1.5,0.8},{0.8,4.4},{0,4.5},{-1,3.8},{-1.8,3.3},{0,0}},{{2.5,3},{1.1,5.6},{0.1,5.3},{-0.9,4.9},{-1.7,4.6}},{{1.7,6.8},{1,2.6},{0.3,1.8},{-0.3,2},{-0.9,2}}}

Graphing Signs

These points provide a frame for graphs of the signs. J and Z are the special cases here, since they can't be represented as static images.
Function for graphing a sign:
In[]:=
sign[letter_]:=Module[{points=signPoints[letter]},​​Which[​​ContainsNone[Flatten[Position[Alphabet[],ToLowerCase[ToString[letter]]]],{10,26}],​​ListLinePlot[Table[{#,points[[1]][[i]],points[[2]][[i]],points[[3]][[i]]},{i,5}]&[points[[1]][[-1]]], PlotRange->{{-5,5},{0,10}}],​​letter===j||letter===J,animate[{animationFrames[graphFrames[letter],pointFrames[letter],a,a]}],​​letter===z||letter===Z,animate[zFrames]]]
In[]:=
sign[A]
Out[]=
-4
-2
0
2
4
2
4
6
8
10
I started by making sure the static signs would work since I knew that J and Z would be outliers to the rest of the signs. For both of the aforementioned signs, I used my animation functions from in a later section.

Special Coordinates

The first row of knuckles (genK) are in the same position across nearly every letter but differ on a few. Specifically, this includes the signs for G, H, O, P, and Q. This basically means that a separate genK variable was needed for them.
For example, these are the coordinates for the sign for A and B, and both use genK1:
In[]:=
signPoints[A]
Out[]=
{{{1.5,0.8},{0.8,4.4},{0,4.5},{-1,3.8},{-1.8,3.3},{0,0}},{{2.5,3},{1.1,5.6},{0.1,5.3},{-0.9,4.9},{-1.7,4.6}},{{1.7,6.8},{1,2.6},{0.3,1.8},{-0.3,2},{-0.9,2}}}
In[]:=
signPoints[B]
Out[]=
{{{1.5,0.8},{0.8,4.4},{0,4.5},{-1,3.8},{-1.8,3.3},{0,0}},{{1.8,2.4},{1,6.5},{0,6.6},{-1,6},{-1.8,4.8}},{{0.8,3.5},{1,8.9},{0,9.2},{-1,8.3},{-1.8,6.5}}}
And these are the coordinates for the sign for G, which has to use genK2:
In[]:=
signPoints[G]
Out[]=
{{{1.2,3.4},{0.2,4.6},{0.7,3.7},{0.9,2.9},{1.5,1.9},{0,0}},{{2.3,3.9},{2.3,5.2},{3,4},{3.4,2.8},{3.2,1.8}},{{3.5,4.6},{4.8,5.7},{2.1,3.2},{2.3,2.2},{2.4,1.4}}}
Notice the placement of the first row of knuckles on the signs:
In[]:=
sign/@{A,B,G}
Out[]=

-4
-2
0
2
4
2
4
6
8
10
,
-4
-2
0
2
4
2
4
6
8
10
,
-4
-2
0
2
4
2
4
6
8
10

Additionally, there are two signs that involve movement, J and Z. For these, I plotted the stopping points of both signs and later used these points to create frames for an animation. This meant I found the coordinates of the beginning and end positions of the sign for J and found the coordinates of the four positions that a hand stops in when signing Z.
For instance, these are the points for J (double the normal amount of coordinates):
In[]:=
signPoints[J]
Out[]=
{{{{1.5,0.8},{0.8,4.4},{0,4.5},{-1,3.8},{-1.8,3.3},{0,0}},{{2.5,3},{1.1,5.6},{0.1,5.3},{-0.9,4.9},{-1.9,4.8}},{{1.8,6},{1,2.6},{0.3,1.8},{-0.3,2},{-1.9,6.8}}},{{{-1,3.5},{-2.5,3.2},{-2.1,2.6},{-1.2,1.8},{-0.4,1.6}},{{-1.2,5.2},{-0.9,4.9},{-0.3,4.3},{0,3.8},{0.5,2.8}},{{-0.2,5.6},{0.8,4.7},{0.8,4},{0.9,3.2},{1.4,4.2}}}}
And this is the sign for J, which uses animations from the next section:
In[]:=
sign[J]
Out[]=
-4
-2
0
2
4
2
4
6
8
10
-4
-2
0
2
4
2
4
6
8
10
-4
-2
0
2
4
2
4
6
8
10
-4
-2
0
2
4
2
4
6
8
10
-4
-2
0
2
4
2
4
6
8
10

Animations

This is where it gets fun (and even more messy). My goal was to be able to create an animation moving from one sign to another. I was also able to use this method for the outlier signs, J and Z.

Information for Frames

I needed some way of obtaining the in-between frames from one letter to the next. I started by creating pointFrames, which takes in two letters and gives out a list of pairs of coordinates where the first coordinate is the the starting position and the second coordinate is the final position.
​
The same logic was applied to J and Z, and with their starting/ending positions. What further complicated things, however, was having to move from one letter then to J or Z, or vise versa. For these particular situations, I just changed the output slightly. For instance, if I wanted to move from A to J, then the output would include a list of pairs of coordinates that start at the points for A to the starting points for J, and a list of pairs of coordinates that move from the starting points at J to the ending points for J.
Functions for pointFrames:
In[]:=
pointFrames[letter_]:=Transpose[Table[signPoints[letter][[All,first]][[All,sec]],{first,3},{sec,5}]];​​​​pointFrames[letter1_,letter2_]:=​​Which[letter1=!=j&&letter1=!=z&&letter2=!=j&&letter2=!=z,​​Transpose[Table[isFunc[signPoints[letter1],signPoints[letter2]][[All,first]][[All,sec]],{first,3},{sec,5}]],​​(letter1===j||letter1===z)&&letter2=!=j&&letter2=!=z,​​Transpose[Table[isFunc[signPoints[letter1][[-1]],signPoints[letter2]][[All,first]][[All,sec]],{first,3},{sec,5}]],​​letter1=!=j&&letter1=!=z&&(letter2===j||letter2===z),​​Transpose[Table[isFunc[signPoints[letter1],signPoints[letter2][[1]]][[All,first]][[All,sec]],{first,3},{sec,5}]],​​(letter1===j||letter1===z)&&(letter2===j||letter2===z),​​Transpose[Table[isFunc[signPoints[letter1][[-1]],signPoints[letter2][[1]]][[All,first]][[All,sec]],{first,3},{sec,5}]]]
I used a very similar approach to graphFrames when involving J and Z. This function takes in a starting letter and an ending letter, and interpolates them into functions of paths from each starting to ending point.
Functions for graphFrames:
An issue I quicky ran into was that interpolation doesn't work when the two inputted points do not create a function. That is, it cannot generate a function if the two signs have a point (such as a knuckle or fingertip) that is in the same place or that moves along a directly vertical path.
​
To accommodate for this, I added isFunc inside of both pointFrames and graphFrames to adjust an X coordinate by 0.0001 points when the beginning and ending points aren't able to be interpolated.
Making sure the two points make a valid function:

Animation Frames

To finally animate the signs, I created animationFrames, which takes in graphFrames, pointFrames, and a letter as the input and outputs six frames. The letter is only for changes in wrist placement, while graphFrames determines the path of each point and pointFrames determines the beginning and end of the path.
Finding in-between frames:
There is another small change that has to be made for J and Z when using animationFrames. They contain more nested lists than the other signs do, which means the last element of their first list is not a point for the bottom of the hand (wrist variables). Like many other signs, both J and Z need their wrist variable to be set to {0,0}, so I defaulted to inputting A for letter1 or letter2 instead of inputting J or Z.
Examples using A and J:

Animating Letters

Finally, this is where all the previous information is used to animate from one sign to another. This, as always, has exceptions for signs for J and Z. This just includes more animationFrames inside the animate function since the original J and Z have their own animationFrames.
Put frames together in an animation:
Animating from letter 1 to letter 2:

Extensions

Some possible extensions include:
◼
  • Create a machine learning model that takes in images of hands and outputs the sign or a model of the sign.
    • ◼
  • A limitation of this project's mapping is the lack of depth, so a possibility could be creating a 3D model of a real hand to show the signs.
    • ◼
  • I would also love to see the same concepts to more signs beyond the alphabet, which would make it much more applicable.

    • Acknowledgements

    I would like to thank Eli Morton, my mentor, for helping me through this project. I also want to thank the Wolfram Summer Research Staff, specifically Nicoló and Zoya for walking me through problems with machine learning and animations.

    CITE THIS NOTEBOOK

    Mapping the motion of hands between ASL alphabet signs​
    by Jenny Wang​
    Wolfram Community, STAFF PICKS, July 13, 2023
    ​https://community.wolfram.com/groups/-/m/t/2963751