Human pose estimation is the task of estimating the positions and orientations of a person’s body joints from images or videos. This project attempts to tackle the 3-dimensional variant of this problem using state-of-the-art computer vision techniques. The CenterNet model is used to obtain 2D keypoints from image frames, estimate the depth of each keypoint using the MiDaS depth estimation model imported via ONNX, and finally scale and visualize the 3D skeletons in an interactive display.
Introduction
Introduction
Human pose estimation is a critical computer vision task that involves estimating the positions and orientations of a person’s body joints from an image or a video. It plays a crucial role in a wide range of applications, including action recognition, virtual reality, human-computer interaction, and surveillance systems. The 2D version of pose estimation focuses on inferring joint locations in a 2D image, while the 3D version aims to estimate the 3D positions of the joints, providing a more detailed representation of human movement in physical space. Both variants contribute to understanding human behavior, enabling various practical applications and enhancing our interaction with technology.
This project adopts a two-step approach to estimate 3D human pose. Initially, the 2D coordinates of 17 keypoints on the human body are estimated using CenterNet, a deep learning model designed for object recognition and pose estimation. These 2D coordinates serve as a foundation for understanding joint locations. To extract depth information, we utilize MiDaS, a state-of-the-art monocular depth estimation model. By appending the depths to the 2D coordinates, we establish linkages between body joints and visualize the pose in 3D. The project focuses on single person 3D human pose estimation from images within the Summer School, with preliminary results on single person videos. This integration of 2D and 3D pose estimation techniques contributes to advancing human pose estimation and its practical applications, offering insights into human movement and enabling immersive technologies.
The image below provides a summary of the pipeline for extracting the 3D keypoints.
This project adopts a two-step approach to estimate 3D human pose. Initially, the 2D coordinates of 17 keypoints on the human body are estimated using CenterNet, a deep learning model designed for object recognition and pose estimation. These 2D coordinates serve as a foundation for understanding joint locations. To extract depth information, we utilize MiDaS, a state-of-the-art monocular depth estimation model. By appending the depths to the 2D coordinates, we establish linkages between body joints and visualize the pose in 3D. The project focuses on single person 3D human pose estimation from images within the Summer School, with preliminary results on single person videos. This integration of 2D and 3D pose estimation techniques contributes to advancing human pose estimation and its practical applications, offering insights into human movement and enabling immersive technologies.
The image below provides a summary of the pipeline for extracting the 3D keypoints.
2D Human Keypoint Estimation
2D Human Keypoint Estimation
As is the case with most of the state art approaches, the 3D pose estimation problem in this project has been split into two modules. The first module focuses on estimating the 2D keypoints corresponding the joints of humans identified in any image frame. This is done using the CenterNet NetModel.
CenterNet Pose Estimation Net
CenterNet Pose Estimation Net
CenterNet is a family of models centered around image recognition, object detection and keypoints estimation released in 2019 and incrementally extended since then. The Wolfram Neural Repository contains a NetModel implementation i.e, CenterNet Pose Estimation Net that has been trained on MS-COCO Data to detect and localize human joints and objects in an image.
In[]:=
centernetModel=NetModel["CenterNet Pose Estimation Nets Trained on MS-COCO Data"]
The model takes an image of size 512x512 and gives 8 outputs. Out of these 8, the most relevant to our task is the PoseOffsets, PosePeaks and PoseHeatmaps. The other outputs ports such as ObjectHeatmaps, ObjectPeaks, BoxSizes and BoxOffsets aid in object recognition and mapping in the images.
Post processing for 2D keypoint extraction
Post processing for 2D keypoint extraction
There are 17 heatmaps each corresponding to a joint/keypoint on the human body. Top peaks are extracted by indexing the PosePeaks at high intensity values of the PoseHeatmaps. During the extraction process, peaks below a certain threshold are omitted. The PosePeaks for each joint are then corrected/adjusted using the values from the PoseOffsets array.
Keypoint Extraction
Keypoint Extraction
As a part of post processing, an alignment operation is done where the person dependent keypoints are best matched to person agnostic accurate keypoint values. Points which are occluded and are unidentifiable are replaced with the missing values.
Keypoint Alignment
Keypoint Alignment
A top level function takes in the image and returns predictions for object detection, keypoint estimation and keypoint confidence.
netevaluate
In[]:=
Options[netevaluate]=Join[Options[alignKeypoints],Options[decode]];netevaluate[net_,img_Image,opts:OptionsPattern[]]:=Block[{scale,predictions},predictions=decode[net,img,Sequence@@DeleteCases[{opts},"NeighborhoodRadius"|"FilterOutsideDetections"->_]];scale=Max@N[ImageDimensions[img]/Reverse@Rest@NetExtract[net,NetPort["PosePeaks"]]];alignKeypoints[predictions,"FilterOutsideDetections"->OptionValue["FilterOutsideDetections"],"NeighborhoodRadius"->OptionValue["NeighborhoodRadius"]*scale]]
The following lines of code demonstrate the model in action
In[]:=
testImage=
;
In[]:=
predictions=netevaluate[centernetModel,testImage];
In[]:=
classes=DeleteDuplicates@Flatten@predictions[["ObjectDetection",All,2]]
Out[]=
{person}
In[]:=
keypoints=predictions["KeypointEstimation"];
In[]:=
HighlightImage[testImage,DeleteMissing[keypoints,2]]//ImageResize[#,{128,Automatic}]&
Out[]=
Using the joint connection information as a list of tuples, we can now plot the skeleton.
In[]:=
jointIndices=;getSkeleton[personKeypoints_]:=Line[DeleteMissing[Map[personKeypoints[[#]]&,jointIndices],1,2]]
In[]:=
HighlightImage[testImage,Append[AssociationThread[Range[Length[#]]->#]&/@{keypoints,Map[getSkeleton,keypoints]},GroupBy[predictions["ObjectDetection"][[All,;;2]],Last->First]],ImageLabels->None]
Out[]=
Depth Estimation of the Keypoints
Depth Estimation of the Keypoints
The next step involves obtaining the depths i.e, z coordinates of the 2D keypoints extracted from the CenterNet. The general idea was to use a monocular depth estimation model to obtain the z value at (x,y) joint location in the image. First attempt was using the Single Image Depth Perceptron Net trained on NYU Depth V2 and Depth in the Wild Data from the Wolfram Neural Network Repository. Due to unsatisfactory results and large inference time, this approach was dropped and alternatively, MiDaS, a monocular depth estimation vision transformer released in 2022, was imported via onnx and used to capture the depths.
Single Image Depth Perceptron Net
Single Image Depth Perceptron Net
Released in 2016, this neural net was trained to predict the relative depth map from a single image using a novel technique based on sparse ordinal annotations. The model takes in an RGB image of size 3 x 320 x 240 and returns a depth map of size 320 x 240. To make it consistent with the CenterNet framework, the encoder was replaced to accept images of 512 x 512 dimensions
In[]:=
depthNet=NetModel["Single-Image Depth Perception Net Trained on NYU Depth V2 and Depth in the Wild Data"]
Out[]=
In[]:=
depthNet=NetReplacePart[depthNet,"Input"->NetEncoder[{"Image",{512,512}}]]
Out[]=
Testing it on the from the previous section. Some images have an alpha channel in addition to RGB. It is removed before passing to the depthMap.
testImage
In[]:=
rescaledTestImage=ImageResize[RemoveAlphaChannel[testImage],{512,512}];
In[]:=
depthMap=depthNet[rescaledTestImage];
The average time it takes for the Depth Perceptron net to inference depths from an image on my machine is
Plotting a histogram of the image depths reveals their concentration in the range of approximately -10 to 10. The variance, calculated at just 14, significantly constrains the potential for visualizing depth variations.
MiDaS - Monocular Depth Estimation Net
MiDaS - Monocular Depth Estimation Net
Implemented using pytorch, MiDaS is a state of art monocular depth estimation model released in 2022 that estimates depth differences in different parts of the foreground objects with good accuracy and more efficiently. The model was exported to ONNX (Open Neural Network Exchange) format using this script provided in the repository.
The model can be downloaded from here. The model can be imported as a NetGraph using the following code by providing the path.
The model can be downloaded from here. The model can be imported as a NetGraph using the following code by providing the path.
The model takes in an RGB image of dimensions 3 x 384 x 384 and returns a depth map of size 384 x 384. Let’s scale the image and reorder the channels as needed by the model.
The average time it takes for the MiDaS mode to inference depths from an image on my machine is
The inference using MiDaS is experimentally 1.5 times faster than on the Depth Perception Net.
The image depth histogram shows concentrations primarily within the 0 to 5000 range. With a variance of around 1700000, even the smallest depth changes can be visualized effectively. This surpasses the previous depth model by offering a wider range and better sensitivity to subtle depth variations.
Combining the models together
Combining the models together
Now let’s head to the next section to see the results!
Visualization and Results
Visualization and Results
Let’s start with this test image.
Looking at the keypoints, we can now observe that they are 3-dimensional,
Next, we scale the z values obtained from the depth model ranging from 0 to any positive number to between 0.4 to 0.6 inclusive. Since x and y are already scaled from 0 to 1, we must scale z within the same unit square. The center remains at 0.5 to keep the skeleton centered in the visualization. The 0.2 depth range is based on the average human height-to-depth ratio of 1:7. This means the height should theoretically span a range of 0.14. However, accounting for the possibility of outstretched arms or legs, we round this up to 0.2.
We can try plotting the points in a 3D plot to see the preliminary results.
This view isn’t very helpful. For a decent visualization, joints can be plotted as spheres in 3D and the bones/edges can be plotted as cylinders. The two utility functions defined below take the keypoints and return the corresponding Graphics3D objects.
Combining all of these utilities into a single function to plot the skeleton.
Another meaningful visualization is to have skeleton stand in front of the image for better comparison. The following compact function implements a mesh passing through the two bottom most keypoints and the image overlay in the xy plane given the keypoints.
Extension to Videos
Extension to Videos
The same utility functions could be used to apply pose estimation on videos by applying them on each frame and animating the output. Let’s import a short video.
You can play the transformed frames as an animation.
Exporting the resultant video as mp4.
Future Directions
Future Directions
This project can be extended in several directions.
1
.Multi-person support: The centerNet model has the ability to identify 2D keypoints of multiple persons. Beyond WSS, the project can be extended to estimate 3D keypoints from multiple people within the image yet still group and view them as separate human skeletons.
2
.Occlusion handling: The framework does not give good results when the body parts an occluded. A challenging yet useful future direction would be to implement an occlusion resistant model that can estimate keypoint even if they are hidden by obstacles.
3
.Video Support: Potential ways could be explored to optimize and implement the 3D estimation pipeline for videos.
4
.Mesh Regression: With the 3D points obtained, work can potentially be done to lift the 3D pose to a typical human mesh.
Concluding remarks
Concluding remarks
Conclusively, this project explored 3D human pose estimation in the wolfram language. To accomplish this, first 2D keypoints were obtained using the CenterNet model from the Wolfram Neural Network Repository. Next, the pretrained MiDaS Monocular Depth Estimation model was exported as an .onnx file and imported as a NetGraph into WL. This was used to obtain the z-coordinate for the 2D keypoints. The 3D keypoints were visualized using an interactive display. This project serves as a baseline on 3D human pose estimation, especially in WL where no such implementation exists. This can be further built to do complex tasks like mesh regression and human reconstruction.
Keywords
Keywords
◼
Pose Estimation
◼
Depth Estimation
◼
Computer Vision
◼
Convolutional Neural Networks
◼
3D Visualization
Acknowledgment
Acknowledgment
I would like to thank my mentor, Maria Sargsyan, for her unconditional support and guidance. Thank you for brainstorming with me! I also appreciate our TA, Mark Greenberg, for his helpful tips and debugging. Lastly, I am grateful to Stephen Wolfram for suggesting and inspiring this project.
References
References
◼
[1] Ranftl, René, Alexey Bochkovskiy, and Vladlen Koltun. “Vision transformers for dense prediction.” Proceedings of the IEEE/CVF international conference on computer vision. 2021.
◼
[2] Zhou, Xingyi, Dequan Wang, and Philipp Krähenbühl. “Objects as points.” arXiv preprint arXiv:1904.07850 (2019).
CITE THIS NOTEBOOK
CITE THIS NOTEBOOK
3D human pose estimation using machine learning
by Fizza Rubab
Wolfram Community, STAFF PICKS, July 12, 2023
https://community.wolfram.com/groups/-/m/t/2958416
by Fizza Rubab
Wolfram Community, STAFF PICKS, July 12, 2023
https://community.wolfram.com/groups/-/m/t/2958416