Goal: Make a 3D reconstruction of an object, from a pair of images taken from slightly different points of view.
Main results in detail: In this project we present a method for reconstructing a 3D object, from a pair of images. We first align the images, so corresponding points are aligned horizontally, then we calculate the disparity map, or information about horizontal differences between images, then we make a 3D reconstruction from the depth map derived from the disparity map. We also compare different approaches for calculating the disparity map of the pair of images, from Neural Networks approaches to more classical techniques.
Future work: Fully 3D object reconstruction from consecutive pairs of images. Use the depth information for computational photography applications, i.e. depth of field effect.
Main results in detail: In this project we present a method for reconstructing a 3D object, from a pair of images. We first align the images, so corresponding points are aligned horizontally, then we calculate the disparity map, or information about horizontal differences between images, then we make a 3D reconstruction from the depth map derived from the disparity map. We also compare different approaches for calculating the disparity map of the pair of images, from Neural Networks approaches to more classical techniques.
Future work: Fully 3D object reconstruction from consecutive pairs of images. Use the depth information for computational photography applications, i.e. depth of field effect.
The term steropsis refers to the perception of depth and 3-Dimensional structure. Human eyes are located at different horizontal positions in the head, resulting in two slightly different images projected to the retinas of the eyes. These positional difference are referred to as disparities, and the differences are mainly in the horizontal position of objects in the two images. These disparities are processed in the brain to yield depth perception with the following logic: the larger the disparity is, the closer the object is located. Estimating depth is a long-standing problem in computer vision. Traditionally, multiple images have been used to estimate depth, and a common technique is the stereo technique, based in the above explained steropsis. Today, mobile phones play a key role in our lives and we use them extensively everyday. Therefore, we propose a method that given a pair of mobile photographs of one object, from different but nearby view points, it reconstructs a 3D model of the image. It consists of 3 steps: - The first step is to vertically align the obtained images: The different techniques explained here search for horizontal differences between images, therefore the two images have to be aligned vertically. - Secondly, the horizontal displacements are measured: We will compare different techniques to calculate the horizontal differences between images. From Neural Networks approaches to more classical techniques. In this step we will obtain a disparity map, which encodes the difference in horizontal coordinates of corresponding image points. The depth map of the image is obtained from the disparity map. - Finally, we make a 3D reconstruction of the object, using the depth information of the scene.
Image alignment
Image alignment
Given a pair of mobile phone images, of the same scene, taken from different but nearby points of view, the first step will be to align them vertically.
In[]:=
{left,right}=
,
;
First, we have to find corresponding image key-points. Then, a transformation consisting of of a rotation, translation, rotation followed by scaling is calculated, and applied to the left image so the key-points are aligned horizontally.
{leftAdjusted,right}
Depth map
Depth map
In this section, we compare different techniques for obtaining a depth map.
EMPIRIC: Ground truth depth map, obtained from a two-camera mobile phone.
In[]:=
depth=
;
ImageAdjust@depth
Out[]=
MONO-OCULAR: This Neural Net from Mathematica repository was trained to predict the relative depth map from a single image using a novel technique based on sparse ordinal annotations.
MONO-OCULAR: This Neural Net from Mathematica repository was trained to predict the relative depth map from a single image using a novel technique based on sparse ordinal annotations.
depthNN=NetModel["Single-Image Depth Perception Net Trained on Depth in the Wild Data"];
NNdepth=ImageResize[ImageAdjust@Image[depthNN[First[leftRight]]],ImageDimensions[First[leftRight]]]
STEREO: The ImageDisplacements Mathematica built-in function gives estimated horizontal and vertical displacements between consecutive images. We calculate a depth map from the above pair of images.
STEREO: The ImageDisplacements Mathematica built-in function gives estimated horizontal and vertical displacements between consecutive images. We calculate a depth map from the above pair of images.
Create a displacement map:
parallax=First@ImageDisplacements[ConformImages[{leftAdjusted,right}]];
Only consider the horizontal displacement and delete unreasonable value:
depth2=ImageAdjust@Image[Clip[parallax[[All,All,1]],{-64,64}]]
REDTAIL DNN: This semi-supervised net is based on the paper "On the importance of stereo for accurate depth estimation: An efficient semi-supervised deep neural network approach". Given a pair of horizontally aligned images, it produces a depth map.
REDTAIL DNN: This semi-supervised net is based on the paper "On the importance of stereo for accurate depth estimation: An efficient semi-supervised deep neural network approach". Given a pair of horizontally aligned images, it produces a depth map.
Architecture of the REDTAIL binocular stereo network to estimate disparity (and hence depth) .
We tried to implement a scaled down version of the DNN described in the article, since DeconvolutionLayers in WL cannot handle 3D deconvolutions. However, this scaled-down version was not described in detail and there appears to be a crucial ingredient missing. We were not able to produce useful results yet.
We tried to implement a scaled down version of the DNN described in the article, since DeconvolutionLayers in WL cannot handle 3D deconvolutions. However, this scaled-down version was not described in detail and there appears to be a crucial ingredient missing. We were not able to produce useful results yet.
3D reconstruction
3D reconstruction
A 3D scene can be reconstructed from a pair of 2D views taken from slightly different points of view. First, a disparity map is calculated and then the depth map is derived. With this depth information, you can extrude the vertices of an object mesh obtained through ImageMesh and TriangulateMesh. The result is displayed via texture mapping on the 3D mesh object.
From the depth map, we create a mask:
depth=ColorNegate[depth];
mask=Erosion[Binarize[depth,0],3]
We estimate the needed resolution in the mesh, according the the details in the depth map. The more details, the more refined mesh. For estimating the details in the depth map, we apply a gradient filter to it.
resolution=ImageClip[0.005+GradientFilter[depth,12],{0,0.01}]//ImageAdjust
Finally, the object mesh is extruded with the depth function, and the texture is applied to it:
Future work
Future work
Complete object 3D reconstruction: The aim of this work would be to be able to fully reconstruct the 3D structure of objects, by applying iteratively this method, to pairs of consecutive images.
"Depth of field" analog photography effect for portraits: Some effects of analog photography, like the depth of field or the "bokeh" effect, can be simulated by computational photography methods. We can reuse our depth estimation method to simulate the depth of field effect, where the objects in the foreground appear sharp and in-focus, and the elements in the background are out of focus. Then, the attention of the image goes to the elements in the foreground. This effect is widely used in portrait photography.