Creating a 3D Image from a DepthMap






4.94/5 (49 votes)
Introducing a class for generating 3D images (Stereoscopic and Anaglyph) from DepthMaps.
- Download C# sample bin + library - 18.8 KB
- Download source code of library and samples (C# and VB.NET) - 43.3 KB
Introduction
When we come to 3D images/videos we must know that a simple 3D image is created by two images from an object, but from different angles, one per eye. Then brain can use this difference and create a depth map for itself. This will give it an idea of the outer world, how objects are near or how they are far.
But in this article, we don’t have two images. We only have one. But we have a depth map too. A depth map is a gray scale image that describes the distance of objects in an image. For example, you can see an image and its related depth map below:
Images are under copyright of http://www.dofpro.com/.
And with this class, you can generate a 3D stereoscopic image from these two images.
Samples
There are two samples in the source code. One in C# and the other in VB.NET. You can test the library before coding. Also there is a good library of images/depth maps on the DOF Pro web site. You can use them for testing.
- http://www.dofpro.com/cgigallery.htm
- http://www.dofpro.com/complexgallery.htm
- http://www.dofpro.com/cagallery.htm
But do not forget the Copyright. You should also read the "Terms of use" here: http://www.dofpro.com/terms.htm
Using the code
Using this class is simple. You can generate a 3D image with a maximum of three lines of code:
- First you need to create a new instance of this class:
_3DImageGenerator.c_3DGenerator gen = new _3DImageGenerator.c_3DGenerator();
Dim gen as New _3DImageGenerator.c_3DGenerator()
bool success = gen.GenerateStereoscopic((Bitmap)i_image, (Bitmap)i_depth);
Dim success As Boolean = gen.GenerateStereoscopic(i_image, i_depth)
pn_output.BackgroundImage = gen.Stereoscopic_SideBySide;
gen.SaveStereoscopic("c:\\example.jps", 80);
pn_output.BackgroundImage = gen.Stereoscopic_SideBySide
gen.SaveStereoscopic("c:\example.jps", 80)
Please note that there are two types of functions in the class for creating:
- Stereoscope images (two images for left and right eye): http://en.wikipedia.org/wiki/Stereoscopy
- Anaglyph images: a stereoscopic image that is merged and can be seen by a simple anaglyph glasses as a print or digital image in monitor: http://en.wikipedia.org/wiki/Anaglyph_image
You can read more about properties and functions in the next section.
Class properties and methods
This class has a number of properties and methods that can be used to change the process of generating 3D images:
Anaglyph
: A bitmap object. Output of the anaglyph generation process. You first must callGenerateAnaglyph
or this property will remainnull
.Stereoscopic_RightChannel
: A bitmap object. One of the outputs of the stereoscopic image generation process. You must first callGenerateStereoscopic
or this property will remainnull
.Stereoscopic_LeftChannel
: A bitmap object. One of the outputs of the stereoscopic image generation process. You must first callGenerateStereoscopic
or this property will remainnull
.Stereoscopic_SideBySide
: A bitmap object. Contains both right and left images of the stereoscopic image generation process. You must first callGenerateStereoscopic
or this property will remainnull
.MaxPixelDisplacement
: Get or set the max movement of pixels in displacement.Smoothing
: Fill between gaps after displacement with edge pixels or leave them black.SwapRightLeft
: Swapping right and left channels.InverseDepthMap
: Can be used for inverting pixels of the input depth map. Very usable when you don’t know exactly how the depth map is generated. (Some cameras create a depth map with “More Near-More Black” rules and you need to inverse them.)
SaveStereoscopic
: Will save a stereoscopic image as a jps file. A jps file is a simple JPG file that counts both right and left channels. Can be opened by programs like nVidia 3D Vision Photo Viewer.SaveAnaglyph
: Will save the anaglyph output as file.GenerateStereoscopic
: Will create/update stereoscopic outputs by given image and depth map. Must be called before using outputs. Return value is boolean.GenerateStereoscopicAsync
: Asynchronous version ofGenerateStereoscopic
. After calling this method, you must wait for theStereoscopicComplete
event.GenerateAnaglyph
: Will create/update an anaglyph output by given image and depth map. Must be called before using any output. Return value is boolean.GenerateAnaglyphAsync
: Asynchronous version ofGenerateAnaglyph
. After calling this method, you must wait for theAnaglyphComplete
event.
How this works
What does the brain want to estimate an object’s distance?! Difference. How can we give this difference to the brain?! By making two images from two angles. And what is the difference of two images from a different angle?! Place of objects of-course. In this code, we try to move objects depending on how much they are near or far to us. This operation is called displacement in Image Processing and we do it dynamically based on the depth map, and you can call it Dynamic Image Displacement. And at end, if we have “Smoothing” enabled, we will try to fill the remaining spaces by pixels created from both edges of the row. You can see parts of the code in the next section. Also source code is documented so you can easily read it, even if you are not a VB.NET programmer.
A little code
Here is the GenerateStereoscopic
method from the class. Written in VB.NET but each line documented so you can understand it easily.
It will give you an idea from the last part about how this class works:
'' Checking if image and depthmap have same size
If Image.Width <> DepthMap.Width OrElse Image.Height <> DepthMap.Height Then
Throw New ArgumentException("Size of Image and DepthMap are not same.")
'' Check if image and depthmap are 24bitRGB or not
If Image.PixelFormat <> PixelFormat.Format24bppRgb OrElse _
Image.PixelFormat <> PixelFormat.Format24bppRgb Then
Throw New ArgumentException("Image and/or DepthMap are/is not 24bitRGB")
Try
'' Locking image and depthmap so other threads
'' cant access them when we work on them
SyncLock Image : SyncLock DepthMap
'' Create CH2 bitmap for saving output
b_CH2 = New Bitmap(DepthMap.Width, DepthMap.Height)
'' Create a rect object, Same size as CH2 bitmap.
'' Need for direct access in memory
Dim r_CH2 As Rectangle = _
New Rectangle(0, 0, DepthMap.Width, DepthMap.Height)
'' Create CH1 bitmap for saving output
b_CH1 = New Bitmap(DepthMap.Width, DepthMap.Height)
'' Create a rect object, Same size as CH1 bitmap.
'' Need for direct access in memory
Dim r_CH1 As Rectangle = _
New Rectangle(0, 0, DepthMap.Width, DepthMap.Height)
'' Calculating real width of image (By byte)
Dim i_width As Integer = DepthMap.Width * 3
If i_width Mod 4 <> 0 Then
i_width = 4 * (i_width / 4 + 1)
End If
'' How much we need to move each pixel per depth byte
Dim hsrate As Double = i_maxDisplacement / 255
'' Creating a rect object with same size. For Depth map
Dim r_depth As Rectangle = _
New Rectangle(0, 0, DepthMap.Width, DepthMap.Height)
'' Opening direct access to bitmap data in memory for Depth map
Dim d_depth As BitmapData = DepthMap.LockBits(r_depth, _
ImageLockMode.ReadOnly, _
System.Drawing.Imaging.PixelFormat.Format24bppRgb)
'' Creating a rect object with same size. For Image
Dim r_image As Rectangle = New Rectangle(0, 0, Image.Width, Image.Height)
'' Opening direct access to bitmap data in memory for Image
Dim d_image As BitmapData = Image.LockBits(r_image, _
ImageLockMode.ReadOnly, _
System.Drawing.Imaging.PixelFormat.Format24bppRgb)
'' Opening direct access to bitmap data in memory for CH2
Dim d_ch2 As BitmapData = b_CH2.LockBits(r_CH2, ImageLockMode.ReadWrite, _
System.Drawing.Imaging.PixelFormat.Format24bppRgb)
'' Opening direct access to bitmap data in memory for CH1
Dim d_ch1 As BitmapData = b_CH1.LockBits(r_CH1, _
ImageLockMode.ReadWrite, _
System.Drawing.Imaging.PixelFormat.Format24bppRgb)
Dim sfp As Integer
For y As Integer = 0 To DepthMap.Height - 1
'' Calculate location of current line's last free pixel
Dim rLPDest As IntPtr = (y + 1) * i_width - (i_maxDisplacement * 3) - 3
'' Calculate location of current line's first free pixel
Dim rFPDest As IntPtr = y * i_width + (i_maxDisplacement * 3)
'' Count for each pixel on width of image.
'' Cut MaxDisplacementfrom from both sides
For x As Integer = i_maxDisplacement To _
DepthMap.Width - 1 - i_maxDisplacement
''''''''''''''''''''''''''''''''''' Right 2 Left
'' Read Depth, Right to Left
Dim depthrgb As Byte = ReadByte(d_depth.Scan0 + rLPDest + 1)
If InverseDepthMap Then depthrgb = 255 - depthrgb
'' Calculate displacement offset, Right to Left
sfp = depthrgb * hsrate
'' Read a pixel from image, Right to Left
Dim imagergb(2) As Byte
Copy(d_image.Scan0 + rLPDest, imagergb, 0, 3)
'' Correct CH2 Displacement, Right to Left
Copy(imagergb, 0, d_ch1.Scan0 + rLPDest + ((sfp) * 3), 3)
'' Smoothing
If b_Smoothing And sfp <> 0 Then
'' Calculate color changes between pixels (For better
'' smoothing we use 4 pixel and then get an average)
Dim ich2Rgrate, ich2Ggrate, ich2Bgrate As Double
Dim db(11) As Byte
Copy(d_image.Scan0 + rLPDest - 12, db, 0, 12)
db(11) = CType((CType(db(11), Integer) + _
CType(db(8), Integer)) / 2, Byte)
db(10) = CType((CType(db(10), Integer) + _
CType(db(7), Integer)) / 2, Byte)
db(9) = CType((CType(db(9), Integer) + _
CType(db(6), Integer)) / 2, Byte)
db(2) = CType((CType(db(2), Integer) + _
CType(db(5), Integer)) / 2, Byte)
db(1) = CType((CType(db(1), Integer) + _
CType(db(4), Integer)) / 2, Byte)
db(0) = CType((CType(db(0), Integer) + _
CType(db(3), Integer)) / 2, Byte)
'' Split color changes between pixels that we
'' need to write. So we can create a gradient effect
ich2Rgrate = (CType(db(2), Integer) - _
CType(db(11), Integer)) / (sfp + 1)
ich2Ggrate = (CType(db(1), Integer) - _
CType(db(10), Integer)) / (sfp + 1)
ich2Bgrate = (CType(db(0), Integer) - _
CType(db(9), Integer)) / (sfp + 1)
'' Apply Smoothing
For i As Integer = 0 To sfp - 1
'' CH2 Smoothing
db(0) = db(9) + (i * ich2Bgrate)
db(1) = db(10) + (i * ich2Ggrate)
db(2) = db(11) + (i * ich2Rgrate)
Copy(db, 0, d_ch1.Scan0 + rLPDest + _
(((sfp - 1) - i) * 3), 3)
Next
End If
'' Go to Last Pixel
rLPDest -= 3
''''''''''''''''''''''''''''''''''' Left 2 Right
'' Read Depth, Left to Right
depthrgb = ReadByte(d_depth.Scan0 + rFPDest + 1)
If InverseDepthMap Then depthrgb = 255 - depthrgb
'' Calculate displacement offset, Left to Right
sfp = depthrgb * hsrate
'' Read a pixel from image, Left to Right
Copy(d_image.Scan0 + rFPDest, imagergb, 0, 3)
'' Correct CH1 Displacement, Left to Right
Copy(imagergb, 0, d_ch2.Scan0 + rFPDest + (-sfp * 3), 3)
'' Smoothing
If b_Smoothing And sfp <> 0 Then
'' Calculate color changes between pixels
'' (For better smoothing we use
'' 4 pixel and then get an average)
Dim ich1Rgrate, ich1Ggrate, ich1Bgrate As Double
Dim db(11) As Byte
Copy(d_image.Scan0 + rFPDest + 3, db, 0, 12)
db(11) = CType((CType(db(11), Integer) + _
CType(db(8), Integer)) / 2, Byte)
db(10) = CType((CType(db(10), Integer) + _
CType(db(7), Integer)) / 2, Byte)
db(9) = CType((CType(db(9), Integer) + _
CType(db(6), Integer)) / 2, Byte)
db(2) = CType((CType(db(2), Integer) + _
CType(db(5), Integer)) / 2, Byte)
db(1) = CType((CType(db(1), Integer) + _
CType(db(4), Integer)) / 2, Byte)
db(0) = CType((CType(db(0), Integer) + _
CType(db(3), Integer)) / 2, Byte)
'' Split color changes between pixels that
'' we need to write. So we can create a gradient effect
ich1Rgrate = (CType(db(2), Integer) - _
CType(db(11), Integer)) / (sfp + 1)
ich1Ggrate = (CType(db(1), Integer) - _
CType(db(10), Integer)) / (sfp + 1)
ich1Bgrate = (CType(db(0), Integer) - _
CType(db(9), Integer)) / (sfp + 1)
'' Apply Smoothing
For i As Integer = 0 To sfp - 1
'' CH1 Smoothing
db(0) = db(9) + ((sfp - i) * ich1Bgrate)
db(1) = db(10) + ((sfp - i) * ich1Ggrate)
db(2) = db(11) + ((sfp - i) * ich1Rgrate)
Copy(db, 0, d_ch2.Scan0 + rFPDest + (-i * 3), 3)
Next
End If
'' Go to Next Pixel
rFPDest += 3
Next
Next
'' Closing direct access
DepthMap.UnlockBits(d_depth)
Image.UnlockBits(d_image)
b_CH2.UnlockBits(d_ch2)
b_CH1.UnlockBits(d_ch1)
End SyncLock : End SyncLock
Return True
Catch ex As Exception
Return False
End Try
Points of interest
- One of the interesting things in this code is using the
System.Runtime.InteropServices.Marshal
class for doing unsafe operations which are normally not supported in VB.NET; this will give us the ability to access data directly in memory that is much faster than using functions likeGetPixel
orSetPixel
. - I originally wrote this class for an application that can get the depth and RGB stream from Kinect (Microsoft’s motion capture device, used for an Xbox gaming console) and then can create a 3D image based on that information. You can download the source code from this address: http://www.kinectdevs.com/forums/filebase.php.