Creating a 3D Image from a DepthMap

• Download C# sample bin + library - 18.8 KB
• Download source code of library and samples (C# and VB.NET) - 43.3 KB

Introduction

When we come to 3D images/videos we must know that a simple 3D image is created by two images from an object, but from different angles, one per eye. Then brain can use this difference and create a depth map for itself. This will give it an idea of the outer world, how objects are near or how they are far.

But in this article, we don’t have two images. We only have one. But we have a depth map too. A depth map is a gray scale image that describes the distance of objects in an image. For example, you can see an image and its related depth map below:

Images are under copyright of http://www.dofpro.com/.

And with this class, you can generate a 3D stereoscopic image from these two images.

Samples

There are two samples in the source code. One in C# and the other in VB.NET. You can test the library before coding. Also there is a good library of images/depth maps on the DOF Pro web site. You can use them for testing.

• http://www.dofpro.com/cgigallery.htm
• http://www.dofpro.com/complexgallery.htm
• http://www.dofpro.com/cagallery.htm

But do not forget the Copyright. You should also read the "Terms of use" here: http://www.dofpro.com/terms.htm

Using the code

Using this class is simple. You can generate a 3D image with a maximum of three lines of code:

• First you need to create a new instance of this class:
C#:
C#

_3DImageGenerator.c_3DGenerator gen = new _3DImageGenerator.c_3DGenerator();

VB.NET:
VB

Dim gen as New _3DImageGenerator.c_3DGenerator()

• Generate a stereoscopic picture and check if this process was completed successfully:
C#:
C#

bool success = gen.GenerateStereoscopic((Bitmap)i_image, (Bitmap)i_depth);

VB.NET:
VB

Dim success As Boolean = gen.GenerateStereoscopic(i_image, i_depth)

• Use the image or save it:
C#:
C#

pn_output.BackgroundImage = gen.Stereoscopic_SideBySide;
gen.SaveStereoscopic("c:\\example.jps", 80);

VB.NET
VB

pn_output.BackgroundImage = gen.Stereoscopic_SideBySide
gen.SaveStereoscopic("c:\example.jps", 80)

Please note that there are two types of functions in the class for creating:

1. Stereoscope images (two images for left and right eye): http://en.wikipedia.org/wiki/Stereoscopy
2. Anaglyph images: a stereoscopic image that is merged and can be seen by a simple anaglyph glasses as a print or digital image in monitor: http://en.wikipedia.org/wiki/Anaglyph_image

You can read more about properties and functions in the next section.

Class properties and methods

This class has a number of properties and methods that can be used to change the process of generating 3D images:

• Anaglyph: A bitmap object. Output of the anaglyph generation process. You first must call GenerateAnaglyph or this property will remain null.
• Stereoscopic_RightChannel: A bitmap object. One of the outputs of the stereoscopic image generation process. You must first call GenerateStereoscopic or this property will remain null.
• Stereoscopic_LeftChannel: A bitmap object. One of the outputs of the stereoscopic image generation process. You must first call GenerateStereoscopic or this property will remain null.
• Stereoscopic_SideBySide: A bitmap object. Contains both right and left images of the stereoscopic image generation process. You must first call GenerateStereoscopic or this property will remain null.
• MaxPixelDisplacement: Get or set the max movement of pixels in displacement.
• Smoothing: Fill between gaps after displacement with edge pixels or leave them black.
• SwapRightLeft: Swapping right and left channels.
• InverseDepthMap: Can be used for inverting pixels of the input depth map. Very usable when you don’t know exactly how the depth map is generated. (Some cameras create a depth map with “More Near-More Black” rules and you need to inverse them.)

• SaveStereoscopic: Will save a stereoscopic image as a jps file. A jps file is a simple JPG file that counts both right and left channels. Can be opened by programs like nVidia 3D Vision Photo Viewer.
• SaveAnaglyph: Will save the anaglyph output as file.
• GenerateStereoscopic: Will create/update stereoscopic outputs by given image and depth map. Must be called before using outputs. Return value is boolean.
• GenerateStereoscopicAsync: Asynchronous version of GenerateStereoscopic. After calling this method, you must wait for the StereoscopicComplete event.
• GenerateAnaglyph: Will create/update an anaglyph output by given image and depth map. Must be called before using any output. Return value is boolean.
• GenerateAnaglyphAsync: Asynchronous version of GenerateAnaglyph. After calling this method, you must wait for the AnaglyphComplete event.

How this works

What does the brain want to estimate an object’s distance?! Difference. How can we give this difference to the brain?! By making two images from two angles. And what is the difference of two images from a different angle?! Place of objects of-course. In this code, we try to move objects depending on how much they are near or far to us. This operation is called displacement in Image Processing and we do it dynamically based on the depth map, and you can call it Dynamic Image Displacement. And at end, if we have “Smoothing” enabled, we will try to fill the remaining spaces by pixels created from both edges of the row. You can see parts of the code in the next section. Also source code is documented so you can easily read it, even if you are not a VB.NET programmer.

A little code

Here is the GenerateStereoscopic method from the class. Written in VB.NET but each line documented so you can understand it easily.

It will give you an idea from the last part about how this class works:

'' Checking if image and depthmap have same size
If Image.Width <> DepthMap.Width OrElse Image.Height <> DepthMap.Height Then
    Throw New ArgumentException("Size of Image and DepthMap are not same.")
'' Check if image and depthmap are 24bitRGB or not
If Image.PixelFormat <> PixelFormat.Format24bppRgb OrElse _
            Image.PixelFormat <> PixelFormat.Format24bppRgb Then
    Throw New ArgumentException("Image and/or DepthMap are/is not 24bitRGB")
Try
    '' Locking image and depthmap so other threads
    '' cant access them when we work on them
    SyncLock Image : SyncLock DepthMap
        '' Create CH2 bitmap for saving output
        b_CH2 = New Bitmap(DepthMap.Width, DepthMap.Height)
        '' Create a rect object, Same size as CH2 bitmap.
        '' Need for direct access in memory
        Dim r_CH2 As Rectangle = _
            New Rectangle(0, 0, DepthMap.Width, DepthMap.Height)
    
        '' Create CH1 bitmap for saving output 
        b_CH1 = New Bitmap(DepthMap.Width, DepthMap.Height)
        '' Create a rect object, Same size as CH1 bitmap.
        '' Need for direct access in memory
        Dim r_CH1 As Rectangle = _
            New Rectangle(0, 0, DepthMap.Width, DepthMap.Height)
    
        '' Calculating real width of image (By byte)
        Dim i_width As Integer = DepthMap.Width * 3
        If i_width Mod 4 <> 0 Then
            i_width = 4 * (i_width / 4 + 1)
        End If  

        '' How much we need to move each pixel per depth byte 
        Dim hsrate As Double = i_maxDisplacement / 255
        '' Creating a rect object with same size. For Depth map
        Dim r_depth As Rectangle = _
            New Rectangle(0, 0, DepthMap.Width, DepthMap.Height)
        '' Opening direct access to bitmap data in memory for Depth map
        Dim d_depth As BitmapData = DepthMap.LockBits(r_depth, _
            ImageLockMode.ReadOnly, _
            System.Drawing.Imaging.PixelFormat.Format24bppRgb)

        '' Creating a rect object with same size. For Image
        Dim r_image As Rectangle = New Rectangle(0, 0, Image.Width, Image.Height)
        '' Opening direct access to bitmap data in memory for Image
        Dim d_image As BitmapData = Image.LockBits(r_image, _
            ImageLockMode.ReadOnly, _
            System.Drawing.Imaging.PixelFormat.Format24bppRgb)

        '' Opening direct access to bitmap data in memory for CH2
        Dim d_ch2 As BitmapData = b_CH2.LockBits(r_CH2, ImageLockMode.ReadWrite, _
            System.Drawing.Imaging.PixelFormat.Format24bppRgb)
        '' Opening direct access to bitmap data in memory for CH1
        Dim d_ch1 As BitmapData = b_CH1.LockBits(r_CH1, _
            ImageLockMode.ReadWrite, _
            System.Drawing.Imaging.PixelFormat.Format24bppRgb)

        Dim sfp As Integer
        For y As Integer = 0 To DepthMap.Height - 1
            '' Calculate location of current line's last free pixel
            Dim rLPDest As IntPtr = (y + 1) * i_width - (i_maxDisplacement * 3) - 3

            '' Calculate location of current line's first free pixel 
            Dim rFPDest As IntPtr = y * i_width + (i_maxDisplacement * 3)

            '' Count for each pixel on width of image.
            '' Cut MaxDisplacementfrom from both sides
            For x As Integer = i_maxDisplacement To _
                     DepthMap.Width - 1 - i_maxDisplacement

                ''''''''''''''''''''''''''''''''''' Right 2 Left
                '' Read Depth, Right to Left
                Dim depthrgb As Byte = ReadByte(d_depth.Scan0 + rLPDest + 1)
                If InverseDepthMap Then depthrgb = 255 - depthrgb
                '' Calculate displacement offset, Right to Left 
                sfp = depthrgb * hsrate 
                '' Read a pixel from image, Right to Left 
                Dim imagergb(2) As Byte
                Copy(d_image.Scan0 + rLPDest, imagergb, 0, 3)

                '' Correct CH2 Displacement, Right to Left 
                Copy(imagergb, 0, d_ch1.Scan0 + rLPDest + ((sfp) * 3), 3)
    
                '' Smoothing  
                If b_Smoothing And sfp <> 0 Then
                   '' Calculate color changes between pixels (For better
                   '' smoothing we use 4 pixel and then get an average)
                    Dim ich2Rgrate, ich2Ggrate, ich2Bgrate As Double
                    Dim db(11) As Byte 
                    Copy(d_image.Scan0 + rLPDest - 12, db, 0, 12) 
                    db(11) = CType((CType(db(11), Integer) + _
                             CType(db(8), Integer)) / 2, Byte)
                    db(10) = CType((CType(db(10), Integer) + _
                             CType(db(7), Integer)) / 2, Byte) 
                    db(9) = CType((CType(db(9), Integer) + _
                            CType(db(6), Integer)) / 2, Byte) 
                    db(2) = CType((CType(db(2), Integer) + _
                            CType(db(5), Integer)) / 2, Byte) 
                    db(1) = CType((CType(db(1), Integer) + _
                            CType(db(4), Integer)) / 2, Byte)
                    db(0) = CType((CType(db(0), Integer) + _
                            CType(db(3), Integer)) / 2, Byte)

                    '' Split color changes between pixels that we
                    '' need to write. So we can create a gradient effect 
                    ich2Rgrate = (CType(db(2), Integer) - _
                                  CType(db(11), Integer)) / (sfp + 1) 
                    ich2Ggrate = (CType(db(1), Integer) - _
                                  CType(db(10), Integer)) / (sfp + 1)
                    ich2Bgrate = (CType(db(0), Integer) - _
                                  CType(db(9), Integer)) / (sfp + 1) 

                    '' Apply Smoothing
                    For i As Integer = 0 To sfp - 1
                        '' CH2 Smoothing
                        db(0) = db(9) + (i * ich2Bgrate)
                        db(1) = db(10) + (i * ich2Ggrate)
                        db(2) = db(11) + (i * ich2Rgrate)
                        Copy(db, 0, d_ch1.Scan0 + rLPDest + _
                             (((sfp - 1) - i) * 3), 3)
                    Next
                End If
                '' Go to Last Pixel
                rLPDest -= 3

                ''''''''''''''''''''''''''''''''''' Left 2 Right
                '' Read Depth, Left to Right
                depthrgb = ReadByte(d_depth.Scan0 + rFPDest + 1)
                If InverseDepthMap Then depthrgb = 255 - depthrgb

                '' Calculate displacement offset, Left to Right  
                sfp = depthrgb * hsrate 

                '' Read a pixel from image, Left to Right
                Copy(d_image.Scan0 + rFPDest, imagergb, 0, 3) 

                '' Correct CH1 Displacement, Left to Right 
                Copy(imagergb, 0, d_ch2.Scan0 + rFPDest + (-sfp * 3), 3) 

                '' Smoothing 
                If b_Smoothing And sfp <> 0 Then 

                    '' Calculate color changes between pixels
                    '' (For better smoothing we use
                    '' 4 pixel and then get an average) 
                    Dim ich1Rgrate, ich1Ggrate, ich1Bgrate As Double
                    Dim db(11) As Byte
                    Copy(d_image.Scan0 + rFPDest + 3, db, 0, 12)
                    db(11) = CType((CType(db(11), Integer) + _
                             CType(db(8), Integer)) / 2, Byte)
                    db(10) = CType((CType(db(10), Integer) + _
                             CType(db(7), Integer)) / 2, Byte)
                    db(9) = CType((CType(db(9), Integer) + _
                            CType(db(6), Integer)) / 2, Byte)
                    db(2) = CType((CType(db(2), Integer) + _
                            CType(db(5), Integer)) / 2, Byte)
                    db(1) = CType((CType(db(1), Integer) + _
                            CType(db(4), Integer)) / 2, Byte) 
                    db(0) = CType((CType(db(0), Integer) + _
                            CType(db(3), Integer)) / 2, Byte)

                    '' Split color changes between pixels that
                    '' we need to write. So we can create a gradient effect
                    ich1Rgrate = (CType(db(2), Integer) - _
                                  CType(db(11), Integer)) / (sfp + 1)
                    ich1Ggrate = (CType(db(1), Integer) - _
                                  CType(db(10), Integer)) / (sfp + 1)
                    ich1Bgrate = (CType(db(0), Integer) - _
                                  CType(db(9), Integer)) / (sfp + 1)

                    '' Apply Smoothing
                    For i As Integer = 0 To sfp - 1
                        '' CH1 Smoothing
                        db(0) = db(9) + ((sfp - i) * ich1Bgrate)
                        db(1) = db(10) + ((sfp - i) * ich1Ggrate)
                        db(2) = db(11) + ((sfp - i) * ich1Rgrate)
                        Copy(db, 0, d_ch2.Scan0 + rFPDest + (-i * 3), 3)
                    Next
                End If
                '' Go to Next Pixel
                rFPDest += 3
            Next
        Next

        '' Closing direct access 
        DepthMap.UnlockBits(d_depth)
        Image.UnlockBits(d_image)
        b_CH2.UnlockBits(d_ch2)
        b_CH1.UnlockBits(d_ch1)
    End SyncLock : End SyncLock
    Return True
Catch ex As Exception
        Return False
End Try

Points of interest

1. One of the interesting things in this code is using the System.Runtime.InteropServices.Marshal class for doing unsafe operations which are normally not supported in VB.NET; this will give us the ability to access data directly in memory that is much faster than using functions like GetPixel or SetPixel.
2. I originally wrote this class for an application that can get the depth and RGB stream from Kinect (Microsoft’s motion capture device, used for an Xbox gaming console) and then can create a 3D image based on that information. You can download the source code from this address: http://www.kinectdevs.com/forums/filebase.php.