This article describes a texture transfer program I developed based on Efros & Freeman's SIGGRAPH 2001 paper:
- Image Quilting for texture synthesis and transfer: pdf, ppt
Texture transfer is the process by which the texture of a source image is transferred to a target image. The result is that we get an image that looks like the target image but it is made out of the texture of the source image e.g. below are shown two images -- source, target and the result of transferring the texture of the source onto the target:
|source||target||first iteration||second iteration||third iteration|
The source is an image of rice and the target is the man's face. After texture transfer we get a face that looks as though it is made out of rice. How do we do this? The idea is to take patches of rice and weave them together seamlessly into the image of the face. Bright patches of rice will goto areas where the face is bright and dark patches will goto areas where the face is dark. To illustrate the concept, in the image below the boundaries of the individual patches have been colored in red. You can find each of these patches in the rice image if you look for them hard enough.
The Texture Transfer Algorithm
The texture transfer algorithm has to synthesize an image from the pixels of the source. Locally the result should have the appearance of the source texture but when looked at globally the result should look like the target. The texture transfer proceeds by synthesizing an image in faster scan order in units of block. You can control the size of the block in the setting window and it is set to 27x27 pixels by default. Whenever a block in the result has to be synthesized the algorithm needs to find a suitable block in the source texture that satisfies two criteria:
- The block should match the target, meaning bright patches of rice should go to bright patches on the man's face.
- After making some cuts, the block should fit in seamlessly with its adjoining neighbors so that we don't see any visual discontinuities and don't even realize that the result is actually made up of many blocks put size by side like a jigsaw puzzle.
To this end for every patch in the source image the algorithm computes two difference measures d1, d2 that quantify how well criterion 1 and 2 above are satisfied and their weighted sum
The algorithm chooses the patch that minimizes d. The function
GetMatchingBlock in TextureTransferTool.cs does this computation. The next step after choosing a matching block is to cut it appropriately so that it fits in seamlessly with the adjoining neighbors. It is because of these cuts that the patches in figure have jagged boundaries. If no cuts were made the boundaries would be straight lines and we would get something that looked like following image:
The algorithm synthesizes the result in several passes or iterations starting with a low value of a (a=0.1) and initial block size set to 27x27 pixels by default and decreasing block size by 1/3 rd in every iteration and increasing alpha until a=0.9 in the final iteration. The number of iterations to do is set to 3 by default. The following code snippet from function
OnDoWork in TextureTransferTool.cs shows the main texture transfer code. The texture transfer runs in a separate
BackgroundWorker thread so that it does not block the UI thread from processing keyboard and mouse events.
for (int i = 0; i < numIterations; i++)
bool isFirstIteration = i == 0;
double alpha = 0.8 * i / (numIterations - 1) + 0.1;
nextX = 0; nextY = 0; nextBlockWidth = blockWidth; nextBlockHeight =
blockHeight; nextOverlapX = 0; nextOverlapY = 0;
bool completed = false;
while (!completed && !CancellationPending)
Point q = new Point(nextX, nextY);
p = GetMatchingBlock(img, src, target, q, nextOverlapX, nextOverlapY,
nextBlockWidth, nextBlockHeight, alpha, isFirstIteration, probe,
out eh, out ev);
int Hcut = findHcut(eh);
int Vcut = findVcut(ev);
cutandpaste(src, p, img, q, Hcut, Vcut, nextBlockWidth,
double fractionComplete = CalculateFractionComplete(i, numIterations,
nextX, nextY, nextBlockWidth,
nextBlockHeight, imgWidth, imgHeight);
int percentComplete = (int)Math.Round(fractionComplete * 100);
ReportProgress(percentComplete, new ProgressReport(fractionComplete,
Next(X, Y, blockWidth, blockHeight, imgWidth, imgHeight, overlapX,
overlapY, out nextX, out nextY, out nextBlockWidth,
out nextBlockHeight, out nextOverlapX, out nextOverlapY,
X = nextX; Y = nextY;
e.Cancel = true;
X = 0; Y = 0;
blockWidth = (int)Math.Round(blockWidth * blockReductionFactor);
blockHeight = (int)Math.Round(blockHeight * blockReductionFactor);
probeWidth = srcWidth - blockWidth;
probeHeight = srcHeight - blockHeight;
overlapX = (int)Math.Round(overlapX * blockReductionFactor);
overlapY = (int)Math.Round(overlapY * blockReductionFactor);
if (overlapX >= blockWidth || overlapY >= blockHeight || blockWidth ==
0 || blockHeight == 0)
Steps to Perform Texture Transfer
In the following table I show some results I have obtained using the texture transfer program. These results are generated with amount to probe set to 10% and distance metric set to SSD. Note that the images have been resized to fit on the screen. Therefore if you download these images and run the program it is likely that you may not get the exact same results.
|Source||Target||Difference=SSD; Amount to probe=10%|
This was a fun project for me as it introduced me to Computer Graphics and some neat features in C# 2.0 such as the
BackgroundWorker class, dynamic layout and panels. I hope some of you find it useful in your work. Suggestions for improvement and bug reports are always welcome.
Updates & Revision history
3/7/08: I created this project using Visual C# Express Edition 2008 and have noticed if you attempt to open the solution using VS 2005 it gives an error message. If you encounter this just create a new project and add the source code contained in the zip file, compile and run. The only references you should need are System, System.Windows.Forms and System.Drawing.
3/11/08: Related links: Rob Burke's C# texture transfer code
7/26/10: I am attaching a zip file that has source code + Visual Studio Project for my texture transfer article. What I have done here is ported the application to use .NET 4.0 TPL (Task Parallel Library). My hope was that the application would run faster as it consumes all the CPUs of the machine. However, the new code's performance is only marginally better and in many cases even worse. Taking advantage of parallelism is no easy task, and full of gotcha's e.g. who knows maybe my new code is suffering from false sharing. But I would still like to upload this source code as a TPL tutorial, and maybe someone can make it faster.