Dynamic Cropping for VirtualDub

Dynamic cropping for VirtualDub
===============================

This archive contains a patch for the VirtualDub 1.9.9 source code tree that 
enables the software with a new dynamic cropping feature. 



LICENSE AND WARRANTIES
======================

The patch is distributed under the GNU general public license (just like VirtualDub). 
In case, if it is going to be merged into the official source tree of VirtualDub, 
some personal credits are expected to appear in the About box. Otherwise, use it 
at your own risk with no warranty of any kind.



STORY
=====

One day I came across a video (http://www.youtube.com/watch?v=A23SPewFZhU) that
was a dub of an old 8mm film we used to make as kids. The dub was made 
unprofessionally with no special equipment or even a tripod, so it suffered from a
number of problems. The worst was sporadic non-compensated motion of the dubbed 
screen area. I tried to find software that would allow me to crop the moving areas
dynamically but had no luck. There was no free software with such a feature, and all
commercial software was way too expensive. Then I came to an idea of writing such
software by myself. As a base, I chose VirtualDub (http://www.virtualdub.org/) 
because it already provided some static cropping functionality. All I needed to do
was to make it dynamic, e.g. with placement of the cropped area defined as a 
function of time. I decided to make it as flexible as possible, so one could edit,
save & restore the dynamic cropping settings, and to design it as an extendable 
feature.



DESIGN
======

The "work horse" of the solution is a new IVDDynClippingStorage interface defined 
in ClippingControl.h :

class VDINTERFACE IVDDynClippingStorage : public vdrefcounted<IVDRefCount> {
public:
	// Initialize clipping map
	virtual void Init(int sourceW, int sourceH) = 0;
	// Get source width
	virtual int GetWidth() = 0;
	// Get source height
	virtual int GetHeight() = 0;
	// Load clipping map from a file
	virtual VDStringW Load(wchar_t *filename, HWND parent) = 0;
	// Save clipping map to a file
	virtual VDStringW Dump(wchar_t *filename, HWND parent) = 0;
	// Display configuration interface
	virtual void Configure(HWND parent) = 0;
	// Set clipping bounds for position
	virtual int SetClipBounds(sint64 pos, const vdrect32& r) = 0;
	// Get clipping bounds for position
	virtual int GetClipBounds(sint64 pos, vdrect32& r) = 0;
	// Get position for clipping map index
	virtual sint64 GetClipPos(int idx) = 0;
	// Get clipping map size
	virtual int GetTotalClipPos() = 0;
	// Get the resulted crop size
	virtual void GetCrop(vdrect32& r) = 0;
	// Delete clipping map entry by index
	virtual void DelClipBounds(int idx) = 0;
	// Crop pixmap for given position
	virtual bool DoCrop(VDPixmap& dst, sint64 pos, vdrect32& crop_area) = 0;
	// Add/drop range to/from clipping map
	virtual void ChangeTimeline(sint64 pos, sint64 nframes, bool add) = 0;
};


It defines a set of methods to manage the clipping map and to perform the actual
dynamic cropping. The VDDynClippingStorage class defined in ClippingControl.cpp 
is implementing the interface. An instance of the class can be created per 
FilterInstance (however, a need in having more than one filter with dynamic 
cropping in the chain of filters is highly doubtful).

The following diagram is showing relations between the components 
of the solution:


 +--------------+                                                +--------------+
 |              |  VDGetDynClippingStorage()                     |              |
 |  Script.cpp  +-----------+                                    |    Job.cpp   |
 |              |           |                                    |              |
 |              |        +--|---------------------------+        |              |
 |              |        |  |                           |        |              |
 |              |        |  | ClippingControl.cpp       |        |              |
 |              |        |  V                           |        |              |
 |              | Load() | +--------------------------+ | Dump() |              |
 |              +--------->|  IVDDynClippingStorage   |<---------|              |
 |              |        | | +----------------------+ | |        |              |
 +------------+-+        | | | VDDynClippingStorage | | |        +--+-----------+
              |          | | |                      | | |           |
 SetClippingStorage()    | | |                      | | |    GetClippingStorage()
              |          | | |                      | | |           |
              |    +------>| +----------------------+ | |           |
              |    |     | +---^-------^----------^---+ |           |
              |    |     |     |       |          |     |           |
              | DoCrop() | Configure() |          |     |           |
              |    |     |     | GetClipBounds()  |     |           |
              |    |     |     |       | SetClipBounds()|           |
              |    |     |     |       |          |     |           |
              |    |     | +---|-------|----------|---+ |           |
              |    |     | |   |IVDClippingControl|   | |           |
              |    |     | | +-+-------+----------+-+ | |           |
              |    |     | | |  VDClippingControl   | | |           |
              |    |     | | |                      | | |           |
              |    |     | | +----------------------+ | |           |
              |    |     | +----^----------------^----+ |           |
              |    |     +------|----------------|------+           |
              |    |            |                |                  |
              |    | GetClippingStorage()  SetClippingStorage()     |
              |    |            |                |                  |
              |    |     +------+----------------+------+           |
              |    |     |                              |           |
              |    |     |          Filtdlg.cpp         |           |
              |    |     |                              |           |
              |    |     +------+----------------+------+           |
              |    |            |                |                  |
              |    | SetClippingStorage()  GetClippingStorage()     |
              |    |            |                |                  |
           +--V----+------------V----------------V------------------V--+
           |                                                           |
           |                    FilterInstance.cpp                     |
           |                                                           |
           +-----------------------------------------------------------+                                        


There are some changes made to the basic functionality of the cropping control
as well. They are not strictly related to dynamic cropping, but are very handy:

1) From now on you will be able not only stretch the cropped area by dragging 
its sides and corners with left mouse button, but also move the complete
cropped area using the right mouse button! Try it and you will see how useful
it is for dynamic cropping when you have to go frame-by-frame adjusting the 
cropped area placement.

2) A new "lock aspect ratio" checkbox allows the cropped area to have the aspect
ratio of the original video. It is suggested to always check it with dynamic
cropping enabled.


The following VirtualDub components are affected with the patch:

- Cropping control (ClippingControl.h/.cpp, FiltDlg.cpp, resources),
- Filter instance class (FilterInstance.h/.cpp),
- Saving (Job.cpp) and loading (Script.cpp) of the processing settings.

The following project files are modified:

src\virtualdub\h\clippingcontrol.h
src\virtualdub\h\filterinstance.h
src\virtualdub\res\resource.h
src\virtualdub\res\virtualdub.rc
src\virtualdub\source\clippingcontrol.cpp
src\virtualdub\source\filtdlg.cpp
src\virtualdub\source\filterinstance.cpp
src\virtualdub\source\job.cpp
src\virtualdub\source\script.cpp



BUILDING
========

You should patch the source tree of VirtualDub and re-build it.

There are two ways of patching the source tree, you can either

1) unzip the archive into the root of VirtualDub 1.9.9 source tree overwriting 
its files, or

2) you can use the provided dyncropping.patch file that contains diff info 
produced with GNU diff (http://gnuwin32.sourceforge.net/). Run it with the Win32
version of patch from the root of VirtualDub source tree as follows -

C:\VirtualDub-1.9.9-src> patch -p1 -i dyncropping.patch

, this may work well with versions of VirtualDub other than 1.9.9 too.


After that run the building process as usual.



RUNNING
=======

Load a video, add any video filter, select "Cropping..." and check the "Dynamic
cropping" checkbox in the "Filter input cropping" control. Check also "Lock
aspect ratio" for better results. Walk the video frame-by-frame inside of the
cropping control and adjust the sizes and position of the cropped area 
accordingly to your needs. All position and size changes will be saved in the
map for rendering. If you rewind the video back to the beginning and go again
frame-by-frame you will notice how the cropped area is moving to follow your
adjustments! You can click "Options..." and select the desired settings. Once
done, click "OK" to close "Filter input cropping" and once again "OK" to close
"Filters". Run "Preview filtered..." and watch what is happening on the left
and right UI panels. Use "Save as AVI..." to save the final rendering.

You can also use the provided testing sample Roof.zip to evaluate the 
functionality of the patch. Just open the Roof.avi file, load its processing
settings from Roof.vcf. It will load the dynamic cropping map from Roof.dcrp.
Run "Preview filtered..." and watch what is happening on the left and right UI
panels. Open "Cropping..." of the Warp Sharp filter and play with the new
dynamic cropping control. Walk the clipping map vertex-by-vertex with "<" and
">" buttons. If you go frame-by-frame with the position control you will
notice how the cropped area is moving around. You can adjust it at any position
modifying its size and placement. All changes will be saved in the cropping map
and used for rendering.



CONCLUSION
==========

In general, this feature can be useful for manual "motion post-compensation" in
the following cases:

- Old videos made with camcorders with no hardware motion compensation
  available,

- Videos made in complex motion situations (in car, boat, on the run),

- Videos made by kids or impaired people,

- "Noisy", dark, low-contrast videos where automatic algorithms usually fail or 
  produce bad results.

Also, it can be handy for some special video effects.

It is very basic, but it is doing what it is supposed to do. Here is a couple
of more things I wanted to share.

- Currently only linear interpolation between vertexes of the clipping map is 
  implemented. One can try to implement a cubic spline-based interpolation to
  see if it improves the rendering results and reduces shaking. Also, one can
  try to implement a real motion-compensation algorithm inside of 
  IVDDynClippingStorage::DoCrop that will cache and analyze frames.

- There is also a "no interpolation" mode available that can be used for videos
  that contain sets of scenes in which static cropping is required - this avoids
  cutting / cropping / gluing them piece-by-piece. 

- With linear interpolation you usually achieve the best results with two 
  cropping adjustment passes. 

- The resampling selection has three settings: original frame size, maximal
  cropping size and custom. In any case, the result will be resampled to the
  rounded sizes divisible by 16 for general compatibility with compression
  algorithms. An attempt to force custom sizes to values non-divisible by 16
  or bigger than those of the source will result in an error.

- There are two resampling algorithms built into VirtualDub - "bilinear" and
  "nearest". The "bicubic" is not implemented in the core of VirtualDub but
  instead as a plugin and therefore is not supported.

- It appears to be useful to have a line grid optionally appearing on the
  cropped area when you slide it with the right mouse button (maybe it can
  appear when you press and hold the right mouse button and then the left one).
  It would allow to position the cropped area more precisely. I did not
  implement it, but left it as an idea for others.

- The IVDDynClippingStorage::ChangeTimeline method is defined and implemented
  in VDDynClippingStorage but it is not been used for anything - it can be a 
  future extension, just in case.

- All clipping map vertex indexes are 1-based. That is, what index you see on
  the interface is what you have in the map.


ABOUT THE AUTHOR
================

I am a Software Engineer with 20+ years of experience in different fields of 
programming. Presently I live in the Toronto area.

Ways to contact me - 

by email:              SergeyStrakhov (at) yahoo (dot) ca
via LiveJournal:       http://Strakhov.livejournal.com/
per personal website:  http://Strakhov.longmusic.com/   (sometimes outdated)