Introduction - C# Video TimeLine DirectShow
Updated Article. I added a new version of the DirectShow.Net library and brought these projects into VS2015. In addition, I will be adding another sample of this timeline to this project shortly using an HTML5 canvas for the timeline with editing code in both C# and C++.
Checkout my related video I posted on YouTube that has over 1 million views, Click Here.
Note: This project is about creating a Video TimeLine in C# and I only added directShow to illustrate how to use my Video TimeLine Control. I added 2 demo projects illustrating how to use this TimeLine Control. These sample projects work with .wmv video files but you can easily extend the demos to work with any video file type. I am planing to post another article for a new type of Video TimeLine Control using html5 in the near future.
This article is now updated with a full control panel and is about an epiphany I had awhile back when I read an article by Alex Hildyard called:
Build a Reusable Graphical Charting Engine with C#.
I realized that the TimeLines in a video editing programs like Adobe AfterEffects, Premiere, AVID, and Media 100 are just ordinary horizontal bar graphs like the one described in Alex Hildyard's article with a movable Thumb added. WOW! In this article I created a basic Video TmeLine Control using the code from Alex Hildyard's article as a starting point and I added a Thumb Slider to finish it off. Essentially it is an ordinary horizontal bar graph with zoom that displays video frames as horizontal bars on a graph.
This article and the enclosed sample project are designed to illustrate how to create a horizontal bar graph with zoom as a TimeLine Control. I used the DirectShowLib-2005.dll from the DirectShow.NET library to play the video in this sample but this project is NOT intended as a DirectShow project. The sample I included will extract any frames, play video backwards and forwards, advances frames backward and forward, and allows you to add any type of filter you want all in C#. But this control ad its code are completelty independent of any code to play video and you can use this control with VideoLAN or any other means of playing video. TimeControl, A Horizontal Bar Graph with Thumb Slider & Zoom
There are 3 parts to the TimeLine, namely, Graph, Left-Side Control Panel, and SnapShots Controls or Frames Display. There are 2 basic designs in TimeLines, namely the TimeLine in Adobe AfterEffects, and Media 100. In Media 100, the TimeLime has multiple files on the same horizontal bar in the graph where all video files go, and in AfterEffects, each horizontal bar is a separate video or audio file. The worst TimeLine design, in my opinion, is the one in FinalCutPro because it is so busy and cluttered. The Best TimeLine for straight editing is Media 100 which, in my opinion is superior to AVID's design. My TimeLine is based on AfterEffects but it would be easy to re-arrange it to work like the TimeLine in Media 100.
I added a Thumb Slider using part of the code from an article by Michal Brylka here on CodeProject, namely:
Owner-drawn trackbar(slider) By Michal Brylka
I also used the following control for sliders on the TimeLine Control, namely:
Advanced TrackBar Control with MAC Style By NicolNghia
In this updated version I replaced the control buttons on the left side with a DataGridView which works much better and allows me to display a lot more information.
TimeLine GUI Design - AfterEffects vs. Media 100, AVID, etc.
There are 2 basic designs in TimeLines that I have seen, the design used my Media 100, AVID, Final CutPro, and Premiere, versus the TimeLine design in Adobe AfterEffects. The layout in AfterEffects is for composting multiple layers of video frame by frame and that is design I implemented in this control because, well, I am in love with Adobe AfterEffects! And it is fairly easy to modify this control to change the layout to the Media 100 layout. I will post a separate article with a very different type of TimeLine based on Media and AVID non-linear editing systems where all the video files are placed on only 2 horizontal bars and a 3rd bar between them is used for transitions. The TimeLine shown in the first image below is Adobe AfterEffects TimeLine Layout where each bar below is a separate video file and when you play each frame of every video is composited together frame by frame. This layout is great for build special effects but not practical for just editing video cuts and other basic transititions.
In the Media 100 layout below there is only one bar of video files shown in gray and below that are 3 bars or tracks,namely, the "A track", the "Fx track", and "B track." Multiple video files are placed on the SAME bar as opposed to AfterEffects where each video file resides on a separate bar in the graph.
Bar Graph, SMTP Time Code, and Dropped Frames
The painted bars represent video or audio files and are designed to allow a user to go to any frame of a video where they want to place an "edit" or transsition. The first problem is understanding how to represent "time" in relationship to frames on a TimeLine. I did this first version of this control to display frames in non-drop frame mode frame accurate control.
Non-Drop Frame Mode vs. Drop-Frame Mode
Drop-Frame Mode was invented when color TV was invented to retain compatibility with existing monochrome TVs. Broadcasting color took 0.03 seconds every second to transmit the color information in the signal, or 3.58sec per hour. Technically, the 3.58 MHz (actually 315/88 MHz = 3.57954545 MHz) color subcarrier would absorb common-phase noise from the harmonics of the line scan frequency. Rather than adjusting the audio or chroma subcarriers, they adjusted everything else, including the frame rate, which was set to 30×1.000/1.001 Hz.
So an "hour of timecode" at a frame rate of 30 frame/seconds was longer than an hour of wall-clock time by 3.59 seconds, leading to an error of almost a minute and a half over a day. To correct this, drop frame SMPTE timecode was invented. HOWEVER, NO video frames are dropped or skipped using Drop-Frame Mode. What's actually being dropped are some of the timecode "labels". In order to make an hour of timecode match an hour on the clock, drop-frame timecode drops frame numbers 0 and 1 of the first second of every minute, and includes them when the number of minutes is divisible by ten. This achieves an "easy-to-track" drop frame rate of 18 frames each ten minutes (18,000 frames @ 30frame/s) and almost perfectly compensates for the difference in rate, leaving a residual timing error of roughly 86.4 milliseconds per day, an error of only 1.0 ppm.
Drop frame TC drops two frames every minute, except every tenth minute, achieving 29.97frame/second. Drop frame is usually represented with a semi-colon (;), i.e., HH;MM;SS;FF whereas Non-Drop Frame Mode is represented with a colon (
, i.e., HH:MM:SS:FF. In the next update I will add support for drop-frame mode.
To start I decided to make the Graph display actual time as the X-axis of the Graph and convert from "actual time" on the wall clock to frames where I might need the frame number itself to display. I pla to update how the SMTP timecod eis painted on the TineLine soon now that I have had a chance to play around with the control.
The most interesting and difficult code was in actually drawing the vertical marks on the TimeLine in the Graph and writing under the division marks the SMTP code vales because it was necessary to calculate how much space was available to write in and to determine the spacing between time codes based on the zoom factor.
As discussed in depth in the article by Alex Hildyard, a graph performs a transformation between two datasets, namely, a set of 'data' points to 'screen' points where you want to render that data. I strongly recommend reading his article as it is excellent and very enlightening. One thing to note in a TimeLine is that we are ONLY "transforming" data on the X-axis and on the Y-axis we are "translating" data points which is trivial to do. The key to the graph is the transformation routine which basically does all of the work, namely:
public PointF Transform(RectangleF from, RectangleF to, PointF pt_src)
{
PointF pt_dst = new PointF();
pt_dst.X = (((pt_src.X - from.Left) / from.Width) * to.Width) + to.Left;
pt_dst.Y = (((pt_src.Y - from.Top) / from.Height) * to.Height) + to.Top;
return pt_dst;
}
When the user mousedowns on the selected bar(orange) the user can move the selected bar to the left or right if the bar isn't the Thumb and the Thumb will not move. If the user MouseClicks on any bar then the Thumb will move to the Clicked Position. Remember that the Thumb is just another bar in the graph. Using the function "IsPointInRect(e.Location, thumbRect)" we identify the Thumb and we can do this for any of the bars by replacing "thumbRect" by the corresponding rectangle variable for a given bar to slide a bar left or right if it isn't the Thumb bar. The code for moving the Thumb Slider (or a bar in the the next update) is as follows:
protected override void OnMouseMove(MouseEventArgs e)
{
base.OnMouseMove(e);
mouseInThumbRegion = IsPointInRect(e.Location, thumbRect);
mouseInSelectedRegion = IsPointInRect(e.Location, selectedRect);
if((mouseInSelectedRegion) && (Capture & e.Button == MouseButtons.Left))
{
selectedRegionDrag = true;
ScrollEventType set = ScrollEventType.ThumbPosition;
Point pt = e.Location;
int p = pt.X;
int margin = thumbSize >> 2;
float coef = (float)(m_data_rect.Width) / (float)((ClientSize.Width) - (float)2 * m_margin_left);
float trackerValueF = coef * ((float)pt.X - (float)m_margin_left) + (float)m_data_rect.X;
int iTrackerValue = Convert.ToInt32(trackerValueF);
if (iTrackerValue <= barMinimum) iTrackerValue = barMinimum;
else if (iTrackerValue >= barMaximum) iTrackerValue = barMaximum;
if ((barIndex > -1))
bars[barIndex].startSeconds = trackerValueF;
}
else if ((selectedRegionDrag) && (Capture & e.Button == MouseButtons.Left))
{
selectedRegionDrag = true;
ScrollEventType set = ScrollEventType.ThumbPosition;
Point pt = e.Location;
int p = pt.X;
int margin = thumbSize >> 2;
float coef = (float)(m_data_rect.Width) / (float)((ClientSize.Width) - (float)2 * m_margin_left);
float trackerValueF = coef * ((float)pt.X - (float)m_margin_left) + (float)m_data_rect.X;
int iTrackerValue = Convert.ToInt32(trackerValueF);
if (iTrackerValue <= barMinimum) iTrackerValue = barMinimum;
else if (iTrackerValue >= barMaximum) iTrackerValue = barMaximum;
if ((barIndex > -1))
bars[barIndex].startSeconds = trackerValueF;
}
Invalidate();
}
The Video Frames SnapShots Control
The SnapShots Control is just a sliding control with the video frames painted on the control. This control is designed to allow you to work with large video files because it only paints the frames visible to the user in a range of time. The loading should be on a thread but I didn't want to take the extra time for this sample project since it behaved fairly well without threading for a simple demo. Double clicking on an image will move the position to the correspong video frame. Notice that I added the 35mm film registration marks to create a partial border. The 2 main manufacturers of 35mm film cameras in use are Panavision and Arriflex. And the difference between these cameras are that Panavision only rents their cameras and uses dual registration whereas Arriflex sells their cameras and they use single registration. When people would tell me they were producer or directors I would ask them what's the difference between Pnanavision and Arriflex? Only the Germans, i.e., Arriflex, could make single registration work. I added the registration marks because I think it is a cool effect. You can easily link the SnapShot control to the Thumb Slider so that the frames displayed matches the Thumb position--I just forgot to do this in the demo but it's just one line of code!
You can easily customize the code below to dynamically display any subset of video frames even if the video file has millions of frames. First, I load the frames into a List as follows:
public void LoadSnapShots()
{
double interval = 1;
double fps = 0;
double duration = 0;
fps = this.getFramePerSecond();
duration = this.getDuration();
if (duration < 101)
interval = 1;
if (duration > 100)
interval = duration / 100;
List<frames> list = new List<frames>();
if (fileType == FileType.Video)
{
Bitmap snapshot;
for (double i = 0; i < duration; i = i + interval)
{
snapshot = this.SnapShot(i);
list.Add(new Frames(i, snapshot));
}
}
this.segmentationImages = list;
}
Second, we have to dynamically paint each frame of Video with an image for the registration marks added to each frame as follows:
protected override void OnPaint(PaintEventArgs pe)
{
if ((images == null) || (images.Count < 1))
return;
Graphics g = pe.Graphics;
Brush theBrush = Brushes.Black;
StringFormatFlags flags = StringFormatFlags.MeasureTrailingSpaces;
StringFormat sf = new StringFormat(flags);
sf.Alignment = StringAlignment.Center; base.OnPaint(pe);
Point pt = AutoScrollPosition;
g.TranslateTransform(pt.X, pt.Y);
int startX = 0;
int startY = 0;
_cumulativeHeight = _squareHeight + (_margin*2);
_cumulativeWidth = 0;
for (int index = 1; index < (_numSquares+1); index++)
{
if ((!_horizontalMode) && (startX > (_verticalWidth-1)))
{
startX = 0;
startY += _squareHeight + _margin;
_cumulativeHeight += (_squareHeight + _margin);
}
Rectangle theRect = new Rectangle(_squareWidth * (startX++) +
_margin, startY, _squareWidth, _squareHeight);
Rectangle textRect = new
Rectangle(theRect.X + 50, theRect.Y + 50, 20, 20);
Image imageBack = Image.FromFile(exePath + "/frame.gif");
g.DrawImage(imageBack, theRect);
Rectangle textRectInnner =
new Rectangle(theRect.X, theRect.Y+9, 68, 45);
int i = index - 1;
g.DrawImage(images[i].image, textRectInnner);
Pen borderPen = new Pen(theBrush);
g.DrawRectangle(borderPen, theRect);
g.DrawString(index.ToString(), this.Font, theBrush, textRect, sf);
_cumulativeWidth += _squareWidth + _margin;
}
AutoScrollMinSize = new Size(_cumulativeWidth, _cumulativeHeight);
}
Next is implementing the MouseClick Handler when a user clicks on an image in the SnapShots control to jump to a particular frameof video and that is done as follows:
protected override void OnMouseClick(MouseEventArgs e)
{
base.OnMouseClick(e);
Point mouseLocation = GetVirtualMouseLocation(e);
int squareID = getSquareID(mouseLocation);
if (squareID > 0)
{
int index = squareID - 1;
if ((((MainForm)this.Parent).mediaManager.currentMedia != null) && (((MainForm)this.Parent).images != null))
((MainForm)this.Parent).mediaManager.currentMedia.setPosition((((MainForm)this.Parent).images[index]).time);
}
}
Left-Side Control Panel
The Control Panel on the left side are just 2 DataGridViews. The top one is used to only display the Headers for each column and the one under it is used to displayed the bars representing the video files that have been added to the control. It is easy to add an icon like an arrow to a row that when clicked will expand the row to show subrows. I turned off Scroll in both directions on both DataGridViews and the vertical slider moves the DataGridView up and down so that the rows of the DataGridView stay locked with the rows of the Graph as the Graph slides up and down vertically. I still need to add 2 more columns, namely Start Time and End Time in SMTP code for when a user slides a bar to the left or right since these values will change as the bar is moved left or right. I also added code to allow you to show a DropDown of the Columns you want to show or hide. Since I added this ability to show/hide columns we need to first determine if a column is hidden before we allow the vertical slider on the right side of the control to move the DataGridView as follows:
private void sliderInfo_Scroll(object sender, ScrollEventArgs e)
{
if (e.NewValue < PANEL_Buttons.ColumnCount)
{
if (this.PANEL_Buttons.Columns[e.NewValue].Visible)
{
PANEL_Buttons.FirstDisplayedScrollingColumnIndex = e.NewValue;
dgvHeaders.FirstDisplayedScrollingColumnIndex = e.NewValue;
}
}
}
The next tricky thing was determing the row of the DataGridView that was clicked. It turns out the using "rowcount" is worthless to determine if we have files loaded so I checked instead to see if the DataSource was null or not as follows:
private void PANEL_Buttons_RowEnter(object sender, DataGridViewCellEventArgs e)
{
int oldBarIndex = this.BarIndex;
if (PANEL_Buttons.DataSource == null)
return;
if (e.RowIndex < 0)
return;
barIndex = e.RowIndex;
PANEL_Buttons.Rows[barIndex].Selected = true;
this.videoSlider1.SelectBar(barIndex);
if (ValidClick != null)
{
FireEventArgs ee = new FireEventArgs(barIndex, oldBarIndex);
ValidClick(this, ee);
}
}
Getting Frames from VLC (VideoLAN)
Most people may not realize that it is very easy to get frames individually from VLC. To get frames from VLC you need to place some code into VLC that calls the VLC internal functions for extracting frames and recompile VLC for Windows. It is actullay VERY easy to compile VLC for Windows if you use Ubunto on a memory drive--it's works the first time! In fact, I am thinking about writing an article on How To Compile VLC for Windows using Ubunto on A Memory Drive. I have a version of this Editing control with the latest version of VLC where I modified VLC to allow frame by frame advance forward and backward with frame extraction and to play in reverse or forward. This are some major advantages to using VLC over DirectShow such as the fact that I also modified VLC to act as a DirectShow Source Filter for RTSP streams and it works perfectlly. I plan to post this modified version of VLC in a seprate article because I have to explain how to compile VLC for Windows using Ubuntu on a memory drive.
Further Development
This TimeLine Control was based on Adobe AfterEffects and is useful for compositing multiple layers of video. However I have developed another TimeLine based on AVID and Media 100 designs that is useful for straight editing of multiple video files and I will write a separate article and project for that one because it works very differently from this one. In that verson track manipulation to the TimeLIne Control use the Splicer.dll on CodePlex for combining tracks with special effects. The Splicer.dll gives you a full-blown Video Editing Program including special effects.
Summary
This project is designed to give you some idea of how to create a TimeLine Control in C#. If you have any questions, please feel free to contact me at: tvmogul1@yahoo.com. I will be posting new versions of this and other video controls on my website at: www.SerGioApps.com
