Click here to Skip to main content
Click here to Skip to main content

Downloader Component

By , 14 Jan 2007
 
Sample Image - Downloader.jpg

Introduction

Downloading files over the network could be a cumbersome task. Trying to implement other features like auto-resume and support for proxy servers, etc. makes it even harder. Using this component can make things a lot easier for yourself. You can download files over proxies, password protected sites or untrusted SSL connections simply by setting a few properties. The example that is provided shows how to do the job and how to handle events over a simple GUI.

Since the download operation takes place in a worker thread asynchronously, you don't need to worry about downloading large files or blocking the user interface. There are some points you need to have in mind though.

Aborting the Worker Thread

Stopping a download operation that is running is tricky. You could just call the DownloadThread.Abort() method on the downloader thread to stop the work, but this is not the recommended way. The main reason for this is that Thread.Abort throws an exception which can occur at any time during your work and leaves your running state inconsistent. The easier way is to check for a boolean field, and terminate the operation when the field is set, or run the job on an entirely different process. To read more about this idea, check Ian Griffiths blog post here.

Updating GUI

When running a download job, the Downloader component provides its internal progress and error state through events and custom eventargs. Since the eventargs are constructed on the worker thread, trying to update the user-interface from the worker thread throws a Cross-Thread exception. The workaround to update the user interface is to use the Invoke method (inherited from Control), a Delegate and call to the necessary method(s), like this:

private delegate void ParamMethodInvoker
    (long fileSize, long progressValue, string message);

/// <summary>
/// Call UpdateProgress method using a delegate and provide method parameters
/// </summary>
private void OnDownloadProgressChanged
    (object sender, DownloadProgressEventArgs e)
{
    Invoke(new ParamMethodInvoker(UpdateProgress), new object[] 
        { e.BytesRead, e.TotalBytes, e.Message });
}

/// <summary>
/// Normally update the user interface
/// </summary>
private void UpdateProgress(long bytesRead, long totalBytes, string message)
{
    int kbRead = Convert.ToInt32(bytesRead) / 1024;
    int kbTotal = Convert.ToInt32(totalBytes) / 1024;
    int percent = Convert.ToInt32((bytesRead * 100) / totalBytes);

    progress.Value = percent;
    lblDownloadMessage.Text = 
        string.Format("{0:#,###} of {1:#,###} KBytes ({2}%)", 
        kbRead, kbTotal, percent);
}

Resuming a Download

To resume a download job, we can simply continue where we left off. When restarting a download job, the downloader component first checks for the existing file at the specified location (through DownloadPath property). If the file exists, it reads the Length of the existing file and sets the HttpWebRequest's range using AddRange method. Overloads of the AddRange method could be used to handle multi-part downloading of a single file.

long startingPoint = 0;

if(File.Exists(DownloadPath))
{
    startingPoint = new FileInfo(DownloadPath).Length;
}

HttpWebRequest _Request = (HttpWebRequest)HttpWebRequest.Create(url);
_Request.AddRange(startingPoint);

Points of Interest

Implementing other features such as multi-part downloading (like download accelerator applications) and FTP downloads seems like a good idea. Any suggestions/comments/feedback are highly appreciated.

History

Version 1

  • Provided the component with base features and events
  • Provided a basic form to show features of the component

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

About the Author

Hadi Eskandari
Software Developer (Senior) SAM Enterprise
Iran (Islamic Republic Of) Iran (Islamic Republic Of)
Member
Working on both Java and .NET Technology, I have developed various enterprise level applications on both platforms. Currently, I am working as a Senior Developer at SAM Enterprise company which resides in Tehran, Iran, and is a partner with S4M GmbH which develops solutions for media and is a Microsoft Gold Partner. I'm also a Microsoft Certified Professional Developer for Web, Windows and Enterprise applications.

Sign Up to vote   Poor Excellent
Add a reason or comment to your vote: x
Votes of 3 or less require a comment

Comments and Discussions

 
You must Sign In to use this message board.
Search this forum  
    Spacing  Noise  Layout  Per page   
GeneralMy vote of 5mvpMichael Haephrati7 Mar '13 - 11:03 
Great article!
QuestionMy vote of 5memberPouriya.GH30 Apr '12 - 3:27 
Another great job by H.Eskandari, thank you my dear compatriot Blush | :O Thumbs Up | :thumbsup:
GeneralMy vote of 5memberShahin Khorshidnia3 Apr '11 - 19:18 
Thank you
GeneralError Downloading file greater than 2GB [modified]memberakshayoh12 Feb '11 - 8:43 
when i tried to download a file greater than 2GB it gives an error.... it says that the value to int is either greater or smaller than int32..... please help regarding this....
modified on Saturday, February 12, 2011 3:52 PM

GeneralRe: Error Downloading file greater than 2GB [modified]memberKOS_void28 Feb '12 - 4:39 
Maybe because int32 is too small?
 
Have a look at Int64...
If electricity comes from electrons, does morality come from morons?

Generaldoc requiredmemberprudhvi1234516 Mar '10 - 9:03 
hi guys
need a final document for this project
 
plz forward this to prudhviraju1987@gmail.com if u hav
GeneralImplementing Speed Limiting (aka Throttling/Shaping)memberTrendyTim14 Sep '07 - 16:30 
I wanted to use your code to download in the background while still allowing the internet to be responsive and i finally figured out how to limit the download speeds, it may turn out its not a one size fits all solution but it in limited testing (2 machines on opposite sides of the world, 1 remote file and results mirror each other) it looks good.
 
The gist of the solution is to have the thread sleep after writing the chunk to disk, delaying its next read.
 
I also made a change so that instead of calling _SaveFileStream.Length every time for the progress changed, it keeps a running tally as the .Length causes it to check the file size from the file system which i would presume is a little less efficient than incrementing and checking a variable.
 
With a chunk/packet length/buffer size (whatever you want to call it) of 2048 the following is true on my 512k DSL with a normal speed of around 35kb/s before the speed limit is enabled.
 
100ms = 20kb/s
200ms = 10kb/s
 

On a connection where the normal download speed is in excess of 200kbs for the same file.
25ms = 80kb/s
100ms = 20kb/s
200ms = 10kb/s
 
So clearly sleep time is independent of connection speed which makes it a feasible method.
 
When halving the chunk size from 2048 to 1024, the KB/s rate halves for the same sleep time, so for limiting to a speed > 320k it would probably require a larger chunk size (though it would start to loose accuracy after 80k since you cant sleep for 12.5ms, unless you alternate between 12ms and 13ms)
 

I have used NetLimiter to monitor the download speeds of the application and it appears to be mostly accurate.
 
The only problem is i haven't yet figured out how to create a formula from that for converting a speed limit into a sleep time/chunk size and for now limiting to 20kb/s is good enough, perhaps if what i have written makes sense to someone (i daresay it could be explained better) they might be able to help in that regard.
 

And thanks for the initial code Hadi.
Generalpoint of interestmemberjugomkd7510 Aug '07 - 8:25 
Points of Interest
 
Implementing other features such as multi-part downloading (like download accelerator applications) and ftp downloads seems like a good idea.

 
So my question is how this can be used out for development of a download accelerator?
Actually how the accelerators work at all? I am pretty sure that they duplicates the download process/thread asynchronous but i was wondering if is there something else that i am missing?
Some start points will be highly appreciated.
 
Thanks Smile | :)
 

 
What?

GeneralRe: point of interestmemberjugomkd7510 Aug '07 - 21:20 
Overloads of the AddRange method could be used to handle multi-part downloading of a single file.
 
Could you please elaborate this?
 
Thanks Smile | :)
 
What?

QuestionHow about FTP ??member_Thurein_9 May '07 - 17:09 
I found an error when I'm downloading from FTP website .....
Please show me some clues ....
 
Thanks !!!
AnswerRe: How about FTP ??memberh.eskandari9 May '07 - 19:52 
Hi
 
What's the nature of the problem you have with FTP downloading??
 

GeneralRe: How about FTP ??member_Thurein_9 May '07 - 21:36 
Problem downloading and copying file from:
 
ftp://updatesrv/updates/update01.dll

to

d:\RecentUpdates\update01.dll
 
System.InvalidCastException:
 
Unable to cast object of type System.Net.HttpWebRequest to type System.Net.FtpWebRequest .....

GeneralRe: How about FTP ??memberh.eskandari9 May '07 - 22:41 
I see now...Okay...gimme a little time to fix this.
GeneralRe: How about FTP ??memberGaara [BTK]12 Oct '07 - 0:28 
Hi...I dont know if it is exactly what u meant...but i just created a FtpWebRequest,FtpWebResponse and a public bool FTPEnabled.
 
and then i changed the private void DownloadFile(string url, string path, long startPoint) method in the downloader.cs like that:

private void DownloadFile(string url, string path, long startPoint)
{
int startingPoint = Convert.ToInt32(startPoint);
 
try
{
//For using untrusted SSL Certificates
ServicePointManager.ServerCertificateValidationCallback += new RemoteCertificateValidationCallback(OnCheckRemoteCallback);
 
if (FTPEnabled)
{
_FTPRequest = (FtpWebRequest)WebRequest.Create(url);
_FTPRequest.Method = WebRequestMethods.Ftp.DownloadFile;
//_FTPRequest.EnableSsl = true;
}
else
{
_Request = (HttpWebRequest)HttpWebRequest.Create(url);
_Request.AddRange(startingPoint);
}
 
if (!string.IsNullOrEmpty(LoginUsername))
{
if (FTPEnabled)
{
_FTPRequest.Credentials = new NetworkCredential(LoginUsername, LoginPassword);
}
else
_Request.Credentials = new NetworkCredential(LoginUsername, LoginPassword);
}
else
{
if (FTPEnabled)
{
//_FTPRequest.Credentials = CredentialCache.DefaultCredentials;
}
else
_Request.Credentials = CredentialCache.DefaultCredentials;
}
if (!FTPEnabled)
{
if (ProxyEnabled)
{
_Request.Proxy = new WebProxy(ProxyURL);
}
}
 
if (FTPEnabled)
{
// Get the ServicePoint object used for this request, and limit it to one connection.
// In a real-world application you might use the default number of connections (2),
// or select a value that works best for your application.
 
ServicePoint sp = _FTPRequest.ServicePoint;
sp.ConnectionLimit = 1;
 
_FTPResponse = (FtpWebResponse)_FTPRequest.GetResponse();
}
else
{
_Response = (HttpWebResponse)_Request.GetResponse();
 
}
 
Stream responseSteam;
if(FTPEnabled)
responseSteam = _FTPResponse.GetResponseStream();
else
responseSteam = _Response.GetResponseStream();

if (startingPoint == 0)
{
_SaveFileStream = new FileStream(path, FileMode.Create, FileAccess.Write, FileShare.ReadWrite);
}
else
{
_SaveFileStream = new FileStream(path, FileMode.Append, FileAccess.Write, FileShare.ReadWrite);
}
 
int bytesSize;
long fileSize;
 
if(FTPEnabled)
fileSize = _FTPResponse.ContentLength;
else
fileSize = _Response.ContentLength;
 
byte[] downloadBuffer = new byte[DEF_PACKET_LENGTH];
 
OnProgressChanged(new DownloadProgressEventArgs(fileSize, 0, "Starting the download..."));
 
while ((bytesSize = responseSteam.Read(downloadBuffer, 0, downloadBuffer.Length)) > 0 && !_ExitThread)
{
if (_ExitThread)
{
break;
}
 
_SaveFileStream.Write(downloadBuffer, 0, bytesSize);
OnProgressChanged(new DownloadProgressEventArgs(_SaveFileStream.Length, fileSize + startingPoint, "Download in progress..."));
}
 
if (_ExitThread)
{
_SaveFileStream.Close();
_SaveFileStream.Dispose();
OnDownloadCompleted(new DownloadCompleteEventArgs(false, true));
 
return;
}
 
OnDownloadCompleted(new DownloadCompleteEventArgs(true, false));
}
catch (Exception ex)
{
OnDownloadError(new DownloadErrorEventArgs(ex, string.Format("Problem downloading and copying file from {0} to {1}.", url, path)));
OnDownloadCompleted(new DownloadCompleteEventArgs(false, false));
}
finally
{
if (_SaveFileStream != null)
{
_SaveFileStream.Close();
_SaveFileStream.Dispose();
}
 
if (_Response != null)
{
_Response.Close();
}
if (_FTPResponse != null)
{
_FTPResponse.Close();
}
}
}

I tried it and it worked very well.Dont forget to switch the mean settings in the beginning -> DownloadForm.cs

if (cbUserFTP.Checked)
{
downloader.LoginUsername = txtUsername.Text;
downloader.LoginPassword = txtPassword.Text;
downloader.FTPEnabled = true;
}

Questionwhat if from a webdav servermemberroychoo16 Mar '07 - 4:11 
hi, what if it is from a webdav server?
when one need username and password
how do i include that in ur code?
AnswerRe: what if from a webdav servermemberHadi Eskandari16 Mar '07 - 18:51 
Hi,
 
Did you check the demo app? There's already a LoginUsername and LoginPassword property in the downloader component which you can set if the site is password protected.

GeneralTest for file size if resumingmemberJeffrey Scott Flesher23 Jan '07 - 18:14 
When resuming test for file size to make sure it's not already at the end; otherwise it will crash.
something like this:
private void DownloadOperation()
{
long startingPoint = 0;
if(File.Exists(DownloadPath))
{
startingPoint = new FileInfo(DownloadPath).Length;
}
if (startingPoint > 0)
{
long FileSize = GetFileSize(FileURL);
if (startingPoint >= FileSize)
{
// Create Event to send back to user with options; overwrite, skip
return;
}
}
...
Good job.
Thanks
 
Lessons learned from 911:

1. United We Stand.

2. United’s We Fall.

Gulf War Syndrome survivors never have a good day. http://www.vetshelpcenter.com/

GeneralVery similar...memberScott Dorman15 Jan '07 - 6:30 
This article is very similar to some of the other articles here that cover this same topic. It does seem like it is most similar to this article:
 
An easy to use URL file downloader class for .NET[^]
 
If you derived your code from either this article, or any of the articles referenced in it, you really should give references to the original articles.
 
-----------------------------
In just two days, tomorrow will be yesterday.

GeneralRe: Very similar...memberHadi Eskandari16 Jan '07 - 19:22 
Hi,
 
Actually, I was preparing an article for my autoupdate component, and I thought it'd be a good idea to split it into two parts (a downloader and an autoupdate component). I saw your article as soon as 'Vertyg0' started mentioning it, and I admit they're very similar, but the whole idea is similar, don't you agree??
 
Best Regards,
Hadi Eskandari
GeneralRe: Very similar...memberScott Dorman18 Jan '07 - 4:09 
Yes, the whole idea is similar. I only mentioned something as both the article and the code had a lot of similarities. There are several articles on this topic and they all provide slightly different implemenations and features. My only comment was that if you used any of those articles as a reference, to simply state that fact. I was by no means implying anything.
 
What I find even more coincidental is that this downloader component was also developed as part of a larger system to provide auto update capabilities. The only difference is that the auto update component in this case had too much proprietary stuff to be able to turn into an article. It is, indeed, a small world. Smile | :)
 
-----------------------------
In just two days, tomorrow will be yesterday.

GeneralTnx For This Articlememberyunas14 Jan '07 - 2:36 
Thankes for This Article.
i am a VC++ Programer and i little Know C# Sigh | :sigh: But i found it very usefull.Smile | :)
GeneralJust a little problemmemberVertyg014 Jan '07 - 1:34 
Here you can find almost same component as you`rs and almost the same problem. You are running download on separate thread but still if I try to download file from my local IIS server (localhost) speed is about 3 mb/s and application utilize my CPU to 100%. Try it you`rself use some big file around 1 GB and try it downloading from http://localhost/thatfile.ext and you will see that application will utilize 100% of CPU.
You are maybe thinking who will download files from localhost, well there are people with some really fast connections and they can make it 2-3 mb/s for sure, I have developed once such application for one dutch guy and he reported me that my application hanged his CPU. Then I have asked him if he is downloading from localhost he said no I`m downloading from net by speed of 2 mb/s.
GeneralRe: Just a little problemmemberHadi Eskandari14 Jan '07 - 19:14 
Hello,
 
Actually, I tried it with my local IIS with large files, but my files weren't THAT large I guess Wink | ;)
I give it another shot and try to find a workaround.
 
Regards,
H. Eskandari
GeneralRe: Just a little problemmemberHadi Eskandari11 Feb '07 - 20:50 
Hi,
 
I've just found some time to do a little test on this. I've tested with a big file (Around 2GB) in my local IIS (On a P4 3300Mhz with 120GB SATA), and tried to download the file. Everything worked smoothly. TaskManager shows that the application is taking about 50% of CPU. I tried testing with even bigger files but seems my server didn't like it (abour 4GB server started reporting Server Error message).
 
If you still have problems, I could advise you to change the code to update the UI less frequently. This may solve the problem you may have. By the way, what's the configuration of the computer you tested this on?
 
Regards

GeneralRe: Just a little problemmemberVertyg011 Feb '07 - 21:42 
What is you`r computer configuration ?

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Permalink | Advertise | Privacy | Mobile
Web04 | 2.6.130516.1 | Last Updated 14 Jan 2007
Article Copyright 2007 by Hadi Eskandari
Everything else Copyright © CodeProject, 1999-2013
Terms of Use
Layout: fixed | fluid