Click here to Skip to main content
Click here to Skip to main content

MultiThread Download Accelerator Console

, 27 Feb 2012 GPL3
Rate this:
Please Sign up or sign in to vote.
Porting Axel a lighter download accelerator for Linux and other Unices to NATIVE WIN32

Introduction

Axel is a lighter download accelerator for Linux and other Unices. It should compile (and run) on BSD, Solaris, Darwin (Mac OS X) and Win32 (Cygwin) systems.

Porting to NATIVE WIN32 without Cygwin, what I did in the #if WIN32 #endif, for example:

  • Changed Linux file descriptor for the new socket to WIN32 SOCKET descriptor referencing the new socket
  • Changed close, write, read to closesocket, send, recv
  • Changed select blocking thread to WSAEventSelect Asynchronous I/O
  • Changed pthread_t for the thread to HANDLE referenced the thread
  • Changed pthread_create, pthread_join, pthread_cancel to CreateThread, WaitForSingleObject, TerminateThread
  • Changed Linux file descripter for the saving file to HANDLE referenced the saving state
  • Changed open, read, write, close to CreateFile, ReadFile, WriteFile, CloseHandle

Background

DEBUG Axel to see multi-thread downloading way:

  • TCP connect (socket, bind, connect) to host such as http://localhost provided by IIS
  • Send http header request:
    GET / HTTP/1.0
    Host: www.codeproject.com
    Range: bytes=1-
    User-Agent: Axel 2.4 (WIN32)
  • Get the content length from header reply:
    HTTP/1.1 206 Partial Content
    Content-Type: text/html
    Last-Modified: Tue, 20 Sep 2011 03:39:05 GMT
    Accept-Ranges: bytes
    ETag: "3c2bb1d84677cc1:0"
    Server: Microsoft-IIS/7.5
    X-Powered-By: ASP.NET
    Date: Fri, 24 Feb 2012 08:41:12 GMT
    Connection: close
    Content-Length: 688 /* Here is what I need */
    Content-Range: bytes 1-688/689
  • Divide the content range into threads' number for instance 10:
    Downloading 0-67 using conn. 0
    Downloading 68-136 using conn. 1
    Downloading 137-205 using conn. 2
    Downloading 206-274 using conn. 3
    Downloading 275-343 using conn. 4
    Downloading 344-412 using conn. 5
    Downloading 413-481 using conn. 6
    Downloading 482-550 using conn. 7
    Downloading 551-619 using conn. 8
    Downloading 620-688 using conn. 9
  • Send http header request with range [from, to] via one of the threads:
    GET / HTTP/1.0
    Host: localhost
    Range: bytes=206-274
    User-Agent: Axel 2.4 (WIN32)

recv the buffer from SOCKET, WriteFile the buffer from http header's reply to HANDLE CreateFile with temporary file *.st (Axel style), terminate && close the thread when downloaded the content range. Temporary file is helpful to the server host resuming available.

Using the Code

The source file text.c is the main entry considering as a good example and testcase about how to use Axel API.

conf_t conf;
conf_init(&conf);
conf.num_connections = 10;  /* multi-thread numbers */
conf.add_header_count = 0;
/* axel_t object construct */
axel_t *axel = axel_new(&conf, 0, "http://localhost");
strcpy(axel->filename, "filename.htm");
axel_open(axel);
axel_start(axel);
/* Have not downloaded the total content */
while (!axel->ready) 
{
    axel_do(axel);
    /* printf some verbose about download threads info such as speed and complete percent */
}
/* axel_t object destruct */
axel_close(axel);

Points of Interest

Cross platform development - played with Axel console under Gentoo Linux to see the Multi-thread download accelerator working way, then ported with NATIVE WIN32 API to Windows. I complied the source code with VS2005 (Choose mbstring project setting instead of Unicode which brings ^M CTRL-V CTRL-M, unrecognized Chinese character, and binary file trash issue) under Windows 7 64-bit, so it needs more contributors to debug under other Windows distribution.

Improvement

There is WSAPoll supported only by Windows Vista, 7 && Server 2008 to determine the status of one or more sockets. It might be more efficient than WSAEventSelect.

History

  • 2012-02-27 xzhai BUG FREE Smile | :)
  • 2012-02-26 xzhai 
    • Fixed Re-connection bug
    • Fixed BIGSIZE file freezing bug
  • 2012-02-24 xzhai
    • Ported Axel v2.4 to WIN32 without Cygwin

License

This article, along with any associated source code and files, is licensed under The GNU General Public License (GPLv3)

Share

About the Author

Leslie Zhai
Engineer
China China
An individual human existence should be like a river - small at first, narrowly contained within its banks, and rushing passionately past boulders and over waterfalls. Gradually the river grows wider, the banks recede, the waters flow more quietly, and in the end, without any visible break, they become merged in the sea, and painlessly lose their individual being.
Follow on   Twitter   Google+   LinkedIn

Comments and Discussions

 
GeneralMy vote of 4 Pinmemberfreeliey24-Feb-12 20:09 
GeneralRe: My vote of 4 PinmemberXiang Zhai24-Feb-12 21:09 
GeneralTODO PinmemberXiang Zhai24-Feb-12 20:03 
QuestionAny measurements? PinmemberJamming124-Feb-12 17:34 
AnswerRe: Any measurements? PinmemberXiang Zhai24-Feb-12 18:40 
GeneralRe: Any measurements? PinmemberJamming124-Feb-12 18:58 
GeneralRe: Any measurements? PinmemberXiang Zhai24-Feb-12 19:54 
Sorry for my simple reply, there is no comparison in my article, but there is already the comparison in UNIX/Linux geek`s mind Smile | :)
 
If you experienced Ubuntu a Linux distribution based on Debian, when installed software package under Gentoo Linux, it is better to use emerge to automatically install dependence. And emerge script choose wget as the default downloader that only support one connection. It is much slower especially when install Linux Kernel, Linux desktop such as Gnome or KDE.
 
So I simply answered your question as there is a comparison between wget and axel, BUT IN MY ARTICLE there is NO comparison between select and poll mentioned in the Improvement section.
GeneralRe: Any measurements? PinmemberMichael McConnell27-Feb-12 4:06 
GeneralRe: Any measurements? PinmemberXiang Zhai27-Feb-12 18:44 
QuestionHUNTING BUG Pinmemberxiangzhai24-Feb-12 16:28 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web03 | 2.8.141223.1 | Last Updated 27 Feb 2012
Article Copyright 2012 by Leslie Zhai
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid