MultiThread Download Accelerator Console






4.94/5 (5 votes)
Porting Axel a lighter download accelerator for Linux and other Unices to NATIVE WIN32
Introduction
Axel is a lighter download accelerator for Linux and other Unices. It should compile (and run) on BSD, Solaris, Darwin (Mac OS X) and Win32 (Cygwin) systems.
Porting to NATIVE WIN32 without Cygwin, what I did in the #if WIN32 #endif
, for example:
- Changed Linux file descriptor for the new socket to WIN32
SOCKET
descriptor referencing the new socket - Changed
close
,write
,read
toclosesocket
,send
,recv
- Changed
select
blocking thread toWSAEventSelect
Asynchronous I/O - Changed
pthread_t
for the thread toHANDLE
referenced the thread - Changed
pthread_create
,pthread_join
,pthread_cancel
toCreateThread
,WaitForSingleObject
,TerminateThread
- Changed Linux file descripter for the saving file to
HANDLE
referenced the saving state - Changed
open
,read
,write
,close
toCreateFile
,ReadFile
,WriteFile
,CloseHandle
Background
DEBUG Axel to see multi-thread downloading way:
- TCP connect (
socket
,bind
,connect
) to host such as http://localhost provided by IIS - Send http header request:
GET / HTTP/1.0 Host: www.codeproject.com Range: bytes=1- User-Agent: Axel 2.4 (WIN32)
- Get the content length from header reply:
HTTP/1.1 206 Partial Content Content-Type: text/html Last-Modified: Tue, 20 Sep 2011 03:39:05 GMT Accept-Ranges: bytes ETag: "3c2bb1d84677cc1:0" Server: Microsoft-IIS/7.5 X-Powered-By: ASP.NET Date: Fri, 24 Feb 2012 08:41:12 GMT Connection: close Content-Length: 688 /* Here is what I need */ Content-Range: bytes 1-688/689
- Divide the content range into threads' number for instance 10:
Downloading 0-67 using conn. 0 Downloading 68-136 using conn. 1 Downloading 137-205 using conn. 2 Downloading 206-274 using conn. 3 Downloading 275-343 using conn. 4 Downloading 344-412 using conn. 5 Downloading 413-481 using conn. 6 Downloading 482-550 using conn. 7 Downloading 551-619 using conn. 8 Downloading 620-688 using conn. 9
- Send http header request with range [from, to] via one of the threads:
GET / HTTP/1.0 Host: localhost Range: bytes=206-274 User-Agent: Axel 2.4 (WIN32)
recv
the buffer from SOCKET
, WriteFile
the buffer from http header's reply to HANDLE
CreateFile
with temporary file *.st (Axel style), terminate && close the thread when downloaded the content range. Temporary file is helpful to the server host resuming available.
Using the Code
The source file text.c is the main entry considering as a good example and testcase about how to use Axel API.
conf_t conf;
conf_init(&conf);
conf.num_connections = 10; /* multi-thread numbers */
conf.add_header_count = 0;
/* axel_t object construct */
axel_t *axel = axel_new(&conf, 0, "http://localhost");
strcpy(axel->filename, "filename.htm");
axel_open(axel);
axel_start(axel);
/* Have not downloaded the total content */
while (!axel->ready)
{
axel_do(axel);
/* printf some verbose about download threads info such as speed and complete percent */
}
/* axel_t object destruct */
axel_close(axel);
Points of Interest
Cross platform development - played with Axel console under Gentoo Linux to see the Multi-thread download accelerator working way, then ported with NATIVE WIN32 API to Windows. I complied the source code with VS2005 (Choose mbstring
project setting instead of Unicode which brings ^M CTRL-V CTRL-M, unrecognized Chinese character, and binary file trash issue) under Windows 7 64-bit, so it needs more contributors to debug under other Windows distribution.
Improvement
There is WSAPoll
supported only by Windows Vista, 7 && Server 2008 to determine the status of one or more sockets. It might be more efficient than WSAEventSelect
.
History
- 2012-02-27 xzhai BUG FREE :)
- 2012-02-26 xzhai
- Fixed Re-connection bug
- Fixed BIGSIZE file freezing bug
- 2012-02-24 xzhai
- Ported Axel v2.4 to WIN32 without Cygwin