Ducascopy tickdata downloader

• Download TickDataDownloader.zip

Introduction

If you design an automatic trading system, the most common problem that you face is where to get the data for the testing (backtesting to be precise).  There are several sources of this data (called tickdata) but none of them is user friendly. Typically the task involves a lot of manual work, like downloading the files, unpacking them, building the database etc. Also the quality of the data almost always is far from perfect. Perhaps one of the best free sources of the tick data is a Swiss company Ducascopy. However if you decide to go for its data, you will be disappointed. The retrieval of the data is so user unfriendly and time consuming that you would think twice before going for it.

However this task can be easily automated. The structure of the data  stored on the Ducascopy servers is simple.  Each hour of tick data is represented by the individual file and the file's Url is constructed from the date and a symbol name. The only intricacy is that the months start with zero. Once this file is downloaded, it has to be unpacked by LZMA compressor. 

Using the code

 Constructing the URL for one day 

string BaseUrl = "http://www.dukascopy.com/datafeed/"; 
DateTime lowValue = start; 

while (lowValue <= end) 
{ 
    string dr_val = symbol + "/" + lowValue.Year.ToString() + 
    "/" + (lowValue.Month - 1).ToString("00") + 
    "/" + (lowValue.Day).ToString("00") + "/"; 
    
    DateTime Dt_base = new DateTime(lowValue.Year, lowValue.Month, lowValue.Day); 
    
    for (int h = 0; h < 24; h++) 
    { 
        SymbolAndURLContainer dc = new SymbolAndURLContainer(); 
        dc.Url = dr_val + h.ToString("00") + "h_ticks.bi5"; 
        TimeSpan ts = TimeSpan.FromHours(h); dc.dt = Dt_base + ts; Files_lst.Add(dc); 
    } 
}  

Running the application 

Just load the solution to Visual Studio 2012, compile and run. You have to chose the output folder before starting the app. 

Downloading the file  

This is done by  

public class HttpDownloader : BaseDownloader 

This class downloads the file asynchronously. It runs in its own thread. 

Unpacking

I use this implementation of LZMA

http://code.google.com/p/managed-lzma/ 

After the file is unpacked, it has to be parsed and interpreted. The bytes are stored as bigendian, so they have to be reversed.  When we finish with unpacking, the data is written to the csv file for further use. 

Where to get the latest version of this software

The latest release of this software can be found here: http://www.binomsoftware.com/download/Democode/DucasDNL/TickDataDownloader.zip

Also you can find the tick data server and the clients for MBTrading, Gaincapital, Histdata and Pepperstone brokers. Under the certain conditions the above products can be obtained free of charge. (and many other useful components and tips) 

Current version 1.0.04