Click here to Skip to main content
Click here to Skip to main content

A UDP-based Reliable Data Transfer Library

, 21 Mar 2006
Rate this:
Please Sign up or sign in to vote.
Introduces an open source UDP-based data transfer library.

Introduction

This article introduces an open source UDP-based data transfer library, namely UDT (UDP-based Data Transfer).

Currently, there are two major Internet transport protocols, TCP and UDP. TCP provides connection oriented reliable data streaming service, whereas UDP provides connection-less unreliable messaging service. Most applications use TCP to transfer data because they require data reliability.

However, when using TCP some applications suffer from two problems. First, TCP demonstrates poor performance in long distance links, especially when the bandwidth is high (e.g., 1GB/s or higher). Second, when incorporating multiple concurrent TCP flows with different RTTs in the same application, different TCP flows may have different data transfer rates (also known as RTT bias).

There are other situations where people do not want to use either TCP or UDP. For example, an application may require data reliability (so cannot use UDP directly) but also wants data to arrive at a user-controlled rate or the message boundaries to be conserved. Furthermore, sometimes it is easier to use UDP to traverse firewalls than TCP.

It may be helpful to build an open source user space transport protocol above UDP, but with the data reliability control and network flow/congestion control. Because it is at user space, it is easy to get installed. Because it is open source, it can be easily modified to meet various requirements from applications.

Example applications

Here we present two example applications in which you may need UDT.

a. Bulk data transfer and online streaming data processing

Suppose a company has their data stored in multiple branches around the world, each branch having its own part of the dataset. The datasets are very large (terabytes) and the company has a 1GB/s intranet to deliver them. Now one of its departments at Chicago branch wants to analyze its own dataset and the dataset of the London branch. This data analysis application will need to read two datasets (at London and Chicago, respectively) using the company's 1GB/s intranet to a computer at Chicago.

Recall the two problems of TCP mentioned above. First, the link must be extremely clean (very little packet loss) for TCP to fully utilize the 1GB/s bandwidth between London and Chicago. Second, when the two TCP streams (London->Chicago, Chicago->Chicago) start at the same time, the London-Chicago stream will be starved due to the RTT bias problem, thus the data analysis process will have to wait for the slower data stream.

b. Application awareness

A streaming video server is sending data to many clients. The server may choose a specific sending rate for each client during any specific period. You may have trouble asking TCP to send data at a fixed rate. And using UDP you have to do most of the data reliability control work by yourself. In addition, video frames are time sensitive data, if the frame is delayed too long, it will not be needed any more. That is, complete data reliability is not desired in this situation.

Using the UDT library, the programming work is simply several lines of option setting code.

UDT

UDT is a transport protocol with its own reliability control and flow/congestion control built above UDP. UDT provides both reliable data streaming service quite similar to TCP and partial reliable messaging. The latter allows users to send data as messages with specified delivery order and time-to-live value.

The position of UDT in the layered architecture of internet reference model is shown in the figure below. Applications use UDT socket to transfer their data, which is passed to the UDP socket. (Effort was made to avoid one time memory copy here.) A congestion control (CC) algorithm can be provided by applications; otherwise the default control algorithm is used. If user-defined control algorithm is provided, UDT will call the callback functions in CC once a control event occurs:

The default UDT control algorithm is designed for high performance data transfer over wide area high speed networks (e.g. in the case of example a). However, UDT's congestion control algorithm is configurable such that you can add your own control algorithm with ease (e.g., in the case of example b).

UDT provides a set of socket-like API. Using UDT in your application is simple as long as you are familiar with BSD socket. For example: With BSD socket, you write:

int s = socket(AF_INET, SOCK_STREAM, 0);

Its counterpart in UDT is:

UDTSOCKET u = UDT::socket(AF_INET, SOCK_STREAM, 0);

UDT is currently implemented using C++ and it supports Linux, Windows (2000, XP and above), and OS X. The current version is 3.0 Beta. After you download UDT, please read the "readme" file or the documentation for detailed installation and usage information. Note that if you are using Windows, you will need VC7 to get it compiled. With VC6, you will have to fix several incompatible C++ grammar by yourself.

If you are interested in the project, please refer to this site for more information.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

lilyco
Web Developer
United States United States
No Biography provided

Comments and Discussions

 
Questionerror code 1001 PinmemberMember 107820505-May-14 3:11 
GeneralMy vote of 5 PinmemberMember 432084428-Jun-12 22:47 
GeneralConnecting to a wireless pc from a pc inside a network trough a server. Pinmemberlukekando1-Jul-10 3:12 
GeneralRe: Connecting to a wireless pc from a pc inside a network trough a server. PinmemberMember 432084428-Jun-12 22:44 
QuestionDoes anyone know how to port and compile as a module? Pinmemberhookarm3-Mar-10 3:54 
QuestionUDT::select doesn't work? Pinmemberthunder5400726-Sep-07 2:31 
AnswerRe: UDT::select doesn't work? Pinmemberrealscan28-Mar-11 22:58 
NewsHint: UDT v3.3 works good, the UDT-m/UDT v4 did not work well at the moment. PinmemberJens.Schneeweiss17-Jul-07 0:02 
Generalconflict with the Queue and Multithreading PinmemberJens.Schneeweiss16-Jul-07 23:52 
Generalbreaking protocol from applications PinmemberGeode8511-Nov-06 3:58 
Questionmultiple UDT sockets PinmemberWee Lio31-Oct-06 19:58 
QuestionIs UDT Support Punching Firewall/NAT only in Rendezvous mode? PinmemberChina_Boy31-Aug-06 16:22 
QuestionHow to add multiple connection support in Rendezvous mode PinmemberHing17-Jul-06 16:54 
AnswerRe: How to add multiple connection support in Rendezvous mode Pinmemberlilyco17-Jul-06 17:12 
QuestionRe: How to add multiple connection support in Rendezvous mode PinmemberHing17-Jul-06 22:22 
GeneralRedezvous connection again PinmemberHing16-Jul-06 21:36 
GeneralRe: Redezvous connection again Pinmemberlilyco17-Jul-06 5:48 
GeneralQuestion on broken client [modified] PinmemberHing12-Jul-06 17:19 
GeneralRe: Question on broken client Pinmemberlilyco12-Jul-06 17:47 
GeneralRe: Question on broken client [modified] PinmemberHing12-Jul-06 18:45 
GeneralRe: Question on broken client Pinmemberlilyco13-Jul-06 13:17 
GeneralRe: Question on broken client PinmemberHing13-Jul-06 23:21 
GeneralRe: Question on broken client Pinmemberlilyco17-Jul-06 5:07 
GeneralQuestion on RENDEZVOUS connection [modified] PinmemberHing9-Jul-06 18:05 
GeneralRe: Question on RENDEZVOUS connection Pinmemberlilyco10-Jul-06 12:51 
GeneralRe: Question on RENDEZVOUS connection PinmemberHing10-Jul-06 18:04 
GeneralRe: Question on RENDEZVOUS connection PinmemberHing10-Jul-06 23:31 
GeneralRe: Question on RENDEZVOUS connection Pinmemberlilyco11-Jul-06 4:51 
GeneralRe: Question on RENDEZVOUS connection [modified] PinmemberHing11-Jul-06 5:29 
GeneralRe: Question on RENDEZVOUS connection [modified] Pinmemberlilyco11-Jul-06 19:55 
GeneralRe: Question on RENDEZVOUS connection [modified] PinmemberHing11-Jul-06 20:26 
GeneralRe: Question on RENDEZVOUS connection [modified] PinmemberHing11-Jul-06 22:13 
GeneralRe: Question on RENDEZVOUS connection Pinmemberlilyco12-Jul-06 3:35 
GeneralRe: Question on RENDEZVOUS connection [modified] PinmemberHing13-Jul-06 23:58 
GeneralRe: Question on RENDEZVOUS connection PinmemberHing11-Jul-06 22:30 
GeneralRe: Question on RENDEZVOUS connection Pinmemberlilyco12-Jul-06 3:38 
GeneralRe: Question on RENDEZVOUS connection Pinmemberzhangtieying15-Jul-06 15:41 
GeneralRe: Question on RENDEZVOUS connection PinmemberHing11-Jul-06 15:43 
GeneralWay to make it compatible with VC6 PinmemberHing7-Jul-06 18:23 
GeneralRe: Way to make it compatible with VC6 Pinmemberlilyco8-Jul-06 18:53 
GeneralVC6++ Pinmemberguojm88888824-Apr-06 19:16 
Questionproof of correctness? Pinmemberbelingueres28-Mar-06 5:35 
AnswerRe: proof of correctness? Pinmemberlilyco28-Mar-06 5:47 
GeneralA related protocol Pinmembermaihem23-Mar-06 13:23 
GeneralRe: A related protocol Pinmemberlilyco23-Mar-06 15:22 
GeneralRe: A related protocol PinmemberHing8-Jul-06 6:20 
GeneralC++ 6.0 Pinmemberhector santos22-Mar-06 18:34 
GeneralRe: C++ 6.0 Pinmemberlilyco23-Mar-06 11:13 
QuestionSome Questions Pinmemberwinni7815-Mar-06 9:26 
AnswerRe: Some Questions Pinmemberlilyco15-Mar-06 17:18 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web03 | 2.8.141015.1 | Last Updated 22 Mar 2006
Article Copyright 2005 by lilyco
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid