Click here to Skip to main content
Click here to Skip to main content

Commands Transfer Protocol (CTP) - A New Networking Protocol for Distributed or Parallel Computations

, 2 Feb 2005
Rate this:
Please Sign up or sign in to vote.
In this article, an improved version of a new networking protocol for distributed or parallel computations is presented. In common, it is suitable just for fast, reliable and featureful interchange of small messages. The protocol's implementation and demo project are provided.

CTP - sequence matters

Visit project's home site.

Introduction

First of all, what is meant here as "computational cluster". Cluster is a union of workstations, which is formed for some definite purposes. Computational cluster is a cluster, which is built for heavy computations. It is a specific system that asserts special requirements for network functionality. Main properties of the networking mechanism for a good quality cluster are:

  1. Fast data interchange.
  2. Reliable data transfer.
  3. Broadcasting support. As usual, all workstations inside some net take part in the computational experiment, so broadcasting makes controlling much easier.
  4. Huge data blocks interchange support. Sometimes, for example, initial conditions of experiment can be represented by such a block.
  5. Peer-to-peer networking. Any workstation can be the data source and the data destination, so they all are clients and servers simultaneously.

In fact, majority of parallel computing software toolkits are represented as libraries, which use standard networking protocol TCP/IP [1]. There are a lot of disadvantages of using this protocol:

  • Low speed of data interchange. The "reliability" and "universality" of TCP has a lot of overhead charges. This protocol is a general-purpose one, so it is suitable for working in such unstable matter as Internet, but in a constant (or quite constant) system, which was developed for computations, it is possible to get more benefits.
  • TCP does not support broadcasting. UDP does, but it is not reliable and the size of UDP datagram is limited by 65467 bytes [1].
  • Ideology of logical channel creation before data interchange is redundant for cluster computations. Firstly because cluster, as usual, is a well tuned, good working net. Secondly because, some strategies of cluster computing lead to disordered interchange between workstations.
  • TCP is a stream-based protocol, but, for determined tasks, bounded blocks interchange is preferable, because it allows to say definitely, when all data, necessary for further operations, have arrived.

Of course, a specialized networking protocol can be adapted for special requirements, which arise for cluster computations. So, CTP is a protocol that is to satisfy needs of arbitrary tasks, which, need support of rapid messages interchange and which, can start heavy computations,s as a reaction for message receipt. Despite the fact that the letter "P" from its name means "protocol", it is not just a specification. CTP is an ideology and toolkit, which allows to use it. So, it is able to replace such products as MPI implementations, PVM, and so on.

Ideology

The majority of existing toolkits for parallel computations use, the so named "message", as a basic abstraction. The basic abstraction used in CTP is "command". Command is an order from somebody to someone to do something (in most cases, workstations in clusters are communicating exactly in this way) or the response for such an order. From the last sentence, it is possible to conclude that a command is characterized by the following parameters:

  • "somebody" - sender.
  • "someone" - recipient.
  • "something" - command's description.

So, first of all, it is needed to define the sender and the recipient somehow. For this purpose, IP addresses will be used. The reason is that IP is used extremely widely and it fully satisfies the requirements (gives unique identifiers to all workstations). Commands will be identified by integer numbers.

In terms of the discussed protocol words, "command" and "message" are, actually, synonyms. "Command" is "message", but not always vice versa.

CTP needs to satisfy the cluster networking requirements, listed above. The way in which this will be achieved follows (in the same order as in the introduction):

  1. For incrementing the speed of interchange, UDP will be used as the basis of the protocol. Moreover, the usage of UDP, without going down to raw networking, will save the user in future from additional problems with the protocol support toolkit's installation.
  2. Reliability of data interchange is to be implemented. Each sent packet will be stored until the recipient has not confirmed the receipt of the data. To maintain this mechanism, packets are to be provided with identifiers. Identification will be performed by assigning integer numbers on the sender-side. These IDs cannot be unique in general, but are to be unique for each sender.
  3. Broadcasting support is one more argument to use UDP as the basic protocol.
  4. Huge data interchange support is to be implemented. If a message that is greater than some limit (65400 bytes, by default) is going to be sent, then it is to be divided into smaller parts. These parts will be enumerated and sent to the recipient separately, one by one. On the recipient's side, they will be united to arrange the initial command. An important note is that the recipient application will get information about the command's arrival only after all its parts have been received. Such commands will be named as "large commands", but on practice, the majority of commands are "normal" (need a single packet for its transfer).
  5. For peer-to-peer interchange, CTP's implementations are to include both client and server functionality, as a solid unity.

CTP/IP's relationship with the OSI-model [2] and UDP/IP ideology is shown on fig. 1.

Fig. 1. Relationship between OSI-model, UDP/IP-model and CTP/IP-model

Fig. 1. Relationship between OSI-model, UDP/IP-model and CTP/IP-model.

The fact that CTP covers a number of layers, from transport layer to application layer, proves that the area of its responsibility starts from relatively low level and goes to a high one.

Internal World

The following discussion is carried out in the form of chaotic descriptions of CTP basic concepts, their properties, and functionality. Reading of all this is to form a solid idea on what is going on inside. Such a strange form of statement was chosen, because systematic discussion is to be started from several points simultaneously, but this is, unfortunately, impossible.

Debug abilities of CTP may help to better understand the sequence of operations. There is a feature to place descriptions of all actions and events to a given output stream.

Header

Each CTP-packet is represented, as usual, as header plus body (data). The structure of the header is shown in table 1 (in the order of appearance).

Name Size (in bits) Comment
Packet size 16 Unsigned integer. Size of the packet (including header).
Command's number 16 Unsigned integer. Command's number (from 0 to 32767, highest bit is not set). If highest bit is set, then packet represents a confirmation for the message with the corresponding command's number.
Packet's number inside the large message 32 Unsigned integer. For large commands - packets number, from zero to amount of packets (given by next field of the header) minus unity. For normal commands - zero.
Amount of packets for the message 32 Unsigned integer. For large commands - amount of packets needed for its sending. For normal commands - zero or unity.
ID 32 Unsigned integer. Identifier of packet. Must be unique for each sender.
Message size 64 (only 48 are used) Unsigned integer. Size of the whole command's data (without any headers). So, size of the biggest command is the maximal amount of packets multiplied by the maximal size of packet, namely 232*65400. This equals 280890861158400 bytes, or more than 255 terabytes. Last value allows to consider the size of message as unlimited. It is obvious that 48 bits are enough to store size value, but 64 bits were apportioned for alignment.
Options 8 Set of bits. Each bit determines if corresponding option is set or not. Options will be discussed later.

Table 1. CTP packet's header

It is possible to calculate that the total size of the header is 25 bytes. All important transfer parameters, as IP addresses and ports of sender and recipient, are stored in the UDP header.

Each packet can be fully identified by its sender, its receiver, and ID. Sender provides uniqueness of ID for each recipient in the following way: initial value of ID for next packet that will be sent is taken as pseudorandom number. After sending each packet, it is to be incremented. The very first packet sent to the recipient, is to be marked with special option (see below) to allow recipient to learn the value of the starting ID.

Storages

The flowchart, which illustrates the sequence of operations CTP does for packet interchange, is shown on fig. 2:

Fig. 2. Flowchart of CTP's implementation

Fig. 2. Flowchart of CTP's implementation.

In the flowchart, so named, "storages" are mentioned. There are four storages for data, cumulated during lifetime, which provide functionality.

  1. Session information storage. It stores description of each workstation the current one communicates with. Among description, next packet ID, interchange timeout, and description of packets received from this recipient, are meant here.

    Interchange timeout is used to determine when the sent packets need to be resend if they have not been confirmed. This timeout is adoptive (because a cluster can be rather heterogeneous and can involve workstations via both intranet and Internet). Initially, default timeout is taken (100 milliseconds, by default). After the first interchange, its value is taken as time, needed for it, multiplied by coefficient (3, by default).

    If confirmation of packet's arrival will not be received during timeout, then the packet is to be resent. The period between resending will grow exponentially. If packet is not be confirmed after 8 re-sendings (255 timeouts will pass), then an error message "Command is not confirmed too long" will be generated. If timeout is set to zero, then this feature is switched off.

    Messages can be resent. So, it is necessary to protect the user from receiving one message several times. That is why, descriptions of received packets are stored for each addressee. It is implemented as an ordered list. First element contains the maximal ID of the packet, received in sequence. After this element, there can be more IDs, corresponding to packets, which have been received, but which are greater than the first element. After insertion of each new ID in this list, the sequence, which begins from the first element, is to be truncated. For example, let's assume that this storage contains {7, 9, 10, 11, 13, 14}. This means that all packets with ID less or equal to 7 and equal to 9, 10, 11, 13 and 14 already have been received. After receiving the packet with ID 8, the list will take the form {11, 13, 14}. If all packets arrive in sequence, then the list always contains a single element.

    Values of IDs are to be in the endless loop (after 232-1 goes 0). Determining of starting ID, which was generated by the sender, is very important in this stage.

    A new entry is added to session information storage when the first message is going to be sent or was received from workstation, or is unknown yet. There is to be a special entry for broadcasted messages.

  2. Sent commands storage. To send the command, packets are to be arranged. Some memory is to be allocated and filled with packet headers and data. The fact is that it will not be freed and unallocated just after sending, but stored to the sent commands storage. A record can be removed from sent commands storage only after all its packets arrival have been confirmed.

    This ideology can be implemented not for "each command", but for "each packet" (like in CTP 1.0), but first variant is preferable. In this case, so named "smart buffers" can keep from redundant memory allocations, by reserving and guarding memory needed for headers, while doing packets data arrangement.

  3. Large commands storage is used to arrange the whole large message, when receiving it part by part. It stores the total amount, the vector of parts receiving status, and a buffer for compiling. Each part of the message, except, probably, the last one, is of maximal data size, so parts can easily find their places in the buffer, knowing their numbers. When all parts have been received, the message is considered to be arranged and the server informs the application about data arrival.
  4. Deliveries storage. The whole received message or error description is, so named, "delivery". After generating, they will be added into deliveries list. Then special deliverer threads will take them from the list and pass them to the application.

    Speaking in terms of object-oriented programming: objects of classes, which implement the recipient application, can subscribe to get deliveries for given command. In this case, the corresponding object will receive information about the command's arrival, and about errors which are related to this command.

    Also, there is to be a default receiver that gets information about the command, which has no subscribers, and about common errors (like error while sockets creation) which have no related commands.

Confirmations

Confirmation is a packet with empty body (header only), which has only three differences from headers of the packet having been confirmed. In confirmations header:

  • Packet's size is set to header's size.
  • In command's number, highest bit is set.
  • Message size is set to zero.

It can be considered as an inefficient solution - to confirm each packet with separate confirmation, but it was done to provide more features by using options. Do not forget that CTP is not only a protocol, but also a toolkit.

After The Packet Have Been Sent

First thing the recipient does when it receives a packet is check if the same packet has been already received. If such a packet was already received, that means that the sender failed to get the confirmation, so confirmation has to be sent again and this packet's receiving procedure can be skipped.

Confirmation has to be, exactly, "sent again", not "resent". Confirmations are not stored in the sent packets storage, but are generated when needed.

If such a packet has not been received earlier, then information about its arrival needs to be stored.

If the got packet represent the normal command, then the server informs the application about data arrival (creates delivery and puts it to the deliveries storage). If it is a part of a large command, then the server stores it to the large command storage. If it is the last remainder part of the message, then delivery also is to be generated.

After the packet has found its place, confirmation has to be sent, and the recipient begins to wait for the next packet.

When the sender receives any confirmation, it is to delete the corresponding record from the sent packets storage. The mechanism, like a physical system, aspires to minimize its potential energy, to free all storages as soon as possible.

Threads

Implementation of protocol functionality is to be multithreaded. There are three types of threads:

  • Server threads receive packets, implement confirmation support, large commands arrangement and so on. If data arrives or error information appears, the thread adds it to the deliveries storage. Once per some period (100 milliseconds, by default), this thread checks the existence of packets that need resending, and resends them if necessary. There can be an arbitrary amount of server threads, depending on the task.
  • Deliverer threads check deliveries list, and if it is not empty, then delivers the first delivery to the corresponding subscriber or to the default receiver. If the thread does nothing for a long time (20 seconds, by default) then it will be terminated. On idle loop, a thread can fall asleep for some period (20 milliseconds, by default).

    Each command is an order (or response). It can order to do something difficult and enduring. So implementation of deliverers as separate threads allows to compute something "by order", just in place and moment, when it has been requested. Nevertheless, it is strongly recommended not to waste this feature. Do not use, for example, modal dialogs in command receiving handlers, because it will keep the deliverer busy uselessly.

  • Delivery manager thread creates additional deliverer threads if all existing deliverers are busy and deliveries list is not empty. Of course, the maximum amount of deliverers is limited by some value (50, by default). The protocol's mechanism aspires to reduce loading. On idle loop, a thread can fall asleep for some time (10 milliseconds, by default).

Options

Options allow to add an interesting functionality to the networking. There are five possible options:

  • DelAfterError - if set, then information about sending this packet will be removed for sent packets storage after the receiver has been informed that its arrival was not confirmed (after error description has been generated). So, this error will be delivered only once, and if data has not arrived, it will be lost.
  • NoResend - if set, this packet will not be resent even if it was not confirmed.
  • UniqueCommand - if set, then confirmation of this packet's command will confirm all packets with the same command's number that was sent to the given recipient. It is not allowed to use this option for large messages to protect from integrity corruption.
  • Broadcast - if set, then this packet is to be broadcasted. Option itself does not have any influence on the networking. As usual, for broadcasting, the user has to specify the recipient's IP address, like: 255.255.255.255. But this option provides two important things which are necessary for broadcasting with CTP. First of all, sent message has to get its ID and has to be stored in session information storage in entry, which corresponds to "broadcasting". Secondly, this message will get the ability to be confirmed by an arbitrary workstation, because it is impossible to know beforehand, who will get it. The command is considered to be confirmed if at least one workstation confirms it. It is not recommended (but possible) to use it for large commands.
  • StartSession - if set, then this packet is the first one which is sent by the sender to this recipient. So, ID, brought by it, can be taken as the minimal ID for the session with this sender. This is taken into account while checking, if packet has been received earlier or not.

    Note that the concept of "session" means one-way channel. If two workstations are exchanging some data, then both have entries for each other in the session information storage.

These options can be used in any combination: separately, all jointly, and so on.

For example, options set ErrorOnce, NoResend and UniqueCommand, ORed together, can be useful for commands like: "answer me if you are alive" (it is also called "ping"). For commands that are sent often, which are small and which does not bring information, but does response or confirmation - the recipient is working.

Implementation for Windows

Initially, CTP was implemented for the Windows operating system to become the basis of the networking mechanism used in (Cellular Automata Modeling Environment & Library) project [3]. See project's home site. Of course, it also could be used for arbitrary applications, which needed rapid messages interchange and heavy computations "by order".

The protocol's implementation is represented by a set of classes. The class which implements the main functionality of CTP has the name CCTPNet. The description of all classes which are involved in the CTP's implementation follows.

IPAddr Class

Objects of IPAddr class represent the IP-address of the workstation. This class does not need any explanations except the source.

union IPAddr
{
    // Data type for solid representation
    typedef unsigned __int32 IPSolid;

    // Actual data
    struct IPBytes {
        unsigned char b1,b2,b3,b4;
    } Bytes;
    IPSolid Solid;

    // Constructors
    IPAddr() {SetLocalhost();};
    IPAddr(unsigned char b1, unsigned char b2, 
           unsigned char b3, unsigned char b4)
      {Bytes.b1=b1;Bytes.b2=b2;Bytes.b3=b3;Bytes.b4=b4;};
    IPAddr(IPSolid l) {Solid=l;};

    // Returns true is this ip-address refers to localhost (127.0.0.1)
    inline bool IsLocalhost()
      {return Bytes.b1==127&&Bytes.b2==0&&Bytes.b3==0&&Bytes.b4==1;};

    // Returns true is this ip-address refers to broadcasting address
    // (255.255.255.255)
    inline bool IsBroadcasting()
      {return Bytes.b1==255&&Bytes.b2==255&&Bytes.b3==255&&Bytes.b4==255;};

    // Returns via s and return value dotted
    // string representation of ip-address
    inline LPTSTR GetString(LPTSTR s)
      {sprintf(s,"%d.%d.%d.%d",Bytes.b1, 
               Bytes.b2,Bytes.b3,Bytes.b4); return s;};

    // Set stored ip address to value, represented with string s (in
    // dot-separated format). Returns true if succeeded and false otherwise
    bool FromString(LPTSTR s);

    // Set ip address to localhost (127.0.0.1)
    inline void SetLocalhost()
      {Bytes.b1=127;Bytes.b2=0;Bytes.b3=0;Bytes.b4=1;};

    // Set ip address to broadcasting address (255.255.255.255)
    inline void SetBroadcast()
      {Bytes.b1=255;Bytes.b2=255;Bytes.b3=255;Bytes.b4=255;};

    // Operators
    bool operator ==(unsigned long ip) {return Solid==ip;};
    bool operator ==(IPAddr ip) {return Solid==ip.Solid;};
    bool operator !=(unsigned long ip) {return Solid!=ip;};
    bool operator !=(IPAddr ip) {return Solid!=ip.Solid;};
    IPAddr& operator =(const unsigned long ip) {Solid=ip; return *this;};
    IPAddr& operator =(const IPAddr ip) {Solid=ip.Solid; return *this;};
};

SmartBuffer Class

Objects of the class SmartBuffer represent "smart buffers" that save CTP's implementation from redundant memory allocations. It reserves the place for the packet's header once per definite size of the data (maximum amount of data in a single packet) on the fly. So, the user just puts data to the smart buffer. Then sending function inserts headers, and packets are ready to go out. This class' definition follows:

class SmartBuffer
{
public:
    // Constructor. Parameters:
    // +) datasize - amount of data to be used;
    // +) autodel - if true then this buffer will be freed automatically by
    //    protocol's implementation after it will be sent and confirmed if
    //    needed. So, working with such buffer, you will have to create it with
    //    operator new, but do not to delete it. If this parameter equals false
    //    you will have to do new and delete manually
    // +) headsize - size of header of each packet;
    // +) maxdatasize - maximum size of data in single packet (without header)
    SmartBuffer(unsigned int datasize=0, bool autodel=true, 
      unsigned int headsize=25, unsigned int maxdatasize=65400);


    // Constructor. Parameters:
    // +) fname - name of file to be stored in internal buffer as data;
    // +) datasize - amount of data to be stroed just before the file;
    // +) autodel - if true then this buffer will be freed automatically by
    //    protocol's implementation after it will be sent and confirmed if
    //    needed. So, working with such buffer, you will have to create it with
    //    operator new, but do not to delete it. If this parameter equals false
    //    you will have to do new and delete manually
    // +) headsize - size of header of each packet;
    // +) maxdatasize - maximum size of data in single packet (without header)
    SmartBuffer(LPCTSTR fname, unsigned int datasize=0, bool autodel=true, 
      unsigned int headsize=25, unsigned int maxdatasize=65400);

    // Destructor
    virtual ~SmartBuffer() {delete[] m_pBuffer;};

// Access to key values
    // Header size
    inline unsigned int GetHeadSize() {return m_uHeadSize;};

    // Data size
    inline unsigned int GetDataSize() {return m_uDataSize;};
    void SetDataSize(unsigned int datasize);

    // Maximum data size for single packet
    inline unsigned int GetMaxDataSize() {return m_uMaxDataSize;};

    // Allocated buffer size
    inline unsigned int GetBufferSize() {return m_uBufferSize;};

    // Auto deleting
    inline bool GetAutoDel() {return m_bAutoDel;}
    inline void SetAutoDel(bool autodel) {m_bAutoDel=autodel;}

// Access to key pointers
    // Returns begining of the buffer
    inline char* GetBufferBegin() {return m_pBuffer;};

    // Returns current pointer
    inline void* GetCurPtr() {return m_pCurPtr;};

    // Sets current pointer to pointer to data of i-th packet (i from zero to
    // GetPacketsCount()-1). Result false if index i is out of bounds
    inline bool CurPtrToDataPtr(unsigned int i)
      {char* res=GetDataPtr(i); 
       if (res) {m_pCurPtr=res; return true;} else return false;};
    
    // Sets current pointer to the begining of data
    inline void CurPtrToDataBegin() {m_pCurPtr=m_pBuffer+m_uHeadSize;};

// Access to packets
    // Returns amount of packets
    inline unsigned int GetPacketsCount()
      {return m_uBufferSize/(m_uHeadSize+m_uMaxDataSize)+
        ((m_uBufferSize%(m_uHeadSize+m_uMaxDataSize))?1:0);};

    // Returns pointer to header of i-th packet (i from zero to
    // GetPacketsCount()-1). Result is zero if index i is out of bounds
    inline char* GetHeadPtr(unsigned int i)
      {char* res=i*(m_uHeadSize+m_uMaxDataSize)+m_pBuffer; 
        if (res>m_pBuffer+m_uBufferSize) return NULL; else return res;};

    // Returns pointer to data of i-th packet (i from zero to
    // GetPacketsCount()-1). Result is zero if index i is out of bounds
    inline char* GetDataPtr(unsigned int i)
      {char* res=GetHeadPtr(i);
       if (res) return res+m_uHeadSize; else return NULL;};

    // Returns size of i-th packet (i from zero to GetPacketsCount()-1),
    // including header. Only last packet's size can differ from header's size
    // plus maximum data size. Result is zero if index i is out of bounds
    inline unsigned int GetPacketSize(unsigned int i)
      {if (i<(m_uBufferSize)/(m_uHeadSize+m_uMaxDataSize)) 
        return m_uHeadSize+m_uMaxDataSize; 
        else if (i==(m_uBufferSize)/(m_uHeadSize+m_uMaxDataSize))
          return (m_uBufferSize)%(m_uHeadSize+m_uMaxDataSize);
        else return 0;};

// Data and header access routines
    // Put data of size GetHeadSize() from src to the are of i-th header.
    // Returns true if data was copied successfully to the existing header etc
    // and false otherwise
    bool PutHead(void* src, unsigned int i);

    // Put byte of data bt to internal buffer from current pointer (if dest is
    // negative) or from dest-th byte.. Current pointer will be moved to the
    // end of put data (skipping headers) if movecur equals true. Returns true
    // if data was copied successfully
    bool PutDataByte(unsigned char bt, bool movecur=true, int dest=-1);

    // Put data of size size from src to internal buffer from current pointer
    // (if dest is negative) or from dest-th byte. Current pointer will be
    // moved to the end of put data (skipping headers) if movecur equals true.
    // Returns true if data was copied successfully, without truncation etc and
    // false otherwise
    bool PutData(void* src, unsigned int size, bool movecur=true, int dest=-1);

    // Put string to internal buffer from current pointer (if dest is negative)
    // or from dest-th byte. Current pointer will be moved to the end of put
    // data (skipping headers) if movecur equals true. Returns true if data was
    // copied successfully, without truncation etc and false otherwise
    inline bool PutDataString(char* str, bool movecur=true, int dest=-1)
      {return PutData(str,strlen(str)+1,movecur,dest);};

    // Put data from file fname to internal buffer from current pointer
    // (if dest is negative) or from dest-th byte. Current pointer will be
    // moved to the end of put data (skipping headers) if movecur equals true.
    // Returns true if data was copied successfully, without truncation etc and
    // false otherwise
    bool PutDataFile(LPCTSTR fname, bool movecur=true, int dest=-1);

    // Trim the buffer, by cutting the content, excluding the part of buffer
    // from current pointer to the end. It is strongly to perform this
    // operation only after all buffer's modifications
    void Trim();

protected:
    // Calculates needed buffer size
    inline unsigned int GetNeededBufferSize(unsigned int datasize, 
           unsigned int headsize, unsigned int maxdatasize)
      {return datasize?(datasize/maxdatasize*(headsize+maxdatasize)+
        ((datasize%maxdatasize>0)?(datasize%maxdatasize+ 
                                  headsize):0)):headsize;};

    // Calculate pointer by dest (if negative - then by m_pCurPtr). Result
    // pointer will be put to ptr. If prtnsize is not NULL then portion size
    // will be also calculated and put to variable, pointed by prtnsize
    inline void DestToPtr(int dest, char*& ptr, unsigned int* prtnsize);

// Data members
    bool m_bAutoDel; // Detele it automatically or not
    unsigned int m_uHeadSize; // Size of header
    unsigned int m_uDataSize; // Size of data
    unsigned int m_uMaxDataSize; // Maximum size of data in single packet

    unsigned int m_uBufferSize; // Size of allocated internal buffer
    char* m_pBuffer; // Pointer to internal buffer
    char* m_pCurPtr; // Current pointer
};

NetSender Class

Class NetSender is a base class for CCTPNet, that implements the main functionality of CTP. NetSender is used only to describe the interface of the common network sending class for an arbitrary protocol.

class NetSender
{
public:
    // Send smart buffer sb to address to. Parameter command represents command
    // id. Parameter options - sending options. If parameter storeiffail is
    // true then sent command will be stored in sent packets storage even if
    // sending fails and will not overwise. Returns true if succeeded (message
    // has gone) and false otherwise
    virtual bool Send(SmartBuffer& sb, unsigned __int16 command, 
      IPAddr to, unsigned __int8 options=0, bool storeiffail=true)=0;

    // Returns true if sender is working (is not suspended and so on) and false
    // otherwise. Default implementation returns true, because implementation
    // of many protocols cannot be switched off and cannot handle errors at all
    virtual bool IsWorking() {return true;};
};

NetReceiver Class

If there is NetSender class, then there must be NetReceiver class also. NetReceiver is used to describe the interface of objects that can subscribe for the delivery of information about data arrival and errors.

class NetReceiver
{
public:
    // Is called when have received data pointed by data
    virtual void OnReceive(void* data)=0;

    // Is called when there was an error, described with data pointed by data
    virtual void OnError(void* data)=0;
};

If an object is to subscribe for deliveries, it is necessary for its class to be NetReceiver's descendant (support of multiple inheritance in C++ allows to add an additional ancestor to any class). Member-functions OnReceive and OnError will be called on message arrival and on error, respectively. In the first case, pointer data will point to the description of arrived data; in the second case, data will point to some error description generated by NetSender's descendant.

Parameters of these member-functions are pointers to void, not to some concrete class, because NetReceiver class is also oriented to arbitrary protocols. When working with CTP, OnReceive's parameter will point to an object of CCTPReceivedData class, and OnError's parameter will point to an object of CCTPErrorInfo class.

CCTPReceivedData Class

Objects of this class describe and give access to the received data. It needs no explanation.

struct CCTPReceivedData {
    // Constructor. Parameters:
    // +) command - command;
    // +) size - amount of data to be stored;
    // +) from - ip address of host, that sends this data;
    // +) buf - points to buffer, which stores received data. If NULL then
    //    data copying will be skipped (only allocation performs)
    CCTPReceivedData(unsigned __int16 command, 
      unsigned __int64 size, unsigned long from, char* buf);
    // Destructor
    virtual ~CCTPReceivedData() {delete[] pBuf;};

    unsigned __int16 command; // Command
    unsigned __int64 size; // Message size (48 bit)
    IPAddr from; // Host, that had sent this data
    char* pBuf; // Data
};

CCTPErrorInfo Class

Objects of this class describe the error occurred while networking.

struct CTCPErrorInfo {
    // Constructor. Parameters:
    // +) type - error type. When it occurs:
    //    +) 0 - on socket creation;
    //    +) 1 - on socket binding or tuning;
    //    +) 2 - on data sending;
    //    +) 3 - on data receiving;
    // +) code - WinSock error code;
    // +) addr - address of host, which causes error (not always can be
    // interpreted, if can not the just equals localhost)
    CTCPErrorInfo(unsigned char type,int code,IPAddr addr)
      {this->type=type; this->code=code; 
       this->addr=addr; GetTimeStamp(timestamp);};

    // Put time stamp to string s and returns it
    static char* GetTimeStamp(char* s);

    unsigned char type; // Error type
    int code; // WinSock error code
    IPAddr addr; // Address of host, which causes error
    char timestamp[22]; // Time stamp, when error occurred
};

Everything must be clear above, except one circumstance: why timestamp is needed as a field of this class? It is the fact that the moment, when a network error occurred, may be very important, for example, for building log files. But the time when the error's description has been delivered may differ greatly from the time when it had taken place. To avoid such mistakes, timestamp was decided to be included as a field of CCTPErrorInfo class, and will be filled just during the object's construction. Timestamp for the current moment can be always retrieved with the help of the static member-function GetTimeStamp with the accuracy of thousandth of a second.

CCTPNet Class

This class implements CTP's main functions (client and server simultaneously). The following definition describes the members:

class CCTPNet: public NetSender
{
public:
    // Data structures

    // Packet header
#pragma pack(push)
#pragma pack(1)
    struct Header {
        // Constructor
        Header() {size=0;command=0;number=0;amount=0;id=0;messize=0;options=0;}

        void ToStream(ostream& out);

        unsigned __int16 size    ; // Packet size (16)
        unsigned __int16 command ; // Command (16)
        unsigned __int32 number  ; // Packet number (from zero to amount-1) (32)
        unsigned __int32 amount  ; // Amount of packets in the command (32)
        unsigned __int32 id      ; // Packet id (32)
        unsigned __int64 messize ; // Message size (64, but 48 are used)
        unsigned __int8  options ; // Options (8)
    };
#pragma pack(pop)

    // Options bits
    enum Options {
        // Delete sent command from the storage after error was
        // generated
        DelAfterError=0x01,

        // Do not resend this packet even if it was not confirmed
        NoResend=0x02,

        // Confirmation of this packets command will confirm all packets with
        // the same command, that was sent to same recipient.
        // NB: It is not recommended to use this option with multipacket
        // messages to protect it from integrity corruption
        UniqueCommand=0x04,

        // Broadcast this message (message with this option will be confirmed
        // from arbitrary recipient)
        Broadcast=0x08,

        // Mark packet, which is first in the session (in the interchange with
        // given recipient)
        // Note: This option is used by CTP internal world and is not needed to
        // be set by user
        StartSession=0x10
    };

    // Set of options, which appropriate for ping
    static const unsigned __int8 OptPing;

    // Structures for messages and error information delivery
    enum DeliveryType {
        ReceivedData,
        ErrorInfo
    };
    struct Delivery {
        // Constructor
        Delivery(NetReceiver* target,CCTPErrorInfo* data)
          {this->target=target; this->data=data; 
           this->type=DeliveryType::ErrorInfo;};
        Delivery(NetReceiver* target,CCTPReceivedData* data)
          {this->target=target; this->data=data; 
           this->type=DeliveryType::ReceivedData;};
        Delivery() {target=NULL; data=NULL; type=(DeliveryType)NULL;};
        // Only for STL compliance

        NetReceiver* target; // Receiver
        void* data; // Data
        DeliveryType type; // Delivery type
    };
    typedef list<Delivery> DeliveriesList;

    // Time settings storage structure
    struct Times {
        // Constructor, which sets defaults
        Times() {
            uMultiplier=       3;
            uDefTimeout=       100;
            uSleepOnDestroy=   50;
            uSleepSuspended=   10;
            uSleepDelMan=      10;
            uSleepNothing=     20;
            uPeriodDestroy=    2000;
            uPeriodAutoDest=   20000;
            uPeriodCheckResend=100;
        };

        // Multiplier for the time, needed for single transfer, to determine
        // timeout
        unsigned int uMultiplier;

        // Default timeout
        unsigned int uDefTimeout;
    
        // Sleeping time during waiting for desroying
        unsigned int uSleepOnDestroy;

        // Sleeping time when suspended
        unsigned int uSleepSuspended;

        // Sleeping time in deriveries manager
        unsigned int uSleepDelMan;

        // Sleeping time when server has nothing to do
        unsigned int uSleepNothing;

        // Period while waiting for desroying, after which working threads will
        // be stopped forcedly
        unsigned int uPeriodDestroy;

        // If deliverer will do nothing during this period it will be destroyed 
        unsigned int uPeriodAutoDest;

        // Period of checking if some packets are to be resent
        unsigned int uPeriodCheckResend;
    };

    // Constructor creates server with all necessary parameter and tunes client
    // for fast data sending:
    // +) receiver - default receiver of arrived data and errors;
    // +) port - number of port, which to listen;
    // +) servers - amount of setvers to be started;
    // +) times - pointer to time settings storage structure (if NULL then
    //    defaults will be used);
    // +) log - points to output stream for gebug log building (if NULL then no
    //    output will be produced.
    // +) packetdatasize - value of maximum data size to be send in single
    //    packet (if message is bigger than it is the "large message");
    // +) maxthreads - maximum amount of deliverers threads
    CCTPNet(NetReceiver* receiver, unsigned short port, 
         unsigned short servers=1,Times* times=NULL,ostream* log=NULL, 
         unsigned __int16 packetdatasize=65400, 
         unsigned short maxthreads=50);

    // Destructor
    virtual ~CCTPNet();

    // Parameter access routines

    // Time settins routines
    const Times& GetTimes() {return m_Times;}
    void SetTimes(Times& times) {m_Times=times;}

    // Port routines
    unsigned short GetPort() {return m_uPort;}
    void SetPort(unsigned short port)
      {closesocket(m_SendSocket); closesocket(m_RecvSocket); 
       FreeSntCommands(); FreeSessions(); FreeLargeCommands(); 
       m_uPort=port; CreateSockets();}

    // Packet data size routines
    unsigned __int16 GetPacketDataSize() {return m_uPacketDataSize;}
    void SetPacketDataSize(unsigned __int16 ps)
      {delete[] m_pBuffer; m_uPacketDataSize=ps; 
       m_pBuffer=new char[m_uPacketDataSize+GetHeaderSize()];}

    // Maximal threads amount routines
    void SetMaxDeliverers(unsigned short maxthreads)
      {m_uMaxDeliverers=maxthreads;};
    unsigned short GetMaxDeliverers() {return m_uMaxDeliverers;};

    // Info target routines
    NetReceiver* GetDefaultReceiver() {return m_DefReceiver;}
    void SetDefaultReceiver(NetReceiver* receiver) {m_DefReceiver=receiver;}
    // Sets special receiver receiver for command command and type type. If
    // receiver for definite command and delivery type already exists it will
    // be replaced
    void AddSpecialReceiver(unsigned __int16 command, 
                 NetReceiver* receiver, DeliveryType type);
    // Delete receiver receiver from special receivers list
    void DeleteSpecialReceiver(NetReceiver* receiver);
    // Returns receiver for command command and type type
    NetReceiver* GetReceiver(unsigned __int16 command, DeliveryType type);

    // Suspending status routines
    bool GetSuspended() {return m_bSuspended;};
    void SetSuspended(bool suspended) {m_bSuspended=suspended;};
    virtual bool IsWorking() {return !m_bSuspended;};

    // Operations

    // Returns size or datagrams header
    static unsigned __int16 GetHeaderSize() {return sizeof(Header);}

    // Send smart buffer sb to address to. Parameter command represents command
    // id. Parameter options - sending options. If parameter storeiffail is
    // true then sent command will be stored in sent packets storage even if
    // sending fails and will not otherwise. Returns true if succeeded (message
    // has gone) and false otherwise
    virtual bool Send(SmartBuffer& sb, unsigned __int16 command, 
         IPAddr to, unsigned __int8 options=0, bool storeiffail=true);

    // Send dataless packet (header only). Parameter command represents command
    // id. Parameter options - sending options. If parameter storeiffail is
    // true then sent command will be stored in sent packets storage even if
    // sending fails and will not otherwise. Returns true if succeeded (message
    // has gone) and false otherwise
    bool Send(unsigned __int16 command, IPAddr to, unsigned __int8 options=0, 
         bool storeiffail=true) {return Send(*(new SmartBuffer()), 
         command,to,options,storeiffail);};

    // Save information about packet received from from with header head.
    // Returns true if new message was received and false otherwise
    bool SaveRcvPacket(unsigned long from,Header* head);

    // Mark packet sent to to with header pointed by header as confirmed.
    // Memory will be cleared if possible
    void ConfirmSntPacket(unsigned long to,Header* header);

    // Send to to confirmation of receipt of the packet with header, pointed by
    // header
    void SendConfirmation(unsigned long to,Header header);

    // Arrange large packet received from from with hearer pointed by head.
    // Function returns true if solid message is arranged after last packet
    bool ArrangeLargeCommand(unsigned long from,Header* head);

    // Resend packets, which have not got confimational commands
    void ResendNotConfirmedData();

    // Determines is command is confirmation of command or not
    inline bool IsConfirmation(unsigned __int16 command)
      {return (command&m_iConfirm)!=0;};

    // Retrieving status information functions

    // Returns amount of entries in sent commands storage
    inline unsigned int GetSntCommandsCount() {return m_SntCommands.size();};

    // Returns amount of entries in created sessions information storege
    inline unsigned int GetSessionsCount() {return m_Sessions.size();};

    // Returns amount of entries in large messages storage
    inline unsigned int GetLrgMessagesCount() {return m_LargeCommands.size();};

    // Returns current amount of deliverer threads
    inline unsigned int GetDelThreadsCount() {return m_pDeliverTrds.size();};

    // Returns current amount of busy deliverer threads
    inline unsigned int GetBusyDelThreadsCount() {return m_uBusy;};

    // Returns current amount of deliveries
    inline unsigned int GetDelCount() {return m_Deliveries.size();};

protected:
    // Free buffers of all sent packets
    void FreeSntCommands();

    // Free information about received packets
    void FreeSessions();

    // Free buffers of large packets
    void FreeLargeCommands();

    // Free all planned deliveries
    void FreeDeliveries();

    // Actually send packet pointed with buf to recipient to. Returns true if
    // succeeded and false otherwise
    bool SendPacket(char* buf, unsigned long to);

    // Creates sending and receiving sockets. Returns true if succeeded and
    // false otherwise
    bool CreateSockets();

    // Check the options validity in the given header
    void CheckupOptions(Header& header);

public:
    // Socket for data receiving
    SOCKET m_RecvSocket;

    // Receiving buffer
    char* m_pBuffer;

    // Equals true if server and delivery threads needs to be finished
    bool m_bKill;

    // Maximal amount of delivery threads
    unsigned short m_uMaxDeliverers;

    //Output stream for log building
    ostream* m_pLog;

    // Deliveries (received messages and error information) storage
    DeliveriesList m_Deliveries;

    // Threads handles
    vector<CWinThread*> m_pServerTrds; // Server threads
    CWinThread* m_pDelManTrd; // Delivery manager thread
    vector<CWinThread*> m_pDeliverTrds; // Deliverers threads

    // Critical section for server threads access
    CCriticalSection m_csServerTrds;

    // Critical section for deliverers threads access
    CCriticalSection m_csDeliverTrds;

    // Critical section for deliveries access
    CCriticalSection m_csDeliveries;

    // Critical section for sent packets storage access
    CCriticalSection m_csSntCommands;

    // Critical section for sent recipients access
    CCriticalSection m_csSessions;

    // Critical section for sent large commands storage access
    CCriticalSection m_csLargeCommands;

    // Critical section for network access
    CCriticalSection m_csNetwork;

    // Critical section for log access
    CCriticalSection m_csLog;

    // Stores amount of busy deliverers threads
    unsigned short m_uBusy;

    // Determines is a<b or not, taking overruning in the account (0xffffffff
    // is less than zero)
    inline static bool Less(unsigned __int32 a,unsigned __int32 b)
      {if (max(a,b)-min(a,b)>0x7fffffff) 
        return !(a<b); else return a<b;}

    // Bit mask which is to be set for confirmations
    static const unsigned __int16 m_iConfirm;

protected:
    // Fills id field of header, refered by head for data, which will be sent
    // to recipient addr. If recipient's address is for broadcasting, then
    // option Broadcast is to be set set beforehand. This function will also
    // set StartSession option, if needed
    void GetNextID(Header& head, IPAddr addr);

    // Returns timeout for the session with workstation addr or for
    // broadcasting, if parameter bcast equals true
    unsigned int GetTimeout(IPAddr addr, bool bcast);

    // Sets timeout to the value of parameter timeout for the session with
    // workstation addr or for broadcasting, if parameter bcast equals true.
    // Value will be set only if current value is zero
    void SetTimeout(IPAddr addr, bool bcast, unsigned int timeout);

    // Type definitions

    // Structures for sent packets
    struct SntCommandInfo {
        // Constructors
        SntCommandInfo():sbBody(*(new SmartBuffer()))
          {CI=NULL; uCount=0;} // Only for STL compliance
        SntCommandInfo(SmartBuffer& sb, DWORD time, 
                    unsigned long to):sbBody(sb)
          {ipTo=to; uCount=sb.GetPacketsCount(); CI=new CommandInfo[uCount]; 
           for (unsigned int i=0; i<uCount; i++)
             {CI[i].dwTime=time; CI[i].dwLTime=time;}}

        // Confirms receiving of i-th packet. Returns true if this object can
        // be excluded from sent commands list and false otherwise
        bool Confirm(unsigned int i);

        // Free memory, controlled by this sent command information storage
        inline void Free() {delete[] CI; if (sbBody.GetAutoDel()) delete &sbBody;};

        // Representation of recipients IP address
        unsigned long ipTo;
        // Reference to smart buffer
        SmartBuffer& sbBody;
        // Amount of messages in this command
        unsigned __int32 uCount;

        // Structure for single command info
        struct CommandInfo {
            // Constructor
            CommandInfo() {uResend=1; dwTime=0; dwLTime=0; bConfirmed=false;};
        
            // Increment period between sendings
            void IncResend() {if (uResend<16384) uResend<<=1;}
            // Dead timeout has elapsed
            bool IsDeadTimeout() {return uResend>=256;}

            // Period between sendings
            unsigned int uResend;
            // Creation time
            DWORD dwTime;
            // Last sending (or resending) time
            DWORD dwLTime;
            // Was confirmed or not
            bool bConfirmed;
        };

        // Array of commands' information
        CommandInfo* CI;
    };
    typedef list<SntCommandInfo> SntCommandInfoList;

    // Structures for session description
    struct SessionInfo {
        // Constructor
        SessionInfo()
          {id=rand()*rand(); timeout=0; 
           received.clear(); minwasset=false;}

        // Type for received messages list
        typedef list<unsigned __int32> RcvList;

        unsigned __int32 id; // Next id
        bool minwasset; // Was minimal id already set or it was not 
        unsigned int timeout; // Timeout
        RcvList received; // Ids of received packets
    };
    typedef map<IPAddr::IPSolid,SessionInfo> SessionsInfo;

    // Structures for storing parts of large packets
    struct LargeCommandInfo {
        // Constructors. Parameters:
        // +) command - command;
        // +) size - amount of data to be stored;
        // +) from - ip address of host, that sends this data;
        // +) id - first packet's id;
        // +) amount - amount of packets left in the message
        LargeCommandInfo(unsigned __int16 command, unsigned __int64 size, 
            unsigned long from, unsigned __int32 id, unsigned __int32 amount)
          {pRD=new CCTPReceivedData(command,size,from,NULL); 
           this->id=id; uCount=amount; received=new bool[uCount];
           for (unsigned int i=0; i<uCount; i++)
             received[i]=false;};
        LargeCommandInfo()
          {id=0; uCount=0; received=NULL; 
           pRD=NULL;}; // Only for STL compliance

        // Mark i-th part of commang
        // as received one. Returns true if all parts
        // were received and false otherwise
        inline bool GotPart(unsigned int i);
    
        // Free memory, controlled by this
        // large command information storage
        // (message's body is not destroyed)
        inline void Free() {delete[] received;};

        unsigned __int32 id; // First packet's id
        unsigned __int32 uCount; // Amount of packets for command
        bool* received; // Array of flags, which shows received or not
        CCTPReceivedData* pRD; // Points to received data
    };
    typedef list<LargeCommandInfo> LargeCommandInfoList;

    // Structures for special receivers
    struct SpecialReceiver {
        // Constructor
        SpecialReceiver(unsigned __int16 command, 
              NetReceiver* receiver, DeliveryType type)
          {this->command=command; 
           this->receiver=receiver;
           this->type=type;};
        SpecialReceiver()
        {command=0;receiver=NULL;
         type=(DeliveryType)NULL;};
         // For STL compliance

        unsigned __int16 command; // Command
        NetReceiver* receiver; // Receiver
        DeliveryType type;
        // Type of delivery to be sent to the receiver
    };
    typedef list<SpecialReceiver> SpecialReceiversList;

    // Storages and other data structures

    // Sessions information storage
    SessionsInfo m_Sessions;

    // Sent commands storage
    SntCommandInfoList m_SntCommands;

    // Large packets storage
    LargeCommandInfoList m_LargeCommands;

    // Points to receiver which will get messages and error information by
    // default (if no special receiver will be present)
    NetReceiver* m_DefReceiver;

    // Special receivers
    SpecialReceiversList m_Receivers;

    // Socket for data sending
    SOCKET m_SendSocket;

    // Local address, used for data receiving
    SOCKADDR_IN m_Local;

    // Port on which to work;
    unsigned short m_uPort;

    // Size of the data in packet
    unsigned __int16 m_uPacketDataSize;

    // Time settings
    Times m_Times;

    // Determines if this workstation is offline or not
    bool m_bSuspended;

    // Returns reference to corresponding session information. Broadcasting
    // session will be returned, if parameter bcast equals true
    SessionInfo& GetSessionInfo(IPAddr addr, bool bcast);
};

CCTPStatusDlg Class

This class allows to display a dialog which shows CTP loading: the amount of elements in storages and the amount of deliverer threads. It also gives the ability to suspend servers. The class has a constructor, which takes a reference to an object of CCTPNet class, to keep an eye on. CCTPStatusDlg class definition follows:

class CCTPStatusDlg : public CDialog
{
public:
    // Constructor. Parameter ctp gives
    // a reverence to CTP fuctionality class,
    // to keep an eye on it. It will be refreshed
    // each cycle milliseconds
    CCTPStatusDlg(CCTPNet& ctp, UINT cycle, 
                     CWnd* pParent = NULL);

// Dialog Data
    //{{AFX_DATA(CCTPStatusDlg)
    enum { IDD = IDD_CTPSTATUS };
    //}}AFX_DATA

// Overrides
    //{{AFX_VIRTUAL(CCTPStatusDlg)
    public:
    virtual BOOL DestroyWindow();
    protected:
    virtual void DoDataExchange(CDataExchange* pDX);
    //}}AFX_VIRTUAL

// Implementation
protected:
    // Set suspend status
    void SetSuspendStatus();

    // Reference to CTP
    CCTPNet& m_CTP;

    // Refreshment timer
    UINT m_uTimer;

    // Refreshment cycle
    UINT m_uCycle;

    // Generated message map functions
    //{{AFX_MSG(CCTPStatusDlg)
    afx_msg void OnTimer(UINT nIDEvent);
    virtual BOOL OnInitDialog();
    afx_msg void OnShowWindow(BOOL bShow, UINT nStatus);
    afx_msg void OnBsuspend();
    //}}AFX_MSG
    DECLARE_MESSAGE_MAP()
};

Remember, that if you want to use this dialog in your project, you have to carry the dialog template IDD_CTPSTATUS and strings IDS_CTP_SUSPEND and IDD_CTP_RESUME from the demo application to your one.

How to use all this?

First of all, add files CTPNet.h, CTPNet.cpp, NetBasics.h, and NetBasics.cpp to your project. Then put the following directives to StdAfx.h:

// Include MFC multithreading support
#include <afxmt.h>

// Include STL
#pragma warning(disable: 4786)
#pragma warning(push)
// Disable STL-critical warnings
#pragma warning(disable: 4245)
#pragma warning(disable: 4100)
#pragma warning(disable: 4663)
#pragma warning(disable: 4018)
#pragma warning(disable: 4097)
#include <map>
#include <list>
#include <vector>
#include <algorithm>
#include <iostream>
using namespace std;
// Enable STL-critical warnings
#pragma warning(pop)

// CRT library includes
#include <sys/timeb.h>
#include <time.h>
#include <math.h>

// Include Windows Sockets
#include <Winsock2.h>
#include <Ws2tcpip.h>

This code is provided in the file NetIncludes.h. It is also necessary to link WinSockets library ws2_32.lib to your project (choose "Project" | "Settings" | "Link", then type "ws2_32.lib" in "Object/library modules" edit field).

Then, start Winsock up. For this purpose, for example, put the following code in the project's main window's initialization function:

WSADATA wsaData;
WSAStartup(MAKEWORD(2,2),&wsaData);

After this, place the following code to start the CTP server up:

m_pCTP = new CCTPNet(m_pCTPReceiver,1515);
// Server created suspended so it needs to be started manually
m_pCTP->SetSuspended(false);

The last code is correct in the assumption that m_pCTPReceiver is NetReceiver's descendant. For example, you can add NetReceiver to your main window's parents.

Measurements

The demo application which is provided with this article allows to try the described CTP implementation. It also includes implementations of TCP and UDP in the same framework, so all these protocols can be used together and, obviously, can be compared (fig. 3).

Fig. 3. Time of interchange via CTP, TCP and UDP (where possible)

Fig. 3. Time of interchange via CTP, TCP and UDP (where possible) in microseconds versus size of command’s data.

For this experiment, system clocks of two workstations were synchronized via SNTP with the same time server, using NetTime 2.0. Extreme values of interchange time were taken for the diagram.

A similar result of CTP and UDP shows that CTP's implementation doesn't use a critical amount of resources. Its overhead expenses are small enough to be ignored.

CTP is twice faster than TCP while working with normal commands and not very large commands that can be brought by several packets. That is great result, because the overwhelming majority of interactions in clusters are performed using normal messages. Order to start computations, query of some values, and response for such queries are small.

TCP is better for large commands. Nevertheless, huge data blocks appear in cluster computations rarely, for example, on the stage of task separation (and even here, not always). An important note is that CTP is not critically slow for large blocks, so it can be used as networking mechanism in clusters, paying attention to the previous paragraph.

Besides, a test has been performed on two nodes, because this is more interesting here: pure protocol's implementation throughput. Results of comparison for rapid interchange between dozens of nodes are to be more pleasant for CTP, because its activities will stay the same, but TCP will loose a lot on channels creating and recreating. For CTP, it does not matter who the recipient is.

Reason why it is impossible to overcome TCP is because it is on kernel level, but CTP is implemented by the application. From one side - this is a disadvantage of the last one, but from another, it is absolutely independent and complete.

References

  1. Jones A., Ohlund J. Network Programming for Windows - Microsoft Press. 2000.
  2. Nemeth E., Snyder G., Seebass S., Hein T. R. Unix System Administration Handbook. Third Edition - Prentice Hall PTR. 2001.
  3. Naumov L. CAME&L - Cellular Automata Modeling Environment & Library // Cellular Automata. Sixth International Conference on Cellular Automata for Research and Industry (ACRI-2004). 2004. Available from project home site.

History

  • 19 April 2004 - Release of CTP v. 1.0.
  • 16 September 2004 - Release of CTP v. 1.1.

    Improvements:

    • "smart buffers" support;
    • timing performance becomes a bit better;
    • some bugs fixed.
  • 31 January 2005 - Release of CTP v. 1.2.

    Improvements:

    • session information is taken into account now; so now we have adaptive timeouts, intelligent resending, and so on;
    • now there is no need to store anything in the system registry;
    • confirmations interchange ameliorated;
    • received packets storage concept was substituted and improved;
    • full-scale broadcasting support;
    • support of multiple servers;
    • more features for "smart buffers";
    • featurefull debug and logging interface;
    • all timeouts and delays become tunable;
    • timing performance becomes significantly better;
    • a lot of bugs fixed.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

About the Author

dum
Web Developer
Russian Federation Russian Federation
Lev Naumov.
MSc in Computer Science. Graduated from Computer Technologies Department of Saint-Petersburg State University of Information Technologies, Mechanics and Optics.
Worked as C/C++ and Java programmer. Now - the research worker in "CAMEL Laboratory" and PhD student in Computer Technologies Department of Saint-Petersburg State University of Information Technologies, Mechanics and Optics.
Has scientific achievements in field of physics, automata theory, cellular automata theory, cluster computing. There are some publications.

Comments and Discussions

 
GeneralMy vote of 3 Pinmembersathitchai.Thailand12-Nov-13 19:07 
QuestionErrors when building CTPDemo in VS2008 PinmemberMember 14020222-Aug-09 16:34 
AnswerRe: Errors when building CTPDemo in VS2008 Pinmemberdum24-Aug-09 13:03 
GeneralUDP Message Size PinmemberJust someone else2-Jan-09 3:30 
GeneralRe: UDP Message Size Pinmemberdum13-Jan-09 6:13 
Generalhi Pinmemberhasangunes29-Oct-08 22:21 
Generalhi Pinmemberhasangunes29-Oct-08 22:21 
GeneralPermission require for publishing an article on CTP PinmemberTeenu Tyagi9-Sep-08 21:58 
GeneralRe: Permission require for publishing an article on CTP Pinmemberdum13-Sep-08 7:01 
GeneralRe: Permission require for publishing an article on CTP [modified] PinmemberTeenu Tyagi15-Sep-08 22:22 
GeneralRe: Permission require for publishing an article on CTP Pinmemberdum13-Oct-08 15:13 
Questionhow to use this for parallel computation. Pinmemberakhshay gandhi4-Mar-08 15:59 
hello
i am working on cluster computation from last 4-5 months, i need to know how this protocol is useful for cluster computation and better than the message passing.
i am using lam mpi for my programming.
 
thank you
Generaldebug assertion in demo file PinmemberRSE Thomas24-Nov-06 16:30 
GeneralRe: debug assertion in demo file Pinmemberdum24-Nov-06 16:38 
GeneralRe: debug assertion in demo file PinmemberRSE Thomas5-Jan-07 0:17 
GeneralRe: debug assertion in demo file Pinmemberdum5-Jan-07 1:07 
GeneralRe: debug assertion in demo file Pinmemberdickty27-May-07 20:39 
GeneralRe: debug assertion in demo file Pinmemberdum10-Jun-07 11:48 
GeneralNAT connection PinmemberHing11-Jul-06 23:40 
GeneralRe: NAT connection Pinmemberdum14-Jul-06 2:13 
Generallicense problem PinmemberHing11-Jul-06 17:01 
AnswerRe: license problem Pinmemberdum14-Jul-06 1:48 
QuestionProblem??? PinmemberCosescu Bogdan3-Jun-06 22:53 
AnswerRe: Problem??? Pinmemberdum8-Jun-06 8:28 
AnswerRe: Problem??? Pinmemberdum8-Jun-06 8:30 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web03 | 2.8.140709.1 | Last Updated 3 Feb 2005
Article Copyright 2004 by dum
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid