Click here to Skip to main content
15,867,308 members
Articles / Programming Languages / C

A Small C-Language TCP Server Framework

Rate me:
Please Sign up or sign in to vote.
4.81/5 (27 votes)
30 Apr 2010CPOL8 min read 72K   3.2K   102   13
A framework for writing small to medium size cross-platform TCP servers

Introduction

This article describes a framework for writing small to medium size TCP servers, written in the C language with an object oriented approach, using the standard Berkeley sockets interface, and intended to be cross-platform. The development was done in a system with the Ubuntu Linux distribution installed, with later porting and a little testing in Windows 2003 with Visual Studio Express 2008.

Background

This framework comes as a consolidation of the many TCP servers I've come across in the many years that I have worked in the banking and retail automation fields. Those servers typically handled from a dozen to some hundred connections, running on private site-local networks.

Description

The framework uses non-blocking sockets to perform all communications functions, and all connections are managed by one thread only, therefore providing a high degree of scalability. Such scalability, however, bumps into the inherent limitations built into the standard Berkeley socket functions. Specifically, the limitations of the select socket function, which can handle connections in the hundreds range, but typically, not in the thousands range.

When writing a server using this framework, the business logic is provided by the application in the form of a dynamically configurable number of threads. Such threads are decoupled from the actual connections, so that applications don't have to worry about connection issues. It is relatively easy to configure the number and the type of the application threads.

Buffer management is also simplified, therefore minimizing a number of errors related to memory management. It is possible to configure static buffers, buffers associated to a connection, and ad hoc buffers, allocated if the other schemes fail. Also, copying is minimized: the only needed copy is the one from the operating system buffers to the framework buffers.

The messages exchanged on the wire use a counter and a flag, to provide for record boundaries and additional reliability.

A number of convenience functions are also provided, to aid in the writing of client applications, although the main goal of the framework is writing servers.

It is assumed that the servers written with the framework will generally be servers tied to specific business needs, and are meant for in house applications running on private networks. These servers are not meant to be general purpose servers running on the internet, like web servers and such.

Below is an informal UML structure class diagram, depicting the overall classes that comprise the framework.

Click to enlarge image

The heart of the framework is a ConnectionManager object. It owns a ConnectionTable instance which owns many Connection objects. Each connection in turn owns a Socket instance which does the actual communications work. Each connection also owns either one or two Message instances, one for receiving messages from and other for sending messages to the connected client.

The connection manager, as its name implies, manages the TCP connections. For each connection, it keeps track of the messages being sent and received: how many bytes were already transferred, and how many are still outstanding. It also manages new connection requests, and handles the errors spawned by all socket operations, closing connections in error if needed.

The framework's general flow of operation is illustrated by the sequence diagram below:

sequence.png

Every application Thread sits in a loop, waiting for input messages to become available for processing. The waiting is done by calling the QueueManager::waitInputMessage method. When the connection manager finishes assembling an input message, it signals the queue manager by invoking the QueueManager::addInputMessage method. This method adds the complete message to the QueueManager's input message queue, and awakens an application thread to process the input message. Notice that the message is not guaranteed to be processed right away: this depends on the availability of an idle application thread.

The application thread, after processing an input message, may generate an output message that is to be sent to the client application. The application thread makes the output message available to the framework by calling the QueueManager::dispatchOutputMessage method. Upon receiving this call, the queue manager adds the output message to its output message queue, and notifies the connection manager of the existence of the new output message by calling the ConnectionManager::notifyOutputMessage method. The connection manager then removes the output message from the output message queue, and schedules its transmission by the connection associated with the message.

Notice that the application thread may reuse the input message as an output message. Or it can dispose of the input message by calling the QueueManager ::disposeMessage method (which returns the message to the free message queue) and then request a free message to use as output message, by calling the QueueManager::getFreeMessage method. It's up to the application thread how it manages its messages, as long as it does something with them: either calls dispatchOutputMessage or disposeMessage on any message it owns, before blocking again after processing an input message. Notice also that a Message object always belongs to another object: either to a MessageQueue, or to a Thread, or to a Connection.

There are three other classes in the class diagram that merit some explanation.

The Server class is a façade class providing convenient access to the most important methods for writing server applications. All the methods in the server class are in fact #defines of methods belonging to another classes.

The Client class is also a façade, like the server class, but for writing client applications. Although the framework's aim is to make it easy to write server applications, all its infrastructure can be easily taken advantage of, to also write client applications. The client class has a bunch of #defines to aid its role as a façade class, but also adds some functionality of its own, in the form of specific methods for writing client applications.

Finally, the Log class writes to a log file a huge amount of information about what the framework does when an application that uses the framework is run. This log facility has the usual levels of severity: information, warning, error, fatal, debugging. It can also be used to trace the communications buffers, and can easily be called from inside the application.

For more complete documentation about the framework, it's easy to change the doxygen configuration file to get a lot more information. (Doxygen is the utility used to generate the documentation for this project.) Specifically, two useful options that can be activated are "CALL_GRAPH" and "CALLER_GRAPH". The output generated by these options are highly useful, but for large projects it can mess up with the layout of the output shown in the browser, because of the big size of the images created.

A First Example: Hello ?

Ok, now for something a little more interesting. Let's see how to create a "hello world"-type application.

First, the main program:

C++
int main(void)
{
   // does whatever initialization the framework needs
   server_init();

   // starts 3 threads of the same code
   server_addThreads(3, threadFunc, "example thread");

   // runs the server
   server_run();

   // that's it!
   return 0;
}

Now, the application thread. Remember, the framework will start 3 thread instances:

C++
// a minimal thread

static threadfunc threadFunc(void* arg)
{
   for (;;)
   {
      // uses "server_printf" (and not plain printf) because of
      // contention issues between threads
      server_printf("Hello from a minimal thread...\n");

      server_sleep(3);
   }

   return 0;
}

Each of the threads above will print a message on the console, sleep for 3 seconds, and will keep doing it over and over again.

Now the header files used by this little program:

C++
// common configuration options & declarations (always include first)
#include "config.h"

// application includes
#include "Server.h" /* server_xxx functions */

// prototypes
static threadfunc threadFunc(void*);

Yep, that's it, the whole program.

The points of interest are the server_xxx functions, and the threadfunc typedef, which are provided by the framework

Another Example Example Example: Echo

Now, the first server program in any book on TCP/IP: an echo server.

First, the main program:

C++
int main(void)
{
   // the port on which the server waits for connection requests
   server_setServicePort(12345);

   // how much detail is written to the log file
   server_setLogLevel(LOG_LEVEL_DEBUG);

   // does whatever initialization it's needed
   server_init();

   // starts 5 threads of the same code
   server_addThreads(5, threadFunc, "example thread");

   // runs the server
   server_run();

   // that's it!
   return 0;
}

Now the application thread again:

C++
static threadfunc threadFunc(void* arg)
{
   Message *msg;

   for (;;) // the thread main loop
   {
      // waits for a message from a client
      msg = server_waitInputMessage();
      server_logDebug("received a message");

      // echoes back the received message
      server_logInfo("ok, replying");
      server_dispatchOutputMessage(msg);
   }

   // yep, that's it!
   return 0;
}

The point of interest now is the Message typedef, which is provided by the framework, and contains the data sent by the client.

Another Example: Still Echoing, But Doing a Little More of Nothing

Now only the application thread is shown. It still doesn't do much, but at least it shows how to access and modify the data sent by the client. (In fact, the way it is shown here is a little insecure, but easy. There are some other methods that access the message contents in a slightly safer way, although as always, with C being C, you can do pretty much everything you want, including shooting yourself in the foot. And you can do it very fast, of course).

C++
threadfunc threadFunc1(void* arg)
{
   uint size, count = 0;
   char *bufIn, *bufOut;
   Message *msgIn, *msgOut;

   for (;;)
   {
      // waits for a message from a client
      msgIn = server_waitInputMessage();

      // ok, message received
      bufIn = server_messageBuffer(msgIn);
      size = server_messageSize(msgIn);

      // requests an (output) message to be sent as reply to the client
      msgOut = server_getFreeMessage();
      bufOut = server_messageBuffer(msgOut);

      // processes the (input) message received from the client
      server_printf("* message: length=%02d buf=[%.20s]\n", size, bufIn);
      // blah blah blah

      // creates the (output) message to be sent as reply to the client
      // (uses the same bytes and size, just so the client can check them)
      memcpy(bufOut, bufIn, size);
      server_setMessageSize(msgOut, size);

      if (!(++count % 10))
         server_logInfo("%d messages processed now", count);

      // copies connection information from the input message to the output
      // message (this is needed so that the framework knows which client to
      // send the output message to)
      server_copyConnectionFromMessage(msgOut, msgIn);

      // releases the input message, it's not needed anymore
      // (this wouldn't be needed if the input message were reused)
      server_disposeMessage(msgIn);

      // makes the message available to be sent to the client
      server_dispatchOutputMessage(msgOut);
   }

   // that's it
   return 0;
}

There are several points of interest here.

  • The server_messageBuffer and server_messageSize methods, that provides access to the raw data sent by the client, and to its size.
  • The server_getFreeMessage method, through which the application thread acquires a new Message instance that it will use as reply to the client.
  • The server_setMessageSize which sets the size of the reply message.
  • The server_copyConnectionFromMessage which sets the IP address of the client to which the reply will be sent.
  • The server_disposeMessage that is used by the application thread to return an unused message to the framework.

Now What ?

There are several enhancements that can be done to the code as it is.

There are a great number of assert calls sprinkled liberally throughout the code. These calls should be reviewed, and proper error handling should be provided for the cases that make sense.

The central class ConnectionManager can be made more class-like by getting rid of the global static variables that exist now. Doing so will enable us to write applications having more than one connection manager, that is, serving in two or more IP addresses or service ports.

Writing some unit tests wouldn't hurt either...There's only one class with tests written, because they were absolutely necessary.

One thing of note is that the code uses lots and lot of uints and ushorts, something that may annoy people who likes their ints and shorts raw.

History

  • 7th April, 2010: First version of the article, and covers the version 1.0.0 of the framework
  • 14th April, 2010: Version 1.01 - Updated source code 
    • Correction: Code inside "assert" calls taken outside
    • Included release builds
    • Compatibility of includes with the C++ language
    • Included C++ examples
  • 29th April, 2010: Version 1.02 - Updated source code
    • Correction: checks number of connections before creating new connection
    • Inclusion of the "generic client" class and example
    • Changed "CreateThread" to "_beginthreadex" in Windows
    • Reorganization of the code in ConnectionManager.c

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
zvx
Software Developer
Brazil Brazil
I'm a long-time software developer living in Brazil.

I've been developing software for retail and banking automation in C/C++ for many years now. In the old days I even did some COBOL programming, and some assembly for the 8080.

My experience ranges from low level software such as interface code for serial devices for DOS and Windows (bar code scanners, printers, cash dispensers, etc) and goes to writing end user applications for POS terminals and bank ATMs. In between I've done a great deal of TCP/IP programming using the basic Berkeley sockets interface, which is my main interest nowadays.

Comments and Discussions

 
QuestionVery well done! Pin
Member 6301288-Oct-13 13:13
Member 6301288-Oct-13 13:13 
QuestionTCP Speed Performance with Multiple Work Stations and Servers like Distributed Systems Pin
MrKyaw11-Sep-13 21:51
MrKyaw11-Sep-13 21:51 
Hi Jose,

Very good contribution and I would like to know whether the frame work is compatible for TCP Speed Performance with Multiple Work Stations(30,000) and Servers like Distributed Systems.

Since I started doing APIs similar to your concept with the above specification(subject) and pls advise me since I am quite new in TCP with beginning point.


Thanks and best regards
Kyaw
Questionhow to compile this source Pin
dhjang513-Mar-13 17:57
dhjang513-Mar-13 17:57 
QuestionThank's Pin
Boubakr9-May-12 8:15
Boubakr9-May-12 8:15 
QuestionGood code can It work in UDP mode? Pin
maplewang11-Jan-12 18:11
maplewang11-Jan-12 18:11 
AnswerRe: Good code can It work in UDP mode? Pin
zvx12-Jan-12 1:59
zvx12-Jan-12 1:59 
QuestionEmprego Pin
mbsteireira20-Dec-11 7:53
mbsteireira20-Dec-11 7:53 
GeneralMy vote of 5 Pin
CShawky18-Aug-11 4:54
professionalCShawky18-Aug-11 4:54 
GeneralGreat work! Some small problems. Pin
y5219-Apr-10 3:29
y5219-Apr-10 3:29 
GeneralRe: Great work! Some small problems. Pin
zvx19-Apr-10 14:38
zvx19-Apr-10 14:38 
GeneralRe: Great work! Some small problems. Pin
zvx20-Apr-10 11:57
zvx20-Apr-10 11:57 
GeneralSome assertion may error used Pin
y529-Apr-10 2:33
y529-Apr-10 2:33 
GeneralRe: Some assertion may error used Pin
zvx9-Apr-10 6:56
zvx9-Apr-10 6:56 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.