This article describes a framework for writing small to medium size TCP servers, written in the C language with an object oriented approach, using the standard Berkeley sockets interface, and intended to be cross-platform. The development was done in a system with the Ubuntu Linux distribution installed, with later porting and a little testing in Windows 2003 with Visual Studio Express 2008.
This framework comes as a consolidation of the many TCP servers I've come across in the many years that I have worked in the banking and retail automation fields. Those servers typically handled from a dozen to some hundred connections, running on private site-local networks.
The framework uses non-blocking sockets to perform all communications functions, and all connections are managed by one thread only, therefore providing a high degree of scalability. Such scalability, however, bumps into the inherent limitations built into the standard Berkeley socket functions. Specifically, the limitations of the select socket function, which can handle connections in the hundreds range, but typically, not in the thousands range.
When writing a server using this framework, the business logic is provided by the application in the form of a dynamically configurable number of threads. Such threads are decoupled from the actual connections, so that applications don't have to worry about connection issues. It is relatively easy to configure the number and the type of the application threads.
Buffer management is also simplified, therefore minimizing a number of errors related to memory management. It is possible to configure static buffers, buffers associated to a connection, and ad hoc buffers, allocated if the other schemes fail. Also, copying is minimized: the only needed copy is the one from the operating system buffers to the framework buffers.
The messages exchanged on the wire use a counter and a flag, to provide for record boundaries and additional reliability.
A number of convenience functions are also provided, to aid in the writing of client applications, although the main goal of the framework is writing servers.
It is assumed that the servers written with the framework will generally be servers tied to specific business needs, and are meant for in house applications running on private networks. These servers are not meant to be general purpose servers running on the internet, like web servers and such.
Below is an informal UML structure class diagram, depicting the overall classes that comprise the framework.
The heart of the framework is a
ConnectionManager object. It owns a
ConnectionTable instance which owns many
Connection objects. Each connection in turn owns a
Socket instance which does the actual communications work. Each connection also owns either one or two
Message instances, one for receiving messages from and other for sending messages to the connected client.
The connection manager, as its name implies, manages the TCP connections. For each connection, it keeps track of the messages being sent and received: how many bytes were already transferred, and how many are still outstanding. It also manages new connection requests, and handles the errors spawned by all socket operations, closing connections in error if needed.
The framework's general flow of operation is illustrated by the sequence diagram below:
Thread sits in a loop, waiting for input messages to become available for processing. The waiting is done by calling the
QueueManager::waitInputMessage method. When the connection manager finishes assembling an input message, it signals the queue manager by invoking the
QueueManager::addInputMessage method. This method adds the complete message to the
QueueManager's input message queue, and awakens an application thread to process the input message. Notice that the message is not guaranteed to be processed right away: this depends on the availability of an idle application thread.
The application thread, after processing an input message, may generate an output message that is to be sent to the client application. The application thread makes the output message available to the framework by calling the
QueueManager::dispatchOutputMessage method. Upon receiving this call, the queue manager adds the output message to its output message queue, and notifies the connection manager of the existence of the new output message by calling the
ConnectionManager::notifyOutputMessage method. The connection manager then removes the output message from the output message queue, and schedules its transmission by the connection associated with the message.
Notice that the application thread may reuse the input message as an output message. Or it can dispose of the input message by calling the
QueueManager ::disposeMessage method (which returns the message to the free message queue) and then request a free message to use as output message, by calling the
QueueManager::getFreeMessage method. It's up to the application thread how it manages its messages, as long as it does something with them: either calls
disposeMessage on any message it owns, before blocking again after processing an input message. Notice also that a
Message object always belongs to another object: either to a
MessageQueue, or to a
Thread, or to a
There are three other classes in the class diagram that merit some explanation.
Server class is a façade class providing convenient access to the most important methods for writing server applications. All the methods in the server class are in fact #defines of methods belonging to another classes.
Client class is also a façade, like the server class, but for writing client applications. Although the framework's aim is to make it easy to write server applications, all its infrastructure can be easily taken advantage of, to also write client applications. The client class has a bunch of #defines to aid its role as a façade class, but also adds some functionality of its own, in the form of specific methods for writing client applications.
Log class writes to a log file a huge amount of information about what the framework does when an application that uses the framework is run. This log facility has the usual levels of severity: information, warning, error, fatal, debugging. It can also be used to trace the communications buffers, and can easily be called from inside the application.
For more complete documentation about the framework, it's easy to change the doxygen configuration file to get a lot more information. (Doxygen is the utility used to generate the documentation for this project.) Specifically, two useful options that can be activated are "
CALL_GRAPH" and "
CALLER_GRAPH". The output generated by these options are highly useful, but for large projects it can mess up with the layout of the output shown in the browser, because of the big size of the images created.
A First Example: Hello ?
Ok, now for something a little more interesting. Let's see how to create a "
hello world"-type application.
First, the main program:
server_addThreads(3, threadFunc, "example thread");
Now, the application thread. Remember, the framework will start 3 thread instances:
static threadfunc threadFunc(void* arg)
server_printf("Hello from a minimal thread...\n");
Each of the threads above will print a message on the console, sleep for 3 seconds, and will keep doing it over and over again.
Now the header files used by this little program:
#include "Server.h" /* server_xxx functions */
static threadfunc threadFunc(void*);
Yep, that's it, the whole program.
The points of interest are the
server_xxx functions, and the
threadfunc typedef, which are provided by the framework
Another Example Example Example: Echo
Now, the first server program in any book on TCP/IP: an echo server.
First, the main program:
server_addThreads(5, threadFunc, "example thread");
Now the application thread again:
static threadfunc threadFunc(void* arg)
msg = server_waitInputMessage();
server_logDebug("received a message");
The point of interest now is the
Message typedef, which is provided by the framework, and contains the data sent by the client.
Another Example: Still Echoing, But Doing a Little More of Nothing
Now only the application thread is shown. It still doesn't do much, but at least it shows how to access and modify the data sent by the client. (In fact, the way it is shown here is a little insecure, but easy. There are some other methods that access the message contents in a slightly safer way, although as always, with C being C, you can do pretty much everything you want, including shooting yourself in the foot. And you can do it very fast, of course).
threadfunc threadFunc1(void* arg)
uint size, count = 0;
char *bufIn, *bufOut;
Message *msgIn, *msgOut;
msgIn = server_waitInputMessage();
bufIn = server_messageBuffer(msgIn);
size = server_messageSize(msgIn);
msgOut = server_getFreeMessage();
bufOut = server_messageBuffer(msgOut);
server_printf("* message: length=%02d buf=[%.20s]\n", size, bufIn);
memcpy(bufOut, bufIn, size);
if (!(++count % 10))
server_logInfo("%d messages processed now", count);
There are several points of interest here.
server_messageSize methods, that provides access to the raw data sent by the client, and to its size.
server_getFreeMessage method, through which the application thread acquires a new
Message instance that it will use as reply to the client.
server_setMessageSize which sets the size of the reply message.
server_copyConnectionFromMessage which sets the IP address of the client to which the reply will be sent.
server_disposeMessage that is used by the application thread to return an unused message to the framework.
Now What ?
There are several enhancements that can be done to the code as it is.
There are a great number of assert calls sprinkled liberally throughout the code. These calls should be reviewed, and proper error handling should be provided for the cases that make sense.
The central class
ConnectionManager can be made more class-like by getting rid of the global static variables that exist now. Doing so will enable us to write applications having more than one connection manager, that is, serving in two or more IP addresses or service ports.
Writing some unit tests wouldn't hurt either...There's only one class with tests written, because they were absolutely necessary.
One thing of note is that the code uses lots and lot of uints and ushorts, something that may annoy people who likes their ints and shorts raw.
- 7th April, 2010: First version of the article, and covers the version 1.0.0 of the framework
- 14th April, 2010: Version 1.01 - Updated source code
- Correction: Code inside "
assert" calls taken outside
- Included release builds
- Compatibility of includes with the C++ language
- Included C++ examples
- 29th April, 2010: Version 1.02 - Updated source code
- Correction: checks number of connections before creating new connection
- Inclusion of the "generic client" class and example
- Changed "
CreateThread" to "
_beginthreadex" in Windows
- Reorganization of the code in ConnectionManager.c