The following source was built using Visual Studio 6.0 SP5 and Visual Studio .Net. You need to have a version of the Microsoft Platform SDK installed
When a server has to deal with lots of short lived client connections it's advisable to use the Microsoft extension function for WinSock,
AcceptEx(), to accept connections. Creating a socket is a relatively "expensive" operation and by using
AcceptEx() you can create the socket before the connection occurs rather than as it occurs, thus speeding the establishment of the connection. What's more,
AcceptEx() can perform an initial data read at the same time as doing the connection establishment which means you can accept a connection and retrieve data with a single call.
In this article we develop a socket server class that uses
AcceptEx() and the related Microsoft extension functions. The resulting server class is similar to the one we developed in the previous article in that it does all of the hard work for you and provides a simple way to develop powerful and scalable socket servers.
Documentation bug, or undocumented behavior?
The documentation for AcceptEx() states:
"When this operation is successfully completed,
sAcceptHandle can be passed, but to the following functions only:
WSASend are conspicuous by their absence, and so it
DisconnectEx. This article assumes that this is due to a documentation bug and that
AcceptEx is intended to operate with these functions. Either way, we're into undocumented behaviour, so if that's important to you then you may not wish to do things this way. What we've found is that it works on the platforms that we need it to work on.
Using Microsoft extension functions with WinSock
The Windows Sockets 2 specification defines an extension mechanism that allows Windows Sockets service providers to expose advanced transport functionality to application programmers. Microsoft provides several of these extension functions but by using them you are limiting your software to running on a Windows Sockets provider that supports these functions. Generally this isn't a problem...
Several of the extension functions have been available since WinSock 1.1 and are exported from MSWsock.dll, however it's not advisable to link directly to this dll as this ties you to the Microsoft WinSock provider. A provider neutral way of accessing these extension functions is to load them dynamically via
WSAIoctl using the
SIO_GET_EXTENSION_FUNCTION_POINTER op code. This should, theoretically, allow you to access these functions from any provider that supports them...
With WindowsXP Microsoft has added several new WinSock extension functions and these are only available via the
WSAIoctl route so, in the interest of consistency and portability we'll access all of the extension functions using a simple wrapper class,
CMSWinSock, which wraps the required calls to
AcceptEx() can reuse sockets that have been prepared for reuse in the appropriate way. WindowsXP provides
DisconnectEX() as a way to prepare a socket handle for reuse. Prior to WindowsXP the only function that could prepare a socket for reuse was
TransmitFile() could be used to prepare a socket for reuse without actually having to transmit a file... To make the code easier to understand
CMSWinSock provides a function to disconnect a socket for reuse.
DisconnectSocketForReuse() will call
DisconnectEx() if it's available and otherwise call
TransmitFile() with the appropriate arguments to simply reuse the socket.
Accepting connections with AcceptEx()
Our previous servers have used a blocking loop on
WSAAccept() to accept connections. When a connection occurs, a socket is created and returned from the call to
WSAAccept(), our accepting thread then loops around to call
WSAAccept() again for the next connection. Not only is creating a socket a time consuming operation but the design means that all connection establishment must go through a single piece of code in a single thread.
AcceptEx() uses a different model, you create your sockets first, then "post" accept requests onto the listening socket and when these requests complete you receieve IO completion packets on the associated IO Completion Port. I've no idea how this works under the hood, but at the very least, the use of an IO Completion Port for notification allows us to multi thread the work that we need to do in relation to establishing a connection.
So, how many sockets do we need to create in advance and how do we know when we need to create more? The number of sockets that you need to create in advance will depend on the number of connections that your server has to handle and as such is a configurable parameter. You don't want to create too many sockets as this wastes server resources, but if you create too few then your server will run slower, or refuse connections... We keep a track of the number of sockets that we have created and as accepts complete we move the sockets from a "pending accept" list onto an "active" list. In this way we can monitor when we need to create more sockets and issue more calls to
AcceptEx(). However, if we are expecting a client to immediately send data after it connects then the accept doesn't complete until at least one byte arrives on the connection. A malicious client could thus attempt a denial of service attack on our server by opening connections and not sending any data. This would eventually use up all of the accepts we have posted and cause our server to start to use the listen backlog queue. Eventually the server will fill the listen backlog queue and begin to reject connections attempts. To avoid this situation we can register for notification when a connection attempt occurs and there are no outstanding accepts available. When this happens the backlog queue will have queued the connection request and we can post more calls to
AcceptEx() so that the connection will be accepted. We use
WSAEventSelect() to register for
FD_ACCEPT events - these are reported by an event being set. We can then structure our accept loop something like this:
WSAEventSelect(m_listeningSocket, m_acceptEvent.GetEvent(), FD_ACCEPT);
for (size_t i = 0; i < numAcceptsToPost; ++i)
handlesToWaitFor = m_postMoreAcceptsEvent.GetEvent();
handlesToWaitFor = m_acceptEvent.GetEvent();
waitResult = ::WaitForMultipleObjects(2, handlesToWaitFor, false, INFINITE);
if (waitResult != WAIT_OBJECT_0 &&
waitResult != WAIT_OBJECT_0 + 1)
OnError(_T("CSocketServerEx::Run() - WaitForMultipleObjects: ")
if (waitResult == WAIT_OBJECT_0 + 1)
while (waitResult == WAIT_OBJECT_0 || waitResult == WAIT_OBJECT_0 + 1);
We've only moved the denial of service attack from causing our server to refuse connections to causing our server to run out of resources by accepting an infinite number of malicious connections. To address this problem we need to be able to determine if a socket that is pending an accept completion has had the connection established and is now waiting for data to arrive, and if it is, how long it's been waiting... For this we use
getsockopt() with the
SO_CONNECT_TIME option (available from Windows NT 4.0) . This returns -1 if the socket is not connected or the number of seconds that it has been connected. If, when we are informed that we need to post more accepts, we post the accepts and then check all of the pending accepts to see how long they have been connected and waiting for data then we can forcibly disconnect sockets that are "taking too long" (a configurable parameter) to send data after connecting...
We now have a server that will post a configurable number of accepts when it first starts listening and, in normal operation, will post more accepts as connections complete. If we get to a point where we have no accepts pending and a connection occurs then we are informed so that we can post more accepts and check to see if any connected sockets have been waiting for data for longer than our configurable timeout.
Accepting and reading data
AcceptEx() you must always pass a buffer to store the local and remote addresses of the resulting connection. For servers that receive data before they send data, such as web servers, for example, you can include space in this buffer for the first batch of data that is read from the connection. As we pointed out above, the accept doesn't complete until at least one byte arrives. The code could look something like this:
Socket *pSocket = AllocateSocket();
CIOBuffer *pBuffer = Allocate();
const size_t sizeOfAddress = GetAddressSize() + 16;
const size_t sizeOfAddresses = 2 * sizeOfAddress;
DWORD bytesReceived = 0;
pBuffer->GetSize() - sizeOfAddresses,
const DWORD lastError = ::WSAGetLastError();
if (ERROR_IO_PENDING != lastError)
Output(_T("CSocketServerEx::Accept() - AcceptEx: ")
Note that we call up to our derived class to provide details of the size of the
sockaddr that we need to make space for, though a default implementation simply returns
sizeof(SOCKADDR_IN). When the accept completes the socket retrieved from the completion key by our worker thread is the listening socket, since that's the device that's associated with the IO Completion Port and generating the completion packet for the accept. We require the accepted socket as well, so we store that in the IO buffer's user data slot. It's a bit crufty in the worker thread as for all other completion packets the completion key is a pointer to a
Socket, but for accepts it's not - a special case that may get refactored away if I get the time... Note that although the expected code path is for
AcceptEx() to return false and for
WSAGetLastError() to return
ERROR_IO_PENDING, we handle the case where the accept completes synchronously by posting to the completion port ourselves, and we do so with the same semantics as the asynchronously generated packet (ie listening socket as completion key). I've never actually been able to get my test harness to generate this situation...
Accepting without reading data
For server's that don't receieve data before sending we can still use
AcceptEx(), we just specify a data buffer size of 0 and no read occurs, the accept completes as soon as the connection is established.
When an accept completes and, if appropriate, data arrives, an IO completion packet is posted and our worker threads complete the accept by setting the socket options on accepted socket to match those of the listening socket (normally
WSAAccept() would do this for us, but
AcceptEx() makes us do it ourselves using
SO_UPDATE_ACCEPT_CONTEXT). We then move the socket from our pending list to our active list, extract the local and remote addresses from the data buffer that we passed to
AcceptEx() and notify the derived class of a new connection and, if appropriate, new data.
The derived class interface
The derived class is almost as straight forward as the one in the previous article except that you can override the creation of the accepted socket and you can override the amount of space you need to reserve for the local and remote addresses (in case you're using a protocol other than TCP/IP).
The example server is another simple echo server, I know what I said about echo servers, but in this case it's an ok example :). The server contains two instances of the socket server class and listens on 5001 and 5002. On 5001 it performs an accept that requires data to arrive before it will complete and on 5002 it performs an accept that returns straight after the connection is established. Note that the server shows how you can package multiple socket servers in the same executable (perhaps one day I'll optimise the class so that all servers are handled by a single pool of IO threads...).
To test the server, telnet to localhost 5001/2 and type some data. If you telnet to 5001 a few times and dont type any data then you should be able to see the
FD_ACCEPT notification and connection timeout checking in operation. As always, the ServerShutdown program lets you pause, resume and shutdown the server.
- 3rd Jun 2002 - Initial revision.
- 26th June 2002 - Removed call to
ReuseAddress() during the creation of the listening socket as it not required - Thanks to Alun Jones for pointing this out to me.
- 28th June 2002 - Adjusted how we handle socket closure. We now issue async disconnects.
- 30th June 2002 - Removed the requirement for users to subclass the socket server's worker thread class. All of the work can now be done by simply subclassing the socket server class.
- 15th July 2002 - Socket closure notifications now occur when the server shuts down whilst there are active connections.
SocketServer can now be set to ensure read and write packet sequences
- 19th July 2002 - Merged with latest Socket Server code - still need to do the refactoring job to remove the duplication. Tweaked the
AcceptEx repost logic so that the server runs 'smoother'. Updated the article to indicate the undocumented nature of the example code.