Click here to Skip to main content
Click here to Skip to main content

A reusable, high performance, socket server class - Part 3

By , 18 Jul 2002
Rate this:
Please Sign up or sign in to vote.

The following source was built using Visual Studio 6.0 SP5 and Visual Studio .Net. You need to have a version of the Microsoft Platform SDK installed

Overview

When a server has to deal with lots of short lived client connections it's advisable to use the Microsoft extension function for WinSock, AcceptEx(), to accept connections. Creating a socket is a relatively "expensive" operation and by using AcceptEx() you can create the socket before the connection occurs rather than as it occurs, thus speeding the establishment of the connection. What's more, AcceptEx() can perform an initial data read at the same time as doing the connection establishment which means you can accept a connection and retrieve data with a single call.

In this article we develop a socket server class that uses AcceptEx() and the related Microsoft extension functions. The resulting server class is similar to the one we developed in the previous article in that it does all of the hard work for you and provides a simple way to develop powerful and scalable socket servers.

Documentation bug, or undocumented behavior?

The documentation for AcceptEx() states:

"When this operation is successfully completed, sAcceptHandle can be passed, but to the following functions only:
ReadFile
WriteFile
send
recv
TransmitFile
closesocket"

Notice that WSARecv and WSASend are conspicuous by their absence, and so it DisconnectEx. This article assumes that this is due to a documentation bug and that AcceptEx is intended to operate with these functions. Either way, we're into undocumented behaviour, so if that's important to you then you may not wish to do things this way. What we've found is that it works on the platforms that we need it to work on.

Using Microsoft extension functions with WinSock

The Windows Sockets 2 specification defines an extension mechanism that allows Windows Sockets service providers to expose advanced transport functionality to application programmers. Microsoft provides several of these extension functions but by using them you are limiting your software to running on a Windows Sockets provider that supports these functions. Generally this isn't a problem...

Several of the extension functions have been available since WinSock 1.1 and are exported from MSWsock.dll, however it's not advisable to link directly to this dll as this ties you to the Microsoft WinSock provider. A provider neutral way of accessing these extension functions is to load them dynamically via WSAIoctl using the SIO_GET_EXTENSION_FUNCTION_POINTER op code. This should, theoretically, allow you to access these functions from any provider that supports them...

With WindowsXP Microsoft has added several new WinSock extension functions and these are only available via the WSAIoctl route so, in the interest of consistency and portability we'll access all of the extension functions using a simple wrapper class, CMSWinSock, which wraps the required calls to WSAIoctl.

AcceptEx() can reuse sockets that have been prepared for reuse in the appropriate way. WindowsXP provides DisconnectEX() as a way to prepare a socket handle for reuse. Prior to WindowsXP the only function that could prepare a socket for reuse was TransmitFile(). Fortunately, TransmitFile() could be used to prepare a socket for reuse without actually having to transmit a file... To make the code easier to understand CMSWinSock provides a function to disconnect a socket for reuse. DisconnectSocketForReuse() will call DisconnectEx() if it's available and otherwise call TransmitFile() with the appropriate arguments to simply reuse the socket.

Accepting connections with AcceptEx()

Our previous servers have used a blocking loop on WSAAccept() to accept connections. When a connection occurs, a socket is created and returned from the call to WSAAccept(), our accepting thread then loops around to call WSAAccept() again for the next connection. Not only is creating a socket a time consuming operation but the design means that all connection establishment must go through a single piece of code in a single thread. AcceptEx() uses a different model, you create your sockets first, then "post" accept requests onto the listening socket and when these requests complete you receieve IO completion packets on the associated IO Completion Port. I've no idea how this works under the hood, but at the very least, the use of an IO Completion Port for notification allows us to multi thread the work that we need to do in relation to establishing a connection.

So, how many sockets do we need to create in advance and how do we know when we need to create more? The number of sockets that you need to create in advance will depend on the number of connections that your server has to handle and as such is a configurable parameter. You don't want to create too many sockets as this wastes server resources, but if you create too few then your server will run slower, or refuse connections... We keep a track of the number of sockets that we have created and as accepts complete we move the sockets from a "pending accept" list onto an "active" list. In this way we can monitor when we need to create more sockets and issue more calls to AcceptEx(). However, if we are expecting a client to immediately send data after it connects then the accept doesn't complete until at least one byte arrives on the connection. A malicious client could thus attempt a denial of service attack on our server by opening connections and not sending any data. This would eventually use up all of the accepts we have posted and cause our server to start to use the listen backlog queue. Eventually the server will fill the listen backlog queue and begin to reject connections attempts. To avoid this situation we can register for notification when a connection attempt occurs and there are no outstanding accepts available. When this happens the backlog queue will have queued the connection request and we can post more calls to AcceptEx() so that the connection will be accepted. We use WSAEventSelect() to register for FD_ACCEPT events - these are reported by an event being set. We can then structure our accept loop something like this:

   WSAEventSelect(m_listeningSocket, m_acceptEvent.GetEvent(), FD_ACCEPT);

   do
   {
      for (size_t i = 0; i < numAcceptsToPost; ++i)
      {
         Accept();
      }

      m_postMoreAcceptsEvent.Reset();
      m_acceptEvent.Reset();

      HANDLE handlesToWaitFor[2];

      handlesToWaitFor[0] = m_postMoreAcceptsEvent.GetEvent();
      handlesToWaitFor[1] = m_acceptEvent.GetEvent();

      waitResult = ::WaitForMultipleObjects(2, handlesToWaitFor, false, INFINITE);

      if (waitResult != WAIT_OBJECT_0 &&
          waitResult != WAIT_OBJECT_0 + 1)
      {
         OnError(_T("CSocketServerEx::Run() - WaitForMultipleObjects: ") 
                 + GetLastErrorMessage(::GetLastError()));
      }

      if (waitResult == WAIT_OBJECT_0 + 1)
      {
         Output(_T("Accept..."));
      }
   }
   while (waitResult == WAIT_OBJECT_0 || waitResult == WAIT_OBJECT_0 + 1);

We've only moved the denial of service attack from causing our server to refuse connections to causing our server to run out of resources by accepting an infinite number of malicious connections. To address this problem we need to be able to determine if a socket that is pending an accept completion has had the connection established and is now waiting for data to arrive, and if it is, how long it's been waiting... For this we use getsockopt() with the SO_CONNECT_TIME option (available from Windows NT 4.0) . This returns -1 if the socket is not connected or the number of seconds that it has been connected. If, when we are informed that we need to post more accepts, we post the accepts and then check all of the pending accepts to see how long they have been connected and waiting for data then we can forcibly disconnect sockets that are "taking too long" (a configurable parameter) to send data after connecting...

We now have a server that will post a configurable number of accepts when it first starts listening and, in normal operation, will post more accepts as connections complete. If we get to a point where we have no accepts pending and a connection occurs then we are informed so that we can post more accepts and check to see if any connected sockets have been waiting for data for longer than our configurable timeout.

Accepting and reading data

When calling AcceptEx() you must always pass a buffer to store the local and remote addresses of the resulting connection. For servers that receive data before they send data, such as web servers, for example, you can include space in this buffer for the first batch of data that is read from the connection. As we pointed out above, the accept doesn't complete until at least one byte arrives. The code could look something like this:

   void CSocketServerEx::Accept()
   {
      Socket *pSocket = AllocateSocket();

      {
         CCriticalSection::Owner lock(m_listManipulationSection);

         m_pendingList.PushNode(pSocket);
      }

      // allocate a buffer
      CIOBuffer *pBuffer = Allocate();

      pBuffer->SetOperation(IO_Accept_Completed);

      pBuffer->SetUserPtr(pSocket);

      const size_t sizeOfAddress = GetAddressSize() + 16;
      const size_t sizeOfAddresses = 2 * sizeOfAddress;

      DWORD bytesReceived = 0;

      if (!CMSWinSock::AcceptEx(
         m_listeningSocket,
         pSocket->m_socket,
         reinterpret_cast<void*>(const_cast<BYTE*>(pBuffer->GetBuffer())),
         pBuffer->GetSize() - sizeOfAddresses,
         sizeOfAddress,
         sizeOfAddress,
         &bytesReceived,
         pBuffer->GetAsOverlapped()))
      {
         const DWORD lastError = ::WSAGetLastError();

         if (ERROR_IO_PENDING != lastError)
         {
            Output(_T("CSocketServerEx::Accept() - AcceptEx: ") 
                    + GetLastErrorMessage(lastError));

            pSocket->Close();
            pSocket->Release();  
            pBuffer->Release();
         }
      }
      else
      {
         // Accept completed synchronously. We need to marshal the data received 
         // over to the worker thread ourselves...

         m_iocp.PostStatus((ULONG_PTR)m_listeningSocket, bytesReceived, 
                           pBuffer->GetAsOverlapped());
      }
   }

Note that we call up to our derived class to provide details of the size of the sockaddr that we need to make space for, though a default implementation simply returns sizeof(SOCKADDR_IN). When the accept completes the socket retrieved from the completion key by our worker thread is the listening socket, since that's the device that's associated with the IO Completion Port and generating the completion packet for the accept. We require the accepted socket as well, so we store that in the IO buffer's user data slot. It's a bit crufty in the worker thread as for all other completion packets the completion key is a pointer to a Socket, but for accepts it's not - a special case that may get refactored away if I get the time... Note that although the expected code path is for AcceptEx() to return false and for WSAGetLastError() to return ERROR_IO_PENDING, we handle the case where the accept completes synchronously by posting to the completion port ourselves, and we do so with the same semantics as the asynchronously generated packet (ie listening socket as completion key). I've never actually been able to get my test harness to generate this situation...

Accepting without reading data

For server's that don't receieve data before sending we can still use AcceptEx(), we just specify a data buffer size of 0 and no read occurs, the accept completes as soon as the connection is established.

Accept completion

When an accept completes and, if appropriate, data arrives, an IO completion packet is posted and our worker threads complete the accept by setting the socket options on accepted socket to match those of the listening socket (normally WSAAccept() would do this for us, but AcceptEx() makes us do it ourselves using setsockopt() and SO_UPDATE_ACCEPT_CONTEXT). We then move the socket from our pending list to our active list, extract the local and remote addresses from the data buffer that we passed to AcceptEx() and notify the derived class of a new connection and, if appropriate, new data.

The derived class interface

The derived class is almost as straight forward as the one in the previous article except that you can override the creation of the accepted socket and you can override the amount of space you need to reserve for the local and remote addresses (in case you're using a protocol other than TCP/IP).

The example...

The example server is another simple echo server, I know what I said about echo servers, but in this case it's an ok example Smile | :) . The server contains two instances of the socket server class and listens on 5001 and 5002. On 5001 it performs an accept that requires data to arrive before it will complete and on 5002 it performs an accept that returns straight after the connection is established. Note that the server shows how you can package multiple socket servers in the same executable (perhaps one day I'll optimise the class so that all servers are handled by a single pool of IO threads...).

To test the server, telnet to localhost 5001/2 and type some data. If you telnet to 5001 a few times and dont type any data then you should be able to see the FD_ACCEPT notification and connection timeout checking in operation. As always, the ServerShutdown program lets you pause, resume and shutdown the server.

Revision history

  • 3rd Jun 2002 - Initial revision.
  • 26th June 2002 - Removed call to ReuseAddress() during the creation of the listening socket as it not required - Thanks to Alun Jones for pointing this out to me.
  • 28th June 2002 - Adjusted how we handle socket closure. We now issue async disconnects.
  • 30th June 2002 - Removed the requirement for users to subclass the socket server's worker thread class. All of the work can now be done by simply subclassing the socket server class.
  • 15th July 2002 - Socket closure notifications now occur when the server shuts down whilst there are active connections. SocketServer can now be set to ensure read and write packet sequences
  • 19th July 2002 - Merged with latest Socket Server code - still need to do the refactoring job to remove the duplication. Tweaked the AcceptEx repost logic so that the server runs 'smoother'. Updated the article to indicate the undocumented nature of the example code.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

Len Holgate
Software Developer (Senior) JetByte Limited
United Kingdom United Kingdom
Len has been programming for over 30 years, having first started with a Sinclair ZX-80. Now he runs his own consulting company, JetByte Limited and has a technical blog here.
 
JetByte provides contract programming and consultancy services. We can provide experience in COM, Corba, C++, Windows NT and UNIX. Our speciality is the design and implementation of systems but we are happy to work with you throughout the entire project life-cycle. We are happy to quote for fixed price work, or, if required, can work for an hourly rate.
 
We are based in London, England, but, thanks to the Internet, we can work 'virtually' anywhere...
 
Please note that many of the articles here may have updated code available on Len's blog and that the IOCP socket server framework is also available in a licensed, much improved and fully supported version, see here for details.

Comments and Discussions

 
GeneralMy vote of 3 Pinmemberdavyas4-Oct-13 2:22 
GeneralRe: My vote of 3 PinmemberLen Holgate4-Oct-13 2:46 
General[Message Deleted] Pinmemberit.ragester2-Apr-09 21:45 
QuestionQuestion on WinAPI ConnectEx PinmemberSchiener12-Dec-08 1:29 
AnswerRe: Question on WinAPI ConnectEx PinmemberLen Holgate12-Dec-08 3:22 
GeneralRe: Question on WinAPI ConnectEx PinmemberSchiener12-Dec-08 3:52 
QuestionRe: Question on WinAPI ConnectEx PinmemberSchiener18-Dec-08 9:41 
AnswerRe: Question on WinAPI ConnectEx PinmemberLen Holgate18-Dec-08 9:50 
Answermsdn give a detail for the problem! [modified] Pinmemberpenghong16-Apr-09 23:55 
GeneralWrong usage AcceptEx PinmemberAlezis5-Apr-08 6:47 
sorry for my bad english.
When AcceptEx function completed successfully and a value of TRUE is returned, AcceptEx also post to the completion port message.
Need :
 
if (!CMSWinSock::AcceptEx(
m_listeningSocket,
pSocket->m_socket,
reinterpret_cast(const_cast(pBuffer->GetBuffer())),
bufferSize,
sizeOfAddress,
sizeOfAddress,
&bytesReceived,
pBuffer))
{
const DWORD lastError = ::WSAGetLastError();
 
if (ERROR_IO_PENDING != lastError)
{
Output(_T("CSocketServerEx::Accept() - AcceptEx: ") + GetLastErrorMessage(lastError));
 
pSocket->Release();
pBuffer->Release();
}
}

/*
// this code waste
else
{
// Accept completed synchronously. We need to marshal the data recieved over to the
// worker thread ourselves...
 
m_iocp.PostStatus((ULONG_PTR)m_listeningSocket, bytesReceived, pBuffer);
}
*/

GeneralRe: Wrong usage AcceptEx PinmemberLen Holgate5-Apr-08 8:25 
Questionwhy doesnt popNode? Pinmemberminkun14-Oct-07 23:22 
AnswerRe: why doesnt popNode? PinmemberLen Holgate15-Oct-07 11:04 
GeneralDialog or MFC PinmemberBeerFizz13-Nov-06 12:32 
GeneralRe: Dialog or MFC PinmemberLen Holgate13-Nov-06 19:34 
GeneralRe: Dialog or MFC PinmemberBeerFizz14-Nov-06 3:40 
GeneralRe: Dialog or MFC PinmemberLen Holgate14-Nov-06 6:22 
GeneralRe: Dialog or MFC PinmemberBeerFizz14-Nov-06 11:49 
AnswerRe: Dialog or MFC PinmemberLen Holgate15-Nov-06 6:53 
GeneralRe: Dialog or MFC PinmemberLen Holgate16-Nov-06 19:53 
QuestionLocalhost plus... PinmemberBeerFizz26-Oct-06 9:07 
AnswerRe: Localhost plus... PinmemberLen Holgate27-Oct-06 1:41 
GeneralRe: Localhost plus... PinmemberBeerFizz27-Oct-06 2:03 
GeneralRe: Localhost plus... PinmemberLen Holgate27-Oct-06 2:04 
GeneralTimeout and disconnect problems PinmemberZelius10-Oct-06 5:51 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web04 | 2.8.140415.2 | Last Updated 19 Jul 2002
Article Copyright 2002 by Len Holgate
Everything else Copyright © CodeProject, 1999-2014
Terms of Use
Layout: fixed | fluid