IO Completion Port Example code walkthrough

1.      Initialize thread handler to invalid value

2.      Call ValidOptions(argc, argv) to make sure command is in correct format

3.      Set control handler function CtrlHandler (DWORD dwEvent)
This function will catch Ctrl-Break and closing event of the console
In case of Ctrl-Break, g_bRestart will be set to true, where g_bEndServer will be set to true and g_hCleanupEvent will be set to true.At the same time, g_hCleanupEvent is signaled by calling WSASetEvent

4.      Initialize the g_hCleanupEvent object by calling WSACreateEvent

5.      Startup Winsock, require version 2+

6.      Line 119, set g_hCleanupEvent to non-signaled state

7.      Line 127, create empty IOCompletionPort, this is similar to a initialization call

8.      Line 145, Call CreateThread, using WorkerThread function, pass the initialized g_hIOCP to the thread. Thread runs immediately after creation

9.      Line 155, create Listening socket. Calls function CreateListenSocket at line 378. See section 1.2 for details

10.   Line 158, call CreateAccpetSocket on line 462. See section 1.3.
This function will create a socket and invoke AccpetEx.

11.   Line 161, block, until g_hCleanupEvent is signaled

12.   When the event is signaled, goto line 166, set flag to true. Then post a NULL competionstatus on line 173 for each thread. When the worker thread receives a NULL completion key, it’ll exit.

13.   On line 179, wait for threads to exit.

14.   Close all thread handlers, close the listening socket, then clean up the global context list. Close the completion port

15.   If there is a restart, loopback to step 6, line 116. Otherwise cleanup the mutex object g_CriticalSection. Close the cleanup event handler, remove the ctrl handler event and call WSACleanup() to finish the server

Section 1.2 CreateListenSocket @ 378

1.      We need to call getaddrinfo() to get a the info for the listening port to bind to. Getaddinfo takes four parameters. The first one is host name or ip, which we are not using in this case. Second one is the port that the server listens to, then the third one is a “hint” for the function where the last one is a linked list to all matching result.
First we fill in the hint structure, starting line 391.
According to MSDN Library:
Setting the AI_PASSIVE flag indicates the caller intends to use the returned socket address structure in a call to the bind function. When the AI_PASSIVE flag is set and nodename is a null pointer, the IP address portion of the of the socket address structure is set to INADDR_ANY for IPv4 addresses and IN6ADDR_ANY_INIT for IPv6 addresses.
When the AI_PASSIVE flag is not set, the returned socket address structure is ready for a call to the connect function for a connection-oriented protocol, or ready for a call to either the connect, sendto, or send functions for a connectionless protocol. If the nodename parameter is a null pointer in this case, the IP address portion of the socket address structure is set to the loopback address.
In simple words, we set AI_PASSIVE flag in order to get a socket for bind to listening rather than connection. Then line 393à395 tells getaddrinfo that we are using IP protocol over internet.

2.      At line 407, we call function CreateSocket at line 308, section 1.4

3.      Line 413, call bind, use the info provided in step 1

4.      Line 420, call listen, set backlog to 5 and starts listening

Section 1.3 CreateAcceptSocket @ 462

1.      This function takes one parameter.
Note: This is only set to true in the first call to CreateAcceptSocket in main function. When this function is called in a worker thread, this parameter is set to false. In other words, the listening socket is added to the io completion port once.
This is done by calling UpdateCompletionPort@787 with the last parameter set to false. Please see section 1.5 for details of this function.

2.      Line 485, calls WSAIoctl. According to MSDN Library:
A call to WSAIoctl function will load the AcceptEx function into memory using WSAIoctl.  The WSAIoctl function is an extension of the ioctlsocket() function that can use overlapped I/O. The function's 3rd through 6th parameters are input and output buffers where we pass the pointer to our AcceptEx function. This is used so that we can call the AcceptEx function directly, rather than refer to the Mswsock.lib library.
In simple words, use the GUID provided to the function, we’ll load the AccpetEx function into the memory and save the pointer to the function in a place in the g_pCtxtListenSocket struct. Which is a PER_SOCKET_CONTEXT. Please see line 51 of IocpServer.h for details.

3.      Line 503, call to CreateSocket@308 to create a overlapped socket, save the result in the listening socket’s global context structure.

4.      Line 512. Invoke AccpetEx by calling the function pointer strored in step2.
Note: AcceptEx has 8 parameters. The details can be found in MSDN library. What we used here are:
The listening socket, The socket for accept created in step3, An receiving buffer created when we setup the first Accept port in main function.
The next 3 parameters are very important.
1. dwReceiveDataLengthàMAX_BUFF_SIZE - (2 * (sizeof(SOCKADDR_STORAGE) + 16)
2. dwLocalAddressLengthà sizeof(SOCKADDR_STORAGE) + 16
3. dwRemoteAddressLengthà sizeof(SOCKADDR_STORAGE) + 16
In other words, the ReceiveDataLength does not include the LocalAddress and RemoteAddress. Thus we have to leave space for them in the buffer. And since LocalAddressLength and RemoteAddressLength must be at least 16 bytes more than the maximum address length for the transport protocol in use, we have to include them in the calculation as well.
The next two are simply a receiving data counter and a overlapped structure as required by the AccpetEx call.

Section 1.4 Create Socket @ 308

1.      Line 313, we call WSASocket to create a new socket. Notice the final parameter in the call, we use WSA_FLAG_OVERLAPPED. This indicates that we are using overlapped socket.

2.      Line 333, we set socket option to disable sending buffer. This is a performance tweak, we can ignore it for now.

Section 1.5 UpdateCompletionPort@787

This is a function used throughout the whole project. It’ll allocate the structures for a socket and add the socket to IOCP. And it’ll add the context structure to a global list of context structures if ask to do so. Now let’s take a look at it.

1.      Line 792, call function CtxtAllocate@863, pass in the socket and given IO_OPERATION. This function will allocate a socket context in the heap. See section 1.6 for CtxtAllocate function.

2.      Line 796, call CreateIoCompletionPort function, it’ll associate the socket with the g_hIOCP and set the lpPerSocketContext as the completion key for that port.
Note 1: This function is quite interesting. It is used for thread pooling in our case. When we call this function the first time in the main function line 127, we pass everthing as null or invalid, this will actually initialize the io completion port. And it’ll return a handler for this port. If we call this function again with that handler and a file handler, the file handler will be added to the io completion port, and a new handler will be returned. This is exactly what’s happening at line 796.
Note 2: The last parameter as stated in MSDN Library, “Maximum number of threads that the operating system can allow to concurrently process I/O completion packets for the I/O completion port.” When set to 0, it’ll allow as many thread as avaible.

3.      Line 809, if asked to, the current context will be added to a global list. This is done by calling function CtxtListAddTo@909, please see section 1.7 for this function.

4.      When complete, the new socket context is returned.

Section 1.6 CtxtAllocate@863

This is rather a “long and boring” function.

1.      At line 867, it’ll get a lock on the global critical section, then calls the macro xmalloc to allocate memory for this socket’s context in the heap.

2.      At line 871, creates a IOContext for this Socket context. This IOContext is actually the very first element in a linked list in this Socket Context.

3.      Returns a initialized SocketContext

Section 1.7 CtxtListAddTo@909

1.      Add a SocketContext to the global linked list.

Section 2.0 Worker thread@529

This is the core function or the actual running function for the server.

1.      Line 531, the g_hIOCP has been passed as a parameter to the threads when they are created. Now we cast the thread context to a local variable, hIOCP.

2.      Line 545, enter a loop

3.      Line 550, call GetQueuedCompletionStatus to get a completion key from the completion port. This function will block infinitely until there is a complete key.

4.      Line 560 & 569, check if thread should exit in case the global restart flag is set

5.      Line 557, cast the received overlapped structure to a IOContext.

6.      Line 583, first we check if this is an Accpet operation.
When calling GetQueuedCompletionStatus function, we get a return value and a data size. A false of this return value or 0 in data size means the client has dropped it’s connection. In this case we call CloseClient@822 to clean up the client abortively and continue to server the next completion port. See section 2.1 for CloseClient

7.      Line 599, enter the FSM of the thread see next section

Section 2.0a WorkerThread @529 à @600 FSM= ClientIoAccept

1.      Line 611, use setsockopt to copy state from the listening socket to the accept socket.

2.      Line 629, call UpdateCompletionPort@787 (Section 1.5) to associate the newly updated accept socket.

3.      Line 643, when dwIoSize is not 0, that means the client has sent data upon the connection is made. Goto step 4. If dwIoSize is 0, goto step 5.

4.      We set the operation to ClientIoWrite. Then on line 648, we copy the input buffer of socket context to our accept socket context. In other words, we copy the state from the listening socket and data from the completion key to form a new accept socket context. Line 651, we post an WSASend then goto step 6.

5.      Line 671, if there was no data coming in when the socket was accepted. Post a read operation to read the incoming data

6.      Line 691, post an new Accept call using CreateAcceptSocket@462 (Section 1.3)

Section 2.0b WorkerThread @529 à @699 FSM= ClientIoRead

1.      Post a write operation when the read completes

Section 2.0c WorkerThread @529 à @724 FSM=ClientIoWrite

1.      Line 731, the sent size is returned by the function of GetQueuedCompletionStatus. dwIoSize.

2.      Line 733, if the sent bytes is less than the total bytes in the buffer, then post another sending. Goto step 3, otherwise, step 4.

3.      Line 739, move the pointer to the position in the buffer that’s not been sent, also update the sent data on line 740.

4.      If the sending are complete, post a read operation on line 763.

Section 2.1 CloseClient@822

1.      Line 830, when we asked to close the socket abortively (i.e. Close without sending all the pending data, which is common when the client closes the connection), we call setsocketopt to set linger. Then close both the accept socket and all the other socket opened during this operation.

2.      Line 849, calls CtxtListDeleteFrom@945 to remove the context from the global list.
Note: CloseClient calls EnterCriticalSection once, and CtxtListDeleteFrom@945 calls EnterCriticalSection once more. This will not result in a dead lock. According to MSDN Library, This is a built in feature of this function, a thread that has the ownership of the critical section will not block if this function is called.