Initialize thread handler to invalid
Call ValidOptions(argc, argv) to
make sure command is in correct format
control handler function CtrlHandler (DWORD dwEvent)
This function will catch
Ctrl-Break and closing event of the console
In case of Ctrl-Break, g_bRestart
will be set to true, where g_bEndServer will be set to true and g_hCleanupEvent
will be set to true.At the same time, g_hCleanupEvent is signaled by calling
Initialize the g_hCleanupEvent
object by calling WSACreateEvent
Startup Winsock, require version
Line 119, set g_hCleanupEvent to
Line 127, create empty
IOCompletionPort, this is similar to a initialization
Line 145, Call CreateThread, using
WorkerThread function, pass the initialized g_hIOCP to the thread. Thread runs
immediately after creation
Line 155, create Listening socket.
Calls function CreateListenSocket at line 378. See section 1.2 for
10. Line 158, call CreateAccpetSocket
on line 462. See section 1.3.
This function will create a socket and invoke
11. Line 161, block, until g_hCleanupEvent is
12. When the event is signaled, goto line 166, set flag to
true. Then post a NULL competionstatus on line 173 for each thread. When the
worker thread receives a NULL completion key, it’ll
13. On line 179, wait for threads to
14. Close all thread handlers, close the listening socket,
then clean up the global context list. Close the completion
15. If there is a restart, loopback to step 6, line 116.
Otherwise cleanup the mutex object g_CriticalSection. Close the cleanup event
handler, remove the ctrl handler event and call WSACleanup() to finish the
Section 1.2 CreateListenSocket @
need to call getaddrinfo() to get a the info for the listening port to bind to.
Getaddinfo takes four parameters. The first one is host name or ip, which we are
not using in this case. Second one is the port that the server listens to, then
the third one is a “hint” for the function where the last one is a linked list
to all matching result.
First we fill in the hint structure, starting line
According to MSDN Library:
Setting the AI_PASSIVE
flag indicates the caller intends to use the returned socket address structure
in a call to the bind
function. When the AI_PASSIVE flag is set and nodename is a null pointer,
the IP address portion of the of the socket address structure is set to
INADDR_ANY for IPv4 addresses and IN6ADDR_ANY_INIT for IPv6 addresses.
the AI_PASSIVE flag is not set, the returned socket address structure is ready
for a call to the connect
function for a connection-oriented protocol, or ready for a call to either the
connect, sendto, or
send functions for a connectionless protocol. If the
nodename parameter is a null pointer in this case, the IP address portion
of the socket address structure is set to the loopback
In simple words, we set AI_PASSIVE flag in order to get a socket for bind
to listening rather than connection. Then line 393à395 tells getaddrinfo that we are using
IP protocol over internet.
At line 407, we call function
CreateSocket at line 308, section 1.4
Line 413, call bind, use the info
provided in step 1
Line 420, call listen, set backlog
to 5 and starts listening
Section 1.3 CreateAcceptSocket @
function takes one parameter.
Note: This is only set to true in the first
call to CreateAcceptSocket in main function. When this function is called in a
worker thread, this parameter is set to false. In other words, the listening
socket is added to the io completion port once.
This is done by calling
UpdateCompletionPort@787 with the last parameter set to false. Please see
section 1.5 for details of this function.
485, calls WSAIoctl. According to MSDN Library:
A call to WSAIoctl function
will load the AcceptEx function into memory using WSAIoctl. The WSAIoctl
function is an extension of the ioctlsocket() function that can use overlapped
I/O. The function's 3rd through 6th parameters are input and output buffers
where we pass the pointer to our AcceptEx function. This is used so that we can
call the AcceptEx function directly, rather than refer to the Mswsock.lib
In simple words, use the GUID provided to the function, we’ll load
the AccpetEx function into the memory and save the pointer to the function in a
place in the g_pCtxtListenSocket struct. Which is a PER_SOCKET_CONTEXT. Please
see line 51 of IocpServer.h for details.
Line 503, call to CreateSocket@308
to create a overlapped socket, save the result in the listening socket’s global
Line 512. Invoke AccpetEx by calling the function pointer
strored in step2.
Note: AcceptEx has 8 parameters. The details can be found
in MSDN library. What we used here are:
The listening socket, The socket for
accept created in step3, An receiving buffer created when we setup the first
Accept port in main function.
The next 3 parameters are very important.
dwReceiveDataLengthàMAX_BUFF_SIZE - (2 *
(sizeof(SOCKADDR_STORAGE) + 16)
2. dwLocalAddressLengthà sizeof(SOCKADDR_STORAGE) + 16
dwRemoteAddressLengthà sizeof(SOCKADDR_STORAGE) + <st1:chmetcnv w:st="on" TCSC="0"
NumberType="1" Negative="False" HasSpace="False" SourceValue="16"
In other words, the ReceiveDataLength does not include the
LocalAddress and RemoteAddress. Thus we have to leave space for them in the
buffer. And since LocalAddressLength and RemoteAddressLength must be at least 16
bytes more than the maximum address length for the transport protocol in use, we
have to include them in the calculation as well.
The next two are simply a
receiving data counter and a overlapped structure as required by the AccpetEx
Section 1.4 Create Socket @
Line 313, we call WSASocket to
create a new socket. Notice the final parameter in the call, we use
WSA_FLAG_OVERLAPPED. This indicates that we are using overlapped socket.
Line 333, we set socket option to
disable sending buffer. This is a performance tweak, we can ignore it for
This is a function used throughout the whole
project. It’ll allocate the structures for a socket and add the socket to IOCP.
And it’ll add the context structure to a global list of context structures if
ask to do so. Now let’s take a look at it.
Line 792, call function
CtxtAllocate@863, pass in the socket and given IO_OPERATION. This function will
allocate a socket context in the heap. See section 1.6 for CtxtAllocate
796, call CreateIoCompletionPort function, it’ll associate the socket with the
g_hIOCP and set the lpPerSocketContext as the completion key for that
Note 1: This function is quite interesting. It is used for thread
pooling in our case. When we call this function the first time in the main
function line 127, we pass everthing as null or invalid, this will actually
initialize the io completion port. And it’ll return a handler for this port. If
we call this function again with that handler and a file handler, the file
handler will be added to the io completion port, and a new handler will be
returned. This is exactly what’s happening at line 796.
Note 2: The last
parameter as stated in MSDN Library, “Maximum number of threads that the
operating system can allow to concurrently process I/O completion packets for
the I/O completion port.” When set to 0, it’ll allow as many thread as
Line 809, if asked to, the current
context will be added to a global list. This is done by calling function
CtxtListAddTo@909, please see section 1.7 for this
When complete, the new socket
context is returned.
This is rather a “long and boring” function.
At line 867, it’ll get a lock on the
global critical section, then calls the macro xmalloc to allocate memory for
this socket’s context in the heap.
At line 871, creates a IOContext for
this Socket context. This IOContext is actually the very first element in a
linked list in this Socket Context.
Returns a initialized
Add a SocketContext to the
global linked list.
Section 2.0 Worker
This is the core function or the actual running
function for the server.
Line 531, the g_hIOCP has been
passed as a parameter to the threads when they are created. Now we cast the
thread context to a local variable, hIOCP.
Line 545, enter a
Line 550, call
GetQueuedCompletionStatus to get a completion key from the completion port. This
function will block infinitely until there is a complete
Line 560 & 569, check if thread
should exit in case the global restart flag is set
Line 557, cast the received
overlapped structure to a IOContext.
583, first we check if this is an Accpet operation.
GetQueuedCompletionStatus function, we get a return value and a data size. A
false of this return value or <st1:chmetcnv w:st="on" TCSC="0" NumberType="1"
Negative="False" HasSpace="True" SourceValue="0" UnitName="in">0 in data size
means the client has dropped it’s connection. In this case we call
CloseClient@822 to clean up the client abortively and continue to server the
next completion port. See section 2.1 for CloseClient
Line 599, enter the FSM of the
thread see next section
Section <st1:chmetcnv w:st="on" TCSC="0"
NumberType="1" Negative="False" HasSpace="False" SourceValue="2"
UnitName="a">2.0a WorkerThread @529 à @600 FSM= ClientIoAccept
Line 611, use setsockopt to copy
state from the listening socket to the accept socket.
Line 629, call
UpdateCompletionPort@787 (Section 1.5) to associate the newly updated accept
Line 643, when dwIoSize is not 0,
that means the client has sent data upon the connection is made. Goto step 4. If
dwIoSize is 0, goto step 5.
We set the operation to
ClientIoWrite. Then on line 648, we copy the input buffer of socket context to
our accept socket context. In other words, we copy the state from the listening
socket and data from the completion key to form a new accept socket context.
Line 651, we post an WSASend then goto step 6.
Line 671, if there was no data
coming in when the socket was accepted. Post a read operation to read the
Line 691, post an new Accept call
using CreateAcceptSocket@462 (Section 1.3)
Section 2.0b WorkerThread @529
à @699 FSM= ClientIoRead
Post a write operation when the read
Section <st1:chmetcnv w:st="on" TCSC="0"
NumberType="1" Negative="False" HasSpace="False" SourceValue="2"
UnitName="C">2.0c WorkerThread @529 à @724 FSM=ClientIoWrite
Line 731, the sent size is returned
by the function of GetQueuedCompletionStatus. dwIoSize.
Line 733, if the sent bytes is less
than the total bytes in the buffer, then post another sending. Goto step 3,
otherwise, step 4.
Line 739, move the pointer to the
position in the buffer that’s not been sent, also update the sent data on line
If the sending are complete, post a
read operation on line 763.
Line 830, when we asked to close the
socket abortively (i.e. Close without sending all the pending data, which is
common when the client closes the connection), we call setsocketopt to set
linger. Then close both the accept socket and all the other socket opened during
849, calls CtxtListDeleteFrom@945 to remove the context from the global list.
Note: CloseClient calls EnterCriticalSection once, and
CtxtListDeleteFrom@945 calls EnterCriticalSection once more. This will not
result in a dead lock. According to MSDN Library, This is a built in feature of
this function, a thread that has the ownership of the critical section will not
block if this function is