This problem is solved very elegantly, but the whole problem looks different, because the design I would advise is not based on events. Basically, you make applications multi-threading; and the service needs two threads; one for listening for new connections, one reading/writing from/to a network stream. The idea is to use exception-driven handle of disconnections, which excludes the need in graceful disconnections (which is impossible to guarantee anyway, because disconnection can be also accidental). The exception handler looks at the handle representing the remote client sockets and removes it from the container holding such offending client socket.
For some more detail, please see my past solution for this problem: using api in c# and socketprogramming