Abstracting TCP/IP Communications, and Adding What Should Be the New Basics

• Download AbstractTcpLib_1.5.5.zip - 775.8 KB
• Download Usage_Documentation_pdf.zip - 313.4 KB

Introduction

AbstractTcpLib is a TCP communications library that provides the functionality I have come to believe is the bare minimum that should be available in a TCP library. Using this library, you will be able to easily create and manage TCP connections between client and server, and communicate not only between the client and server, but also client to client. This library also provides AES256 encrypted communication, Windows authentication, multiple concurrent file transfers to and from the same client, and object serialization for convenient transfer of data between remote machines.

Documentation

This version of AbstractTcpLib includes a documentation PDF. If you have downloaded this library already, and only need or want the documentation, you can download it separately above. This is a more complicated library than most you will find here on CodeProject. If you would like to use it, but are having a hard time understanding how and when to use the various features, have a look at the PDF.

What If Sessions Could Contain Other Sessions?

When I first started working with TCP communications, I hated the idea of creating several TCP connections from a single application to a server. It seemed sloppy. It also seemed like I was cheating, because I believed that I should be able to do all the communicating I needed to over a single session. After all, bytes are bytes, right? If you have a connection to a server, you have a connection to a server.

Over the years, I came up with different ways to logically separate and manage the data I was sending - but there's just nothing like having an entire session to work with when sending whatever you have to send. So instead of coming up with yet another way to synchronize data between the client and the server, I built a session management system into this library. In this library, sessions really can contain other sessions. They're called Sub-Sessions, and these are the communication channels you will use for everything that you need to logically separate. The beauty of this is that your network hardware and the routers between your local machine and your remote machine will handle the bandwidth balancing between data sent over your subsessions, so while your overall bandwidth per subsession may decrease, they all get as much bandwidth as your switch and router(s) can provide no matter what is being sent over them.

Sub-Session Linking

Once I had the subsession management system in place, another thought occurred to me. If I have a client that handles and manages (potentially) lots of TCP sessions, what if I logically connected two subsessions belonging to different clients at the server? If it was done correctly, data sent from client 1 over this linked subsession would naturally flow to the other client... and the other client would be able to send data to client 1. All we need is a server that is smart enough to do the associations when we ask it to. We would also need to be able to get a list of connected clients, and clear error messages in the event that we asked the server to create a linked subsession, and for some reason it couldn't.

Of course, this is peer to peer communication. In this library, we're calling it "Sub-Session linking". Linked subsessions can be created between any two connected clients.

Bytes? Who Needs Bytes?

Most TCP libraries out there will allow you to send text, but if you want to send anything else, you need to convert the data into a string or a byte array. I spent a long time doing that, and it gets old. It forces you to do quite a bit of parsing, converting strings to byte arrays and converting classes to XML, and XML back into classes in your code. The more different kinds of data you need to send, the more of this there is. Things tend to get ugly fast.

This library has only a single .Send() function in the Client and Session classes for sending data. If you use it the way I intended, you will never even think about sending bytes again - because this library's .Send() function accepts any .NET serializable object. You drop your string into the Send() function, and you get a string on the other end. Have a DataTable to send? Drop it in the send function. Have some of your own classes to send? Mark them as serializable, and drop them into the Send() function - and off they go.

For this reason, I named the library AbstractTcpLib; Because it abstracts you from the details of sending your data.

But What If I WANT to Work With Bytes?

Of course, serializing objects and deserializing them at the remote end uses some CPU resourses. I'm using the .NET binary serializer, and although it is relatively fast and efficient, those among us who are interested in raw performance will see this as an issue. The good news is that byte arrays aren't serialized. Why? Because, well - that would be silly. The socket class sends byte arrays. Serializing a byte array would just use CPU resourses needlessly, and add a few bytes to the byte array to include information about the byte array object - all for nothing.

So if you really want to send Byte[] because you want the best possible network performance, or you're concerned about the binary serializer for one reason or another, or you just would rather work with bytes - then go right ahead. Just drop your byte array into the .Send() function.

Request / Response Transactions

Request / Response Transactions provide easy and powerful question and answer communication from client to server or client to client. Send a SQL query to the server and get a Datatable with just a few lines of code. Ask the server for an image and get it quickly and easily. Define an amount of time to wait for your reply and also supply a delegate with code to run if it's been too long. All wrapped up with a bow.

1. Create a "Request": A request contains a type string, a request object that you provide, and delegates that you provide containing code to run when the response arrives, and if the request times out. You must also specify an amount of time to wait for a reply.
2. On the remote machine, you define RequestProcessors: A request processor runs when a request arrives with a matching type string. You provide the type string and a delegate containing code to handle the request, and process the request object that has been sent.
3. Respond with a serializable object: If you choose to send a reply object using the .Respond(Object o) function, it will be sent back to the requesting client and processed by the delegate you provided in the initial Request.

Security - Encryption and Authentication

Everyone is concerned about security these days. I consult for an access control and video surveillance company, and of course, security is at the top of their list of priorities. Everything must be encrypted. Everything must be authenticated. It's so important in our society today, that it seems to me that no communications library can be considered complete without security functionality of some kind. In this library, I included two different methods of implementing security measures:

AES256 Encryption: The server class works in two modes - encrypted mode and mixed mode. In encrypted mode, all clients must have the correct pre-shared key to connect. Connecting clients are required to register themselves with the server as either a session or a subsession immediately upon connection. If the server cannot decrypt this registration request with its own PSK, the connection is rejected and the client will simply be disconnected with no error message whatsoever. If the server can decrypt the registration request, it immediately changes the session PSK to a new randomly generated one using RNGCryptoServiceProvider.

LDAP Authentication: The server can be configured to require Windows authentication. Before connecting with your Client, enter credentials using the .Login(username, password) function. Credentials are immediately stored as encrypted strings, and passed to the server in the registration request. The server will attempt to authenticate the credentials against the domain the server machine is in, so sending a domain along with the username (i.e.: domain\username) is not necessary. If the server is not in a domain, it will attempt to authenticate against the windows workstation it is running on. If the client's authentication fails, it receives an "authentication failure" message, and is disconnected.

Mixed Mode vs Encrypted Mode: In mixed mode, clients can create encrypted sessions or subsessions if they choose. It's important to know that if a server is configured to require authentication, but not to require encryption and you create non-encrypted sessions or subsessions, then your Windows credentials are being sent as serialized strings - and this is not secure. If you configure the server to require authentication, use encryption also.

Files, Files and More Files: Concurrent File Transfers per Client

So you need to send a file. Ok, great - this library has you covered. Need to send two? Sure, of course. How would you like to send them - one at a time, or both at the same time? Or maybe three or four at the same time? It's completely up to you, and what you think your hardware can handle.

AbstractTcpLib sends files over subsessions. In fact, if you look at the Client class, you won't see a SendFile() or GetFile() function there at all - those functions live in the Session class. You create a file transfer by first creating a subsession or two. Then you get a reference to your subsessions (which are Session objects) by using Client.GetSubSession(), and then call Session.SendFile() or Session.GetFile().

You can create as many as you like, and you can transfer files between your Client and your Server, or between your Client and another connected client.

To transfer files between clients, create a linked subsession first. Then get a reference to it using Client.GetSubSession() the same way you would any other subsession.

Subscribing to File Transfer Events: There are three delegates to subscribe to when initiating or receiving a file transfer: TransferProgress(Uint16 percentComplete), TransferError(String errorMessage), and TransferComplete(). I believe this is self explanatory, and they work the way you think they will. If you have any questions, please feel free to ask.

Files are transferred using the FileTransfer class. During a transfer, a file transfer object will be associated with each end of a subsession. Subscribing to the Server.receivingAFile(FileTransfer) or Client.receivingAFile(FileTransfer) delegate will allow you to become aware of when you are receiving a file, and you can also subscribe to the transferProgress, transferError and transferComplete delegates on the fileTransfer object so you can track the progress of the incoming file. If you don't want to allow the transfer to continue, you can always .Cancel() the transfer on the receiving end before it completes.

Some Details

The Client, Server and Session classes talk to each other using XML. Because this library serializes sent objects, I never needed to parse any XML myself. Instead, I used an XML parser I built called XmlObject. Using XmlObject and this library's ability to serialize objects and pass them back and fourth, I was able to easily and clearly create XmlObject(s), add parameters and other data, pass them to a remote machine where they arrive as XmlObjects, use the tools available in the XmlObject class to easily get at the passed data.

This library allows some of that communication to be filtered up to you, so that you are notified when sessions disconnect, when they connect and register themselves, when subsessions are created, when there is a connection failure due to incompatible encryption keys, authentication failure, etc. So your client and server will be receiving XmlObjects in your callbacks. Don't be afraid... they are your friends.

Using the Code

Both the server and the client use a delegate to pass you incoming data (your sent objects) as it comes in. These delegates have a single CommData object as a parameter. A CommData object is just a wrapper for your passed object. It contains the serialized byte array that came in, a deSerialized Object that is the object you put into sendbytes, only it isn't your String - it's an Object. You could simply test it to make sure it's your String using typeof(String), and then cast it to a String or var, or use the CommData.GetObject(), like this:

this.client = new Client((Core.CommData data) =>
{
    // Get the passed object:
    var o = data.deSerialized;
    
    if (o.GetType() == typeof(XmlObject) && ((XmlObject)o).Name.Equals("ATcpLib"))
    {
        XmlObject xml = (XmlObject)data.deSerialized;
        
        String msg      = xml.GetAttribute("", "internal");
        String originId = xml.GetAttribute("", "id");
        
        // Are we shutting down?
        if(msg.Contains("disconnected") && originId.Equals(client.GetConnectionId()))
        {
            UI(() =>
            {
                lblStatus.Text = "Disconnected.";
                btConnect.Text = "Connect";
                lbSubSessions.Items.Clear();
            });
        }
        
        if (msg.Contains("CreateSubSession") && originId.Equals(client.GetConnectionId()) 
        && xml.GetAttribute("", "status").Equals("true"))
        {
            UI(() =>
            {
                lbSubSessions.Items.Add(xml.GetAttribute("", "subSessionName"));
            });
        }
        
        // Is a SubSession shutting down?
        if (msg.Contains("disconnected") && !originId.Equals(client.GetConnectionId()))

Me.client = New Client(Function(ByVal data As Core.CommData)
    Dim o = data.deSerialized
    If o.[GetType]() = GetType(XmlObject) _
        AndAlso (CType(o, XmlObject)).Name.Equals("ATcpLib") Then
        Dim xml As XmlObject = CType(data.deSerialized, XmlObject)
        Dim msg As String = xml.GetAttribute("", "internal")
        Dim originId As String = xml.GetAttribute("", "id")
        If msg.Contains("disconnected") AndAlso originId.Equals(client.GetConnectionId()) Then
            UI(Function()
                lblStatus.Text = "Disconnected."
                btConnect.Text = "Connect"
                lbSubSessions.Items.Clear()
            End Function)
        End If

        If msg.Contains("CreateSubSession") AndAlso originId.Equals(client.GetConnectionId()) _
        AndAlso xml.GetAttribute("", "status").Equals("true") Then
            UI(Function()
                lbSubSessions.Items.Add(xml.GetAttribute("", "subSessionName"))
            End Function)
        End If

        If msg.Contains("disconnected") _
           AndAlso Not originId.Equals(client.GetConnectionId()) Then

As you see, the client's constructor takes the incoming data delegate. You connect to the server as follows:

String errMsg = "";
client.Login(tbUserName.Text, tbPassword.Text);
if (!client.Connect(System.Net.IPAddress.Parse(tbIpAddress.Text.Trim()), 
ushort.Parse(tbPort.Text.Trim()), tbSessionId.Text.Trim(), 
    out errMsg, cbUseEncryption.Checked, tbPsk.Text))
{
    MessageBox.Show(errMsg, "Connection failed.", MessageBoxButtons.OK, MessageBoxIcon.Error);
    return;
}

Dim errMsg As String = ""

client.Login(tbUserName.Text, tbPassword.Text)
If Not Me.client.Connect(System.Net.IPAddress.Parse(tbIpAddress.Text.Trim()), _
UShort.Parse(tbPort.Text.Trim()), tbSessionId.Text.Trim(), _
    errMsg, cbUseEncryption.Checked, tbPsk.Text) Then
    client.Close()
    MessageBox.Show(errMsg, "Connection failed.", _
                    MessageBoxButtons.OK, MessageBoxIcon.[Error])
    Return
End If

When you connect, a session object is created to handle your client's connection to the server, and added to the server's SessionCollection. You can send objects to the server using your session if you choose, or you can create subsessions.

Subsessions are sessions also. When you create a subsession with your client, a Session object is created and added to your client's subsession collection. On the server, your subsession is registered and added to your session's subsession collection.

To send data to the server (or to a peer) over your client's session, use the Client.Send() (or Session.Send()) function, as follows:

String errMsg = "";
if(!client.Send(tbMessage.Text, out errMsg))
{
    MessageBox.Show(errMsg, "Send failed.", MessageBoxButtons.OK, MessageBoxIcon.Error);
}

Dim errMsg As String = ""
If Not client.Send(tbMessage.Text, errMsg) Then
    MessageBox.Show(errMsg, "Send failed.", MessageBoxButtons.OK, MessageBoxIcon.Error)
End If

To send data over a subsession, first get the subsession using its name (String sessionId) as follows:

Session session   = null;
String errMsg     = "";
if(!client.GetSubSession(lbSubSessions.SelectedItems[0].ToString(), out session, out errMsg))
{
    MessageBox.Show(errMsg, "Could not get subsession.", _
                    MessageBoxButtons.OK, MessageBoxIcon.Error);
} else
{
    if(!session.Send(tbMessage.Text, out errMsg))
{
    MessageBox.Show(errMsg, "Send failed.", MessageBoxButtons.OK, MessageBoxIcon.Error);
}
}

Dim session As Session = Nothing
Dim errMsg As String = ""
If Not client.GetSubSession(lbSubSessions.SelectedItems(0).ToString(), session, errMsg) Then
    MessageBox.Show(errMsg, "Could not get subsession.", _
                    MessageBoxButtons.OK, MessageBoxIcon.Error)
Else
    If Not session.Send(tbMessage.Text, errMsg) Then
        MessageBox.Show(errMsg, "Send failed.", MessageBoxButtons.OK, MessageBoxIcon.Error)
    End If
End If

To subscribe to the server (or client's) Incoming FileTransfer delegate, do it as follows (this example is taken from the example application. As such, it is updating a listview with the transfer's information):

server.receivingAFile = (FileTransfer transfer) =>
{
    ListViewItem lvi                                = new ListViewItem(transfer.FileName());
    FileTransfer.TransferComplete complete          = null;
    FileTransfer.TransferProgress updateProgress    = null;
    FileTransfer.TransferError transferError        = null;
    
    lvi.SubItems.Add("0%");
    lvi.SubItems.Add(transfer.DestinationFolder());
    lvi.SubItems.Add("Transferring file");
    
    complete = () =>
    {
        UI(() => lvi.SubItems[3].Text = "Complete");
        
        transfer.transferComplete   -= complete;
        transfer.transferError      -= transferError;
        transfer.transferProgress   -= updateProgress;
    };
    
    updateProgress = (ushort percentComplete) =>
    {
        UI(() =>
        {
            lvIncommingFiles.BeginUpdate();
            lvi.SubItems[1].Text = percentComplete.ToString() + "%";
            lvIncommingFiles.EndUpdate();
        });
    };
    
    transferError = (String errorMessage) =>
    {
        UI(() =>
        {
            lvi.SubItems[2].Text    = "Error: " + errorMessage;
            lvi.ForeColor           = Color.Red;
        });
        
        transfer.transferComplete   -= complete;
        transfer.transferError      -= transferError;
        transfer.transferProgress   -= updateProgress;
    };
    
    transfer.transferComplete   += complete;
    transfer.transferError      += transferError;
    transfer.transferProgress   += updateProgress;
    
    UI(() => lvIncommingFiles.Items.Add(lvi));
};

server.receivingFile = Sub(transfer As FileTransfer)
        Dim lvi As New ListViewItem(transfer.FileName())
        Dim complete As FileTransfer.FileTransferComplete = Nothing
        Dim updateProgress As FileTransfer.FileTransferProgress = Nothing
        Dim transferError As FileTransfer.FileTransferError = Nothing

        lvi.SubItems.Add("0%")
        lvi.SubItems.Add(transfer.DestinationFolder())
        lvi.SubItems.Add("Transferring file")

        complete = Sub()
                       UI(Sub()
                              lvi.SubItems(3).Text = "Complete"
                       End Sub)

                       transfer.transferComplete = _
                                    [Delegate].Remove(transfer.transferComplete, complete)
                       transfer.transferError    = _
                                    [Delegate].Remove(transfer.transferError, transferError)
                       transfer.transferProgress = _
                                    [Delegate].Remove(transfer.transferProgress, updateProgress)

                   End Sub

        updateProgress = Sub(percentComplete As UShort)
                             UI(Sub()
                                    lvIncommingFiles.BeginUpdate()
                                    lvi.SubItems(1).Text = percentComplete.ToString() + "%"
                                    lvIncommingFiles.EndUpdate()
                                End Sub)

                         End Sub

        transferError = Sub(errorMessage As String)
                            UI(Sub()
                                   lvi.SubItems(2).Text = "Error: " + errorMessage
                                   lvi.ForeColor = Color.Red
                               End Sub)

                               transfer.transferComplete = _
                                  [Delegate].Remove(transfer.transferComplete, complete)
                               transfer.transferError = _
                                  [Delegate].Remove(transfer.transferError, transferError)
                               transfer.transferProgress = _
                                  [Delegate].Remove(transfer.transferProgress, updateProgress)

                        End Sub

                        transfer.transferComplete = _
                                 [Delegate].Combine(transfer.transferComplete, complete)
                        transfer.transferError = _
                                 [Delegate].Combine(transfer.transferError, transferError)
                        transfer.transferProgress = _
                                 [Delegate].Combine(transfer.transferProgress, updateProgress)

                        UI(Function() lvIncommingFiles.Items.Add(lvi))

       End Sub

How to Send Your Own Custom Objects

I didn't initially include any information about this because - like most developers, I think - I just assumed that everyone would just automatically understand how that worked. Well, today I learned that it's not quite as intuitive as I imagined, so I'm going to outline exactly how to do it here:

First, to send your custom classes from your clients to your server, (or the reverse), both your server and your client have to know about them. You can't just create a class in your client and send it. You're server won't understand what it's receiving.

1. Create a new class file in your project. When I write "your project", I mean the project in which you will be adding a reference to AnstractTcpLib, and creating an AbstractTcpLib.Client() or an AbstractTcpLib.Server(). If you want to send your own custom class objects back and forth between client and server, you must create this custom class in both the project you are creating an AbstractTcpLib.Client() in, and the project you are creating an AbstractTcpLib.Server() in, and they must be identical.
2. Change the default namespace to something descriptive and useful in both your server and client applications. In this example project, there is now a "MyObj" class. This class is in the "MyObjects" namespace. There is an identical MyObj class in both the server and the client applications, so you can see how this is done.
3. When your server (or client) receives an object, you need to test for it using typeof. Something like this will work:

if (data.type == typeof(MyObjects.MyObject))
{
    MyObjects.MyObject obj = (MyObjects.MyObject)data.deSerialized;
    // Do something with obj here.
}

If data.type = GetType(MyObjects.MyObject) Then
    Dim obj As MyObjects.MyObject = DirectCast(data.deSerialized, MyObjects.MyObject)
    ' Do something with obj here.
End If

And that's it. If you attempt to send a class or object that the receiving assembly cannot understand, you will instead be handed an XmlObject containing the exception information generated when deserialization failed.

Wait - Isn't There More?

Of course. Please see the example application for anything else you need to know how to do. If you can't find it there, or if you run into an issue, please feel free to ask below.

Will You Be Adding Anything in the Future?

Definitely!

Cooperative throttling, for starters. Encrypted registration requests for clients using LDAP authentication (at the moment they are only deflated, preventing clear text transmission of login credentials), and more as I think of it, or you ask for it, and I have the time.

Found a Bug?

This library is cool... if I do say so myself. I have enjoyed building it, and to a small degree testing it. But there's a lot of code here, and there's just no way I could test it as thoroughly as I would like... and it's brand new.

I'm going to use it in future projects, and post fixes as bugs appear. If you find one, please feel free to let me know. I'll fix it and post a new build as I'm able.

Thanks for reading!

Changes

The following list contains changes in version 1.5.5 of this library:

• Resolved a memory leak, discovered by CosmoMaster.

The following list contains changes in version 1.5.4 of this library:

• Removed some code that was in the library for testing purposes, which would prevent file transfers in some circumstances.
• Added a documentation PDF.

The following list contains changes in version 1.5.3 of this library:

• Added server event handlers for Session connections, Session disconnections and general notifications. This goes a long way toward uncluttering the main server callback, and dedicating all of that space to the processing of user data arriving at the server. Please have a look at the example projects to see how it's done for your language.

The following list contains changes in version 1.5.2 of this library:

• Fixed a bug in the server causing a crash if a client attempted to connect to a server with an incorrect psk while configured for non-encrypted communication when the server was configured for encryption only (reported by Toron).
• Did some manual refactoring of the new RequestResponseTransactions class.

The following list contains changes in version 1.5.1 of this library:

• Fixed a bug in Client.Close() that throws an exception if close() is called before a connection is attempted.
• Added New Functionality: client.connect() now accepts a string as an address. V4 ip addresses or hostnames will be accepted and parsed or resolved into an ip address.

The following list contains changes in version 1.5 of this library:

• Fixed a bug preventing a subsession from being created with the same name as one that had been previously closed.
• Added New Functionality: The Request/Response Transaction system.

The following list contains changes in version 1.4.2 of this library:

• Changed the error response behaviour when the server is started using a port that is already in use. Previously, an exception was thrown. Now when the exception is thrown, it's caught and passed to the end user via the data in delegate as an XmlObject containing the exception data.

The following list contains changes in version 1.4.1 of this library:

• Corrected issue in the VB.NET example project preventing custom objects from being transferred properly between VB.NET applications.
• Changed the names of the test objects so they are identical between C# and VB.NET example applications.

The following list contains changes in version 1.4 of this library:

• Resolved a crash while making an encrypted connection to a server that isn't configured to accept encrypted connections.
• Added new example client and server projects in VB.NET. They look and behave identically to their C# counterparts in every way, except they are done in VB.NET.

The following list contains changes in version 1.3.1 of this library:

• Resolved a crash while sending custom objects.
• AbstractTcpLib catches deserialization failures, and reports them using an XmlObject containing the exception information.

The following list contains changes in version 1.2 of this library:

• This library will no longer accept serialized objects as registration requests, and will disconnect any other kind of initial connection.
• Internal library communication is no longer done using serialized objects. Instead, XmlObject(s) are converted to strings, then to byte arrays and deflated before being passed, and parsed back to XmlObject(s) in the remote machine. This change is seamless for the end user.
• These changes are to protect servers configured for LDAP authentication only from DOS attacks.

The following list contains changes in version 1.1 of this library:

• This library now Deflates serialized objects before sending them, and Inflates them before deserializing.