Unit test and the man in the middle

• Download ProxyRecorderConsole.zip - 298.8 KB

(xkcd.com)

Unit test and the man in the middle

• The pain
• The cure : A real proxy man-in-the-middle
• Not the silver bullet yet
• Show me the code
• More to come
• History

The pain

I hate slow tests... You know, the kind of test that cause a coffee addiction... Not because you like coffee, you are just bored to death by waiting.

Some among you will preach the benefit of mock testing. That's great, but mock testing work in test, and this stupid MockMailSender will not work so well in real life.

Above all, I don't like writing code, it's buggy, and there is always someone to put you down on each line of code.
"What's your solution Nicolas ?" I can read in your curious eyes.

The cure : A real proxy man-in-the-middle

Well, just run the test one time, with a man-in-the-middle (proxy) between your networks ressource and your test, record messages in a local folder. And next time you run the test, just ask man-in-the-middle to send again the same sequence of byte.
Picture maestro !

Step 1 : You record messages between your machine and the server.

Step 2 : You replay messages from your local store. (By default, a folder)

How cool is that ?

I used it in my own code to test code that interact with SMTP/IMAP server, so how it works in real code ?

static void Main(string[] args)
{
    string dataFromCP;
    string dataFromProxy;

    ProxyRecorder recorder = new ProxyRecorder(8172, "www.codeproject.com", 80);

    using(recorder.Record("CPData"))
    {
        WebClient client = new WebClient();
        dataFromCP = client.DownloadString("http://localhost:8172/");
    }

    //You can cut the network connection, reboot your computer, or work on the beach here !
    using(recorder.Replay("CPData"))
    {
        WebClient client = new WebClient();
        dataFromProxy = client.DownloadString("http://localhost:8172/");
    }
    AssertAreEqual(dataFromCP, dataFromProxy);
}

However, there are some problems to be aware of with this approach...

Not the silver bullet yet

The problem is that some protocol are replay replay proof. In other words, the client will detect that these messages are not the real one.
That's the case with SSL, but don't be sad so fast, I've not told your the end of the story.

My code works with SMTP and IMAP over SSL.

To do that, I create a fake certificate with my proxy, and tell my client to trust it.
My proxy then, can decrypt and encrypt client data, and negotiate its own SSL session with the real server to reencrypt it.

ProxyRecorder recorder = new ProxyRecorder(8172, "smtp.codeproject.com", 593);
recorder.UseSmtpOverSSL = true;
ServicePointManager.ServerCertificateValidationCallback += (a, b, c, d) => true; //Accept invalid certificate
using(recorder.Record("CPData"))
{
    SmtpClient client = new SmtpClient();
    client.Host = "localhost";
    client.Port = 8172;
    client.EnableSsl = true;
    //...do your stuff

But wait, there is more : in the case of HTTP, I cheated on my previous example...

Imagine you want to go on a website like www.google.fr, it will try to redirect you to www.google.com... And so, the WebClient will then send a request to google and not to your proxy !

No problem I thought about it (thanks to this article) you just have to specify that you are using a proxy to WebClient :

static void Main(string[] args)
{
    string dataFromCP;
    string dataFromProxy;

    ProxyRecorder recorder = new ProxyRecorder(8172, "www.codeproject.com", 593);
    using(recorder.Record("CPData"))
    {
        WebClient client = new WebClient();
        client.Proxy = recorder.CreateHttpWebProxy();
        dataFromCP = client.DownloadString("http://www.codeproject.com");
    }

    //You can cut the network connection here !
    using(recorder.Replay("CPData"))
    {
        WebClient client = new WebClient();
        client.Proxy = recorder.CreateHttpWebProxy();
        dataFromProxy = client.DownloadString("http://www.codeproject.com");
    }
    AssertAreEqual(dataFromCP, dataFromProxy);
}

All HTTP requests will endup with the right domain, and will pass through your proxy.

For WCF and SOAP message with WS-Security, you will need to deactivate detect replay attack, and set the ClockMaxSkew to maximum value on your client.

Show me the code

That's very simple, you only have one extension point to change the message persistence store.

By default all messages are stored in a folder a zip file. And I would appreciate if someone could develop the IMessageRepository to store in a zip file instead... (Done since 22 may)

The actual implementation is very simple, it just open two socket : one for the client, and one for the server, then copy the data from one to another.

Here is the code of the Recorder: (parent.Protocol.CreateWrapperStream will wrap the network stream depending on the ProxyRecorder.Protocol used. It can be Http, Smtp, default -automatic detection with port number- or simple -protocol with no SSL bootstrapping like IMAP-)

void ClientAccepted(IAsyncResult ar)
{
    var clientStream = parent.Protocol.CreateWrapperStream(new NetworkStream(socket.EndAccept(ar)), false);
    var targetStream = parent.Protocol.CreateWrapperStream(new NetworkStream(parent.ConnectToTarget()), true);
    Route(clientStream, targetStream, false);
    Route(targetStream, clientStream, true);
}

object _lock = new object();

private void Route(Stream source, Stream target, bool sourceIsTarget)
{
    var buffer = new byte[10000];
    Route(source, target, buffer, sourceIsTarget);
}
private void Route(Stream source, Stream target, byte[] buffer, bool fromTarget)
{
    ReThrowIfNotDisposed(() =>
    {
        source.BeginRead(buffer, 0, buffer.Length, ar =>
        {
            ReThrowIfNotDisposed(() =>
            {
                var count = source.EndRead(ar);
                lock(_lock)
                {
                    target.Write(buffer, 0, count);
                    _Stream = _MessageStream.CreateNextMessage(fromTarget);
                    _Stream.Write(buffer, 0, count);
                }
                Route(source, target, buffer, fromTarget);
            });
        }, null);
    });
}

For the Replay part, I just iterate over all messages from the store and dispose them or send it to client.

private void Listen(Stream clientStream, byte[] buffer)
{
    ReThrowIfNotDisposed(() =>
    {
        var message = _MessageStream.GetNextMessage();
        if(message != null)
        {
            if(message.IsTarget)
            {
                clientStream.Write(message.Content, 0, message.Content.Length);
                Listen(clientStream, buffer);
            }
            else
            {
                clientStream.Read(buffer, 0, buffer.Length);
                Listen(clientStream, buffer);
            }
        }
    });
}

More to come

I did not test yet on database connection, but I think it should work, since there is no credential negotiation with anti replay by default if I remember.

Anyway, thanks for reading, I hope it will change your life, and that you will live longer. If some protocols are not supported, just tell me.

And last thing : I want to thank code project for changing the article submission wizard, that was a moment of joy for me :')

History

22 May 2012 : Lots of bug fix, sockets are properly disposed on recorder and replayer dispose. Support saving traffic in .zip by default instead of folder. Support HTTPS.