Roselyn CTP vs. Microsoft.CSharp (C# native compiler)

Here is a small comparison between the new Roselyn CTP compiler from Microsoft (http://www.microsoft.com/en-us/download/details.aspx?id=27746) and the native C# compiler which is integrated into .NET.

Here is the code used for testing compilation between the "two" compilers:

static class Program
{
    static void Main()
    {
        int pass = 50;
        List<long> rList = new List<long>();
        List<long> oList = new List<long>();
 
        for (int i = 1; i < pass; i++)
        {
            long roselynVal = TestRoselyn();
            long oldVal = TestOld();
 
            rList.Add(roselynVal);
            oList.Add(oldVal);
 
            Console.WriteLine("Roselyn Pass " + i + ":  " + roselynVal.ToString() + 
                     "\t\tOld Microsoft compiler Pass " + i + ":  " + oldVal.ToString());
        }
 
        var rosleynAvg = rList.Average();
        var oldAvg = oList.Average();
        //var rAvg = from r in rList
        Console.WriteLine("Roselyn average ms:\t" + rosleynAvg);
        Console.WriteLine("Microsoft old compiler average ms:\t" + oldAvg);
 
        Console.ReadKey();
    }
 
    /// <summary>
    /// Test roselyn
    /// </summary>
    /// <returns></returns>
    private static long TestRoselyn()
    {
        //Crate stopwatch and evaluate script using Roselyn
        Stopwatch sw = new Stopwatch();
 
        sw.Start();
        var engine = new ScriptEngine();
        var result = engine.Execute<bool>("var x = 10; x == 10");
        sw.Stop();
        return sw.ElapsedMilliseconds;
    }
 
    /// <summary>
    /// Test old one
    /// </summary>
    /// <returns></returns>
    private static long TestOld()
    {
        Stopwatch sw2 = new Stopwatch();
 
        sw2.Start();
        Microsoft.CSharp.CSharpCodeProvider cp = new CSharpCodeProvider();
        CompilerParameters cparam = new CompilerParameters { GenerateInMemory = true };
        CompilerResults cr = cp.CompileAssemblyFromSource(cparam,
           "namespace RoselynTemplate { class Main { public bool " + 
           "GetSmt() { var x = 10; return x == 10; } }}");
        var instance = cr.CompiledAssembly.CreateInstance("RoselynTemplate.Main");
        var method = instance.GetType().GetMethod("GetSmt");
        object res = method.Invoke(instance, null);
        sw2.Stop();
        return sw2.ElapsedMilliseconds;
    }
}

And here are the test results:

Considering code execution, it seems that Roselyn uses some kind of caching mechanism, also I need to mention that every time at “Pass 1” the old Microsoft compiler has beaten Roselyn by 6-7ms which I presume is spent on caching and other operations.