An Objective Analysis of Language Performance - 1) Math Functions

• Download mathperformance_data_summary - 13.6 KB
• Download mathperformance_src_c_dotnet - 24.66 KB
• Download mathperformance_src_java - 16.27 KB

Introduction

This series of articles presents an objective analytical analysis of language performance for Java and C# .NET (2.0, 3.0, and 3.5). The aim of these articles is too end the age old dispute of .Net is better than Java or vica versa. The article analyses performance of standard base mathematical operators common to all languages but specifically mathematical operators that form the basic components of most programs.

Scope of the Analysis

In this article the following functions are tested for performance;

Testing Procedure

Key Points

• The testing is performed via a console all cases with system load at minimal prior to test start
• All system auto-updates are disabled
• Task manager is run and set to show cpu usage of all processes ordered by cpu usage descending; if any other process exceeds 5% usage then the test is restarted
• The test process is set to a single core affinity (to reduce cpu effects)
• All tests are performed 5 times to reduce data noise and increase accuracy
• All tests are performed on the same multi-core system (reduces influences of other processes)
• Each mathematical function is tested 1000000000 times to reduce inaccuracy in system time calculation and increase accuracy of measures
• Effects of looping over 1000000000 times is removed in control case

Test System

Test Code

Control Case

double y = 0;
int controlStart = System.Environment.TickCount;
for (int i = 0; i < 1000000000; i++)
{
   y = y + 1;
}
int controlEnd = System.Environment.TickCount;

Console.WriteLine(y.ToString());

The control case is intended to be used to remove the effects of the loop over the test function. To prevent compiler optimisations from removing the loop the loop performs a simple add function which is used in all function tests (see below). Furthermore the value of the add function is printed at the end, again the prevent compiler optimisations from removing y due to non-use and then the loop itself. The control case is repeated 5 times and the average of all (controlEnd - controlStart) is used as the controlTime. y is reset to 0 at the start of each test.

Test Case

y = 0;
double x = 0;
int addStart = System.Environment.TickCount;
for (int i = 0; i < 1000000000; i++)
{
   y = y + 1;
   x = x + 1;
}
int addEnd = System.Environment.TickCount;

Console.WriteLine(y.ToString());
Console.WriteLine(x.ToString());

The test case is kept as similar to the control case as possible, only with the addition of the mathematical function being tested. In this example the add function is being tested, and like the control case the value of x is printed at the end to prevent removal of x due to compiler optimisations. Furthermore comparison of resultant x values may be used to confirm equal functionality between the various languages being tested. The test time is calculated as ((addEnd - addStart) - controlTime).

The other tests are performed as follows;

Results Analysis

All times are in milliseconds and are averages of the 5 test passes.

Lower values are better.

In these graphs relative performance of functions is shown, and counter to perhaps popular belief is that in all but the power and multiply cases Java actually appears to out-perform C# .Net languages. It is worth noting however that the cases of add and subtract for java are actually lower than 0 indicating that either the control cases have run particularly slowly or perhaps some optimisation has occurred. Since the control cases for Java show good cohesion, it is unlikely to be the former.

Lower values are better.

In this graph the total time for completing the control cases are shown over the 5 passes. It is interesting to note that some form of optimisation has occurred in the .Net languages due to the consistent reduction of overall time from pass 1 to the latter passes; this effect cannot be seen in Java which appears to perform consistently better (perhaps due to the control case being based on the add function itself which according to the results above performs significantly better).

Summary

• The control case, i.e. a loop and add function, performs consistently better in Java
• .Net performs some optimisation on repeated functions, while Java does not
• Java performs better at divide, add, and subtract functions (although further testing is required to confirm)
• .Net performs better at multiply and power functions

Other References

Check back soon for other articles on comparison of disk access and interface update performance.

History

Version 1.0.0.1 - Testing +, -, *, /, and power.