Comparative Speed Testing

• Download SpeedTests_Article - 129.65 KB
• Download SpeedTests_EXE - 22.47 KB
• Download SpeedTests_Src_CS - 21.28 KB
• Download SpeedTests_Src_VB - 21.46 KB

Abstract

This article presents an unsophisticated class for comparing the raw speed of two programming design alternatives. It is not a scientific benchmarking exercise, but is a very simple way of implementing two coded processes to gauge which is probably faster and approximately how much faster. The class allows the comparison with a minimum of infrastructure coding, it is not rocket-science.

Contents

• Introduction
• Some Surprising Results
• Example - property versus direct member access:
• Example - recursion versus stack-and-loop:
• Class SpeedTestsAB
• Using SpeedTestsAB
• The Downloads
• Points to Note
• Sample Test Results
• Speed Test: If, ElseIf compared with Select Case.
• Speed Test: If, ElseIf compared with nested IIF.
• Speed Test: Compare Shift-Right 4 (X >> 4) with Divide by 16.
• History

Introduction

In the process of writing some imaging filter methods I continually found myself in the position of having to optimise my code for raw speed. Although the approach I had taken was fundamentally quite an efficient one, the need to regularly process more than six million pixels made speed paramount, sometimes at the expense of transparency and good programming practice. The simple process of converting a 24-bit image to 8-bit, indexed grayscale then finding edges, meant 20 million pixel manipulations. One microsecond gained per pixel saved 20 seconds of processing time.

Routine practices soon became critical decisions, for example: do I protect a class member/field (like a pixel array) and access it via a read-write property, or do I make it public and allow read-write directly to the data? Until I knew what the time penalty was for access via a property, I had no more than a strong suspicion that reading-writing directly would be slightly faster, and even less idea of how much faster it might be.

Eventually I had such a backlog of outstanding comparisons to make that I wrote a simple abstract class that enabled me to get through them as quickly as possible with sufficiently scientific results that I could choose which options to use.

Some Surprising Results

Well they were to me anyway...

I was not interested in any rigorous scientific comparison that would withstand serious academic scrutiny, I was merely interested in making some design choices that would result in a shorter wait for some imaging filter to execute. Some of the differences in the tests I ran were a little more pronounced than I expected, some quite surprising.

Example - property versus direct member access:

An example; direct, unprotected read access to class members is about five times as efficient as the same access via a property. I did not test write-access; the test outcome was sufficient evidence for me to make several class members public.

Later I discovered, to my surprise, that if the ÔÇ£ReleaseÔÇØ version of the executable is run the outcome is indeterminate - one pass gives property access the advantage, another direct access. My conclusion is that either will do, and my assumption is that the optimiser builds similar code for both in the release binary. I have included two executables in the download, SpeedTests_Debug.exe and SpeedTests_Release.exe, so you can see the disparity for yourself.

Example - recursion versus stack-and-loop:

Another example: a simple recursive method surprisingly seemed ten times as efficient as pushing a value on a dotNet Stack instance and looping. I did not test a hand-crafted stack which may have led to different results; I just went with the recursive method.

Class SpeedTestsAB

The speed test runs a rudimentary control method (SpeedTestControl) which is simply an empty method, to provide an overhead metric which can be subtracted from the total time of each test to provide a net running time.

The abstract class SpeedTestsAB defines four abstract/mustoverride methods:

Method	Description
SetUpTestA	Initialisation for SpeedTestA, for example, CS: base.f_DescribeA = "Access a public field directly.";
SetUpTestB	Initialisation for SpeedTestB, for example, VB: me.f_DescribeB = "Access a field via a property."
SpeedTestA	The speed-test A code to execute.
SpeedTestB	The speed-test B code to execute.

In addition there are methods and properties that enable simple reporting of the results.

Property	Description
TotalTimeA TotalTimeB TotalTimeControl	The total times taken for each test.
NetTimeA NetTimeB	The net time taken for each test - TotalTime - TotalTimeControl.
Repetitions	The number of repetitions.
Method	Description
Results	Returns a very simple results report string.
AppendResults	Append the result string to a file.
ShowResults	Show the result string in a MessageBox.
WriteResults	Write the result string to a file or stream.

Results output looks like:

TEST RESULTS:

10,000,000 repetitions.
Test A: Access a public field directly.
Test B: Access a field via a property.
00:00:00.0937500 hh:mm:ss.ff Equivalent Elapsed Time Control Process.
00:00:00.1406250 hh:mm:ss.ff Total Elapsed Time Process A.
00:00:00.1718750 hh:mm:ss.ff Total Elapsed Time Process B. 00:00:00.0468750 hh:mm:ss.ff
Net Elapsed Time Process A.
00:00:00.0625000 hh:mm:ss.ff Net Elapsed Time Process B.
Net Unit Processing Time A: 4.688 nanosecs
Net Unit Processing Time B: 6.250 nanosecs
75.000% Percentage: Process A divided by Process B.

Using SpeedTestsAB

Create a class that inherits clsBaseSpeedTestsAB, e.g.:

Public Class clsSpeedTestAB_Properties
  Inherits clsBaseSpeedTestAB

Create a code section that defines properties, methods and data required to run the tests, e.g.:

#Region "[=== SPEED TEST COMPONENTS ===]"

  Protected f_SomeInteger As Int32 = 123456

  Public Property SomeInteger() As Int32
    Get
      Return Me.f_SomeInteger
    End Get
    Set(ByVal value As Int32)
      Me.f_SomeInteger = value
    End Set
  End Property

#End Region

Override methods SetUpTestA and SetUpTestB with, at least, the description of the tests, e.g.:

  ''' <summary>
  ''' Overriden speed test A setup.
  ''' </summary>
  ''' <remarks></remarks>
  Protected Overrides Sub SetUpTestB()
    Me.f_DescribeB = "Access a field via a property."
  End Sub

Override methods SpeedTestA and SpeedTestB with the code to be tested, e.g.:

  ''' <summary>
  ''' Overridden Speed Test A.
  ''' </summary>
  ''' <remarks></remarks>
  Protected Overrides Sub SpeedTestA()
    Dim xInt As Int32
    xInt = Me.f_SomeInteger
  End Sub

  ''' <summary>
  ''' Overridden Speed Test B.
  ''' </summary>
  ''' <remarks></remarks>
  Protected Overrides Sub SpeedTestB()
    Dim xInt As Int32
    xInt = Me.SomeInteger
  End Sub

Define the class somewhere and call the RunTest method, e.g.:

  Private Sub Button1_Click(ByVal sender As System.Object _
        , ByVal e As System.EventArgs) Handles Button3.Click
    Dim xTest As New clsSpeedTestAB_Properties(10000000)
    xTest.RunTest()
    xTest.ShowResults
  End Sub

The Downloads

The down-loads are in four packs:

Pack	Contents
SpeedTests_Src_CS.zip	C# source code for the demonstration, including: clsBaseSpeedTestAB.cs clsSpeedTestAB_Delegates.cs clsSpeedTestAB_Properties.cs clsSpeedTestAB_Recursion.cs Form1.cs Please note that the C# source code is the disassembler output from Lutz RoederÔÇÖs dotNET Reflector v5.1, and not the result of my porting the source code. As a consequence there may be some errors in the code as the disassembly created several which I have tried to fix.
SpeedTests_Src_VB.zip	C# source code for the demonstration, including: clsBaseSpeedTestAB.vb clsSpeedTestAB_Delegates.vb clsSpeedTestAB_Properties.vb clsSpeedTestAB_Recursion.vb Form1.vb
SpeedTests_EXE.zip	The two speed test compiles, one in debug mode (SpeedTests_Debug.exe), and one in release mode (SpeedTests_Release.exe).
SpeedTests_Article.zip	This article.

Points to Note

This is not an example of proper scientific speed benchmarking, but is only a mechanism for making a choice as to which code design to use when raw speed is of primary importance.

There is often quite a difference between the efficiency/speed of code which is compiled in debug mode when compared to the same code compiled in release mode. Assuming that the final version of your software will be compiled in release mode, I recommend speed testing in that mode. It is of interest however, to compare execution speed in the two modes.

Sample Test Results

(In debug Mode.)

These results are from an earlier incarnation of the class and were actively used in making image coding design decisions. The differences may disappear when tested with a release compiled version.

Speed Test: If, ElseIf compared with Select Case.

    Public Overrides Sub SpeedTestA()
      If Me.f_Int32 = 0 Then
        Me.f_Int32 = 1
      ElseIf Me.f_Int32 = 1 Then
        Me.f_Int32 = 1
      ElseIf Me.f_Int32 = 2 Then
        Me.f_Int32 = 2
      ElseIf Me.f_Int32 = 3 Then
        Me.f_Int32 = 3
      ElseIf Me.f_Int32 = 4 Then
        Me.f_Int32 = 4
      ElseIf Me.f_Int32 = 5 Then
        Me.f_Int32 = 5
      ElseIf Me.f_Int32 = 6 Then
        Me.f_Int32 = 6
      ElseIf Me.f_Int32 = 7 Then
        Me.f_Int32 = 7
      ElseIf Me.f_Int32 = 8 Then
        Me.f_Int32 = 8
      ElseIf Me.f_Int32 = 9 Then
        Me.f_Int32 = 9
      End If
    End Sub

   Public Overrides Sub SpeedTestB()
      Select Case Me.f_Int32
        Case 0
          Me.f_Int32 = 0
        Case 1
          Me.f_Int32 = 1
        Case 2
          Me.f_Int32 = 2
        Case 3
          Me.f_Int32 = 3
        Case 4
          Me.f_Int32 = 4
        Case 5
          Me.f_Int32 = 5
        Case 6
          Me.f_Int32 = 6
        Case 7
          Me.f_Int32 = 7
        Case 8
          Me.f_Int32 = 8
        Case 9
          Me.f_Int32 = 9
      End Select
    End Sub

    Protected f_Int32 As Int32

TEST RESULTS:

600,000,000 repetitions.
Test A: Use if, elseif.
Test B: Use select case.
00:00:07.4270833 hh:mm:ss.ff Equivalent Elapsed Time Control Process.
00:00:10.0312500 hh:mm:ss.ff Total Elapsed Time Process A.
00:00:09.2968750 hh:mm:ss.ff Total Elapsed Time Process B.
00:00:02.6041667 hh:mm:ss.ff Net Elapsed Time Process A.
00:00:01.8697917 hh:mm:ss.ff Net Elapsed Time Process B.
Net Unit Processing Time A: 2.604 secs
Net Unit Processing Time B: 1.870 secs
139.276% Percentage: Process A divided by Process B.

CONCLUSION:

Select Case may be approximately 40% faster than If, ElseIf.

Speed Test: If, ElseIf compared with nested IIF.

    Public Overrides Sub SpeedTestA()
      If Me.f_Int32 = 0 Then
        Me.f_Int32 = 1
      ElseIf Me.f_Int32 = 1 Then
        Me.f_Int32 = 1
      ElseIf Me.f_Int32 = 2 Then
        Me.f_Int32 = 2
      ElseIf Me.f_Int32 = 3 Then
        Me.f_Int32 = 3
      ElseIf Me.f_Int32 = 4 Then
        Me.f_Int32 = 4
      ElseIf Me.f_Int32 = 5 Then
        Me.f_Int32 = 5
      ElseIf Me.f_Int32 = 6 Then
        Me.f_Int32 = 6
      ElseIf Me.f_Int32 = 7 Then
        Me.f_Int32 = 7
      ElseIf Me.f_Int32 = 8 Then
        Me.f_Int32 = 8
      ElseIf Me.f_Int32 = 9 Then
        Me.f_Int32 = 9
      End If
    End Sub

    Public Overrides Sub SpeedTestB()
      IIf(Me.f_Int32 = 0, Me.f_Int32 = 0 _
            , IIf(Me.f_Int32 = 1, Me.f_Int32 = 1 _
            , IIf(Me.f_Int32 = 2, Me.f_Int32 = 2 _
            , IIf(Me.f_Int32 = 3, Me.f_Int32 = 3 _
            , IIf(Me.f_Int32 = 4, Me.f_Int32 = 4 _
            , IIf(Me.f_Int32 = 5, Me.f_Int32 = 5 _
            , IIf(Me.f_Int32 = 6, Me.f_Int32 = 6 _
            , IIf(Me.f_Int32 = 7, Me.f_Int32 = 7 _
            , IIf(Me.f_Int32 = 8, Me.f_Int32 = 8 _
            , IIf(Me.f_Int32 = 9, Me.f_Int32 = 9 _
            , Me.f_Int32 = 9))))))))))
    End Sub

    Protected f_Int32 As Int32 = -1

TEST RESULTS:

100,000,000 repetitions.
Test A: Use if, elseif.
Test B: Use nested IIF.
00:00:01.2343750 hh:mm:ss.ff Equivalent Elapsed Time Control Process.
00:00:03.2812500 hh:mm:ss.ff Total Elapsed Time Process A.
00:00:39.7500000 hh:mm:ss.ff Total Elapsed Time Process B.
00:00:02.0468750 hh:mm:ss.ff Net Elapsed Time Process A.
00:00:38.5156250 hh:mm:ss.ff Net Elapsed Time Process B.
Net Unit Processing Time A: 2.047 secs
Net Unit Processing Time B: 38.516 secs
5.314% Percentage: Process A divided by Process B.

CONCLUSION:

Do not use IIF.

Speed Test: Compare Shift-Right 4 (X >> 4) with Divide by 16.

Public Overrides Sub SpeedTestA()
  Dim xInt As Int32 = CInt((((((((Me.f_Int >> 4) >> 4) >> 4) >> 4) >> 4) >> 4) >> 4) >> 4)
End Sub

Public Overrides Sub SpeedTestB()
  Dim xInt As Int32 = CInt((((((((Me.f_Int / 16 / 16 / 16 / 16 / 16 / 16 / 16 / 16)
End Sub

Protected f_Int As Int32 = 123456

TEST RESULTS:

100,000,000 repetitions.
Test A: Shift-Right 4.
Test B: Divide by 16.
00:00:01.2500000 hh:mm:ss.ff Equivalent Elapsed Time Control Process.
00:00:01.3593750 hh:mm:ss.ff Total Elapsed Time Process A.
00:00:20.9843750 hh:mm:ss.ff Total Elapsed Time Process B.
00:00:00.1093750 hh:mm:ss.ff Net Elapsed Time Process A.
00:00:19.7343750 hh:mm:ss.ff Net Elapsed Time Process B.
Net Unit Processing Time A: 109.375 millisecs
Net Unit Processing Time B: 19,734.375 millisecs
0.554% Percentage: Process A divided by Process B.

CONCLUSION:

Getting close to 200 times as fast to shift rather than divide.

History

2008-05-26 Created.

You must Sign In to use this message board.

FAQ    Noise Tolerance Very HighHighMediumLowVery Low   Layout Expand Posts & RepliesThread ViewNo JavascriptNo JS + Preview   Per page 102550

Msgs 1 to 6 of 6 (Total in Forum: 6) (Refresh) FirstPrevNext

Dictionaries Selma 0:51 3 Jun '08   Hi Warrick Have you done any comparison tests against some of the collections, e.g. Dictionary, SortedDictionary, SortedList, etc? Selma Sign In·View Thread·PermaLink Re: Dictionaries Warrick Procter 13:48 5 Jun '08   Hello Selma No, I have not done any comparitive tests on Dictionaries, etc. When/if I do, I'll post the results here. If you or anyone else does, please forward me the code and the results and I will include it. Regards, Warrick Troft not lest ye be sponned on the nurg! (Milligan) Sign In·View Thread·PermaLink >> versus / gugoxx 22:20 2 Jun '08   This last test could be illusory The problem here is that the compiler optimizes subsequent /16 executions in one division only If the optimizer finds ((((val/16)/16)/16)/16), it rewrites (val/65536), with one only division while the same optimization for the >> is not present. Try Disassemblying. Sign In·View Thread·PermaLink Re: >> versus / Warrick Procter 0:17 3 Jun '08   Hello gugoxx, thanks for that input, I think you are correct. I no longer have the code for those tests. I remember that I had quite a backlog of tests and did them all one after the other, editing the same classes and running in the IDE. And I tested them all in debug mode for a start, therefore the results are very suspect, especially anything that would so obviously be optimised. The exercise where the tests are compiled in release mode is still valid I think. However you have highlighted that I may have made some image coding choices that may not be as advantageous as I originally thought. When I get a chance, I'll review the article and make some changes. Cheers Warrick Troft not lest ye be sponned on the nurg! (Milligan) Sign In·View Thread·PermaLink 10,000,000 repetitions quiensabe 2:05 29 May '08   To be honest, in real life I have never seen a program that's running 10,000,000 repetitions of a function. This speedtest is saying more about the behaviour of functions in the cache than about the functions. Why not write a test that measures each function once, twice, five times, 100, 1000, and eventualy 10000 times. You'll see different results. Sign In·View Thread·PermaLink 5.00/5 (3 votes) Re: 10,000,000 repetitions Warrick Procter 10:55 29 May '08   You are quite right, and thanks for your suggestion quiensabe. That would be a scientific approach for measuring the actual time taken - for that purpose this class is useless. All the tests were of a similar small 'code size' and will remain valid as a method of comparing which one probably takes the longest to execute, as long as procedure A and B either both execute within cache or both do not. This is not an exercise in benchmarking. The number of tests was purely a device to get a meaningful elapsed time. The only flaw, I can think of at the moment, is the difference an optimiser will make when compiling a 'release' version. Cheers, Warrick Troft not lest ye be sponned on the nurg! (Milligan) Sign In·View Thread·PermaLink

Last Visit: 20:22 7 Nov '09     Last Update: 20:22 7 Nov '09 1