Simple paradigm of scientific software testing

• Download Source code - 3.32 MB
• Download helper files 96 KB
• Download chart test 2.9 KB
• Download test of orbit determination 235 KB
• Download test of chart reconstruction 216 KB
• Download test of orbit determination and sounds 216 KB

Useful links

• Universal engineering software homepage
• Visual C# Express
• Weifen Luo homepage

1. Introduction

Science and engineering software can have very complicated logics. Unit testing of this software cannot test every execution path of program. This article is devoted to simple paradigm of scientific software testing. This paradigm requires saving calculation result. Testing procedure compares saved data with (new) results of calculation. If modification is correct both results should coincide. This article contains elegant code implementation of this paradigm.

2. Unit testing

Unit testing provides rapid fixing of simple bugs. I think that good testing should include both

• Unit testing
• Complicated testing

This section describes examples of unit testing of formula editor. Any advanced (not only) scientific software operates with formulae. Described here framework contains formula editor. Samples of formulae are presented below. Following picture contains four arithmetic operations:

These and other elementary formulae can be easy tested. Following snippet contains test of these formulae.

/// <summary>
/// Test of double binary
/// </summary>
[TestMethod]
public void TestMethodBinaryDouble()
{
    double type = 0;
    Dictionary<string, object> d = new Dictionary<string, object>()
    {
        {"x", type},
        {"y", type}
    };
    FormulaEditor.VariableDetectors.ExtendedDictionaryVariableDetector det =
        new FormulaEditor.VariableDetectors.ExtendedDictionaryVariableDetector(d);
    FormulaEditor.Interfaces.IFormulaObjectCreator cr = ExtendedFormulaCreator.GetCreator(det);
    XmlDocument doc = new XmlDocument();

    doc.LoadXml(Properties.Resources.BinaryDouble);             // Reading of formulae from XML file
    List<ObjectFormulaTree> l = new List<ObjectFormulaTree>();
    XmlNodeList nl = doc.DocumentElement.ChildNodes;
    foreach (XmlElement e in nl)
    {
        MathFormula f = MathFormula.FromString(sizes, e.OuterXml);   // Creation of formula objects
        ObjectFormulaTree tree = ObjectFormulaTree.CreateTree(f.FullTransform(null), cr);
        l.Add(tree);
    }
    Dictionary<Func<double, double, double>, object[]> dic = new Dictionary<Func<double, double, double>, object[]>()   // Dictionary of tested
    {                                                         // binary functions
        {(double x, double y) => { return x + y;} , new object[]{new object[] {"plus",  l[0]}}},
        {(double x, double y) => { return x - y;} , new object[]{new object[] {"minus",  l[1]}}},
        {(double x, double y) => { return x * y;} , new object[]{new object[] {"mult1",  l[2]}, new object[] {"mult2",  l[3]}}},
        {(double x, double y) => { return x / y;} , new object[]{new object[] {"frac",  l[4]}}},
        {Math.Pow , new object[]{new object[] {"pow",  l[5]}}},
        {Math.Atan2 , new object[]{new object[] {"atan2",  l[6]}}},
    };
    FormulaEditor.Interfaces.ITreeCollectionProxy proxy = l.ToArray().CreateProxy();   // Creation of proxy code
    foreach (Func<double, double, double> func in dic.Keys)
    {
        object[] o = dic[func];
        foreach (object[] ob in o)
        {
            ObjectFormulaTree tree = ob[1] as ObjectFormulaTree;
            string fs = ob[0] + "";
            GetValue g = proxy[tree];                                       // Proxy functions
            for (int i = 0; i < 10; i++)
            {
                double x = 0.007 + 0.07 * i;
                det["x"] = x;
                for (int j = 0; j < 10; j++)
                {
                    double y = 0.34 + 0.031 * j;
                    det["y"] = y;
                    double a = func(x, y);                                      // Calculation of functions
                    object b = tree.Result;                                     // Calculation of tree
                    Assert.AreEqual(a, b);                                      // Comparation of results
                    proxy.Update();
                    object c = g();                                             // Calculation of proxy code
                    Assert.AreEqual(a, c);                                      // Comparation of results
                }
            }
        }
    }
}

This test is very clear. It compares calculatinos obtained from formulae end C# functions.

3. Complicated testing

Unfortunately unit testing is not panacea. Unit testing cannot embrace all complicated tasks. For example determination of orbits of artificial Earth satellites contains lots of elements which are presented below:

Main idea of testing such use cases is recording of results. Testing contains comparation of recorded and calculated results. If calculation framework is scalable then testing should have abstract level. Following code contains this absract level.

/// <summary>
/// Test
/// </summary>
public interface ITest
{
    /// <summary>
    /// Tests collection of components
    /// </summary>
    /// <param name="collection">Collection of components</param>
    /// <returns>Test result</returns>
    object this[IComponentCollection collection]
    {
        get;
    }
}

This interface contains one method which tests collection of components and returns tests results.

3.1 Elementary tests

3.1.1 Test of time dependent function

A lot of engineering tasks contains calculation of time dependent functions. Following picture presents calculation of time dependent function

Input signal is transformed by transfer function. Transformation result is time dependent function which is indicated by red curve on Chart component. Properties of Input are presented below:

Above picture contains formula of input signal with Dirac δ function. The Transfer function object has following properties:

It contains formula of transfer function.

Result of transformation is presented below:

Output function has jump at time= 0.5. This jump is caused by Dirac δ function.

Testing of calculation of time dependent function can be performed by following way:

• Saving of time dependent functions;
• Comparing of new calculation with saved result.

Following snippet contains implementation of this test.

  /// <summary>
 /// Test of time dependent function
 /// </summary>
[Serializable()]
 class LocalChart : ITest, ISerializable
 {

     #region Fields

     /// <summary>
     /// Saved series
     /// </summary>
     Dictionary<string, LocalSeries> series = new Dictionary<string, LocalSeries>();

     string name;

     string argument;

     double start;

     double step;

     int stepCount;

     string[] values = null;



     #endregion

     #region Ctor

     internal LocalChart(string name, double start, double step, int stepCount, string argument, string[] values)
     {
         this.name = name;
         this.start = start;
         this.step = step;
         this.stepCount = stepCount;
         this.argument = argument;
         this.values = values;
     }


     /// <summary>
     /// Loads saved time dependent functions
     /// </summary>
     /// <param name="info">Serialization info</param>
     /// <param name="context">Streaming context</param>
     private LocalChart(SerializationInfo info, StreamingContext context)
     {
         name = info.GetString("Name");
         series = info.GetValue("Series", typeof(Dictionary<string, LocalSeries>)) as Dictionary<string, LocalSeries>;
         argument = info.GetString("Argument");
         start = info.GetDouble("Start");
         step = info.GetDouble("Step");
         stepCount = info.GetInt32("StepCount");
     }

     #endregion

     #region ISerializable Members

     /// <summary>
     /// Saves time dependent functions
     /// </summary>
     /// <param name="info">Serialization info</param>
     /// <param name="context">Streaming context</param>
     void ISerializable.GetObjectData(SerializationInfo info, StreamingContext context)
     {
         info.AddValue("Name", name);
         info.AddValue("Series", series, typeof(Dictionary<string, LocalSeries>));
         info.AddValue("Argument", argument);
         info.AddValue("Start", start);
         info.AddValue("Step", step);
         info.AddValue("StepCount", stepCount);
     }

     #endregion

     #region ITest Members


     /// <summary>
     /// Tests collection of components
     /// </summary>
     /// <param name="collection">Collection of components</param>
     /// <returns>Test result</returns>
     object ITest.this[IComponentCollection collection]
     {
         get
         {
             IDesktop desktop = collection as IDesktop;
             IDataConsumer dataconsumer = desktop[name] as IDataConsumer;
             Dictionary<string, DataPerformer.Basic.Series> d = GetSeries(collection); // Calculation of time dependent function
             List<string> l = new List<string>();
             foreach (string s in d.Keys)
             {
                 if (!series[s].Compare(d[s])) // Comparation of test results
                 {
                     l.Add("Different series values. Object - " + name + ". Series - " + s + ".");
                 }
             }
             if (l.Count == 0)
             {
                 return null;
             }
             return l;
         }
     }

     #endregion

     #region Members

     /// <summary>
     /// Crates tests
     /// </summary>
     /// <param name="collection">Collection of components</param>
     internal void Create(IComponentCollection collection)
     {
         Dictionary<string, DataPerformer.Basic.Series> d = GetSeries(collection);
         series.Clear();
         foreach (string key in d.Keys)
         {
             series[key] = new LocalSeries(d[key]);
         }
     }

     Dictionary<string, DataPerformer.Basic.Series> GetSeries(IComponentCollection collection)
     {
         IDesktop desktop = collection as IDesktop;
         IDataConsumer dataConsumer = desktop.GetObject(name) as IDataConsumer;
         string[] ss = (values == null) ? series.Keys.ToArray() : values;
         return dataConsumer.GetSeries(start, step, stepCount,
             argument, ss);
     }

     internal string Name
     {
         get
         {
             return name;
         }
     }

     #endregion
 }

Essential part of this code is presented below

 /// <summary>
/// Tests collection of components
/// </summary>
/// <param name="collection">Collection of components</param>
/// <returns>Test result</returns>
object ITest.this[IComponentCollection collection]
{
    get
    {
        IDesktop desktop = collection as IDesktop;
        IDataConsumer dataconsumer = desktop[name] as IDataConsumer;
        Dictionary<string, DataPerformer.Basic.Series> d = GetSeries(collection); // Calculation of time dependent function
        List<string> l = new List<string>();
        foreach (string s in d.Keys)
        {
            if (!series[s].Compare(d[s])) // Comparation of test results
            {
                l.Add("Different series values. Object - " + name + ". Series - " + s + ".");
            }
        }
        if (l.Count == 0)
        {
            return null;
        }
        return l;
    }
}

• Download test of chart - 2.85 KB

Our application can be started with test support. If application is started with -tc parameter of command line (Aviation.Wpf3D.Advanced.exe -tc) then it supports tests i.e. any saving initialize request of test parameters. User interface of test request is presented below:

This picture means that current scenario is saved with test of Chart component. Saving of this scenario also saves LocalChart object

Loading of scenario also loads and performs test of Chart object.

3.1.2 Test of test

Above chapter contains tests. But we should test work test whether test works. A test without bug is not test. If we have no but we should make it. Following snippet contains artificial bug.

     //AddXY(Converter.ToDouble(x()()), Converter.ToDouble(y()()));

 /*!!! Test of test   (Artificial bug) */
AddXY(Converter.ToDouble(x()()), Converter.ToDouble(y()()) + 0.0001);

 //End test of test*/

Adding of 0.0001 is necessary calculation bug. If we load above situation with this bug then we obtain following test report:

3.1.3 Test of nonlinear regression

Nonlinear regression in statistics is the problem of fitting a model.

to multidimensional x, y data, where f is a nonlinear function of x, with regression parameter θ. Vector ε=(ε₁,..., ε_n) is called vector of residuals which. Regression algorithm defines such vector θ residual parameter σ²=(ε₁²+...+ε_n²)/n is minimal. Regression algorithm iteratively estimates new value of θ. During one iteration algorithm defines new value of θ and new value of σ². Test of nonlinear regression implies recording number of iterations and residual parameter. Following code snippet contains implementation of this test

/// <summary>
 /// Test of nonlinear regression
 /// </summary>
 [Serializable()]
 class RegressionTest : ITest, ISerializable
 {
     #region Fields

     /// <summary>
     /// Name of component on desktop
     /// </summary>
     string name;

     /// <summary>
     /// Number of iterations
     /// </summary>
     int number;

     /// <summary>
     /// Residual parameter
     /// </summary>
     double value;

     #endregion

     #region Ctor

     /// <summary>
     /// Constructor
     /// </summary>
     /// <param name="name">Name of component on desktop</param>
     /// <param name="number">Number of iterations</param>
     internal RegressionTest(string name, int number)
     {
         this.name = name;
         this.number = number;
     }


     /// <summary>
     /// Deserialization constructor
     /// </summary>
     /// <param name="info">Serialization info</param>
     /// <param name="context">Streaming context</param>
     private RegressionTest(SerializationInfo info, StreamingContext context)
     {
         name = info.GetString("Name");
         number = info.GetInt32("Number");
         value = info.GetDouble("Value");
     }


     #endregion

     #region ITest Members

     /// <summary>
     /// Tests collection of components
     /// </summary>
     /// <param name="collection">Collection of components</param>
     /// <returns>Test result</returns>
     object ITest.this[IComponentCollection collection]
     {
         get
         {
             if (GetValue(collection) != value)  // If calculated value of residual parameter is not equal
                                                     // is not equal to saved value of residual parameter
             {
                 return "Different regression values. Object - " + name;  // Then method returns error message
             }
             return null;        // Null means absence of error
         }
     }

     #endregion

     #region ISerializable Members

     /// <summary>
     /// ISerializable interface implementation
     /// </summary>
     /// <param name="info">Serialization info</param>
     /// <param name="context">Streaming context</param>
     void ISerializable.GetObjectData(SerializationInfo info, StreamingContext context)
     {
         info.AddValue("Name", name);
         info.AddValue("Number", number);
         info.AddValue("Value",value);
     }

     #endregion

     #region Members

     internal int Number
     {
         get
         {
             return number;
         }
     }

     internal string Name
     {
         get
         {
             return name;
         }
     }

     internal void Create(IComponentCollection collection)
     {
         value = GetValue(collection);
     }

     /// <summary>
     /// Calculates value of residual parameter
     /// </summary>
     /// <param name="collection">Collection of objects</param>
     /// <returns>Residual parameter</returns>
     double GetValue(IComponentCollection collection)
     {
         IDesktop desktop = collection as IDesktop;
         AliasRegression reg = desktop.GetObject(name) as AliasRegression; // Regression component
         for (int i = 0; i < number; i++)
         {
             reg.FullIterate();                  // Iteration cylce
         }
         return reg.SquareResidual;              // returns residual parameter
     }

     #endregion

 }

This class is serializable. This test object can be saved to stream. Following snippet contains essential part of above implementation

 /// <summary>
/// Tests collection of components
/// </summary>
/// <param name="collection">Collection of components</param>
/// <returns>Test result</returns>
object ITest.this[IComponentCollection collection]
{
    get
    {
        if (GetValue(collection) != value)  // If calculated value of residual parameter is not equal
                                                // is not equal to saved value of residual parameter
        {
            return "Different regression values. Object - " + name;  // Then method returns error message
        }
        return null;        // Null means absence of error
    }
}

/// <summary>
/// Calculates value of residual parameter
/// </summary>
/// <param name="collection">Collection of objects</param>
/// <returns>Residual parameter</returns>
double GetValue(IComponentCollection collection)
{
    IDesktop desktop = collection as IDesktop;
    AliasRegression reg = desktop.GetObject(name) as AliasRegression; // Regression component
    for (int i = 0; i < number; i++)
    {
        reg.FullIterate();                  // Iteration cylce
    }
    return reg.SquareResidual;              // returns residual parameter
}

In fact this test compares saved value of σ² with calculated one.

3.1.4 Example of regression test. Determination of Orbits of Artificial Satellites

Determination of orbits of artificial satellites is considered in my previous article. Following picture contains elements of orbit determination

Main component is Prosessor which performs determination of orbit.

If we try to save this scenario then obtain following request of test.

This picture has following meaning

• The Processor component is tested
• Count of iteration is 3

If we set to Order default value (=0) then test of processor will be dropped. Saving of this scenario also saves RegressionTest object which contains number of iterations and corresponding σ². If we load saved file then Processor should perform 3 iterations and compares saved σ² with calculated one. Testing of this test can be performed by following artificial bug.

th.UpdateChildrenData(); ///TEST!!! Comment this string for artificial bug of orbit determination

In case of this artificial following test report will occur.

• Download orbit determination test - 230.25 KB

This sample requires installation of helper files. Following archive should be unpacked to directory of Aviation.Wpf3D.Advanced.exe file.

• Download helper files - 94.08 KB

3.2 Integrated tests

Complicated engineering problems as rule contains a set of elements which need tests. In general test result depends on order of testing. For example time dependency can depend on estimated by regression parameters. In this case result of time dependency test depends on whether estimation was performed before test or after it. In this chapter some examples of integrated tests is considered.

3.2.1 Chart reconstruction

Let us consider problem of chart reconstruction from image. Sample of image is presented below:

We would like to find math function which corresponds to this image. We also suppose that image contains errors of measurements. Following picture contains elements of this task.

First of all we need scale this image. Moreover X axis and Y axis are not always horizontal and vertical respectively. So one need image scaling in wide sense (scaling with rotation). The image is marked by colored points for scaling. Zero point is scaled by red color. Top and right image points are marked by green and blue color respectively. Detection of these points is performed by following steps. First of all image is filtered. Following component represents filter or red color

In this formula r, g, b are weights of red, green and blue color respectively. Colors are normed by following way;

• |r|≤1;
• |g|≤1;
• |b|≤1.

Above formula makes black point on place of red point. Other pixels are white. Filtration result is contained in Zero Image component. This component is connected to Zero Selection component which transforms image to chart. Properties of this chart are presented below:

This selection enables us to detect coordinates of red point. Nonlinear regression component Zero Regression is used for this purpose (Zero Regression is connected to Zero Selection). Similraly using filters and regression components one can define coordinates of top and left point. Top Processor and Right Processor are being used for this purpose. Usage of filter which defines black pixels enable us define following chart.

After these actions one can define math dependency of chart. Regression formula is presented below:

Parameters a, b, c, d, k should be estimated. Main Processor is nonlinear regression component which performs estimation. Following chart contains calculated regression result (red curve) and a set of approximated points (blue color).

Testing of this scenario should be ordered by following way:

The Order column means order of test:

• Download chart reconstruction test - 211.84 KB

3.2.2 Test of sounds

Any virtual reality software should support sounds. Also it should support testing of sounds. Let us consider a sample of sound testing. This sample contains following tasks:

• Determination of orbits of artificial satellites;
• Calculation of motion parameters;
• Blowing of words "up" and "down" at virtual intersection of equator. "Up" is phonated if satellite is moved from South hemisphere to North one and "down" is phonated otherwise.

Following picture represents components of this sample

Left part of above picture contains determination of orbit, right one is related to sounds. Following picture represents right part in details. This picture has following meaning

Condition parameter	Meaning	Sound file
Sound formula.Formula_1 = true	Satellite intersects equator plane from South to North	up.wav
Sound formula.Formula_2 = true	Satellite intersects equator plane from North to South	down.wav

Testing of sounds is performed by following way. Preparation of test imply saving of dictionary of sounds. Dictionary keys are points of sound times, dictionary values are names of sound files. Implementation of this test is presented below:

 /// <summary>
/// Test of sound
/// </summary>
[Serializable()]
class SoundTest : ISerializable, ITest
{

    #region Fields

    string name;

    double start;

    double step;

    int stepCount;

    /// <summary>
    /// Dictionary of sounds
    /// Keys are instants of sounds
    /// Values are sound filenames
    /// </summary>
    Dictionary<double, string> soundResults = new Dictionary<double, string>();


    #endregion

    #region Ctor

    /// <summary>
    /// Constructor
    /// </summary>
    /// <param name="collection">Collection of components</param>
    /// <param name="name">Sound component name</param>
    /// <param name="start">Start time</param>
    /// <param name="step">Step</param>
    /// <param name="stepCount">Count of steps</param>
    internal SoundTest(IComponentCollection collection,
        string name, double start, double step, int stepCount)
    {
        this.name = name;
        this.start = start;
        this.step = step;
        this.stepCount = stepCount;
        soundResults = GetSounds(collection);
    }

    /// <summary>
    /// Deserialization constructor
    /// </summary>
    /// <param name="info">Serialization info</param>
    /// <param name="context">Streaming context</param>
    private SoundTest(SerializationInfo info, StreamingContext context)
    {
        name = info.GetString("Name");
        start = info.GetDouble("Start");
        step = info.GetDouble("Step");
        stepCount = info.GetInt32("StepCount");
        soundResults = info.GetValue("Sounds", typeof(Dictionary<double, string>)) as Dictionary<double, string>;
    }



    #endregion

    #region ISerializable Members

    /// <summary>
    /// ISerializable interface implementation
    /// </summary>
    /// <param name="info">Serialization info</param>
    /// <param name="context">Streaming context</param>
    void ISerializable.GetObjectData(SerializationInfo info, StreamingContext context)
    {
        info.AddValue("Name", name);
        info.AddValue("Start", start);
        info.AddValue("Step", step);
        info.AddValue("StepCount", stepCount);

        // Dictionary of sounds saving
        info.AddValue("Sounds", soundResults, typeof(Dictionary<double, string>));
    }

    #endregion

    #region ITest Members

    /// <summary>
    /// Testing
    /// </summary>
    /// <param name="collection">Collection of objects</param>
    /// <returns>Tesing result</returns>
    object ITest.this[IComponentCollection collection]
    {
        get
        {
            List<string> l = new List<string>();
            Dictionary<double, string> d = GetSounds(collection); // Calculates dictionary of sounds


            // Comparation of saved dictionary with calculated one
            if (d.Count != soundResults.Count)
            {
                l.Add("Different number of sounds");
            }
            foreach (double key in d.Keys)
            {
                if (!soundResults.ContainsKey(key))
                {
                    l.Add("Illegal time: " + key);
                }
                else
                {
                    string s = d[key];
                    if (!s.Equals(soundResults[key]))
                    {
                        l.Add("Illegal sound '" + s + "' Time = " + key);
                    }

                }
            }
             if (l.Count == 0)
            {
                return null;
            }
            // Returns list of errors
            return l;
        }
   }

    #endregion

    #region Members

    /// <summary>
    /// Name
    /// </summary>
    internal string Name
    {
        get
        {
            return name;
        }
    }


    /// <summary>
    /// Calculates dictionary of sounds
    /// </summary>
    /// <param name="collection">Collection of objects</param>
    /// <returns>Dictionary of sounds</returns>
    Dictionary<double, string> GetSounds(IComponentCollection collection)
    {
        IDesktop desktop = collection as IDesktop;
        Dictionary<double, string> d = new Dictionary<double, string>();
        SoundCollection coll = desktop.GetObject(name) as SoundCollection;
        IDataConsumer dc = coll;
        ITimeMeasureProvider p = DataPerformer.StaticExtension.Factory.TimeProvider;
        Action<string> act = (string s) => { d[p.Time] = s; };
        coll.PlaySound += act;
        dc.PerformFixed(start, step, stepCount, p,
           DifferentialEquationProcessor.Processor, desktop, 0, () => { });
        coll.PlaySound -= act;
        return d;
    }

    #endregion
}

If application is started with test support then test component have additional tab page of test's parameters.

This page contains start time, count of steps and step of sound test. Test result depends on operation order. Dictionary of sounds depends on determination of orbit. If we would like save above scenario then we obtain following request of test.

This request has following meaning

• First of all test performs orbit determination with 3 iterations;
• Secondly test of sounds is performed.

• Download test of sounds and orbit determination - 610.82 KB

This sample requires installation of helper files. Following archive should be unpacked to directory of Aviation.Wpf3D.Advanced.exe file.

• Download helper files - 94.08 KB

4. Points of interests

I find that testing is like to mammillae of man. Testing is not useful and it do not have beauty. However without advanced software is impossible without testing . Testing is inevitable misfortune.

5. Acknowlegment

I would like to acknowledge Weifen Luo for his very useful software. His software is used in my projects.