Click here to Skip to main content
15,171,993 members
Articles / Programming Languages / C#
Technical Blog
Posted 15 Nov 2017

Tagged as


1 bookmarked

Testing and Validation CNTK Models using C#

Rate me:
Please Sign up or sign in to vote.
5.00/5 (5 votes)
15 Nov 2017CPOL1 min read
Once the model is built and Loss and Validation functions satisfy our expectation, we need to validate and test the model using the data which was not part of the training data set (unseen data).

…continuation from the previous post.

Once the model is built and Loss and Validation functions satisfy our expectation, we need to validate and test the model using the data which was not part of the training data set (unseen data). The model validation is very important because we want to see if our model is trained well, so that can evaluate unseen data approximately same as the training data. Otherwise, the model which cannot predict the output is called overfitted model. Overfitting can happen when the model was trained long enough that shows very high performance for the training data set, but for the testing data, evaluate bad results.

We will continue with the implementation from the prevision two posts, and implement model validation. After the model is trained, the model and the trainer are passed to the Evaluation method. The evaluation method loads the testing data and calculates the output using passed model. Then it compares calculated (predicted) values with the output from the testing data set and calculated the accuracy. The following source code shows the evaluation implementation.

private static void EvaluateIrisModel(Function ffnn_model, Trainer trainer, DeviceDescriptor device)
    var dataFolder = "Data";//files must be on the same folder as program
    var trainPath = Path.Combine(dataFolder, "testIris_cntk.txt");
    var featureStreamName = "features";
    var labelsStreamName = "label";

    //extract features and label from the model
    var feature = ffnn_model.Arguments[0];
    var label = ffnn_model.Output;

    //stream configuration to distinct features and labels in the file
    var streamConfig = new StreamConfiguration[]
            new StreamConfiguration(featureStreamName, feature.Shape[0]),
            new StreamConfiguration(labelsStreamName, label.Shape[0])

    // prepare testing data
    var testMinibatchSource = MinibatchSource.TextFormatMinibatchSource(
        trainPath, streamConfig, MinibatchSource.InfinitelyRepeat, true);
    var featureStreamInfo = testMinibatchSource.StreamInfo(featureStreamName);
    var labelStreamInfo = testMinibatchSource.StreamInfo(labelsStreamName);

    int batchSize = 20;
    int miscountTotal = 0, totalCount = 20;
    while (true)
        var minibatchData = testMinibatchSource.GetNextMinibatch((uint)batchSize, device);
        if (minibatchData == null || minibatchData.Count == 0)
        totalCount += (int)minibatchData[featureStreamInfo].numberOfSamples;

        // expected labels are in the mini batch data.
        var labelData = minibatchData[labelStreamInfo].data.GetDenseData<float>(label);
        var expectedLabels = labelData.Select(l => l.IndexOf(l.Max())).ToList();

        var inputDataMap = new Dictionary<Variable, Value>() {
            { feature, minibatchData[featureStreamInfo].data }

        var outputDataMap = new Dictionary<Variable, Value>() {
            { label, null }

        ffnn_model.Evaluate(inputDataMap, outputDataMap, device);
        var outputData = outputDataMap[label].GetDenseData<float>(label);
        var actualLabels = outputData.Select(l => l.IndexOf(l.Max())).ToList();

        int misMatches = actualLabels.Zip(expectedLabels, (a, b) => a.Equals(b) ? 0 : 1).Sum();

        miscountTotal += misMatches;
        Console.WriteLine($"Validating Model: Total Samples = {totalCount}, 
                                              Mis-classify Count = {miscountTotal}");

        if (totalCount >= 20)
    Console.WriteLine($"------TESTING SUMMARY--------");
    float accuracy = (1.0F - miscountTotal / totalCount);
    Console.WriteLine($"Model Accuracy = {accuracy}");

The implemented method is called in the previous Training method.

EvaluateIrisModel(ffnn_model, trainer, device);

As can be seen, the model validation has shown that the model predicts the data with high accuracy, which is shown in the following image:

This is the latest post in the series of blog posts about using Feed forward neural networks to train the Iris data using CNTK and C#.

The full source code for all three samples can be found here.

Filed under: .NET, C#, CNTK, CodeProject
Tagged: .NET, C#, CNTK, Code Project, CodeProject, Machine Learning


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

Bahrudin Hrnjica
Software Developer (Senior)
Bosnia and Herzegovina Bosnia and Herzegovina
Bahrudin Hrnjica holds a Ph.D. degree in Technical Science/Engineering from University in Bihać.
Besides teaching at University, he is in the software industry for more than two decades, focusing on development technologies e.g. .NET, Visual Studio, Desktop/Web/Cloud solutions.

He works on the development and application of different ML algorithms. In the development of ML-oriented solutions and modeling, he has more than 10 years of experience. His field of interest is also the development of predictive models with the ML.NET and Keras, but also actively develop two ML-based .NET open source projects: GPdotNET-genetic programming tool and ANNdotNET - deep learning tool on .NET platform. He works in multidisciplinary teams with the mission of optimizing and selecting the ML algorithms to build ML models.

He is the author of several books, and many online articles, writes a blog at, regularly holds lectures at local and regional conferences, User groups and Code Camp gatherings, and is also the founder of the Bihac Developer Meetup Group. Microsoft recognizes his work and awarded him with the prestigious Microsoft MVP title for the first time in 2011, which he still holds today.

Comments and Discussions

QuestionI do have a quick question? Pin
asiwel16-Nov-17 10:10
professionalasiwel16-Nov-17 10:10 
AnswerRe: I do have a quick question? Pin
Bahrudin Hrnjica17-Nov-17 1:59
professionalBahrudin Hrnjica17-Nov-17 1:59 
GeneralRe: I do have a quick question? Pin
asiwel17-Nov-17 5:07
professionalasiwel17-Nov-17 5:07 
GeneralRe: I do have a quick question? Pin
Bahrudin Hrnjica21-Nov-17 9:43
professionalBahrudin Hrnjica21-Nov-17 9:43 
GeneralRe: I do have a quick question? Pin
asiwel21-Nov-17 12:15
professionalasiwel21-Nov-17 12:15 
GeneralRe: I do have a quick question? Pin
Bahrudin Hrnjica22-Nov-17 1:43
professionalBahrudin Hrnjica22-Nov-17 1:43 
GeneralRe: Variable Learning rate Pin
asiwel22-Nov-17 8:26
professionalasiwel22-Nov-17 8:26 
There was no link associated with "here" but there was a link to "How to Setup Learning Rate .." at your blog and it was very helpful! Thank you.

The confusion (without clues in regular documentation) results essentially from "concept" unfamiliarity. "Sample" seems to mean data record (in most places) and "miniBatch" means a dataset (in what we have been calling batch mode) or a piece/slice/etc. of that set (in minibatch mode, I think) and "epoch" is an iteration. This is sort of like translating a foreign language!

More specifically, the confusion results from the fact that the CNTK.TrainingParameterScheduleDouble method has 6 overloads.

1) CNTK.TrainingParameterScheduleDouble(double value)
This just sets a fixed learning rate for the whole run and works for me, e.g.,
var learningRate = new TrainingParameterScheduleDouble(0.02);

2) CNTK.TrainingParameterScheduleDouble(double value, uint minibatchSize)
This works when using minibatches (does not work well in batch mode for me).
This is your var learningRate = new TrainingParameterScheduleDouble(0.2, 1);
The "1" is a minbatchsize (i.e., one whole minibatch or maybe for each individual minibatch?)
(Heck, it might be a range too?)

3) CNTK.TrainingParameterScheduleDouble(VectorPairSizeTDouble schedule, uint epochSize)

This one is interesting and it is the one you are using to control the Learning Rate in your blog.

I set my vector up like this:

PairSizeTDouble p1 = new PairSizeTDouble(100, 0.03);
PairSizeTDouble p2 = new PairSizeTDouble(200, 0.02);
PairSizeTDouble p3 = new PairSizeTDouble(1, 0.01);

var vp = new VectorPairSizeTDouble() { p1, p2, p3 };
var learningRatePerSample =
    new CNTK.TrainingParameterScheduleDouble(vp, 1);

(... probably should have renamed that var learningRatePerEpoch)

which says use a multiplier of 100 * an epochSize of 1 so that the first 100 epochs with use a learning rate of .03. The next 200 epochs will use .02 and so forth. Here in this overload, the "1" is an epochSize (or epoch range?) Sure would be nice to read about this stuff first!

But that's learning and experimenting with new code. (CNTK is full of this sort of thing.)

Actually, there is a more interesting (to me) question of why it is designed this way? It seems to me that the "learning rate" should be dynamic. By that, I mean the learner ought to automatically adjust it. (Maybe that's the MomentumSGD learner). How does the user know ahead of time what epochs to pick to change (let alone optimize) the rate? Strange?

Also, these algorithms don't seem to have a meaningful stopping criterion, other than the number of epochs? How does a user know an appropriate number of epochs? Why not set a logical stopping point, like when the "evaluation criterion" stops changing in the 4th or 6th decimal place? I guess I could write my code to do that ... and let the computer just run overnight! Smile | :)

EDIT: I did this easily. Just change the printTrainingProgress() method to return a float and check that against its previous value. The CrossEntropy "trainLossValue" is a much better choice for a stopping criterion than is the "evaluationValue," I think now.

Actually, IMHO, this code was probably written by people with far greater expertise (than I certainly have) and also far more familiarity with using very large data sets for training, validation, and testing. So one has to give the developers (almost Smile | :) ) all benefit of doubt.

modified 23-Nov-17 13:29pm.

GeneralRe: I do have a quick question and a solution Pin
asiwel21-Nov-17 14:59
professionalasiwel21-Nov-17 14:59 
GeneralHa! Feel some vindicated. Pin
asiwel25-Nov-17 12:12
professionalasiwel25-Nov-17 12:12 
QuestionReally Neat Project! Pin
asiwel15-Nov-17 17:50
professionalasiwel15-Nov-17 17:50 
AnswerRe: Really Neat Project! Pin
Bahrudin Hrnjica17-Nov-17 2:04
professionalBahrudin Hrnjica17-Nov-17 2:04 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.