Click here to Skip to main content
11,431,829 members (62,113 online)
Click here to Skip to main content

Convolutional Neural Network Workbench

, 21 Jul 2014 CPOL
Rate this:
Please Sign up or sign in to vote.
A workbench to create, train, and test convolutional neural networks against the MNIST and CIFAR-10 datasets

Download CNNWB Sources (21,3 MB)

Download Setup (7,5 MB)

Introduction

This article is about a framework in C# 4.0 that allows to create, train, and test convolutional neural networks against the MNIST and the CIFAR-10 dataset of 10 different natural objects. I initially based me on an article by Mike O'Neill on the The Code Project and gradually added new features that I've found interesting in research documents found on the internet. Dr. Yann LeCun's paper: Gradient-Based Learning Applied to Document Recognition is a great paper to get a better understanding of the principles of convolutional neural networks and the reason why they are so successful in the area of machine vision.

The Code

The main goal of this project was to build a more flexible and extendable managed version of Mike O'Neill's excellent C++ project. I've included and used the splendid WPF TaskDialog Wrapper from Sean A. Hanley, the Extended WPF Toolkit and for unzipping the CIFAR-10 dataset the open-source SharpDevelop SharpZipLib module. Visual Studio 2012/2013 and/or Windows 7 are the minimum requirements. I made maximal use of the parallel functionality offered in C# 4.0 by letting the user at all times choose how many logical cores are used in the parallel optimized code parts with a simple manipulation of a sliderbar next to the View combobox.

Using the Code

Here is the example code to construct a LeNet-5 network in code (see the InitializeDefaultNeuralNetwork() function in MainViewWindows.xaml.cs):

NeuralNetwork cnn = new NeuralNetwork
(DataProvider, "LeNet-5", 10, 0.8D, LossFunctions.MeanSquareError, 
DataProviderSets.MNIST, TrainingStrategy.SGDLevenbergMarquardt, 0.02D);
cnn.AddLayer(LayerTypes.Input, 1, 32, 32);
cnn.AddLayer(LayerTypes.Convolutional, ActivationFunctions.Tanh, 6, 28, 28, 5, 5);
cnn.AddLayer(LayerTypes.AveragePooling, ActivationFunctions.Tanh, 6, 14, 14, 2, 2);

bool[] maps = new bool[6 * 16] 
{
 true, false,false,false,true, true, true, false,false,true, true, true, true, false,true, true,
 true, true, false,false,false,true, true, true, false,false,true, true, true, true, false,true,
 true, true, true, false,false,false,true, true, true, false,false,true, false,true, true, true,
 false,true, true, true, false,false,true, true, true, true, false,false,true, false,true, true,
 false,false,true, true, true, false,false,true, true, true, true, false,true, true, false,true,
 false,false,false,true, true, true, false,false,true, true, true, true, false,true, true, true
};

cnn.AddLayer(LayerTypes.Convolutional, ActivationFunctions.Tanh, 16, 10, 10, 5, 5, new Mappings(maps));
cnn.AddLayer(LayerTypes.AveragePooling, ActivationFunctions.Tanh, 16, 5, 5, 2, 2);
cnn.AddLayer(LayerTypes.Convolutional, ActivationFunctions.Tanh, 120, 1, 1, 5, 5);
cnn.AddLayer(LayerTypes.FullyConnected, ActivationFunctions.Tanh, 10);
cnn.InitializeWeights(); 

Design View

In Design View you see how your network is defined and get a good picture of the current distribution of weight values in all the layers concerned.

Training View

In Training View you obiviously train the network. The 'Play' button gives you the 'Select Training Parameters' dialog where you define the training parameters. The 'Training Scheme Editor' button gives you the possibility to make training schemes to experiment with. At any time the training can be paused or aborted. If the training is pauzed you can save its weights. The 'Star' button will forget (random reset) all the learned weight values in each layer.

 

Testing View

In Testing View you get a better picture of the testing (or training) samples which are not recognized correctly.

Calculate View

In Calculate View we test a single testing or training sample with the desired properties and get a graphical view of all the output values in every layer.

Final Words

I would love to see a GPU integration for offloading the highly parallel task of learning the neural network. I made an attempt to use a simple MVVM structure in this WPF application. In the Model folder you will find the NeuralNetwork and DataProvider class which provide all the neural network code and deals with loading and providing the necessary training and testing samples. Also a NeuralNetworkDataSet class is used to load and save neural network definitions. The View folder contains four different PageViews and a global PageView which acts as the container for all the different views (Design, Training, Testing and Calculate). Hope there's someone out there who can actually use the code and improve on it. Extend it with an unsupervised learning stage, (encoder/decoder construction), implement better loss-functions, more training strategies (conjugate gradient, l-bgfs, ...), more datasets, better activation fuctions, ...

History

  • 1.0.3.7: (07-08-14) (updated download of Setup & Sources on 07-21-14)
  • BugFix: slow speed resolved in Testing View
  • Added SGDLevenbergMarquardtModA training strategy. This can be used with a softmax output layer
  • Posibility to save the weights while training. Just click on Pause and then Save/Save as...
  • Various smaller fixes and optimizations
  • 1.0.3.6: (05-02-14)
  • Choice between four Training Strategies:
    • SGDLevenbergMarquardt
    • SGDLevenbergMarquardtMiniBatch
    • SGD
    • SGDMiniBatch
  • BugFix: Derivative of ReLU's activation functions
  • Added SoftSign activation function
  • Overall 20% more training performance over previous version
  • Faster binary save of the network weights
  • Various smaller fixes and optimizations
  • 1.0.3.5: (03-13-14)
  • StochasticPooling and L2Pooling layers now correctly implemented
  • Native C++ implementation + managed C++/CLI wrapper (removed from now)
  • 1.0.3.4: (12-10-13)
  • Support for Stochastic Pooling layers
  • Much faster binary load and save of a cnn
  • ReLU activation function now working as expected
  • 1.0.3.3: (11-26-2013)
  • Bugfix: Download datasets now working as expected
  • Bugfix: Softmax function corrected
  • Bugfix: DropOut function corrected
  • 1.0.3.2: (11-15-2013)
  • Bugfix: Local layer and Convolutional layer now works properly
  • Bugfix: Cross Entropy loss now works better (in combination with a SoftMax activation function in a final fully connected layer)
  • Added LeCun's standard LeNet-5 training scheme
  • 1.0.3.1: (11-09-2013)
  • Now the last min/max display preference is saved
  • Added some extra predefined training parameters and schemes
  • Bugfix: Average error not showing correctly after switching between neural networks with a different objective function
  • Bugfix: Sample not always showing correctly in calculate view
  • Bugfix: The end result in testing view is not displaying the correct values
  • 1.0.3.0: (11-06-2013)
  • Supports dropout
  • Supports the Local layer type (Same as the convolution layer but with non-shared kernel weights)
  • Supports padding in the Local and Convolution layers (gives ability to create deep networks)
  • Supports overlapping receptive fields in the pooling layers
  • Supports weightless pooling layers
  • Supports all the commonly used activation functions
  • Supports Cross Entropy objective function in combination with a SoftMax activation function in the output layer
  • Ability to specify a density percentage to generate mappings between layers far more easily
  • Improved DataProvider class with much reduced memory footprint and a common logical functionality shared across all datasets (easier to add datasets)
  • Much improved UI speed and general functionality

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Filip D'haene
Software Developer
Belgium Belgium
No Biography provided

Comments and Discussions

 
GeneralRe: ActivationFunctions Pin
Дарья Прокурат10-May-12 7:24
memberДарья Прокурат10-May-12 7:24 
GeneralRe: ActivationFunctions Pin
Filip D'haene10-May-12 7:50
memberFilip D'haene10-May-12 7:50 
GeneralRe: ActivationFunctions Pin
Дарья Прокурат10-May-12 8:41
memberДарья Прокурат10-May-12 8:41 
AnswerRe: ActivationFunctions Pin
Filip D'haene10-May-12 9:01
memberFilip D'haene10-May-12 9:01 
Questioncrashes while downloading training images Pin
Sperneder Patrick5-May-12 9:33
memberSperneder Patrick5-May-12 9:33 
AnswerRe: crashes while downloading training images Pin
Filip D'haene5-May-12 10:01
memberFilip D'haene5-May-12 10:01 
GeneralRe: crashes while downloading training images Pin
Sperneder Patrick8-May-12 4:25
memberSperneder Patrick8-May-12 4:25 
AnswerRe: crashes while downloading training images Pin
Filip D'haene8-May-12 5:13
memberFilip D'haene8-May-12 5:13 
GeneralRe: crashes while downloading training images Pin
Sperneder Patrick10-May-12 8:30
memberSperneder Patrick10-May-12 8:30 
AnswerRe: crashes while downloading training images [modified] Pin
Filip D'haene10-May-12 9:16
memberFilip D'haene10-May-12 9:16 
GeneralRe: crashes while downloading training images Pin
Sperneder Patrick10-May-12 10:39
memberSperneder Patrick10-May-12 10:39 
AnswerRe: crashes while downloading training images Pin
Filip D'haene14-May-12 19:49
memberFilip D'haene14-May-12 19:49 
QuestionDownload links do not work... please check Pin
fawzi_masri10-Apr-12 11:42
memberfawzi_masri10-Apr-12 11:42 
AnswerRe: Download links do not work... please check Pin
Filip D'haene10-Apr-12 11:50
memberFilip D'haene10-Apr-12 11:50 
This is a CodeProject website issue that for some unknown reason don't let you
download the files you want. I've changed nothing to my article in the mean time.
downloading was working fine a couple of hours ago! D'Oh! | :doh:

I hope it will be fixed soon!
GeneralRe: Download links do not work... please check Pin
fawzi_masri10-Apr-12 11:59
memberfawzi_masri10-Apr-12 11:59 
QuestionDownload links do not seem to work Pin
Bill Dickinson9-Apr-12 15:58
memberBill Dickinson9-Apr-12 15:58 
AnswerRe: Download links do not seem to work Pin
Filip D'haene10-Apr-12 1:47
memberFilip D'haene10-Apr-12 1:47 
GeneralRe: Download links do not seem to work Pin
Bill Dickinson10-Apr-12 3:45
memberBill Dickinson10-Apr-12 3:45 
QuestionMy vote of 5 Pin
Vietdungiitb14-Mar-12 18:55
memberVietdungiitb14-Mar-12 18:55 
AnswerRe: My vote of 5 Pin
Filip D'haene23-Mar-12 11:01
memberFilip D'haene23-Mar-12 11:01 
QuestionI bookmarked this article Pin
Keith.Badeau13-Mar-12 17:14
memberKeith.Badeau13-Mar-12 17:14 
AnswerRe: I bookmarked this article Pin
Filip D'haene23-Mar-12 11:04
memberFilip D'haene23-Mar-12 11:04 
Generalvery cool Pin
dave.dolan25-Feb-12 19:27
memberdave.dolan25-Feb-12 19:27 
GeneralRe: very cool Pin
Filip D'haene26-Feb-12 4:05
memberFilip D'haene26-Feb-12 4:05 
GeneralMy vote of 5 Pin
ErrolErrol13-Feb-12 20:18
memberErrolErrol13-Feb-12 20:18 
GeneralRe: My vote of 5 Pin
Filip D'haene14-Feb-12 4:20
memberFilip D'haene14-Feb-12 4:20 
QuestionTraining Pin
ibnkhaldun31-Jan-12 11:12
memberibnkhaldun31-Jan-12 11:12 
AnswerRe: Training Pin
Filip D'haene2-Feb-12 12:18
memberFilip D'haene2-Feb-12 12:18 
GeneralMy vote of 5 Pin
pothiq24-Jan-12 17:52
memberpothiq24-Jan-12 17:52 
AnswerRe: My vote of 5 [modified] Pin
Filip D'haene25-Jan-12 13:13
memberFilip D'haene25-Jan-12 13:13 
Questionhow to use alternative dataset? Pin
maryam_6818-Jan-12 11:08
membermaryam_6818-Jan-12 11:08 
AnswerRe: how to use alternative dataset? Pin
Filip D'haene18-Jan-12 13:53
memberFilip D'haene18-Jan-12 13:53 
GeneralRe: how to use alternative dataset? Pin
maryam_6819-Jan-12 2:22
membermaryam_6819-Jan-12 2:22 
AnswerRe: how to use alternative dataset? [modified] Pin
Filip D'haene19-Jan-12 7:04
memberFilip D'haene19-Jan-12 7:04 
GeneralRe: how to use alternative dataset? Pin
maryam_6819-Jan-12 20:32
membermaryam_6819-Jan-12 20:32 
AnswerRe: how to use alternative dataset? Pin
Filip D'haene20-Jan-12 4:50
memberFilip D'haene20-Jan-12 4:50 
QuestionTraining with text Pin
ibnkhaldun9-Jan-12 22:17
memberibnkhaldun9-Jan-12 22:17 
AnswerRe: Training with text Pin
Filip D'haene10-Jan-12 12:31
memberFilip D'haene10-Jan-12 12:31 
GeneralMy vote of 5 Pin
hoernchenmeister9-Jan-12 2:39
memberhoernchenmeister9-Jan-12 2:39 
GeneralRe: My vote of 5 Pin
Filip D'haene10-Jan-12 12:09
memberFilip D'haene10-Jan-12 12:09 
GeneralMy vote of 5 Pin
Member 43208442-Jan-12 10:21
memberMember 43208442-Jan-12 10:21 
GeneralRe: My vote of 5 Pin
Filip D'haene4-Jan-12 8:04
memberFilip D'haene4-Jan-12 8:04 
GeneralMy vote of 5 Pin
Vllado229-Dec-11 3:13
memberVllado229-Dec-11 3:13 
GeneralRe: My vote of 5 Pin
Filip D'haene31-Dec-11 10:43
memberFilip D'haene31-Dec-11 10:43 
GeneralMy vote of 5 Pin
Rex_kz21-Dec-11 20:51
memberRex_kz21-Dec-11 20:51 
GeneralRe: My vote of 5 Pin
Filip D'haene22-Dec-11 4:43
memberFilip D'haene22-Dec-11 4:43 
GeneralMy vote of 5 Pin
dovydasm10-Dec-11 7:32
memberdovydasm10-Dec-11 7:32 
GeneralRe: My vote of 5 Pin
Filip D'haene10-Dec-11 13:56
memberFilip D'haene10-Dec-11 13:56 
GeneralMy vote of 5 Pin
Kanasz Robert18-Nov-11 22:34
memberKanasz Robert18-Nov-11 22:34 
GeneralRe: My vote of 5 Pin
Filip D'haene19-Nov-11 14:34
memberFilip D'haene19-Nov-11 14:34 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.150428.2 | Last Updated 21 Jul 2014
Article Copyright 2010 by Filip D'haene
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid