Click here to Skip to main content
11,493,937 members (60,374 online)
Click here to Skip to main content

Convolutional Neural Network MNIST Workbench

, 8 Nov 2011 CPOL 259.5K 27.2K 300
Rate this:
Please Sign up or sign in to vote.
A workbench to create, train, and test convolutional neural networks against the MNIST dataset.
This is an old version of the currently published article.

Introduction

This article is about a Microsoft C# 4.0 WPF implementation of a framework that allows to create, train, and test convolutional neural networks against the MNIST dataset of handwritten digits. There is a magnificent article by Mike O'Neill on the The Code Project about the same subject. Without his great article and C++ demo code, this project wouldn't exist. I also relied heavily on Dr. Yann LeCun's paper: Gradient-Based Learning Applied to Document Recognition to understand more about the principles of convolutional neural networks and the reason why they are so successful in the area of machine vision. Mike O'Neill uses Patrice Simard's implementation where the subsampling step is integrated in the structure of the convolutional layer itself. Dr. Yann LeCun uses in his LeNet-5 a separate subsampling step, and also uses non-fully connected layers. The framework presented allows to use all types of layers, and has an additional Max-Pooling layer that you can use instead of plain Average-Pooling. The default squashing function used is tanh() and the value to train for is set to 0.8 because it is the value at the curvature of the second derivative of the used non-linearity so there is less saturation. The input images are all normalised (-1/1), and the input layer is at a fixed 32x32 window.

The Code

The main goal of this project was to build an enhanced and extended version of Mike O'Neill's excellent C++ project. This time written in C# 4.0 and using WPF with a simple MVVM pattern as the GUI instead of Windows Forms. I also used the Windows API Code Pack for better integration with Windows 7. So Visual Studio 2010 and Windows Vista SP2 are the minimum requirements to use my application. I also made maximal use of the parallel functionality offered in C# 4.0 by letting the user at all times choose how many logical cores are used in the parallel optimised code parts with a simple manipulation of the sliderbar next to the View combobox.

Using the code

Here is the example code to construct a LeNet-5 network in my code (see the InitializeDefaultNeuralNetwork() function in MainViewWindows.xaml.cs):

NeuralNetworks network = new NeuralNetworks("LeNet-5", 0.8D, LossFunctions.MeanSquareError, 0.02D);
network.Layers.Add(new Layers(network, LayerTypes.Input, 1, 32, 32));
network.Layers.Add(new Layers(network, LayerTypes.Convolutional,ActivationFunctions.Tanh, 6, 28, 28, 5, 5));
network.Layers.Add(new Layers(network, LayerTypes.Subsampling, ActivationFunctions.AveragePoolingTanh, 6, 14, 14, 2, 2));

List<bool> mapCombinations = new List<bool>(16 * 6) 
{
 true, false,false,false,true, true, true, false,false,true, true, true, true, false,true, true,
 true, true, false,false,false,true, true, true, false,false,true, true, true, true, false,true,
 true, true, true, false,false,false,true, true, true, false,false,true, false,true, true, true,
 false,true, true, true, false,false,true, true, true, true, false,false,true, false,true, true,
 false,false,true, true, true, false,false,true, true, true, true, false,true, true, false,true,
 false,false,false,true, true, true, false,false,true, true, true, true, false,true, true, true
};

network.Layers.Add(new Layers(network, LayerTypes.Convolutional, ActivationFunctions.Tanh, 16, 10, 10, 5, 5, new Mappings(network, 2, mapCombinations)));
network.Layers.Add(new Layers(network, LayerTypes.Subsampling, ActivationFunctions.AveragePoolingTanh, 16, 5, 5, 2, 2));
network.Layers.Add(new Layers(network, LayerTypes.Convolutional, ActivationFunctions.Tanh, 120, 1, 1, 5, 5));
network.Layers.Add(new Layers(network, LayerTypes.FullyConnected, ActivationFunctions.Tanh, 10));
network.InitWeights();

Design View

This is Design view where you can see how the network is defined and see the weights of the convolutional layers. When you hover with the mouse over a single weight, a tooltip shows the corresponding weight value.

Training View

Training.png

This is Training view where you train the network. The 'Play' button gives you the 'Select Training Parameters' dialog where you can define the basic training parameters. The 'Training Schema Editor' button gives you the possibility to fully define your own training schemas and to save and load them as you want. At any time, the training can be easily aborted by pressing the 'Stop' button.

TrainingSchemaEditor.PNG

Testing View

In Testing view, you can test your network and get a graphical confusion matrix that represents all the misses.

Calculate View

In Calculate view, we can test a single digit with the desired properties and fire it through the network and get a graphical view of all the outputs in every layer.

Final Words

I would love to see a DirectCompute 5.0 integration for offloading the highly parallel task of learning the neural network to a DirectX 11 compliant GPU if one is available. But I've never programmed with DirectX or any other shader based language before, so if there's anyone out there with some more experience in this area, any help is very welcome. I made an attempt to use a simple MVVM structure in this WPF application. In the Model folder, you can find the files for the neural network class and also a DataProvider class which deals with loading and providing the necessary MNIST training and testing samples. There is also a NeuralNetworkDataSet class that is used by the project to load and save neural network definitions, weights, or both (full) from or to a file on disk. Then there is the View folder that contains the four different PageViews in the project and a global PageView which acts as a container for the different views (Design, Training, Testing, and Calculate). In the ViewModel folder, you will find a PageViewModelBase class where the corresponding four ViewModels are derived from. All the rest is found in the MainViewWindows.xaml.cs class. Hope there's someone out there who can actually use this code and improve on it. Extend it with an unsupervised learning stage for example (encoder/decoder construction), or implement a better loss-function (negative log likelihood instead of MSE); extend to more test databases, other than just only the handwritten MNIST; make use of more advanced squashing functions, etc.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Filip D'haene
Software Developer
Belgium Belgium
No Biography provided

Comments and Discussions


Discussions posted for the Published version of this article. Posting a message here will take you to the publicly available article in order to continue your conversation in public.
 
QuestionSeveral bias connections for single neuron Pin
geoyar6-Dec-14 11:48
membergeoyar6-Dec-14 11:48 
AnswerRe: Several bias connections for single neuron Pin
Filip D'haene6-Dec-14 12:14
memberFilip D'haene6-Dec-14 12:14 
GeneralRe: Several bias connections for single neuron Pin
geoyar6-Dec-14 12:46
membergeoyar6-Dec-14 12:46 
AnswerRe: Several bias connections for single neuron Pin
Filip D'haene6-Dec-14 13:44
memberFilip D'haene6-Dec-14 13:44 
BugRe: Several bias connections for single neuron Pin
Filip D'haene8-Dec-14 7:53
memberFilip D'haene8-Dec-14 7:53 
GeneralRe: Several bias connections for single neuron Pin
geoyar9-Dec-14 15:31
membergeoyar9-Dec-14 15:31 
AnswerRe: Several bias connections for single neuron Pin
Filip D'haene11-Dec-14 15:45
memberFilip D'haene11-Dec-14 15:45 
GeneralRe: Several bias connections for single neuron Pin
geoyar11-Dec-14 17:10
membergeoyar11-Dec-14 17:10 
GeneralRe: Several bias connections for single neuron Pin
geoyar23-Dec-14 12:40
membergeoyar23-Dec-14 12:40 
GeneralRe: Several bias connections for single neuron Pin
Filip D'haene23-Dec-14 15:56
memberFilip D'haene23-Dec-14 15:56 
GeneralRe: Several bias connections for single neuron Pin
geoyar23-Dec-14 16:52
membergeoyar23-Dec-14 16:52 
GeneralRe: Several bias connections for single neuron Pin
Filip D'haene23-Dec-14 17:10
memberFilip D'haene23-Dec-14 17:10 
GeneralRe: Several bias connections for single neuron Pin
geoyar24-Dec-14 7:47
membergeoyar24-Dec-14 7:47 
GeneralRe: Several bias connections for single neuron Pin
Filip D'haene25-Dec-14 6:42
memberFilip D'haene25-Dec-14 6:42 
GeneralRe: Several bias connections for single neuron Pin
Nathan Allan27-Feb-15 16:58
memberNathan Allan27-Feb-15 16:58 
QuestionSeems there is a bug Pin
geoyar17-Nov-14 9:36
membergeoyar17-Nov-14 9:36 
AnswerRe: Seems there is a bug Pin
Filip D'haene18-Nov-14 12:17
memberFilip D'haene18-Nov-14 12:17 
GeneralRe: Seems there is a bug Pin
geoyar19-Nov-14 12:08
membergeoyar19-Nov-14 12:08 
AnswerRe: Seems there is a bug Pin
Filip D'haene27-Nov-14 12:35
memberFilip D'haene27-Nov-14 12:35 
QuestionTrainToValue in calculatePseudoHessian Pin
Member 1108296215-Oct-14 23:49
memberMember 1108296215-Oct-14 23:49 
AnswerRe: TrainToValue in calculatePseudoHessian Pin
Filip D'haene25-Oct-14 11:39
memberFilip D'haene25-Oct-14 11:39 
QuestioniNumWeight in Parallel.For? Pin
Member19632543-Oct-14 14:20
memberMember19632543-Oct-14 14:20 
Questioncan we calculate the probability of recognition? Pin
Member 1091229327-Aug-14 22:56
memberMember 1091229327-Aug-14 22:56 
QuestionWrong connections in the second layer of LeCun network Pin
geoyar6-Aug-14 17:13
membergeoyar6-Aug-14 17:13 
AnswerRe: Wrong connections in the second layer of LeCun network Pin
Filip D'haene6-Aug-14 18:00
memberFilip D'haene6-Aug-14 18:00 
GeneralRe: Wrong connections in the second layer of LeCun network Pin
geoyar7-Aug-14 15:17
membergeoyar7-Aug-14 15:17 
GeneralRe: Wrong connections in the second layer of LeCun network Pin
geoyar8-Aug-14 13:09
membergeoyar8-Aug-14 13:09 
QuestionLittle bug Pin
geoyar5-Aug-14 13:35
membergeoyar5-Aug-14 13:35 
AnswerRe: Little bug Pin
Filip D'haene6-Aug-14 17:44
memberFilip D'haene6-Aug-14 17:44 
QuestionPlease give us a little tips to make our NNT Pin
Member 109122935-Aug-14 1:22
memberMember 109122935-Aug-14 1:22 
QuestionIt throw exceptions when working with my own data set. [modified] Pin
Member 1033405331-Jul-14 18:01
memberMember 1033405331-Jul-14 18:01 
AnswerRe: It throw exceptions when working with my own data set. [modified] Pin
Filip D'haene1-Aug-14 11:29
memberFilip D'haene1-Aug-14 11:29 
QuestionCan not setup the app Pin
geoyar20-Jul-14 12:15
membergeoyar20-Jul-14 12:15 
AnswerRe: Can not setup the app Pin
Filip D'haene20-Jul-14 13:49
memberFilip D'haene20-Jul-14 13:49 
GeneralRe: Can not setup the app Pin
geoyar21-Jul-14 11:39
membergeoyar21-Jul-14 11:39 
QuestionCode/Binaries gone? Pin
Member 788540010-Jul-14 4:00
memberMember 788540010-Jul-14 4:00 
AnswerRe: Code/Binaries gone? Pin
Filip D'haene10-Jul-14 5:30
memberFilip D'haene10-Jul-14 5:30 
GeneralRe: Code/Binaries gone? Pin
Member 788540010-Jul-14 15:45
memberMember 788540010-Jul-14 15:45 
AnswerRe: Code/Binaries gone? Pin
Filip D'haene10-Jul-14 16:55
memberFilip D'haene10-Jul-14 16:55 
GeneralMy vote of 5 Pin
CatchExAs9-Jul-14 5:06
professionalCatchExAs9-Jul-14 5:06 
GeneralRe: My vote of 5 Pin
Filip D'haene9-Jul-14 6:12
memberFilip D'haene9-Jul-14 6:12 
GeneralRe: My vote of 5 Pin
CatchExAs12-Jul-14 4:21
professionalCatchExAs12-Jul-14 4:21 
QuestionChange database [modified] Pin
MikeMNN13-Jun-14 3:24
memberMikeMNN13-Jun-14 3:24 
AnswerRe: Change database Pin
Filip D'haene13-Jun-14 12:41
memberFilip D'haene13-Jun-14 12:41 
GeneralRe: Change database Pin
MikeMNN16-Jun-14 4:20
memberMikeMNN16-Jun-14 4:20 
AnswerRe: Change database Pin
Filip D'haene17-Jun-14 5:30
memberFilip D'haene17-Jun-14 5:30 
GeneralRe: Change database [modified] Pin
MikeMNN18-Jun-14 0:09
memberMikeMNN18-Jun-14 0:09 
AnswerRe: Change database Pin
Filip D'haene18-Jun-14 14:08
memberFilip D'haene18-Jun-14 14:08 
QuestionPlease guide me to training CNNs on Sound spectrum Images Pin
Member 1062768312-Apr-14 2:53
memberMember 1062768312-Apr-14 2:53 
AnswerRe: Please guide me to training CNNs on Sound spectrum Images [modified] Pin
Filip D'haene14-Apr-14 9:43
memberFilip D'haene14-Apr-14 9:43 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.150520.1 | Last Updated 8 Nov 2011
Article Copyright 2010 by Filip D'haene
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid