Click here to Skip to main content
15,947,389 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
So I am trying to implement an MLP for XOR. Currently I am not getting any satisfying results. I coded it in java and currently I'm convinced that the backpropagation is correct and so is the forward but I don't get why it doesn't work. The delta calculation is what I had the mot problem with but I think I understand it now. I hope someone could help me. I have set the learning rate from 0.1 to 0.7 and no luck.
I already tested with the inputs keeping it simple I used 2 inputs: 00, 01,10,11. The results I'm getting:
0.03046590405786709
0.9159156257800326
0.8664182057539402
0.7590399051848435

They should be 0110 but the network isn't learning properly. I tested the results from the forward and they were correct so the only one I'm not quite sure how to check are the back propagation results. The delta results seem to be okay as well. Please guide me if my algorithm understanding is wrong.

What I have tried:

Java
public double[] forward(ArrayList<Double> inputs){
		// Loop through the hidden layers 
		for(int i = 0; i < NH; i++){
			Hidden[i] = 0.0;
			for(int j = 0; j < NI; j++){
				Hidden[i] += (input[j] * W1[j][i]);
			} 

			Hidden[i] +=bias;
			//Calculate activation
			Hidden[i] = 1/ (1 + Math.exp(-Hidden[i]));

		} 

		// Outer layer - output calculation 
		for(int i = 0; i < NO; i++){
			Output[i] = 0.0;
			for(int j = 0; j < NH; j++){
                            Output[i] += (Hidden[j] * W2[j][i]);
			} 
			
			//add bias set to 1
			Output[i] +=bias;
                        //Calculate activation 
			Output[i] = 1/ (1 + Math.exp(-Output[i]));
		} 
		return Output;
	}



Java
public double backwards(ArrayList<Double> targetOutputs){
		double error = 0;

		for(int i = 0; i < NO; i++){
			Z2[i] = Output[i]*(1.0-Output[i]) * (targetOutputs.get(i) - Output[i]);

		}
		
		for(int i=0; i< NH; i++) { 
			error = 0.0;
			for(int j=0; j< NO; j++){
				error += W2[i][j] * Z2[j];
			}
			Z1[i] = Hidden[i] * (1.0-Hidden[i]) * error;
		
		}
		//calculate the delta weight change
		for(int i=0; i< NH; i++) {
			for(int j=0; j< NI; j++) {
				dW1[j][i] += Z1[i] * input[j] + LR*dW1[j][i];

			}
		}
		for(int i=0; i< NO; i++){
			for(int j=0; j< NH; j++){
				dW2[j][i] +=  Z2[i] * Hidden[j] + LR*dW2[j][i];
			}
		}
		return Z2[0];
	}

Java
public void updateWeights(double LR){
    for(int i= 0; i< NO; i++){
        for(int j=0; j< NH; j++){
            W2[j][i] += dW2[j][i] ;
            dW2[j][i] = 0;
        }
    }
    for(int i=0; i< NH; i++){
        for(int j=0; j< NI; j++){
            W1[j][i] +=  dW1[j][i];
            dW1[j][i] = 0;
        }
    }
}
Posted
Updated 2-May-16 6:04am
v3
Comments
Patrice T 2-May-16 11:36am    
What is MLP ? My Little Pony ?
Member 12497969 2-May-16 11:42am    
Better(Title)?
Patrice T 2-May-16 11:44am    
sure :)

1 solution

Quote:
I'm convinced that the backpropagation is correct and so is the forward
being convinced is not enough ! you must ensure that things are correct.
Try unit testing, either with real unit testing or manually: get a sample dataset, run the function on it, check that the results match the expectations.

It is also a good idea to use the debugger to see where something strange append.
The debugger allow you to follow the execution line by line, inspect variables and you will see that there is a point where it stop doing what you expect.
Debugger - Wikipedia, the free encyclopedia[^]
Mastering Debugging in Visual Studio 2010 - A Beginner's Guide[^]
http://docs.oracle.com/javase/7/docs/technotes/tools/windows/jdb.html[^]
https://www.jetbrains.com/idea/help/debugging-your-first-java-application.html[^]
 
Share this answer
 
Comments
Member 12497969 2-May-16 11:50am    
I alrady tested with the inputs keeping it simple I used 2 inputs: 00, 01,10,11. The results I'm getting:
0.03046590405786709
0.9159156257800326
0.8664182057539402
0.7590399051848435

They should be 0110 but the network isn't learning properly. I tested the results from the forward and they were correct so the only one Im not quite sure how to check are the back propagation results. The delta results seem to be ok as well. I dont usually ask questions online and this is my first time. Please guide me if my algorithm understanding is wrong.
Patrice T 2-May-16 11:59am    
Use Improve question to update your question.
Member 12497969 4-May-16 10:00am    
Thanks, it turns out that the weights were not randomised properly.
Patrice T 4-May-16 10:04am    
Nice to see you solved the problem;
did you used the debugger ?

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
Top Experts
Last 24hrsThis month


CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900