Shiraz University - CSE Dept - Project # 2

Training Neural Networks Using Genetic Algorithms

M.M.Haji
mehdi.haji@gmail.com

Problem Definition:

Genetic Algorithms can be applied to the problem of neural network design in several ways, in this project we will explore the use of GAs in training a network of known structure ( a 3-2-1 feed forward network ). GAs can also be applied to discover the size, structure and learning parameters of a network to be trained by a separate learning algorithm.

A 3-2-1 Feed Forward Neural Network

The training set is the 3-bit XOR function, in the following form:

0 0 0 0
0 0 1 1
0 1 0 1
0 1 1 0

1 0 0 1
1 0 1 0
1 1 0 0
1 1 1 1

Where in each line the first three numbers are inputs and the last number is the output resulting from XORing these three inputs.

Our aim is to find the set of input-hidden weights, hidden layer biases, hidden-output weights and output layer biases using a GA for the above training set.

Implementation:

The implementation includes three java files:

• NNUtils.java which contains some useful functions such as matrix multiplication method.

• FFNN.java which is the Feed Forward Neural Network class, and each FFNN object is an individual in our GA. The network may have no more than one hidden layer, but the number of neurons in each layer (i.e.  input, hidden and output) is adjustable. The activation function of the hidden layer is tansig but the activation function of the output layer may be tansig, logsig, hardlim or purelin. Where:
   

  tansig(x) = 2/(1+exp(-2*x))-1     logsig(x) = 1/(1+exp(-x))     purelin(x) = x     hardlim(x) = 1, if x >= 0                  0, otherwise

  Uniform cross-over is used as the main operator in our GA which will be done by the xover_uniform method of FFNN.

• GAPopulation.java which is the main class, and in its main method a simple GA is implemented to find the set of weights and biases (i.e. print the best individual that is a feed forward neural network):
   

  public static void main(String[] args) {   1     FFNN.out_act_fun = FFNN.LOGSIG;   2     FFNN.tr_set = NNUtils.read_training_set("xor.txt");   3     GAPopulation pop = new GAPopulation(300,3,2,1);   4      for(int i=0; i < 500; i++){   5         System.out.println(pop.ind[pop.best_index]);   6         if( pop.ind[pop.best_index].eval_error() == 0.0f )   8             break;   10        pop = pop.generate(pop,80,10);   11    }   12    System.out.println(pop.ind[pop.best_index]);   }

  In line 1 we set the activation function of the output layer to logsig, which is a good choice for the XOR problem.
  The training set is read from a file ( xor.txt in this case ) for convenience. It is done by line 2.
  In line 3, we create the first generation ( generation # 0 ) of evolution that contains 300 individuals, each individual is a 3-2-1 feed forward neural network.
  Lines 4 through 11 implements evolution loop. The evolution terminates when an ideal individual ( an individual of fitness 1.0 ) is found. Maximum generations are 500 and will be reached when an ideal individual is not found. By line 10 we mean: "generate the new population from the current population by 80% cross-over and 10% mutation and the remaining 10% by reproduction". After the evolution loop, in line 12, the best individual of the last generation is printed.

As you see, the implementation is so flexible and general and is not limited to this specific XOR problem. By only modifying lines 1,2 and 3, the program can be used to train another function to the network

Download class files here: NNUtils.class, FFNN.class and GAPopulation.class.

Compile and Run:

To compile and run the program, java 2 complier must be installed on your system. If so, execute the following commands:

Results:

With 2 neurons in the hidden layer, it is hard for GA to find a set pf appropriate weights, but after some runs, the following weights was found:

input to hidden weight matrix:

-0.983598    4.8899565
1.0686378    -6.306935
-1.001853    5.6389494

hidden layer bias:

0.40927365    -1.8566227

hidden to output weight matrix:

15.7031975
11.211046

output layer bias:

0.32423866

But with 3 neurons in the hidden layer, almost in each run this GA converges (i.e. finds the set of optimal weights).