In our last part we went over the mathematical design of the neurons and the network itself. Now we are going to build our network in MatLab and test it out on a real world problem.
Let’s say that we work in a chemical plant. We are creating some compound and we want to anticipate and optimize our production. The compound is synthesized in a fluidized bed reactor. For those of you without a chemical engineering background, think of a tube that contains tons of pellets. Fluid then runs over these pellets and turns into a new compound. Your boss comes to you and tells you that there is too much impurity in our output stream. There are two things you can change to reduce the impurity, catalyst (the pellets in our tube) amount and stabilizer amount.In the pilot scale facility, you run a few tests varying the amount of catalyst and stabilizer. You come up with the following table of your results.| Catalyst | Stabilizer | Impurities % |
| 0.57 | 3.41 | 3.7 |
| 3.41 | 3.41 | 4.8 |
| 0 | 2 | 3.7 |
| 4 | 2 | 8.9 |
| 2 | 0 | 6.6 |
| 2 | 4 | 3.6 |
| 2 | 2 | 4.2 |
After looking at the results you decide to create a neural network to predict and optimize these values. As we know we have two inputs, catalyst and stabilizer, and one output, impurity percent. From our last part on structures of neural networks we decided that we need two neurons in our input layer (one for catalyst and one for stabilizer), and one neuron in our output layer (impurity percent). That only leaves our hidden layer, since we do not expect a complex difficult problem that requires deep learning we only choose one layer. As for neurons we will choose 3 neurons to make the problem a little more interesting. The structure is shown below.
Now that we have the structure let us build our network in MatLab. The code is actually quite simple for this part. First we input our two variables in a x by 2 matrix. We then multiply these by our first weights from our hidden layer and pass them through our sigmoid function. These values are then multiplied by the weights from the output layer then passed through the sigmoid function again. After they pass through they become our output, impurity %. So lets see how our network performs the vector on the left is our actual values (scaled to the max) and on the right is what our network determined.
As you can see, the network did not guess even remotely correctly. Well we are missing the most important part of the neural network, the training. We must train our network to get the right predictions. In order to do this we need to do our favorite thing, optimize.-MarcelloHeres the code:
% ANN code
% structure:
% 2 input
% 3 hidden nodes
% 1 output
%initial input data [ catalyst, stabilizer]
input_t0 = [0.57 3.41; 3.41 3.41;0 2;4 2;2 0;2 4;2 2];
%normalize input data
input_t0(:,1) = input_t0/max(input_t0);
input_t0(:,2) = input_t0/max(input_t0);
%normalize output data
output_t0 = [3.7 4.8 3.7 8.9 6.6 3.6 4.2];
output_t0 = output_t0/max(output_t0);
%randomly assigned weights
weight_in = [.3 .6 .7;.2 .8 .5];
weight_out = [ .4 .6 .7]';
%initialize matrices
actHidSig = zeros(7,3);
actOutSig=zeros(7,1);
%find activation for hidden layer
act_hid = input_t0*weight_in ;
%apply sigmoid to hidden activation
for i = 1:7
for j = 1:3
actHidSig(i,j) = 1/(1+exp(-act_hid(i,j)));
end
end
%find activation for output layer
act_out = actHidSig*weight_out;
%apply sigmid to output activation
for i = 1:7
actOutSig(i) = 1/(1+exp(-act_out(i)));
end
%show results
output_t0'
actOutSig
Once you get home you open up chrome and check out some of the nutritional facts on the items from the store. You pop open excel and make a spread sheet that lists all the nutritional facts broke down into our four “macros”. The spreadsheet is shown below.
As you can see the method is flying back and forth past our minima. However, the method eventually converges at (0.9987, 0.4987), our target minima. Below I have included a matrix containing our initial step and all of the following steps (taken from matlab output).
We now take the inverse of the Hessian and then find our next point. The iterative equation is listed below as well as the output.
This looks awfully like our iterative steps from our one dimensional Newton’s method. . As we get closer to our minimum the gradient, g, of our function moves toward 0 which makes xk≈ xk+1. This is when we know our method has found a minimum (hopefully). Hopefully this all feels familiar. Next post we will see how Newton Raphson works.-Marcello
From this point we must calculate the gradient. I have taken the gradient of our function (f(x,y) = x^2 -x+cos(y+x)+y^2). This gradient must now be evaluated at our point (8,8)
