Data Mining - Mehmed Kantardzic [122]
Samplek xk1, xk2, … , xkm dk
Processing the input vector X(n), a neuron k produces the output that is denoted by yk(n):
It represents the only output of this simple network, and it is compared with a desired response or target output dk(n) given in the sample. An error ek(n) produced at the output is by definition
The error signal produced actuates a control mechanism of the learning algorithm, the purpose of which is to apply a sequence of corrective adjustments to the input weights of a neuron. The corrective adjustments are designed to make the output signal yk(n) come closer to the desired response dk(n) in a step-by-step manner. This objective is achieved by minimizing a cost function E(n), which is the instantaneous value of error energy, defined for this simple example in terms of the error ek(n):
The learning process based on a minimization of the cost function is referred to as error-correction learning. In particular, minimization of E(n) leads to a learning rule commonly referred to as the delta rule or Widrow-Hoff rule. Let wkj(n) denote the value of the weight factor for neuron k excited by input xj(n) at time step n. According to the delta rule, the adjustment Δwkj(n) is defined by
where η is a positive constant that determines the rate of learning. Therefore, the delta rule may be stated as: The adjustment made to a weight factor of an input neuron connection is proportional to the product of the error signal and the input value of the connection in question.
Having computed the adjustment Δwkj(n), the updated value of synaptic weight is determined by
In effect, wkj(n) and wkj(n + 1) may be viewed as the old and new values of synaptic weight wkj, respectively. From Figure 7.6 we recognize that error-correction learning is an example of a closed-loop feedback system. Control theory explains that the stability of such a system is determined by those parameters that constitute the feedback loop. One of those parameters of particular interest is the learning rate η. This parameter has to be carefully selected to ensure that the stability of convergence of the iterative-learning process is achieved. Therefore, in practice, this parameter plays a key role in determining the performance of error-correction learning.
Figure 7.6. Error-correction learning performed through weights adjustments.
Let us analyze one simple example of the learning process performed on a single artificial neuron in Figure 7.7a, with a set of the three training (or learning) examples given in Figure 7.7b.
Figure 7.7. Initialization of the error correction-learning process for a single neuron. (a) Artificial neuron with the feedback; (b) training data set for a learning process.
The process of adjusting the weight factors for a given neuron will be performed with the learning rate η = 0.1. The bias value for the neuron is equal 0, and the activation function is linear. The first iteration of a learning process, and only for the first training example, is performed with the following steps:
Similarly, it is possible to continue with the second and third examples (n = 2 and n = 3). The results of the learning corrections Δw together with new weight factors w are given in Table 7.2.
TABLE 7.2. Adjustment of Weight Factors with Training Examples in Figure 7.7b
Parameter n = 2 n = 3
x1 −1 0.3
x2 0.7 0.3
x3 −0.5 −0.3
y −1.1555 −0.18
d 0.2 0.5
e 1.3555 0.68
Δw1(n) −0.14 0.02
Δw2(n) 0.098 0.02
Δw3(n) −0.07 −0.02
w1(n + 1) 0.37 0.39
w2(n + 1) −0.19 −0.17
w3(n + 1) 0.735 0.715
Error-correction learning can be applied on much more complex ANN architecture, and its implementation is discussed in Section 7.5, where the basic principles of multilayer feedforward ANNs with backpropagation are introduced. This example only shows how weight factors change with every training (learning) sample. We gave the results only for the first iteration. The weight-correction process will continue either by using new training samples or by using the same data samples in the next iterations. As to when to finish the iterative