Masters Theses

Date of Award

12-1991

Degree Type

Thesis

Degree Name

Master of Science

Major

Computer Science

Major Professor

Reinhold Mann

Committee Members

Michael Thomason, Bruce MacLennan

Abstract

Artificial neural networks exhibit important classification capabilities in a variety of pattern recognition applications, due in large part to their ability to adapt to a changing environment without requiring either a complex set of programmed steps or underlying sample distribution information. Their adaptive learning capability may allow them to be successfully applied to problems in such areas as speech and pattern recognition [Lippmann 87]. Multi-layer feed-forward neural networks classifiers require a training rule for adjusting weights in internal laxers. One such training rule, the back-propagation of error algorithm, also known as the generalized delta rule [Rumelhart 86a], has become one of the most widely used training algorithms for neural networks. However, it suffers from two major drawbacks: slow convergence rates and convergence to non-optimal solutions, or local minima. The objective of this research was to investigate the effect different heuristics have on the performance of the standard back-propagation (BP) algorithm. The methods that were studied included modification of both learning rate and momentum parameters, several adaptive techniques (e.g. the Delta-Bar-Delta and Extended Delta-Bar-Delta algorithms) for dynamically adjusting these parameters, modification of the sigmoid function, and replacement of the random initial weights with values which pre-partition the inputs into separate decision regions. Results from the use of these heuristics were compared in terms of both their overall impact on convergence rates and their effect on convergence to local minima. These heuristics were implemented and tested on three benchmark problems, each with different characteristics which presented varying levels o{ difficulty to the neural network. The three problem types used to test and compare the algorithms were the XOR (parity), multiplexer, and encoder/decoder problems. After benchmarking performance on small problem sets, problem size was increased to examine the effect of scaling on the different heuristics. The adaptive algorithms were shown to achieve reduced convergence times compared to BP on all problem types and sizes. The improvement was more pronounced for the larger problems. The Delta-Bar-Delta algorithm often produced the best results of any of the heuristics using only three additional inputs. The large number of parameters required by the Extended Delta-Bar-Delta algorithm made it difficult to tune. In order to converge to solutions on scaled-up problem sizes, the BP algorithm was shown to require normalization of the weight update value, as well as an individual per-pattern error criterion in place of the sum of squared error criterion. Computational requirements were shown to increase exponentially with additional input lines for the XOR/Parity and multiplexer problems, and polynomially for the encoder/decoder problem.

Download
Files over 3MB may be slow to open. For best results, right-click and select "save as..."

Share

COinS