Adaptive Simulated Annealing: An Alternative Approach for the Error Minimization of Neural Networks
Date of Award
Master of Science
Dr. Bruce A. Whitehead
Dr. Kenneth R. Kimble, Dr. Monty L. Smith
This work introduces an alternative algorithm, simulated annealing, to minimize the prediction error from neural networks that traditionally use back-propagation methods. The simulated annealing algorithm stochastically samples the parameter space formed by weights of the neural network until a minimal error is found.
Three problems were investigated in this work: the radiator problem, the spiral problem, and the time series prediction problem. Each of them was examined using the same neural network architecture, i.e., a 2-layer network, and trained by both back-propagation and simulated annealing.
The simulated annealing algorithm consumes longer computation time in searching for the global minimum than the back-propagation method. It may not obtain as small an error as back-propagation in a reasonable amount ot time though it theoretically guarantees the location of the global minimum, given enough searching time. It was also found that simulated annealing can more effectively optimize neural networks with a moderate number of weights (less than 32) than those with a large number of parameters, e.g., in the radiator problem, and the spiral problem. Also, simulated annealing is suitable to solve problems with a discrete parameter space.
During the process of tuning the simulated annealing package, configurations with low initial temperatures give faster location of minima, and hence more effective searching, than those with high initial temperatures.
Sun, Yuxing, "Adaptive Simulated Annealing: An Alternative Approach for the Error Minimization of Neural Networks. " Master's Thesis, University of Tennessee, 2001.