A Resource Efficient Localized Recurrent Neural Network Architecture and Learning Algorithm

Recurrent neural networks (RNNs) are widely acknowledged as an effective tool that can be employed by a wide range of applications that store and process temporal sequences. The ability of RNNs to capture complex, nonlinear system dynamics has served as a driving motivation for their study. RNNs have the potential to be effectively used in modeling, system identification and adaptive control applications, to name a few, where other techniques fall short. Most of the proposed RNN learning algorithms rely on the calculation of error gradients with respect to the network weights. What distinguishes recurrent neural networks from static, or feedforward networks, is the fact that the gradients are time-dependent or dynamic. This implies that the current error gradient does not only depend on the current input, output and targets, but rather on its possibly infinite past. How to effectively train RNNs remains a challenging and active research topic.

This thesis introduces TRTRL, an efficient, low-complexity online learning algorithm for recurrent neural networks. The approach is based on the real-time recurrent learning (RTRL) algorithm, whereby the sensitivity set of each neuron is reduced to weights associated either with its input or output links. As a consequence, storage requirements are reduced from O(N³) to O(N²) and the computational complexity is reduced from O(N⁴) to O(N²). Despite the radical reduction in resource requirements, it is shown through simulations results that the overall performance degradation of the truncated real-time recurrent learning (TRTRL) algorithm is minor. Moreover, the scheme lends itself to efficient hardware realization by virtue of the localized property that is inherent to the approach. The TRTRL algorithm is first implemented and evaluated using a multi-purpose CPU. Next, the framework is extended to a hardware implementation that scales to high network densities without compromising computation speed and overall performance.