Neural grammatical inference using network-adapted encoding

Other researchers have recently trained recurrent artificial neural networks with gradient descent techniques to infer simple abstract machines. This thesis examines the effects of a dimensionality reduction technique on the performance of such a network trained to recognize the odd-parity language. The network used is a second-order recurrent network similar to that of Giles et al. (1990) with the same complete-gradient training method. The differences are: 1) a hidden layer (the encoding layer) inserted between the input layer and the original second layer (the state layer) and 2) a separate output layer--a single processing element (PE) with first-order connections from the state layer. The encoding layer has first order connections from the input layer. Hypothesis: The encoding layer should improve the performance of the network.

According to two measures of performance--the fraction of correctly classified strings and the total epoch error--the network performed better without encoding PEs. The network converged 70% of the time for 0 encoding PEs (i.e., without any compression method) and did not converge at all when using encoding PEs. The error and classification accuracy plots (with respect to a varying number of encoding PEs) reveal a pattern whose general shape is not heavily dependent on the data set used for testing. Thus, the accuracy and error information may be useful in characterizing network behavior, with or without convergence.