Robustness

The testing data was made from letters shown in Fig. 6.2, resulting in a 6045×3 input matrix U_test and a 6045×2 output matrix V_test with n_gap = 16 being used. For non-ROL topology the test errors were found to be mse_test,1 ≈ 1.2×10⁻², mse_test,2 ≈ 3.5×10⁻². Using ROL reduced the test errors to mse_test,1 ≈ 0.9×10⁻³, mse_test,2 ≈ 3.0×10⁻². The errors mse_test,i provide only a rough indication of network performance. A visual comparison between these two trials is shown in Fig. 6.2. It can be observed that the network also produces appropriate naturalness for letters on which it had not been trained.

To better understand the impact of ROL on RNN performance, a setup with no internal units was tested first. This configuration reveals the im-portance of the output from previous time step (=ROL) more clearly due to the lack of involvement from any internal states. Further, this setup also demonstrates the importance of the internal states themselves. For the non-ROL setup, the output weights W^out were computed using the input unit states u(n) only. This boils down to a regression problem with three predictor variables (input activations) and two response variables (output activations). For the ROL setup, the output weights W^out were computed using the input states u(n) and output activations y(n−1). This is equiv-alent to a regression problem with five predictor variables (input activations and output activations from the previous step) and two response variables (output activations). To better evaluate the performance of the models, we also computed the correlations between the teacher signals and the produced output signals. The results are given in the Table 6.1. Poor training per-formance of the non-ROL setup shows that the input activations u(n) alone are not sufficient for computing the outputy(n) and suggests that nonlinear expansion of the input activations into a space of higher dimension might be necessary. Using the output activations y(n−1) from the previous step (ROL configuration) decreased the training error and increased the correla-tions significantly. This result indicates that the activation y(n−1) is vital for computing the successive output y(n). The non-ROL performed roughly the same on the testing data as it did on the training data. In contrast, the ROL configuration performed significantly worse on the testing data than it did on the training data. This drop in performance can be attributed to the fact that during the testing phase the ROL model could not generate an appropriate output, which, in turn, could be used to compute the next

output. Relatively high testing errors and low correlations for both setups suggest that nonlinear expansion (=internal state, Eq. (6.2)) of both input activationsu(n) and output activations y(n−1) is necessary to achieve bet-ter performance. Comparisons of performance with testing data for both configurations shows that even in this trivial case ROL performs slightly better³.

Table 6.1: Training and testing errors and correlations for configuration with no internal units. See text for details.

train test

mse_1,2 corr_1,2 mse_1,2 corr_1,2

non-rol 0.0278 0.0247 0.1761 -0.0525 0.0272 0.0216 0.0989 0.0004 rol 0.0029 0.0025 0.9477 0.9466 0.0307 0.0212 0.2480 0.0770

In the ESN approach, every initialization of a network produces slightly different weight matrices, and thus, it is common practice to run the same setup several times and keep the network, which performed the best. To find out whether ROL is superior to non-ROL in general or only for a particular configuration with particular weights, four different setups were tested. Each configuration was tested 25 times with and without ROL, and the network was reinitialized for each new trial. In total, 100 trials were carried out.

Within each trial, the same network was used without reinitialization of any weights so as to ensure that having or not having ROL was the only difference between the compared networks.

Configuration 1 is identical to the one introduced in Section 6.3.2. For configuration 2, the spectral radius of matrix Wwas set to 0.95. All remain-ing parameters were identical to configuration 1. For configuration 3, the

3Comparable testing errorsmsetest,i but higher correlations corrtest,i for ROL config-urations (i= 1,2).

spectral radius of matrix W was set to 0.75 with all remaining parameters being identical to configuration 1. Finally, for configuration 4, the activation function for the internal units was set totanh. All remaining parameters were identical to configuration 1. The results are given in the Table 6.2. As we can see, for each configuration, ROL performed better. Fig. 6.3, 6.4 and 6.5 illustrate visually the results of randomly selected trials of configurations 2, 3 and 4, respectively. Visual comparison of the results obtained using configu-ration 1 is possible using Fig. 6.1 and 6.2. The superior performance of ROL is clearly visible for configurations 1, 2 and 3 (Fig. 6.2, 6.3 and 6.4). With configuration 4, the difference between the performance of the configuration with and without ROL might not be obvious at first look. Configuration 4 without ROL was often found, however, to produce misshapen letters and consequently higher mse_test,i.

Throughout the 100 trials, some trials where ROLmse_test,iwas very simi-lar or slightly higher than nonROLmsetest,iwere also found to occur. Visual comparison of these few cases nevertheless confirmed superiority of ROL;

ROL configurations always produced outputs that were visually more ap-pealing.

Stability of configurations that use ROL is another important issue. Here, the spectral radius of the internal weights matrix W was found to be of special significance. Configurations with spectral radius higher than 0.8 were found to very often generate stable solutions. Changing the spectral radius to lower values was found to increase the number of non-stable solutions.

Networks that exhibited non-stable behavior with ROL usually produced numerically stable solutions without ROL, but the testing errors of such networks was rather high and their visually evaluated performance was rather poor. On the other hand, cases where networks exhibited stable behavior

with ROL but non-stable without were also found to occur. Neither of the above cases were included in the averaged results shown in the Table 6.2 because the high testing errors of those trials would significantly bias the results towards ROL or nonROL configurations.

In general, ROL setups were found to perform better numerically (lower mse) and visually (they produced visually more appealing characters). Con-cerning stability, the spectral radius of internal weights matrixW was found to be of significance, and the existence/non-existence of ROL has not found to have any impact on stability.

Table 6.2: Errors and correlations for training and testing averaged over 25 trials for configurations 1,2,3 and 4. See text for details.

configuration train

mse_1,2 corr_1,2

1 non-rol 7.90×10⁻⁴ 2.48×10⁻³ 0.979 0.978 rol 5.79×10⁻⁴ 1.97×10⁻³ 0.985 0.983 2 non-rol 1.02×10⁻³ 2.89×10⁻³ 0.973 0.975 rol 6.03×10⁻⁴ 2.18×10⁻³ 0.984 0.981 3 non-rol 6.85×10⁻⁴ 2.10×10⁻³ 0.982 0.982 rol 5.48×10⁻⁴ 1.75×10⁻³ 0.985 0.985 4 non-rol 1.18×10⁻³ 2.34×10⁻³ 0.968 0.980 rol 1.11×10⁻³ 2.05×10⁻³ 0.970 0.982

configuration test

mse_1,2 corr_1,2

1 non-rol 2.22×10⁻² 6.08×10⁻² 0.450 0.410 rol 1.15×10⁻² 4.18×10⁻² 0.574 0.446 2 non-rol 1.66×10⁻² 4.18×10⁻² 0.504 0.469 rol 1.18×10⁻² 3.87×10⁻² 0.568 0.476 3 non-rol 1.86×10⁻² 6.33×10⁻² 0.510 0.418 rol 1.19×10⁻² 4.23×10⁻² 0.568 0.450 4 non-rol 2.25×10⁻² 4.39×10⁻² 0.371 0.407 rol 1.25×10⁻² 3.91×10⁻² 0.483 0.421

Figure 6.3: Setup 2