Assignment 5 -- Simple Recurrent Networks

In this assignment you will experiment with the simple recurrent network (SRN) architecture described by Elman in his paper Finding Structure in Time.  You may use the code from Assignment 4 as your starting point for this assignment if you wish, or you may implement your network from scratch.

You will need to add a context layer to make your network an SRN.  This is just an additional set of input units, which store a copy of the hidden unit activations from the previous time step.  Be sure to copy the hidden unit activations to the context layer only after the weights have been adjusted for a given input.

Experiment with two of the following problems:

• Sequence prediction.  The input consists of a repeated sequence of patterns such as A B C B A B C B...  The output unit(s) predict the next element in the sequence.

• Sequence categorization.  The input consists of several short fixed-length sequences presented in random order.  The output unit(s) categorize the given input sequence.

• Meter recognition.  The input pattern consists of duple (2-beat) and triple (3-beat) meters as shown below.  The output unit(s) categorize the current input as duple or triple.


    1 0 1 0 1 0 1 0 1 0 0 1 0 0 1 0 0 1 0 0 1 0 1 0 1 0 1 0 ...
• Learning English word boundaries.  The input consists of a stream of letters (each represented as a 5-bit random vector) taken from a corpus of English text.  The output units predict the next letter in the sequence.  This experiment is described on pages 10-12 in the Elman article.  You should first test the network on short sentences generated from a limited vocabulary (as described in the article) before trying it out on real English text.

Turning in Your Homework