Assignment 1 -- Properties of Random Vectors

Due by class time Monday, September 23


A critical aspect of neural net models in general is the structure and generation of the state vectors used to represent information in the system.

We make the assumption that elements in a state vector are related to the activities of neurons in the brain. In the nervous system, cortical neurons represent highly processed information. The activities of single neurons represent coded information that is the result of evolutionary experience and environmental modification during development.

Single unit neurophysiology has given us some insight into what these codings contain -- in the visual system, for example, one finds cells that are motion-sensitive, orientation-sensitive, etc. What this means is that we can legitimately assume that many important aspects of the world have been directly coded for us in the elements of our inputs and we need not worry about deriving them. That is, the representation of information, if done properly, may do much of the work for us. For example, many cells at the low levels of the visual system respond to motion. We can assume that this is part of the higher-level input vector; we do not need to form the difference between successive images to infer motion directly from image displacement. Presumably the coding used has been evolved to make it easy for higher levels to function. How sensitivity to motion is generated in the nervous system in the first place is an interesting and important question in itself.

At present we know only enough about coding to make very crude descriptions of the low-level analysis of the visual system. Other representations used by the brain, especially those used in high-level cognitive tasks, are almost a complete mystery.

In this assignment we will study a simple but useful kind of state vector: one whose elements are random variables. This statistically random vector is easy to analyze and generate, and gives useful information about the behavior of distributed systems at a basic level. It can also represent our ignorance of details of interesting codings. Real codings that have statistics like vectors with random elements turn out to be optimal in some situations. In any case, such vectors serve to demonstrate useful properties of distributed systems.

Part One

Write a program to generate a large number of random values in the range zero to one using your particular programming language's built-in random number generator. As the simplest possible check of the generator, make sure that the values indeed are uniformly distributed between zero and one. Make a histogram of values for, say, 50,000 calls to the generator. A simple histogram program with 10 bins can be constructed by multiplying the value of the random number by 10, truncating, and adding one to the appropriate bin. That is, all values between 0.0000 and 0.0999 will go in the 0th bin, between 0.1000 and 0.1999 in the first bin, etc. With a uniform distribution, all the bins should contain roughly the same count. Those with a statistical background can analyze this matter further, but this is not necessary to do the assignment. A classic reference to random number generators is found in Donald Knuth's The Art of Computer Programming, Volume 2, Addison-Wesley (2nd. edition, 1981), pp. 1-178, where generators and tests of generators are covered in more detail than you can possibly imagine.

Part Two

Uniform random number distributions are not terribly interesting in many applications, but they turn out to be about the only kind you can generate easily by numerical methods.  Of course, as everyone is aware, it is impossible to actually generate random numbers on the computer, merely sequences that are sufficiently random for our purposes.

A more generally applicable distribution is the normal ("Gaussian", "bell-shaped") distribution. It is relatively easy to generate a normally distributed random variable from a uniformly distributed one. The simplest way is to add up a bunch of uniformly distributed random variables, wave your hands, invoke the Central Limit Theorem and claim that a normal distribution is shown by the sum.  Actually, this technique gives a pretty good approximation to the center of the normal distribution, but leaves something to be desired at the tails of the distribution, where often important things happen.

Write a procedure that will return normally distributed random values. Generate a histogram of values returned by the function. (Remember: These values are no longer necessarily between 0 and 1. You may need more than 10 bins on your histogram and you may have some extreme values.) Compare the values with what you expect from a normal distribution.  The mathematical form of the standard normal distribution, with mean 0 and variance 1, is given below:

Tables of values and copious commentary about the normal distribution can be found in any elementary statistics textbook.

Part Three

Now that you have your generators working, the next step is to construct random stimuli (i.e., state vectors). For our purposes, this means constructing vectors with random elements. It is usually convenient to construct stimuli having a distribution of mean zero. This means that we should subtract the mean of the vector elements from the element values, or, very slightly differently, we could subtract 0.5 from the values generated by a uniform random distribution. In practice, for reasonably large dimensionality, these techniques do not differ in their effects on  neural models.

Your specific assignment will now be to study the properties of vectors containing random elements. I would like you to generate normalized (i.e., length 1) random vectors whose elements are taken from a distribution with mean zero. It is probably easiest to use the uniform distribution to generate the elements.

Generate many pairs of such vectors and generate their dot product. What does this dot product actually mean, geometrically? (Remember: the length of each vector is 1.) Generate a histogram of dot products and compute the mean and standard deviation of the dot product. Use the following dimensionalities: 10, 20, 50, 100 250, 500, 1000 and 2000.

It is trivial to compute what the mean of the resulting distribution of dot products should be, given the constraints on the vectors. Tell me what it should be (and why) and compare it with your results. Computing the expected standard deviation of the distribution of dot products is not so easy but not hard if you know statistics. If you can figure it out, tell me and compare it with your simulation. Otherwise, see if you can guess roughly what it should be from your data. Try to guess how the "width" (standard deviation) of the distribution changes with the dimensionality of the vectors used in the dot product.

Turning in Your Homework

You may use Java, C/C++, MATLAB, Perl, or any other reasonable programming language for this assignment.  Your code should be handed in electronically, using the cs152submit (or cs152submitall) command on turing.  In addition, please turn in a writeup of your experiments in class. Your writeup should explain as clearly as possible what experiments you did, what results you got, and what you concluded from it all. Think of it as a science lab report.

Based on an assignment by James A. Anderson