Introduction to Neural Networks

Introduction to Neural Networks

What are neural networks?

Neural networks are computational models inspired by the massively parallel architecture of the brain. They differ radically from traditional, von Neumann-style sequential algorithms.

Why is brain-style computation interesting?

Because the brain is an "existence proof" that AI is possible!

Brains are fantastically good at:

face recognition
speech recognition
language understanding
language generation
pattern recognition
abstract, high-level perception
analogical thinking

Despite our best efforts, computers are still moderately to abysmally bad at these tasks. Why?

Brains have a number of other advantages as well:

robust and fault tolerant: nerve cells die every day without significantly affecting performance
can easily adapt to new environmental conditions by learning
can deal gracefully with incomplete, noisy, or inconsistent information
processing is highly parallel
fits inside a space of about 1400cc and weighs around 1.4kg

Why pay attention to neurobiology? (or: "Is this a CS course, or what?!")

Because arguments from biology can provide important insights. Simple example:

neurons are metabolically very expensive to maintain (25% of body's energy is used by the CNS)
strong evolutionary pressure exists to minimize the number of neurons needed for survival
human brains nevertheless contain an extraordinary number of neurons (on the order of 100 billion!)
perhaps this means that all 100 billion are necessary to attain human-level intelligence
number of simulated neurons in artificial neural networks doesn't even approach this scale
most neural net research has concentrated on developing or analyzing clever learning algorithms
maybe we're overlooking the fundamental importance of sheer network size...?

What can neural net models do?

Learn to perform tasks from examples (supervised learning)
Discover hidden regularities in data (unsupervised learning)
Behave as associative, content-addressable memories
Example:

Field is inherently interdisciplinary, with connections to:

neuroscience
cognitive science
computer science
mathematics
psychology
physics
linguistics
signal processing
information theory
statistics
philosophy

Overview of the Brain

humans have on the order of 10-100 billion neurons packed into a 3-4 pound chunk of matter
each neuron has on the order of 1000-10,000 connections to other neurons
10¹³ - 10¹⁵ total connections (compare to ~10¹¹ stars in the Milky Way galaxy)
nevertheless, the brain is a very sparsely-connected network (connectivity less than 0.0001 %)
glia cells serve as physical "scaffolding" for neurons
even more glia cells than neurons
neurons do not regenerate (with a few exceptions)
massive neuron death before birth (up to half of all neurons die)

Terminology

soma - cell body
dendrites ("processes") - input fibers to the neuron cell body
dendritic tree - full branching system of dendrites
axon - single output fiber (may be several feet long)
terminal arborization - branching structure at the far end of the axon
synapse - connection between a terminal axon branch and the dendrite of another neuron
presynaptic cell - the neuron providing the signal at a synapse
postsynaptic cell - the neuron receiving the signal at a synapse
synaptic cleft - the space between the presynaptic membrane and the postsynaptic membrane of a synapse
synaptic vesicles - small bundles of chemical neurotransmitters stored in the synapse
dendritic spines - small nodules protruding from dendrites; locus of synapses