S-72.340 Information Theory (3 cr) P fall 2004
In 1948, Claude Shannon published his paper "A mathematical theory of
communication" -- and information theory was born. The results took the
scientific community by surprise. It was generally believed that
increasing the transmission rate of information over a communication
channel increased the probability of error. But Shannon proved that
this is not true as long as the communication rate is below channel
This course gives an introduction to information theory and its most
important applications to communications. The topic is mathematically
oriented. The basic concepts of entropy, relative entropy, and mutual
information are defined, and their connections to channel capacity,
coding, and data compression are presented. In addition to limits for
error-free communication, information theory also presents limits for
data compression. Whereas coding methods for error control are
discussed in the companion course S-72.341, data compression methods
are here discussed in more detail, including Huffman, Lempel-Ziv, and
Shannon coding. The course book provides a variety of interesting areas
of application outside communications, including gambling and
investment (stock market).
The teaching language is English. (P = may be included in
- Learning objectives: Upon completion of the course, the
students will be able to
- define and apply the basic concepts of information theory
- differentiate between lossy and lossless data compression
methods, and describe the most common such methods;
- design an efficient data compression scheme for a given
- calculate the capacity of communication channels;
- sketch Shannon's proof regarding the limits of error-free
- explain the impact of feedback and/or many senders or
receivers on the communication problem.
- Prerequisites: Basic courses in mathematics.
- Literature: T. M. Cover and J. A. Thomas, Elements of
Information Theory, Wiley, New York, 1991.
- Lectures: Thursdays 9-12, room H402, first lecture:
16.9.2004 (no lectures on 21.10, 11.11). Slides: Lecture 1. Old
slides: Lecture 2,
- Teacher: Prof. Patric
Östergård, room I447B, tel. 451 2341, e-mail:
- Tutorials: Wednesdays 10-12, room F201, first tutorial:
22.9.2004. Tutorial 1 (ps)
Tutorial 2 (ps) (pdf) Tutorial 3 (ps) (pdf) Tutorial 4 (ps) (pdf) Tutorial 5 (ps) (pdf) Tutorial 6 (ps) (pdf) Tutorial 7 (ps) (pdf) Tutorial 8 (ps) (pdf) Tutorial 9 (ps) (pdf) Tutorial 10 (ps) (pdf)
- Assistant: Petri Lindroos, M.Sc., room E206B, tel. 451
6221, e-mail: pjlindroQhut.fi (Q=@)
- To pass the course: Project, three
home assignments, home exam.
If you have any questions, do not hesitate to contact the teacher
(or the assistant).
Last update: September 16, 2004.