Title: Introduction to Information Theory
Catalog Description: EE 478 Introduction to Information Theory (3+0+0)3
Information Content, conditional, joint and mutual entropy. Binary symmetric Channels;channels with and without memory. Source coding algorithms and rate-distortion bounds. Channel capacity and Shannon law. Block code, cyclic codes, convolution codes.
Coordinator: Ayşin Ertüzün, Professor of Electrical Engineering
Goals: The objective of this course is to discuss fundamental concepts and limits in information theory. It starts with basic concepts of information theory such as information content, entroy and contiues to discuss Shannon’s theorems and the rate distortion theory, to illustrate the role of coding for efficient and reliable communication, to give methods to construct good codes.
At the end of this course, students will be able to:
1. Compute information content and entropy of different sources
2. Design source codes
3. Design error correcting and detecting codes
4. Commute channel capacity of various channels
Textbook: Jan C. A. van der Lubbe, Information Theory, Cambridge Universty Press, 1997.
1. Thomas M. Cover and Joy A. Thomas, “Elements of Information Theory”, John Wiley and Sons, Inc. 1991.
2. Robert M. Gray, “Entropy and Information Theory”, Springer –Verlag, 1990.
3. Robert Ash, “Information Theory”, Dover Publications
4. R. Galleger, “Information Theory and Reliable Communication”, John Wiley and Sons
5. Abramson, ”Information Theory and Coding,”
6. Masud Mansuripur , “Introduction to Information Theory”, Prentice-Hall Inc.
Prerequisites by Topic:
1. Basic Concepts in Information Theory (2 weeks)
2. Variable-Length Source Coding (2 week)
3. Information Channels and Channel capacity (3 weeks)
4. Rate Distortion Theory (2 weeks)
5. Error-Correcting Codes
• Linear Block Codes (1 week)
• Cyclic Codes (1 week)
• Convolutional Codes (1 week)
Course Structure: The class meets for three 50-minute sessions per week. 5-6 sets of homework problems are assigned per semester. There are two in-class mid-term exams and a final exam.
Laboratory Resources: None.
1. Homework sets (10%)
2. Two Midterms (25% each).
3. Final exam (40%).
(a) Apply math, science and engineering knowledge. This course is covers the principles of information theory and coding. Basic concepts of information theory, Shannon’s law, source and channel coding, channel capacity are heavily emphasized in lectures, homework sets and exams.
(c) Design a system, component or process to meet desired needs. In this course students are equipped with knowledge to design source codes and or channel code to achieve the desired needs.
Prepared By: Ayşın Ertüzün