Abstract

This monograph is based on lecture notes of a graduate course, which focuses on the relations between information theory and statistical physics. The course was delivered at the Technion during the Spring of 2010 for the first time, and its target audience consists of EE graduate students in the area of communications and information theory, as well as graduate students in Physics who have basic background in information theory. Strong emphasis is given to the analogy and parallelism between information theory and statistical physics, as well as to the insights, the analysis tools and techniques that can be borrowed from statistical physics and 'imported' to certain problem areas in information theory. This is a research trend that has been very active in the last few decades, and the hope is that by exposing the students to the meeting points between these two disciplines, their background and perspective may be expanded and enhanced. This monograph is substantially revised and expanded relative to an earlier version posted in arXiv (1006.1565v1 [cs.iT]).

Statistical Physics and Information Theory

Statistical Physics and Information Theory focuses on some of the relationships and the interplay between information theory and statistical physics - a branch of physics that deals with many-particle systems using probabilistic and statistical methods in the microscopic level. The author interlaces the physics and the information-theoretic subjects with each other, rather than giving them in two continuous, separate parts. This makes the relations between information theory and statistical physics more apparent. He also shows that, not only are the relations between information theory and statistical physics interesting academically in their own right, but moreover, they prove useful and beneficial in that they provide information-theorists with new insights and mathematical tools to deal with information-theoretic problems. These mathematical tools sometimes prove a lot more efficient than traditional tools used in information theory, and they may give either simpler expressions for performance analysis, or improved bounds, or both. The author provides examples of the techniques and insights. One example is the use of integrals in the complex plane and the saddle-point method. Another example is the analysis technique of error exponents, which stems from the random energy model, along with its insights about phase transitions.

Statistical Physics and Information Theory highlights to the reader techniques that have been been used in one branch of science which can be applied effectively in another. The point is that it is not the physics itself that may be useful, it is the way in which physicists use mathematical tools. This will bring new insights to all students and researchers in the field of information theory and communications.