Abstract

During Parts I and II, we have investigated the celebrated theorems in information theory: (i) source coding theorem; (ii) channel coding theorem; and (iii) source-channel separation theorem. In Part I, we introduced prominent information-theoretic notions such as entropy, mutual information and KL divergence. We also learned about the concept of prefix-free codes, typical sequences, and a couple of lower-bound&upper-bound techniques. We then employed such tools to prove the source coding theorem as well as to study an explicit code (the Huffman code) together with its Python implementation.