Arbitrary source models and Bayesian codebooks in rate-distortion theory

Ioannis Kontoyiannis, Junshan Zhang

Research output: Contribution to journalArticlepeer-review

21 Scopus citations


We characterize the best achievable performance of lossy compression algorithms operating on arbitrary random sources, and with respect to general distortion measures. Direct and converse coding theorems are given for variable-rate codes operating at a fixed distortion level, emphasizing: a) nonasymptotic results, b) optimal or near-optimal redundancy bounds, and c) results with probability one. This development is based in part on the observation that there is a precise correspondence between compression algorithms and probability measures on the reproduction alphabet. This is analogous to the Kraft inequality in lossless data compression. In the case of stationary ergodic sources our results reduce to the classical coding theorems. As an application of these general results, we examine the performance of codes based on mixture codebooks for discrete memoryless sources. A mixture codebook (or Bayesian codebook) is a random codebook generated from a mixture over some class of reproduction distributions. We demonstrate the existence of universal mixture codebooks, and show that it is possible to universally encode memoryless sources with redundancy of approximately (d/2) log n bits, where d is the dimension of the simplex of probability distributions on the reproduction alphabet.

Original languageEnglish (US)
Pages (from-to)2276-2290
Number of pages15
JournalIEEE Transactions on Information Theory
Issue number8
StatePublished - Aug 2002


  • Data compression
  • Mixture codebooks
  • Rate-distortion theory
  • Redundancy rate

ASJC Scopus subject areas

  • Information Systems
  • Computer Science Applications
  • Library and Information Sciences


Dive into the research topics of 'Arbitrary source models and Bayesian codebooks in rate-distortion theory'. Together they form a unique fingerprint.

Cite this