Data processing using information theory functionals

This paper presents an overview of three information‐theoretic methods, which have been used extensively in many areas such as signal/image processing, pattern recognition and statistical inference. These are: the maximum entropy (ME), minimum cross‐entropy (MCE) and mutual information (MI) methods. The development history of these techniques is reviewed, their essential philosophy is explained, and typical applications, supported by simulation results, are discussed.