eit12 - Maximum entropy
Maximum entropy distribution
Let $f$ be a probability density satisfying the constraints
\[\int_Sf(x)r_i(x)=\alpha_i,~\mathbf{for}~~1\le i\le m~~(1.1)\]Let $f^\star(x)=f_\lambda(x)=e^{\lambda_0+\sum_{i=1}^{m}\lambda_ir_i(x)},~x\in S$, and let $\lambda_0,…,\lambda_m$ be chosen so that $f^\star$ satisfies (1.1). Then $f^\star$ uniquely maximizes $h(f)$ over all $f$ satisfying these constraints.
A tricky problem
Considering a tricky problem in which the $\lambda_i$ cannot be chosen to satisfy the constraints. Nonetheless, the "maximum" entropy can be found.
【reason】The entropy has a least upper bound under these constraints, but it's not possible to attain it. -> We cannot achieve it, but we can come arbitrarily close.
Consider a normal distribution with a small "wiggle" at a very high value of $x$. The moments of the new distribution are almost the same as those of the old one, the biggest change being in the third moment. We can bring the first and second moments back to their original values by adding new wiggles to balance out the changes caused by the first. By choosing the position of the wiggles, we can get any value of the third moment without reducing the entropy significantly below that of the associated normal. Using this method, we can come arbitrarily close to the upper bound for the maximum entropy distribution. We conclude that
$\sup h(f)=h(\mathcal{N}(0,\alpha_2-\alpha_1^2))=\frac{1}{2}\ln 2\pi e(\alpha_2-\alpha_1^2)$
Spectrum estimation
Given a stationary zero-mean stochastic process ${X_i}$, we define the autocorrelation function as
\[R(k)=EX_iX_{i+k}\]Differential entropy rate
The differential entropy rate of a stochastic process ${X_i}$, $X_i\in\mathcal{R}$, is defined to be
\[\begin{aligned} h(\mathcal{X})&=\lim_{n\to\infty}\frac{h(X_1,X_2,...,X_n)}{n}~~(3.1)\\ &=\lim_{n\to\infty}h(X_n\vert X_{n-1},...,X_1) \end{aligned}\]if the limit exists.
Burg's maximum entropy theorem
The maximum entropy rate stochastic process ${X_i}$ satisfying hte constraints
\[EX_iX_{i+k}=\alpha_k,k=0,1,...,p~~for~all~i\]is the pth-order Gauss-Markov process of the form
\[X_i=-\sum_{k=1}^pa_kX_{i-k}+Z_i\]where the $Z_i$ are $i.i.d.\sim\mathcal{N}(0,\sigma^2)$ and $a_1,a_2,…,a_p, \sigma^2$ are chosen to satisfy.
The entropy of a finite segment of a stochastic process is bounded above by the entropy of a segment of a Gaussian random process with the same covariance structure. This entropy is in turn bounded above by the entropy of the minimal order Gauss-Markov process satisfying the given covariance constraints.
Maximum entropy spectral density estimation
The entropy rate of a stochastic process subject to autocorrelation constraints $R_0,R_1,…,R_p$ is maximized by the $p$th order zero-mean Gauss-Markov process satisfying these constraints. The maximum entropy rate is
\[h^*=\frac{1}{2}\log(2\pi e)\frac{\vert K_p\vert}{\vert K_{p-1}\vert}~~(4.1)\]and the maximum entropy spectral density is
\[S(\lambda)=\frac{\sigma^2}{\vert 1+\sum_{k=1}^pa_ke^{-ik\lambda}\vert^2}~~(4.2)\]