Optimal Upper Bounds for the Divergence of Finite-Dimensional Distributions under a Given Variational Distance

Abstract

We consider the problem of finding the maximum values of divergences D(P‖Q) and D(Q‖P) for probability distributions P and Q ranging in the finite set \(\mathcal{N}=\left\{1,\;2,...,n\right\}\) provided that both the variation distance V (P,Q) between them and either the probability distribution Q or (in the case of D(P‖Q)) only the value of the minimal component q_min of the probability distribution Q are given. Precise expressions for the maximum values of these divergences are obtained. In several cases these expressions allow us to write out some explicit formulas and simple upper and lower bounds for them. Moreover, explicit formulas for the maximum of D(P‖Q) for given V (P,Q) and q_min and also for the maximum of D(Q‖P) for given Q and V (P,Q) are obtained for all possible values of these parameters.