10811:
3359:
9872:
10281:
7353:
16631:
9234:
4139:
4554:
6799:
13876:
10806:{\displaystyle {\begin{aligned}\sum _{n=2}^{\infty }\left\vert {\frac {1}{n(n-1)}}\sum _{x\in {\mathcal {X}}}{\frac {(Q(x)-P(x))^{n}}{Q(x)^{n-1}}}\right\vert &=\sum _{n=2}^{\infty }{\frac {1}{n(n-1)}}\sum _{x\in {\mathcal {X}}}\left\vert Q(x)-P(x)\right\vert \left\vert 1-{\frac {P(x)}{Q(x)}}\right\vert ^{n-1}\\&\leq \sum _{n=2}^{\infty }{\frac {1}{n(n-1)}}\sum _{x\in {\mathcal {X}}}\left\vert Q(x)-P(x)\right\vert \\&\leq \sum _{n=2}^{\infty }{\frac {1}{n(n-1)}}\\&=1\end{aligned}}}
12966:
16248:
3732:
4147:
9867:{\displaystyle {\begin{aligned}f''(\alpha )&=\sum _{x\in {\mathcal {X}}}{\frac {(P(x)-Q(x))^{2}}{(1-\alpha )Q(x)+\alpha P(x)}}\\f''(0)&=\sum _{x\in {\mathcal {X}}}{\frac {(P(x)-Q(x))^{2}}{Q(x)}}\\f^{(n)}(\alpha )&=(-1)^{n}(n-2)!\sum _{x\in {\mathcal {X}}}{\frac {(P(x)-Q(x))^{n}}{\left((1-\alpha )Q(x)+\alpha P(x)\right)^{n-1}}}\\f^{(n)}(0)&=(-1)^{n}(n-2)!\sum _{x\in {\mathcal {X}}}{\frac {(P(x)-Q(x))^{n}}{Q(x)^{n-1}}}\end{aligned}}}
9229:
18752:
7348:{\displaystyle {\begin{aligned}D_{\text{KL}}(P\parallel Q)&=\int _{x_{a}}^{x_{b}}p(x)\log \left({\frac {p(x)}{q(x)}}\right)\,dx\\&=\int _{x_{a}}^{x_{b}}{\tilde {p}}(y(x))|{\frac {dy}{dx}}(x)|\log \left({\frac {{\tilde {p}}(y(x))\,|{\frac {dy}{dx}}(x)|}{{\tilde {q}}(y(x))\,|{\frac {dy}{dx}}(x)|}}\right)\,dx\\&=\int _{y_{a}}^{y_{b}}{\tilde {p}}(y)\log \left({\frac {{\tilde {p}}(y)}{{\tilde {q}}(y)}}\right)\,dy\end{aligned}}}
5130:
13570:
22616:
17563:
12680:
16626:{\displaystyle {\begin{aligned}\left|{\frac {\partial p}{\partial \rho }}\right|&<F(x):\int _{x=0}^{\infty }F(x)\,dx<\infty ,\\\left|{\frac {\partial ^{2}p}{\partial \rho ^{2}}}\right|&<G(x):\int _{x=0}^{\infty }G(x)\,dx<\infty \\\left|{\frac {\partial ^{3}\log(p)}{\partial \rho ^{3}}}\right|&<H(x):\int _{x=0}^{\infty }p(x,0)H(x)\,dx<\xi <\infty \end{aligned}}}
4134:{\displaystyle {\begin{aligned}D_{\text{KL}}(P\parallel Q)&=\sum _{x\in {\mathcal {X}}}P(x)\ln \left({\frac {P(x)}{Q(x)}}\right)\\&={\frac {9}{25}}\ln \left({\frac {9/25}{1/3}}\right)+{\frac {12}{25}}\ln \left({\frac {12/25}{1/3}}\right)+{\frac {4}{25}}\ln \left({\frac {4/25}{1/3}}\right)\\&={\frac {1}{25}}\left(32\ln(2)+55\ln(3)-50\ln(5)\right)\approx 0.0852996,\end{aligned}}}
8864:
4549:{\displaystyle {\begin{aligned}D_{\text{KL}}(Q\parallel P)&=\sum _{x\in {\mathcal {X}}}Q(x)\ln \left({\frac {Q(x)}{P(x)}}\right)\\&={\frac {1}{3}}\ln \left({\frac {1/3}{9/25}}\right)+{\frac {1}{3}}\ln \left({\frac {1/3}{12/25}}\right)+{\frac {1}{3}}\ln \left({\frac {1/3}{4/25}}\right)\\&={\frac {1}{3}}\left(-4\ln(2)-6\ln(3)+6\ln(5)\right)\approx 0.097455.\end{aligned}}}
10248:
18314:
8391:
5531:
2417:
13871:{\displaystyle D_{\text{KL}}\left({\mathcal {N}}\left(\left(\mu _{1},\ldots ,\mu _{k}\right)^{\mathsf {T}},\operatorname {diag} \left(\sigma _{1}^{2},\ldots ,\sigma _{k}^{2}\right)\right)\parallel {\mathcal {N}}\left(\mathbf {0} ,\mathbf {I} \right)\right)={1 \over 2}\sum _{i=1}^{k}\left(\sigma _{i}^{2}+\mu _{i}^{2}-1-\ln \left(\sigma _{i}^{2}\right)\right).}
16854:
13555:
14077:
12961:{\displaystyle D_{\text{KL}}\left({\mathcal {N}}_{0}\parallel {\mathcal {N}}_{1}\right)={\frac {1}{2}}\left(\operatorname {tr} \left(\Sigma _{1}^{-1}\Sigma _{0}\right)-k+\left(\mu _{1}-\mu _{0}\right)^{\mathsf {T}}\Sigma _{1}^{-1}\left(\mu _{1}-\mu _{0}\right)+\ln \left({\frac {\det \Sigma _{1}}{\det \Sigma _{0}}}\right)\right).}
17298:
15963:
21959:
9985:
8156:
1978:
18747:{\displaystyle {\begin{aligned}\mathrm {H} (X\mid Y)&=\log(N)-D_{\text{KL}}(P(X,Y)\parallel P_{U}(X)P(Y))\\&=\log(N)-D_{\text{KL}}(P(X,Y)\parallel P(X)P(Y))-D_{\text{KL}}(P(X)\parallel P_{U}(X))\\&=\mathrm {H} (X)-\operatorname {I} (X;Y)\\&=\log(N)-\operatorname {E} _{Y}\left\end{aligned}}}
24286:
16241:
20834:
5089:
Consider a growth-optimizing investor in a fair game with mutually exclusive outcomes (e.g. a “horse race” in which the official odds add up to one). The rate of return expected by such an investor is equal to the relative entropy between the investor's believed probabilities and the official odds.
17891:
5120:
via information geometry. Investors' views, the prevailing market view, and risky scenarios form triangles on the relevant manifold of probability distributions. The shape of the triangles determines key financial risks (both qualitatively and quantitatively). For instance, obtuse triangles in which
23703:
The resulting contours of constant relative entropy, shown at right for a mole of Argon at standard temperature and pressure, for example put limits on the conversion of hot to cold as in flame-powered air-conditioning or in the unpowered device to convert boiling-water to ice-water discussed here.
9224:{\displaystyle {\begin{aligned}f'(\alpha )&=\sum _{x\in {\mathcal {X}}}(P(x)-Q(x))\left(\log \left({\frac {(1-\alpha )Q(x)+\alpha P(x)}{Q(x)}}\right)+1\right)\\&=\sum _{x\in {\mathcal {X}}}(P(x)-Q(x))\log \left({\frac {(1-\alpha )Q(x)+\alpha P(x)}{Q(x)}}\right)\\f'(0)&=0\end{aligned}}}
5311:
20007:
15784:
8646:
2210:
2851:
15630:
16996:
23949:
are particularly well described in papers and a book by
Burnham and Anderson. In a nutshell the relative entropy of reality from a model may be estimated, to within a constant additive term, by a function of the deviations observed between data and the model's predictions (like the
16649:
13291:
1719:
13899:
23925:
Just as relative entropy of "actual from ambient" measures thermodynamic availability, relative entropy of "reality from a model" is also useful even if the only clues we have about reality are some experimental measurements. In the former case relative entropy describes
14273:
21467:
1207:
1060:
17157:
247:
17558:{\displaystyle {\begin{aligned}\operatorname {I} (X;Y)&=D_{\text{KL}}(P(X,Y)\parallel P(X)P(Y))\\&=\operatorname {E} _{X}\{D_{\text{KL}}(P(Y\mid X)\parallel P(Y))\}\\&=\operatorname {E} _{Y}\{D_{\text{KL}}(P(X\mid Y)\parallel P(X))\}\end{aligned}}}
7976:
15799:
24490:
20520:
21741:
20389:
12466:
25007:– the absolute entropy of a set of data in this sense being the data required to reconstruct it (minimum compressed size), while the relative entropy of a target set of data, given a source set of data, is the data required to reconstruct the target
14485:
19205:
15409:
18203:
19692:
6576:
1821:
12290:
10243:{\displaystyle {\begin{aligned}D_{\text{KL}}(P\parallel Q)&=\sum _{n=0}^{\infty }{\frac {f^{(n)}(0)}{n!}}\\&=\sum _{n=2}^{\infty }{\frac {1}{n(n-1)}}\sum _{x\in {\mathcal {X}}}{\frac {(Q(x)-P(x))^{n}}{Q(x)^{n-1}}}\end{aligned}}}
8465:
8386:{\displaystyle D_{\text{KL}}(\lambda P_{1}+(1-\lambda )P_{2}\parallel \lambda Q_{1}+(1-\lambda )Q_{2})\leq \lambda D_{\text{KL}}(P_{1}\parallel Q_{1})+(1-\lambda )D_{\text{KL}}(P_{2}\parallel Q_{2}){\text{ for }}0\leq \lambda \leq 1.}
24124:
6732:
15229:
20636:
16099:
20641:
11709:
24751:
17723:
23842:
11827:
5526:{\displaystyle {\begin{aligned}D_{\text{KL}}(P\parallel Q)&=\sum _{x\in {\mathcal {X}}}p(x)\log {\frac {1}{q(x)}}-\sum _{x\in {\mathcal {X}}}p(x)\log {\frac {1}{p(x)}}\\&=\mathrm {H} (P,Q)-\mathrm {H} (P)\end{aligned}}}
5121:
investors' views and risk scenarios appear on “opposite sides” relative to the market describe negative risks, acute triangles describe positive exposure, and the right-angled situation in the middle corresponds to zero risk.
24903:
21021:
11257:
7573:
is dimensionless. The argument of the logarithmic term is and remains dimensionless, as it must. It can therefore be seen as in some ways a more fundamental quantity than some other properties in information theory (such as
5871:
Arthur Hobson proved that relative entropy is the only measure of difference between probability distributions that satisfies some desired properties, which are the canonical extension to those appearing in a commonly used
22465:
19825:
2412:{\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{x\in {\mathcal {X}}}{\frac {P(\mathrm {d} \ \!x)}{Q(\mathrm {d} \ \!x)}}\ \log \left({\frac {P(\mathrm {d} \ \!x)}{Q(\mathrm {d} \ \!x)}}\right)\ Q(\mathrm {d} \ \!x)\ ,}
21483:
there is very little difference between near certainty and absolute certainty—coding according to a near certainty requires hardly any more bits than coding according to an absolute certainty. On the other hand, on the
15645:
2060:
14950:
12148:
8782:
845:. The term "divergence" is in contrast to a distance (metric), since the symmetrized divergence does not satisfy the triangle inequality. Numerous references to earlier uses of the symmetrized divergence and to other
19549:
2694:
319:
Relative entropy is always a non-negative real number, with value 0 if and only if the two distributions in question are identical. It has diverse applications, both theoretical, such as characterizing the relative
24047:
18319:
15510:
16865:
16849:{\displaystyle \left.\int _{x=0}^{\infty }{\frac {\partial p(x,\rho )}{\partial \rho }}\right|_{\rho =0}\,dx=\left.\int _{x=0}^{\infty }{\frac {\partial ^{2}p(x,\rho )}{\partial \rho ^{2}}}\right|_{\rho =0}\,dx=0}
13550:{\displaystyle D_{\text{KL}}\left({\mathcal {N}}_{0}\parallel {\mathcal {N}}_{1}\right)={\frac {1}{2}}\left(\sum _{i,j=1}^{k}(M_{ij})^{2}-k+|y|^{2}+2\sum _{i=1}^{k}\ln {\frac {(L_{1})_{ii}}{(L_{0})_{ii}}}\right).}
14822:
2644:
2564:
841:, the symmetrized form is again referred to as the "divergence", and the relative entropies in each direction are referred to as a "directed divergences" between two distributions; Kullback preferred the term
17303:
6804:
2190:
1579:
20916:
is to maximise the expected relative entropy between the prior and the posterior. When posteriors are approximated to be
Gaussian distributions, a design maximising the expected relative entropy is called
22203:
22780:
15294:
14072:{\displaystyle D_{\text{KL}}\left({\mathcal {p}}\parallel {\mathcal {q}}\right)=\log {\frac {\sigma _{1}}{\sigma _{0}}}+{\frac {\sigma _{0}^{2}+(\mu _{0}-\mu _{1})^{2}}{2\sigma _{1}^{2}}}-{\frac {1}{2}}}
21488:
scale implied by weight of evidence, the difference between the two is enormous – infinite perhaps; this might reflect the difference between being almost sure (on a probabilistic level) that, say, the
7852:
5316:
2932:
14135:
12524:
24804:
17728:
16253:
10286:
9990:
9239:
8869:
4152:
3737:
23218:
21316:
11889:
5946:
18294:
1071:
927:
12040:
8839:
21599:
17073:
3135:
3023:
117:
1556:
853:, pp. 6–7, §1.3 Divergence). The asymmetric "directed divergence" has come to be known as the Kullback–Leibler divergence, while the symmetrized "divergence" is now referred to as the
5293:
in bits. Therefore, relative entropy can be interpreted as the expected extra message-length per datum that must be communicated if a code that is optimal for a given (wrong) distribution
15958:{\displaystyle g_{jk}(\theta _{0})=\left.{\frac {\partial ^{2}}{\partial \theta _{j}\,\partial \theta _{k}}}\right|_{\theta =\theta _{0}}D_{\text{KL}}(P(\theta )\parallel P(\theta _{0}))}
23891:
22568:
22526:
14681:
14639:
9914:
8434:
8020:
7807:
7719:
6370:
5992:
5843:
5667:
5021:
4979:
4848:
4775:
4701:
3702:
3660:
416:
94:
23338:
13176:
13121:
11581:
11471:
11429:
14128:
24392:
19741:
14871:
13283:
12668:
3319:
21954:{\displaystyle D_{\text{KL}}(q(x\mid a)u(a)\parallel p(x,a))=\operatorname {E} _{u(a)}\left\{D_{\text{KL}}(q(x\mid a)\parallel p(x\mid a))\right\}+D_{\text{KL}}(u(a)\parallel p(a)),}
14748:
5233:
24551:
20902:
20394:
20148:
6790:
22259:
20239:
831:
23914:
23279:
22342:. In particular, it is the natural extension of the principle of maximum entropy from discrete to continuous distributions, for which Shannon entropy ceases to be so useful (see
15468:
12301:
4639:
1793:
911:
19324:
15502:
14394:
11157:
11001:
6149:
6087:
2110:
1445:
23698:
22605:
12622:
6284:
5571:
665:
585:
22913:
20196:
15075:
3468:
23581:
23411:
19246:
19014:
16036:
14401:
6247:
5614:
3496:
3440:
21733:
11376:
10914:
7571:
7451:
7402:
3613:
3585:
3557:
2476:
1415:
24581:
23461:
16091:
8151:
8105:
5118:
24676:
24667:
24380:
23022:
19817:
19430:
19085:
18815:
15323:
15033:
13227:
11283:
10966:
5772:
1374:
22704:
22528:
subject to some constraint. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be
20083:
18072:
15997:
15122:
13066:
7847:
7631:
625:
24351:
22123:
22088:
20231:
19776:
19389:
14557:
14524:
10940:
1315:
1274:
24817:
23085:
22954:
9980:
6193:
5264:
1973:{\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{x\in {\mathcal {X}}}\ \log \left({\frac {P(\mathrm {d} \ \!x)}{Q(\mathrm {d} \ \!x)}}\right)\ P(\mathrm {d} \ \!x)\ ,}
24325:
24116:
19564:
18894:
18023:
17950:
17645:
11102:
10839:
10276:
8675:
2678:
1483:
1343:
754:
727:
545:
24929:
15314:
15095:
14998:
12557:
11937:
11352:
11332:
11073:
11027:
10868:
21637:
18931:
18850:
17692:
17604:
13019:
11523:
11497:
6415:
5718:
700:
23542:
23515:
23488:
23376:
23143:
23116:
22831:
22644:
21550:
21275:
21248:
21218:
21191:
21160:
21133:
21106:
21079:
21052:
12156:
11968:
11122:
11047:
8059:
5291:
5170:
3353:
3233:
518:
491:
24281:{\displaystyle D_{\lambda }(P\parallel Q)=\lambda D_{\text{KL}}(P\parallel \lambda P+(1-\lambda )Q)+(1-\lambda )D_{\text{KL}}(Q\parallel \lambda P+(1-\lambda )Q),}
22317:
22288:
22053:
22024:
21991:
21678:
21308:
18052:
17979:
17252:
17189:
7517:
7488:
6410:
6212:
5801:
3529:
3412:
2874:
2452:
22980:
20040:
19350:
17278:
17219:
15435:
8460:
6021:
3207:
22800:
14356:
14318:
3260:
16236:{\displaystyle {\frac {\partial \log(p)}{\partial \rho }},{\frac {\partial ^{2}\log(p)}{\partial \rho ^{2}}},{\frac {\partial ^{3}\log(p)}{\partial \rho ^{3}}}}
11177:
10888:
9954:
9934:
8859:
8695:
6581:
21476:
in
Bayesian experimental design, to choose an optimal next question to investigate: but they will in general lead to rather different experimental strategies.
20829:{\displaystyle D_{\text{KL}}{\big (}p(x\mid y_{1},y_{2},I)\parallel p(x\mid y_{1},I){\big )}+D_{\text{KL}}{\big (}p(x\mid y_{1},I)\parallel p(x\mid I){\big )}}
17886:{\displaystyle {\begin{aligned}\mathrm {H} (X)&=\operatorname {E} \left\\&=\log(N)-D_{\text{KL}}\left(p_{X}(x)\parallel P_{U}(X)\right)\end{aligned}}}
15134:
20535:
11589:
23490:
is the probability of a given state under ambient conditions. For instance, the work available in equilibrating a monatomic ideal gas to ambient values of
23745:
12985:
logarithms of expressions that are either factors of the density function or otherwise arise naturally. The equation therefore gives a result measured in
11720:
26665:
25718:
23941:
Although this tool for evaluating models against systems that are accessible experimentally may be applied in any field, its application to selecting a
25687:
20932:
20002:{\displaystyle D_{\text{KL}}{\big (}p(x\mid y,I)\parallel p(x\mid I){\big )}=\sum _{x}p(x\mid y,I)\log \left({\frac {p(x\mid y,I)}{p(x\mid I)}}\right)}
11182:
15779:{\displaystyle D_{\text{KL}}(P(\theta )\parallel P(\theta _{0}))={\frac {1}{2}}\,\Delta \theta _{j}\,\Delta \theta _{k}\,g_{jk}(\theta _{0})+\cdots }
22385:
8641:{\displaystyle D_{\text{KL}}(P\parallel Q)=\sum _{n=2}^{\infty }{\frac {1}{n(n-1)}}\sum _{x\in {\mathcal {X}}}{\frac {(Q(x)-P(x))^{n}}{Q(x)^{n-1}}}}
26511:
24631:
19555:
7453:. Although it was assumed that the transformation was continuous, this need not be the case. This also shows that the relative entropy produces a
2846:{\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{x\in {\mathcal {X}}}p(x)\ \log \left({\frac {\ p(x)\ }{q(x)}}\right)\ \mu (\mathrm {d} \ \!x)~.}
1986:
14886:
12045:
8706:
15625:{\displaystyle \left.{\frac {\partial }{\partial \theta _{j}}}\right|_{\theta =\theta _{0}}D_{\text{KL}}(P(\theta )\parallel P(\theta _{0}))=0,}
19442:
16991:{\displaystyle {\mathcal {D}}(p(x,0)\parallel p(x,\rho ))={\frac {c\rho ^{2}}{2}}+{\mathcal {O}}\left(\rho ^{3}\right){\text{ as }}\rho \to 0.}
5024:
26157:
3036:, but this fails to convey the fundamental asymmetry in the relation. Sometimes, as in this article, it may be described as the divergence of
23982:
19217:
Under this scenario, relative entropies (kl-divergence) can be interpreted as the extra number of bits, on average, that are needed (beyond
14756:
6302:; and the Kullback–Leibler divergence therefore represents the expected number of extra bits that must be transmitted to identify a value
2569:
2489:
13564:, is the relative entropy between a diagonal multivariate normal, and a standard normal distribution (with zero mean and unit variance):
18965:(e.g.: the events (A, B, C) with probabilities p = (1/2, 1/4, 1/4) can be encoded as the bits (0, 10, 11)). If we know the distribution
5900:
1714:{\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{-\infty }^{\infty }p(x)\ \log \left({\frac {p(x)}{q(x)}}\right)\ \mathrm {d} \ \!x\ ,}
2115:
288:, which is the most familiar and formal type of distance. In particular, it is not symmetric in the two distributions (in contrast to
25731:
Donsker, Monroe D.; Varadhan, SR Srinivasa (1983). "Asymptotic evaluation of certain Markov process expectations for large time. IV".
26652:
17659:
if they are coded using only their marginal distributions instead of the joint distribution. Equivalently, if the joint probability
26532:
24614:
Furthermore, the Jensen–Shannon divergence can be generalized using abstract statistical M-mixtures relying on an abstract mean M.
22136:
22713:
15237:
19743:. However, from the standpoint of the new probability distribution one can estimate that to have used the original code based on
23957:
When trying to fit parametrized models to data there are various estimators which attempt to minimize relative entropy, such as
5881:
439:
19059:
needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution
14268:{\displaystyle D_{\text{KL}}\left({\mathcal {p}}\parallel {\mathcal {q}}\right)=\log _{2}k+(k^{-2}-1)/2/\ln(2)\mathrm {bits} }
4884:
typically represents the "true" distribution of data, observations, or a precisely calculated theoretical distribution, while
26586:
26487:
26446:
26433:, Lecture Notes in Mathematics (in French), vol. 649, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 342–363,
26261:
25978:
25649:
25075:
21493:
is correct, compared to being certain that it is correct because one has a mathematical proof. These two different scales of
17030:
Many of the other quantities of information theory can be interpreted as applications of relative entropy to specific cases.
2879:
284:. While it is a measure of how different two distributions are, and in some sense is thus a "distance", it is not actually a
25997:
Thermostatics and
Thermodynamics: An Introduction to Energy, Information and States of Matter, with Engineering Applications
5873:
21462:{\displaystyle D_{\text{KL}}(p(x\mid H_{1})\parallel p(x\mid H_{0}))\neq IG=D_{\text{KL}}(p(H\mid x)\parallel p(H\mid I)).}
12474:
24756:
23954:) . Estimates of such divergence for models that share the same additive term can in turn be used to select among models.
1202:{\displaystyle D_{\text{KL}}(P\parallel Q)=-\sum _{x\in {\mathcal {X}}}P(x)\ \log \left({\frac {\ Q(x)\ }{P(x)}}\right)~.}
1055:{\displaystyle D_{\text{KL}}(P\parallel Q)=\sum _{x\in {\mathcal {X}}}P(x)\ \log \left({\frac {\ P(x)\ }{Q(x)}}\right)\ ,}
18973:). Meaning the messages we encode will have the shortest length on average (assuming the encoded events are sampled from
18059:
23148:
19024:
will be used (on average) to identify an event from a set of possibilities. This new (larger) number is measured by the
17152:{\displaystyle \operatorname {\operatorname {I} } (m)=D_{\text{KL}}\left(\delta _{\text{im}}\parallel \{p_{i}\}\right),}
11839:
242:{\displaystyle D_{\text{KL}}(P\parallel Q)=\sum _{x\in {\mathcal {X}}}P(x)\ \log \left({\frac {\ P(x)\ }{Q(x)}}\right).}
25671:
25065:
18214:
15411:
the corresponding rate of change in the probability distribution. Since relative entropy has an absolute minimum 0 for
6330:
are two discrete probability distributions built by distributing the same discrete quantity, then the maximum value of
4778:
11977:
10841:
a.s. is also a necessary condition for convergence of the series by the following proof by contradiction. Assume that
8787:
3239:
is the distribution on the right side of the figure, a discrete uniform distribution with the three possible outcomes
26614:
25479:
25373:
25170:
24056:
for the evolution of the term). This function is symmetric and nonnegative, and had already been defined and used by
21555:
18961:. This compresses the data by replacing each fixed-length input symbol with a corresponding unique, variable-length,
17648:
3092:
2980:
866:
24052:
which they referred to as the "divergence", though today the "KL divergence" refers to the asymmetric function (see
7971:{\displaystyle D_{\text{KL}}(P\parallel Q)=D_{\text{KL}}(P_{1}\parallel Q_{1})+D_{\text{KL}}(P_{2}\parallel Q_{2}).}
1491:
26686:
24811:
22920:
22331:
12581:
17:
24485:{\displaystyle D_{\text{JS}}={\frac {1}{2}}D_{\text{KL}}(P\parallel M)+{\frac {1}{2}}D_{\text{KL}}(Q\parallel M)}
24087:
of a feature under two different classes. In the
Banking and Finance industries, this quantity is referred to as
23854:
22531:
22489:
14644:
14602:
9877:
8397:
7983:
7724:
7636:
6333:
5955:
5806:
5630:
4984:
4942:
4811:
4738:
4664:
3665:
3623:
379:
57:
23284:
13126:
13071:
11544:
11434:
11392:
25045:
25030:
20515:{\displaystyle \displaystyle \sum _{x}p(x\mid y_{1},I)\log \left({\frac {p(x\mid y_{1},I)}{p(x\mid I)}}\right)}
17569:
14085:
20012:
to the message length. This therefore represents the amount of useful information, or information gain, about
19700:
14834:
13232:
12630:
3276:
26696:
25080:
24383:
20384:{\displaystyle \sum _{x}p(x\mid y_{1},y_{2},I)\log \left({\frac {p(x\mid y_{1},y_{2},I)}{p(x\mid I)}}\right)}
15125:
14701:
5179:
5142:
5056:
1800:
25758:
Lee, Se Yoon (2021). "Gibbs sampler and coordinate ascent variational inference: A set-theoretical review".
24505:
20846:
20092:
12461:{\displaystyle E_{Q}-D_{\text{KL}}(Q\parallel P)=\log E_{P}-D_{\text{KL}}(Q\parallel Q^{*})\leq \log E_{P},}
10278:
a.s. is a sufficient condition for convergence of the series by the following absolute convergence argument
6737:
5090:
This is a special case of a much more general connection between financial returns and divergence measures.
25025:
24084:
23946:
23934:, while in the latter case it tells you about surprises that reality has up its sleeve or, in other words,
22335:
22208:
20913:
13028:
In a numerical implementation, it is helpful to express the result in terms of the
Cholesky decompositions
7590:
2423:
2063:
1733:
1447:
is infinite in extent. Analogous comments apply to the continuous and general measure cases defined below.
321:
22619:
Pressure versus volume plot of available work from a mole of argon gas relative to ambient, calculated as
759:
26203:
26106:
A Method of
Geometrical Representation of the Thermodynamic Properties of Substances by Means of Surfaces
24961:
23915:
Maximum likelihood estimation § Relation to minimizing
Kullback–Leibler divergence and cross entropy
23848:
23223:
15440:
5074:
4588:
1765:
883:
19279:
15477:
14480:{\displaystyle D_{\text{KL}}\left({\mathcal {p}}\parallel {\mathcal {q}}\right)=\log {\frac {D-C}{B-A}}}
14361:
11127:
10971:
6092:
6030:
5880:
is the only measure of mutual dependence that obeys certain related conditions, since it can be defined
2085:
1420:
24072:
23962:
23589:
22573:
17053:
12587:
6375:
Relative entropy remains well-defined for continuous distributions, and furthermore is invariant under
6252:
5539:
1570:
630:
550:
305:
22874:
20153:
15038:
3446:
25315:
24967:
24600:
23547:
23384:
21501:
useful, according to how well each reflects the particular circumstances of the problem in question.
19220:
19200:{\displaystyle \mathrm {H} (p,q)=\operatorname {E} _{p}=\mathrm {H} (p)+D_{\text{KL}}(p\parallel q).}
18988:
16002:
15404:{\displaystyle P_{j}\left(\theta _{0}\right)={\frac {\partial P}{\partial \theta _{j}}}(\theta _{0})}
14969:
14695:
12976:
6221:
5588:
5036:
3709:
3474:
3418:
418:
is then interpreted as the average difference of the number of bits required for encoding samples of
21691:
21509:
The idea of relative entropy as discrimination information led
Kullback to propose the Principle of
18198:{\displaystyle \lim _{N\rightarrow \infty }H_{N}(X)=\log(N)-\int p(x)\log {\frac {p(x)}{m(x)}}\,dx,}
11357:
10893:
7522:
7407:
7358:
3591:
3563:
3535:
2457:
1379:
25085:
25055:
24559:
23728:
23416:
19044:
17064:
17007:
16057:
15969:
8110:
8064:
5096:
4570:
3358:
1213:
289:
101:
24637:
24359:
22992:
19781:
19394:
18772:
15003:
13189:
11262:
10945:
5727:
1350:
26701:
24670:
23087:
is minimized instead. The change in free energy under these conditions is a measure of available
22660:
20048:
19687:{\displaystyle \mathrm {H} {\big (}p(x\mid y,I){\big )}=-\sum _{x}p(x\mid y,I)\log p(x\mid y,I),}
15975:
15100:
14877:
14580:
13031:
7812:
7596:
5682:
5145:
establishes that any directly decodable coding scheme for coding a message to identify one value
4920:
590:
301:
31:
24330:
23091:
that might be done in the process. Thus available work for an ideal gas at constant temperature
22093:
22058:
20201:
19746:
19359:
14529:
14496:
10919:
1279:
1238:
26691:
25471:
25102:
25070:
25050:
24583:
as the capacity of a noisy information channel with two inputs giving the output distributions
23951:
23046:
22933:
19273:
9959:
6571:{\displaystyle P(dx)=p(x)\,dx={\tilde {p}}(y)\,dy={\tilde {p}}(y(x))|{\tfrac {dy}{dx}}(x)|\,dx}
6154:
5242:
5078:
5052:
4905:
4858:
3083:
25804:
25604:
24310:
24101:
18863:
17992:
17919:
17609:
12285:{\displaystyle D_{\text{KL}}(Q\parallel Q^{*})-D_{\text{KL}}(Q\parallel P)=-E_{Q}+\log E_{P}.}
11078:
10815:
10252:
8651:
2657:
1453:
1325:
732:
705:
523:
24914:
23088:
22987:
15299:
15080:
14983:
13561:
12529:
11910:
11337:
11317:
11052:
11006:
10844:
3180:
2959:
1756:
21607:
18899:
18820:
17662:
17574:
12992:
11502:
11476:
5688:
2958:
etc. for continuous distributions. The logarithms in these formulae are usually taken to
670:
368:
represents the data, the observations, or a measured probability distribution. Distribution
26601:
26381:
26310:
26179:
26130:
26077:
26027:
25904:
25545:
25201:
24623:
24608:
23898:
23520:
23493:
23466:
23354:
23121:
23094:
22868:
22864:
22805:
22622:
22344:
21528:
21253:
21226:
21196:
21169:
21138:
21111:
21084:
21057:
21030:
18978:
18954:
18063:
14576:
14572:
11946:
11107:
11032:
8032:
7454:
5949:
5670:
5269:
5148:
5048:
5044:
4916:
4912:
3723:
3324:
3212:
846:
496:
469:
443:
297:
285:
97:
26253:
Model selection and multimodel inference : a practical information-theoretic approach
25690:(2020). "Kullback-Leibler divergence between quantum distributions, and its upper-bound".
22293:
22264:
22029:
22000:
21967:
21654:
21284:
18028:
17955:
17228:
17165:
7493:
7464:
6727:{\displaystyle Q(dx)=q(x)\,dx={\tilde {q}}(y)\,dy={\tilde {q}}(y)|{\tfrac {dy}{dx}}(x)|dx}
6386:
6197:
5777:
3505:
3388:
2859:
2437:
8:
25924:
22959:
20019:
19329:
19265:
18305:
17257:
17198:
17049:
17039:
17011:
15414:
14596:
14588:
8439:
8027:
6376:
6000:
5617:
5134:
4936:
4928:
3186:
2974:. Most formulas involving relative entropy hold regardless of the base of the logarithm.
1318:
313:
293:
26385:
26314:
26183:
26134:
26081:
26031:
25549:
24908:
22785:
21281:
the same as the information gain expected per sample about the probability distribution
17697:
known, it is the expected number of extra bits that must on average be sent to identify
15224:{\displaystyle P(\theta )=P(\theta _{0})+\Delta \theta _{j}\,P_{j}(\theta _{0})+\cdots }
14323:
14285:
3242:
26606:
26566:
26549:
26527:
26505:
26452:
26404:
26371:
26357:
26333:
26300:
26286:
26195:
26169:
26142:
26043:
25785:
25767:
25691:
25568:
25535:
25523:
25508:
25464:
25332:
25189:
25060:
25035:
24936:
23958:
21490:
20631:{\displaystyle D_{\text{KL}}{\big (}p(x\mid y_{1},y_{2},I)\parallel p(x\mid I){\big )}}
19269:
17289:
17015:
11387:
11162:
10873:
9939:
9919:
8844:
8680:
5877:
5063:
5051:), and this allows one to minimize relative entropy by geometric means, for example by
4805:
3072:
3064:
309:
26358:"On a Generalization of the Jensen–Shannon Divergence and the Jensen–Shannon Centroid"
14691:
11704:{\displaystyle \log E_{P}={\text{sup}}_{Q\ll P}\{E_{Q}-D_{\text{KL}}(Q\parallel P)\}.}
26610:
26582:
26493:
26483:
26456:
26442:
26409:
26338:
26267:
26257:
25974:
25968:
25878:
25850:
25842:
25789:
25667:
25645:
25641:
25600:
25573:
25475:
25432:
25369:
25012:
25001:
24980:
24955:
24746:{\displaystyle \delta (P,Q)\leq {\sqrt {{\frac {1}{2}}D_{\text{KL}}(P\parallel Q)}}.}
24068:
23942:
23041:
19433:
19268:, relative entropy can be used as a measure of the information gain in moving from a
19076:
17019:
16039:
12625:
6215:
4851:
3705:
438:
can be reversed in some situations where that is easier to compute, such as with the
26573:. Applied Mathematical Sciences. Vol. 194. Springer Japan. pp. XIII, 374.
26553:
26427:
Bretagnolle, J.; Huber, C. (1978), "Estimation des densités : Risque minimax",
26219:"Kullback–Leibler information as a basis for strong inference in ecological studies"
26199:
26047:
23837:{\displaystyle D_{\text{KL}}(P\parallel Q)=\operatorname {Tr} (P(\log(P)-\log(Q))).}
22290:; and we note that this result incorporates Bayes' theorem, if the new distribution
15504:. More formally, as for any minimum, the first derivatives of the divergence vanish
11822:{\displaystyle {\frac {Q(d\theta )}{P(d\theta )}}={\frac {\exp h(\theta )}{E_{P}}},}
2200:. Also we assume the expression on the right-hand side exists. Equivalently (by the
26574:
26541:
26434:
26399:
26389:
26328:
26318:
26230:
26187:
26138:
26085:
26062:
26035:
26012:
25964:
25943:
25933:
25834:
25777:
25740:
25618:
Sanov, I.N. (1957). "On the probability of large deviations of random magnitudes".
25563:
25553:
25504:
25328:
25324:
25310:
25179:
25157:
25137:
25090:
24990:
24945:
24604:
23341:
21473:
20918:
18962:
18958:
17985:
the expected number of bits saved, which would have had to be sent if the value of
17045:
7575:
5865:
5849:
loss. The resulting function is asymmetric, and while this can be symmetrized (see
4732:
3148:
2943:
2939:
2935:
1760:
455:
345:
26653:
Matlab code for calculating
Kullback–Leibler divergence for discrete distributions
25781:
24898:{\displaystyle \delta (P,Q)\leq {\sqrt {1-e^{-D_{\mathrm {KL} }(P\parallel Q)}}}.}
21016:{\textstyle D_{\text{KL}}{\bigl (}p(x\mid H_{1})\parallel p(x\mid H_{0}){\bigr )}}
18856:
the expected number of bits saved which would have had to be sent if the value of
14972:. This can be made explicit as follows. Assume that the probability distributions
1485:
is zero the contribution of the corresponding term is interpreted as zero because
26104:
25995:
25363:
25197:
25161:
25094:
24057:
23920:
21525:
should be chosen which is as hard to discriminate from the original distribution
17714:
17192:
14957:
11714:
Further, the supremum on the right-hand side is attained if and only if it holds
11538:
11252:{\displaystyle {\frac {1}{n(n-1)}}\rho \left(1+{\frac {\epsilon }{U}}\right)^{n}}
8023:
7579:
834:
459:
333:
26657:
25715:
20233:, it turns out that it may be either greater or less than previously estimated:
4585:(drawn from one of them) is through the log of the ratio of their likelihoods:
24626:. Some of these are particularly connected with relative entropy. For example:
24095:), and is used to assess distributional shifts in model features through time.
23713:
18970:
17909:
15790:
12564:
11104:
is bounded, so we need only concern ourselves with the behavior of the measure
6287:
5995:
5028:
4708:
4573:
states that the most powerful way to distinguish between the two distributions
341:
265:
26578:
26545:
25882:
25822:
25436:
25184:
25165:
25126:"I-Divergence Geometry of Probability Distributions and Minimization Problems"
22460:{\displaystyle \mathrm {H} (p,m)=\mathrm {H} (p)+D_{\text{KL}}(p\parallel m),}
21081:: the mean information per sample for discriminating in favor of a hypothesis
17651:— i.e. the expected number of extra bits that must be transmitted to identify
4789:
which is currently used. By analogy with information theory, it is called the
26680:
26497:
26287:"On the Jensen–Shannon Symmetrization of Distances Relying on Abstract Means"
26271:
25846:
25838:
25142:
25125:
25040:
23724:
22470:
which is appropriate if one is trying to choose an adequate approximation to
21494:
19040:
19025:
15636:
5861:
5721:
5574:
5032:
4889:
4658:
26661:
26089:
26039:
25938:
25919:
25711:
24911:
generalize relative entropy. Depending on the value of a certain parameter,
22653:
add where probabilities multiply. The surprisal for an event of probability
2055:{\displaystyle \ {\frac {\ P(\mathrm {d} \ \!x)\ }{Q(\mathrm {d} \ \!x)\ }}}
26413:
26342:
25854:
25744:
25577:
25389:
22844:
Best-guess states (e.g. for atoms in a gas) are inferred by maximizing the
14945:{\displaystyle P_{n}\xrightarrow {D} P\Rightarrow P_{n}\xrightarrow {TV} P}
12143:{\displaystyle Q^{*}(d\theta )={\frac {\exp h(\theta )}{E_{P}}}P(d\theta )}
8777:{\displaystyle f(\alpha ):=D_{\text{KL}}((1-\alpha )Q+\alpha P\parallel Q)}
7586:
5894:
5067:
2951:
877:
372:
represents instead a theory, a model, a description or an approximation of
337:
4850:
is a measure of the information gained by revising one's beliefs from the
26648:
Jon Shlens' tutorial on Kullback–Leibler divergence and likelihood theory
25407:
22339:
19544:{\displaystyle p(x\mid y,I)={\frac {p(y\mid x,I)p(x\mid I)}{p(y\mid I)}}}
18969:
in advance, we can devise an encoding that would be optimal (e.g.: using
18055:
17057:
7582:), which can become undefined or negative for non-discrete probabilities.
6322:
No upper-bound exists for the general case. However, it is shown that if
5685:: an asymmetric, generalized form of squared distance. The cross-entropy
325:
26642:
26428:
25948:
25524:"Economics of Disagreement—Financial Intuition for the Rényi Divergence"
23351:
relative to some ambient is obtained by multiplying ambient temperature
21552:
as possible; so that the new data produces as small an information gain
4649:. Kullback motivated the statistic as an expected log likelihood ratio.
26438:
26191:
25963:
Press, W.H.; Teukolsky, S.A.; Vetterling, W.T.; Flannery, B.P. (2007).
25336:
25193:
24042:{\displaystyle D_{\text{KL}}(P\parallel Q)+D_{\text{KL}}(Q\parallel P)}
21163:
19352:
is discovered, it can be used to update the posterior distribution for
12986:
3719:
2971:
2201:
26530:(2000). "The earth mover's distance as a metric for image retrieval".
26394:
26323:
25872:
25558:
25426:
24617:
14980:
are both parameterized by some (possibly multi-dimensional) parameter
2934:
although in practice it will usually be one that in the context like
26671:
26477:
26251:
25591:
Soklakov, A. N. (2023). "Information Geometry of Risks and Returns".
25495:
Kelly, J. L. Jr. (1956). "A New Interpretation of Information Rate".
22650:
19020:) when creating the entropy encoding scheme, then a larger number of
18757:
is the number of bits which would have to be transmitted to identify
17896:
is the number of bits which would have to be transmitted to identify
17221:— i.e. the number of extra bits that must be transmitted to identify
17025:
14817:{\displaystyle \lim _{n\to \infty }D_{\text{KL}}(P_{n}\parallel Q)=0}
12972:
12674:, then the relative entropy between the distributions is as follows:
3060:
2955:
2876:
for which densities can be defined always exists, since one can take
329:
269:
26637:
26235:
26218:
25823:"Distributions of the Kullback-Leibler divergence with applications"
22615:
18949:
When we have a set of possible events, coming from the distribution
14930:
14904:
14853:
5066:
generated by the negative entropy, but it is also of the form of an
2639:{\displaystyle \ Q(\mathrm {d} \ \!x)=q(x)\mu (\mathrm {d} \ \!x)\ }
2559:{\displaystyle \ P(\mathrm {d} \ \!x)=p(x)\mu (\mathrm {d} \ \!x)\ }
26376:
26305:
25772:
25696:
25540:
24810:. For such distributions, an alternative bound can be used, due to
23704:
Thus relative entropy measures thermodynamic availability in bits.
21680:, then the relative entropy between the new joint distribution for
6793:
5129:
5043:. Relative entropy satisfies a generalized Pythagorean theorem for
3167:
Kullback gives the following example (Table 2.1, Example 2.1). Let
26647:
26174:
21504:
11049:. The previous proof of sufficiency demonstrated that the measure
6310:, if a code is used corresponding to the probability distribution
26362:
26291:
22852:
22327:
11179:
th term of this component of the series is then lower bounded by
6792:
is the absolute value of the derivative or more generally of the
5297:
is used, compared to using a code based on the true distribution
5176:
can be seen as representing an implicit probability distribution
2185:{\displaystyle \ P(\mathrm {d} \ \!x)=r(x)Q(\mathrm {d} \ \!x)\ }
25962:
24291:
which can be interpreted as the expected information gain about
22478:
the task one is trying to achieve. Instead, just as often it is
22055:) of the relative entropy of the prior conditional distribution
25390:"What's the maximum value of Kullback-Leibler (KL) divergence?"
25313:(1987). "Letter to the Editor: The Kullback–Leibler distance".
25266:
25264:
23348:
18058:'s alternative generalization to continuous distributions, the
14579:
on the space of probability distributions, but instead it is a
11298:
The following result, due to Donsker and Varadhan, is known as
4641:. The KL divergence is the expected value of this statistic if
2947:
26121:
Tribus, M.; McIrvine, E. C. (1971). "Energy and information".
25874:
An intuition for physicists: information gain from experiments
24607:(in the sense that it induces the same affine connection on a
22348:), but the relative entropy continues to be just as relevant.
22026:, plus the expected value (using the probability distribution
5023:, and the asymmetry is an important part of the geometry. The
22198:{\displaystyle D_{\text{KL}}(q(x\mid a)\parallel p(x\mid a))}
21485:
14526:
bits. This connects with the use of bits in computing, where
5077:, it is unique in being a member of both of these classes of
4703:
can be constructed by measuring the expected number of extra
25261:
24753:
Pinsker's inequality is vacuous for any distributions where
22775:{\displaystyle \left\{1,1/\ln 2,1.38\times 10^{-23}\right\}}
19697:
which may be less than or greater than the original entropy
19016:). However, if we use a different probability distribution (
15289:{\displaystyle \Delta \theta _{j}=(\theta -\theta _{0})_{j}}
11293:
9231:
Further derivatives may be derived and evaluated as follows
4864:. In other words, it is the amount of information lost when
328:, and information gain when comparing statistical models of
23040:
are held constant (say during processes in your body), the
22351:
In the engineering literature, MDI is sometimes called the
16743:
16655:
15837:
15516:
5624:(which is the same as the cross-entropy of P with itself).
25827:
British Journal of Mathematical and Statistical Psychology
23930:
or (when multiplied by ambient temperature) the amount of
23908:
22125:. (Note that often the later expected value is called the
20085:, subsequently comes in, the probability distribution for
4919:, the most familiar type of distance, but instead it is a
25281:
25279:
19056:
19021:
13022:
6379:. For example, if a transformation is made from variable
6291:
4704:
2963:
1231:
Relative entropy is only defined in this way if, for all
1224:, where the expectation is taken using the probabilities
25368:(1st ed.). Cambridge University Press. p. 34.
23032:
is entropy) is minimized as a system "equilibrates." If
21162:
is true. Another name for this quantity, given to it by
18054:. This definition of Shannon entropy forms the basis of
17908:
the relative entropy of the uniform distribution on the
17162:
is the relative entropy of the probability distribution
15097:
differs by only a small amount from the parameter value
14282:
Consider two uniform distributions, with the support of
1216:
of the logarithmic difference between the probabilities
25408:"In what situations is the integral equal to infinity?"
22867:, both classically and quantum mechanically, minimizes
22855:) for a given set of control parameters (like pressure
22837:
bits of surprisal for landing all "heads" on a toss of
20150:. If one reinvestigates the information gain for using
16046:
parameter space, called the Fisher information metric.
14687:), the asymmetry is an important part of the geometry.
14591:, divergences are asymmetric in general and generalize
5845:
agree more closely with our notion of distance, as the
2927:{\displaystyle \ \mu ={\frac {1}{2}}\left(P+Q\right)\ }
308:, and for certain classes of distributions (notably an
26672:
A modern summary of info-theoretic divergence measures
25365:
Information Theory, Inference, and Learning Algorithms
25276:
25215:
25213:
25211:
20935:
6747:
6683:
6526:
4888:
typically represents a theory, model, description, or
3179:
is the distribution on the left side of the figure, a
627:
are the hypotheses that one is selecting from measure
25821:
Belov, Dmitry I.; Armstrong, Ronald D. (2011-04-15).
25443:
25343:
25251:
25249:
25236:
25234:
25232:
25230:
25228:
24917:
24820:
24759:
24679:
24640:
24562:
24508:
24395:
24362:
24333:
24313:
24127:
24104:
23985:
23857:
23748:
23592:
23550:
23523:
23496:
23469:
23419:
23387:
23357:
23287:
23226:
23151:
23124:
23097:
23049:
22995:
22962:
22936:
22877:
22808:
22788:
22716:
22663:
22625:
22576:
22534:
22492:
22388:
22296:
22267:
22211:
22139:
22096:
22061:
22032:
22003:
21970:
21744:
21694:
21657:
21610:
21558:
21531:
21319:
21287:
21256:
21229:
21199:
21172:
21141:
21114:
21087:
21060:
21033:
20849:
20644:
20538:
20398:
20397:
20242:
20204:
20156:
20095:
20051:
20022:
19828:
19784:
19749:
19703:
19567:
19445:
19397:
19362:
19332:
19282:
19223:
19088:
18991:
18902:
18866:
18823:
18775:
18317:
18217:
18075:
18031:
17995:
17958:
17922:
17726:
17665:
17612:
17577:
17301:
17260:
17231:
17201:
17168:
17076:
16868:
16652:
16251:
16102:
16060:
16005:
15978:
15802:
15648:
15513:
15480:
15443:
15417:
15326:
15302:
15240:
15137:
15124:. Specifically, up to first order one has (using the
15103:
15083:
15041:
15006:
14986:
14956:
where the latter stands for the usual convergence in
14889:
14837:
14759:
14704:
14647:
14605:
14532:
14499:
14404:
14364:
14326:
14288:
14138:
14088:
14082:
In the case of co-centered normal distributions with
13902:
13573:
13294:
13235:
13192:
13129:
13074:
13034:
12995:
12683:
12633:
12590:
12532:
12519:{\displaystyle D_{\text{KL}}(Q\parallel Q^{*})\geq 0}
12477:
12304:
12159:
12048:
11980:
11949:
11913:
11842:
11723:
11592:
11547:
11505:
11479:
11437:
11395:
11360:
11340:
11320:
11265:
11185:
11165:
11130:
11110:
11081:
11055:
11035:
11009:
10974:
10948:
10922:
10896:
10876:
10847:
10818:
10284:
10255:
9988:
9962:
9942:
9922:
9880:
9237:
8867:
8847:
8790:
8709:
8683:
8654:
8468:
8442:
8400:
8159:
8113:
8067:
8035:
7986:
7855:
7815:
7727:
7639:
7599:
7525:
7496:
7467:
7410:
7361:
6802:
6740:
6584:
6418:
6389:
6336:
6255:
6224:
6200:
6157:
6095:
6033:
6003:
5958:
5903:
5809:
5780:
5730:
5691:
5633:
5591:
5542:
5314:
5272:
5245:
5182:
5151:
5099:
4987:
4945:
4814:
4741:
4667:
4591:
4150:
3735:
3668:
3626:
3594:
3566:
3538:
3508:
3477:
3449:
3421:
3391:
3327:
3279:
3245:
3215:
3189:
3095:
2983:
2882:
2862:
2697:
2660:
2572:
2492:
2460:
2440:
2213:
2118:
2088:
1989:
1824:
1768:
1582:
1494:
1456:
1423:
1382:
1353:
1328:
1282:
1241:
1074:
930:
886:
762:
735:
708:
673:
633:
593:
553:
526:
499:
472:
382:
120:
60:
26643:
Ruby gem for calculating Kullback–Leibler divergence
25291:
24799:{\displaystyle D_{\mathrm {KL} }(P\parallel Q)>2}
22319:
is in fact a δ function representing certainty that
16049:
12575:
10890:. It then follows that there must exist some values
5724:), but it cannot be thought of as a distance, since
3175:
be the distributions shown in the table and figure.
3025:
in words. Often it is referred to as the divergence
466:
as "the mean information for discrimination between
107:
is different from a second probability distribution
25412:
Mathematics Stack Exchange (math.stackexchange.com)
25394:
Statistics Stack Exchange (stats.stackexchange.com)
25208:
24618:
Relationship to other probability-distance measures
21647:, and subsequently learnt the true distribution of
4904:, we can minimize the KL divergence and compute an
547:", where one is comparing two probability measures
26525:
25970:Numerical Recipes: The Art of Scientific Computing
25917:
25638:Extreme Value Methods with Applications to Finance
25463:
25246:
25225:
25150:
24923:
24897:
24798:
24745:
24661:
24575:
24545:
24484:
24374:
24345:
24319:
24280:
24110:
24041:
23885:
23836:
23692:
23575:
23536:
23509:
23482:
23455:
23405:
23370:
23332:
23273:
23213:{\displaystyle W=\Delta G=NkT_{o}\Theta (V/V_{o})}
23212:
23137:
23110:
23079:
23016:
22974:
22948:
22907:
22825:
22794:
22774:
22698:
22638:
22599:
22562:
22520:
22459:
22311:
22282:
22253:
22197:
22117:
22082:
22047:
22018:
21985:
21953:
21727:
21672:
21631:
21593:
21544:
21461:
21302:
21269:
21242:
21212:
21185:
21154:
21127:
21100:
21073:
21046:
21015:
20896:
20828:
20630:
20514:
20383:
20225:
20190:
20142:
20077:
20034:
20001:
19811:
19770:
19735:
19686:
19543:
19424:
19383:
19344:
19318:
19240:
19199:
19008:
18925:
18888:
18844:
18809:
18746:
18288:
18197:
18046:
18017:
17973:
17944:
17885:
17686:
17639:
17598:
17557:
17272:
17246:
17213:
17183:
17151:
17026:Relation to other quantities of information theory
16990:
16848:
16625:
16235:
16085:
16030:
15991:
15957:
15778:
15624:
15496:
15462:
15429:
15403:
15308:
15288:
15223:
15116:
15089:
15069:
15027:
14992:
14944:
14865:
14816:
14742:
14675:
14633:
14559:bits would be needed to identify one element of a
14551:
14518:
14479:
14388:
14350:
14312:
14267:
14122:
14071:
13870:
13549:
13277:
13221:
13170:
13115:
13060:
13013:
12960:
12670:If the two distributions have the same dimension,
12662:
12616:
12551:
12518:
12460:
12284:
12142:
12034:
11962:
11931:
11884:{\displaystyle {\frac {Q(d\theta )}{P(d\theta )}}}
11883:
11832:almost surely with respect to probability measure
11821:
11703:
11575:
11517:
11491:
11465:
11423:
11370:
11346:
11326:
11277:
11251:
11171:
11151:
11116:
11096:
11067:
11041:
11021:
10995:
10960:
10934:
10908:
10882:
10862:
10833:
10805:
10270:
10242:
9974:
9948:
9928:
9908:
9866:
9223:
8853:
8833:
8776:
8689:
8669:
8640:
8454:
8428:
8385:
8145:
8099:
8053:
8014:
7970:
7841:
7801:
7713:
7625:
7565:
7511:
7482:
7445:
7396:
7347:
6784:
6726:
6570:
6404:
6364:
6278:
6241:
6206:
6187:
6143:
6081:
6015:
5986:
5941:{\displaystyle D_{\text{KL}}(P\parallel Q)\geq 0,}
5940:
5837:
5795:
5766:
5712:
5661:
5608:
5565:
5525:
5285:
5258:
5227:
5164:
5112:
5015:
4973:
4842:
4769:
4695:
4633:
4548:
4133:
3696:
3654:
3607:
3579:
3551:
3523:
3490:
3462:
3434:
3406:
3347:
3313:
3254:
3227:
3201:
3129:
3017:
2926:
2868:
2845:
2672:
2638:
2558:
2470:
2446:
2411:
2184:
2104:
2054:
1972:
1787:
1713:
1550:
1477:
1439:
1409:
1368:
1337:
1309:
1268:
1201:
1054:
905:
825:
748:
721:
694:
659:
619:
579:
539:
512:
485:
410:
241:
88:
26063:"Information theory and statistical mechanics II"
25805:"Derivations for Linear Algebra and Optimization"
25760:Communications in Statistics - Theory and Methods
22610:
22486:that one is attempting to optimise by minimising
22371:is equivalent to minimizing the cross-entropy of
20089:can be updated further, to give a new best guess
18860:were coded according to the uniform distribution
18769:the relative entropy of the product distribution
18289:{\displaystyle \log(N)-D_{\text{KL}}(p(x)||m(x))}
17989:were coded according to the uniform distribution
14595:distance, in some cases satisfying a generalized
4935:distance, in some cases satisfying a generalized
3704:are calculated as follows. This example uses the
2830:
2626:
2590:
2546:
2510:
2396:
2365:
2342:
2302:
2279:
2172:
2136:
2039:
2013:
1957:
1926:
1903:
1701:
1573:, relative entropy is defined to be the integral
324:in information systems, randomness in continuous
26678:
26426:
24806:, since the total variation distance is at most
24295:from discovering which probability distribution
22482:that is some fixed prior reference measure, and
19252:for constructing the encoding scheme instead of
18077:
17254:is available to the receiver, not the fact that
14761:
12930:
12915:
12035:{\displaystyle {\frac {\exp h(\theta )}{E_{P}}}}
8834:{\displaystyle D_{\text{KL}}(P\parallel Q)=f(1)}
6249:thus sets a minimum value for the cross-entropy
3362:Two distributions to illustrate relative entropy
1496:
26216:
26158:"Thermal roots of correlation-based complexity"
25730:
25461:
25270:
25156:
24935:Other notable measures of distance include the
23973:
21735:, and the earlier prior distribution would be:
21594:{\displaystyle D_{\text{KL}}(f\parallel f_{0})}
21505:Principle of minimum discrimination information
20907:
16093:satisfies the following regularity conditions:
15999:vary (and dropping the subindex 0) the Hessian
8436:may be Taylor expanded about its minimum (i.e.
3130:{\displaystyle \ D_{\text{KL}}(P\parallel Q)\ }
3018:{\displaystyle \ D_{\text{KL}}(P\parallel Q)\ }
2654:are both absolutely continuous with respect to
463:
26120:
26013:"Information theory and statistical mechanics"
25894:
25892:
25820:
25733:Communications on Pure and Applied Mathematics
25710:See the section "differential entropy – 4" in
24669:. This is connected to the divergence through
21472:Either of the two quantities can be used as a
5774:is not zero. This can be fixed by subtracting
5040:
1551:{\displaystyle \lim _{x\to 0^{+}}x\log(x)=0~.}
25000:entropy serves as theoretical background for
24989:entropy serves as theoretical background for
23707:
21604:For example, if one had a prior distribution
21008:
20948:
20924:
20821:
20762:
20742:
20657:
20623:
20551:
19893:
19841:
19819:would have added an expected number of bits:
19606:
19575:
17067:, the self-information can be represented as
14583:. While metrics are symmetric and generalize
14493:times narrower uniform distribution contains
12981:since all terms apart from the last are base-
5133:Illustration of the relative entropy for two
4923:. While metrics are symmetric and generalize
833:, which had already been defined and used by
332:; and practical, such as applied statistics,
26510:: CS1 maint: multiple names: authors list (
25973:(3rd ed.). Cambridge University Press.
25704:
22820:
22789:
22359:for short. Minimising relative entropy from
18066:), which defines the continuous entropy as
17548:
17496:
17470:
17418:
17138:
17125:
17056:is defined as the negative logarithm of the
14968:Relative entropy is directly related to the
14963:
14737:
14705:
11907:For a short proof assuming integrability of
11695:
11645:
8153:are two pairs of probability measures then
7593:in much the same way as Shannon entropy. If
4931:, divergences are asymmetric and generalize
3308:
3290:
26625:(Second ed.). Oxford University Press.
26349:
26278:
25965:"Section 14.7.2. Kullback–Leibler Distance"
25898:
25889:
25680:
25303:
24622:There are many other important measures of
23886:{\displaystyle D_{\text{KL}}(P\parallel Q)}
22563:{\displaystyle D_{\text{KL}}(p\parallel m)}
22521:{\displaystyle D_{\text{KL}}(p\parallel m)}
19248:) for encoding the events because of using
17001:
14684:
14676:{\displaystyle D_{\text{KL}}(Q\parallel P)}
14634:{\displaystyle D_{\text{KL}}(P\parallel Q)}
13186:solutions to the triangular linear systems
12526:, for which equality occurs if and only if
9909:{\displaystyle D_{\text{KL}}(P\parallel Q)}
8429:{\displaystyle D_{\text{KL}}(P\parallel Q)}
8015:{\displaystyle D_{\text{KL}}(P\parallel Q)}
7802:{\displaystyle Q(dx,dy)=Q_{1}(dx)Q_{2}(dy)}
7714:{\displaystyle P(dx,dy)=P_{1}(dx)P_{2}(dy)}
6365:{\displaystyle D_{\text{KL}}(P\parallel Q)}
5987:{\displaystyle D_{\text{KL}}(P\parallel Q)}
5853:), the asymmetric form is more useful. See
5850:
5838:{\displaystyle D_{\text{KL}}(P\parallel Q)}
5662:{\displaystyle D_{\text{KL}}(P\parallel Q)}
5137:. The typical asymmetry is clearly visible.
5027:form of relative entropy, specifically its
5016:{\displaystyle D_{\text{KL}}(Q\parallel P)}
4974:{\displaystyle D_{\text{KL}}(P\parallel Q)}
4843:{\displaystyle D_{\text{KL}}(P\parallel Q)}
4770:{\displaystyle D_{\text{KL}}(P\parallel Q)}
4696:{\displaystyle D_{\text{KL}}(P\parallel Q)}
3697:{\displaystyle D_{\text{KL}}(Q\parallel P)}
3655:{\displaystyle D_{\text{KL}}(P\parallel Q)}
2977:Various conventions exist for referring to
411:{\displaystyle D_{\text{KL}}(P\parallel Q)}
351:
89:{\displaystyle D_{\text{KL}}(P\parallel Q)}
26638:Information Theoretical Estimators Toolbox
25355:
24591:. The Jensen–Shannon divergence, like all
24067:This quantity has sometimes been used for
23976:also considered the symmetrized function:
23333:{\displaystyle \Theta (x)=x-1-\ln x\geq 0}
20525:and so the combined information gain does
19067:. The cross entropy for two distributions
13171:{\displaystyle \Sigma _{1}=L_{1}L_{1}^{T}}
13116:{\displaystyle \Sigma _{0}=L_{0}L_{0}^{T}}
12989:. Dividing the entire expression above by
11576:{\displaystyle (\Theta ,{\mathcal {F}},P)}
11466:{\displaystyle (\Theta ,{\mathcal {F}},Q)}
11424:{\displaystyle (\Theta ,{\mathcal {F}},P)}
11300:Donsker and Varadhan's variational formula
5857:for more on the geometric interpretation.
26571:Information Geometry and Its Applications
26403:
26393:
26375:
26332:
26322:
26304:
26234:
26173:
26155:
25947:
25937:
25899:Cover, Thomas M.; Thomas, Joy A. (1991),
25771:
25695:
25567:
25557:
25539:
25466:Model Selection and Multi-Model Inference
25183:
25141:
25119:
25117:
24499:is the average of the two distributions,
23968:
21166:, is the expected weight of evidence for
19210:For explicit derivation of this, see the
18896:rather than the conditional distribution
18185:
16833:
16731:
16600:
16456:
16339:
15865:
15740:
15726:
15712:
15188:
14750:is a sequence of distributions such that
14683:, and while this can be symmetrized (see
14123:{\displaystyle k=\sigma _{1}/\sigma _{0}}
13560:A special case, and a common quantity in
11529:is absolutely continuous with respect to
11294:Duality formula for variational inference
7556:
7334:
7203:
7156:
7084:
6926:
6796:, the relative entropy may be rewritten:
6646:
6615:
6561:
6480:
6449:
5720:is itself such a measurement (formally a
2962:2 if information is measured in units of
2196:is absolutely continuous with respect to
26620:
26595:
26533:International Journal of Computer Vision
26479:Introduction to nonparametric estimation
26476:B.), Tsybakov, A. B. (Alexandre (2010).
25920:"Bayesian experimental design: a review"
25590:
25521:
25462:Burnham, K. P.; Anderson, D. R. (2002).
25449:
25428:Pattern recognition and machine learning
25349:
25309:
25297:
25285:
25219:
22614:
21964:i.e. the sum of the relative entropy of
21023:can also be interpreted as the expected
19736:{\displaystyle \mathrm {H} (p(x\mid I))}
17006:Another information-theoretic metric is
14866:{\displaystyle P_{n}{\xrightarrow {D}}Q}
14277:
13881:For two univariate normal distributions
13278:{\displaystyle L_{1}y=\mu _{1}-\mu _{0}}
12663:{\displaystyle \Sigma _{0},\Sigma _{1}.}
11891:denotes the Radon-Nikodym derivative of
8861:may be derived and evaluated as follows
5673:, a measure of how far the distribution
5128:
5084:
3357:
3314:{\displaystyle {\mathcal {X}}=\{0,1,2\}}
850:
838:
702:, and defined the "'divergence' between
27:Mathematical statistics distance measure
26355:
26284:
26249:
25870:
25866:
25864:
25686:
25123:
24307:, if they currently have probabilities
23909:Relationship between models and reality
22131:conditional Kullback–Leibler divergence
21521:): given new facts, a new distribution
20016:, that has been learned by discovering
17010:, which is roughly a symmetrization of
15000:. Consider then two close by values of
14743:{\displaystyle \{P_{1},P_{2},\ldots \}}
14489:Intuitively, the information gain to a
12975:in the last term must be taken to base
12471:where the last inequality follows from
5882:in terms of Kullback–Leibler divergence
5228:{\displaystyle q(x_{i})=2^{-\ell _{i}}}
4875:
454:The relative entropy was introduced by
440:expectation–maximization algorithm (EM)
356:Consider two probability distributions
14:
26679:
26217:Burnham, K.P.; Anderson, D.R. (2001).
26060:
26010:
25993:
25661:
25361:
25114:
24931:, various inequalities may be deduced.
24546:{\displaystyle M={\frac {1}{2}}(P+Q).}
24060:in 1948; it is accordingly called the
22090:from the new conditional distribution
20897:{\displaystyle p(y_{2}\mid y_{1},x,I)}
20143:{\displaystyle p(x\mid y_{1},y_{2},I)}
19063:, rather than the "true" distribution
18299:
17705:is not already known to the receiver.
14566:
13643:
12842:
6785:{\displaystyle |{\tfrac {dy}{dx}}(x)|}
6314:, rather than the "true" distribution
316:(which applies to squared distances).
26565:
26519:
26250:Burnham, Kenneth P. (December 2010).
26102:
25918:Chaloner, K.; Verdinelli, I. (1995).
25630:
25617:
25611:
25494:
25255:
25240:
25076:Information theory and measure theory
22646:times the Kullback–Leibler divergence
22254:{\displaystyle q(x\mid a)=p(x\mid a)}
17283:
17225:if only the probability distribution
11334:be a set endowed with an appropriate
5669:can be thought of geometrically as a
2942:or a convenient variant thereof like
1321:). Otherwise, it is often defined as
667:(respectively). They denoted this by
25861:
25802:
24974:
21223:The expected weight of evidence for
19259:
11583:. Then the following equality holds
6294:required when using a code based on
5854:
826:{\displaystyle J(1,2)=I(1:2)+I(2:1)}
253:
26475:
25757:
23936:how much the model has yet to learn
23274:{\displaystyle V_{o}=NkT_{o}/P_{o}}
22326:MDI can be seen as an extension of
18060:limiting density of discrete points
17033:
15463:{\displaystyle \theta =\theta _{0}}
10870:with measure strictly greater than
7633:are independent distributions, and
4719:rather than the code optimized for
4634:{\displaystyle \log P(Y)-\log Q(Y)}
2078:almost everywhere defined function
1788:{\displaystyle \ {\mathcal {X}}\ ,}
906:{\displaystyle \ {\mathcal {X}}\ ,}
111:. Mathematically, it is defined as
24:
26143:10.1038/scientificamerican0971-179
25509:10.1002/j.1538-7305.1956.tb03809.x
25470:(2nd ed.). Springer. p.
25424:
25066:Information gain in decision trees
24867:
24864:
24769:
24766:
23654:
23613:
23593:
23567:
23388:
23288:
23183:
23158:
22578:
22413:
22390:
22353:Principle of Minimum Cross-Entropy
21816:
21513:Minimum Discrimination Information
20904:, the two sides will average out.
19705:
19569:
19319:{\displaystyle p(x)\to p(x\mid I)}
19225:
19153:
19113:
19090:
18993:
18654:
18604:
18588:
18323:
18087:
18025:rather than the true distribution
17764:
17752:
17732:
17708:
17649:marginal probability distributions
17484:
17406:
17306:
17078:
16951:
16871:
16800:
16771:
16762:
16705:
16682:
16674:
16616:
16565:
16507:
16481:
16466:
16439:
16381:
16367:
16349:
16322:
16271:
16263:
16217:
16191:
16168:
16142:
16126:
16106:
15866:
15852:
15842:
15727:
15713:
15524:
15520:
15497:{\displaystyle \Delta \theta _{j}}
15481:
15369:
15361:
15241:
15175:
14771:
14432:
14422:
14389:{\displaystyle C\leq A<B\leq D}
14261:
14258:
14255:
14252:
14166:
14156:
13930:
13920:
13717:
13591:
13330:
13313:
13131:
13076:
12934:
12919:
12849:
12783:
12765:
12719:
12702:
12648:
12635:
11559:
11551:
11449:
11441:
11407:
11399:
11363:
11321:
11272:
11152:{\displaystyle P\geq 2Q+\epsilon }
10996:{\displaystyle P\geq 2Q+\epsilon }
10955:
10754:
10684:
10639:
10508:
10463:
10355:
10305:
10156:
10111:
10041:
9780:
9573:
9420:
9279:
9067:
8909:
8558:
8513:
6257:
6226:
6144:{\displaystyle Q(dx)=q(x)\mu (dx)}
6082:{\displaystyle P(dx)=p(x)\mu (dx)}
5593:
5544:
5506:
5483:
5426:
5364:
5073:. For probabilities over a finite
4896:. In order to find a distribution
4859:posterior probability distribution
4559:
4200:
3785:
3282:
2823:
2739:
2680:), then the relative entropy from
2619:
2583:
2539:
2503:
2463:
2389:
2358:
2335:
2295:
2272:
2255:
2165:
2129:
2105:{\displaystyle \ {\mathcal {X}}\ }
2094:
2032:
2006:
1950:
1919:
1896:
1866:
1774:
1694:
1624:
1619:
1440:{\displaystyle \ {\mathcal {X}}\ }
1429:
1360:
1332:
1119:
972:
892:
867:discrete probability distributions
376:. The Kullback–Leibler divergence
162:
25:
26713:
26631:
26598:Information Theory and Statistics
25664:Concepts in statistical mechanics
25171:Annals of Mathematical Statistics
23897:can also be used as a measure of
23693:{\displaystyle \Delta I=Nk\left.}
22919:is a constrained multiplicity or
22833:so that, for instance, there are
22600:{\displaystyle \mathrm {H} (p,m)}
22474:. However, this is just as often
21220:to be expected from each sample.
19055:) measures the average number of
18817:from the true joint distribution
17052:of a signal, random variable, or
16050:Fisher information metric theorem
14396:). Then the information gain is:
12617:{\displaystyle \mu _{0},\mu _{1}}
12582:multivariate normal distributions
12576:Multivariate normal distributions
7519:are also dimensioned, since e.g.
6279:{\displaystyle \mathrm {H} (P,Q)}
5566:{\displaystyle \mathrm {H} (P,Q)}
5093:Financial risks are connected to
3089:. Another common way to refer to
1807:, then the relative entropy from
660:{\displaystyle \mu _{1},\mu _{2}}
580:{\displaystyle \mu _{1},\mu _{2}}
100:: a measure of how one reference
25871:Buchner, Johannes (2022-04-29).
25166:"On information and sufficiency"
24098:An alternative is given via the
23413:defined as the average value of
22908:{\displaystyle A\equiv -k\ln(Z)}
22332:Principle of Insufficient Reason
20191:{\displaystyle p(x\mid y_{1},I)}
19391:to a new posterior distribution
18944:
17060:of the given outcome occurring.
16038:defines a (possibly degenerate)
15070:{\displaystyle Q=P(\theta _{0})}
13736:
13728:
5041:§ Fisher information metric
4569:In the field of statistics, the
3463:{\displaystyle {\frac {12}{25}}}
3162:
280:when the actual distribution is
26469:
26420:
26243:
26210:
26149:
26114:
26096:
26054:
26004:
25987:
25956:
25911:
25814:
25796:
25751:
25724:
25666:. New York: Gordon and Breach.
25655:
25584:
25515:
25488:
25455:
25418:
25400:
25382:
23576:{\displaystyle W=T_{o}\Delta I}
23406:{\displaystyle \Delta I\geq 0,}
19778:instead of a new code based on
19241:{\displaystyle \mathrm {H} (p)}
19211:
19009:{\displaystyle \mathrm {H} (p)}
17568:is the relative entropy of the
17014:. It is a metric on the set of
16031:{\displaystyle g_{jk}(\theta )}
11378:, and two probability measures
8648:which converges if and only if
6242:{\displaystyle \mathrm {H} (P)}
5609:{\displaystyle \mathrm {H} (P)}
3491:{\displaystyle {\frac {4}{25}}}
3435:{\displaystyle {\frac {9}{25}}}
2938:for discrete distributions, or
26103:Gibbs, Josiah Willard (1871).
25901:Elements of Information Theory
25329:10.1080/00031305.1987.10475510
25046:Deviance information criterion
25031:Bayesian information criterion
25011:the source (minimum size of a
24885:
24873:
24836:
24824:
24787:
24775:
24735:
24723:
24695:
24683:
24656:
24644:
24537:
24525:
24479:
24467:
24441:
24429:
24272:
24266:
24254:
24236:
24223:
24211:
24205:
24199:
24187:
24169:
24150:
24138:
24053:
24036:
24024:
24008:
23996:
23880:
23868:
23828:
23825:
23822:
23816:
23804:
23798:
23789:
23783:
23771:
23759:
23450:
23429:
23297:
23291:
23207:
23186:
22902:
22896:
22871:availability in entropy units
22693:
22679:
22611:Relationship to available work
22594:
22582:
22557:
22545:
22515:
22503:
22451:
22439:
22423:
22417:
22406:
22394:
22306:
22300:
22277:
22271:
22248:
22236:
22227:
22215:
22192:
22189:
22177:
22168:
22156:
22150:
22112:
22100:
22077:
22065:
22042:
22036:
22013:
22007:
21997:from the updated distribution
21980:
21974:
21945:
21942:
21936:
21927:
21921:
21915:
21894:
21891:
21879:
21870:
21858:
21852:
21829:
21823:
21809:
21806:
21794:
21785:
21779:
21773:
21761:
21755:
21728:{\displaystyle q(x\mid a)u(a)}
21722:
21716:
21710:
21698:
21667:
21661:
21626:
21614:
21588:
21569:
21453:
21450:
21438:
21429:
21417:
21411:
21386:
21383:
21364:
21355:
21336:
21330:
21297:
21291:
21003:
20984:
20975:
20956:
20891:
20853:
20816:
20804:
20795:
20770:
20737:
20712:
20703:
20665:
20618:
20606:
20597:
20559:
20529:obey the triangle inequality:
20501:
20489:
20481:
20456:
20437:
20412:
20371:
20359:
20351:
20313:
20294:
20256:
20220:
20208:
20185:
20160:
20137:
20099:
19989:
19977:
19969:
19951:
19932:
19914:
19888:
19876:
19867:
19849:
19806:
19788:
19765:
19753:
19730:
19727:
19715:
19709:
19678:
19660:
19648:
19630:
19601:
19583:
19535:
19523:
19515:
19503:
19497:
19479:
19467:
19449:
19419:
19401:
19378:
19366:
19313:
19301:
19295:
19292:
19286:
19235:
19229:
19191:
19179:
19163:
19157:
19146:
19143:
19137:
19125:
19106:
19094:
19003:
18997:
18920:
18913:
18906:
18883:
18877:
18839:
18827:
18804:
18798:
18792:
18786:
18765:equally likely possibilities,
18727:
18721:
18647:
18641:
18622:
18610:
18598:
18592:
18574:
18571:
18565:
18549:
18543:
18537:
18521:
18518:
18512:
18506:
18500:
18491:
18479:
18473:
18457:
18451:
18432:
18429:
18423:
18417:
18411:
18395:
18383:
18377:
18361:
18355:
18339:
18327:
18283:
18280:
18274:
18267:
18262:
18258:
18252:
18246:
18230:
18224:
18179:
18173:
18165:
18159:
18144:
18138:
18126:
18120:
18108:
18102:
18084:
18041:
18035:
18012:
18006:
17968:
17962:
17939:
17933:
17904:equally likely possibilities,
17871:
17865:
17849:
17843:
17812:
17806:
17782:
17776:
17742:
17736:
17681:
17669:
17634:
17628:
17622:
17616:
17593:
17581:
17570:joint probability distribution
17545:
17542:
17536:
17527:
17515:
17509:
17467:
17464:
17458:
17449:
17437:
17431:
17392:
17389:
17383:
17377:
17371:
17362:
17350:
17344:
17324:
17312:
17241:
17235:
17178:
17172:
17091:
17085:
16982:
16918:
16915:
16903:
16894:
16882:
16876:
16795:
16783:
16700:
16688:
16597:
16591:
16585:
16573:
16543:
16537:
16502:
16496:
16453:
16447:
16417:
16411:
16336:
16330:
16300:
16294:
16212:
16206:
16163:
16157:
16121:
16115:
16078:
16066:
16025:
16019:
15952:
15949:
15936:
15927:
15921:
15915:
15829:
15816:
15767:
15754:
15696:
15693:
15680:
15671:
15665:
15659:
15610:
15607:
15594:
15585:
15579:
15573:
15474:order in the small parameters
15398:
15385:
15277:
15257:
15212:
15199:
15169:
15156:
15147:
15141:
15064:
15051:
15022:
15016:
14913:
14805:
14786:
14768:
14670:
14658:
14628:
14616:
14345:
14333:
14307:
14295:
14248:
14242:
14220:
14198:
14024:
13997:
13524:
13510:
13496:
13482:
13436:
13427:
13408:
13391:
13008:
13002:
12507:
12488:
12452:
12440:
12418:
12399:
12383:
12371:
12349:
12337:
12321:
12315:
12276:
12264:
12242:
12236:
12217:
12205:
12189:
12170:
12137:
12128:
12119:
12107:
12092:
12086:
12068:
12059:
12026:
12014:
11999:
11993:
11926:
11920:
11875:
11866:
11858:
11849:
11810:
11798:
11783:
11777:
11756:
11747:
11739:
11730:
11692:
11680:
11664:
11658:
11621:
11609:
11570:
11548:
11460:
11438:
11418:
11396:
11371:{\displaystyle {\mathcal {F}}}
11269:
11207:
11195:
11124:component of the series where
11075:component of the series where
10909:{\displaystyle \epsilon >0}
10780:
10768:
10720:
10714:
10705:
10699:
10665:
10653:
10590:
10584:
10576:
10570:
10544:
10538:
10529:
10523:
10489:
10477:
10417:
10410:
10396:
10392:
10386:
10377:
10371:
10365:
10336:
10324:
10218:
10211:
10197:
10193:
10187:
10178:
10172:
10166:
10137:
10125:
10071:
10065:
10060:
10054:
10015:
10003:
9903:
9891:
9842:
9835:
9821:
9817:
9811:
9802:
9796:
9790:
9761:
9749:
9740:
9730:
9720:
9714:
9709:
9703:
9672:
9666:
9654:
9648:
9642:
9630:
9614:
9610:
9604:
9595:
9589:
9583:
9554:
9542:
9533:
9523:
9513:
9507:
9502:
9496:
9481:
9475:
9461:
9457:
9451:
9442:
9436:
9430:
9397:
9391:
9373:
9367:
9355:
9349:
9343:
9331:
9320:
9316:
9310:
9301:
9295:
9289:
9256:
9250:
9204:
9198:
9176:
9170:
9162:
9156:
9144:
9138:
9132:
9120:
9104:
9101:
9095:
9086:
9080:
9074:
9023:
9017:
9009:
9003:
8991:
8985:
8979:
8967:
8946:
8943:
8937:
8928:
8922:
8916:
8886:
8880:
8828:
8822:
8813:
8801:
8771:
8750:
8738:
8735:
8719:
8713:
8620:
8613:
8599:
8595:
8589:
8580:
8574:
8568:
8539:
8527:
8491:
8479:
8423:
8411:
8360:
8334:
8321:
8309:
8303:
8277:
8258:
8245:
8233:
8201:
8189:
8170:
8140:
8114:
8094:
8068:
8048:
8036:
8009:
7997:
7962:
7936:
7920:
7894:
7878:
7866:
7809:for independent distributions
7796:
7787:
7774:
7765:
7749:
7731:
7708:
7699:
7686:
7677:
7661:
7643:
7566:{\displaystyle P(dx)=p(x)\,dx}
7553:
7547:
7538:
7529:
7506:
7500:
7477:
7471:
7446:{\displaystyle y_{b}=y(x_{b})}
7440:
7427:
7397:{\displaystyle y_{a}=y(x_{a})}
7391:
7378:
7324:
7318:
7312:
7301:
7295:
7289:
7267:
7261:
7255:
7192:
7188:
7182:
7158:
7153:
7150:
7144:
7138:
7132:
7120:
7116:
7110:
7086:
7081:
7078:
7072:
7066:
7060:
7037:
7033:
7027:
7003:
6999:
6996:
6990:
6984:
6978:
6916:
6910:
6902:
6896:
6877:
6871:
6829:
6817:
6778:
6774:
6768:
6742:
6714:
6710:
6704:
6678:
6674:
6668:
6662:
6643:
6637:
6631:
6612:
6606:
6597:
6588:
6557:
6553:
6547:
6521:
6517:
6514:
6508:
6502:
6496:
6477:
6471:
6465:
6446:
6440:
6431:
6422:
6399:
6393:
6359:
6347:
6273:
6261:
6236:
6230:
6182:
6176:
6167:
6161:
6138:
6129:
6123:
6117:
6108:
6099:
6076:
6067:
6061:
6055:
6046:
6037:
5981:
5969:
5926:
5914:
5832:
5820:
5790:
5784:
5761:
5755:
5746:
5734:
5707:
5695:
5656:
5644:
5603:
5597:
5560:
5548:
5516:
5510:
5499:
5487:
5466:
5460:
5442:
5436:
5404:
5398:
5380:
5374:
5341:
5329:
5266:is the length of the code for
5199:
5186:
5172:out of a set of possibilities
5047:(geometrically interpreted as
5010:
4998:
4968:
4956:
4852:prior probability distribution
4837:
4825:
4764:
4752:
4690:
4678:
4628:
4622:
4607:
4601:
4528:
4522:
4507:
4501:
4486:
4480:
4255:
4249:
4241:
4235:
4216:
4210:
4177:
4165:
4110:
4104:
4089:
4083:
4068:
4062:
3840:
3834:
3826:
3820:
3801:
3795:
3762:
3750:
3691:
3679:
3649:
3637:
3608:{\displaystyle {\frac {1}{3}}}
3580:{\displaystyle {\frac {1}{3}}}
3552:{\displaystyle {\frac {1}{3}}}
3518:
3512:
3401:
3395:
3137:is as the relative entropy of
3121:
3109:
3009:
2997:
2970:if information is measured in
2946:or the uniform measure on the
2834:
2819:
2803:
2797:
2786:
2780:
2755:
2749:
2720:
2708:
2630:
2615:
2609:
2603:
2594:
2579:
2550:
2535:
2529:
2523:
2514:
2499:
2471:{\displaystyle {\mathcal {X}}}
2434:. Continuing in this case, if
2400:
2385:
2369:
2354:
2346:
2331:
2306:
2291:
2283:
2268:
2236:
2224:
2176:
2161:
2155:
2149:
2140:
2125:
2043:
2028:
2017:
2002:
1961:
1946:
1930:
1915:
1907:
1892:
1847:
1835:
1680:
1674:
1666:
1660:
1638:
1632:
1605:
1593:
1533:
1527:
1503:
1469:
1463:
1410:{\displaystyle \ Q(x)\neq 0\ }
1395:
1389:
1295:
1289:
1254:
1248:
1183:
1177:
1166:
1160:
1135:
1129:
1097:
1085:
1036:
1030:
1019:
1013:
988:
982:
953:
941:
820:
808:
799:
787:
778:
766:
756:" as the symmetrized quantity
689:
677:
426:rather than one optimized for
405:
393:
312:), it satisfies a generalized
226:
220:
209:
203:
178:
172:
143:
131:
83:
71:
13:
1:
26668:2009. One-hour video lecture.
26430:Séminaire de Probabilités XII
25782:10.1080/03610926.2021.1921214
25108:
24576:{\displaystyle D_{\text{JS}}}
23974:Kullback & Leibler (1951)
23456:{\displaystyle k\ln(p/p_{o})}
22956:) is also minimized. Thus if
18953:, we can encode them (with a
17952:, from the true distribution
16086:{\displaystyle p_{(x,\rho )}}
15126:Einstein summation convention
14685:§ Symmetrised divergence
8146:{\displaystyle (P_{2},Q_{2})}
8100:{\displaystyle (P_{1},Q_{1})}
5887:
5860:Relative entropy relates to "
5851:§ Symmetrised divergence
5124:
5113:{\displaystyle D_{\text{KL}}}
5057:maximum likelihood estimation
4804:Expressed in the language of
4564:
860:
464:Kullback & Leibler (1951)
26609:in 1968; reprinted in 1978:
25124:Csiszar, I (February 1975).
25026:Akaike information criterion
24662:{\displaystyle \delta (p,q)}
24375:{\displaystyle \lambda =0.5}
23947:Akaike information criterion
23017:{\displaystyle F\equiv U-TS}
22336:Principle of Maximum Entropy
22127:conditional relative entropy
20914:Bayesian experimental design
20908:Bayesian experimental design
20638:may be <, = or > than
20045:If a further piece of data,
19812:{\displaystyle p(x\mid y,I)}
19554:This distribution has a new
19425:{\displaystyle p(x\mid y,I)}
19079:is thus defined as follows.
18810:{\displaystyle P_{U}(X)P(Y)}
17195:representing certainty that
15028:{\displaystyle P=P(\theta )}
14571:While relative entropy is a
13222:{\displaystyle L_{1}M=L_{0}}
12563:For alternative proof using
11537:be a real-valued integrable
11278:{\displaystyle n\to \infty }
11159:. The absolute value of the
10961:{\displaystyle U<\infty }
9916:via the Taylor expansion of
5767:{\displaystyle H(P,P)=:H(P)}
5062:The relative entropy is the
4726:
1369:{\displaystyle \ +\infty \ }
449:
292:), and does not satisfy the
7:
25362:MacKay, David J.C. (2003).
25271:Kullback & Leibler 1951
25018:
24962:Kolmogorov–Smirnov distance
23849:quantum information science
22699:{\displaystyle s=k\ln(1/p)}
21993:the prior distribution for
20839:All one can say is that on
20078:{\displaystyle Y_{2}=y_{2}}
15992:{\displaystyle \theta _{0}}
15639:one has up to second order
15117:{\displaystyle \theta _{0}}
13061:{\displaystyle L_{0},L_{1}}
12570:
11307:
7842:{\displaystyle Q_{1},Q_{2}}
7626:{\displaystyle P_{1},P_{2}}
7461:is a dimensioned variable,
5893:Relative entropy is always
5874:characterization of entropy
5141:In information theory, the
4715:using a code optimized for
620:{\displaystyle H_{1},H_{2}}
444:evidence lower bound (ELBO)
422:using a code optimized for
10:
26718:
26596:Kullback, Solomon (1959),
24978:
24346:{\displaystyle 1-\lambda }
24089:Population Stability Index
23918:
23912:
23893:over all separable states
23708:Quantum information theory
22323:has one particular value.
22261:over the whole support of
22118:{\displaystyle q(x\mid a)}
22083:{\displaystyle p(x\mid a)}
21025:discrimination information
20925:Discrimination information
20226:{\displaystyle p(x\mid I)}
19771:{\displaystyle p(x\mid I)}
19384:{\displaystyle p(x\mid I)}
18977:), which will be equal to
17037:
14552:{\displaystyle \log _{2}k}
14519:{\displaystyle \log _{2}k}
12559:. The conclusion follows.
11285:, so the series diverges.
10935:{\displaystyle \rho >0}
8841:. The first derivative of
2204:), this can be written as
1571:continuous random variable
1417:everywhere, provided that
1310:{\displaystyle \ P(x)=0\ }
1269:{\displaystyle \ Q(x)=0\ }
1212:In other words, it is the
913:the relative entropy from
843:discrimination information
430:. Note that the roles of
26621:Jeffreys, Harold (1948).
26579:10.1007/978-4-431-55978-8
25378:– via Google Books.
25316:The American Statistician
25081:Jensen–Shannon divergence
24601:Fisher information metric
24384:Jensen–Shannon divergence
23583:, where relative entropy
23080:{\displaystyle G=U+PV-TS}
22949:{\displaystyle T\times A}
19045:probability distributions
18955:lossless data compression
18062:(as opposed to the usual
14970:Fisher information metric
14964:Fisher information metric
14696:probability distributions
14587:distance, satisfying the
13021:yields the divergence in
12580:Suppose that we have two
9975:{\displaystyle \alpha =1}
7591:independent distributions
6377:parameter transformations
6188:{\displaystyle p(x)=q(x)}
5677:is from the distribution
5259:{\displaystyle \ell _{i}}
5037:Fisher information metric
4927:distance, satisfying the
4785:would be used instead of
4652:
3321:), each with probability
2856:Note that such a measure
26526:Rubner, Y.; Tomasi, C.;
25839:10.1348/000711010x522227
25522:Soklakov, A. N. (2020).
25086:Quantum relative entropy
25056:Entropy power inequality
24632:total-variation distance
24320:{\displaystyle \lambda }
24111:{\displaystyle \lambda }
23729:quantum relative entropy
22982:and number of molecules
21479:On the entropy scale of
18889:{\displaystyle P_{U}(X)}
18208:which is equivalent to:
18018:{\displaystyle P_{U}(X)}
17945:{\displaystyle P_{U}(X)}
17640:{\displaystyle P(X)P(Y)}
17065:discrete random variable
17008:variation of information
17002:Variation of information
13893:the above simplifies to
12624:and with (non-singular)
11259:, which is unbounded as
11097:{\displaystyle P\leq 2Q}
10834:{\displaystyle P\leq 2Q}
10271:{\displaystyle P\leq 2Q}
8670:{\displaystyle P\leq 2Q}
7455:dimensionally consistent
5681:. Geometrically it is a
4581:based on an observation
2673:{\displaystyle \ \mu \ }
2064:Radon–Nikodym derivative
1478:{\displaystyle \ P(x)\ }
1338:{\displaystyle +\infty }
749:{\displaystyle \mu _{2}}
722:{\displaystyle \mu _{1}}
540:{\displaystyle \mu _{1}}
352:Introduction and context
290:variation of information
256:of the KL divergence of
102:probability distribution
26687:Entropy and information
26546:10.1023/A:1026543900054
26356:Nielsen, Frank (2020).
26285:Nielsen, Frank (2019).
26090:10.1103/physrev.108.171
26040:10.1103/physrev.106.620
25662:Hobson, Arthur (1971).
25425:Bishop, Christopher M.
25185:10.1214/aoms/1177729694
24951:quadratic form distance
24924:{\displaystyle \alpha }
24814:(see, also, Tsybakov):
24603:. It is similar to the
23928:distance to equilibrium
23378:by relative entropy or
22930:is fixed, free energy (
22205:) This is minimized if
15309:{\displaystyle \theta }
15090:{\displaystyle \theta }
14993:{\displaystyle \theta }
12552:{\displaystyle Q=Q^{*}}
11932:{\displaystyle \exp(h)}
11347:{\displaystyle \sigma }
11327:{\displaystyle \Theta }
11068:{\displaystyle 1-\rho }
11022:{\displaystyle Q\leq U}
10863:{\displaystyle P>2Q}
5079:statistical divergences
4868:is used to approximate
4645:is actually drawn from
1065:which is equivalent to
296:. Instead, in terms of
32:mathematical statistics
26156:Fraundorf, P. (2007).
26061:Jaynes, E. T. (1957).
26011:Jaynes, E. T. (1957).
25994:Tribus, Myron (1959).
25745:10.1002/cpa.3160360204
25143:10.1214/aop/1176996454
25103:Bhattacharyya distance
25071:Information gain ratio
25051:Entropic value at risk
24968:earth mover's distance
24941:histogram intersection
24925:
24899:
24800:
24747:
24663:
24577:
24556:We can also interpret
24547:
24486:
24376:
24347:
24321:
24282:
24112:
24043:
23969:Symmetrised divergence
23952:mean squared deviation
23887:
23838:
23694:
23577:
23538:
23511:
23484:
23457:
23407:
23372:
23334:
23275:
23214:
23139:
23112:
23081:
23018:
22976:
22950:
22909:
22827:
22796:
22776:
22700:
22647:
22640:
22601:
22564:
22522:
22461:
22313:
22284:
22255:
22199:
22119:
22084:
22049:
22020:
21987:
21955:
21729:
21674:
21633:
21632:{\displaystyle p(x,a)}
21595:
21546:
21463:
21304:
21271:
21244:
21214:
21187:
21156:
21129:
21102:
21075:
21048:
21017:
20898:
20830:
20632:
20516:
20391:may be ≤ or > than
20385:
20227:
20192:
20144:
20079:
20036:
20003:
19813:
19772:
19737:
19688:
19545:
19426:
19385:
19346:
19320:
19274:posterior distribution
19242:
19201:
19010:
18927:
18926:{\displaystyle P(X|Y)}
18890:
18846:
18845:{\displaystyle P(X,Y)}
18811:
18748:
18290:
18199:
18048:
18019:
17975:
17946:
17887:
17688:
17687:{\displaystyle P(X,Y)}
17641:
17600:
17599:{\displaystyle P(X,Y)}
17559:
17274:
17248:
17215:
17185:
17153:
16992:
16850:
16627:
16237:
16087:
16032:
15993:
15959:
15780:
15626:
15498:
15464:
15431:
15405:
15310:
15290:
15225:
15118:
15091:
15077:so that the parameter
15071:
15029:
14994:
14946:
14867:
14818:
14744:
14698:. More concretely, if
14677:
14635:
14553:
14520:
14481:
14390:
14352:
14314:
14269:
14130:, this simplifies to:
14124:
14073:
13872:
13783:
13551:
13472:
13390:
13279:
13223:
13172:
13117:
13062:
13015:
13014:{\displaystyle \ln(2)}
12962:
12664:
12618:
12553:
12520:
12462:
12286:
12144:
12036:
11964:
11933:
11885:
11823:
11705:
11577:
11519:
11518:{\displaystyle Q\ll P}
11493:
11492:{\displaystyle Q\ll P}
11467:
11425:
11386:, which formulate two
11372:
11348:
11328:
11279:
11253:
11173:
11153:
11118:
11098:
11069:
11043:
11023:
10997:
10962:
10936:
10910:
10884:
10864:
10835:
10807:
10758:
10643:
10467:
10309:
10272:
10244:
10115:
10045:
9976:
9950:
9930:
9910:
9868:
9225:
8855:
8835:
8778:
8691:
8671:
8642:
8517:
8456:
8430:
8387:
8147:
8101:
8055:
8016:
7972:
7843:
7803:
7715:
7627:
7567:
7513:
7484:
7447:
7398:
7349:
6786:
6728:
6572:
6406:
6366:
6280:
6243:
6208:
6189:
6145:
6083:
6017:
5988:
5942:
5855:§ Interpretations
5839:
5797:
5768:
5714:
5713:{\displaystyle H(P,Q)}
5663:
5610:
5567:
5527:
5287:
5260:
5229:
5166:
5143:Kraft–McMillan theorem
5138:
5114:
5053:information projection
5017:
4975:
4906:information projection
4844:
4771:
4697:
4635:
4550:
4135:
3698:
3656:
3609:
3581:
3553:
3525:
3492:
3464:
3436:
3408:
3363:
3349:
3315:
3256:
3229:
3203:
3131:
3019:
2928:
2870:
2847:
2674:
2640:
2560:
2472:
2448:
2413:
2186:
2106:
2056:
1974:
1789:
1715:
1552:
1479:
1441:
1411:
1370:
1339:
1311:
1270:
1203:
1056:
907:
827:
750:
723:
696:
695:{\displaystyle I(1:2)}
661:
621:
581:
541:
514:
487:
412:
304:, a generalization of
276:as a model instead of
243:
90:
26623:Theory of Probability
26602:John Wiley & Sons
25939:10.1214/ss/1177009939
25905:John Wiley & Sons
24946:Chi-squared statistic
24926:
24900:
24812:Bretagnolle and Huber
24801:
24748:
24664:
24578:
24548:
24487:
24377:
24348:
24322:
24283:
24113:
24044:
23919:Further information:
23888:
23839:
23695:
23578:
23539:
23537:{\displaystyle T_{o}}
23512:
23510:{\displaystyle V_{o}}
23485:
23483:{\displaystyle p_{o}}
23458:
23408:
23373:
23371:{\displaystyle T_{o}}
23335:
23276:
23215:
23140:
23138:{\displaystyle P_{o}}
23113:
23111:{\displaystyle T_{o}}
23082:
23019:
22988:Helmholtz free energy
22977:
22951:
22910:
22828:
22826:{\displaystyle J/K\}}
22797:
22782:then surprisal is in
22777:
22701:
22641:
22639:{\displaystyle T_{o}}
22618:
22602:
22565:
22523:
22462:
22314:
22285:
22256:
22200:
22120:
22085:
22050:
22021:
21988:
21956:
21730:
21675:
21634:
21596:
21547:
21545:{\displaystyle f_{0}}
21464:
21305:
21272:
21270:{\displaystyle H_{0}}
21245:
21243:{\displaystyle H_{1}}
21215:
21213:{\displaystyle H_{0}}
21188:
21186:{\displaystyle H_{1}}
21157:
21155:{\displaystyle H_{1}}
21130:
21128:{\displaystyle H_{0}}
21108:against a hypothesis
21103:
21101:{\displaystyle H_{1}}
21076:
21074:{\displaystyle H_{0}}
21049:
21047:{\displaystyle H_{1}}
21018:
20899:
20831:
20633:
20517:
20386:
20228:
20193:
20145:
20080:
20037:
20004:
19814:
19773:
19738:
19689:
19546:
19427:
19386:
19347:
19321:
19243:
19202:
19011:
18928:
18891:
18847:
18812:
18749:
18291:
18200:
18049:
18020:
17976:
17947:
17888:
17689:
17642:
17601:
17560:
17275:
17249:
17216:
17186:
17154:
16993:
16851:
16628:
16238:
16088:
16033:
15994:
15970:positive semidefinite
15960:
15781:
15627:
15499:
15470:, it changes only to
15465:
15432:
15406:
15311:
15291:
15226:
15119:
15092:
15072:
15030:
14995:
14947:
14868:
14828:then it is said that
14819:
14745:
14678:
14636:
14554:
14521:
14482:
14391:
14353:
14315:
14278:Uniform distributions
14270:
14125:
14074:
13873:
13763:
13562:variational inference
13552:
13452:
13364:
13280:
13224:
13173:
13118:
13063:
13016:
12963:
12665:
12619:
12554:
12521:
12463:
12287:
12145:
12037:
11965:
11963:{\displaystyle Q^{*}}
11934:
11886:
11824:
11706:
11578:
11520:
11494:
11468:
11426:
11373:
11349:
11329:
11280:
11254:
11174:
11154:
11119:
11117:{\displaystyle \rho }
11099:
11070:
11044:
11042:{\displaystyle \rho }
11024:
10998:
10963:
10937:
10911:
10885:
10865:
10836:
10808:
10738:
10623:
10447:
10289:
10273:
10245:
10095:
10025:
9977:
9951:
9931:
9911:
9869:
9226:
8856:
8836:
8779:
8692:
8672:
8643:
8497:
8457:
8431:
8388:
8148:
8102:
8056:
8054:{\displaystyle (P,Q)}
8017:
7973:
7844:
7804:
7716:
7628:
7568:
7514:
7485:
7448:
7399:
7350:
6787:
6729:
6573:
6407:
6367:
6281:
6244:
6209:
6190:
6146:
6084:
6018:
5989:
5943:
5840:
5798:
5769:
5715:
5664:
5627:The relative entropy
5611:
5568:
5528:
5288:
5286:{\displaystyle x_{i}}
5261:
5230:
5167:
5165:{\displaystyle x_{i}}
5132:
5115:
5085:Finance (game theory)
5049:dually flat manifolds
5018:
4976:
4845:
4772:
4698:
4636:
4551:
4136:
3699:
3657:
3610:
3582:
3554:
3526:
3493:
3465:
3437:
3409:
3361:
3350:
3348:{\displaystyle p=1/3}
3316:
3257:
3230:
3228:{\displaystyle p=0.4}
3204:
3181:binomial distribution
3132:
3046:or as the divergence
3020:
2929:
2871:
2848:
2675:
2641:
2561:
2473:
2449:
2414:
2192:which exists because
2187:
2107:
2057:
1975:
1801:absolutely continuous
1790:
1734:probability densities
1716:
1553:
1480:
1442:
1412:
1371:
1340:
1312:
1271:
1204:
1057:
908:
847:statistical distances
828:
751:
724:
697:
662:
622:
582:
542:
520:per observation from
515:
513:{\displaystyle H_{2}}
488:
486:{\displaystyle H_{1}}
413:
244:
91:
26697:Information geometry
25392:. Machine learning.
24915:
24818:
24757:
24677:
24671:Pinsker's inequality
24638:
24624:probability distance
24609:statistical manifold
24599:proportional to the
24560:
24506:
24393:
24360:
24331:
24311:
24125:
24102:
24083:are the conditional
23983:
23855:
23746:
23590:
23548:
23521:
23494:
23467:
23417:
23385:
23355:
23285:
23224:
23149:
23122:
23095:
23047:
22993:
22960:
22934:
22875:
22865:entropy maximization
22863:). This constrained
22806:
22786:
22714:
22661:
22623:
22574:
22532:
22490:
22386:
22345:differential entropy
22312:{\displaystyle u(a)}
22294:
22283:{\displaystyle u(a)}
22265:
22209:
22137:
22094:
22059:
22048:{\displaystyle u(a)}
22030:
22019:{\displaystyle u(a)}
22001:
21986:{\displaystyle p(a)}
21968:
21742:
21692:
21673:{\displaystyle u(a)}
21655:
21608:
21556:
21529:
21497:for uncertainty are
21317:
21303:{\displaystyle p(H)}
21285:
21254:
21227:
21197:
21170:
21139:
21112:
21085:
21058:
21031:
20933:
20847:
20642:
20536:
20395:
20240:
20202:
20154:
20093:
20049:
20020:
19826:
19782:
19747:
19701:
19565:
19443:
19395:
19360:
19330:
19280:
19221:
19086:
18989:
18900:
18864:
18821:
18773:
18315:
18215:
18073:
18064:differential entropy
18047:{\displaystyle P(X)}
18029:
17993:
17974:{\displaystyle P(X)}
17956:
17920:
17724:
17663:
17610:
17575:
17299:
17258:
17247:{\displaystyle P(i)}
17229:
17199:
17184:{\displaystyle P(i)}
17166:
17074:
17048:, also known as the
16866:
16650:
16249:
16100:
16058:
16003:
15976:
15800:
15646:
15511:
15478:
15441:
15415:
15324:
15300:
15238:
15135:
15101:
15081:
15039:
15004:
14984:
14887:
14878:Pinsker's inequality
14835:
14757:
14702:
14645:
14603:
14573:statistical distance
14530:
14497:
14402:
14362:
14324:
14286:
14136:
14086:
13900:
13571:
13292:
13233:
13190:
13127:
13072:
13032:
12993:
12681:
12631:
12588:
12530:
12475:
12302:
12157:
12046:
11978:
11947:
11911:
11840:
11721:
11590:
11545:
11503:
11477:
11435:
11393:
11358:
11338:
11318:
11263:
11183:
11163:
11128:
11108:
11079:
11053:
11033:
11007:
10972:
10946:
10920:
10894:
10874:
10845:
10816:
10282:
10253:
9986:
9960:
9940:
9920:
9878:
9235:
8865:
8845:
8788:
8707:
8681:
8677:almost surely w.r.t
8652:
8466:
8440:
8398:
8157:
8111:
8065:
8033:
8028:probability measures
7984:
7853:
7813:
7725:
7637:
7597:
7585:Relative entropy is
7523:
7512:{\displaystyle q(x)}
7494:
7483:{\displaystyle p(x)}
7465:
7408:
7359:
6800:
6738:
6582:
6416:
6405:{\displaystyle y(x)}
6387:
6334:
6253:
6222:
6207:{\displaystyle \mu }
6198:
6155:
6093:
6031:
6001:
5956:
5901:
5807:
5796:{\displaystyle H(P)}
5778:
5728:
5689:
5671:statistical distance
5631:
5589:
5540:
5312:
5270:
5243:
5180:
5149:
5135:normal distributions
5097:
5045:exponential families
4985:
4943:
4913:statistical distance
4876:Information geometry
4812:
4777:is often called the
4739:
4665:
4589:
4571:Neyman–Pearson lemma
4148:
3733:
3724:units of information
3666:
3624:
3592:
3564:
3536:
3524:{\displaystyle Q(x)}
3506:
3475:
3447:
3419:
3407:{\displaystyle P(x)}
3389:
3325:
3277:
3243:
3213:
3187:
3093:
3059:. This reflects the
2981:
2880:
2869:{\displaystyle \mu }
2860:
2695:
2658:
2646:exist (meaning that
2570:
2490:
2478:for which densities
2458:
2447:{\displaystyle \mu }
2438:
2211:
2116:
2086:
1987:
1822:
1766:
1580:
1492:
1454:
1421:
1380:
1376:is possible even if
1351:
1326:
1280:
1239:
1072:
928:
884:
876:defined on the same
760:
733:
706:
671:
631:
591:
551:
524:
497:
470:
380:
298:information geometry
118:
98:statistical distance
58:
26386:2020Entrp..22..221N
26315:2019Entrp..21..485N
26184:2008Cmplx..13c..18F
26135:1971SciAm.225c.179T
26123:Scientific American
26082:1957PhRv..108..171J
26032:1957PhRv..106..620J
25925:Statistical Science
25636:Novak S.Y. (2011),
25550:2020Entrp..22..860S
24062:Jeffreys divergence
23347:More generally the
22975:{\displaystyle T,V}
21310:of the hypotheses,
20035:{\displaystyle Y=y}
19345:{\displaystyle Y=y}
19326:. If some new fact
19266:Bayesian statistics
18306:conditional entropy
18300:Conditional entropy
17273:{\displaystyle i=m}
17214:{\displaystyle i=m}
17050:information content
17040:Information content
17012:conditional entropy
16766:
16678:
16569:
16443:
16326:
15430:{\displaystyle P=Q}
14937:
14908:
14857:
14597:Pythagorean theorem
14589:triangle inequality
14567:Relation to metrics
14052:
13993:
13855:
13821:
13803:
13701:
13677:
13167:
13112:
12865:
12781:
12626:covariance matrices
11312: —
8455:{\displaystyle P=Q}
7457:quantity, since if
7248:
6971:
6867:
6016:{\displaystyle P=Q}
5864:" in the theory of
4937:Pythagorean theorem
4929:triangle inequality
4900:that is closest to
3620:Relative entropies
3202:{\displaystyle N=2}
1747:More generally, if
1628:
1319:absolute continuity
855:Jeffreys divergence
314:Pythagorean theorem
294:triangle inequality
26607:Dover Publications
26439:10.1007/bfb0064610
26192:10.1002/cplx.20195
25642:Chapman & Hall
25497:Bell Syst. Tech. J
25396:. Cross validated.
25061:Hellinger distance
25036:Bregman divergence
24937:Hellinger distance
24921:
24895:
24796:
24743:
24659:
24573:
24543:
24482:
24372:
24343:
24317:
24278:
24108:
24039:
23959:maximum likelihood
23883:
23834:
23690:
23573:
23534:
23507:
23480:
23453:
23403:
23368:
23330:
23271:
23210:
23135:
23108:
23077:
23014:
22986:are constant, the
22972:
22946:
22921:partition function
22905:
22823:
22795:{\displaystyle \{}
22792:
22772:
22696:
22648:
22636:
22597:
22560:
22518:
22457:
22309:
22280:
22251:
22195:
22115:
22080:
22045:
22016:
21983:
21951:
21725:
21670:
21629:
21591:
21542:
21491:Riemann hypothesis
21459:
21300:
21267:
21240:
21210:
21183:
21152:
21135:, when hypothesis
21125:
21098:
21071:
21044:
21013:
20894:
20843:, averaging using
20826:
20628:
20512:
20511:
20408:
20381:
20252:
20223:
20188:
20140:
20075:
20032:
19999:
19910:
19809:
19768:
19733:
19684:
19626:
19541:
19422:
19381:
19342:
19316:
19270:prior distribution
19238:
19197:
19006:
18923:
18886:
18842:
18807:
18744:
18742:
18286:
18195:
18091:
18044:
18015:
17971:
17942:
17883:
17881:
17684:
17637:
17596:
17555:
17553:
17290:mutual information
17284:Mutual information
17270:
17244:
17211:
17181:
17149:
17063:When applied to a
16988:
16846:
16746:
16658:
16640:is independent of
16623:
16621:
16549:
16423:
16306:
16233:
16083:
16028:
15989:
15955:
15793:of the divergence
15776:
15622:
15494:
15460:
15427:
15401:
15306:
15296:a small change of
15286:
15221:
15114:
15087:
15067:
15025:
14990:
14942:
14863:
14814:
14775:
14740:
14673:
14631:
14549:
14516:
14477:
14386:
14351:{\displaystyle q=}
14348:
14313:{\displaystyle p=}
14310:
14265:
14120:
14069:
14038:
13979:
13868:
13841:
13807:
13789:
13687:
13663:
13547:
13275:
13219:
13168:
13153:
13113:
13098:
13058:
13011:
12958:
12848:
12764:
12660:
12614:
12549:
12516:
12458:
12282:
12140:
12032:
11960:
11929:
11905:
11881:
11819:
11701:
11573:
11515:
11489:
11463:
11421:
11388:probability spaces
11368:
11344:
11324:
11310:
11275:
11249:
11169:
11149:
11114:
11094:
11065:
11039:
11019:
10993:
10958:
10932:
10906:
10880:
10860:
10831:
10803:
10801:
10690:
10514:
10361:
10268:
10240:
10238:
10162:
9972:
9946:
9926:
9906:
9874:Hence solving for
9864:
9862:
9786:
9579:
9426:
9285:
9221:
9219:
9073:
8915:
8851:
8831:
8774:
8687:
8667:
8638:
8564:
8452:
8426:
8383:
8143:
8097:
8051:
8012:
7968:
7839:
7799:
7711:
7623:
7563:
7509:
7480:
7443:
7394:
7345:
7343:
7220:
6943:
6839:
6782:
6766:
6724:
6702:
6568:
6545:
6402:
6372:can be calculated.
6362:
6276:
6239:
6204:
6185:
6141:
6079:
6027:In particular, if
6013:
5984:
5948:a result known as
5938:
5878:mutual information
5835:
5793:
5764:
5710:
5659:
5606:
5563:
5523:
5521:
5432:
5370:
5283:
5256:
5225:
5162:
5139:
5110:
5064:Bregman divergence
5013:
4971:
4840:
4806:Bayesian inference
4767:
4731:In the context of
4693:
4657:In the context of
4631:
4546:
4544:
4206:
4131:
4129:
3791:
3718:to get results in
3694:
3652:
3605:
3577:
3549:
3521:
3488:
3460:
3432:
3404:
3364:
3345:
3311:
3255:{\displaystyle x=}
3252:
3225:
3199:
3127:
3065:Bayesian inference
3015:
2924:
2866:
2843:
2670:
2636:
2556:
2468:
2454:is any measure on
2444:
2409:
2182:
2102:
2074:, i.e. the unique
2052:
1970:
1785:
1711:
1611:
1561:For distributions
1548:
1517:
1475:
1437:
1407:
1366:
1335:
1307:
1266:
1199:
1125:
1052:
978:
903:
823:
746:
719:
692:
657:
617:
577:
537:
510:
483:
408:
310:exponential family
300:, it is a type of
239:
168:
86:
26605:. Republished by
26588:978-4-431-55977-1
26489:978-1-4419-2709-5
26448:978-3-540-08761-8
26395:10.3390/e22020221
26324:10.3390/e21050485
26263:978-1-4419-2973-0
26223:Wildlife Research
26111:footnote page 52.
25980:978-0-521-88068-8
25714:video lecture by
25650:978-1-4398-3574-6
25559:10.3390/e22080860
24981:Data differencing
24975:Data differencing
24909:Rényi divergences
24890:
24738:
24720:
24711:
24595:-divergences, is
24570:
24523:
24464:
24455:
24426:
24417:
24403:
24233:
24166:
24069:feature selection
24021:
23993:
23943:statistical model
23865:
23756:
23739:is defined to be
23676:
23652:
23635:
23042:Gibbs free energy
22926:When temperature
22846:average surprisal
22542:
22500:
22436:
22147:
22133:) and denoted by
21912:
21849:
21752:
21566:
21408:
21327:
20943:
20929:Relative entropy
20912:A common goal in
20757:
20652:
20546:
20505:
20399:
20375:
20243:
19993:
19901:
19836:
19617:
19539:
19260:Bayesian updating
19176:
19077:probability space
18979:Shannon's Entropy
18678:
18534:
18470:
18374:
18243:
18183:
18076:
17825:
17606:from the product
17506:
17428:
17341:
17119:
17104:
17020:probability space
16977:
16944:
16814:
16712:
16521:
16395:
16278:
16231:
16182:
16133:
16040:Riemannian metric
15912:
15880:
15710:
15656:
15570:
15538:
15383:
14938:
14909:
14858:
14783:
14760:
14655:
14613:
14475:
14412:
14146:
14067:
14054:
13971:
13910:
13761:
13581:
13537:
13357:
13302:
12944:
12746:
12691:
12485:
12396:
12334:
12202:
12167:
12123:
12030:
11903:
11879:
11814:
11760:
11677:
11631:
11306:
11236:
11211:
11172:{\displaystyle n}
10883:{\displaystyle 0}
10784:
10671:
10669:
10594:
10495:
10493:
10433:
10342:
10340:
10234:
10143:
10141:
10083:
10000:
9949:{\displaystyle 0}
9929:{\displaystyle f}
9888:
9858:
9767:
9692:
9560:
9485:
9407:
9377:
9266:
9180:
9054:
9027:
8896:
8854:{\displaystyle f}
8798:
8732:
8690:{\displaystyle Q}
8636:
8545:
8543:
8476:
8408:
8366:
8331:
8274:
8167:
7994:
7980:Relative entropy
7933:
7891:
7863:
7328:
7315:
7292:
7258:
7197:
7180:
7135:
7108:
7063:
7025:
6981:
6920:
6814:
6765:
6701:
6665:
6634:
6544:
6499:
6468:
6344:
6216:almost everywhere
5966:
5950:Gibbs' inequality
5911:
5817:
5641:
5470:
5413:
5408:
5351:
5326:
5107:
4995:
4953:
4880:In applications,
4822:
4749:
4675:
4461:
4437:
4397:
4380:
4340:
4323:
4283:
4259:
4187:
4162:
4046:
4022:
3982:
3965:
3925:
3908:
3868:
3844:
3772:
3747:
3676:
3634:
3618:
3617:
3603:
3575:
3547:
3486:
3458:
3430:
3126:
3106:
3098:
3014:
2994:
2986:
2923:
2900:
2885:
2839:
2829:
2815:
2807:
2791:
2776:
2760:
2705:
2669:
2663:
2635:
2625:
2589:
2575:
2555:
2545:
2509:
2495:
2405:
2395:
2381:
2373:
2364:
2341:
2314:
2310:
2301:
2278:
2221:
2181:
2171:
2135:
2121:
2101:
2091:
2050:
2048:
2038:
2022:
2012:
1998:
1992:
1966:
1956:
1942:
1934:
1925:
1902:
1875:
1832:
1781:
1771:
1707:
1700:
1692:
1684:
1643:
1590:
1544:
1495:
1474:
1459:
1436:
1426:
1406:
1385:
1365:
1356:
1306:
1285:
1265:
1244:
1195:
1187:
1171:
1156:
1140:
1106:
1082:
1048:
1040:
1024:
1009:
993:
959:
938:
921:is defined to be
899:
889:
390:
322:(Shannon) entropy
230:
214:
199:
183:
149:
128:
68:
16:(Redirected from
26709:
26662:Relative Entropy
26626:
26604:
26592:
26567:Amari, Shun-ichi
26558:
26557:
26523:
26517:
26515:
26509:
26501:
26473:
26467:
26465:
26464:
26463:
26424:
26418:
26417:
26407:
26397:
26379:
26353:
26347:
26346:
26336:
26326:
26308:
26282:
26276:
26275:
26247:
26241:
26240:
26238:
26214:
26208:
26207:
26202:. Archived from
26177:
26153:
26147:
26146:
26118:
26112:
26110:
26100:
26094:
26093:
26067:
26058:
26052:
26051:
26017:
26008:
26002:
26001:
25991:
25985:
25984:
25960:
25954:
25953:
25951:
25941:
25915:
25909:
25908:
25896:
25887:
25886:
25868:
25859:
25858:
25818:
25812:
25811:
25809:
25800:
25794:
25793:
25775:
25766:(6): 1549–1568.
25755:
25749:
25748:
25728:
25722:
25712:Relative Entropy
25708:
25702:
25701:
25699:
25684:
25678:
25677:
25659:
25653:
25634:
25628:
25627:
25615:
25609:
25608:
25588:
25582:
25581:
25571:
25561:
25543:
25519:
25513:
25512:
25492:
25486:
25485:
25469:
25459:
25453:
25447:
25441:
25440:
25422:
25416:
25415:
25404:
25398:
25397:
25386:
25380:
25379:
25359:
25353:
25347:
25341:
25340:
25307:
25301:
25295:
25289:
25283:
25274:
25268:
25259:
25253:
25244:
25238:
25223:
25217:
25206:
25205:
25187:
25154:
25148:
25147:
25145:
25121:
25091:Solomon Kullback
24930:
24928:
24927:
24922:
24904:
24902:
24901:
24896:
24891:
24889:
24888:
24872:
24871:
24870:
24843:
24809:
24805:
24803:
24802:
24797:
24774:
24773:
24772:
24752:
24750:
24749:
24744:
24739:
24722:
24721:
24718:
24712:
24704:
24702:
24668:
24666:
24665:
24660:
24605:Hellinger metric
24594:
24590:
24586:
24582:
24580:
24579:
24574:
24572:
24571:
24568:
24552:
24550:
24549:
24544:
24524:
24516:
24498:
24491:
24489:
24488:
24483:
24466:
24465:
24462:
24456:
24448:
24428:
24427:
24424:
24418:
24410:
24405:
24404:
24401:
24381:
24379:
24378:
24373:
24352:
24350:
24349:
24344:
24326:
24324:
24323:
24318:
24306:
24302:
24298:
24294:
24287:
24285:
24284:
24279:
24235:
24234:
24231:
24168:
24167:
24164:
24137:
24136:
24117:
24115:
24114:
24109:
24082:
24078:
24075:problems, where
24054:§ Etymology
24048:
24046:
24045:
24040:
24023:
24022:
24019:
23995:
23994:
23991:
23904:
23896:
23892:
23890:
23889:
23884:
23867:
23866:
23863:
23843:
23841:
23840:
23835:
23758:
23757:
23754:
23738:
23734:
23722:
23718:
23714:density matrices
23699:
23697:
23696:
23691:
23686:
23682:
23681:
23677:
23675:
23674:
23662:
23653:
23645:
23640:
23636:
23634:
23633:
23621:
23582:
23580:
23579:
23574:
23566:
23565:
23543:
23541:
23540:
23535:
23533:
23532:
23516:
23514:
23513:
23508:
23506:
23505:
23489:
23487:
23486:
23481:
23479:
23478:
23462:
23460:
23459:
23454:
23449:
23448:
23439:
23412:
23410:
23409:
23404:
23377:
23375:
23374:
23369:
23367:
23366:
23342:Gibbs inequality
23339:
23337:
23336:
23331:
23280:
23278:
23277:
23272:
23270:
23269:
23260:
23255:
23254:
23236:
23235:
23219:
23217:
23216:
23211:
23206:
23205:
23196:
23182:
23181:
23144:
23142:
23141:
23136:
23134:
23133:
23117:
23115:
23114:
23109:
23107:
23106:
23086:
23084:
23083:
23078:
23039:
23035:
23031:
23027:
23023:
23021:
23020:
23015:
22985:
22981:
22979:
22978:
22973:
22955:
22953:
22952:
22947:
22929:
22918:
22914:
22912:
22911:
22906:
22862:
22858:
22850:
22840:
22836:
22832:
22830:
22829:
22824:
22816:
22801:
22799:
22798:
22793:
22781:
22779:
22778:
22773:
22771:
22767:
22766:
22765:
22735:
22709:
22705:
22703:
22702:
22697:
22689:
22656:
22645:
22643:
22642:
22637:
22635:
22634:
22606:
22604:
22603:
22598:
22581:
22569:
22567:
22566:
22561:
22544:
22543:
22540:
22527:
22525:
22524:
22519:
22502:
22501:
22498:
22485:
22481:
22473:
22466:
22464:
22463:
22458:
22438:
22437:
22434:
22416:
22393:
22378:
22374:
22370:
22367:with respect to
22366:
22362:
22322:
22318:
22316:
22315:
22310:
22289:
22287:
22286:
22281:
22260:
22258:
22257:
22252:
22204:
22202:
22201:
22196:
22149:
22148:
22145:
22124:
22122:
22121:
22116:
22089:
22087:
22086:
22081:
22054:
22052:
22051:
22046:
22025:
22023:
22022:
22017:
21996:
21992:
21990:
21989:
21984:
21960:
21958:
21957:
21952:
21914:
21913:
21910:
21901:
21897:
21851:
21850:
21847:
21833:
21832:
21754:
21753:
21750:
21734:
21732:
21731:
21726:
21687:
21683:
21679:
21677:
21676:
21671:
21650:
21646:
21642:
21638:
21636:
21635:
21630:
21600:
21598:
21597:
21592:
21587:
21586:
21568:
21567:
21564:
21551:
21549:
21548:
21543:
21541:
21540:
21524:
21515:
21514:
21481:information gain
21474:utility function
21468:
21466:
21465:
21460:
21410:
21409:
21406:
21382:
21381:
21354:
21353:
21329:
21328:
21325:
21309:
21307:
21306:
21301:
21276:
21274:
21273:
21268:
21266:
21265:
21249:
21247:
21246:
21241:
21239:
21238:
21219:
21217:
21216:
21211:
21209:
21208:
21192:
21190:
21189:
21184:
21182:
21181:
21161:
21159:
21158:
21153:
21151:
21150:
21134:
21132:
21131:
21126:
21124:
21123:
21107:
21105:
21104:
21099:
21097:
21096:
21080:
21078:
21077:
21072:
21070:
21069:
21053:
21051:
21050:
21045:
21043:
21042:
21022:
21020:
21019:
21014:
21012:
21011:
21002:
21001:
20974:
20973:
20952:
20951:
20945:
20944:
20941:
20903:
20901:
20900:
20895:
20878:
20877:
20865:
20864:
20835:
20833:
20832:
20827:
20825:
20824:
20788:
20787:
20766:
20765:
20759:
20758:
20755:
20746:
20745:
20730:
20729:
20696:
20695:
20683:
20682:
20661:
20660:
20654:
20653:
20650:
20637:
20635:
20634:
20629:
20627:
20626:
20590:
20589:
20577:
20576:
20555:
20554:
20548:
20547:
20544:
20521:
20519:
20518:
20513:
20510:
20506:
20504:
20484:
20474:
20473:
20451:
20430:
20429:
20407:
20390:
20388:
20387:
20382:
20380:
20376:
20374:
20354:
20344:
20343:
20331:
20330:
20308:
20287:
20286:
20274:
20273:
20251:
20232:
20230:
20229:
20224:
20197:
20195:
20194:
20189:
20178:
20177:
20149:
20147:
20146:
20141:
20130:
20129:
20117:
20116:
20088:
20084:
20082:
20081:
20076:
20074:
20073:
20061:
20060:
20041:
20039:
20038:
20033:
20015:
20008:
20006:
20005:
20000:
19998:
19994:
19992:
19972:
19946:
19909:
19897:
19896:
19845:
19844:
19838:
19837:
19834:
19818:
19816:
19815:
19810:
19777:
19775:
19774:
19769:
19742:
19740:
19739:
19734:
19708:
19693:
19691:
19690:
19685:
19625:
19610:
19609:
19579:
19578:
19572:
19550:
19548:
19547:
19542:
19540:
19538:
19518:
19474:
19431:
19429:
19428:
19423:
19390:
19388:
19387:
19382:
19355:
19351:
19349:
19348:
19343:
19325:
19323:
19322:
19317:
19255:
19251:
19247:
19245:
19244:
19239:
19228:
19206:
19204:
19203:
19198:
19178:
19177:
19174:
19156:
19121:
19120:
19093:
19074:
19070:
19066:
19062:
19054:
19050:
19035:
19031:
19019:
19015:
19013:
19012:
19007:
18996:
18984:
18976:
18968:
18963:prefix-free code
18959:entropy encoding
18952:
18940:
18936:
18932:
18930:
18929:
18924:
18916:
18895:
18893:
18892:
18887:
18876:
18875:
18859:
18851:
18849:
18848:
18843:
18816:
18814:
18813:
18808:
18785:
18784:
18764:
18760:
18753:
18751:
18750:
18745:
18743:
18739:
18735:
18734:
18730:
18720:
18719:
18707:
18703:
18680:
18679:
18676:
18662:
18661:
18628:
18591:
18580:
18564:
18563:
18536:
18535:
18532:
18472:
18471:
18468:
18438:
18410:
18409:
18376:
18375:
18372:
18326:
18295:
18293:
18292:
18287:
18270:
18265:
18245:
18244:
18241:
18204:
18202:
18201:
18196:
18184:
18182:
18168:
18154:
18101:
18100:
18090:
18053:
18051:
18050:
18045:
18024:
18022:
18021:
18016:
18005:
18004:
17988:
17980:
17978:
17977:
17972:
17951:
17949:
17948:
17943:
17932:
17931:
17915:
17903:
17899:
17892:
17890:
17889:
17884:
17882:
17878:
17874:
17864:
17863:
17842:
17841:
17827:
17826:
17823:
17793:
17789:
17785:
17772:
17771:
17735:
17704:
17701:if the value of
17700:
17693:
17691:
17690:
17685:
17658:
17654:
17646:
17644:
17643:
17638:
17605:
17603:
17602:
17597:
17564:
17562:
17561:
17556:
17554:
17508:
17507:
17504:
17492:
17491:
17476:
17430:
17429:
17426:
17414:
17413:
17398:
17343:
17342:
17339:
17279:
17277:
17276:
17271:
17253:
17251:
17250:
17245:
17224:
17220:
17218:
17217:
17212:
17190:
17188:
17187:
17182:
17158:
17156:
17155:
17150:
17145:
17141:
17137:
17136:
17121:
17120:
17117:
17106:
17105:
17102:
17081:
17046:self-information
17034:Self-information
16997:
16995:
16994:
16989:
16978:
16975:
16973:
16969:
16968:
16955:
16954:
16945:
16940:
16939:
16938:
16925:
16875:
16874:
16855:
16853:
16852:
16847:
16832:
16831:
16820:
16816:
16815:
16813:
16812:
16811:
16798:
16779:
16778:
16768:
16765:
16760:
16730:
16729:
16718:
16714:
16713:
16711:
16703:
16680:
16677:
16672:
16643:
16639:
16632:
16630:
16629:
16624:
16622:
16568:
16563:
16526:
16522:
16520:
16519:
16518:
16505:
16489:
16488:
16478:
16442:
16437:
16400:
16396:
16394:
16393:
16392:
16379:
16375:
16374:
16364:
16325:
16320:
16283:
16279:
16277:
16269:
16261:
16242:
16240:
16239:
16234:
16232:
16230:
16229:
16228:
16215:
16199:
16198:
16188:
16183:
16181:
16180:
16179:
16166:
16150:
16149:
16139:
16134:
16132:
16124:
16104:
16092:
16090:
16089:
16084:
16082:
16081:
16045:
16037:
16035:
16034:
16029:
16018:
16017:
15998:
15996:
15995:
15990:
15988:
15987:
15964:
15962:
15961:
15956:
15948:
15947:
15914:
15913:
15910:
15904:
15903:
15902:
15901:
15885:
15881:
15879:
15878:
15877:
15864:
15863:
15850:
15849:
15840:
15828:
15827:
15815:
15814:
15785:
15783:
15782:
15777:
15766:
15765:
15753:
15752:
15739:
15738:
15725:
15724:
15711:
15703:
15692:
15691:
15658:
15657:
15654:
15637:Taylor expansion
15631:
15629:
15628:
15623:
15606:
15605:
15572:
15571:
15568:
15562:
15561:
15560:
15559:
15543:
15539:
15537:
15536:
15535:
15519:
15503:
15501:
15500:
15495:
15493:
15492:
15469:
15467:
15466:
15461:
15459:
15458:
15436:
15434:
15433:
15428:
15410:
15408:
15407:
15402:
15397:
15396:
15384:
15382:
15381:
15380:
15367:
15359:
15354:
15350:
15349:
15336:
15335:
15319:
15315:
15313:
15312:
15307:
15295:
15293:
15292:
15287:
15285:
15284:
15275:
15274:
15253:
15252:
15230:
15228:
15227:
15222:
15211:
15210:
15198:
15197:
15187:
15186:
15168:
15167:
15123:
15121:
15120:
15115:
15113:
15112:
15096:
15094:
15093:
15088:
15076:
15074:
15073:
15068:
15063:
15062:
15034:
15032:
15031:
15026:
14999:
14997:
14996:
14991:
14979:
14975:
14951:
14949:
14948:
14943:
14926:
14925:
14924:
14900:
14899:
14898:
14872:
14870:
14869:
14864:
14859:
14849:
14847:
14846:
14823:
14821:
14820:
14815:
14798:
14797:
14785:
14784:
14781:
14774:
14749:
14747:
14746:
14741:
14730:
14729:
14717:
14716:
14694:on the space of
14682:
14680:
14679:
14674:
14657:
14656:
14653:
14640:
14638:
14637:
14632:
14615:
14614:
14611:
14562:
14558:
14556:
14555:
14550:
14542:
14541:
14525:
14523:
14522:
14517:
14509:
14508:
14492:
14486:
14484:
14483:
14478:
14476:
14474:
14463:
14452:
14441:
14437:
14436:
14435:
14426:
14425:
14414:
14413:
14410:
14395:
14393:
14392:
14387:
14357:
14355:
14354:
14349:
14320:enclosed within
14319:
14317:
14316:
14311:
14274:
14272:
14271:
14266:
14264:
14235:
14227:
14213:
14212:
14188:
14187:
14175:
14171:
14170:
14169:
14160:
14159:
14148:
14147:
14144:
14129:
14127:
14126:
14121:
14119:
14118:
14109:
14104:
14103:
14078:
14076:
14075:
14070:
14068:
14060:
14055:
14053:
14051:
14046:
14033:
14032:
14031:
14022:
14021:
14009:
14008:
13992:
13987:
13977:
13972:
13970:
13969:
13960:
13959:
13950:
13939:
13935:
13934:
13933:
13924:
13923:
13912:
13911:
13908:
13892:
13886:
13877:
13875:
13874:
13869:
13864:
13860:
13859:
13854:
13849:
13820:
13815:
13802:
13797:
13782:
13777:
13762:
13754:
13749:
13745:
13744:
13740:
13739:
13731:
13721:
13720:
13711:
13707:
13706:
13702:
13700:
13695:
13676:
13671:
13648:
13647:
13646:
13640:
13636:
13635:
13634:
13616:
13615:
13595:
13594:
13583:
13582:
13579:
13556:
13554:
13553:
13548:
13543:
13539:
13538:
13536:
13535:
13534:
13522:
13521:
13508:
13507:
13506:
13494:
13493:
13480:
13471:
13466:
13445:
13444:
13439:
13430:
13416:
13415:
13406:
13405:
13389:
13384:
13358:
13350:
13345:
13341:
13340:
13339:
13334:
13333:
13323:
13322:
13317:
13316:
13304:
13303:
13300:
13284:
13282:
13281:
13276:
13274:
13273:
13261:
13260:
13245:
13244:
13228:
13226:
13225:
13220:
13218:
13217:
13202:
13201:
13185:
13181:
13177:
13175:
13174:
13169:
13166:
13161:
13152:
13151:
13139:
13138:
13122:
13120:
13119:
13114:
13111:
13106:
13097:
13096:
13084:
13083:
13067:
13065:
13064:
13059:
13057:
13056:
13044:
13043:
13020:
13018:
13017:
13012:
12984:
12979:
12967:
12965:
12964:
12959:
12954:
12950:
12949:
12945:
12943:
12942:
12941:
12928:
12927:
12926:
12913:
12898:
12894:
12893:
12892:
12880:
12879:
12864:
12856:
12847:
12846:
12845:
12839:
12835:
12834:
12833:
12821:
12820:
12796:
12792:
12791:
12790:
12780:
12772:
12747:
12739:
12734:
12730:
12729:
12728:
12723:
12722:
12712:
12711:
12706:
12705:
12693:
12692:
12689:
12673:
12669:
12667:
12666:
12661:
12656:
12655:
12643:
12642:
12623:
12621:
12620:
12615:
12613:
12612:
12600:
12599:
12558:
12556:
12555:
12550:
12548:
12547:
12525:
12523:
12522:
12517:
12506:
12505:
12487:
12486:
12483:
12467:
12465:
12464:
12459:
12439:
12438:
12417:
12416:
12398:
12397:
12394:
12370:
12369:
12336:
12335:
12332:
12314:
12313:
12291:
12289:
12288:
12283:
12263:
12262:
12235:
12234:
12204:
12203:
12200:
12188:
12187:
12169:
12168:
12165:
12149:
12147:
12146:
12141:
12124:
12122:
12106:
12105:
12095:
12075:
12058:
12057:
12041:
12039:
12038:
12033:
12031:
12029:
12013:
12012:
12002:
11982:
11973:
11969:
11967:
11966:
11961:
11959:
11958:
11942:
11939:with respect to
11938:
11936:
11935:
11930:
11898:
11895:with respect to
11894:
11890:
11888:
11887:
11882:
11880:
11878:
11861:
11844:
11835:
11828:
11826:
11825:
11820:
11815:
11813:
11797:
11796:
11786:
11766:
11761:
11759:
11742:
11725:
11710:
11708:
11707:
11702:
11679:
11678:
11675:
11657:
11656:
11644:
11643:
11632:
11629:
11608:
11607:
11582:
11580:
11579:
11574:
11563:
11562:
11536:
11532:
11528:
11524:
11522:
11521:
11516:
11498:
11496:
11495:
11490:
11472:
11470:
11469:
11464:
11453:
11452:
11430:
11428:
11427:
11422:
11411:
11410:
11385:
11381:
11377:
11375:
11374:
11369:
11367:
11366:
11353:
11351:
11350:
11345:
11333:
11331:
11330:
11325:
11313:
11309:
11284:
11282:
11281:
11276:
11258:
11256:
11255:
11250:
11248:
11247:
11242:
11238:
11237:
11229:
11212:
11210:
11187:
11178:
11176:
11175:
11170:
11158:
11156:
11155:
11150:
11123:
11121:
11120:
11115:
11103:
11101:
11100:
11095:
11074:
11072:
11071:
11066:
11048:
11046:
11045:
11040:
11028:
11026:
11025:
11020:
11002:
11000:
10999:
10994:
10967:
10965:
10964:
10959:
10941:
10939:
10938:
10933:
10915:
10913:
10912:
10907:
10889:
10887:
10886:
10881:
10869:
10867:
10866:
10861:
10840:
10838:
10837:
10832:
10812:
10810:
10809:
10804:
10802:
10789:
10785:
10783:
10760:
10757:
10752:
10731:
10727:
10723:
10689:
10688:
10687:
10670:
10668:
10645:
10642:
10637:
10616:
10612:
10611:
10600:
10596:
10595:
10593:
10579:
10565:
10551:
10547:
10513:
10512:
10511:
10494:
10492:
10469:
10466:
10461:
10439:
10435:
10434:
10432:
10431:
10430:
10405:
10404:
10403:
10363:
10360:
10359:
10358:
10341:
10339:
10316:
10308:
10303:
10277:
10275:
10274:
10269:
10249:
10247:
10246:
10241:
10239:
10235:
10233:
10232:
10231:
10206:
10205:
10204:
10164:
10161:
10160:
10159:
10142:
10140:
10117:
10114:
10109:
10088:
10084:
10082:
10074:
10064:
10063:
10047:
10044:
10039:
10002:
10001:
9998:
9981:
9979:
9978:
9973:
9955:
9953:
9952:
9947:
9935:
9933:
9932:
9927:
9915:
9913:
9912:
9907:
9890:
9889:
9886:
9873:
9871:
9870:
9865:
9863:
9859:
9857:
9856:
9855:
9830:
9829:
9828:
9788:
9785:
9784:
9783:
9748:
9747:
9713:
9712:
9693:
9691:
9690:
9679:
9675:
9623:
9622:
9621:
9581:
9578:
9577:
9576:
9541:
9540:
9506:
9505:
9486:
9484:
9470:
9469:
9468:
9428:
9425:
9424:
9423:
9390:
9378:
9376:
9329:
9328:
9327:
9287:
9284:
9283:
9282:
9249:
9230:
9228:
9227:
9222:
9220:
9197:
9185:
9181:
9179:
9165:
9118:
9072:
9071:
9070:
9047:
9043:
9039:
9032:
9028:
9026:
9012:
8965:
8914:
8913:
8912:
8879:
8860:
8858:
8857:
8852:
8840:
8838:
8837:
8832:
8800:
8799:
8796:
8783:
8781:
8780:
8775:
8734:
8733:
8730:
8696:
8694:
8693:
8688:
8676:
8674:
8673:
8668:
8647:
8645:
8644:
8639:
8637:
8635:
8634:
8633:
8608:
8607:
8606:
8566:
8563:
8562:
8561:
8544:
8542:
8519:
8516:
8511:
8478:
8477:
8474:
8461:
8459:
8458:
8453:
8435:
8433:
8432:
8427:
8410:
8409:
8406:
8392:
8390:
8389:
8384:
8367:
8364:
8359:
8358:
8346:
8345:
8333:
8332:
8329:
8302:
8301:
8289:
8288:
8276:
8275:
8272:
8257:
8256:
8229:
8228:
8213:
8212:
8185:
8184:
8169:
8168:
8165:
8152:
8150:
8149:
8144:
8139:
8138:
8126:
8125:
8106:
8104:
8103:
8098:
8093:
8092:
8080:
8079:
8060:
8058:
8057:
8052:
8021:
8019:
8018:
8013:
7996:
7995:
7992:
7977:
7975:
7974:
7969:
7961:
7960:
7948:
7947:
7935:
7934:
7931:
7919:
7918:
7906:
7905:
7893:
7892:
7889:
7865:
7864:
7861:
7848:
7846:
7845:
7840:
7838:
7837:
7825:
7824:
7808:
7806:
7805:
7800:
7786:
7785:
7764:
7763:
7721:, and likewise
7720:
7718:
7717:
7712:
7698:
7697:
7676:
7675:
7632:
7630:
7629:
7624:
7622:
7621:
7609:
7608:
7576:self-information
7572:
7570:
7569:
7564:
7518:
7516:
7515:
7510:
7489:
7487:
7486:
7481:
7460:
7452:
7450:
7449:
7444:
7439:
7438:
7420:
7419:
7403:
7401:
7400:
7395:
7390:
7389:
7371:
7370:
7354:
7352:
7351:
7346:
7344:
7333:
7329:
7327:
7317:
7316:
7308:
7304:
7294:
7293:
7285:
7281:
7260:
7259:
7251:
7247:
7246:
7245:
7235:
7234:
7233:
7213:
7202:
7198:
7196:
7195:
7181:
7179:
7171:
7163:
7161:
7137:
7136:
7128:
7124:
7123:
7109:
7107:
7099:
7091:
7089:
7065:
7064:
7056:
7052:
7040:
7026:
7024:
7016:
7008:
7006:
6983:
6982:
6974:
6970:
6969:
6968:
6958:
6957:
6956:
6936:
6925:
6921:
6919:
6905:
6891:
6866:
6865:
6864:
6854:
6853:
6852:
6816:
6815:
6812:
6791:
6789:
6788:
6783:
6781:
6767:
6764:
6756:
6748:
6745:
6733:
6731:
6730:
6725:
6717:
6703:
6700:
6692:
6684:
6681:
6667:
6666:
6658:
6636:
6635:
6627:
6577:
6575:
6574:
6569:
6560:
6546:
6543:
6535:
6527:
6524:
6501:
6500:
6492:
6470:
6469:
6461:
6411:
6409:
6408:
6403:
6382:
6371:
6369:
6368:
6363:
6346:
6345:
6342:
6329:
6325:
6317:
6313:
6309:
6305:
6301:
6297:
6285:
6283:
6282:
6277:
6260:
6248:
6246:
6245:
6240:
6229:
6213:
6211:
6210:
6205:
6194:
6192:
6191:
6186:
6150:
6148:
6147:
6142:
6088:
6086:
6085:
6080:
6022:
6020:
6019:
6014:
5993:
5991:
5990:
5985:
5968:
5967:
5964:
5947:
5945:
5944:
5939:
5913:
5912:
5909:
5876:. Consequently,
5866:large deviations
5844:
5842:
5841:
5836:
5819:
5818:
5815:
5802:
5800:
5799:
5794:
5773:
5771:
5770:
5765:
5719:
5717:
5716:
5711:
5680:
5676:
5668:
5666:
5665:
5660:
5643:
5642:
5639:
5623:
5615:
5613:
5612:
5607:
5596:
5584:
5580:
5572:
5570:
5569:
5564:
5547:
5532:
5530:
5529:
5524:
5522:
5509:
5486:
5475:
5471:
5469:
5452:
5431:
5430:
5429:
5409:
5407:
5390:
5369:
5368:
5367:
5328:
5327:
5324:
5300:
5296:
5292:
5290:
5289:
5284:
5282:
5281:
5265:
5263:
5262:
5257:
5255:
5254:
5238:
5234:
5232:
5231:
5226:
5224:
5223:
5222:
5221:
5198:
5197:
5175:
5171:
5169:
5168:
5163:
5161:
5160:
5119:
5117:
5116:
5111:
5109:
5108:
5105:
5070:
5035:that equals the
5022:
5020:
5019:
5014:
4997:
4996:
4993:
4980:
4978:
4977:
4972:
4955:
4954:
4951:
4903:
4899:
4895:
4887:
4883:
4871:
4867:
4863:
4856:
4849:
4847:
4846:
4841:
4824:
4823:
4820:
4800:
4797:with respect to
4796:
4791:relative entropy
4788:
4784:
4779:information gain
4776:
4774:
4773:
4768:
4751:
4750:
4747:
4733:machine learning
4722:
4718:
4714:
4702:
4700:
4699:
4694:
4677:
4676:
4673:
4648:
4644:
4640:
4638:
4637:
4632:
4584:
4580:
4576:
4555:
4553:
4552:
4547:
4545:
4535:
4531:
4462:
4454:
4446:
4442:
4438:
4436:
4432:
4423:
4419:
4410:
4398:
4390:
4385:
4381:
4379:
4375:
4366:
4362:
4353:
4341:
4333:
4328:
4324:
4322:
4318:
4309:
4305:
4296:
4284:
4276:
4268:
4264:
4260:
4258:
4244:
4230:
4205:
4204:
4203:
4164:
4163:
4160:
4140:
4138:
4137:
4132:
4130:
4117:
4113:
4047:
4039:
4031:
4027:
4023:
4021:
4017:
4008:
4004:
3995:
3983:
3975:
3970:
3966:
3964:
3960:
3951:
3947:
3938:
3926:
3918:
3913:
3909:
3907:
3903:
3894:
3890:
3881:
3869:
3861:
3853:
3849:
3845:
3843:
3829:
3815:
3790:
3789:
3788:
3749:
3748:
3745:
3717:
3712:
3703:
3701:
3700:
3695:
3678:
3677:
3674:
3661:
3659:
3658:
3653:
3636:
3635:
3632:
3614:
3612:
3611:
3606:
3604:
3596:
3586:
3584:
3583:
3578:
3576:
3568:
3558:
3556:
3555:
3550:
3548:
3540:
3530:
3528:
3527:
3522:
3497:
3495:
3494:
3489:
3487:
3479:
3469:
3467:
3466:
3461:
3459:
3451:
3441:
3439:
3438:
3433:
3431:
3423:
3413:
3411:
3410:
3405:
3371:
3366:
3365:
3354:
3352:
3351:
3346:
3341:
3320:
3318:
3317:
3312:
3286:
3285:
3272:
3268:
3264:
3261:
3259:
3258:
3253:
3238:
3234:
3232:
3231:
3226:
3208:
3206:
3205:
3200:
3178:
3174:
3170:
3158:
3154:
3149:information gain
3146:
3140:
3136:
3134:
3133:
3128:
3124:
3108:
3107:
3104:
3096:
3088:
3077:
3058:
3052:
3045:
3039:
3035:
3031:
3024:
3022:
3021:
3016:
3012:
2996:
2995:
2992:
2984:
2969:
2944:Gaussian measure
2940:Lebesgue measure
2936:counting measure
2933:
2931:
2930:
2925:
2921:
2920:
2916:
2901:
2893:
2883:
2875:
2873:
2872:
2867:
2852:
2850:
2849:
2844:
2837:
2827:
2826:
2813:
2812:
2808:
2806:
2792:
2789:
2774:
2772:
2758:
2745:
2744:
2743:
2742:
2707:
2706:
2703:
2687:
2683:
2679:
2677:
2676:
2671:
2667:
2661:
2653:
2649:
2645:
2643:
2642:
2637:
2633:
2623:
2622:
2587:
2586:
2573:
2565:
2563:
2562:
2557:
2553:
2543:
2542:
2507:
2506:
2493:
2485:
2481:
2477:
2475:
2474:
2469:
2467:
2466:
2453:
2451:
2450:
2445:
2433:
2429:
2418:
2416:
2415:
2410:
2403:
2393:
2392:
2379:
2378:
2374:
2372:
2362:
2361:
2349:
2339:
2338:
2326:
2312:
2311:
2309:
2299:
2298:
2286:
2276:
2275:
2263:
2261:
2260:
2259:
2258:
2223:
2222:
2219:
2199:
2195:
2191:
2189:
2188:
2183:
2179:
2169:
2168:
2133:
2132:
2119:
2111:
2109:
2108:
2103:
2099:
2098:
2097:
2089:
2081:
2077:
2073:
2070:with respect to
2069:
2061:
2059:
2058:
2053:
2051:
2049:
2046:
2036:
2035:
2023:
2020:
2010:
2009:
1996:
1994:
1990:
1979:
1977:
1976:
1971:
1964:
1954:
1953:
1940:
1939:
1935:
1933:
1923:
1922:
1910:
1900:
1899:
1887:
1873:
1872:
1871:
1870:
1869:
1834:
1833:
1830:
1814:
1810:
1806:
1803:with respect to
1798:
1794:
1792:
1791:
1786:
1779:
1778:
1777:
1769:
1761:measurable space
1755:are probability
1754:
1750:
1743:
1739:
1731:
1727:
1720:
1718:
1717:
1712:
1705:
1698:
1697:
1690:
1689:
1685:
1683:
1669:
1655:
1641:
1627:
1622:
1592:
1591:
1588:
1568:
1564:
1557:
1555:
1554:
1549:
1542:
1516:
1515:
1514:
1484:
1482:
1481:
1476:
1472:
1457:
1446:
1444:
1443:
1438:
1434:
1433:
1432:
1424:
1416:
1414:
1413:
1408:
1404:
1383:
1375:
1373:
1372:
1367:
1363:
1354:
1346:
1344:
1342:
1341:
1336:
1316:
1314:
1313:
1308:
1304:
1283:
1275:
1273:
1272:
1267:
1263:
1242:
1234:
1227:
1223:
1219:
1208:
1206:
1205:
1200:
1193:
1192:
1188:
1186:
1172:
1169:
1154:
1152:
1138:
1124:
1123:
1122:
1084:
1083:
1080:
1061:
1059:
1058:
1053:
1046:
1045:
1041:
1039:
1025:
1022:
1007:
1005:
991:
977:
976:
975:
940:
939:
936:
920:
916:
912:
910:
909:
904:
897:
896:
895:
887:
875:
871:
832:
830:
829:
824:
755:
753:
752:
747:
745:
744:
728:
726:
725:
720:
718:
717:
701:
699:
698:
693:
666:
664:
663:
658:
656:
655:
643:
642:
626:
624:
623:
618:
616:
615:
603:
602:
586:
584:
583:
578:
576:
575:
563:
562:
546:
544:
543:
538:
536:
535:
519:
517:
516:
511:
509:
508:
492:
490:
489:
484:
482:
481:
456:Solomon Kullback
437:
433:
429:
425:
421:
417:
415:
414:
409:
392:
391:
388:
375:
371:
367:
363:
359:
346:machine learning
306:squared distance
283:
279:
275:
263:
259:
248:
246:
245:
240:
235:
231:
229:
215:
212:
197:
195:
181:
167:
166:
165:
130:
129:
126:
110:
106:
95:
93:
92:
87:
70:
69:
66:
48:relative entropy
36:Kullback–Leibler
21:
18:Relative entropy
26717:
26716:
26712:
26711:
26710:
26708:
26707:
26706:
26677:
26676:
26634:
26629:
26589:
26561:
26524:
26520:
26503:
26502:
26490:
26474:
26470:
26461:
26459:
26449:
26425:
26421:
26354:
26350:
26283:
26279:
26264:
26248:
26244:
26236:10.1071/WR99107
26215:
26211:
26154:
26150:
26119:
26115:
26101:
26097:
26070:Physical Review
26065:
26059:
26055:
26020:Physical Review
26015:
26009:
26005:
26000:. Van Nostrand.
25992:
25988:
25981:
25961:
25957:
25916:
25912:
25897:
25890:
25869:
25862:
25819:
25815:
25807:
25801:
25797:
25756:
25752:
25729:
25725:
25709:
25705:
25685:
25681:
25674:
25660:
25656:
25635:
25631:
25616:
25612:
25589:
25585:
25520:
25516:
25493:
25489:
25482:
25460:
25456:
25448:
25444:
25423:
25419:
25410:. Integration.
25406:
25405:
25401:
25388:
25387:
25383:
25376:
25360:
25356:
25348:
25344:
25308:
25304:
25296:
25292:
25284:
25277:
25269:
25262:
25254:
25247:
25239:
25226:
25218:
25209:
25155:
25151:
25122:
25115:
25111:
25099:
25095:Richard Leibler
25021:
24983:
24977:
24916:
24913:
24912:
24863:
24862:
24858:
24854:
24850:
24842:
24819:
24816:
24815:
24807:
24765:
24764:
24760:
24758:
24755:
24754:
24717:
24713:
24703:
24701:
24678:
24675:
24674:
24639:
24636:
24635:
24620:
24592:
24588:
24584:
24567:
24563:
24561:
24558:
24557:
24515:
24507:
24504:
24503:
24496:
24461:
24457:
24447:
24423:
24419:
24409:
24400:
24396:
24394:
24391:
24390:
24361:
24358:
24357:
24353:respectively.
24332:
24329:
24328:
24312:
24309:
24308:
24304:
24300:
24299:is drawn from,
24296:
24292:
24230:
24226:
24163:
24159:
24132:
24128:
24126:
24123:
24122:
24103:
24100:
24099:
24080:
24076:
24058:Harold Jeffreys
24018:
24014:
23990:
23986:
23984:
23981:
23980:
23971:
23963:maximum spacing
23923:
23921:Model selection
23917:
23911:
23902:
23894:
23862:
23858:
23856:
23853:
23852:
23851:the minimum of
23753:
23749:
23747:
23744:
23743:
23736:
23732:
23720:
23716:
23710:
23670:
23666:
23661:
23657:
23644:
23629:
23625:
23620:
23616:
23612:
23608:
23591:
23588:
23587:
23561:
23557:
23549:
23546:
23545:
23528:
23524:
23522:
23519:
23518:
23501:
23497:
23495:
23492:
23491:
23474:
23470:
23468:
23465:
23464:
23444:
23440:
23435:
23418:
23415:
23414:
23386:
23383:
23382:
23362:
23358:
23356:
23353:
23352:
23286:
23283:
23282:
23265:
23261:
23256:
23250:
23246:
23231:
23227:
23225:
23222:
23221:
23201:
23197:
23192:
23177:
23173:
23150:
23147:
23146:
23129:
23125:
23123:
23120:
23119:
23102:
23098:
23096:
23093:
23092:
23048:
23045:
23044:
23037:
23033:
23029:
23025:
22994:
22991:
22990:
22983:
22961:
22958:
22957:
22935:
22932:
22931:
22927:
22916:
22876:
22873:
22872:
22860:
22856:
22848:
22838:
22834:
22812:
22807:
22804:
22803:
22802:nats, bits, or
22787:
22784:
22783:
22758:
22754:
22731:
22721:
22717:
22715:
22712:
22711:
22707:
22685:
22662:
22659:
22658:
22654:
22630:
22626:
22624:
22621:
22620:
22613:
22577:
22575:
22572:
22571:
22539:
22535:
22533:
22530:
22529:
22497:
22493:
22491:
22488:
22487:
22483:
22479:
22471:
22433:
22429:
22412:
22389:
22387:
22384:
22383:
22376:
22372:
22368:
22364:
22360:
22320:
22295:
22292:
22291:
22266:
22263:
22262:
22210:
22207:
22206:
22144:
22140:
22138:
22135:
22134:
22095:
22092:
22091:
22060:
22057:
22056:
22031:
22028:
22027:
22002:
21999:
21998:
21994:
21969:
21966:
21965:
21909:
21905:
21846:
21842:
21841:
21837:
21819:
21815:
21749:
21745:
21743:
21740:
21739:
21693:
21690:
21689:
21685:
21681:
21656:
21653:
21652:
21648:
21644:
21640:
21609:
21606:
21605:
21582:
21578:
21563:
21559:
21557:
21554:
21553:
21536:
21532:
21530:
21527:
21526:
21522:
21512:
21511:
21507:
21405:
21401:
21377:
21373:
21349:
21345:
21324:
21320:
21318:
21315:
21314:
21286:
21283:
21282:
21261:
21257:
21255:
21252:
21251:
21234:
21230:
21228:
21225:
21224:
21204:
21200:
21198:
21195:
21194:
21177:
21173:
21171:
21168:
21167:
21146:
21142:
21140:
21137:
21136:
21119:
21115:
21113:
21110:
21109:
21092:
21088:
21086:
21083:
21082:
21065:
21061:
21059:
21056:
21055:
21038:
21034:
21032:
21029:
21028:
21007:
21006:
20997:
20993:
20969:
20965:
20947:
20946:
20940:
20936:
20934:
20931:
20930:
20927:
20919:Bayes d-optimal
20910:
20873:
20869:
20860:
20856:
20848:
20845:
20844:
20820:
20819:
20783:
20779:
20761:
20760:
20754:
20750:
20741:
20740:
20725:
20721:
20691:
20687:
20678:
20674:
20656:
20655:
20649:
20645:
20643:
20640:
20639:
20622:
20621:
20585:
20581:
20572:
20568:
20550:
20549:
20543:
20539:
20537:
20534:
20533:
20485:
20469:
20465:
20452:
20450:
20446:
20425:
20421:
20403:
20396:
20393:
20392:
20355:
20339:
20335:
20326:
20322:
20309:
20307:
20303:
20282:
20278:
20269:
20265:
20247:
20241:
20238:
20237:
20203:
20200:
20199:
20173:
20169:
20155:
20152:
20151:
20125:
20121:
20112:
20108:
20094:
20091:
20090:
20086:
20069:
20065:
20056:
20052:
20050:
20047:
20046:
20021:
20018:
20017:
20013:
19973:
19947:
19945:
19941:
19905:
19892:
19891:
19840:
19839:
19833:
19829:
19827:
19824:
19823:
19783:
19780:
19779:
19748:
19745:
19744:
19704:
19702:
19699:
19698:
19621:
19605:
19604:
19574:
19573:
19568:
19566:
19563:
19562:
19519:
19475:
19473:
19444:
19441:
19440:
19396:
19393:
19392:
19361:
19358:
19357:
19353:
19331:
19328:
19327:
19281:
19278:
19277:
19262:
19253:
19249:
19224:
19222:
19219:
19218:
19214:section above.
19173:
19169:
19152:
19116:
19112:
19089:
19087:
19084:
19083:
19072:
19068:
19064:
19060:
19052:
19048:
19033:
19029:
19017:
18992:
18990:
18987:
18986:
18982:
18974:
18966:
18950:
18947:
18938:
18934:
18912:
18901:
18898:
18897:
18871:
18867:
18865:
18862:
18861:
18857:
18822:
18819:
18818:
18780:
18776:
18774:
18771:
18770:
18762:
18758:
18741:
18740:
18715:
18711:
18693:
18689:
18685:
18681:
18675:
18671:
18670:
18666:
18657:
18653:
18626:
18625:
18587:
18578:
18577:
18559:
18555:
18531:
18527:
18467:
18463:
18436:
18435:
18405:
18401:
18371:
18367:
18342:
18322:
18318:
18316:
18313:
18312:
18302:
18266:
18261:
18240:
18236:
18216:
18213:
18212:
18169:
18155:
18153:
18096:
18092:
18080:
18074:
18071:
18070:
18030:
18027:
18026:
18000:
17996:
17994:
17991:
17990:
17986:
17957:
17954:
17953:
17927:
17923:
17921:
17918:
17917:
17913:
17910:random variates
17901:
17897:
17880:
17879:
17859:
17855:
17837:
17833:
17832:
17828:
17822:
17818:
17791:
17790:
17767:
17763:
17762:
17758:
17745:
17731:
17727:
17725:
17722:
17721:
17715:Shannon entropy
17711:
17709:Shannon entropy
17702:
17698:
17664:
17661:
17660:
17656:
17652:
17611:
17608:
17607:
17576:
17573:
17572:
17552:
17551:
17503:
17499:
17487:
17483:
17474:
17473:
17425:
17421:
17409:
17405:
17396:
17395:
17338:
17334:
17327:
17302:
17300:
17297:
17296:
17286:
17259:
17256:
17255:
17230:
17227:
17226:
17222:
17200:
17197:
17196:
17193:Kronecker delta
17167:
17164:
17163:
17132:
17128:
17116:
17112:
17111:
17107:
17101:
17097:
17077:
17075:
17072:
17071:
17042:
17036:
17028:
17004:
16974:
16964:
16960:
16956:
16950:
16949:
16934:
16930:
16926:
16924:
16870:
16869:
16867:
16864:
16863:
16821:
16807:
16803:
16799:
16774:
16770:
16769:
16767:
16761:
16750:
16745:
16742:
16741:
16719:
16704:
16681:
16679:
16673:
16662:
16657:
16654:
16653:
16651:
16648:
16647:
16641:
16637:
16620:
16619:
16564:
16553:
16527:
16514:
16510:
16506:
16484:
16480:
16479:
16477:
16473:
16470:
16469:
16438:
16427:
16401:
16388:
16384:
16380:
16370:
16366:
16365:
16363:
16359:
16356:
16355:
16321:
16310:
16284:
16270:
16262:
16260:
16256:
16252:
16250:
16247:
16246:
16224:
16220:
16216:
16194:
16190:
16189:
16187:
16175:
16171:
16167:
16145:
16141:
16140:
16138:
16125:
16105:
16103:
16101:
16098:
16097:
16065:
16061:
16059:
16056:
16055:
16052:
16043:
16010:
16006:
16004:
16001:
16000:
15983:
15979:
15977:
15974:
15973:
15943:
15939:
15909:
15905:
15897:
15893:
15886:
15873:
15869:
15859:
15855:
15851:
15845:
15841:
15839:
15836:
15835:
15823:
15819:
15807:
15803:
15801:
15798:
15797:
15761:
15757:
15745:
15741:
15734:
15730:
15720:
15716:
15702:
15687:
15683:
15653:
15649:
15647:
15644:
15643:
15601:
15597:
15567:
15563:
15555:
15551:
15544:
15531:
15527:
15523:
15518:
15515:
15514:
15512:
15509:
15508:
15488:
15484:
15479:
15476:
15475:
15454:
15450:
15442:
15439:
15438:
15416:
15413:
15412:
15392:
15388:
15376:
15372:
15368:
15360:
15358:
15345:
15341:
15337:
15331:
15327:
15325:
15322:
15321:
15320:direction, and
15317:
15301:
15298:
15297:
15280:
15276:
15270:
15266:
15248:
15244:
15239:
15236:
15235:
15206:
15202:
15193:
15189:
15182:
15178:
15163:
15159:
15136:
15133:
15132:
15108:
15104:
15102:
15099:
15098:
15082:
15079:
15078:
15058:
15054:
15040:
15037:
15036:
15005:
15002:
15001:
14985:
14982:
14981:
14977:
14973:
14966:
14958:total variation
14920:
14916:
14894:
14890:
14888:
14885:
14884:
14848:
14842:
14838:
14836:
14833:
14832:
14793:
14789:
14780:
14776:
14764:
14758:
14755:
14754:
14725:
14721:
14712:
14708:
14703:
14700:
14699:
14690:It generates a
14652:
14648:
14646:
14643:
14642:
14641:does not equal
14610:
14606:
14604:
14601:
14600:
14569:
14560:
14537:
14533:
14531:
14528:
14527:
14504:
14500:
14498:
14495:
14494:
14490:
14464:
14453:
14451:
14431:
14430:
14421:
14420:
14419:
14415:
14409:
14405:
14403:
14400:
14399:
14363:
14360:
14359:
14325:
14322:
14321:
14287:
14284:
14283:
14280:
14251:
14231:
14223:
14205:
14201:
14183:
14179:
14165:
14164:
14155:
14154:
14153:
14149:
14143:
14139:
14137:
14134:
14133:
14114:
14110:
14105:
14099:
14095:
14087:
14084:
14083:
14059:
14047:
14042:
14034:
14027:
14023:
14017:
14013:
14004:
14000:
13988:
13983:
13978:
13976:
13965:
13961:
13955:
13951:
13949:
13929:
13928:
13919:
13918:
13917:
13913:
13907:
13903:
13901:
13898:
13897:
13888:
13882:
13850:
13845:
13837:
13816:
13811:
13798:
13793:
13788:
13784:
13778:
13767:
13753:
13735:
13727:
13726:
13722:
13716:
13715:
13696:
13691:
13672:
13667:
13662:
13658:
13642:
13641:
13630:
13626:
13611:
13607:
13606:
13602:
13601:
13600:
13596:
13590:
13589:
13588:
13584:
13578:
13574:
13572:
13569:
13568:
13527:
13523:
13517:
13513:
13509:
13499:
13495:
13489:
13485:
13481:
13479:
13467:
13456:
13440:
13435:
13434:
13426:
13411:
13407:
13398:
13394:
13385:
13368:
13363:
13359:
13349:
13335:
13329:
13328:
13327:
13318:
13312:
13311:
13310:
13309:
13305:
13299:
13295:
13293:
13290:
13289:
13269:
13265:
13256:
13252:
13240:
13236:
13234:
13231:
13230:
13213:
13209:
13197:
13193:
13191:
13188:
13187:
13183:
13179:
13162:
13157:
13147:
13143:
13134:
13130:
13128:
13125:
13124:
13107:
13102:
13092:
13088:
13079:
13075:
13073:
13070:
13069:
13052:
13048:
13039:
13035:
13033:
13030:
13029:
12994:
12991:
12990:
12982:
12977:
12937:
12933:
12929:
12922:
12918:
12914:
12912:
12908:
12888:
12884:
12875:
12871:
12870:
12866:
12857:
12852:
12841:
12840:
12829:
12825:
12816:
12812:
12811:
12807:
12806:
12786:
12782:
12773:
12768:
12763:
12759:
12752:
12748:
12738:
12724:
12718:
12717:
12716:
12707:
12701:
12700:
12699:
12698:
12694:
12688:
12684:
12682:
12679:
12678:
12671:
12651:
12647:
12638:
12634:
12632:
12629:
12628:
12608:
12604:
12595:
12591:
12589:
12586:
12585:
12578:
12573:
12561:
12543:
12539:
12531:
12528:
12527:
12501:
12497:
12482:
12478:
12476:
12473:
12472:
12434:
12430:
12412:
12408:
12393:
12389:
12365:
12361:
12331:
12327:
12309:
12305:
12303:
12300:
12299:
12258:
12254:
12230:
12226:
12199:
12195:
12183:
12179:
12164:
12160:
12158:
12155:
12154:
12101:
12097:
12096:
12076:
12074:
12053:
12049:
12047:
12044:
12043:
12008:
12004:
12003:
11983:
11981:
11979:
11976:
11975:
11971:
11954:
11950:
11948:
11945:
11944:
11940:
11912:
11909:
11908:
11901:
11896:
11892:
11862:
11845:
11843:
11841:
11838:
11837:
11833:
11792:
11788:
11787:
11767:
11765:
11743:
11726:
11724:
11722:
11719:
11718:
11674:
11670:
11652:
11648:
11633:
11628:
11627:
11603:
11599:
11591:
11588:
11587:
11558:
11557:
11546:
11543:
11542:
11539:random variable
11534:
11530:
11526:
11525:indicates that
11504:
11501:
11500:
11478:
11475:
11474:
11448:
11447:
11436:
11433:
11432:
11406:
11405:
11394:
11391:
11390:
11383:
11379:
11362:
11361:
11359:
11356:
11355:
11339:
11336:
11335:
11319:
11316:
11315:
11311:
11296:
11290:
11288:
11287:
11264:
11261:
11260:
11243:
11228:
11221:
11217:
11216:
11191:
11186:
11184:
11181:
11180:
11164:
11161:
11160:
11129:
11126:
11125:
11109:
11106:
11105:
11080:
11077:
11076:
11054:
11051:
11050:
11034:
11031:
11030:
11008:
11005:
11004:
10973:
10970:
10969:
10947:
10944:
10943:
10921:
10918:
10917:
10895:
10892:
10891:
10875:
10872:
10871:
10846:
10843:
10842:
10817:
10814:
10813:
10800:
10799:
10787:
10786:
10764:
10759:
10753:
10742:
10729:
10728:
10695:
10691:
10683:
10682:
10675:
10649:
10644:
10638:
10627:
10614:
10613:
10601:
10580:
10566:
10564:
10557:
10553:
10552:
10519:
10515:
10507:
10506:
10499:
10473:
10468:
10462:
10451:
10440:
10420:
10416:
10406:
10399:
10395:
10364:
10362:
10354:
10353:
10346:
10320:
10315:
10314:
10310:
10304:
10293:
10285:
10283:
10280:
10279:
10254:
10251:
10250:
10237:
10236:
10221:
10217:
10207:
10200:
10196:
10165:
10163:
10155:
10154:
10147:
10121:
10116:
10110:
10099:
10086:
10085:
10075:
10053:
10049:
10048:
10046:
10040:
10029:
10018:
9997:
9993:
9989:
9987:
9984:
9983:
9961:
9958:
9957:
9941:
9938:
9937:
9921:
9918:
9917:
9885:
9881:
9879:
9876:
9875:
9861:
9860:
9845:
9841:
9831:
9824:
9820:
9789:
9787:
9779:
9778:
9771:
9743:
9739:
9723:
9702:
9698:
9695:
9694:
9680:
9629:
9625:
9624:
9617:
9613:
9582:
9580:
9572:
9571:
9564:
9536:
9532:
9516:
9495:
9491:
9488:
9487:
9471:
9464:
9460:
9429:
9427:
9419:
9418:
9411:
9400:
9383:
9380:
9379:
9330:
9323:
9319:
9288:
9286:
9278:
9277:
9270:
9259:
9242:
9238:
9236:
9233:
9232:
9218:
9217:
9207:
9190:
9187:
9186:
9166:
9119:
9117:
9113:
9066:
9065:
9058:
9045:
9044:
9013:
8966:
8964:
8960:
8953:
8949:
8908:
8907:
8900:
8889:
8872:
8868:
8866:
8863:
8862:
8846:
8843:
8842:
8795:
8791:
8789:
8786:
8785:
8729:
8725:
8708:
8705:
8704:
8682:
8679:
8678:
8653:
8650:
8649:
8623:
8619:
8609:
8602:
8598:
8567:
8565:
8557:
8556:
8549:
8523:
8518:
8512:
8501:
8473:
8469:
8467:
8464:
8463:
8441:
8438:
8437:
8405:
8401:
8399:
8396:
8395:
8365: for
8363:
8354:
8350:
8341:
8337:
8328:
8324:
8297:
8293:
8284:
8280:
8271:
8267:
8252:
8248:
8224:
8220:
8208:
8204:
8180:
8176:
8164:
8160:
8158:
8155:
8154:
8134:
8130:
8121:
8117:
8112:
8109:
8108:
8088:
8084:
8075:
8071:
8066:
8063:
8062:
8034:
8031:
8030:
8026:in the pair of
7991:
7987:
7985:
7982:
7981:
7956:
7952:
7943:
7939:
7930:
7926:
7914:
7910:
7901:
7897:
7888:
7884:
7860:
7856:
7854:
7851:
7850:
7833:
7829:
7820:
7816:
7814:
7811:
7810:
7781:
7777:
7759:
7755:
7726:
7723:
7722:
7693:
7689:
7671:
7667:
7638:
7635:
7634:
7617:
7613:
7604:
7600:
7598:
7595:
7594:
7580:Shannon entropy
7524:
7521:
7520:
7495:
7492:
7491:
7466:
7463:
7462:
7458:
7434:
7430:
7415:
7411:
7409:
7406:
7405:
7385:
7381:
7366:
7362:
7360:
7357:
7356:
7342:
7341:
7307:
7306:
7305:
7284:
7283:
7282:
7280:
7276:
7250:
7249:
7241:
7237:
7236:
7229:
7225:
7224:
7211:
7210:
7191:
7172:
7164:
7162:
7157:
7127:
7126:
7125:
7119:
7100:
7092:
7090:
7085:
7055:
7054:
7053:
7051:
7047:
7036:
7017:
7009:
7007:
7002:
6973:
6972:
6964:
6960:
6959:
6952:
6948:
6947:
6934:
6933:
6906:
6892:
6890:
6886:
6860:
6856:
6855:
6848:
6844:
6843:
6832:
6811:
6807:
6803:
6801:
6798:
6797:
6777:
6757:
6749:
6746:
6741:
6739:
6736:
6735:
6713:
6693:
6685:
6682:
6677:
6657:
6656:
6626:
6625:
6583:
6580:
6579:
6556:
6536:
6528:
6525:
6520:
6491:
6490:
6460:
6459:
6417:
6414:
6413:
6388:
6385:
6384:
6380:
6341:
6337:
6335:
6332:
6331:
6327:
6323:
6315:
6311:
6307:
6303:
6299:
6295:
6256:
6254:
6251:
6250:
6225:
6223:
6220:
6219:
6199:
6196:
6195:
6156:
6153:
6152:
6094:
6091:
6090:
6032:
6029:
6028:
6002:
5999:
5998:
5963:
5959:
5957:
5954:
5953:
5908:
5904:
5902:
5899:
5898:
5890:
5814:
5810:
5808:
5805:
5804:
5779:
5776:
5775:
5729:
5726:
5725:
5690:
5687:
5686:
5678:
5674:
5638:
5634:
5632:
5629:
5628:
5621:
5592:
5590:
5587:
5586:
5582:
5578:
5543:
5541:
5538:
5537:
5520:
5519:
5505:
5482:
5473:
5472:
5456:
5451:
5425:
5424:
5417:
5394:
5389:
5363:
5362:
5355:
5344:
5323:
5319:
5315:
5313:
5310:
5309:
5298:
5294:
5277:
5273:
5271:
5268:
5267:
5250:
5246:
5244:
5241:
5240:
5236:
5217:
5213:
5209:
5205:
5193:
5189:
5181:
5178:
5177:
5173:
5156:
5152:
5150:
5147:
5146:
5127:
5104:
5100:
5098:
5095:
5094:
5087:
5068:
4992:
4988:
4986:
4983:
4982:
4981:does not equal
4950:
4946:
4944:
4941:
4940:
4901:
4897:
4893:
4885:
4881:
4878:
4869:
4865:
4861:
4854:
4819:
4815:
4813:
4810:
4809:
4798:
4794:
4786:
4782:
4746:
4742:
4740:
4737:
4736:
4729:
4720:
4716:
4712:
4672:
4668:
4666:
4663:
4662:
4655:
4646:
4642:
4590:
4587:
4586:
4582:
4578:
4574:
4567:
4562:
4560:Interpretations
4543:
4542:
4467:
4463:
4453:
4444:
4443:
4428:
4424:
4415:
4411:
4409:
4405:
4389:
4371:
4367:
4358:
4354:
4352:
4348:
4332:
4314:
4310:
4301:
4297:
4295:
4291:
4275:
4266:
4265:
4245:
4231:
4229:
4225:
4199:
4198:
4191:
4180:
4159:
4155:
4151:
4149:
4146:
4145:
4128:
4127:
4052:
4048:
4038:
4029:
4028:
4013:
4009:
4000:
3996:
3994:
3990:
3974:
3956:
3952:
3943:
3939:
3937:
3933:
3917:
3899:
3895:
3886:
3882:
3880:
3876:
3860:
3851:
3850:
3830:
3816:
3814:
3810:
3784:
3783:
3776:
3765:
3744:
3740:
3736:
3734:
3731:
3730:
3715:
3710:
3673:
3669:
3667:
3664:
3663:
3631:
3627:
3625:
3622:
3621:
3595:
3593:
3590:
3589:
3567:
3565:
3562:
3561:
3539:
3537:
3534:
3533:
3507:
3504:
3503:
3478:
3476:
3473:
3472:
3450:
3448:
3445:
3444:
3422:
3420:
3417:
3416:
3390:
3387:
3386:
3369:
3337:
3326:
3323:
3322:
3281:
3280:
3278:
3275:
3274:
3270:
3266:
3262:
3244:
3241:
3240:
3236:
3214:
3211:
3210:
3188:
3185:
3184:
3176:
3172:
3168:
3165:
3156:
3152:
3144:
3142:with respect to
3138:
3103:
3099:
3094:
3091:
3090:
3086:
3075:
3067:, which starts
3056:
3050:
3043:
3037:
3033:
3029:
2991:
2987:
2982:
2979:
2978:
2967:
2906:
2902:
2892:
2881:
2878:
2877:
2861:
2858:
2857:
2822:
2793:
2773:
2771:
2767:
2738:
2737:
2730:
2726:
2702:
2698:
2696:
2693:
2692:
2685:
2681:
2659:
2656:
2655:
2651:
2647:
2618:
2582:
2571:
2568:
2567:
2538:
2502:
2491:
2488:
2487:
2483:
2479:
2462:
2461:
2459:
2456:
2455:
2439:
2436:
2435:
2431:
2427:
2388:
2357:
2350:
2334:
2327:
2325:
2321:
2294:
2287:
2271:
2264:
2262:
2254:
2253:
2246:
2242:
2218:
2214:
2212:
2209:
2208:
2197:
2193:
2164:
2128:
2117:
2114:
2113:
2093:
2092:
2087:
2084:
2083:
2079:
2075:
2071:
2067:
2031:
2024:
2005:
1995:
1993:
1988:
1985:
1984:
1949:
1918:
1911:
1895:
1888:
1886:
1882:
1865:
1864:
1857:
1853:
1829:
1825:
1823:
1820:
1819:
1812:
1808:
1804:
1796:
1773:
1772:
1767:
1764:
1763:
1752:
1748:
1741:
1737:
1729:
1725:
1693:
1670:
1656:
1654:
1650:
1623:
1615:
1587:
1583:
1581:
1578:
1577:
1566:
1562:
1510:
1506:
1499:
1493:
1490:
1489:
1455:
1452:
1451:
1428:
1427:
1422:
1419:
1418:
1381:
1378:
1377:
1352:
1349:
1348:
1327:
1324:
1323:
1322:
1281:
1278:
1277:
1240:
1237:
1236:
1232:
1225:
1221:
1217:
1173:
1153:
1151:
1147:
1118:
1117:
1110:
1079:
1075:
1073:
1070:
1069:
1026:
1006:
1004:
1000:
971:
970:
963:
935:
931:
929:
926:
925:
918:
914:
891:
890:
885:
882:
881:
873:
869:
863:
839:Kullback (1959)
835:Harold Jeffreys
761:
758:
757:
740:
736:
734:
731:
730:
713:
709:
707:
704:
703:
672:
669:
668:
651:
647:
638:
634:
632:
629:
628:
611:
607:
598:
594:
592:
589:
588:
571:
567:
558:
554:
552:
549:
548:
531:
527:
525:
522:
521:
504:
500:
498:
495:
494:
477:
473:
471:
468:
467:
460:Richard Leibler
452:
435:
431:
427:
423:
419:
387:
383:
381:
378:
377:
373:
369:
365:
361:
357:
354:
334:fluid mechanics
281:
277:
273:
261:
257:
216:
196:
194:
190:
161:
160:
153:
125:
121:
119:
116:
115:
108:
104:
96:, is a type of
65:
61:
59:
56:
55:
28:
23:
22:
15:
12:
11:
5:
26715:
26705:
26704:
26702:Thermodynamics
26699:
26694:
26689:
26675:
26674:
26669:
26655:
26650:
26645:
26640:
26633:
26632:External links
26630:
26628:
26627:
26618:
26593:
26587:
26562:
26560:
26559:
26518:
26516:Equation 2.25.
26488:
26468:
26447:
26419:
26348:
26277:
26262:
26242:
26229:(2): 111–119.
26209:
26206:on 2011-08-13.
26148:
26129:(3): 179–186.
26113:
26109:. The Academy.
26095:
26076:(2): 171–190.
26053:
26026:(4): 620–630.
26003:
25986:
25979:
25955:
25932:(3): 273–304.
25910:
25888:
25860:
25833:(2): 291–309.
25813:
25795:
25750:
25739:(2): 183–212.
25723:
25703:
25679:
25673:978-0677032405
25672:
25654:
25629:
25610:
25583:
25514:
25503:(4): 917–926.
25487:
25480:
25454:
25442:
25431:. p. 55.
25417:
25399:
25381:
25374:
25354:
25342:
25323:(4): 340–341.
25302:
25290:
25288:, p. 158.
25275:
25260:
25245:
25224:
25207:
25149:
25136:(1): 146–158.
25112:
25110:
25107:
25106:
25105:
25098:
25097:
25088:
25083:
25078:
25073:
25068:
25063:
25058:
25053:
25048:
25043:
25038:
25033:
25028:
25022:
25020:
25017:
24979:Main article:
24976:
24973:
24956:match distance
24933:
24932:
24920:
24907:The family of
24905:
24894:
24887:
24884:
24881:
24878:
24875:
24869:
24866:
24861:
24857:
24853:
24849:
24846:
24841:
24838:
24835:
24832:
24829:
24826:
24823:
24795:
24792:
24789:
24786:
24783:
24780:
24777:
24771:
24768:
24763:
24742:
24737:
24734:
24731:
24728:
24725:
24716:
24710:
24707:
24700:
24697:
24694:
24691:
24688:
24685:
24682:
24658:
24655:
24652:
24649:
24646:
24643:
24619:
24616:
24566:
24554:
24553:
24542:
24539:
24536:
24533:
24530:
24527:
24522:
24519:
24514:
24511:
24493:
24492:
24481:
24478:
24475:
24472:
24469:
24460:
24454:
24451:
24446:
24443:
24440:
24437:
24434:
24431:
24422:
24416:
24413:
24408:
24399:
24371:
24368:
24365:
24342:
24339:
24336:
24316:
24289:
24288:
24277:
24274:
24271:
24268:
24265:
24262:
24259:
24256:
24253:
24250:
24247:
24244:
24241:
24238:
24229:
24225:
24222:
24219:
24216:
24213:
24210:
24207:
24204:
24201:
24198:
24195:
24192:
24189:
24186:
24183:
24180:
24177:
24174:
24171:
24162:
24158:
24155:
24152:
24149:
24146:
24143:
24140:
24135:
24131:
24107:
24073:classification
24050:
24049:
24038:
24035:
24032:
24029:
24026:
24017:
24013:
24010:
24007:
24004:
24001:
23998:
23989:
23970:
23967:
23932:available work
23910:
23907:
23882:
23879:
23876:
23873:
23870:
23861:
23845:
23844:
23833:
23830:
23827:
23824:
23821:
23818:
23815:
23812:
23809:
23806:
23803:
23800:
23797:
23794:
23791:
23788:
23785:
23782:
23779:
23776:
23773:
23770:
23767:
23764:
23761:
23752:
23709:
23706:
23701:
23700:
23689:
23685:
23680:
23673:
23669:
23665:
23660:
23656:
23651:
23648:
23643:
23639:
23632:
23628:
23624:
23619:
23615:
23611:
23607:
23604:
23601:
23598:
23595:
23572:
23569:
23564:
23560:
23556:
23553:
23531:
23527:
23504:
23500:
23477:
23473:
23452:
23447:
23443:
23438:
23434:
23431:
23428:
23425:
23422:
23402:
23399:
23396:
23393:
23390:
23365:
23361:
23349:work available
23329:
23326:
23323:
23320:
23317:
23314:
23311:
23308:
23305:
23302:
23299:
23296:
23293:
23290:
23268:
23264:
23259:
23253:
23249:
23245:
23242:
23239:
23234:
23230:
23209:
23204:
23200:
23195:
23191:
23188:
23185:
23180:
23176:
23172:
23169:
23166:
23163:
23160:
23157:
23154:
23132:
23128:
23105:
23101:
23076:
23073:
23070:
23067:
23064:
23061:
23058:
23055:
23052:
23028:is energy and
23013:
23010:
23007:
23004:
23001:
22998:
22971:
22968:
22965:
22945:
22942:
22939:
22904:
22901:
22898:
22895:
22892:
22889:
22886:
22883:
22880:
22822:
22819:
22815:
22811:
22791:
22770:
22764:
22761:
22757:
22753:
22750:
22747:
22744:
22741:
22738:
22734:
22730:
22727:
22724:
22720:
22695:
22692:
22688:
22684:
22681:
22678:
22675:
22672:
22669:
22666:
22657:is defined as
22633:
22629:
22612:
22609:
22596:
22593:
22590:
22587:
22584:
22580:
22570:, rather than
22559:
22556:
22553:
22550:
22547:
22538:
22517:
22514:
22511:
22508:
22505:
22496:
22468:
22467:
22456:
22453:
22450:
22447:
22444:
22441:
22432:
22428:
22425:
22422:
22419:
22415:
22411:
22408:
22405:
22402:
22399:
22396:
22392:
22308:
22305:
22302:
22299:
22279:
22276:
22273:
22270:
22250:
22247:
22244:
22241:
22238:
22235:
22232:
22229:
22226:
22223:
22220:
22217:
22214:
22194:
22191:
22188:
22185:
22182:
22179:
22176:
22173:
22170:
22167:
22164:
22161:
22158:
22155:
22152:
22143:
22114:
22111:
22108:
22105:
22102:
22099:
22079:
22076:
22073:
22070:
22067:
22064:
22044:
22041:
22038:
22035:
22015:
22012:
22009:
22006:
21982:
21979:
21976:
21973:
21962:
21961:
21950:
21947:
21944:
21941:
21938:
21935:
21932:
21929:
21926:
21923:
21920:
21917:
21908:
21904:
21900:
21896:
21893:
21890:
21887:
21884:
21881:
21878:
21875:
21872:
21869:
21866:
21863:
21860:
21857:
21854:
21845:
21840:
21836:
21831:
21828:
21825:
21822:
21818:
21814:
21811:
21808:
21805:
21802:
21799:
21796:
21793:
21790:
21787:
21784:
21781:
21778:
21775:
21772:
21769:
21766:
21763:
21760:
21757:
21748:
21724:
21721:
21718:
21715:
21712:
21709:
21706:
21703:
21700:
21697:
21669:
21666:
21663:
21660:
21628:
21625:
21622:
21619:
21616:
21613:
21590:
21585:
21581:
21577:
21574:
21571:
21562:
21539:
21535:
21506:
21503:
21470:
21469:
21458:
21455:
21452:
21449:
21446:
21443:
21440:
21437:
21434:
21431:
21428:
21425:
21422:
21419:
21416:
21413:
21404:
21400:
21397:
21394:
21391:
21388:
21385:
21380:
21376:
21372:
21369:
21366:
21363:
21360:
21357:
21352:
21348:
21344:
21341:
21338:
21335:
21332:
21323:
21299:
21296:
21293:
21290:
21264:
21260:
21237:
21233:
21207:
21203:
21180:
21176:
21149:
21145:
21122:
21118:
21095:
21091:
21068:
21064:
21041:
21037:
21010:
21005:
21000:
20996:
20992:
20989:
20986:
20983:
20980:
20977:
20972:
20968:
20964:
20961:
20958:
20955:
20950:
20939:
20926:
20923:
20909:
20906:
20893:
20890:
20887:
20884:
20881:
20876:
20872:
20868:
20863:
20859:
20855:
20852:
20837:
20836:
20823:
20818:
20815:
20812:
20809:
20806:
20803:
20800:
20797:
20794:
20791:
20786:
20782:
20778:
20775:
20772:
20769:
20764:
20753:
20749:
20744:
20739:
20736:
20733:
20728:
20724:
20720:
20717:
20714:
20711:
20708:
20705:
20702:
20699:
20694:
20690:
20686:
20681:
20677:
20673:
20670:
20667:
20664:
20659:
20648:
20625:
20620:
20617:
20614:
20611:
20608:
20605:
20602:
20599:
20596:
20593:
20588:
20584:
20580:
20575:
20571:
20567:
20564:
20561:
20558:
20553:
20542:
20523:
20522:
20509:
20503:
20500:
20497:
20494:
20491:
20488:
20483:
20480:
20477:
20472:
20468:
20464:
20461:
20458:
20455:
20449:
20445:
20442:
20439:
20436:
20433:
20428:
20424:
20420:
20417:
20414:
20411:
20406:
20402:
20379:
20373:
20370:
20367:
20364:
20361:
20358:
20353:
20350:
20347:
20342:
20338:
20334:
20329:
20325:
20321:
20318:
20315:
20312:
20306:
20302:
20299:
20296:
20293:
20290:
20285:
20281:
20277:
20272:
20268:
20264:
20261:
20258:
20255:
20250:
20246:
20222:
20219:
20216:
20213:
20210:
20207:
20187:
20184:
20181:
20176:
20172:
20168:
20165:
20162:
20159:
20139:
20136:
20133:
20128:
20124:
20120:
20115:
20111:
20107:
20104:
20101:
20098:
20072:
20068:
20064:
20059:
20055:
20031:
20028:
20025:
20010:
20009:
19997:
19991:
19988:
19985:
19982:
19979:
19976:
19971:
19968:
19965:
19962:
19959:
19956:
19953:
19950:
19944:
19940:
19937:
19934:
19931:
19928:
19925:
19922:
19919:
19916:
19913:
19908:
19904:
19900:
19895:
19890:
19887:
19884:
19881:
19878:
19875:
19872:
19869:
19866:
19863:
19860:
19857:
19854:
19851:
19848:
19843:
19832:
19808:
19805:
19802:
19799:
19796:
19793:
19790:
19787:
19767:
19764:
19761:
19758:
19755:
19752:
19732:
19729:
19726:
19723:
19720:
19717:
19714:
19711:
19707:
19695:
19694:
19683:
19680:
19677:
19674:
19671:
19668:
19665:
19662:
19659:
19656:
19653:
19650:
19647:
19644:
19641:
19638:
19635:
19632:
19629:
19624:
19620:
19616:
19613:
19608:
19603:
19600:
19597:
19594:
19591:
19588:
19585:
19582:
19577:
19571:
19552:
19551:
19537:
19534:
19531:
19528:
19525:
19522:
19517:
19514:
19511:
19508:
19505:
19502:
19499:
19496:
19493:
19490:
19487:
19484:
19481:
19478:
19472:
19469:
19466:
19463:
19460:
19457:
19454:
19451:
19448:
19434:Bayes' theorem
19421:
19418:
19415:
19412:
19409:
19406:
19403:
19400:
19380:
19377:
19374:
19371:
19368:
19365:
19341:
19338:
19335:
19315:
19312:
19309:
19306:
19303:
19300:
19297:
19294:
19291:
19288:
19285:
19261:
19258:
19237:
19234:
19231:
19227:
19208:
19207:
19196:
19193:
19190:
19187:
19184:
19181:
19172:
19168:
19165:
19162:
19159:
19155:
19151:
19148:
19145:
19142:
19139:
19136:
19133:
19130:
19127:
19124:
19119:
19115:
19111:
19108:
19105:
19102:
19099:
19096:
19092:
19075:over the same
19005:
19002:
18999:
18995:
18971:Huffman coding
18946:
18943:
18922:
18919:
18915:
18911:
18908:
18905:
18885:
18882:
18879:
18874:
18870:
18841:
18838:
18835:
18832:
18829:
18826:
18806:
18803:
18800:
18797:
18794:
18791:
18788:
18783:
18779:
18755:
18754:
18738:
18733:
18729:
18726:
18723:
18718:
18714:
18710:
18706:
18702:
18699:
18696:
18692:
18688:
18684:
18674:
18669:
18665:
18660:
18656:
18652:
18649:
18646:
18643:
18640:
18637:
18634:
18631:
18629:
18627:
18624:
18621:
18618:
18615:
18612:
18609:
18606:
18603:
18600:
18597:
18594:
18590:
18586:
18583:
18581:
18579:
18576:
18573:
18570:
18567:
18562:
18558:
18554:
18551:
18548:
18545:
18542:
18539:
18530:
18526:
18523:
18520:
18517:
18514:
18511:
18508:
18505:
18502:
18499:
18496:
18493:
18490:
18487:
18484:
18481:
18478:
18475:
18466:
18462:
18459:
18456:
18453:
18450:
18447:
18444:
18441:
18439:
18437:
18434:
18431:
18428:
18425:
18422:
18419:
18416:
18413:
18408:
18404:
18400:
18397:
18394:
18391:
18388:
18385:
18382:
18379:
18370:
18366:
18363:
18360:
18357:
18354:
18351:
18348:
18345:
18343:
18341:
18338:
18335:
18332:
18329:
18325:
18321:
18320:
18301:
18298:
18297:
18296:
18285:
18282:
18279:
18276:
18273:
18269:
18264:
18260:
18257:
18254:
18251:
18248:
18239:
18235:
18232:
18229:
18226:
18223:
18220:
18206:
18205:
18194:
18191:
18188:
18181:
18178:
18175:
18172:
18167:
18164:
18161:
18158:
18152:
18149:
18146:
18143:
18140:
18137:
18134:
18131:
18128:
18125:
18122:
18119:
18116:
18113:
18110:
18107:
18104:
18099:
18095:
18089:
18086:
18083:
18079:
18043:
18040:
18037:
18034:
18014:
18011:
18008:
18003:
17999:
17970:
17967:
17964:
17961:
17941:
17938:
17935:
17930:
17926:
17894:
17893:
17877:
17873:
17870:
17867:
17862:
17858:
17854:
17851:
17848:
17845:
17840:
17836:
17831:
17821:
17817:
17814:
17811:
17808:
17805:
17802:
17799:
17796:
17794:
17792:
17788:
17784:
17781:
17778:
17775:
17770:
17766:
17761:
17757:
17754:
17751:
17748:
17746:
17744:
17741:
17738:
17734:
17730:
17729:
17710:
17707:
17683:
17680:
17677:
17674:
17671:
17668:
17636:
17633:
17630:
17627:
17624:
17621:
17618:
17615:
17595:
17592:
17589:
17586:
17583:
17580:
17566:
17565:
17550:
17547:
17544:
17541:
17538:
17535:
17532:
17529:
17526:
17523:
17520:
17517:
17514:
17511:
17502:
17498:
17495:
17490:
17486:
17482:
17479:
17477:
17475:
17472:
17469:
17466:
17463:
17460:
17457:
17454:
17451:
17448:
17445:
17442:
17439:
17436:
17433:
17424:
17420:
17417:
17412:
17408:
17404:
17401:
17399:
17397:
17394:
17391:
17388:
17385:
17382:
17379:
17376:
17373:
17370:
17367:
17364:
17361:
17358:
17355:
17352:
17349:
17346:
17337:
17333:
17330:
17328:
17326:
17323:
17320:
17317:
17314:
17311:
17308:
17305:
17304:
17285:
17282:
17269:
17266:
17263:
17243:
17240:
17237:
17234:
17210:
17207:
17204:
17180:
17177:
17174:
17171:
17160:
17159:
17148:
17144:
17140:
17135:
17131:
17127:
17124:
17115:
17110:
17100:
17096:
17093:
17090:
17087:
17084:
17080:
17038:Main article:
17035:
17032:
17027:
17024:
17018:of a discrete
17003:
17000:
16999:
16998:
16987:
16984:
16981:
16976: as
16972:
16967:
16963:
16959:
16953:
16948:
16943:
16937:
16933:
16929:
16923:
16920:
16917:
16914:
16911:
16908:
16905:
16902:
16899:
16896:
16893:
16890:
16887:
16884:
16881:
16878:
16873:
16857:
16856:
16845:
16842:
16839:
16836:
16830:
16827:
16824:
16819:
16810:
16806:
16802:
16797:
16794:
16791:
16788:
16785:
16782:
16777:
16773:
16764:
16759:
16756:
16753:
16749:
16744:
16740:
16737:
16734:
16728:
16725:
16722:
16717:
16710:
16707:
16702:
16699:
16696:
16693:
16690:
16687:
16684:
16676:
16671:
16668:
16665:
16661:
16656:
16634:
16633:
16618:
16615:
16612:
16609:
16606:
16603:
16599:
16596:
16593:
16590:
16587:
16584:
16581:
16578:
16575:
16572:
16567:
16562:
16559:
16556:
16552:
16548:
16545:
16542:
16539:
16536:
16533:
16530:
16528:
16525:
16517:
16513:
16509:
16504:
16501:
16498:
16495:
16492:
16487:
16483:
16476:
16472:
16471:
16468:
16465:
16462:
16459:
16455:
16452:
16449:
16446:
16441:
16436:
16433:
16430:
16426:
16422:
16419:
16416:
16413:
16410:
16407:
16404:
16402:
16399:
16391:
16387:
16383:
16378:
16373:
16369:
16362:
16358:
16357:
16354:
16351:
16348:
16345:
16342:
16338:
16335:
16332:
16329:
16324:
16319:
16316:
16313:
16309:
16305:
16302:
16299:
16296:
16293:
16290:
16287:
16285:
16282:
16276:
16273:
16268:
16265:
16259:
16255:
16254:
16244:
16227:
16223:
16219:
16214:
16211:
16208:
16205:
16202:
16197:
16193:
16186:
16178:
16174:
16170:
16165:
16162:
16159:
16156:
16153:
16148:
16144:
16137:
16131:
16128:
16123:
16120:
16117:
16114:
16111:
16108:
16080:
16077:
16074:
16071:
16068:
16064:
16051:
16048:
16027:
16024:
16021:
16016:
16013:
16009:
15986:
15982:
15966:
15965:
15954:
15951:
15946:
15942:
15938:
15935:
15932:
15929:
15926:
15923:
15920:
15917:
15908:
15900:
15896:
15892:
15889:
15884:
15876:
15872:
15868:
15862:
15858:
15854:
15848:
15844:
15838:
15834:
15831:
15826:
15822:
15818:
15813:
15810:
15806:
15791:Hessian matrix
15787:
15786:
15775:
15772:
15769:
15764:
15760:
15756:
15751:
15748:
15744:
15737:
15733:
15729:
15723:
15719:
15715:
15709:
15706:
15701:
15698:
15695:
15690:
15686:
15682:
15679:
15676:
15673:
15670:
15667:
15664:
15661:
15652:
15633:
15632:
15621:
15618:
15615:
15612:
15609:
15604:
15600:
15596:
15593:
15590:
15587:
15584:
15581:
15578:
15575:
15566:
15558:
15554:
15550:
15547:
15542:
15534:
15530:
15526:
15522:
15517:
15491:
15487:
15483:
15457:
15453:
15449:
15446:
15426:
15423:
15420:
15400:
15395:
15391:
15387:
15379:
15375:
15371:
15366:
15363:
15357:
15353:
15348:
15344:
15340:
15334:
15330:
15305:
15283:
15279:
15273:
15269:
15265:
15262:
15259:
15256:
15251:
15247:
15243:
15232:
15231:
15220:
15217:
15214:
15209:
15205:
15201:
15196:
15192:
15185:
15181:
15177:
15174:
15171:
15166:
15162:
15158:
15155:
15152:
15149:
15146:
15143:
15140:
15111:
15107:
15086:
15066:
15061:
15057:
15053:
15050:
15047:
15044:
15024:
15021:
15018:
15015:
15012:
15009:
14989:
14965:
14962:
14954:
14953:
14941:
14936:
14933:
14929:
14923:
14919:
14915:
14912:
14907:
14903:
14897:
14893:
14875:
14874:
14862:
14856:
14852:
14845:
14841:
14826:
14825:
14813:
14810:
14807:
14804:
14801:
14796:
14792:
14788:
14779:
14773:
14770:
14767:
14763:
14739:
14736:
14733:
14728:
14724:
14720:
14715:
14711:
14707:
14672:
14669:
14666:
14663:
14660:
14651:
14630:
14627:
14624:
14621:
14618:
14609:
14575:, it is not a
14568:
14565:
14548:
14545:
14540:
14536:
14515:
14512:
14507:
14503:
14473:
14470:
14467:
14462:
14459:
14456:
14450:
14447:
14444:
14440:
14434:
14429:
14424:
14418:
14408:
14385:
14382:
14379:
14376:
14373:
14370:
14367:
14347:
14344:
14341:
14338:
14335:
14332:
14329:
14309:
14306:
14303:
14300:
14297:
14294:
14291:
14279:
14276:
14263:
14260:
14257:
14254:
14250:
14247:
14244:
14241:
14238:
14234:
14230:
14226:
14222:
14219:
14216:
14211:
14208:
14204:
14200:
14197:
14194:
14191:
14186:
14182:
14178:
14174:
14168:
14163:
14158:
14152:
14142:
14117:
14113:
14108:
14102:
14098:
14094:
14091:
14080:
14079:
14066:
14063:
14058:
14050:
14045:
14041:
14037:
14030:
14026:
14020:
14016:
14012:
14007:
14003:
13999:
13996:
13991:
13986:
13982:
13975:
13968:
13964:
13958:
13954:
13948:
13945:
13942:
13938:
13932:
13927:
13922:
13916:
13906:
13879:
13878:
13867:
13863:
13858:
13853:
13848:
13844:
13840:
13836:
13833:
13830:
13827:
13824:
13819:
13814:
13810:
13806:
13801:
13796:
13792:
13787:
13781:
13776:
13773:
13770:
13766:
13760:
13757:
13752:
13748:
13743:
13738:
13734:
13730:
13725:
13719:
13714:
13710:
13705:
13699:
13694:
13690:
13686:
13683:
13680:
13675:
13670:
13666:
13661:
13657:
13654:
13651:
13645:
13639:
13633:
13629:
13625:
13622:
13619:
13614:
13610:
13605:
13599:
13593:
13587:
13577:
13558:
13557:
13546:
13542:
13533:
13530:
13526:
13520:
13516:
13512:
13505:
13502:
13498:
13492:
13488:
13484:
13478:
13475:
13470:
13465:
13462:
13459:
13455:
13451:
13448:
13443:
13438:
13433:
13429:
13425:
13422:
13419:
13414:
13410:
13404:
13401:
13397:
13393:
13388:
13383:
13380:
13377:
13374:
13371:
13367:
13362:
13356:
13353:
13348:
13344:
13338:
13332:
13326:
13321:
13315:
13308:
13298:
13272:
13268:
13264:
13259:
13255:
13251:
13248:
13243:
13239:
13216:
13212:
13208:
13205:
13200:
13196:
13165:
13160:
13156:
13150:
13146:
13142:
13137:
13133:
13110:
13105:
13101:
13095:
13091:
13087:
13082:
13078:
13055:
13051:
13047:
13042:
13038:
13010:
13007:
13004:
13001:
12998:
12969:
12968:
12957:
12953:
12948:
12940:
12936:
12932:
12925:
12921:
12917:
12911:
12907:
12904:
12901:
12897:
12891:
12887:
12883:
12878:
12874:
12869:
12863:
12860:
12855:
12851:
12844:
12838:
12832:
12828:
12824:
12819:
12815:
12810:
12805:
12802:
12799:
12795:
12789:
12785:
12779:
12776:
12771:
12767:
12762:
12758:
12755:
12751:
12745:
12742:
12737:
12733:
12727:
12721:
12715:
12710:
12704:
12697:
12687:
12659:
12654:
12650:
12646:
12641:
12637:
12611:
12607:
12603:
12598:
12594:
12577:
12574:
12572:
12569:
12565:measure theory
12546:
12542:
12538:
12535:
12515:
12512:
12509:
12504:
12500:
12496:
12493:
12490:
12481:
12469:
12468:
12457:
12454:
12451:
12448:
12445:
12442:
12437:
12433:
12429:
12426:
12423:
12420:
12415:
12411:
12407:
12404:
12401:
12392:
12388:
12385:
12382:
12379:
12376:
12373:
12368:
12364:
12360:
12357:
12354:
12351:
12348:
12345:
12342:
12339:
12330:
12326:
12323:
12320:
12317:
12312:
12308:
12293:
12292:
12281:
12278:
12275:
12272:
12269:
12266:
12261:
12257:
12253:
12250:
12247:
12244:
12241:
12238:
12233:
12229:
12225:
12222:
12219:
12216:
12213:
12210:
12207:
12198:
12194:
12191:
12186:
12182:
12178:
12175:
12172:
12163:
12139:
12136:
12133:
12130:
12127:
12121:
12118:
12115:
12112:
12109:
12104:
12100:
12094:
12091:
12088:
12085:
12082:
12079:
12073:
12070:
12067:
12064:
12061:
12056:
12052:
12028:
12025:
12022:
12019:
12016:
12011:
12007:
12001:
11998:
11995:
11992:
11989:
11986:
11957:
11953:
11928:
11925:
11922:
11919:
11916:
11902:
11877:
11874:
11871:
11868:
11865:
11860:
11857:
11854:
11851:
11848:
11830:
11829:
11818:
11812:
11809:
11806:
11803:
11800:
11795:
11791:
11785:
11782:
11779:
11776:
11773:
11770:
11764:
11758:
11755:
11752:
11749:
11746:
11741:
11738:
11735:
11732:
11729:
11712:
11711:
11700:
11697:
11694:
11691:
11688:
11685:
11682:
11673:
11669:
11666:
11663:
11660:
11655:
11651:
11647:
11642:
11639:
11636:
11626:
11623:
11620:
11617:
11614:
11611:
11606:
11602:
11598:
11595:
11572:
11569:
11566:
11561:
11556:
11553:
11550:
11514:
11511:
11508:
11488:
11485:
11482:
11462:
11459:
11456:
11451:
11446:
11443:
11440:
11420:
11417:
11414:
11409:
11404:
11401:
11398:
11365:
11343:
11323:
11304:
11295:
11292:
11274:
11271:
11268:
11246:
11241:
11235:
11232:
11227:
11224:
11220:
11215:
11209:
11206:
11203:
11200:
11197:
11194:
11190:
11168:
11148:
11145:
11142:
11139:
11136:
11133:
11113:
11093:
11090:
11087:
11084:
11064:
11061:
11058:
11038:
11018:
11015:
11012:
10992:
10989:
10986:
10983:
10980:
10977:
10957:
10954:
10951:
10931:
10928:
10925:
10905:
10902:
10899:
10879:
10859:
10856:
10853:
10850:
10830:
10827:
10824:
10821:
10798:
10795:
10792:
10790:
10788:
10782:
10779:
10776:
10773:
10770:
10767:
10763:
10756:
10751:
10748:
10745:
10741:
10737:
10734:
10732:
10730:
10726:
10722:
10719:
10716:
10713:
10710:
10707:
10704:
10701:
10698:
10694:
10686:
10681:
10678:
10674:
10667:
10664:
10661:
10658:
10655:
10652:
10648:
10641:
10636:
10633:
10630:
10626:
10622:
10619:
10617:
10615:
10610:
10607:
10604:
10599:
10592:
10589:
10586:
10583:
10578:
10575:
10572:
10569:
10563:
10560:
10556:
10550:
10546:
10543:
10540:
10537:
10534:
10531:
10528:
10525:
10522:
10518:
10510:
10505:
10502:
10498:
10491:
10488:
10485:
10482:
10479:
10476:
10472:
10465:
10460:
10457:
10454:
10450:
10446:
10443:
10441:
10438:
10429:
10426:
10423:
10419:
10415:
10412:
10409:
10402:
10398:
10394:
10391:
10388:
10385:
10382:
10379:
10376:
10373:
10370:
10367:
10357:
10352:
10349:
10345:
10338:
10335:
10332:
10329:
10326:
10323:
10319:
10313:
10307:
10302:
10299:
10296:
10292:
10288:
10287:
10267:
10264:
10261:
10258:
10230:
10227:
10224:
10220:
10216:
10213:
10210:
10203:
10199:
10195:
10192:
10189:
10186:
10183:
10180:
10177:
10174:
10171:
10168:
10158:
10153:
10150:
10146:
10139:
10136:
10133:
10130:
10127:
10124:
10120:
10113:
10108:
10105:
10102:
10098:
10094:
10091:
10089:
10087:
10081:
10078:
10073:
10070:
10067:
10062:
10059:
10056:
10052:
10043:
10038:
10035:
10032:
10028:
10024:
10021:
10019:
10017:
10014:
10011:
10008:
10005:
9996:
9992:
9991:
9971:
9968:
9965:
9945:
9925:
9905:
9902:
9899:
9896:
9893:
9884:
9854:
9851:
9848:
9844:
9840:
9837:
9834:
9827:
9823:
9819:
9816:
9813:
9810:
9807:
9804:
9801:
9798:
9795:
9792:
9782:
9777:
9774:
9770:
9766:
9763:
9760:
9757:
9754:
9751:
9746:
9742:
9738:
9735:
9732:
9729:
9726:
9724:
9722:
9719:
9716:
9711:
9708:
9705:
9701:
9697:
9696:
9689:
9686:
9683:
9678:
9674:
9671:
9668:
9665:
9662:
9659:
9656:
9653:
9650:
9647:
9644:
9641:
9638:
9635:
9632:
9628:
9620:
9616:
9612:
9609:
9606:
9603:
9600:
9597:
9594:
9591:
9588:
9585:
9575:
9570:
9567:
9563:
9559:
9556:
9553:
9550:
9547:
9544:
9539:
9535:
9531:
9528:
9525:
9522:
9519:
9517:
9515:
9512:
9509:
9504:
9501:
9498:
9494:
9490:
9489:
9483:
9480:
9477:
9474:
9467:
9463:
9459:
9456:
9453:
9450:
9447:
9444:
9441:
9438:
9435:
9432:
9422:
9417:
9414:
9410:
9406:
9403:
9401:
9399:
9396:
9393:
9389:
9386:
9382:
9381:
9375:
9372:
9369:
9366:
9363:
9360:
9357:
9354:
9351:
9348:
9345:
9342:
9339:
9336:
9333:
9326:
9322:
9318:
9315:
9312:
9309:
9306:
9303:
9300:
9297:
9294:
9291:
9281:
9276:
9273:
9269:
9265:
9262:
9260:
9258:
9255:
9252:
9248:
9245:
9241:
9240:
9216:
9213:
9210:
9208:
9206:
9203:
9200:
9196:
9193:
9189:
9188:
9184:
9178:
9175:
9172:
9169:
9164:
9161:
9158:
9155:
9152:
9149:
9146:
9143:
9140:
9137:
9134:
9131:
9128:
9125:
9122:
9116:
9112:
9109:
9106:
9103:
9100:
9097:
9094:
9091:
9088:
9085:
9082:
9079:
9076:
9069:
9064:
9061:
9057:
9053:
9050:
9048:
9046:
9042:
9038:
9035:
9031:
9025:
9022:
9019:
9016:
9011:
9008:
9005:
9002:
8999:
8996:
8993:
8990:
8987:
8984:
8981:
8978:
8975:
8972:
8969:
8963:
8959:
8956:
8952:
8948:
8945:
8942:
8939:
8936:
8933:
8930:
8927:
8924:
8921:
8918:
8911:
8906:
8903:
8899:
8895:
8892:
8890:
8888:
8885:
8882:
8878:
8875:
8871:
8870:
8850:
8830:
8827:
8824:
8821:
8818:
8815:
8812:
8809:
8806:
8803:
8794:
8784:and note that
8773:
8770:
8767:
8764:
8761:
8758:
8755:
8752:
8749:
8746:
8743:
8740:
8737:
8728:
8724:
8721:
8718:
8715:
8712:
8701:
8700:
8699:
8698:
8686:
8666:
8663:
8660:
8657:
8632:
8629:
8626:
8622:
8618:
8615:
8612:
8605:
8601:
8597:
8594:
8591:
8588:
8585:
8582:
8579:
8576:
8573:
8570:
8560:
8555:
8552:
8548:
8541:
8538:
8535:
8532:
8529:
8526:
8522:
8515:
8510:
8507:
8504:
8500:
8496:
8493:
8490:
8487:
8484:
8481:
8472:
8451:
8448:
8445:
8425:
8422:
8419:
8416:
8413:
8404:
8393:
8382:
8379:
8376:
8373:
8370:
8362:
8357:
8353:
8349:
8344:
8340:
8336:
8327:
8323:
8320:
8317:
8314:
8311:
8308:
8305:
8300:
8296:
8292:
8287:
8283:
8279:
8270:
8266:
8263:
8260:
8255:
8251:
8247:
8244:
8241:
8238:
8235:
8232:
8227:
8223:
8219:
8216:
8211:
8207:
8203:
8200:
8197:
8194:
8191:
8188:
8183:
8179:
8175:
8172:
8163:
8142:
8137:
8133:
8129:
8124:
8120:
8116:
8096:
8091:
8087:
8083:
8078:
8074:
8070:
8050:
8047:
8044:
8041:
8038:
8011:
8008:
8005:
8002:
7999:
7990:
7978:
7967:
7964:
7959:
7955:
7951:
7946:
7942:
7938:
7929:
7925:
7922:
7917:
7913:
7909:
7904:
7900:
7896:
7887:
7883:
7880:
7877:
7874:
7871:
7868:
7859:
7836:
7832:
7828:
7823:
7819:
7798:
7795:
7792:
7789:
7784:
7780:
7776:
7773:
7770:
7767:
7762:
7758:
7754:
7751:
7748:
7745:
7742:
7739:
7736:
7733:
7730:
7710:
7707:
7704:
7701:
7696:
7692:
7688:
7685:
7682:
7679:
7674:
7670:
7666:
7663:
7660:
7657:
7654:
7651:
7648:
7645:
7642:
7620:
7616:
7612:
7607:
7603:
7583:
7562:
7559:
7555:
7552:
7549:
7546:
7543:
7540:
7537:
7534:
7531:
7528:
7508:
7505:
7502:
7499:
7479:
7476:
7473:
7470:
7442:
7437:
7433:
7429:
7426:
7423:
7418:
7414:
7393:
7388:
7384:
7380:
7377:
7374:
7369:
7365:
7340:
7337:
7332:
7326:
7323:
7320:
7314:
7311:
7303:
7300:
7297:
7291:
7288:
7279:
7275:
7272:
7269:
7266:
7263:
7257:
7254:
7244:
7240:
7232:
7228:
7223:
7219:
7216:
7214:
7212:
7209:
7206:
7201:
7194:
7190:
7187:
7184:
7178:
7175:
7170:
7167:
7160:
7155:
7152:
7149:
7146:
7143:
7140:
7134:
7131:
7122:
7118:
7115:
7112:
7106:
7103:
7098:
7095:
7088:
7083:
7080:
7077:
7074:
7071:
7068:
7062:
7059:
7050:
7046:
7043:
7039:
7035:
7032:
7029:
7023:
7020:
7015:
7012:
7005:
7001:
6998:
6995:
6992:
6989:
6986:
6980:
6977:
6967:
6963:
6955:
6951:
6946:
6942:
6939:
6937:
6935:
6932:
6929:
6924:
6918:
6915:
6912:
6909:
6904:
6901:
6898:
6895:
6889:
6885:
6882:
6879:
6876:
6873:
6870:
6863:
6859:
6851:
6847:
6842:
6838:
6835:
6833:
6831:
6828:
6825:
6822:
6819:
6810:
6806:
6805:
6780:
6776:
6773:
6770:
6763:
6760:
6755:
6752:
6744:
6723:
6720:
6716:
6712:
6709:
6706:
6699:
6696:
6691:
6688:
6680:
6676:
6673:
6670:
6664:
6661:
6655:
6652:
6649:
6645:
6642:
6639:
6633:
6630:
6624:
6621:
6618:
6614:
6611:
6608:
6605:
6602:
6599:
6596:
6593:
6590:
6587:
6567:
6564:
6559:
6555:
6552:
6549:
6542:
6539:
6534:
6531:
6523:
6519:
6516:
6513:
6510:
6507:
6504:
6498:
6495:
6489:
6486:
6483:
6479:
6476:
6473:
6467:
6464:
6458:
6455:
6452:
6448:
6445:
6442:
6439:
6436:
6433:
6430:
6427:
6424:
6421:
6412:, then, since
6401:
6398:
6395:
6392:
6373:
6361:
6358:
6355:
6352:
6349:
6340:
6275:
6272:
6269:
6266:
6263:
6259:
6238:
6235:
6232:
6228:
6218:. The entropy
6203:
6184:
6181:
6178:
6175:
6172:
6169:
6166:
6163:
6160:
6140:
6137:
6134:
6131:
6128:
6125:
6122:
6119:
6116:
6113:
6110:
6107:
6104:
6101:
6098:
6078:
6075:
6072:
6069:
6066:
6063:
6060:
6057:
6054:
6051:
6048:
6045:
6042:
6039:
6036:
6025:
6024:
6012:
6009:
6006:
5996:if and only if
5983:
5980:
5977:
5974:
5971:
5962:
5937:
5934:
5931:
5928:
5925:
5922:
5919:
5916:
5907:
5889:
5886:
5834:
5831:
5828:
5825:
5822:
5813:
5792:
5789:
5786:
5783:
5763:
5760:
5757:
5754:
5751:
5748:
5745:
5742:
5739:
5736:
5733:
5709:
5706:
5703:
5700:
5697:
5694:
5658:
5655:
5652:
5649:
5646:
5637:
5605:
5602:
5599:
5595:
5562:
5559:
5556:
5553:
5550:
5546:
5534:
5533:
5518:
5515:
5512:
5508:
5504:
5501:
5498:
5495:
5492:
5489:
5485:
5481:
5478:
5476:
5474:
5468:
5465:
5462:
5459:
5455:
5450:
5447:
5444:
5441:
5438:
5435:
5428:
5423:
5420:
5416:
5412:
5406:
5403:
5400:
5397:
5393:
5388:
5385:
5382:
5379:
5376:
5373:
5366:
5361:
5358:
5354:
5350:
5347:
5345:
5343:
5340:
5337:
5334:
5331:
5322:
5318:
5317:
5280:
5276:
5253:
5249:
5220:
5216:
5212:
5208:
5204:
5201:
5196:
5192:
5188:
5185:
5159:
5155:
5126:
5123:
5103:
5086:
5083:
5012:
5009:
5006:
5003:
5000:
4991:
4970:
4967:
4964:
4961:
4958:
4949:
4915:, it is not a
4911:While it is a
4877:
4874:
4839:
4836:
4833:
4830:
4827:
4818:
4766:
4763:
4760:
4757:
4754:
4745:
4728:
4725:
4692:
4689:
4686:
4683:
4680:
4671:
4654:
4651:
4630:
4627:
4624:
4621:
4618:
4615:
4612:
4609:
4606:
4603:
4600:
4597:
4594:
4566:
4563:
4561:
4558:
4557:
4556:
4541:
4538:
4534:
4530:
4527:
4524:
4521:
4518:
4515:
4512:
4509:
4506:
4503:
4500:
4497:
4494:
4491:
4488:
4485:
4482:
4479:
4476:
4473:
4470:
4466:
4460:
4457:
4452:
4449:
4447:
4445:
4441:
4435:
4431:
4427:
4422:
4418:
4414:
4408:
4404:
4401:
4396:
4393:
4388:
4384:
4378:
4374:
4370:
4365:
4361:
4357:
4351:
4347:
4344:
4339:
4336:
4331:
4327:
4321:
4317:
4313:
4308:
4304:
4300:
4294:
4290:
4287:
4282:
4279:
4274:
4271:
4269:
4267:
4263:
4257:
4254:
4251:
4248:
4243:
4240:
4237:
4234:
4228:
4224:
4221:
4218:
4215:
4212:
4209:
4202:
4197:
4194:
4190:
4186:
4183:
4181:
4179:
4176:
4173:
4170:
4167:
4158:
4154:
4153:
4142:
4141:
4126:
4123:
4120:
4116:
4112:
4109:
4106:
4103:
4100:
4097:
4094:
4091:
4088:
4085:
4082:
4079:
4076:
4073:
4070:
4067:
4064:
4061:
4058:
4055:
4051:
4045:
4042:
4037:
4034:
4032:
4030:
4026:
4020:
4016:
4012:
4007:
4003:
3999:
3993:
3989:
3986:
3981:
3978:
3973:
3969:
3963:
3959:
3955:
3950:
3946:
3942:
3936:
3932:
3929:
3924:
3921:
3916:
3912:
3906:
3902:
3898:
3893:
3889:
3885:
3879:
3875:
3872:
3867:
3864:
3859:
3856:
3854:
3852:
3848:
3842:
3839:
3836:
3833:
3828:
3825:
3822:
3819:
3813:
3809:
3806:
3803:
3800:
3797:
3794:
3787:
3782:
3779:
3775:
3771:
3768:
3766:
3764:
3761:
3758:
3755:
3752:
3743:
3739:
3738:
3693:
3690:
3687:
3684:
3681:
3672:
3651:
3648:
3645:
3642:
3639:
3630:
3616:
3615:
3602:
3599:
3587:
3574:
3571:
3559:
3546:
3543:
3531:
3520:
3517:
3514:
3511:
3499:
3498:
3485:
3482:
3470:
3457:
3454:
3442:
3429:
3426:
3414:
3403:
3400:
3397:
3394:
3382:
3381:
3378:
3375:
3372:
3344:
3340:
3336:
3333:
3330:
3310:
3307:
3304:
3301:
3298:
3295:
3292:
3289:
3284:
3251:
3248:
3224:
3221:
3218:
3198:
3195:
3192:
3164:
3161:
3123:
3120:
3117:
3114:
3111:
3102:
3011:
3008:
3005:
3002:
2999:
2990:
2919:
2915:
2912:
2909:
2905:
2899:
2896:
2891:
2888:
2865:
2854:
2853:
2842:
2836:
2833:
2825:
2821:
2818:
2811:
2805:
2802:
2799:
2796:
2788:
2785:
2782:
2779:
2770:
2766:
2763:
2757:
2754:
2751:
2748:
2741:
2736:
2733:
2729:
2725:
2722:
2719:
2716:
2713:
2710:
2701:
2666:
2632:
2629:
2621:
2617:
2614:
2611:
2608:
2605:
2602:
2599:
2596:
2593:
2585:
2581:
2578:
2552:
2549:
2541:
2537:
2534:
2531:
2528:
2525:
2522:
2519:
2516:
2513:
2505:
2501:
2498:
2465:
2443:
2420:
2419:
2408:
2402:
2399:
2391:
2387:
2384:
2377:
2371:
2368:
2360:
2356:
2353:
2348:
2345:
2337:
2333:
2330:
2324:
2320:
2317:
2308:
2305:
2297:
2293:
2290:
2285:
2282:
2274:
2270:
2267:
2257:
2252:
2249:
2245:
2241:
2238:
2235:
2232:
2229:
2226:
2217:
2178:
2175:
2167:
2163:
2160:
2157:
2154:
2151:
2148:
2145:
2142:
2139:
2131:
2127:
2124:
2096:
2045:
2042:
2034:
2030:
2027:
2019:
2016:
2008:
2004:
2001:
1981:
1980:
1969:
1963:
1960:
1952:
1948:
1945:
1938:
1932:
1929:
1921:
1917:
1914:
1909:
1906:
1898:
1894:
1891:
1885:
1881:
1878:
1868:
1863:
1860:
1856:
1852:
1849:
1846:
1843:
1840:
1837:
1828:
1815:is defined as
1784:
1776:
1722:
1721:
1710:
1704:
1696:
1688:
1682:
1679:
1676:
1673:
1668:
1665:
1662:
1659:
1653:
1649:
1646:
1640:
1637:
1634:
1631:
1626:
1621:
1618:
1614:
1610:
1607:
1604:
1601:
1598:
1595:
1586:
1559:
1558:
1547:
1541:
1538:
1535:
1532:
1529:
1526:
1523:
1520:
1513:
1509:
1505:
1502:
1498:
1471:
1468:
1465:
1462:
1431:
1403:
1400:
1397:
1394:
1391:
1388:
1362:
1359:
1347:but the value
1334:
1331:
1303:
1300:
1297:
1294:
1291:
1288:
1262:
1259:
1256:
1253:
1250:
1247:
1210:
1209:
1198:
1191:
1185:
1182:
1179:
1176:
1168:
1165:
1162:
1159:
1150:
1146:
1143:
1137:
1134:
1131:
1128:
1121:
1116:
1113:
1109:
1105:
1102:
1099:
1096:
1093:
1090:
1087:
1078:
1063:
1062:
1051:
1044:
1038:
1035:
1032:
1029:
1021:
1018:
1015:
1012:
1003:
999:
996:
990:
987:
984:
981:
974:
969:
966:
962:
958:
955:
952:
949:
946:
943:
934:
902:
894:
862:
859:
851:Kullback (1959
822:
819:
816:
813:
810:
807:
804:
801:
798:
795:
792:
789:
786:
783:
780:
777:
774:
771:
768:
765:
743:
739:
716:
712:
691:
688:
685:
682:
679:
676:
654:
650:
646:
641:
637:
614:
610:
606:
601:
597:
574:
570:
566:
561:
557:
534:
530:
507:
503:
480:
476:
451:
448:
446:computations.
407:
404:
401:
398:
395:
386:
353:
350:
342:bioinformatics
254:interpretation
250:
249:
238:
234:
228:
225:
222:
219:
211:
208:
205:
202:
193:
189:
186:
180:
177:
174:
171:
164:
159:
156:
152:
148:
145:
142:
139:
136:
133:
124:
85:
82:
79:
76:
73:
64:
26:
9:
6:
4:
3:
2:
26714:
26703:
26700:
26698:
26695:
26693:
26692:F-divergences
26690:
26688:
26685:
26684:
26682:
26673:
26670:
26667:
26663:
26659:
26656:
26654:
26651:
26649:
26646:
26644:
26641:
26639:
26636:
26635:
26624:
26619:
26616:
26615:0-8446-5625-9
26612:
26608:
26603:
26599:
26594:
26590:
26584:
26580:
26576:
26572:
26568:
26564:
26563:
26555:
26551:
26547:
26543:
26540:(2): 99–121.
26539:
26535:
26534:
26529:
26528:Guibas, L. J.
26522:
26513:
26507:
26499:
26495:
26491:
26485:
26481:
26480:
26472:
26458:
26454:
26450:
26444:
26440:
26436:
26432:
26431:
26423:
26415:
26411:
26406:
26401:
26396:
26391:
26387:
26383:
26378:
26373:
26369:
26365:
26364:
26359:
26352:
26344:
26340:
26335:
26330:
26325:
26320:
26316:
26312:
26307:
26302:
26298:
26294:
26293:
26288:
26281:
26273:
26269:
26265:
26259:
26255:
26254:
26246:
26237:
26232:
26228:
26224:
26220:
26213:
26205:
26201:
26197:
26193:
26189:
26185:
26181:
26176:
26171:
26167:
26163:
26159:
26152:
26144:
26140:
26136:
26132:
26128:
26124:
26117:
26108:
26107:
26099:
26091:
26087:
26083:
26079:
26075:
26071:
26064:
26057:
26049:
26045:
26041:
26037:
26033:
26029:
26025:
26021:
26014:
26007:
25999:
25998:
25990:
25982:
25976:
25972:
25971:
25966:
25959:
25950:
25945:
25940:
25935:
25931:
25927:
25926:
25921:
25914:
25906:
25902:
25895:
25893:
25884:
25880:
25876:
25875:
25867:
25865:
25856:
25852:
25848:
25844:
25840:
25836:
25832:
25828:
25824:
25817:
25810:. p. 13.
25806:
25799:
25791:
25787:
25783:
25779:
25774:
25769:
25765:
25761:
25754:
25746:
25742:
25738:
25734:
25727:
25720:
25717:
25713:
25707:
25698:
25693:
25689:
25683:
25675:
25669:
25665:
25658:
25651:
25647:
25643:
25639:
25633:
25625:
25621:
25614:
25606:
25602:
25598:
25594:
25587:
25579:
25575:
25570:
25565:
25560:
25555:
25551:
25547:
25542:
25537:
25533:
25529:
25525:
25518:
25510:
25506:
25502:
25498:
25491:
25483:
25481:9780387953649
25477:
25473:
25468:
25467:
25458:
25451:
25450:Kullback 1959
25446:
25438:
25434:
25430:
25429:
25421:
25413:
25409:
25403:
25395:
25391:
25385:
25377:
25375:9780521642989
25371:
25367:
25366:
25358:
25351:
25350:Kullback 1959
25346:
25338:
25334:
25330:
25326:
25322:
25318:
25317:
25312:
25306:
25299:
25298:Kullback 1959
25294:
25287:
25286:Jeffreys 1948
25282:
25280:
25273:, p. 80.
25272:
25267:
25265:
25258:, p. 28.
25257:
25252:
25250:
25243:, p. 11.
25242:
25237:
25235:
25233:
25231:
25229:
25221:
25220:Kullback 1959
25216:
25214:
25212:
25203:
25199:
25195:
25191:
25186:
25181:
25177:
25173:
25172:
25167:
25163:
25162:Leibler, R.A.
25159:
25153:
25144:
25139:
25135:
25131:
25127:
25120:
25118:
25113:
25104:
25101:
25100:
25096:
25092:
25089:
25087:
25084:
25082:
25079:
25077:
25074:
25072:
25069:
25067:
25064:
25062:
25059:
25057:
25054:
25052:
25049:
25047:
25044:
25042:
25041:Cross-entropy
25039:
25037:
25034:
25032:
25029:
25027:
25024:
25023:
25016:
25014:
25010:
25006:
25005:
24999:
24995:
24994:
24988:
24982:
24972:
24970:
24969:
24964:
24963:
24958:
24957:
24952:
24948:
24947:
24942:
24938:
24918:
24910:
24906:
24892:
24882:
24879:
24876:
24859:
24855:
24851:
24847:
24844:
24839:
24833:
24830:
24827:
24821:
24813:
24793:
24790:
24784:
24781:
24778:
24761:
24740:
24732:
24729:
24726:
24714:
24708:
24705:
24698:
24692:
24689:
24686:
24680:
24672:
24653:
24650:
24647:
24641:
24633:
24629:
24628:
24627:
24625:
24615:
24612:
24610:
24606:
24602:
24598:
24564:
24540:
24534:
24531:
24528:
24520:
24517:
24512:
24509:
24502:
24501:
24500:
24476:
24473:
24470:
24458:
24452:
24449:
24444:
24438:
24435:
24432:
24420:
24414:
24411:
24406:
24397:
24389:
24388:
24387:
24386:, defined by
24385:
24369:
24366:
24363:
24354:
24340:
24337:
24334:
24314:
24275:
24269:
24263:
24260:
24257:
24251:
24248:
24245:
24242:
24239:
24227:
24220:
24217:
24214:
24208:
24202:
24196:
24193:
24190:
24184:
24181:
24178:
24175:
24172:
24160:
24156:
24153:
24147:
24144:
24141:
24133:
24129:
24121:
24120:
24119:
24118:-divergence,
24105:
24096:
24094:
24090:
24086:
24074:
24070:
24065:
24063:
24059:
24055:
24033:
24030:
24027:
24015:
24011:
24005:
24002:
23999:
23987:
23979:
23978:
23977:
23975:
23966:
23964:
23960:
23955:
23953:
23948:
23944:
23939:
23937:
23933:
23929:
23922:
23916:
23906:
23901:in the state
23900:
23877:
23874:
23871:
23859:
23850:
23831:
23819:
23813:
23810:
23807:
23801:
23795:
23792:
23786:
23780:
23777:
23774:
23768:
23765:
23762:
23750:
23742:
23741:
23740:
23730:
23726:
23725:Hilbert space
23715:
23705:
23687:
23683:
23678:
23671:
23667:
23663:
23658:
23649:
23646:
23641:
23637:
23630:
23626:
23622:
23617:
23609:
23605:
23602:
23599:
23596:
23586:
23585:
23584:
23570:
23562:
23558:
23554:
23551:
23529:
23525:
23502:
23498:
23475:
23471:
23445:
23441:
23436:
23432:
23426:
23423:
23420:
23400:
23397:
23394:
23391:
23381:
23380:net surprisal
23363:
23359:
23350:
23345:
23343:
23327:
23324:
23321:
23318:
23315:
23312:
23309:
23306:
23303:
23300:
23294:
23266:
23262:
23257:
23251:
23247:
23243:
23240:
23237:
23232:
23228:
23202:
23198:
23193:
23189:
23178:
23174:
23170:
23167:
23164:
23161:
23155:
23152:
23130:
23126:
23118:and pressure
23103:
23099:
23090:
23074:
23071:
23068:
23065:
23062:
23059:
23056:
23053:
23050:
23043:
23011:
23008:
23005:
23002:
22999:
22996:
22989:
22969:
22966:
22963:
22943:
22940:
22937:
22924:
22922:
22899:
22893:
22890:
22887:
22884:
22881:
22878:
22870:
22866:
22854:
22847:
22842:
22817:
22813:
22809:
22768:
22762:
22759:
22755:
22751:
22748:
22745:
22742:
22739:
22736:
22732:
22728:
22725:
22722:
22718:
22690:
22686:
22682:
22676:
22673:
22670:
22667:
22664:
22652:
22631:
22627:
22617:
22608:
22591:
22588:
22585:
22554:
22551:
22548:
22536:
22512:
22509:
22506:
22494:
22477:
22454:
22448:
22445:
22442:
22430:
22426:
22420:
22409:
22403:
22400:
22397:
22382:
22381:
22380:
22358:
22354:
22349:
22347:
22346:
22341:
22337:
22333:
22329:
22324:
22303:
22297:
22274:
22268:
22245:
22242:
22239:
22233:
22230:
22224:
22221:
22218:
22212:
22186:
22183:
22180:
22174:
22171:
22165:
22162:
22159:
22153:
22141:
22132:
22128:
22109:
22106:
22103:
22097:
22074:
22071:
22068:
22062:
22039:
22033:
22010:
22004:
21977:
21971:
21948:
21939:
21933:
21930:
21924:
21918:
21906:
21902:
21898:
21888:
21885:
21882:
21876:
21873:
21867:
21864:
21861:
21855:
21843:
21838:
21834:
21826:
21820:
21812:
21803:
21800:
21797:
21791:
21788:
21782:
21776:
21770:
21767:
21764:
21758:
21746:
21738:
21737:
21736:
21719:
21713:
21707:
21704:
21701:
21695:
21664:
21658:
21623:
21620:
21617:
21611:
21602:
21601:as possible.
21583:
21579:
21575:
21572:
21560:
21537:
21533:
21520:
21516:
21502:
21500:
21496:
21495:loss function
21492:
21487:
21482:
21477:
21475:
21456:
21447:
21444:
21441:
21435:
21432:
21426:
21423:
21420:
21414:
21402:
21398:
21395:
21392:
21389:
21378:
21374:
21370:
21367:
21361:
21358:
21350:
21346:
21342:
21339:
21333:
21321:
21313:
21312:
21311:
21294:
21288:
21280:
21262:
21258:
21235:
21231:
21221:
21205:
21201:
21178:
21174:
21165:
21147:
21143:
21120:
21116:
21093:
21089:
21066:
21062:
21039:
21035:
21026:
20998:
20994:
20990:
20987:
20981:
20978:
20970:
20966:
20962:
20959:
20953:
20937:
20922:
20920:
20915:
20905:
20888:
20885:
20882:
20879:
20874:
20870:
20866:
20861:
20857:
20850:
20842:
20813:
20810:
20807:
20801:
20798:
20792:
20789:
20784:
20780:
20776:
20773:
20767:
20751:
20747:
20734:
20731:
20726:
20722:
20718:
20715:
20709:
20706:
20700:
20697:
20692:
20688:
20684:
20679:
20675:
20671:
20668:
20662:
20646:
20615:
20612:
20609:
20603:
20600:
20594:
20591:
20586:
20582:
20578:
20573:
20569:
20565:
20562:
20556:
20540:
20532:
20531:
20530:
20528:
20507:
20498:
20495:
20492:
20486:
20478:
20475:
20470:
20466:
20462:
20459:
20453:
20447:
20443:
20440:
20434:
20431:
20426:
20422:
20418:
20415:
20409:
20404:
20400:
20377:
20368:
20365:
20362:
20356:
20348:
20345:
20340:
20336:
20332:
20327:
20323:
20319:
20316:
20310:
20304:
20300:
20297:
20291:
20288:
20283:
20279:
20275:
20270:
20266:
20262:
20259:
20253:
20248:
20244:
20236:
20235:
20234:
20217:
20214:
20211:
20205:
20182:
20179:
20174:
20170:
20166:
20163:
20157:
20134:
20131:
20126:
20122:
20118:
20113:
20109:
20105:
20102:
20096:
20070:
20066:
20062:
20057:
20053:
20043:
20029:
20026:
20023:
19995:
19986:
19983:
19980:
19974:
19966:
19963:
19960:
19957:
19954:
19948:
19942:
19938:
19935:
19929:
19926:
19923:
19920:
19917:
19911:
19906:
19902:
19898:
19885:
19882:
19879:
19873:
19870:
19864:
19861:
19858:
19855:
19852:
19846:
19830:
19822:
19821:
19820:
19803:
19800:
19797:
19794:
19791:
19785:
19762:
19759:
19756:
19750:
19724:
19721:
19718:
19712:
19681:
19675:
19672:
19669:
19666:
19663:
19657:
19654:
19651:
19645:
19642:
19639:
19636:
19633:
19627:
19622:
19618:
19614:
19611:
19598:
19595:
19592:
19589:
19586:
19580:
19561:
19560:
19559:
19557:
19532:
19529:
19526:
19520:
19512:
19509:
19506:
19500:
19494:
19491:
19488:
19485:
19482:
19476:
19470:
19464:
19461:
19458:
19455:
19452:
19446:
19439:
19438:
19437:
19435:
19416:
19413:
19410:
19407:
19404:
19398:
19375:
19372:
19369:
19363:
19339:
19336:
19333:
19310:
19307:
19304:
19298:
19289:
19283:
19275:
19271:
19267:
19257:
19232:
19215:
19213:
19194:
19188:
19185:
19182:
19170:
19166:
19160:
19149:
19140:
19134:
19131:
19128:
19122:
19117:
19109:
19103:
19100:
19097:
19082:
19081:
19080:
19078:
19058:
19046:
19042:
19041:cross entropy
19037:
19027:
19026:cross entropy
19023:
19000:
18980:
18972:
18964:
18960:
18956:
18945:Cross entropy
18942:
18917:
18909:
18903:
18880:
18872:
18868:
18855:
18836:
18833:
18830:
18824:
18801:
18795:
18789:
18781:
18777:
18768:
18736:
18731:
18724:
18716:
18712:
18708:
18704:
18700:
18697:
18694:
18690:
18686:
18682:
18672:
18667:
18663:
18658:
18650:
18644:
18638:
18635:
18632:
18630:
18619:
18616:
18613:
18607:
18601:
18595:
18584:
18582:
18568:
18560:
18556:
18552:
18546:
18540:
18528:
18524:
18515:
18509:
18503:
18497:
18494:
18488:
18485:
18482:
18476:
18464:
18460:
18454:
18448:
18445:
18442:
18440:
18426:
18420:
18414:
18406:
18402:
18398:
18392:
18389:
18386:
18380:
18368:
18364:
18358:
18352:
18349:
18346:
18344:
18336:
18333:
18330:
18311:
18310:
18309:
18307:
18277:
18271:
18255:
18249:
18237:
18233:
18227:
18221:
18218:
18211:
18210:
18209:
18192:
18189:
18186:
18176:
18170:
18162:
18156:
18150:
18147:
18141:
18135:
18132:
18129:
18123:
18117:
18114:
18111:
18105:
18097:
18093:
18081:
18069:
18068:
18067:
18065:
18061:
18057:
18038:
18032:
18009:
18001:
17997:
17984:
17965:
17959:
17936:
17928:
17924:
17911:
17907:
17875:
17868:
17860:
17856:
17852:
17846:
17838:
17834:
17829:
17819:
17815:
17809:
17803:
17800:
17797:
17795:
17786:
17779:
17773:
17768:
17759:
17755:
17749:
17747:
17739:
17720:
17719:
17718:
17716:
17706:
17696:
17678:
17675:
17672:
17666:
17650:
17631:
17625:
17619:
17613:
17590:
17587:
17584:
17578:
17571:
17539:
17533:
17530:
17524:
17521:
17518:
17512:
17500:
17493:
17488:
17480:
17478:
17461:
17455:
17452:
17446:
17443:
17440:
17434:
17422:
17415:
17410:
17402:
17400:
17386:
17380:
17374:
17368:
17365:
17359:
17356:
17353:
17347:
17335:
17331:
17329:
17321:
17318:
17315:
17309:
17295:
17294:
17293:
17291:
17281:
17267:
17264:
17261:
17238:
17232:
17208:
17205:
17202:
17194:
17175:
17169:
17146:
17142:
17133:
17129:
17122:
17113:
17108:
17098:
17094:
17088:
17082:
17070:
17069:
17068:
17066:
17061:
17059:
17055:
17051:
17047:
17041:
17031:
17023:
17021:
17017:
17013:
17009:
16985:
16979:
16970:
16965:
16961:
16957:
16946:
16941:
16935:
16931:
16927:
16921:
16912:
16909:
16906:
16900:
16897:
16891:
16888:
16885:
16879:
16862:
16861:
16860:
16843:
16840:
16837:
16834:
16828:
16825:
16822:
16817:
16808:
16804:
16792:
16789:
16786:
16780:
16775:
16757:
16754:
16751:
16747:
16738:
16735:
16732:
16726:
16723:
16720:
16715:
16708:
16697:
16694:
16691:
16685:
16669:
16666:
16663:
16659:
16646:
16645:
16644:
16613:
16610:
16607:
16604:
16601:
16594:
16588:
16582:
16579:
16576:
16570:
16560:
16557:
16554:
16550:
16546:
16540:
16534:
16531:
16529:
16523:
16515:
16511:
16499:
16493:
16490:
16485:
16474:
16463:
16460:
16457:
16450:
16444:
16434:
16431:
16428:
16424:
16420:
16414:
16408:
16405:
16403:
16397:
16389:
16385:
16376:
16371:
16360:
16352:
16346:
16343:
16340:
16333:
16327:
16317:
16314:
16311:
16307:
16303:
16297:
16291:
16288:
16286:
16280:
16274:
16266:
16257:
16245:
16225:
16221:
16209:
16203:
16200:
16195:
16184:
16176:
16172:
16160:
16154:
16151:
16146:
16135:
16129:
16118:
16112:
16109:
16096:
16095:
16094:
16075:
16072:
16069:
16062:
16047:
16041:
16022:
16014:
16011:
16007:
15984:
15980:
15971:
15944:
15940:
15933:
15930:
15924:
15918:
15906:
15898:
15894:
15890:
15887:
15882:
15874:
15870:
15860:
15856:
15846:
15832:
15824:
15820:
15811:
15808:
15804:
15796:
15795:
15794:
15792:
15773:
15770:
15762:
15758:
15749:
15746:
15742:
15735:
15731:
15721:
15717:
15707:
15704:
15699:
15688:
15684:
15677:
15674:
15668:
15662:
15650:
15642:
15641:
15640:
15638:
15619:
15616:
15613:
15602:
15598:
15591:
15588:
15582:
15576:
15564:
15556:
15552:
15548:
15545:
15540:
15532:
15528:
15507:
15506:
15505:
15489:
15485:
15473:
15455:
15451:
15447:
15444:
15424:
15421:
15418:
15393:
15389:
15377:
15373:
15364:
15355:
15351:
15346:
15342:
15338:
15332:
15328:
15303:
15281:
15271:
15267:
15263:
15260:
15254:
15249:
15245:
15218:
15215:
15207:
15203:
15194:
15190:
15183:
15179:
15172:
15164:
15160:
15153:
15150:
15144:
15138:
15131:
15130:
15129:
15127:
15109:
15105:
15084:
15059:
15055:
15048:
15045:
15042:
15019:
15013:
15010:
15007:
14987:
14971:
14961:
14959:
14939:
14934:
14931:
14927:
14921:
14917:
14910:
14905:
14901:
14895:
14891:
14883:
14882:
14881:
14880:entails that
14879:
14860:
14854:
14850:
14843:
14839:
14831:
14830:
14829:
14811:
14808:
14802:
14799:
14794:
14790:
14777:
14765:
14753:
14752:
14751:
14734:
14731:
14726:
14722:
14718:
14713:
14709:
14697:
14693:
14688:
14686:
14667:
14664:
14661:
14649:
14625:
14622:
14619:
14607:
14599:. In general
14598:
14594:
14590:
14586:
14582:
14578:
14574:
14564:
14563:long stream.
14546:
14543:
14538:
14534:
14513:
14510:
14505:
14501:
14487:
14471:
14468:
14465:
14460:
14457:
14454:
14448:
14445:
14442:
14438:
14427:
14416:
14406:
14397:
14383:
14380:
14377:
14374:
14371:
14368:
14365:
14342:
14339:
14336:
14330:
14327:
14304:
14301:
14298:
14292:
14289:
14275:
14245:
14239:
14236:
14232:
14228:
14224:
14217:
14214:
14209:
14206:
14202:
14195:
14192:
14189:
14184:
14180:
14176:
14172:
14161:
14150:
14140:
14131:
14115:
14111:
14106:
14100:
14096:
14092:
14089:
14064:
14061:
14056:
14048:
14043:
14039:
14035:
14028:
14018:
14014:
14010:
14005:
14001:
13994:
13989:
13984:
13980:
13973:
13966:
13962:
13956:
13952:
13946:
13943:
13940:
13936:
13925:
13914:
13904:
13896:
13895:
13894:
13891:
13885:
13865:
13861:
13856:
13851:
13846:
13842:
13838:
13834:
13831:
13828:
13825:
13822:
13817:
13812:
13808:
13804:
13799:
13794:
13790:
13785:
13779:
13774:
13771:
13768:
13764:
13758:
13755:
13750:
13746:
13741:
13732:
13723:
13712:
13708:
13703:
13697:
13692:
13688:
13684:
13681:
13678:
13673:
13668:
13664:
13659:
13655:
13652:
13649:
13637:
13631:
13627:
13623:
13620:
13617:
13612:
13608:
13603:
13597:
13585:
13575:
13567:
13566:
13565:
13563:
13544:
13540:
13531:
13528:
13518:
13514:
13503:
13500:
13490:
13486:
13476:
13473:
13468:
13463:
13460:
13457:
13453:
13449:
13446:
13441:
13431:
13423:
13420:
13417:
13412:
13402:
13399:
13395:
13386:
13381:
13378:
13375:
13372:
13369:
13365:
13360:
13354:
13351:
13346:
13342:
13336:
13324:
13319:
13306:
13296:
13288:
13287:
13286:
13270:
13266:
13262:
13257:
13253:
13249:
13246:
13241:
13237:
13214:
13210:
13206:
13203:
13198:
13194:
13163:
13158:
13154:
13148:
13144:
13140:
13135:
13108:
13103:
13099:
13093:
13089:
13085:
13080:
13053:
13049:
13045:
13040:
13036:
13026:
13024:
13005:
12999:
12996:
12988:
12980:
12974:
12955:
12951:
12946:
12938:
12923:
12909:
12905:
12902:
12899:
12895:
12889:
12885:
12881:
12876:
12872:
12867:
12861:
12858:
12853:
12836:
12830:
12826:
12822:
12817:
12813:
12808:
12803:
12800:
12797:
12793:
12787:
12777:
12774:
12769:
12760:
12756:
12753:
12749:
12743:
12740:
12735:
12731:
12725:
12713:
12708:
12695:
12685:
12677:
12676:
12675:
12657:
12652:
12644:
12639:
12627:
12609:
12605:
12601:
12596:
12592:
12584:, with means
12583:
12568:
12566:
12560:
12544:
12540:
12536:
12533:
12513:
12510:
12502:
12498:
12494:
12491:
12479:
12455:
12449:
12446:
12443:
12435:
12431:
12427:
12424:
12421:
12413:
12409:
12405:
12402:
12390:
12386:
12380:
12377:
12374:
12366:
12362:
12358:
12355:
12352:
12346:
12343:
12340:
12328:
12324:
12318:
12310:
12306:
12298:
12297:
12296:
12279:
12273:
12270:
12267:
12259:
12255:
12251:
12248:
12245:
12239:
12231:
12227:
12223:
12220:
12214:
12211:
12208:
12196:
12192:
12184:
12180:
12176:
12173:
12161:
12153:
12152:
12151:
12134:
12131:
12125:
12116:
12113:
12110:
12102:
12098:
12089:
12083:
12080:
12077:
12071:
12065:
12062:
12054:
12050:
12023:
12020:
12017:
12009:
12005:
11996:
11990:
11987:
11984:
11955:
11951:
11923:
11917:
11914:
11900:
11872:
11869:
11863:
11855:
11852:
11846:
11816:
11807:
11804:
11801:
11793:
11789:
11780:
11774:
11771:
11768:
11762:
11753:
11750:
11744:
11736:
11733:
11727:
11717:
11716:
11715:
11698:
11689:
11686:
11683:
11671:
11667:
11661:
11653:
11649:
11640:
11637:
11634:
11624:
11618:
11615:
11612:
11604:
11600:
11596:
11593:
11586:
11585:
11584:
11567:
11564:
11554:
11540:
11512:
11509:
11506:
11486:
11483:
11480:
11457:
11454:
11444:
11415:
11412:
11402:
11389:
11341:
11303:
11301:
11291:
11286:
11266:
11244:
11239:
11233:
11230:
11225:
11222:
11218:
11213:
11204:
11201:
11198:
11192:
11188:
11166:
11146:
11143:
11140:
11137:
11134:
11131:
11111:
11091:
11088:
11085:
11082:
11062:
11059:
11056:
11036:
11029:with measure
11016:
11013:
11010:
10990:
10987:
10984:
10981:
10978:
10975:
10952:
10949:
10929:
10926:
10923:
10903:
10900:
10897:
10877:
10857:
10854:
10851:
10848:
10828:
10825:
10822:
10819:
10796:
10793:
10791:
10777:
10774:
10771:
10765:
10761:
10749:
10746:
10743:
10739:
10735:
10733:
10724:
10717:
10711:
10708:
10702:
10696:
10692:
10679:
10676:
10672:
10662:
10659:
10656:
10650:
10646:
10634:
10631:
10628:
10624:
10620:
10618:
10608:
10605:
10602:
10597:
10587:
10581:
10573:
10567:
10561:
10558:
10554:
10548:
10541:
10535:
10532:
10526:
10520:
10516:
10503:
10500:
10496:
10486:
10483:
10480:
10474:
10470:
10458:
10455:
10452:
10448:
10444:
10442:
10436:
10427:
10424:
10421:
10413:
10407:
10400:
10389:
10383:
10380:
10374:
10368:
10350:
10347:
10343:
10333:
10330:
10327:
10321:
10317:
10311:
10300:
10297:
10294:
10290:
10265:
10262:
10259:
10256:
10228:
10225:
10222:
10214:
10208:
10201:
10190:
10184:
10181:
10175:
10169:
10151:
10148:
10144:
10134:
10131:
10128:
10122:
10118:
10106:
10103:
10100:
10096:
10092:
10090:
10079:
10076:
10068:
10057:
10050:
10036:
10033:
10030:
10026:
10022:
10020:
10012:
10009:
10006:
9994:
9969:
9966:
9963:
9956:evaluated at
9943:
9923:
9900:
9897:
9894:
9882:
9852:
9849:
9846:
9838:
9832:
9825:
9814:
9808:
9805:
9799:
9793:
9775:
9772:
9768:
9764:
9758:
9755:
9752:
9744:
9736:
9733:
9727:
9725:
9717:
9706:
9699:
9687:
9684:
9681:
9676:
9669:
9663:
9660:
9657:
9651:
9645:
9639:
9636:
9633:
9626:
9618:
9607:
9601:
9598:
9592:
9586:
9568:
9565:
9561:
9557:
9551:
9548:
9545:
9537:
9529:
9526:
9520:
9518:
9510:
9499:
9492:
9478:
9472:
9465:
9454:
9448:
9445:
9439:
9433:
9415:
9412:
9408:
9404:
9402:
9394:
9387:
9384:
9370:
9364:
9361:
9358:
9352:
9346:
9340:
9337:
9334:
9324:
9313:
9307:
9304:
9298:
9292:
9274:
9271:
9267:
9263:
9261:
9253:
9246:
9243:
9214:
9211:
9209:
9201:
9194:
9191:
9182:
9173:
9167:
9159:
9153:
9150:
9147:
9141:
9135:
9129:
9126:
9123:
9114:
9110:
9107:
9098:
9092:
9089:
9083:
9077:
9062:
9059:
9055:
9051:
9049:
9040:
9036:
9033:
9029:
9020:
9014:
9006:
9000:
8997:
8994:
8988:
8982:
8976:
8973:
8970:
8961:
8957:
8954:
8950:
8940:
8934:
8931:
8925:
8919:
8904:
8901:
8897:
8893:
8891:
8883:
8876:
8873:
8848:
8825:
8819:
8816:
8810:
8807:
8804:
8792:
8768:
8765:
8762:
8759:
8756:
8753:
8747:
8744:
8741:
8726:
8722:
8716:
8710:
8684:
8664:
8661:
8658:
8655:
8630:
8627:
8624:
8616:
8610:
8603:
8592:
8586:
8583:
8577:
8571:
8553:
8550:
8546:
8536:
8533:
8530:
8524:
8520:
8508:
8505:
8502:
8498:
8494:
8488:
8485:
8482:
8470:
8449:
8446:
8443:
8420:
8417:
8414:
8402:
8394:
8380:
8377:
8374:
8371:
8368:
8355:
8351:
8347:
8342:
8338:
8325:
8318:
8315:
8312:
8306:
8298:
8294:
8290:
8285:
8281:
8268:
8264:
8261:
8253:
8249:
8242:
8239:
8236:
8230:
8225:
8221:
8217:
8214:
8209:
8205:
8198:
8195:
8192:
8186:
8181:
8177:
8173:
8161:
8135:
8131:
8127:
8122:
8118:
8089:
8085:
8081:
8076:
8072:
8045:
8042:
8039:
8029:
8025:
8006:
8003:
8000:
7988:
7979:
7965:
7957:
7953:
7949:
7944:
7940:
7927:
7923:
7915:
7911:
7907:
7902:
7898:
7885:
7881:
7875:
7872:
7869:
7857:
7834:
7830:
7826:
7821:
7817:
7793:
7790:
7782:
7778:
7771:
7768:
7760:
7756:
7752:
7746:
7743:
7740:
7737:
7734:
7728:
7705:
7702:
7694:
7690:
7683:
7680:
7672:
7668:
7664:
7658:
7655:
7652:
7649:
7646:
7640:
7618:
7614:
7610:
7605:
7601:
7592:
7588:
7584:
7581:
7577:
7560:
7557:
7550:
7544:
7541:
7535:
7532:
7526:
7503:
7497:
7474:
7468:
7456:
7435:
7431:
7424:
7421:
7416:
7412:
7386:
7382:
7375:
7372:
7367:
7363:
7338:
7335:
7330:
7321:
7309:
7298:
7286:
7277:
7273:
7270:
7264:
7252:
7242:
7238:
7230:
7226:
7221:
7217:
7215:
7207:
7204:
7199:
7185:
7176:
7173:
7168:
7165:
7147:
7141:
7129:
7113:
7104:
7101:
7096:
7093:
7075:
7069:
7057:
7048:
7044:
7041:
7030:
7021:
7018:
7013:
7010:
6993:
6987:
6975:
6965:
6961:
6953:
6949:
6944:
6940:
6938:
6930:
6927:
6922:
6913:
6907:
6899:
6893:
6887:
6883:
6880:
6874:
6868:
6861:
6857:
6849:
6845:
6840:
6836:
6834:
6826:
6823:
6820:
6808:
6795:
6771:
6761:
6758:
6753:
6750:
6721:
6718:
6707:
6697:
6694:
6689:
6686:
6671:
6659:
6653:
6650:
6647:
6640:
6628:
6622:
6619:
6616:
6609:
6603:
6600:
6594:
6591:
6585:
6565:
6562:
6550:
6540:
6537:
6532:
6529:
6511:
6505:
6493:
6487:
6484:
6481:
6474:
6462:
6456:
6453:
6450:
6443:
6437:
6434:
6428:
6425:
6419:
6396:
6390:
6378:
6374:
6356:
6353:
6350:
6338:
6321:
6320:
6319:
6293:
6289:
6270:
6267:
6264:
6233:
6217:
6201:
6179:
6173:
6170:
6164:
6158:
6135:
6132:
6126:
6120:
6114:
6111:
6105:
6102:
6096:
6073:
6070:
6064:
6058:
6052:
6049:
6043:
6040:
6034:
6010:
6007:
6004:
5997:
5978:
5975:
5972:
5960:
5951:
5935:
5932:
5929:
5923:
5920:
5917:
5905:
5896:
5892:
5891:
5885:
5883:
5879:
5875:
5869:
5867:
5863:
5862:rate function
5858:
5856:
5852:
5848:
5829:
5826:
5823:
5811:
5787:
5781:
5758:
5752:
5749:
5743:
5740:
5737:
5731:
5723:
5722:loss function
5704:
5701:
5698:
5692:
5684:
5672:
5653:
5650:
5647:
5635:
5625:
5619:
5600:
5576:
5575:cross entropy
5557:
5554:
5551:
5513:
5502:
5496:
5493:
5490:
5479:
5477:
5463:
5457:
5453:
5448:
5445:
5439:
5433:
5421:
5418:
5414:
5410:
5401:
5395:
5391:
5386:
5383:
5377:
5371:
5359:
5356:
5352:
5348:
5346:
5338:
5335:
5332:
5320:
5308:
5307:
5306:
5304:
5278:
5274:
5251:
5247:
5218:
5214:
5210:
5206:
5202:
5194:
5190:
5183:
5157:
5153:
5144:
5136:
5131:
5122:
5101:
5091:
5082:
5080:
5076:
5072:
5065:
5060:
5058:
5054:
5050:
5046:
5042:
5038:
5034:
5033:metric tensor
5030:
5026:
5025:infinitesimal
5007:
5004:
5001:
4989:
4965:
4962:
4959:
4947:
4939:. In general
4938:
4934:
4930:
4926:
4922:
4918:
4914:
4909:
4907:
4891:
4890:approximation
4873:
4860:
4853:
4834:
4831:
4828:
4816:
4807:
4802:
4792:
4780:
4761:
4758:
4755:
4743:
4734:
4724:
4711:samples from
4710:
4706:
4687:
4684:
4681:
4669:
4660:
4659:coding theory
4650:
4625:
4619:
4616:
4613:
4610:
4604:
4598:
4595:
4592:
4572:
4539:
4536:
4532:
4525:
4519:
4516:
4513:
4510:
4504:
4498:
4495:
4492:
4489:
4483:
4477:
4474:
4471:
4468:
4464:
4458:
4455:
4450:
4448:
4439:
4433:
4429:
4425:
4420:
4416:
4412:
4406:
4402:
4399:
4394:
4391:
4386:
4382:
4376:
4372:
4368:
4363:
4359:
4355:
4349:
4345:
4342:
4337:
4334:
4329:
4325:
4319:
4315:
4311:
4306:
4302:
4298:
4292:
4288:
4285:
4280:
4277:
4272:
4270:
4261:
4252:
4246:
4238:
4232:
4226:
4222:
4219:
4213:
4207:
4195:
4192:
4188:
4184:
4182:
4174:
4171:
4168:
4156:
4144:
4143:
4124:
4121:
4118:
4114:
4107:
4101:
4098:
4095:
4092:
4086:
4080:
4077:
4074:
4071:
4065:
4059:
4056:
4053:
4049:
4043:
4040:
4035:
4033:
4024:
4018:
4014:
4010:
4005:
4001:
3997:
3991:
3987:
3984:
3979:
3976:
3971:
3967:
3961:
3957:
3953:
3948:
3944:
3940:
3934:
3930:
3927:
3922:
3919:
3914:
3910:
3904:
3900:
3896:
3891:
3887:
3883:
3877:
3873:
3870:
3865:
3862:
3857:
3855:
3846:
3837:
3831:
3823:
3817:
3811:
3807:
3804:
3798:
3792:
3780:
3777:
3773:
3769:
3767:
3759:
3756:
3753:
3741:
3729:
3728:
3727:
3725:
3721:
3714:, designated
3713:
3707:
3688:
3685:
3682:
3670:
3646:
3643:
3640:
3628:
3600:
3597:
3588:
3572:
3569:
3560:
3544:
3541:
3532:
3515:
3509:
3502:Distribution
3501:
3500:
3483:
3480:
3471:
3455:
3452:
3443:
3427:
3424:
3415:
3398:
3392:
3385:Distribution
3384:
3383:
3379:
3376:
3373:
3368:
3367:
3360:
3356:
3342:
3338:
3334:
3331:
3328:
3305:
3302:
3299:
3296:
3293:
3287:
3249:
3246:
3222:
3219:
3216:
3196:
3193:
3190:
3182:
3163:Basic example
3160:
3150:
3143:
3118:
3115:
3112:
3100:
3085:
3081:
3074:
3070:
3066:
3062:
3055:
3049:
3042:
3028:
3006:
3003:
3000:
2988:
2975:
2973:
2966:, or to base
2965:
2961:
2957:
2953:
2949:
2945:
2941:
2937:
2917:
2913:
2910:
2907:
2903:
2897:
2894:
2889:
2886:
2863:
2840:
2831:
2816:
2809:
2800:
2794:
2783:
2777:
2768:
2764:
2761:
2752:
2746:
2734:
2731:
2727:
2723:
2717:
2714:
2711:
2699:
2691:
2690:
2689:
2664:
2627:
2612:
2606:
2600:
2597:
2591:
2576:
2547:
2532:
2526:
2520:
2517:
2511:
2496:
2441:
2425:
2422:which is the
2406:
2397:
2382:
2375:
2366:
2351:
2343:
2328:
2322:
2318:
2315:
2303:
2288:
2280:
2265:
2250:
2247:
2243:
2239:
2233:
2230:
2227:
2215:
2207:
2206:
2205:
2203:
2173:
2158:
2152:
2146:
2143:
2137:
2122:
2065:
2040:
2025:
2014:
1999:
1967:
1958:
1943:
1936:
1927:
1912:
1904:
1889:
1883:
1879:
1876:
1861:
1858:
1854:
1850:
1844:
1841:
1838:
1826:
1818:
1817:
1816:
1802:
1782:
1762:
1758:
1745:
1735:
1708:
1702:
1686:
1677:
1671:
1663:
1657:
1651:
1647:
1644:
1635:
1629:
1616:
1612:
1608:
1602:
1599:
1596:
1584:
1576:
1575:
1574:
1572:
1545:
1539:
1536:
1530:
1524:
1521:
1518:
1511:
1507:
1500:
1488:
1487:
1486:
1466:
1460:
1448:
1401:
1398:
1392:
1386:
1357:
1329:
1320:
1301:
1298:
1292:
1286:
1260:
1257:
1251:
1245:
1229:
1215:
1196:
1189:
1180:
1174:
1163:
1157:
1148:
1144:
1141:
1132:
1126:
1114:
1111:
1107:
1103:
1100:
1094:
1091:
1088:
1076:
1068:
1067:
1066:
1049:
1042:
1033:
1027:
1016:
1010:
1001:
997:
994:
985:
979:
967:
964:
960:
956:
950:
947:
944:
932:
924:
923:
922:
900:
879:
868:
858:
856:
852:
849:are given in
848:
844:
840:
836:
817:
814:
811:
805:
802:
796:
793:
790:
784:
781:
775:
772:
769:
763:
741:
737:
714:
710:
686:
683:
680:
674:
652:
648:
644:
639:
635:
612:
608:
604:
599:
595:
572:
568:
564:
559:
555:
532:
528:
505:
501:
478:
474:
465:
461:
457:
447:
445:
441:
402:
399:
396:
384:
349:
347:
343:
339:
335:
331:
327:
323:
317:
315:
311:
307:
303:
299:
295:
291:
287:
271:
267:
255:
236:
232:
223:
217:
206:
200:
191:
187:
184:
175:
169:
157:
154:
150:
146:
140:
137:
134:
122:
114:
113:
112:
103:
99:
80:
77:
74:
62:
53:
49:
46:(also called
45:
41:
37:
33:
19:
26658:Sergio Verdú
26622:
26597:
26570:
26537:
26531:
26521:
26482:. Springer.
26478:
26471:
26460:, retrieved
26429:
26422:
26367:
26361:
26351:
26296:
26290:
26280:
26256:. Springer.
26252:
26245:
26226:
26222:
26212:
26204:the original
26168:(3): 18–26.
26165:
26161:
26151:
26126:
26122:
26116:
26105:
26098:
26073:
26069:
26056:
26023:
26019:
26006:
25996:
25989:
25969:
25958:
25949:11299/199630
25929:
25923:
25913:
25907:, p. 22
25900:
25873:
25830:
25826:
25816:
25798:
25763:
25759:
25753:
25736:
25732:
25726:
25716:Sergio Verdú
25706:
25682:
25663:
25657:
25637:
25632:
25626:(84): 11–44.
25623:
25620:Mat. Sbornik
25619:
25613:
25596:
25592:
25586:
25531:
25527:
25517:
25500:
25496:
25490:
25465:
25457:
25452:, p. 5.
25445:
25427:
25420:
25411:
25402:
25393:
25384:
25364:
25357:
25352:, p. 6.
25345:
25320:
25314:
25311:Kullback, S.
25305:
25300:, p. 7.
25293:
25178:(1): 79–86.
25175:
25169:
25158:Kullback, S.
25152:
25133:
25129:
25008:
25004:differencing
25003:
24997:
24992:
24986:
24984:
24966:
24960:
24954:
24950:
24944:
24940:
24934:
24621:
24613:
24596:
24555:
24494:
24355:
24290:
24097:
24092:
24088:
24066:
24061:
24051:
23972:
23965:estimators.
23956:
23940:
23935:
23931:
23927:
23924:
23899:entanglement
23846:
23711:
23702:
23379:
23346:
22925:
22845:
22843:
22649:
22475:
22469:
22356:
22352:
22350:
22343:
22325:
22130:
22126:
21963:
21603:
21518:
21510:
21508:
21498:
21480:
21478:
21471:
21278:
21222:
21024:
20928:
20911:
20840:
20838:
20526:
20524:
20198:rather than
20044:
20011:
19696:
19553:
19263:
19216:
19209:
19043:between two
19038:
18985:(denoted as
18948:
18853:
18766:
18756:
18303:
18207:
17982:
17905:
17895:
17712:
17694:
17567:
17287:
17161:
17062:
17043:
17029:
17005:
16858:
16635:
16053:
15967:
15788:
15634:
15471:
15233:
14967:
14955:
14876:
14827:
14689:
14592:
14584:
14570:
14488:
14398:
14281:
14132:
14081:
13889:
13883:
13880:
13559:
13178:. Then with
13027:
12970:
12579:
12562:
12470:
12294:
11906:
11831:
11713:
11305:
11299:
11297:
11289:
8702:
6383:to variable
6298:rather than
6026:
6023:as measures.
5994:equals zero
5895:non-negative
5870:
5859:
5846:
5626:
5535:
5302:
5301:: it is the
5140:
5092:
5088:
5061:
4932:
4924:
4910:
4879:
4803:
4790:
4781:achieved if
4730:
4707:required to
4656:
4568:
3619:
3166:
3141:
3079:
3078:and updates
3068:
3053:
3047:
3040:
3026:
2976:
2952:Haar measure
2855:
2688:is given as
2430:relative to
2421:
1982:
1746:
1723:
1560:
1449:
1230:
1211:
1064:
878:sample space
864:
854:
842:
837:in 1948. In
453:
355:
338:neuroscience
318:
251:
52:I-divergence
51:
47:
43:
39:
35:
29:
25688:Bonnici, V.
25130:Ann. Probab
24993:compression
22340:E.T. Jaynes
18056:E.T. Jaynes
17647:of the two
17058:probability
15635:and by the
12295:Therefore,
6306:drawn from
5071:-divergence
3706:natural log
1732:denote the
1214:expectation
364:. Usually,
326:time-series
272:from using
54:), denoted
26681:Categories
26462:2023-02-14
26377:1912.00610
26370:(2): 221.
26306:1904.04017
26299:(5): 485.
26162:Complexity
25883:1363563215
25773:2008.01006
25697:2008.05932
25640:ch. 14.5 (
25541:1811.08308
25534:(8): 860.
25437:1334664824
25256:Amari 2016
25241:Amari 2016
25109:References
24382:gives the
24356:The value
23913:See also:
23340:(see also
22859:or volume
22651:Surprisals
22334:, and the
21164:I. J. Good
19212:Motivation
17016:partitions
15972:. Letting
15789:where the
14581:divergence
13068:such that
10968:such that
8061:, i.e. if
6290:number of
5888:Properties
5683:divergence
5125:Motivation
5031:, gives a
4921:divergence
4565:Statistics
3708:with base
2202:chain rule
2112:such that
861:Definition
302:divergence
44:divergence
26506:cite book
26498:757859245
26466:Lemma 2.1
26457:122597694
26272:878132909
26175:1103.2481
25847:0007-1102
25803:Duchi J.
25790:220935477
24919:α
24880:∥
24856:−
24848:−
24840:≤
24822:δ
24782:∥
24730:∥
24699:≤
24681:δ
24642:δ
24474:∥
24436:∥
24364:λ
24341:λ
24338:−
24315:λ
24264:λ
24261:−
24246:λ
24243:∥
24221:λ
24218:−
24197:λ
24194:−
24179:λ
24176:∥
24157:λ
24145:∥
24134:λ
24106:λ
24031:∥
24003:∥
23875:∥
23814:
23808:−
23796:
23781:
23766:∥
23655:Θ
23614:Θ
23594:Δ
23568:Δ
23427:
23395:≥
23389:Δ
23325:≥
23319:
23313:−
23307:−
23289:Θ
23184:Θ
23159:Δ
23069:−
23006:−
23000:≡
22941:×
22894:
22885:−
22882:≡
22760:−
22752:×
22740:
22677:
22552:∥
22510:∥
22446:∥
22355:(MCE) or
22243:∣
22222:∣
22184:∣
22172:∥
22163:∣
22107:∣
22072:∣
21931:∥
21886:∣
21874:∥
21865:∣
21835:
21789:∥
21768:∣
21705:∣
21576:∥
21445:∣
21433:∥
21424:∣
21390:≠
21371:∣
21359:∥
21343:∣
20991:∣
20979:∥
20963:∣
20867:∣
20811:∣
20799:∥
20777:∣
20719:∣
20707:∥
20672:∣
20613:∣
20601:∥
20566:∣
20496:∣
20463:∣
20444:
20419:∣
20401:∑
20366:∣
20320:∣
20301:
20263:∣
20245:∑
20215:∣
20167:∣
20106:∣
19984:∣
19958:∣
19939:
19921:∣
19903:∑
19883:∣
19871:∥
19856:∣
19795:∣
19760:∣
19722:∣
19667:∣
19655:
19637:∣
19619:∑
19615:−
19590:∣
19530:∣
19510:∣
19486:∣
19456:∣
19408:∣
19373:∣
19308:∣
19296:→
19186:∥
19135:
19129:−
19123:
18709:∥
18698:∣
18664:
18651:−
18639:
18608:
18602:−
18553:∥
18525:−
18495:∥
18461:−
18449:
18399:∥
18365:−
18353:
18334:∣
18234:−
18222:
18151:
18133:∫
18130:−
18118:
18088:∞
18085:→
17853:∥
17816:−
17804:
17774:
17756:
17531:∥
17522:∣
17494:
17453:∥
17444:∣
17416:
17366:∥
17310:
17123:∥
17114:δ
17083:
16983:→
16980:ρ
16962:ρ
16932:ρ
16913:ρ
16898:∥
16823:ρ
16805:ρ
16801:∂
16793:ρ
16772:∂
16763:∞
16748:∫
16721:ρ
16709:ρ
16706:∂
16698:ρ
16683:∂
16675:∞
16660:∫
16617:∞
16611:ξ
16566:∞
16551:∫
16512:ρ
16508:∂
16494:
16482:∂
16467:∞
16440:∞
16425:∫
16386:ρ
16382:∂
16368:∂
16350:∞
16323:∞
16308:∫
16275:ρ
16272:∂
16264:∂
16222:ρ
16218:∂
16204:
16192:∂
16173:ρ
16169:∂
16155:
16143:∂
16130:ρ
16127:∂
16113:
16107:∂
16076:ρ
16023:θ
15981:θ
15941:θ
15931:∥
15925:θ
15895:θ
15888:θ
15871:θ
15867:∂
15857:θ
15853:∂
15843:∂
15821:θ
15774:⋯
15759:θ
15732:θ
15728:Δ
15718:θ
15714:Δ
15685:θ
15675:∥
15669:θ
15599:θ
15589:∥
15583:θ
15553:θ
15546:θ
15529:θ
15525:∂
15521:∂
15486:θ
15482:Δ
15452:θ
15445:θ
15390:θ
15374:θ
15370:∂
15362:∂
15343:θ
15304:θ
15268:θ
15264:−
15261:θ
15246:θ
15242:Δ
15219:⋯
15204:θ
15180:θ
15176:Δ
15161:θ
15145:θ
15106:θ
15085:θ
15056:θ
15020:θ
14988:θ
14914:⇒
14800:∥
14772:∞
14769:→
14735:…
14665:∥
14623:∥
14544:
14511:
14469:−
14458:−
14449:
14428:∥
14381:≤
14369:≤
14240:
14215:−
14207:−
14190:
14162:∥
14112:σ
14097:σ
14057:−
14040:σ
14015:μ
14011:−
14002:μ
13981:σ
13963:σ
13953:σ
13947:
13926:∥
13843:σ
13835:
13829:−
13823:−
13809:μ
13791:σ
13765:∑
13713:∥
13689:σ
13682:…
13665:σ
13656:
13628:μ
13621:…
13609:μ
13477:
13454:∑
13418:−
13366:∑
13325:∥
13267:μ
13263:−
13254:μ
13132:Σ
13077:Σ
13000:
12973:logarithm
12935:Σ
12920:Σ
12906:
12886:μ
12882:−
12873:μ
12859:−
12850:Σ
12827:μ
12823:−
12814:μ
12798:−
12784:Σ
12775:−
12766:Σ
12757:
12714:∥
12649:Σ
12636:Σ
12606:μ
12593:μ
12545:∗
12511:≥
12503:∗
12495:∥
12447:
12428:
12422:≤
12414:∗
12406:∥
12387:−
12378:
12359:
12344:∥
12325:−
12271:
12252:
12224:−
12212:∥
12193:−
12185:∗
12177:∥
12135:θ
12114:
12090:θ
12081:
12066:θ
12055:∗
12021:
11997:θ
11988:
11974:-density
11956:∗
11918:
11873:θ
11856:θ
11805:
11781:θ
11772:
11754:θ
11737:θ
11687:∥
11668:−
11638:≪
11616:
11597:
11552:Θ
11510:≪
11484:≪
11442:Θ
11400:Θ
11342:σ
11322:Θ
11273:∞
11270:→
11231:ϵ
11214:ρ
11202:−
11147:ϵ
11135:≥
11112:ρ
11086:≤
11063:ρ
11060:−
11037:ρ
11014:≤
10991:ϵ
10979:≥
10956:∞
10924:ρ
10898:ϵ
10823:≤
10775:−
10755:∞
10740:∑
10736:≤
10709:−
10680:∈
10673:∑
10660:−
10640:∞
10625:∑
10621:≤
10606:−
10562:−
10533:−
10504:∈
10497:∑
10484:−
10464:∞
10449:∑
10425:−
10381:−
10351:∈
10344:∑
10331:−
10306:∞
10291:∑
10260:≤
10226:−
10182:−
10152:∈
10145:∑
10132:−
10112:∞
10097:∑
10042:∞
10027:∑
10010:∥
9964:α
9898:∥
9850:−
9806:−
9776:∈
9769:∑
9756:−
9734:−
9685:−
9661:α
9640:α
9637:−
9599:−
9569:∈
9562:∑
9549:−
9527:−
9511:α
9446:−
9416:∈
9409:∑
9362:α
9341:α
9338:−
9305:−
9275:∈
9268:∑
9254:α
9151:α
9130:α
9127:−
9111:
9090:−
9063:∈
9056:∑
8998:α
8977:α
8974:−
8958:
8932:−
8905:∈
8898:∑
8884:α
8808:∥
8766:∥
8760:α
8748:α
8745:−
8717:α
8659:≤
8628:−
8584:−
8554:∈
8547:∑
8534:−
8514:∞
8499:∑
8486:∥
8418:∥
8378:≤
8375:λ
8372:≤
8348:∥
8319:λ
8316:−
8291:∥
8265:λ
8262:≤
8243:λ
8240:−
8218:λ
8215:∥
8199:λ
8196:−
8174:λ
8004:∥
7950:∥
7908:∥
7873:∥
7313:~
7290:~
7274:
7256:~
7222:∫
7133:~
7061:~
7045:
6979:~
6945:∫
6884:
6841:∫
6824:∥
6663:~
6632:~
6497:~
6466:~
6354:∥
6202:μ
6127:μ
6065:μ
5976:∥
5930:≥
5921:∥
5827:∥
5651:∥
5503:−
5449:
5422:∈
5415:∑
5411:−
5387:
5360:∈
5353:∑
5336:∥
5305:entropy.
5248:ℓ
5215:ℓ
5211:−
5005:∥
4963:∥
4832:∥
4759:∥
4727:Inference
4685:∥
4617:
4611:−
4596:
4540:0.097455.
4537:≈
4520:
4499:
4490:−
4478:
4469:−
4403:
4346:
4289:
4223:
4196:∈
4189:∑
4172:∥
4122:0.0852996
4119:≈
4102:
4093:−
4081:
4060:
3988:
3931:
3874:
3808:
3781:∈
3774:∑
3757:∥
3686:∥
3644:∥
3116:∥
3084:posterior
3061:asymmetry
3004:∥
2956:Lie group
2887:μ
2864:μ
2817:μ
2765:
2735:∈
2728:∫
2715:∥
2665:μ
2613:μ
2533:μ
2442:μ
2319:
2251:∈
2244:∫
2231:∥
1880:
1862:∈
1855:∫
1842:∥
1648:
1625:∞
1620:∞
1617:−
1613:∫
1600:∥
1525:
1504:→
1450:Whenever
1399:≠
1361:∞
1333:∞
1145:
1115:∈
1108:∑
1104:−
1092:∥
998:
968:∈
961:∑
948:∥
738:μ
711:μ
649:μ
636:μ
569:μ
556:μ
529:μ
450:Etymology
400:∥
330:inference
252:A simple
188:
158:∈
151:∑
138:∥
78:∥
26569:(2016).
26554:14106275
26414:33285995
26343:33267199
26200:20794688
26048:17870175
25855:21492134
25578:33286632
25164:(1951).
25019:See also
24998:relative
24987:absolute
24985:Just as
23544:is thus
22379:, since
19028:between
18957:) using
15968:must be
14928:→
14902:→
14851:→
14692:topology
12571:Examples
11836:, where
11308:Theorem
9388:″
9247:″
9195:′
8877:′
7587:additive
6794:Jacobian
6288:expected
5803:to make
5239:, where
5075:alphabet
1757:measures
1276:implies
270:surprise
266:expected
26405:7516653
26382:Bibcode
26363:Entropy
26334:7514974
26311:Bibcode
26292:Entropy
26180:Bibcode
26131:Bibcode
26078:Bibcode
26028:Bibcode
25605:4134885
25569:7517462
25546:Bibcode
25528:Entropy
25337:2684769
25202:0039968
25194:2236703
24597:locally
23024:(where
22853:entropy
22841:coins.
22357:Minxent
22328:Laplace
20841:average
19556:entropy
18852:— i.e.
17981:— i.e.
17191:from a
16042:on the
15437:, i.e.
15316:in the
14593:squared
12567:, see.
12042:, i.e.
11533:.) Let
11473:, with
11354:-field
9982:yields
8703:Denote
6151:, then
5952:, with
5618:entropy
5616:is the
5573:is the
5055:and in
5029:Hessian
4933:squared
4857:to the
3147:or the
3027:between
2424:entropy
2062:is the
268:excess
264:is the
34:, the
26613:
26585:
26552:
26496:
26486:
26455:
26445:
26412:
26402:
26341:
26331:
26270:
26260:
26198:
26046:
25977:
25881:
25853:
25845:
25788:
25670:
25648:
25603:
25576:
25566:
25478:
25435:
25372:
25335:
25200:
25192:
24965:, and
24495:where
23727:, the
23463:where
23220:where
22915:where
19432:using
18937:given
16859:then:
16636:where
16243:exist,
15472:second
14585:linear
14577:metric
13229:, and
11943:, let
10942:, and
9936:about
8024:convex
7355:where
6734:where
6286:, the
5847:excess
5585:, and
5536:where
5303:excess
5039:; see
4925:linear
4917:metric
4653:Coding
3273:(i.e.
3125:
3097:
3013:
2985:
2948:sphere
2922:
2884:
2838:
2828:
2814:
2790:
2775:
2759:
2668:
2662:
2634:
2624:
2588:
2574:
2554:
2544:
2508:
2494:
2404:
2394:
2380:
2363:
2340:
2313:
2300:
2277:
2180:
2170:
2134:
2120:
2100:
2090:
2047:
2037:
2021:
2011:
1997:
1991:
1983:where
1965:
1955:
1941:
1924:
1901:
1874:
1780:
1770:
1724:where
1706:
1699:
1691:
1642:
1543:
1473:
1458:
1435:
1425:
1405:
1384:
1364:
1355:
1305:
1284:
1264:
1243:
1194:
1170:
1155:
1139:
1047:
1023:
1008:
992:
898:
888:
587:, and
344:, and
286:metric
213:
198:
182:
26550:S2CID
26453:S2CID
26372:arXiv
26301:arXiv
26196:S2CID
26170:arXiv
26066:(PDF)
26044:S2CID
26016:(PDF)
25808:(PDF)
25786:S2CID
25768:arXiv
25692:arXiv
25536:arXiv
25333:JSTOR
25190:JSTOR
25013:patch
25009:given
25002:data
24991:data
23731:from
23723:on a
22869:Gibbs
22706:. If
21639:over
21486:logit
21250:over
21193:over
21054:over
19356:from
19272:to a
18761:from
17900:from
17054:event
16054:When
15234:with
12150:Then
11970:have
11904:Proof
8462:) as
7849:then
5235:over
3722:(see
3183:with
3155:over
3151:from
3073:prior
2954:on a
2486:with
1759:on a
1569:of a
260:from
26666:NIPS
26611:ISBN
26583:ISBN
26512:link
26494:OCLC
26484:ISBN
26443:ISBN
26410:PMID
26339:PMID
26268:OCLC
26258:ISBN
25975:ISBN
25879:OCLC
25851:PMID
25843:ISSN
25721:2009
25719:NIPS
25668:ISBN
25646:ISBN
25601:SSRN
25597:June
25593:Risk
25574:PMID
25476:ISBN
25433:OCLC
25370:ISBN
25093:and
24791:>
24630:The
24587:and
24327:and
24085:pdfs
24079:and
23961:and
23945:via
23719:and
23712:For
23517:and
23281:and
23089:work
23036:and
22749:1.38
22375:and
22129:(or
21684:and
21651:was
21643:and
21499:both
21027:for
19071:and
19057:bits
19051:and
19039:The
19032:and
19022:bits
18854:less
18767:less
18304:The
17983:less
17906:less
17713:The
17655:and
17288:The
17044:The
16614:<
16608:<
16532:<
16464:<
16406:<
16347:<
16289:<
15035:and
14976:and
14375:<
13887:and
13653:diag
13182:and
13123:and
13023:bits
12987:nats
12971:The
11431:and
11382:and
11314:Let
11003:and
10953:<
10927:>
10901:>
10852:>
8107:and
7589:for
7490:and
7404:and
6578:and
6326:and
6292:bits
6089:and
5581:and
4709:code
4705:bits
4577:and
3720:nats
3662:and
3209:and
3171:and
3082:the
3069:from
3048:from
3041:from
3032:and
2972:nats
2964:bits
2960:base
2650:and
2566:and
2482:and
1795:and
1751:and
1740:and
1728:and
1565:and
1220:and
872:and
865:For
729:and
493:and
458:and
442:and
434:and
360:and
50:and
26575:doi
26542:doi
26435:doi
26400:PMC
26390:doi
26329:PMC
26319:doi
26231:doi
26188:doi
26139:doi
26127:224
26086:doi
26074:108
26036:doi
26024:106
25944:hdl
25934:doi
25835:doi
25778:doi
25741:doi
25644:).
25564:PMC
25554:doi
25505:doi
25325:doi
25180:doi
25138:doi
25015:).
24611:).
24370:0.5
24303:or
24093:PSI
24071:in
23847:In
23811:log
23793:log
23735:to
23344:).
23145:is
22710:is
22476:not
22363:to
22338:of
22330:'s
21519:MDI
21279:not
21277:is
20527:not
20441:log
20298:log
19936:log
19652:log
19264:In
19132:log
18981:of
18933:of
18636:log
18446:log
18350:log
18219:log
18148:log
18115:log
18078:lim
17912:of
17801:log
16491:log
16201:log
16152:log
16110:log
14762:lim
14535:log
14502:log
14446:log
14181:log
13944:log
12931:det
12916:det
12444:exp
12425:log
12375:exp
12356:log
12268:exp
12249:log
12111:exp
12078:exp
12018:exp
11985:exp
11915:exp
11802:exp
11769:exp
11630:sup
11613:exp
11594:log
11541:on
11499:. (
9108:log
8955:log
8022:is
7578:or
7271:log
7042:log
6881:log
5620:of
5577:of
5446:log
5384:log
4892:of
4793:of
4614:log
4593:log
3726:):
3223:0.4
3063:in
2762:log
2684:to
2426:of
2316:log
2082:on
2066:of
1877:log
1811:to
1799:is
1736:of
1645:log
1522:log
1497:lim
1142:log
995:log
917:to
462:in
185:log
30:In
26683::
26664:,
26660:,
26600:,
26581:.
26548:.
26538:40
26536:.
26508:}}
26504:{{
26492:.
26451:,
26441:,
26408:.
26398:.
26388:.
26380:.
26368:22
26366:.
26360:.
26337:.
26327:.
26317:.
26309:.
26297:21
26295:.
26289:.
26266:.
26227:28
26225:.
26221:.
26194:.
26186:.
26178:.
26166:13
26164:.
26160:.
26137:.
26125:.
26084:.
26072:.
26068:.
26042:.
26034:.
26022:.
26018:.
25967:.
25942:.
25930:10
25928:.
25922:.
25903:,
25891:^
25877:.
25863:^
25849:.
25841:.
25831:64
25829:.
25825:.
25784:.
25776:.
25764:51
25762:.
25737:36
25735:.
25624:42
25622:.
25599:.
25595:.
25572:.
25562:.
25552:.
25544:.
25532:22
25530:.
25526:.
25499:.
25474:.
25472:51
25331:.
25321:41
25319:.
25278:^
25263:^
25248:^
25227:^
25210:^
25198:MR
25196:.
25188:.
25176:22
25174:.
25168:.
25160:;
25132:.
25128:.
25116:^
24996:,
24971:.
24959:,
24953:,
24949:,
24943:,
24939:,
24719:KL
24673::
24634:,
24569:JS
24463:KL
24425:KL
24402:JS
24232:KL
24165:KL
24064:.
24020:KL
23992:KL
23938:.
23905:.
23864:KL
23778:Tr
23755:KL
23424:ln
23316:ln
22923:.
22891:ln
22763:23
22756:10
22737:ln
22674:ln
22607:.
22541:KL
22499:KL
22435:KL
22146:KL
21911:KL
21848:KL
21751:KL
21688:,
21565:KL
21407:KL
21326:KL
20942:KL
20921:.
20756:KL
20651:KL
20545:KL
20042:.
19835:KL
19558::
19436::
19276::
19256:.
19175:KL
19036:.
18941:.
18677:KL
18533:KL
18469:KL
18373:KL
18308:,
18242:KL
17916:,
17824:KL
17717:,
17695:is
17505:KL
17427:KL
17340:KL
17292:,
17280:.
17118:im
17103:KL
17022:.
16986:0.
15911:KL
15655:KL
15569:KL
15128:)
14960:.
14782:KL
14654:KL
14612:KL
14411:KL
14237:ln
14145:KL
13909:KL
13832:ln
13580:KL
13474:ln
13301:KL
13285:,
13025:.
12997:ln
12903:ln
12754:tr
12690:KL
12484:KL
12395:KL
12333:KL
12201:KL
12166:KL
11899:.
11676:KL
11302:.
10916:,
9999:KL
9887:KL
8797:KL
8731:KL
8723::=
8475:KL
8407:KL
8381:1.
8330:KL
8273:KL
8166:KL
7993:KL
7932:KL
7890:KL
7862:KL
6813:KL
6343:KL
6318:.
5965:KL
5910:KL
5897:,
5884:.
5868:.
5816:KL
5750:=:
5640:KL
5325:KL
5106:KL
5081:.
5059:.
4994:KL
4952:KL
4908:.
4872:.
4821:KL
4808:,
4801:.
4748:KL
4735:,
4723:.
4674:KL
4661:,
4517:ln
4496:ln
4475:ln
4434:25
4400:ln
4377:25
4369:12
4343:ln
4320:25
4286:ln
4220:ln
4161:KL
4099:ln
4096:50
4078:ln
4075:55
4057:ln
4054:32
4044:25
4006:25
3985:ln
3980:25
3949:25
3941:12
3928:ln
3923:25
3920:12
3892:25
3871:ln
3866:25
3805:ln
3746:KL
3716:ln
3675:KL
3633:KL
3484:25
3456:25
3453:12
3428:25
3380:2
3355:.
3269:,
3265:,
3235:.
3159:.
3105:KL
3080:to
3071:a
3054:to
2993:KL
2950:,
2704:KL
2220:KL
1831:KL
1744:.
1589:KL
1235:,
1228:.
1081:KL
937:KL
880:,
857:.
389:KL
348:.
340:,
336:,
127:KL
67:KL
42:)
40:KL
26617:.
26591:.
26577::
26556:.
26544::
26514:)
26500:.
26437::
26416:.
26392::
26384::
26374::
26345:.
26321::
26313::
26303::
26274:.
26239:.
26233::
26190::
26182::
26172::
26145:.
26141::
26133::
26092:.
26088::
26080::
26050:.
26038::
26030::
25983:.
25952:.
25946::
25936::
25885:.
25857:.
25837::
25792:.
25780::
25770::
25747:.
25743::
25700:.
25694::
25676:.
25652:.
25607:.
25580:.
25556::
25548::
25538::
25511:.
25507::
25501:2
25484:.
25439:.
25414:.
25339:.
25327::
25222:.
25204:.
25182::
25146:.
25140::
25134:3
24893:.
24886:)
24883:Q
24877:P
24874:(
24868:L
24865:K
24860:D
24852:e
24845:1
24837:)
24834:Q
24831:,
24828:P
24825:(
24808:1
24794:2
24788:)
24785:Q
24779:P
24776:(
24770:L
24767:K
24762:D
24741:.
24736:)
24733:Q
24727:P
24724:(
24715:D
24709:2
24706:1
24696:)
24693:Q
24690:,
24687:P
24684:(
24657:)
24654:q
24651:,
24648:p
24645:(
24593:f
24589:Q
24585:P
24565:D
24541:.
24538:)
24535:Q
24532:+
24529:P
24526:(
24521:2
24518:1
24513:=
24510:M
24497:M
24480:)
24477:M
24471:Q
24468:(
24459:D
24453:2
24450:1
24445:+
24442:)
24439:M
24433:P
24430:(
24421:D
24415:2
24412:1
24407:=
24398:D
24367:=
24335:1
24305:Q
24301:P
24297:X
24293:X
24276:,
24273:)
24270:Q
24267:)
24258:1
24255:(
24252:+
24249:P
24240:Q
24237:(
24228:D
24224:)
24215:1
24212:(
24209:+
24206:)
24203:Q
24200:)
24191:1
24188:(
24185:+
24182:P
24173:P
24170:(
24161:D
24154:=
24151:)
24148:Q
24142:P
24139:(
24130:D
24091:(
24081:Q
24077:P
24037:)
24034:P
24028:Q
24025:(
24016:D
24012:+
24009:)
24006:Q
24000:P
23997:(
23988:D
23903:P
23895:Q
23881:)
23878:Q
23872:P
23869:(
23860:D
23832:.
23829:)
23826:)
23823:)
23820:Q
23817:(
23805:)
23802:P
23799:(
23790:(
23787:P
23784:(
23775:=
23772:)
23769:Q
23763:P
23760:(
23751:D
23737:P
23733:Q
23721:Q
23717:P
23688:.
23684:]
23679:)
23672:o
23668:T
23664:T
23659:(
23650:2
23647:3
23642:+
23638:)
23631:o
23627:V
23623:V
23618:(
23610:[
23606:k
23603:N
23600:=
23597:I
23571:I
23563:o
23559:T
23555:=
23552:W
23530:o
23526:T
23503:o
23499:V
23476:o
23472:p
23451:)
23446:o
23442:p
23437:/
23433:p
23430:(
23421:k
23401:,
23398:0
23392:I
23364:o
23360:T
23328:0
23322:x
23310:1
23304:x
23301:=
23298:)
23295:x
23292:(
23267:o
23263:P
23258:/
23252:o
23248:T
23244:k
23241:N
23238:=
23233:o
23229:V
23208:)
23203:o
23199:V
23194:/
23190:V
23187:(
23179:o
23175:T
23171:k
23168:N
23165:=
23162:G
23156:=
23153:W
23131:o
23127:P
23104:o
23100:T
23075:S
23072:T
23066:V
23063:P
23060:+
23057:U
23054:=
23051:G
23038:P
23034:T
23030:S
23026:U
23012:S
23009:T
23003:U
22997:F
22984:N
22970:V
22967:,
22964:T
22944:A
22938:T
22928:T
22917:Z
22903:)
22900:Z
22897:(
22888:k
22879:A
22861:V
22857:P
22851:(
22849:S
22839:N
22835:N
22821:}
22818:K
22814:/
22810:J
22790:{
22769:}
22746:,
22743:2
22733:/
22729:1
22726:,
22723:1
22719:{
22708:k
22694:)
22691:p
22687:/
22683:1
22680:(
22671:k
22668:=
22665:s
22655:p
22632:o
22628:T
22595:)
22592:m
22589:,
22586:p
22583:(
22579:H
22558:)
22555:m
22549:p
22546:(
22537:D
22516:)
22513:m
22507:p
22504:(
22495:D
22484:p
22480:m
22472:p
22455:,
22452:)
22449:m
22443:p
22440:(
22431:D
22427:+
22424:)
22421:p
22418:(
22414:H
22410:=
22407:)
22404:m
22401:,
22398:p
22395:(
22391:H
22377:m
22373:p
22369:m
22365:p
22361:m
22321:a
22307:)
22304:a
22301:(
22298:u
22278:)
22275:a
22272:(
22269:u
22249:)
22246:a
22240:x
22237:(
22234:p
22231:=
22228:)
22225:a
22219:x
22216:(
22213:q
22193:)
22190:)
22187:a
22181:x
22178:(
22175:p
22169:)
22166:a
22160:x
22157:(
22154:q
22151:(
22142:D
22113:)
22110:a
22104:x
22101:(
22098:q
22078:)
22075:a
22069:x
22066:(
22063:p
22043:)
22040:a
22037:(
22034:u
22014:)
22011:a
22008:(
22005:u
21995:a
21981:)
21978:a
21975:(
21972:p
21949:,
21946:)
21943:)
21940:a
21937:(
21934:p
21928:)
21925:a
21922:(
21919:u
21916:(
21907:D
21903:+
21899:}
21895:)
21892:)
21889:a
21883:x
21880:(
21877:p
21871:)
21868:a
21862:x
21859:(
21856:q
21853:(
21844:D
21839:{
21830:)
21827:a
21824:(
21821:u
21817:E
21813:=
21810:)
21807:)
21804:a
21801:,
21798:x
21795:(
21792:p
21786:)
21783:a
21780:(
21777:u
21774:)
21771:a
21765:x
21762:(
21759:q
21756:(
21747:D
21723:)
21720:a
21717:(
21714:u
21711:)
21708:a
21702:x
21699:(
21696:q
21686:a
21682:x
21668:)
21665:a
21662:(
21659:u
21649:a
21645:a
21641:x
21627:)
21624:a
21621:,
21618:x
21615:(
21612:p
21589:)
21584:0
21580:f
21573:f
21570:(
21561:D
21538:0
21534:f
21523:f
21517:(
21457:.
21454:)
21451:)
21448:I
21442:H
21439:(
21436:p
21430:)
21427:x
21421:H
21418:(
21415:p
21412:(
21403:D
21399:=
21396:G
21393:I
21387:)
21384:)
21379:0
21375:H
21368:x
21365:(
21362:p
21356:)
21351:1
21347:H
21340:x
21337:(
21334:p
21331:(
21322:D
21298:)
21295:H
21292:(
21289:p
21263:0
21259:H
21236:1
21232:H
21206:0
21202:H
21179:1
21175:H
21148:1
21144:H
21121:0
21117:H
21094:1
21090:H
21067:0
21063:H
21040:1
21036:H
21009:)
21004:)
20999:0
20995:H
20988:x
20985:(
20982:p
20976:)
20971:1
20967:H
20960:x
20957:(
20954:p
20949:(
20938:D
20892:)
20889:I
20886:,
20883:x
20880:,
20875:1
20871:y
20862:2
20858:y
20854:(
20851:p
20822:)
20817:)
20814:I
20808:x
20805:(
20802:p
20796:)
20793:I
20790:,
20785:1
20781:y
20774:x
20771:(
20768:p
20763:(
20752:D
20748:+
20743:)
20738:)
20735:I
20732:,
20727:1
20723:y
20716:x
20713:(
20710:p
20704:)
20701:I
20698:,
20693:2
20689:y
20685:,
20680:1
20676:y
20669:x
20666:(
20663:p
20658:(
20647:D
20624:)
20619:)
20616:I
20610:x
20607:(
20604:p
20598:)
20595:I
20592:,
20587:2
20583:y
20579:,
20574:1
20570:y
20563:x
20560:(
20557:p
20552:(
20541:D
20508:)
20502:)
20499:I
20493:x
20490:(
20487:p
20482:)
20479:I
20476:,
20471:1
20467:y
20460:x
20457:(
20454:p
20448:(
20438:)
20435:I
20432:,
20427:1
20423:y
20416:x
20413:(
20410:p
20405:x
20378:)
20372:)
20369:I
20363:x
20360:(
20357:p
20352:)
20349:I
20346:,
20341:2
20337:y
20333:,
20328:1
20324:y
20317:x
20314:(
20311:p
20305:(
20295:)
20292:I
20289:,
20284:2
20280:y
20276:,
20271:1
20267:y
20260:x
20257:(
20254:p
20249:x
20221:)
20218:I
20212:x
20209:(
20206:p
20186:)
20183:I
20180:,
20175:1
20171:y
20164:x
20161:(
20158:p
20138:)
20135:I
20132:,
20127:2
20123:y
20119:,
20114:1
20110:y
20103:x
20100:(
20097:p
20087:x
20071:2
20067:y
20063:=
20058:2
20054:Y
20030:y
20027:=
20024:Y
20014:X
19996:)
19990:)
19987:I
19981:x
19978:(
19975:p
19970:)
19967:I
19964:,
19961:y
19955:x
19952:(
19949:p
19943:(
19933:)
19930:I
19927:,
19924:y
19918:x
19915:(
19912:p
19907:x
19899:=
19894:)
19889:)
19886:I
19880:x
19877:(
19874:p
19868:)
19865:I
19862:,
19859:y
19853:x
19850:(
19847:p
19842:(
19831:D
19807:)
19804:I
19801:,
19798:y
19792:x
19789:(
19786:p
19766:)
19763:I
19757:x
19754:(
19751:p
19731:)
19728:)
19725:I
19719:x
19716:(
19713:p
19710:(
19706:H
19682:,
19679:)
19676:I
19673:,
19670:y
19664:x
19661:(
19658:p
19649:)
19646:I
19643:,
19640:y
19634:x
19631:(
19628:p
19623:x
19612:=
19607:)
19602:)
19599:I
19596:,
19593:y
19587:x
19584:(
19581:p
19576:(
19570:H
19536:)
19533:I
19527:y
19524:(
19521:p
19516:)
19513:I
19507:x
19504:(
19501:p
19498:)
19495:I
19492:,
19489:x
19483:y
19480:(
19477:p
19471:=
19468:)
19465:I
19462:,
19459:y
19453:x
19450:(
19447:p
19420:)
19417:I
19414:,
19411:y
19405:x
19402:(
19399:p
19379:)
19376:I
19370:x
19367:(
19364:p
19354:X
19340:y
19337:=
19334:Y
19314:)
19311:I
19305:x
19302:(
19299:p
19293:)
19290:x
19287:(
19284:p
19254:p
19250:q
19236:)
19233:p
19230:(
19226:H
19195:.
19192:)
19189:q
19183:p
19180:(
19171:D
19167:+
19164:)
19161:p
19158:(
19154:H
19150:=
19147:]
19144:)
19141:q
19138:(
19126:[
19118:p
19114:E
19110:=
19107:)
19104:q
19101:,
19098:p
19095:(
19091:H
19073:q
19069:p
19065:p
19061:q
19053:q
19049:p
19047:(
19034:q
19030:p
19018:q
19004:)
19001:p
18998:(
18994:H
18983:p
18975:p
18967:p
18951:p
18939:Y
18935:X
18921:)
18918:Y
18914:|
18910:X
18907:(
18904:P
18884:)
18881:X
18878:(
18873:U
18869:P
18858:X
18840:)
18837:Y
18834:,
18831:X
18828:(
18825:P
18805:)
18802:Y
18799:(
18796:P
18793:)
18790:X
18787:(
18782:U
18778:P
18763:N
18759:X
18737:]
18732:)
18728:)
18725:X
18722:(
18717:U
18713:P
18705:)
18701:Y
18695:X
18691:(
18687:P
18683:(
18673:D
18668:[
18659:Y
18655:E
18648:)
18645:N
18642:(
18633:=
18623:)
18620:Y
18617:;
18614:X
18611:(
18605:I
18599:)
18596:X
18593:(
18589:H
18585:=
18575:)
18572:)
18569:X
18566:(
18561:U
18557:P
18550:)
18547:X
18544:(
18541:P
18538:(
18529:D
18522:)
18519:)
18516:Y
18513:(
18510:P
18507:)
18504:X
18501:(
18498:P
18492:)
18489:Y
18486:,
18483:X
18480:(
18477:P
18474:(
18465:D
18458:)
18455:N
18452:(
18443:=
18433:)
18430:)
18427:Y
18424:(
18421:P
18418:)
18415:X
18412:(
18407:U
18403:P
18396:)
18393:Y
18390:,
18387:X
18384:(
18381:P
18378:(
18369:D
18362:)
18359:N
18356:(
18347:=
18340:)
18337:Y
18331:X
18328:(
18324:H
18284:)
18281:)
18278:x
18275:(
18272:m
18268:|
18263:|
18259:)
18256:x
18253:(
18250:p
18247:(
18238:D
18231:)
18228:N
18225:(
18193:,
18190:x
18187:d
18180:)
18177:x
18174:(
18171:m
18166:)
18163:x
18160:(
18157:p
18145:)
18142:x
18139:(
18136:p
18127:)
18124:N
18121:(
18112:=
18109:)
18106:X
18103:(
18098:N
18094:H
18082:N
18042:)
18039:X
18036:(
18033:P
18013:)
18010:X
18007:(
18002:U
17998:P
17987:X
17969:)
17966:X
17963:(
17960:P
17940:)
17937:X
17934:(
17929:U
17925:P
17914:X
17902:N
17898:X
17876:)
17872:)
17869:X
17866:(
17861:U
17857:P
17850:)
17847:x
17844:(
17839:X
17835:p
17830:(
17820:D
17813:)
17810:N
17807:(
17798:=
17787:]
17783:)
17780:x
17777:(
17769:X
17765:I
17760:[
17753:E
17750:=
17743:)
17740:X
17737:(
17733:H
17703:X
17699:Y
17682:)
17679:Y
17676:,
17673:X
17670:(
17667:P
17657:Y
17653:X
17635:)
17632:Y
17629:(
17626:P
17623:)
17620:X
17617:(
17614:P
17594:)
17591:Y
17588:,
17585:X
17582:(
17579:P
17549:}
17546:)
17543:)
17540:X
17537:(
17534:P
17528:)
17525:Y
17519:X
17516:(
17513:P
17510:(
17501:D
17497:{
17489:Y
17485:E
17481:=
17471:}
17468:)
17465:)
17462:Y
17459:(
17456:P
17450:)
17447:X
17441:Y
17438:(
17435:P
17432:(
17423:D
17419:{
17411:X
17407:E
17403:=
17393:)
17390:)
17387:Y
17384:(
17381:P
17378:)
17375:X
17372:(
17369:P
17363:)
17360:Y
17357:,
17354:X
17351:(
17348:P
17345:(
17336:D
17332:=
17325:)
17322:Y
17319:;
17316:X
17313:(
17307:I
17268:m
17265:=
17262:i
17242:)
17239:i
17236:(
17233:P
17223:i
17209:m
17206:=
17203:i
17179:)
17176:i
17173:(
17170:P
17147:,
17143:)
17139:}
17134:i
17130:p
17126:{
17109:(
17099:D
17095:=
17092:)
17089:m
17086:(
17079:I
16971:)
16966:3
16958:(
16952:O
16947:+
16942:2
16936:2
16928:c
16922:=
16919:)
16916:)
16910:,
16907:x
16904:(
16901:p
16895:)
16892:0
16889:,
16886:x
16883:(
16880:p
16877:(
16872:D
16844:0
16841:=
16838:x
16835:d
16829:0
16826:=
16818:|
16809:2
16796:)
16790:,
16787:x
16784:(
16781:p
16776:2
16758:0
16755:=
16752:x
16739:=
16736:x
16733:d
16727:0
16724:=
16716:|
16701:)
16695:,
16692:x
16689:(
16686:p
16670:0
16667:=
16664:x
16642:ρ
16638:ξ
16605:x
16602:d
16598:)
16595:x
16592:(
16589:H
16586:)
16583:0
16580:,
16577:x
16574:(
16571:p
16561:0
16558:=
16555:x
16547::
16544:)
16541:x
16538:(
16535:H
16524:|
16516:3
16503:)
16500:p
16497:(
16486:3
16475:|
16461:x
16458:d
16454:)
16451:x
16448:(
16445:G
16435:0
16432:=
16429:x
16421::
16418:)
16415:x
16412:(
16409:G
16398:|
16390:2
16377:p
16372:2
16361:|
16353:,
16344:x
16341:d
16337:)
16334:x
16331:(
16328:F
16318:0
16315:=
16312:x
16304::
16301:)
16298:x
16295:(
16292:F
16281:|
16267:p
16258:|
16226:3
16213:)
16210:p
16207:(
16196:3
16185:,
16177:2
16164:)
16161:p
16158:(
16147:2
16136:,
16122:)
16119:p
16116:(
16079:)
16073:,
16070:x
16067:(
16063:p
16044:θ
16026:)
16020:(
16015:k
16012:j
16008:g
15985:0
15953:)
15950:)
15945:0
15937:(
15934:P
15928:)
15922:(
15919:P
15916:(
15907:D
15899:0
15891:=
15883:|
15875:k
15861:j
15847:2
15833:=
15830:)
15825:0
15817:(
15812:k
15809:j
15805:g
15771:+
15768:)
15763:0
15755:(
15750:k
15747:j
15743:g
15736:k
15722:j
15708:2
15705:1
15700:=
15697:)
15694:)
15689:0
15681:(
15678:P
15672:)
15666:(
15663:P
15660:(
15651:D
15620:,
15617:0
15614:=
15611:)
15608:)
15603:0
15595:(
15592:P
15586:)
15580:(
15577:P
15574:(
15565:D
15557:0
15549:=
15541:|
15533:j
15490:j
15456:0
15448:=
15425:Q
15422:=
15419:P
15399:)
15394:0
15386:(
15378:j
15365:P
15356:=
15352:)
15347:0
15339:(
15333:j
15329:P
15318:j
15282:j
15278:)
15272:0
15258:(
15255:=
15250:j
15216:+
15213:)
15208:0
15200:(
15195:j
15191:P
15184:j
15173:+
15170:)
15165:0
15157:(
15154:P
15151:=
15148:)
15142:(
15139:P
15110:0
15065:)
15060:0
15052:(
15049:P
15046:=
15043:Q
15023:)
15017:(
15014:P
15011:=
15008:P
14978:Q
14974:P
14952:,
14940:P
14935:V
14932:T
14922:n
14918:P
14911:P
14906:D
14896:n
14892:P
14873:.
14861:Q
14855:D
14844:n
14840:P
14824:,
14812:0
14809:=
14806:)
14803:Q
14795:n
14791:P
14787:(
14778:D
14766:n
14738:}
14732:,
14727:2
14723:P
14719:,
14714:1
14710:P
14706:{
14671:)
14668:P
14662:Q
14659:(
14650:D
14629:)
14626:Q
14620:P
14617:(
14608:D
14561:k
14547:k
14539:2
14514:k
14506:2
14491:k
14472:A
14466:B
14461:C
14455:D
14443:=
14439:)
14433:q
14423:p
14417:(
14407:D
14384:D
14378:B
14372:A
14366:C
14358:(
14346:]
14343:D
14340:,
14337:C
14334:[
14331:=
14328:q
14308:]
14305:B
14302:,
14299:A
14296:[
14293:=
14290:p
14262:s
14259:t
14256:i
14253:b
14249:)
14246:2
14243:(
14233:/
14229:2
14225:/
14221:)
14218:1
14210:2
14203:k
14199:(
14196:+
14193:k
14185:2
14177:=
14173:)
14167:q
14157:p
14151:(
14141:D
14116:0
14107:/
14101:1
14093:=
14090:k
14065:2
14062:1
14049:2
14044:1
14036:2
14029:2
14025:)
14019:1
14006:0
13998:(
13995:+
13990:2
13985:0
13974:+
13967:0
13957:1
13941:=
13937:)
13931:q
13921:p
13915:(
13905:D
13890:q
13884:p
13866:.
13862:)
13857:)
13852:2
13847:i
13839:(
13826:1
13818:2
13813:i
13805:+
13800:2
13795:i
13786:(
13780:k
13775:1
13772:=
13769:i
13759:2
13756:1
13751:=
13747:)
13742:)
13737:I
13733:,
13729:0
13724:(
13718:N
13709:)
13704:)
13698:2
13693:k
13685:,
13679:,
13674:2
13669:1
13660:(
13650:,
13644:T
13638:)
13632:k
13624:,
13618:,
13613:1
13604:(
13598:(
13592:N
13586:(
13576:D
13545:.
13541:)
13532:i
13529:i
13525:)
13519:0
13515:L
13511:(
13504:i
13501:i
13497:)
13491:1
13487:L
13483:(
13469:k
13464:1
13461:=
13458:i
13450:2
13447:+
13442:2
13437:|
13432:y
13428:|
13424:+
13421:k
13413:2
13409:)
13403:j
13400:i
13396:M
13392:(
13387:k
13382:1
13379:=
13376:j
13373:,
13370:i
13361:(
13355:2
13352:1
13347:=
13343:)
13337:1
13331:N
13320:0
13314:N
13307:(
13297:D
13271:0
13258:1
13250:=
13247:y
13242:1
13238:L
13215:0
13211:L
13207:=
13204:M
13199:1
13195:L
13184:y
13180:M
13164:T
13159:1
13155:L
13149:1
13145:L
13141:=
13136:1
13109:T
13104:0
13100:L
13094:0
13090:L
13086:=
13081:0
13054:1
13050:L
13046:,
13041:0
13037:L
13009:)
13006:2
13003:(
12983:e
12978:e
12956:.
12952:)
12947:)
12939:0
12924:1
12910:(
12900:+
12896:)
12890:0
12877:1
12868:(
12862:1
12854:1
12843:T
12837:)
12831:0
12818:1
12809:(
12804:+
12801:k
12794:)
12788:0
12778:1
12770:1
12761:(
12750:(
12744:2
12741:1
12736:=
12732:)
12726:1
12720:N
12709:0
12703:N
12696:(
12686:D
12672:k
12658:.
12653:1
12645:,
12640:0
12610:1
12602:,
12597:0
12541:Q
12537:=
12534:Q
12514:0
12508:)
12499:Q
12492:Q
12489:(
12480:D
12456:,
12453:]
12450:h
12441:[
12436:P
12432:E
12419:)
12410:Q
12403:Q
12400:(
12391:D
12384:]
12381:h
12372:[
12367:P
12363:E
12353:=
12350:)
12347:P
12341:Q
12338:(
12329:D
12322:]
12319:h
12316:[
12311:Q
12307:E
12280:.
12277:]
12274:h
12265:[
12260:P
12256:E
12246:+
12243:]
12240:h
12237:[
12232:Q
12228:E
12221:=
12218:)
12215:P
12209:Q
12206:(
12197:D
12190:)
12181:Q
12174:Q
12171:(
12162:D
12138:)
12132:d
12129:(
12126:P
12120:]
12117:h
12108:[
12103:P
12099:E
12093:)
12087:(
12084:h
12072:=
12069:)
12063:d
12060:(
12051:Q
12027:]
12024:h
12015:[
12010:P
12006:E
12000:)
11994:(
11991:h
11972:P
11952:Q
11941:P
11927:)
11924:h
11921:(
11897:P
11893:Q
11876:)
11870:d
11867:(
11864:P
11859:)
11853:d
11850:(
11847:Q
11834:P
11817:,
11811:]
11808:h
11799:[
11794:P
11790:E
11784:)
11778:(
11775:h
11763:=
11757:)
11751:d
11748:(
11745:P
11740:)
11734:d
11731:(
11728:Q
11699:.
11696:}
11693:)
11690:P
11684:Q
11681:(
11672:D
11665:]
11662:h
11659:[
11654:Q
11650:E
11646:{
11641:P
11635:Q
11625:=
11622:]
11619:h
11610:[
11605:P
11601:E
11571:)
11568:P
11565:,
11560:F
11555:,
11549:(
11535:h
11531:P
11527:Q
11513:P
11507:Q
11487:P
11481:Q
11461:)
11458:Q
11455:,
11450:F
11445:,
11439:(
11419:)
11416:P
11413:,
11408:F
11403:,
11397:(
11384:Q
11380:P
11364:F
11267:n
11245:n
11240:)
11234:U
11226:+
11223:1
11219:(
11208:)
11205:1
11199:n
11196:(
11193:n
11189:1
11167:n
11144:+
11141:Q
11138:2
11132:P
11092:Q
11089:2
11083:P
11057:1
11017:U
11011:Q
10988:+
10985:Q
10982:2
10976:P
10950:U
10930:0
10904:0
10878:0
10858:Q
10855:2
10849:P
10829:Q
10826:2
10820:P
10797:1
10794:=
10781:)
10778:1
10772:n
10769:(
10766:n
10762:1
10750:2
10747:=
10744:n
10725:|
10721:)
10718:x
10715:(
10712:P
10706:)
10703:x
10700:(
10697:Q
10693:|
10685:X
10677:x
10666:)
10663:1
10657:n
10654:(
10651:n
10647:1
10635:2
10632:=
10629:n
10609:1
10603:n
10598:|
10591:)
10588:x
10585:(
10582:Q
10577:)
10574:x
10571:(
10568:P
10559:1
10555:|
10549:|
10545:)
10542:x
10539:(
10536:P
10530:)
10527:x
10524:(
10521:Q
10517:|
10509:X
10501:x
10490:)
10487:1
10481:n
10478:(
10475:n
10471:1
10459:2
10456:=
10453:n
10445:=
10437:|
10428:1
10422:n
10418:)
10414:x
10411:(
10408:Q
10401:n
10397:)
10393:)
10390:x
10387:(
10384:P
10378:)
10375:x
10372:(
10369:Q
10366:(
10356:X
10348:x
10337:)
10334:1
10328:n
10325:(
10322:n
10318:1
10312:|
10301:2
10298:=
10295:n
10266:Q
10263:2
10257:P
10229:1
10223:n
10219:)
10215:x
10212:(
10209:Q
10202:n
10198:)
10194:)
10191:x
10188:(
10185:P
10179:)
10176:x
10173:(
10170:Q
10167:(
10157:X
10149:x
10138:)
10135:1
10129:n
10126:(
10123:n
10119:1
10107:2
10104:=
10101:n
10093:=
10080:!
10077:n
10072:)
10069:0
10066:(
10061:)
10058:n
10055:(
10051:f
10037:0
10034:=
10031:n
10023:=
10016:)
10013:Q
10007:P
10004:(
9995:D
9970:1
9967:=
9944:0
9924:f
9904:)
9901:Q
9895:P
9892:(
9883:D
9853:1
9847:n
9843:)
9839:x
9836:(
9833:Q
9826:n
9822:)
9818:)
9815:x
9812:(
9809:Q
9803:)
9800:x
9797:(
9794:P
9791:(
9781:X
9773:x
9765:!
9762:)
9759:2
9753:n
9750:(
9745:n
9741:)
9737:1
9731:(
9728:=
9721:)
9718:0
9715:(
9710:)
9707:n
9704:(
9700:f
9688:1
9682:n
9677:)
9673:)
9670:x
9667:(
9664:P
9658:+
9655:)
9652:x
9649:(
9646:Q
9643:)
9634:1
9631:(
9627:(
9619:n
9615:)
9611:)
9608:x
9605:(
9602:Q
9596:)
9593:x
9590:(
9587:P
9584:(
9574:X
9566:x
9558:!
9555:)
9552:2
9546:n
9543:(
9538:n
9534:)
9530:1
9524:(
9521:=
9514:)
9508:(
9503:)
9500:n
9497:(
9493:f
9482:)
9479:x
9476:(
9473:Q
9466:2
9462:)
9458:)
9455:x
9452:(
9449:Q
9443:)
9440:x
9437:(
9434:P
9431:(
9421:X
9413:x
9405:=
9398:)
9395:0
9392:(
9385:f
9374:)
9371:x
9368:(
9365:P
9359:+
9356:)
9353:x
9350:(
9347:Q
9344:)
9335:1
9332:(
9325:2
9321:)
9317:)
9314:x
9311:(
9308:Q
9302:)
9299:x
9296:(
9293:P
9290:(
9280:X
9272:x
9264:=
9257:)
9251:(
9244:f
9215:0
9212:=
9205:)
9202:0
9199:(
9192:f
9183:)
9177:)
9174:x
9171:(
9168:Q
9163:)
9160:x
9157:(
9154:P
9148:+
9145:)
9142:x
9139:(
9136:Q
9133:)
9124:1
9121:(
9115:(
9105:)
9102:)
9099:x
9096:(
9093:Q
9087:)
9084:x
9081:(
9078:P
9075:(
9068:X
9060:x
9052:=
9041:)
9037:1
9034:+
9030:)
9024:)
9021:x
9018:(
9015:Q
9010:)
9007:x
9004:(
9001:P
8995:+
8992:)
8989:x
8986:(
8983:Q
8980:)
8971:1
8968:(
8962:(
8951:(
8947:)
8944:)
8941:x
8938:(
8935:Q
8929:)
8926:x
8923:(
8920:P
8917:(
8910:X
8902:x
8894:=
8887:)
8881:(
8874:f
8849:f
8829:)
8826:1
8823:(
8820:f
8817:=
8814:)
8811:Q
8805:P
8802:(
8793:D
8772:)
8769:Q
8763:P
8757:+
8754:Q
8751:)
8742:1
8739:(
8736:(
8727:D
8720:)
8714:(
8711:f
8697:.
8685:Q
8665:Q
8662:2
8656:P
8631:1
8625:n
8621:)
8617:x
8614:(
8611:Q
8604:n
8600:)
8596:)
8593:x
8590:(
8587:P
8581:)
8578:x
8575:(
8572:Q
8569:(
8559:X
8551:x
8540:)
8537:1
8531:n
8528:(
8525:n
8521:1
8509:2
8506:=
8503:n
8495:=
8492:)
8489:Q
8483:P
8480:(
8471:D
8450:Q
8447:=
8444:P
8424:)
8421:Q
8415:P
8412:(
8403:D
8369:0
8361:)
8356:2
8352:Q
8343:2
8339:P
8335:(
8326:D
8322:)
8313:1
8310:(
8307:+
8304:)
8299:1
8295:Q
8286:1
8282:P
8278:(
8269:D
8259:)
8254:2
8250:Q
8246:)
8237:1
8234:(
8231:+
8226:1
8222:Q
8210:2
8206:P
8202:)
8193:1
8190:(
8187:+
8182:1
8178:P
8171:(
8162:D
8141:)
8136:2
8132:Q
8128:,
8123:2
8119:P
8115:(
8095:)
8090:1
8086:Q
8082:,
8077:1
8073:P
8069:(
8049:)
8046:Q
8043:,
8040:P
8037:(
8010:)
8007:Q
8001:P
7998:(
7989:D
7966:.
7963:)
7958:2
7954:Q
7945:2
7941:P
7937:(
7928:D
7924:+
7921:)
7916:1
7912:Q
7903:1
7899:P
7895:(
7886:D
7882:=
7879:)
7876:Q
7870:P
7867:(
7858:D
7835:2
7831:Q
7827:,
7822:1
7818:Q
7797:)
7794:y
7791:d
7788:(
7783:2
7779:Q
7775:)
7772:x
7769:d
7766:(
7761:1
7757:Q
7753:=
7750:)
7747:y
7744:d
7741:,
7738:x
7735:d
7732:(
7729:Q
7709:)
7706:y
7703:d
7700:(
7695:2
7691:P
7687:)
7684:x
7681:d
7678:(
7673:1
7669:P
7665:=
7662:)
7659:y
7656:d
7653:,
7650:x
7647:d
7644:(
7641:P
7619:2
7615:P
7611:,
7606:1
7602:P
7561:x
7558:d
7554:)
7551:x
7548:(
7545:p
7542:=
7539:)
7536:x
7533:d
7530:(
7527:P
7507:)
7504:x
7501:(
7498:q
7478:)
7475:x
7472:(
7469:p
7459:x
7441:)
7436:b
7432:x
7428:(
7425:y
7422:=
7417:b
7413:y
7392:)
7387:a
7383:x
7379:(
7376:y
7373:=
7368:a
7364:y
7339:y
7336:d
7331:)
7325:)
7322:y
7319:(
7310:q
7302:)
7299:y
7296:(
7287:p
7278:(
7268:)
7265:y
7262:(
7253:p
7243:b
7239:y
7231:a
7227:y
7218:=
7208:x
7205:d
7200:)
7193:|
7189:)
7186:x
7183:(
7177:x
7174:d
7169:y
7166:d
7159:|
7154:)
7151:)
7148:x
7145:(
7142:y
7139:(
7130:q
7121:|
7117:)
7114:x
7111:(
7105:x
7102:d
7097:y
7094:d
7087:|
7082:)
7079:)
7076:x
7073:(
7070:y
7067:(
7058:p
7049:(
7038:|
7034:)
7031:x
7028:(
7022:x
7019:d
7014:y
7011:d
7004:|
7000:)
6997:)
6994:x
6991:(
6988:y
6985:(
6976:p
6966:b
6962:x
6954:a
6950:x
6941:=
6931:x
6928:d
6923:)
6917:)
6914:x
6911:(
6908:q
6903:)
6900:x
6897:(
6894:p
6888:(
6878:)
6875:x
6872:(
6869:p
6862:b
6858:x
6850:a
6846:x
6837:=
6830:)
6827:Q
6821:P
6818:(
6809:D
6779:|
6775:)
6772:x
6769:(
6762:x
6759:d
6754:y
6751:d
6743:|
6722:x
6719:d
6715:|
6711:)
6708:x
6705:(
6698:x
6695:d
6690:y
6687:d
6679:|
6675:)
6672:y
6669:(
6660:q
6654:=
6651:y
6648:d
6644:)
6641:y
6638:(
6629:q
6623:=
6620:x
6617:d
6613:)
6610:x
6607:(
6604:q
6601:=
6598:)
6595:x
6592:d
6589:(
6586:Q
6566:x
6563:d
6558:|
6554:)
6551:x
6548:(
6541:x
6538:d
6533:y
6530:d
6522:|
6518:)
6515:)
6512:x
6509:(
6506:y
6503:(
6494:p
6488:=
6485:y
6482:d
6478:)
6475:y
6472:(
6463:p
6457:=
6454:x
6451:d
6447:)
6444:x
6441:(
6438:p
6435:=
6432:)
6429:x
6426:d
6423:(
6420:P
6400:)
6397:x
6394:(
6391:y
6381:x
6360:)
6357:Q
6351:P
6348:(
6339:D
6328:Q
6324:P
6316:P
6312:Q
6308:X
6304:x
6300:P
6296:Q
6274:)
6271:Q
6268:,
6265:P
6262:(
6258:H
6237:)
6234:P
6231:(
6227:H
6214:-
6183:)
6180:x
6177:(
6174:q
6171:=
6168:)
6165:x
6162:(
6159:p
6139:)
6136:x
6133:d
6130:(
6124:)
6121:x
6118:(
6115:q
6112:=
6109:)
6106:x
6103:d
6100:(
6097:Q
6077:)
6074:x
6071:d
6068:(
6062:)
6059:x
6056:(
6053:p
6050:=
6047:)
6044:x
6041:d
6038:(
6035:P
6011:Q
6008:=
6005:P
5982:)
5979:Q
5973:P
5970:(
5961:D
5936:,
5933:0
5927:)
5924:Q
5918:P
5915:(
5906:D
5833:)
5830:Q
5824:P
5821:(
5812:D
5791:)
5788:P
5785:(
5782:H
5762:)
5759:P
5756:(
5753:H
5747:)
5744:P
5741:,
5738:P
5735:(
5732:H
5708:)
5705:Q
5702:,
5699:P
5696:(
5693:H
5679:P
5675:Q
5657:)
5654:Q
5648:P
5645:(
5636:D
5622:P
5604:)
5601:P
5598:(
5594:H
5583:Q
5579:P
5561:)
5558:Q
5555:,
5552:P
5549:(
5545:H
5517:)
5514:P
5511:(
5507:H
5500:)
5497:Q
5494:,
5491:P
5488:(
5484:H
5480:=
5467:)
5464:x
5461:(
5458:p
5454:1
5443:)
5440:x
5437:(
5434:p
5427:X
5419:x
5405:)
5402:x
5399:(
5396:q
5392:1
5381:)
5378:x
5375:(
5372:p
5365:X
5357:x
5349:=
5342:)
5339:Q
5333:P
5330:(
5321:D
5299:P
5295:Q
5279:i
5275:x
5252:i
5237:X
5219:i
5207:2
5203:=
5200:)
5195:i
5191:x
5187:(
5184:q
5174:X
5158:i
5154:x
5102:D
5069:f
5011:)
5008:P
5002:Q
4999:(
4990:D
4969:)
4966:Q
4960:P
4957:(
4948:D
4902:P
4898:Q
4894:P
4886:Q
4882:P
4870:P
4866:Q
4862:P
4855:Q
4838:)
4835:Q
4829:P
4826:(
4817:D
4799:Q
4795:P
4787:Q
4783:P
4765:)
4762:Q
4756:P
4753:(
4744:D
4721:P
4717:Q
4713:P
4691:)
4688:Q
4682:P
4679:(
4670:D
4647:P
4643:Y
4629:)
4626:Y
4623:(
4620:Q
4608:)
4605:Y
4602:(
4599:P
4583:Y
4579:Q
4575:P
4533:)
4529:)
4526:5
4523:(
4514:6
4511:+
4508:)
4505:3
4502:(
4493:6
4487:)
4484:2
4481:(
4472:4
4465:(
4459:3
4456:1
4451:=
4440:)
4430:/
4426:4
4421:3
4417:/
4413:1
4407:(
4395:3
4392:1
4387:+
4383:)
4373:/
4364:3
4360:/
4356:1
4350:(
4338:3
4335:1
4330:+
4326:)
4316:/
4312:9
4307:3
4303:/
4299:1
4293:(
4281:3
4278:1
4273:=
4262:)
4256:)
4253:x
4250:(
4247:P
4242:)
4239:x
4236:(
4233:Q
4227:(
4217:)
4214:x
4211:(
4208:Q
4201:X
4193:x
4185:=
4178:)
4175:P
4169:Q
4166:(
4157:D
4125:,
4115:)
4111:)
4108:5
4105:(
4090:)
4087:3
4084:(
4072:+
4069:)
4066:2
4063:(
4050:(
4041:1
4036:=
4025:)
4019:3
4015:/
4011:1
4002:/
3998:4
3992:(
3977:4
3972:+
3968:)
3962:3
3958:/
3954:1
3945:/
3935:(
3915:+
3911:)
3905:3
3901:/
3897:1
3888:/
3884:9
3878:(
3863:9
3858:=
3847:)
3841:)
3838:x
3835:(
3832:Q
3827:)
3824:x
3821:(
3818:P
3812:(
3802:)
3799:x
3796:(
3793:P
3786:X
3778:x
3770:=
3763:)
3760:Q
3754:P
3751:(
3742:D
3711:e
3692:)
3689:P
3683:Q
3680:(
3671:D
3650:)
3647:Q
3641:P
3638:(
3629:D
3601:3
3598:1
3573:3
3570:1
3545:3
3542:1
3519:)
3516:x
3513:(
3510:Q
3481:4
3425:9
3402:)
3399:x
3396:(
3393:P
3377:1
3374:0
3370:x
3343:3
3339:/
3335:1
3332:=
3329:p
3309:}
3306:2
3303:,
3300:1
3297:,
3294:0
3291:{
3288:=
3283:X
3271:2
3267:1
3263:0
3250:=
3247:x
3237:Q
3220:=
3217:p
3197:2
3194:=
3191:N
3177:P
3173:Q
3169:P
3157:Q
3153:P
3145:Q
3139:P
3122:)
3119:Q
3113:P
3110:(
3101:D
3087:P
3076:Q
3057:P
3051:Q
3044:Q
3038:P
3034:Q
3030:P
3010:)
3007:Q
3001:P
2998:(
2989:D
2968:e
2918:)
2914:Q
2911:+
2908:P
2904:(
2898:2
2895:1
2890:=
2841:.
2835:)
2832:x
2824:d
2820:(
2810:)
2804:)
2801:x
2798:(
2795:q
2787:)
2784:x
2781:(
2778:p
2769:(
2756:)
2753:x
2750:(
2747:p
2740:X
2732:x
2724:=
2721:)
2718:Q
2712:P
2709:(
2700:D
2686:P
2682:Q
2652:Q
2648:P
2631:)
2628:x
2620:d
2616:(
2610:)
2607:x
2604:(
2601:q
2598:=
2595:)
2592:x
2584:d
2580:(
2577:Q
2551:)
2548:x
2540:d
2536:(
2530:)
2527:x
2524:(
2521:p
2518:=
2515:)
2512:x
2504:d
2500:(
2497:P
2484:q
2480:p
2464:X
2432:Q
2428:P
2407:,
2401:)
2398:x
2390:d
2386:(
2383:Q
2376:)
2370:)
2367:x
2359:d
2355:(
2352:Q
2347:)
2344:x
2336:d
2332:(
2329:P
2323:(
2307:)
2304:x
2296:d
2292:(
2289:Q
2284:)
2281:x
2273:d
2269:(
2266:P
2256:X
2248:x
2240:=
2237:)
2234:Q
2228:P
2225:(
2216:D
2198:Q
2194:P
2177:)
2174:x
2166:d
2162:(
2159:Q
2156:)
2153:x
2150:(
2147:r
2144:=
2141:)
2138:x
2130:d
2126:(
2123:P
2095:X
2080:r
2076:Q
2072:Q
2068:P
2044:)
2041:x
2033:d
2029:(
2026:Q
2018:)
2015:x
2007:d
2003:(
2000:P
1968:,
1962:)
1959:x
1951:d
1947:(
1944:P
1937:)
1931:)
1928:x
1920:d
1916:(
1913:Q
1908:)
1905:x
1897:d
1893:(
1890:P
1884:(
1867:X
1859:x
1851:=
1848:)
1845:Q
1839:P
1836:(
1827:D
1813:P
1809:Q
1805:Q
1797:P
1783:,
1775:X
1753:Q
1749:P
1742:Q
1738:P
1730:q
1726:p
1709:,
1703:x
1695:d
1687:)
1681:)
1678:x
1675:(
1672:q
1667:)
1664:x
1661:(
1658:p
1652:(
1639:)
1636:x
1633:(
1630:p
1609:=
1606:)
1603:Q
1597:P
1594:(
1585:D
1567:Q
1563:P
1546:.
1540:0
1537:=
1534:)
1531:x
1528:(
1519:x
1512:+
1508:0
1501:x
1470:)
1467:x
1464:(
1461:P
1430:X
1402:0
1396:)
1393:x
1390:(
1387:Q
1358:+
1345:,
1330:+
1317:(
1302:0
1299:=
1296:)
1293:x
1290:(
1287:P
1261:0
1258:=
1255:)
1252:x
1249:(
1246:Q
1233:x
1226:P
1222:Q
1218:P
1197:.
1190:)
1184:)
1181:x
1178:(
1175:P
1167:)
1164:x
1161:(
1158:Q
1149:(
1136:)
1133:x
1130:(
1127:P
1120:X
1112:x
1101:=
1098:)
1095:Q
1089:P
1086:(
1077:D
1050:,
1043:)
1037:)
1034:x
1031:(
1028:Q
1020:)
1017:x
1014:(
1011:P
1002:(
989:)
986:x
983:(
980:P
973:X
965:x
957:=
954:)
951:Q
945:P
942:(
933:D
919:P
915:Q
901:,
893:X
874:Q
870:P
821:)
818:1
815::
812:2
809:(
806:I
803:+
800:)
797:2
794::
791:1
788:(
785:I
782:=
779:)
776:2
773:,
770:1
767:(
764:J
742:2
715:1
690:)
687:2
684::
681:1
678:(
675:I
653:2
645:,
640:1
613:2
609:H
605:,
600:1
596:H
573:2
565:,
560:1
533:1
506:2
502:H
479:1
475:H
436:Q
432:P
428:P
424:Q
420:P
406:)
403:Q
397:P
394:(
385:D
374:P
370:Q
366:P
362:Q
358:P
282:P
278:P
274:Q
262:Q
258:P
237:.
233:)
227:)
224:x
221:(
218:Q
210:)
207:x
204:(
201:P
192:(
179:)
176:x
173:(
170:P
163:X
155:x
147:=
144:)
141:Q
135:P
132:(
123:D
109:Q
105:P
84:)
81:Q
75:P
72:(
63:D
38:(
20:)
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.