10800:
3348:
9861:
10270:
7342:
16620:
9223:
4128:
4543:
6788:
13865:
10795:{\displaystyle {\begin{aligned}\sum _{n=2}^{\infty }\left\vert {\frac {1}{n(n-1)}}\sum _{x\in {\mathcal {X}}}{\frac {(Q(x)-P(x))^{n}}{Q(x)^{n-1}}}\right\vert &=\sum _{n=2}^{\infty }{\frac {1}{n(n-1)}}\sum _{x\in {\mathcal {X}}}\left\vert Q(x)-P(x)\right\vert \left\vert 1-{\frac {P(x)}{Q(x)}}\right\vert ^{n-1}\\&\leq \sum _{n=2}^{\infty }{\frac {1}{n(n-1)}}\sum _{x\in {\mathcal {X}}}\left\vert Q(x)-P(x)\right\vert \\&\leq \sum _{n=2}^{\infty }{\frac {1}{n(n-1)}}\\&=1\end{aligned}}}
12955:
16237:
3721:
4136:
9856:{\displaystyle {\begin{aligned}f''(\alpha )&=\sum _{x\in {\mathcal {X}}}{\frac {(P(x)-Q(x))^{2}}{(1-\alpha )Q(x)+\alpha P(x)}}\\f''(0)&=\sum _{x\in {\mathcal {X}}}{\frac {(P(x)-Q(x))^{2}}{Q(x)}}\\f^{(n)}(\alpha )&=(-1)^{n}(n-2)!\sum _{x\in {\mathcal {X}}}{\frac {(P(x)-Q(x))^{n}}{\left((1-\alpha )Q(x)+\alpha P(x)\right)^{n-1}}}\\f^{(n)}(0)&=(-1)^{n}(n-2)!\sum _{x\in {\mathcal {X}}}{\frac {(P(x)-Q(x))^{n}}{Q(x)^{n-1}}}\end{aligned}}}
9218:
18741:
7337:{\displaystyle {\begin{aligned}D_{\text{KL}}(P\parallel Q)&=\int _{x_{a}}^{x_{b}}p(x)\log \left({\frac {p(x)}{q(x)}}\right)\,dx\\&=\int _{x_{a}}^{x_{b}}{\tilde {p}}(y(x))|{\frac {dy}{dx}}(x)|\log \left({\frac {{\tilde {p}}(y(x))\,|{\frac {dy}{dx}}(x)|}{{\tilde {q}}(y(x))\,|{\frac {dy}{dx}}(x)|}}\right)\,dx\\&=\int _{y_{a}}^{y_{b}}{\tilde {p}}(y)\log \left({\frac {{\tilde {p}}(y)}{{\tilde {q}}(y)}}\right)\,dy\end{aligned}}}
5119:
13559:
22605:
17552:
12669:
16615:{\displaystyle {\begin{aligned}\left|{\frac {\partial p}{\partial \rho }}\right|&<F(x):\int _{x=0}^{\infty }F(x)\,dx<\infty ,\\\left|{\frac {\partial ^{2}p}{\partial \rho ^{2}}}\right|&<G(x):\int _{x=0}^{\infty }G(x)\,dx<\infty \\\left|{\frac {\partial ^{3}\log(p)}{\partial \rho ^{3}}}\right|&<H(x):\int _{x=0}^{\infty }p(x,0)H(x)\,dx<\xi <\infty \end{aligned}}}
4123:{\displaystyle {\begin{aligned}D_{\text{KL}}(P\parallel Q)&=\sum _{x\in {\mathcal {X}}}P(x)\ln \left({\frac {P(x)}{Q(x)}}\right)\\&={\frac {9}{25}}\ln \left({\frac {9/25}{1/3}}\right)+{\frac {12}{25}}\ln \left({\frac {12/25}{1/3}}\right)+{\frac {4}{25}}\ln \left({\frac {4/25}{1/3}}\right)\\&={\frac {1}{25}}\left(32\ln(2)+55\ln(3)-50\ln(5)\right)\approx 0.0852996,\end{aligned}}}
8853:
4538:{\displaystyle {\begin{aligned}D_{\text{KL}}(Q\parallel P)&=\sum _{x\in {\mathcal {X}}}Q(x)\ln \left({\frac {Q(x)}{P(x)}}\right)\\&={\frac {1}{3}}\ln \left({\frac {1/3}{9/25}}\right)+{\frac {1}{3}}\ln \left({\frac {1/3}{12/25}}\right)+{\frac {1}{3}}\ln \left({\frac {1/3}{4/25}}\right)\\&={\frac {1}{3}}\left(-4\ln(2)-6\ln(3)+6\ln(5)\right)\approx 0.097455.\end{aligned}}}
10237:
18303:
8380:
5520:
2406:
13860:{\displaystyle D_{\text{KL}}\left({\mathcal {N}}\left(\left(\mu _{1},\ldots ,\mu _{k}\right)^{\mathsf {T}},\operatorname {diag} \left(\sigma _{1}^{2},\ldots ,\sigma _{k}^{2}\right)\right)\parallel {\mathcal {N}}\left(\mathbf {0} ,\mathbf {I} \right)\right)={1 \over 2}\sum _{i=1}^{k}\left(\sigma _{i}^{2}+\mu _{i}^{2}-1-\ln \left(\sigma _{i}^{2}\right)\right).}
16843:
13544:
14066:
12950:{\displaystyle D_{\text{KL}}\left({\mathcal {N}}_{0}\parallel {\mathcal {N}}_{1}\right)={\frac {1}{2}}\left(\operatorname {tr} \left(\Sigma _{1}^{-1}\Sigma _{0}\right)-k+\left(\mu _{1}-\mu _{0}\right)^{\mathsf {T}}\Sigma _{1}^{-1}\left(\mu _{1}-\mu _{0}\right)+\ln \left({\frac {\det \Sigma _{1}}{\det \Sigma _{0}}}\right)\right).}
17287:
15952:
21948:
9974:
8145:
1967:
18736:{\displaystyle {\begin{aligned}\mathrm {H} (X\mid Y)&=\log(N)-D_{\text{KL}}(P(X,Y)\parallel P_{U}(X)P(Y))\\&=\log(N)-D_{\text{KL}}(P(X,Y)\parallel P(X)P(Y))-D_{\text{KL}}(P(X)\parallel P_{U}(X))\\&=\mathrm {H} (X)-\operatorname {I} (X;Y)\\&=\log(N)-\operatorname {E} _{Y}\left\end{aligned}}}
24275:
16230:
20823:
5078:
Consider a growth-optimizing investor in a fair game with mutually exclusive outcomes (e.g. a “horse race” in which the official odds add up to one). The rate of return expected by such an investor is equal to the relative entropy between the investor's believed probabilities and the official odds.
17880:
5109:
via information geometry. Investors' views, the prevailing market view, and risky scenarios form triangles on the relevant manifold of probability distributions. The shape of the triangles determines key financial risks (both qualitatively and quantitatively). For instance, obtuse triangles in which
23692:
The resulting contours of constant relative entropy, shown at right for a mole of Argon at standard temperature and pressure, for example put limits on the conversion of hot to cold as in flame-powered air-conditioning or in the unpowered device to convert boiling-water to ice-water discussed here.
9213:{\displaystyle {\begin{aligned}f'(\alpha )&=\sum _{x\in {\mathcal {X}}}(P(x)-Q(x))\left(\log \left({\frac {(1-\alpha )Q(x)+\alpha P(x)}{Q(x)}}\right)+1\right)\\&=\sum _{x\in {\mathcal {X}}}(P(x)-Q(x))\log \left({\frac {(1-\alpha )Q(x)+\alpha P(x)}{Q(x)}}\right)\\f'(0)&=0\end{aligned}}}
5300:
19996:
15773:
8635:
2199:
2840:
15619:
16985:
23938:
are particularly well described in papers and a book by
Burnham and Anderson. In a nutshell the relative entropy of reality from a model may be estimated, to within a constant additive term, by a function of the deviations observed between data and the model's predictions (like the
16638:
13280:
1708:
13888:
23914:
Just as relative entropy of "actual from ambient" measures thermodynamic availability, relative entropy of "reality from a model" is also useful even if the only clues we have about reality are some experimental measurements. In the former case relative entropy describes
14262:
21456:
1196:
1049:
17146:
236:
17547:{\displaystyle {\begin{aligned}\operatorname {I} (X;Y)&=D_{\text{KL}}(P(X,Y)\parallel P(X)P(Y))\\&=\operatorname {E} _{X}\{D_{\text{KL}}(P(Y\mid X)\parallel P(Y))\}\\&=\operatorname {E} _{Y}\{D_{\text{KL}}(P(X\mid Y)\parallel P(X))\}\end{aligned}}}
7965:
15788:
24479:
20509:
21730:
20378:
12455:
24996:– the absolute entropy of a set of data in this sense being the data required to reconstruct it (minimum compressed size), while the relative entropy of a target set of data, given a source set of data, is the data required to reconstruct the target
14474:
19194:
15398:
18192:
19681:
6565:
1810:
12279:
10232:{\displaystyle {\begin{aligned}D_{\text{KL}}(P\parallel Q)&=\sum _{n=0}^{\infty }{\frac {f^{(n)}(0)}{n!}}\\&=\sum _{n=2}^{\infty }{\frac {1}{n(n-1)}}\sum _{x\in {\mathcal {X}}}{\frac {(Q(x)-P(x))^{n}}{Q(x)^{n-1}}}\end{aligned}}}
8454:
8375:{\displaystyle D_{\text{KL}}(\lambda P_{1}+(1-\lambda )P_{2}\parallel \lambda Q_{1}+(1-\lambda )Q_{2})\leq \lambda D_{\text{KL}}(P_{1}\parallel Q_{1})+(1-\lambda )D_{\text{KL}}(P_{2}\parallel Q_{2}){\text{ for }}0\leq \lambda \leq 1.}
24113:
6721:
15218:
20625:
16088:
20630:
11698:
24740:
17712:
23831:
11816:
5515:{\displaystyle {\begin{aligned}D_{\text{KL}}(P\parallel Q)&=\sum _{x\in {\mathcal {X}}}p(x)\log {\frac {1}{q(x)}}-\sum _{x\in {\mathcal {X}}}p(x)\log {\frac {1}{p(x)}}\\&=\mathrm {H} (P,Q)-\mathrm {H} (P)\end{aligned}}}
5110:
investors' views and risk scenarios appear on “opposite sides” relative to the market describe negative risks, acute triangles describe positive exposure, and the right-angled situation in the middle corresponds to zero risk.
24892:
21010:
11246:
7562:
is dimensionless. The argument of the logarithmic term is and remains dimensionless, as it must. It can therefore be seen as in some ways a more fundamental quantity than some other properties in information theory (such as
5860:
Arthur Hobson proved that relative entropy is the only measure of difference between probability distributions that satisfies some desired properties, which are the canonical extension to those appearing in a commonly used
22454:
19814:
2401:{\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{x\in {\mathcal {X}}}{\frac {P(\mathrm {d} \ \!x)}{Q(\mathrm {d} \ \!x)}}\ \log \left({\frac {P(\mathrm {d} \ \!x)}{Q(\mathrm {d} \ \!x)}}\right)\ Q(\mathrm {d} \ \!x)\ ,}
21472:
there is very little difference between near certainty and absolute certainty—coding according to a near certainty requires hardly any more bits than coding according to an absolute certainty. On the other hand, on the
15634:
2049:
14939:
12137:
8771:
834:. The term "divergence" is in contrast to a distance (metric), since the symmetrized divergence does not satisfy the triangle inequality. Numerous references to earlier uses of the symmetrized divergence and to other
19538:
2683:
308:
Relative entropy is always a non-negative real number, with value 0 if and only if the two distributions in question are identical. It has diverse applications, both theoretical, such as characterizing the relative
24036:
18308:
15499:
16854:
16838:{\displaystyle \left.\int _{x=0}^{\infty }{\frac {\partial p(x,\rho )}{\partial \rho }}\right|_{\rho =0}\,dx=\left.\int _{x=0}^{\infty }{\frac {\partial ^{2}p(x,\rho )}{\partial \rho ^{2}}}\right|_{\rho =0}\,dx=0}
13539:{\displaystyle D_{\text{KL}}\left({\mathcal {N}}_{0}\parallel {\mathcal {N}}_{1}\right)={\frac {1}{2}}\left(\sum _{i,j=1}^{k}(M_{ij})^{2}-k+|y|^{2}+2\sum _{i=1}^{k}\ln {\frac {(L_{1})_{ii}}{(L_{0})_{ii}}}\right).}
14811:
2633:
2553:
830:, the symmetrized form is again referred to as the "divergence", and the relative entropies in each direction are referred to as a "directed divergences" between two distributions; Kullback preferred the term
17292:
6793:
2179:
1568:
20905:
is to maximise the expected relative entropy between the prior and the posterior. When posteriors are approximated to be
Gaussian distributions, a design maximising the expected relative entropy is called
22192:
22769:
15283:
14061:{\displaystyle D_{\text{KL}}\left({\mathcal {p}}\parallel {\mathcal {q}}\right)=\log {\frac {\sigma _{1}}{\sigma _{0}}}+{\frac {\sigma _{0}^{2}+(\mu _{0}-\mu _{1})^{2}}{2\sigma _{1}^{2}}}-{\frac {1}{2}}}
21477:
scale implied by weight of evidence, the difference between the two is enormous – infinite perhaps; this might reflect the difference between being almost sure (on a probabilistic level) that, say, the
7841:
5305:
2921:
14124:
12513:
24793:
17717:
16242:
10275:
9979:
9228:
8858:
4141:
3726:
23207:
21305:
11878:
5935:
18283:
1060:
916:
12029:
8828:
21588:
17062:
3124:
3012:
106:
1545:
842:, pp. 6–7, §1.3 Divergence). The asymmetric "directed divergence" has come to be known as the Kullback–Leibler divergence, while the symmetrized "divergence" is now referred to as the
5282:
in bits. Therefore, relative entropy can be interpreted as the expected extra message-length per datum that must be communicated if a code that is optimal for a given (wrong) distribution
15947:{\displaystyle g_{jk}(\theta _{0})=\left.{\frac {\partial ^{2}}{\partial \theta _{j}\,\partial \theta _{k}}}\right|_{\theta =\theta _{0}}D_{\text{KL}}(P(\theta )\parallel P(\theta _{0}))}
23880:
22557:
22515:
14670:
14628:
9903:
8423:
8009:
7796:
7708:
6359:
5981:
5832:
5656:
5010:
4968:
4837:
4764:
4690:
3691:
3649:
405:
83:
23327:
13165:
13110:
11570:
11460:
11418:
14117:
24381:
19730:
14860:
13272:
12657:
3308:
21943:{\displaystyle D_{\text{KL}}(q(x\mid a)u(a)\parallel p(x,a))=\operatorname {E} _{u(a)}\left\{D_{\text{KL}}(q(x\mid a)\parallel p(x\mid a))\right\}+D_{\text{KL}}(u(a)\parallel p(a)),}
14737:
5222:
24540:
20891:
20383:
20137:
6779:
22248:
20228:
820:
23903:
23268:
22331:. In particular, it is the natural extension of the principle of maximum entropy from discrete to continuous distributions, for which Shannon entropy ceases to be so useful (see
15457:
12290:
4628:
1782:
900:
19313:
15491:
14383:
11146:
10990:
6138:
6076:
2099:
1434:
23687:
22594:
12611:
6273:
5560:
654:
574:
22902:
20185:
15064:
3457:
23570:
23400:
19235:
19003:
16025:
14390:
6236:
5603:
3485:
3429:
21722:
11365:
10903:
7560:
7440:
7391:
3602:
3574:
3546:
2465:
1404:
24570:
23450:
16080:
8140:
8094:
5107:
24665:
24656:
24369:
23011:
19806:
19419:
19074:
18804:
15312:
15022:
13216:
11272:
10955:
5761:
1363:
22693:
22517:
subject to some constraint. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be
20072:
18061:
15986:
15111:
13055:
7836:
7620:
614:
24340:
22112:
22077:
20220:
19765:
19378:
14546:
14513:
10929:
1304:
1263:
24806:
23074:
22943:
9969:
6182:
5253:
1962:{\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{x\in {\mathcal {X}}}\ \log \left({\frac {P(\mathrm {d} \ \!x)}{Q(\mathrm {d} \ \!x)}}\right)\ P(\mathrm {d} \ \!x)\ ,}
24314:
24105:
19553:
18883:
18012:
17939:
17634:
11091:
10828:
10265:
8664:
2667:
1472:
1332:
743:
716:
534:
24918:
15303:
15084:
14987:
12546:
11926:
11341:
11321:
11062:
11016:
10857:
21626:
18920:
18839:
17681:
17593:
13008:
11512:
11486:
6404:
5707:
689:
23531:
23504:
23477:
23365:
23132:
23105:
22820:
22633:
21539:
21264:
21237:
21207:
21180:
21149:
21122:
21095:
21068:
21041:
12145:
11957:
11111:
11036:
8048:
5280:
5159:
3342:
3222:
507:
480:
24270:{\displaystyle D_{\lambda }(P\parallel Q)=\lambda D_{\text{KL}}(P\parallel \lambda P+(1-\lambda )Q)+(1-\lambda )D_{\text{KL}}(Q\parallel \lambda P+(1-\lambda )Q),}
22306:
22277:
22042:
22013:
21980:
21667:
21297:
18041:
17968:
17241:
17178:
7506:
7477:
6399:
6201:
5790:
3518:
3401:
2863:
2441:
22969:
20029:
19339:
17267:
17208:
15424:
8449:
6010:
3196:
22789:
14345:
14307:
3249:
16225:{\displaystyle {\frac {\partial \log(p)}{\partial \rho }},{\frac {\partial ^{2}\log(p)}{\partial \rho ^{2}}},{\frac {\partial ^{3}\log(p)}{\partial \rho ^{3}}}}
11166:
10877:
9943:
9923:
8848:
8684:
6570:
21465:
in
Bayesian experimental design, to choose an optimal next question to investigate: but they will in general lead to rather different experimental strategies.
20818:{\displaystyle D_{\text{KL}}{\big (}p(x\mid y_{1},y_{2},I)\parallel p(x\mid y_{1},I){\big )}+D_{\text{KL}}{\big (}p(x\mid y_{1},I)\parallel p(x\mid I){\big )}}
17875:{\displaystyle {\begin{aligned}\mathrm {H} (X)&=\operatorname {E} \left\\&=\log(N)-D_{\text{KL}}\left(p_{X}(x)\parallel P_{U}(X)\right)\end{aligned}}}
15123:
20524:
11578:
23479:
is the probability of a given state under ambient conditions. For instance, the work available in equilibrating a monatomic ideal gas to ambient values of
23734:
12974:
logarithms of expressions that are either factors of the density function or otherwise arise naturally. The equation therefore gives a result measured in
11709:
26654:
25707:
23930:
Although this tool for evaluating models against systems that are accessible experimentally may be applied in any field, its application to selecting a
25676:
20921:
19991:{\displaystyle D_{\text{KL}}{\big (}p(x\mid y,I)\parallel p(x\mid I){\big )}=\sum _{x}p(x\mid y,I)\log \left({\frac {p(x\mid y,I)}{p(x\mid I)}}\right)}
11171:
15768:{\displaystyle D_{\text{KL}}(P(\theta )\parallel P(\theta _{0}))={\frac {1}{2}}\,\Delta \theta _{j}\,\Delta \theta _{k}\,g_{jk}(\theta _{0})+\cdots }
22374:
8630:{\displaystyle D_{\text{KL}}(P\parallel Q)=\sum _{n=2}^{\infty }{\frac {1}{n(n-1)}}\sum _{x\in {\mathcal {X}}}{\frac {(Q(x)-P(x))^{n}}{Q(x)^{n-1}}}}
26500:
24620:
19544:
7442:. Although it was assumed that the transformation was continuous, this need not be the case. This also shows that the relative entropy produces a
2835:{\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{x\in {\mathcal {X}}}p(x)\ \log \left({\frac {\ p(x)\ }{q(x)}}\right)\ \mu (\mathrm {d} \ \!x)~.}
1975:
14875:
12034:
8695:
15614:{\displaystyle \left.{\frac {\partial }{\partial \theta _{j}}}\right|_{\theta =\theta _{0}}D_{\text{KL}}(P(\theta )\parallel P(\theta _{0}))=0,}
19431:
16980:{\displaystyle {\mathcal {D}}(p(x,0)\parallel p(x,\rho ))={\frac {c\rho ^{2}}{2}}+{\mathcal {O}}\left(\rho ^{3}\right){\text{ as }}\rho \to 0.}
5013:
26146:
3025:, but this fails to convey the fundamental asymmetry in the relation. Sometimes, as in this article, it may be described as the divergence of
23971:
19206:
Under this scenario, relative entropies (kl-divergence) can be interpreted as the extra number of bits, on average, that are needed (beyond
14745:
6291:; and the Kullback–Leibler divergence therefore represents the expected number of extra bits that must be transmitted to identify a value
2558:
2478:
13553:, is the relative entropy between a diagonal multivariate normal, and a standard normal distribution (with zero mean and unit variance):
18954:(e.g.: the events (A, B, C) with probabilities p = (1/2, 1/4, 1/4) can be encoded as the bits (0, 10, 11)). If we know the distribution
5889:
1703:{\displaystyle D_{\text{KL}}(P\parallel Q)=\int _{-\infty }^{\infty }p(x)\ \log \left({\frac {p(x)}{q(x)}}\right)\ \mathrm {d} \ \!x\ ,}
2104:
277:, which is the most familiar and formal type of distance. In particular, it is not symmetric in the two distributions (in contrast to
25720:
Donsker, Monroe D.; Varadhan, SR Srinivasa (1983). "Asymptotic evaluation of certain Markov process expectations for large time. IV".
26641:
17648:
if they are coded using only their marginal distributions instead of the joint distribution. Equivalently, if the joint probability
26521:
24603:
Furthermore, the Jensen–Shannon divergence can be generalized using abstract statistical M-mixtures relying on an abstract mean M.
22125:
22702:
15226:
19732:. However, from the standpoint of the new probability distribution one can estimate that to have used the original code based on
23946:
When trying to fit parametrized models to data there are various estimators which attempt to minimize relative entropy, such as
5870:
428:
19048:
needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution
14257:{\displaystyle D_{\text{KL}}\left({\mathcal {p}}\parallel {\mathcal {q}}\right)=\log _{2}k+(k^{-2}-1)/2/\ln(2)\mathrm {bits} }
4873:
typically represents the "true" distribution of data, observations, or a precisely calculated theoretical distribution, while
26575:
26476:
26435:
26422:, Lecture Notes in Mathematics (in French), vol. 649, Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 342–363,
26250:
25967:
25638:
25064:
21482:
is correct, compared to being certain that it is correct because one has a mathematical proof. These two different scales of
17019:
Many of the other quantities of information theory can be interpreted as applications of relative entropy to specific cases.
2868:
273:. While it is a measure of how different two distributions are, and in some sense is thus a "distance", it is not actually a
25986:
Thermostatics and
Thermodynamics: An Introduction to Energy, Information and States of Matter, with Engineering Applications
5862:
21451:{\displaystyle D_{\text{KL}}(p(x\mid H_{1})\parallel p(x\mid H_{0}))\neq IG=D_{\text{KL}}(p(H\mid x)\parallel p(H\mid I)).}
12463:
24745:
23943:) . Estimates of such divergence for models that share the same additive term can in turn be used to select among models.
1191:{\displaystyle D_{\text{KL}}(P\parallel Q)=-\sum _{x\in {\mathcal {X}}}P(x)\ \log \left({\frac {\ Q(x)\ }{P(x)}}\right)~.}
1044:{\displaystyle D_{\text{KL}}(P\parallel Q)=\sum _{x\in {\mathcal {X}}}P(x)\ \log \left({\frac {\ P(x)\ }{Q(x)}}\right)\ ,}
18962:). Meaning the messages we encode will have the shortest length on average (assuming the encoded events are sampled from
18048:
23137:
19013:
will be used (on average) to identify an event from a set of possibilities. This new (larger) number is measured by the
17141:{\displaystyle \operatorname {\operatorname {I} } (m)=D_{\text{KL}}\left(\delta _{\text{im}}\parallel \{p_{i}\}\right),}
11828:
231:{\displaystyle D_{\text{KL}}(P\parallel Q)=\sum _{x\in {\mathcal {X}}}P(x)\ \log \left({\frac {\ P(x)\ }{Q(x)}}\right).}
25660:
25054:
18203:
15400:
the corresponding rate of change in the probability distribution. Since relative entropy has an absolute minimum 0 for
6319:
are two discrete probability distributions built by distributing the same discrete quantity, then the maximum value of
4767:
11966:
10830:
a.s. is also a necessary condition for convergence of the series by the following proof by contradiction. Assume that
8776:
3228:
is the distribution on the right side of the figure, a discrete uniform distribution with the three possible outcomes
26603:
25468:
25362:
25159:
24045:
for the evolution of the term). This function is symmetric and nonnegative, and had already been defined and used by
21544:
18950:. This compresses the data by replacing each fixed-length input symbol with a corresponding unique, variable-length,
17637:
3081:
2969:
855:
24041:
which they referred to as the "divergence", though today the "KL divergence" refers to the asymmetric function (see
7960:{\displaystyle D_{\text{KL}}(P\parallel Q)=D_{\text{KL}}(P_{1}\parallel Q_{1})+D_{\text{KL}}(P_{2}\parallel Q_{2}).}
1480:
26675:
24800:
22909:
22320:
12570:
24474:{\displaystyle D_{\text{JS}}={\frac {1}{2}}D_{\text{KL}}(P\parallel M)+{\frac {1}{2}}D_{\text{KL}}(Q\parallel M)}
24076:
of a feature under two different classes. In the
Banking and Finance industries, this quantity is referred to as
23843:
22520:
22478:
14633:
14591:
9866:
8386:
7972:
7713:
7625:
6322:
5944:
5795:
5619:
4973:
4931:
4800:
4727:
4653:
3654:
3612:
368:
46:
23273:
13115:
13060:
11533:
11423:
11381:
25034:
25019:
20504:{\displaystyle \displaystyle \sum _{x}p(x\mid y_{1},I)\log \left({\frac {p(x\mid y_{1},I)}{p(x\mid I)}}\right)}
17558:
14074:
20001:
to the message length. This therefore represents the amount of useful information, or information gain, about
19689:
14823:
13221:
12619:
3265:
26685:
25069:
24372:
20373:{\displaystyle \sum _{x}p(x\mid y_{1},y_{2},I)\log \left({\frac {p(x\mid y_{1},y_{2},I)}{p(x\mid I)}}\right)}
15114:
14690:
5168:
5131:
5045:
1789:
25747:
Lee, Se Yoon (2021). "Gibbs sampler and coordinate ascent variational inference: A set-theoretical review".
24494:
20835:
20081:
12450:{\displaystyle E_{Q}-D_{\text{KL}}(Q\parallel P)=\log E_{P}-D_{\text{KL}}(Q\parallel Q^{*})\leq \log E_{P},}
10267:
a.s. is a sufficient condition for convergence of the series by the following absolute convergence argument
6726:
5079:
This is a special case of a much more general connection between financial returns and divergence measures.
25014:
24073:
23935:
23923:, while in the latter case it tells you about surprises that reality has up its sleeve or, in other words,
22324:
22197:
20902:
13017:
In a numerical implementation, it is helpful to express the result in terms of the
Cholesky decompositions
7579:
2412:
2052:
1722:
1436:
is infinite in extent. Analogous comments apply to the continuous and general measure cases defined below.
310:
22608:
Pressure versus volume plot of available work from a mole of argon gas relative to ambient, calculated as
748:
26192:
26095:
A Method of
Geometrical Representation of the Thermodynamic Properties of Substances by Means of Surfaces
24950:
23904:
Maximum likelihood estimation § Relation to minimizing
Kullback–Leibler divergence and cross entropy
23837:
23212:
15429:
5063:
4577:
1754:
872:
19268:
15466:
14469:{\displaystyle D_{\text{KL}}\left({\mathcal {p}}\parallel {\mathcal {q}}\right)=\log {\frac {D-C}{B-A}}}
14350:
11116:
10960:
6081:
6019:
5869:
is the only measure of mutual dependence that obeys certain related conditions, since it can be defined
2074:
1409:
24061:
23951:
23578:
22562:
17042:
12576:
6364:
Relative entropy remains well-defined for continuous distributions, and furthermore is invariant under
6241:
5528:
1559:
619:
539:
294:
22863:
20142:
15027:
3435:
25304:
24956:
24589:
23536:
23373:
21490:
useful, according to how well each reflects the particular circumstances of the problem in question.
19209:
19189:{\displaystyle \mathrm {H} (p,q)=\operatorname {E} _{p}=\mathrm {H} (p)+D_{\text{KL}}(p\parallel q).}
18977:
15991:
15393:{\displaystyle P_{j}\left(\theta _{0}\right)={\frac {\partial P}{\partial \theta _{j}}}(\theta _{0})}
14958:
14684:
12965:
6210:
5577:
5025:
3698:
3463:
3407:
407:
is then interpreted as the average difference of the number of bits required for encoding samples of
21680:
21498:
The idea of relative entropy as discrimination information led
Kullback to propose the Principle of
18187:{\displaystyle \lim _{N\rightarrow \infty }H_{N}(X)=\log(N)-\int p(x)\log {\frac {p(x)}{m(x)}}\,dx,}
11346:
10882:
7511:
7396:
7347:
3580:
3552:
3524:
2446:
1368:
25074:
25044:
24548:
23717:
23405:
19033:
17053:
16996:
16046:
15958:
8099:
8053:
5085:
4559:
3347:
1202:
278:
90:
24626:
24348:
22981:
19770:
19383:
18761:
14992:
13178:
11251:
10934:
5716:
1339:
26690:
24659:
23076:
is minimized instead. The change in free energy under these conditions is a measure of available
22649:
20037:
19676:{\displaystyle \mathrm {H} {\big (}p(x\mid y,I){\big )}=-\sum _{x}p(x\mid y,I)\log p(x\mid y,I),}
15964:
15089:
14866:
14569:
13020:
7801:
7585:
5671:
5134:
establishes that any directly decodable coding scheme for coding a message to identify one value
4909:
579:
290:
20:
24319:
23080:
that might be done in the process. Thus available work for an ideal gas at constant temperature
22082:
22047:
20190:
19735:
19348:
14518:
14485:
10908:
1268:
1227:
26680:
25460:
25091:
25059:
25039:
24572:
as the capacity of a noisy information channel with two inputs giving the output distributions
23940:
23035:
22922:
19262:
9948:
6560:{\displaystyle P(dx)=p(x)\,dx={\tilde {p}}(y)\,dy={\tilde {p}}(y(x))|{\tfrac {dy}{dx}}(x)|\,dx}
6143:
5231:
5067:
5041:
4894:
4847:
3072:
25793:
25593:
24299:
24090:
18852:
17981:
17908:
17598:
12274:{\displaystyle D_{\text{KL}}(Q\parallel Q^{*})-D_{\text{KL}}(Q\parallel P)=-E_{Q}+\log E_{P}.}
11067:
10804:
10241:
8640:
2646:
1442:
1314:
721:
694:
512:
24903:
23077:
22976:
15288:
15069:
14972:
13550:
12518:
11899:
11326:
11306:
11041:
10995:
10833:
3169:
2948:
1745:
21596:
18888:
18809:
17651:
17563:
12981:
11491:
11465:
5677:
2947:
etc. for continuous distributions. The logarithms in these formulae are usually taken to
659:
357:
represents the data, the observations, or a measured probability distribution. Distribution
26590:
26370:
26299:
26168:
26119:
26066:
26016:
25893:
25534:
25190:
24612:
24597:
23887:
23509:
23482:
23455:
23343:
23110:
23083:
22857:
22853:
22794:
22611:
22333:
21517:
21242:
21215:
21185:
21158:
21127:
21100:
21073:
21046:
21019:
18967:
18943:
18052:
14565:
14561:
11935:
11096:
11021:
8021:
7443:
5938:
5659:
5258:
5137:
5037:
5033:
4905:
4901:
3712:
3313:
3201:
835:
485:
458:
432:
286:
274:
86:
26242:
Model selection and multimodel inference : a practical information-theoretic approach
25679:(2020). "Kullback-Leibler divergence between quantum distributions, and its upper-bound".
22282:
22253:
22018:
21989:
21956:
21643:
21273:
18017:
17944:
17217:
17154:
7482:
7453:
6716:{\displaystyle Q(dx)=q(x)\,dx={\tilde {q}}(y)\,dy={\tilde {q}}(y)|{\tfrac {dy}{dx}}(x)|dx}
6375:
6186:
5766:
3494:
3377:
2848:
2426:
8:
25913:
22948:
20008:
19318:
19254:
18294:
17246:
17187:
17038:
17028:
17000:
15403:
14585:
14577:
8428:
8016:
6365:
5989:
5606:
5123:
4925:
4917:
3175:
2963:. Most formulas involving relative entropy hold regardless of the base of the logarithm.
1307:
302:
282:
26374:
26303:
26172:
26123:
26070:
26020:
25538:
24897:
22774:
21270:
the same as the information gain expected per sample about the probability distribution
17686:
known, it is the expected number of extra bits that must on average be sent to identify
15213:{\displaystyle P(\theta )=P(\theta _{0})+\Delta \theta _{j}\,P_{j}(\theta _{0})+\cdots }
14312:
14274:
3231:
26595:
26555:
26538:
26516:
26494:
26441:
26393:
26360:
26346:
26322:
26289:
26275:
26184:
26158:
26131:
26032:
25774:
25756:
25680:
25557:
25524:
25512:
25497:
25453:
25321:
25178:
25049:
25024:
24925:
23947:
21479:
20620:{\displaystyle D_{\text{KL}}{\big (}p(x\mid y_{1},y_{2},I)\parallel p(x\mid I){\big )}}
19258:
17278:
17004:
11376:
11151:
10862:
9928:
9908:
8833:
8669:
5866:
5052:
5040:), and this allows one to minimize relative entropy by geometric means, for example by
4794:
3061:
3053:
298:
26347:"On a Generalization of the Jensen–Shannon Divergence and the Jensen–Shannon Centroid"
14680:
11693:{\displaystyle \log E_{P}={\text{sup}}_{Q\ll P}\{E_{Q}-D_{\text{KL}}(Q\parallel P)\}.}
26599:
26571:
26482:
26472:
26445:
26431:
26398:
26327:
26256:
26246:
25963:
25957:
25867:
25839:
25831:
25778:
25656:
25634:
25630:
25589:
25562:
25464:
25421:
25358:
25001:
24990:
24969:
24944:
24735:{\displaystyle \delta (P,Q)\leq {\sqrt {{\frac {1}{2}}D_{\text{KL}}(P\parallel Q)}}.}
24057:
23931:
23030:
19422:
19257:, relative entropy can be used as a measure of the information gain in moving from a
19065:
17008:
16028:
12614:
6204:
4840:
3694:
427:
can be reversed in some situations where that is easier to compute, such as with the
26562:. Applied Mathematical Sciences. Vol. 194. Springer Japan. pp. XIII, 374.
26542:
26416:
Bretagnolle, J.; Huber, C. (1978), "Estimation des densités : Risque minimax",
26208:"Kullback–Leibler information as a basis for strong inference in ecological studies"
26188:
26036:
23826:{\displaystyle D_{\text{KL}}(P\parallel Q)=\operatorname {Tr} (P(\log(P)-\log(Q))).}
22279:; and we note that this result incorporates Bayes' theorem, if the new distribution
15493:. More formally, as for any minimum, the first derivatives of the divergence vanish
11811:{\displaystyle {\frac {Q(d\theta )}{P(d\theta )}}={\frac {\exp h(\theta )}{E_{P}}},}
2189:. Also we assume the expression on the right-hand side exists. Equivalently (by the
26563:
26530:
26423:
26388:
26378:
26317:
26307:
26219:
26176:
26127:
26074:
26051:
26024:
26001:
25953:
25932:
25922:
25823:
25766:
25729:
25607:
Sanov, I.N. (1957). "On the probability of large deviations of random magnitudes".
25552:
25542:
25493:
25317:
25313:
25299:
25168:
25146:
25126:
25079:
24979:
24934:
24593:
23330:
21462:
20907:
18951:
18947:
17974:
the expected number of bits saved, which would have had to be sent if the value of
17034:
7564:
5854:
5838:
loss. The resulting function is asymmetric, and while this can be symmetrized (see
4721:
3137:
2932:
2928:
2924:
1749:
444:
334:
26642:
Matlab code for calculating
Kullback–Leibler divergence for discrete distributions
25770:
24887:{\displaystyle \delta (P,Q)\leq {\sqrt {1-e^{-D_{\mathrm {KL} }(P\parallel Q)}}}.}
21005:{\textstyle D_{\text{KL}}{\bigl (}p(x\mid H_{1})\parallel p(x\mid H_{0}){\bigr )}}
18845:
the expected number of bits saved which would have had to be sent if the value of
14961:. This can be made explicit as follows. Assume that the probability distributions
1474:
is zero the contribution of the corresponding term is interpreted as zero because
26093:
25984:
25352:
25186:
25150:
25083:
24046:
23909:
21514:
should be chosen which is as hard to discriminate from the original distribution
17703:
17181:
14946:
11703:
Further, the supremum on the right-hand side is attained if and only if it holds
11527:
11241:{\displaystyle {\frac {1}{n(n-1)}}\rho \left(1+{\frac {\epsilon }{U}}\right)^{n}}
8012:
7568:
823:
448:
322:
26646:
25704:
20222:, it turns out that it may be either greater or less than previously estimated:
4574:(drawn from one of them) is through the log of the ratio of their likelihoods:
24615:. Some of these are particularly connected with relative entropy. For example:
24084:), and is used to assess distributional shifts in model features through time.
23702:
18959:
17898:
15779:
12553:
11093:
is bounded, so we need only concern ourselves with the behavior of the measure
6276:
5984:
5017:
4697:
4562:
states that the most powerful way to distinguish between the two distributions
330:
254:
26567:
26534:
25871:
25811:
25425:
25173:
25154:
25115:"I-Divergence Geometry of Probability Distributions and Minimization Problems"
22449:{\displaystyle \mathrm {H} (p,m)=\mathrm {H} (p)+D_{\text{KL}}(p\parallel m),}
21070:: the mean information per sample for discriminating in favor of a hypothesis
17640:— i.e. the expected number of extra bits that must be transmitted to identify
4778:
which is currently used. By analogy with information theory, it is called the
26669:
26486:
26276:"On the Jensen–Shannon Symmetrization of Distances Relying on Abstract Means"
26260:
25835:
25827:
25131:
25114:
25029:
23713:
22459:
which is appropriate if one is trying to choose an adequate approximation to
21483:
19029:
19014:
15625:
5850:
5710:
5563:
5021:
4878:
4647:
26650:
26078:
26028:
25927:
25908:
25700:
24900:
generalize relative entropy. Depending on the value of a certain parameter,
22642:
add where probabilities multiply. The surprisal for an event of probability
2044:{\displaystyle \ {\frac {\ P(\mathrm {d} \ \!x)\ }{Q(\mathrm {d} \ \!x)\ }}}
26402:
26331:
25843:
25733:
25566:
25378:
22833:
Best-guess states (e.g. for atoms in a gas) are inferred by maximizing the
14934:{\displaystyle P_{n}\xrightarrow {D} P\Rightarrow P_{n}\xrightarrow {TV} P}
12132:{\displaystyle Q^{*}(d\theta )={\frac {\exp h(\theta )}{E_{P}}}P(d\theta )}
8766:{\displaystyle f(\alpha ):=D_{\text{KL}}((1-\alpha )Q+\alpha P\parallel Q)}
7575:
5883:
5056:
2940:
866:
361:
represents instead a theory, a model, a description or an approximation of
326:
4839:
is a measure of the information gained by revising one's beliefs from the
26637:
Jon Shlens' tutorial on Kullback–Leibler divergence and likelihood theory
25396:
22328:
19533:{\displaystyle p(x\mid y,I)={\frac {p(y\mid x,I)p(x\mid I)}{p(y\mid I)}}}
18958:
in advance, we can devise an encoding that would be optimal (e.g.: using
18044:
17046:
7571:), which can become undefined or negative for non-discrete probabilities.
6311:
No upper-bound exists for the general case. However, it is shown that if
5674:: an asymmetric, generalized form of squared distance. The cross-entropy
314:
26631:
26417:
25937:
25513:"Economics of Disagreement—Financial Intuition for the Rényi Divergence"
23340:
relative to some ambient is obtained by multiplying ambient temperature
21541:
as possible; so that the new data produces as small an information gain
4638:. Kullback motivated the statistic as an expected log likelihood ratio.
26427:
26180:
25952:
Press, W.H.; Teukolsky, S.A.; Vetterling, W.T.; Flannery, B.P. (2007).
25325:
25182:
24031:{\displaystyle D_{\text{KL}}(P\parallel Q)+D_{\text{KL}}(Q\parallel P)}
21152:
19341:
is discovered, it can be used to update the posterior distribution for
12975:
3708:
2960:
2190:
26519:(2000). "The earth mover's distance as a metric for image retrieval".
26383:
26312:
25861:
25547:
25415:
24606:
14969:
are both parameterized by some (possibly multi-dimensional) parameter
2923:
although in practice it will usually be one that in the context like
26660:
26466:
26240:
25580:
Soklakov, A. N. (2023). "Information Geometry of Risks and Returns".
25484:
Kelly, J. L. Jr. (1956). "A New Interpretation of Information Rate".
22639:
19009:) when creating the entropy encoding scheme, then a larger number of
18746:
is the number of bits which would have to be transmitted to identify
17885:
is the number of bits which would have to be transmitted to identify
17210:— i.e. the number of extra bits that must be transmitted to identify
17014:
14806:{\displaystyle \lim _{n\to \infty }D_{\text{KL}}(P_{n}\parallel Q)=0}
12961:
12663:, then the relative entropy between the distributions is as follows:
3049:
2944:
2865:
for which densities can be defined always exists, since one can take
318:
258:
26626:
26224:
26207:
25812:"Distributions of the Kullback-Leibler divergence with applications"
22604:
18938:
When we have a set of possible events, coming from the distribution
14919:
14893:
14842:
5055:
generated by the negative entropy, but it is also of the form of an
2628:{\displaystyle \ Q(\mathrm {d} \ \!x)=q(x)\mu (\mathrm {d} \ \!x)\ }
2548:{\displaystyle \ P(\mathrm {d} \ \!x)=p(x)\mu (\mathrm {d} \ \!x)\ }
26365:
26294:
25761:
25685:
25529:
24799:. For such distributions, an alternative bound can be used, due to
23693:
Thus relative entropy measures thermodynamic availability in bits.
21669:, then the relative entropy between the new joint distribution for
6782:
5118:
5032:. Relative entropy satisfies a generalized Pythagorean theorem for
3156:
Kullback gives the following example (Table 2.1, Example 2.1). Let
26636:
26163:
21493:
11038:. The previous proof of sufficiency demonstrated that the measure
6299:, if a code is used corresponding to the probability distribution
26351:
26280:
22841:
22316:
11168:
th term of this component of the series is then lower bounded by
6781:
is the absolute value of the derivative or more generally of the
5286:
is used, compared to using a code based on the true distribution
5165:
can be seen as representing an implicit probability distribution
2174:{\displaystyle \ P(\mathrm {d} \ \!x)=r(x)Q(\mathrm {d} \ \!x)\ }
25951:
24280:
which can be interpreted as the expected information gain about
22467:
the task one is trying to achieve. Instead, just as often it is
22044:) of the relative entropy of the prior conditional distribution
25379:"What's the maximum value of Kullback-Leibler (KL) divergence?"
25302:(1987). "Letter to the Editor: The Kullback–Leibler distance".
25255:
25253:
23337:
18047:'s alternative generalization to continuous distributions, the
14568:
on the space of probability distributions, but instead it is a
11287:
The following result, due to Donsker and Varadhan, is known as
4630:. The KL divergence is the expected value of this statistic if
2936:
26110:
Tribus, M.; McIrvine, E. C. (1971). "Energy and information".
25863:
An intuition for physicists: information gain from experiments
24596:(in the sense that it induces the same affine connection on a
22337:), but the relative entropy continues to be just as relevant.
22015:, plus the expected value (using the probability distribution
5012:, and the asymmetry is an important part of the geometry. The
22187:{\displaystyle D_{\text{KL}}(q(x\mid a)\parallel p(x\mid a))}
21474:
14515:
bits. This connects with the use of bits in computing, where
5066:, it is unique in being a member of both of these classes of
4692:
can be constructed by measuring the expected number of extra
25250:
24742:
Pinsker's inequality is vacuous for any distributions where
22764:{\displaystyle \left\{1,1/\ln 2,1.38\times 10^{-23}\right\}}
19686:
which may be less than or greater than the original entropy
19005:). However, if we use a different probability distribution (
15278:{\displaystyle \Delta \theta _{j}=(\theta -\theta _{0})_{j}}
11282:
9220:
Further derivatives may be derived and evaluated as follows
4853:. In other words, it is the amount of information lost when
317:, and information gain when comparing statistical models of
23029:
are held constant (say during processes in your body), the
22340:
In the engineering literature, MDI is sometimes called the
16732:
16644:
15826:
15505:
5613:(which is the same as the cross-entropy of P with itself).
25816:
British Journal of Mathematical and Statistical Psychology
23919:
or (when multiplied by ambient temperature) the amount of
23897:
22114:. (Note that often the later expected value is called the
20074:, subsequently comes in, the probability distribution for
4908:, the most familiar type of distance, but instead it is a
25270:
25268:
19045:
19010:
13011:
6368:. For example, if a transformation is made from variable
6280:
4693:
2952:
1220:
Relative entropy is only defined in this way if, for all
1213:, where the expectation is taken using the probabilities
25357:(1st ed.). Cambridge University Press. p. 34.
23021:
is entropy) is minimized as a system "equilibrates." If
21151:
is true. Another name for this quantity, given to it by
18043:. This definition of Shannon entropy forms the basis of
17897:
the relative entropy of the uniform distribution on the
17151:
is the relative entropy of the probability distribution
15086:
differs by only a small amount from the parameter value
14271:
Consider two uniform distributions, with the support of
1205:
of the logarithmic difference between the probabilities
25397:"In what situations is the integral equal to infinity?"
22856:, both classically and quantum mechanically, minimizes
22844:) for a given set of control parameters (like pressure
22826:
bits of surprisal for landing all "heads" on a toss of
20139:. If one reinvestigates the information gain for using
16035:
parameter space, called the Fisher information metric.
14676:), the asymmetry is an important part of the geometry.
14580:, divergences are asymmetric in general and generalize
5834:
agree more closely with our notion of distance, as the
2916:{\displaystyle \ \mu ={\frac {1}{2}}\left(P+Q\right)\ }
297:, and for certain classes of distributions (notably an
26661:
A modern summary of info-theoretic divergence measures
25354:
Information Theory, Inference, and Learning Algorithms
25265:
25204:
25202:
25200:
20924:
6736:
6672:
6515:
4877:
typically represents a theory, model, description, or
3168:
is the distribution on the left side of the figure, a
616:
are the hypotheses that one is selecting from measure
25810:
Belov, Dmitry I.; Armstrong, Ronald D. (2011-04-15).
25432:
25332:
25240:
25238:
25225:
25223:
25221:
25219:
25217:
24906:
24809:
24748:
24668:
24629:
24551:
24497:
24384:
24351:
24322:
24302:
24116:
24093:
23974:
23846:
23737:
23581:
23539:
23512:
23485:
23458:
23408:
23376:
23346:
23276:
23215:
23140:
23113:
23086:
23038:
22984:
22951:
22925:
22866:
22797:
22777:
22705:
22652:
22614:
22565:
22523:
22481:
22377:
22285:
22256:
22200:
22128:
22085:
22050:
22021:
21992:
21959:
21733:
21683:
21646:
21599:
21547:
21520:
21308:
21276:
21245:
21218:
21188:
21161:
21130:
21103:
21076:
21049:
21022:
20838:
20633:
20527:
20387:
20386:
20231:
20193:
20145:
20084:
20040:
20011:
19817:
19773:
19738:
19692:
19556:
19434:
19386:
19351:
19321:
19271:
19212:
19077:
18980:
18891:
18855:
18812:
18764:
18306:
18206:
18064:
18020:
17984:
17947:
17911:
17715:
17654:
17601:
17566:
17290:
17249:
17220:
17190:
17157:
17065:
16857:
16641:
16240:
16091:
16049:
15994:
15967:
15791:
15637:
15502:
15469:
15432:
15406:
15315:
15291:
15229:
15126:
15113:. Specifically, up to first order one has (using the
15092:
15072:
15030:
14995:
14975:
14945:
where the latter stands for the usual convergence in
14878:
14826:
14748:
14693:
14636:
14594:
14521:
14488:
14393:
14353:
14315:
14277:
14127:
14077:
14071:
In the case of co-centered normal distributions with
13891:
13562:
13283:
13224:
13181:
13118:
13063:
13023:
12984:
12672:
12622:
12579:
12521:
12508:{\displaystyle D_{\text{KL}}(Q\parallel Q^{*})\geq 0}
12466:
12293:
12148:
12037:
11969:
11938:
11902:
11831:
11712:
11581:
11536:
11494:
11468:
11426:
11384:
11349:
11329:
11309:
11254:
11174:
11154:
11119:
11099:
11070:
11044:
11024:
10998:
10963:
10937:
10911:
10885:
10865:
10836:
10807:
10273:
10244:
9977:
9951:
9931:
9911:
9869:
9226:
8856:
8836:
8779:
8698:
8672:
8643:
8457:
8431:
8389:
8148:
8102:
8056:
8024:
7975:
7844:
7804:
7716:
7628:
7588:
7514:
7485:
7456:
7399:
7350:
6791:
6729:
6573:
6407:
6378:
6325:
6244:
6213:
6189:
6146:
6084:
6022:
5992:
5947:
5892:
5798:
5769:
5719:
5680:
5622:
5580:
5531:
5303:
5261:
5234:
5171:
5140:
5088:
4976:
4934:
4803:
4730:
4656:
4580:
4139:
3724:
3657:
3615:
3583:
3555:
3527:
3497:
3466:
3438:
3410:
3380:
3316:
3268:
3234:
3204:
3178:
3084:
2972:
2871:
2851:
2686:
2649:
2561:
2481:
2449:
2429:
2202:
2107:
2077:
1978:
1813:
1757:
1571:
1483:
1445:
1412:
1371:
1342:
1317:
1271:
1230:
1063:
919:
875:
751:
724:
697:
662:
622:
582:
542:
515:
488:
461:
371:
109:
49:
26632:
Ruby gem for calculating Kullback–Leibler divergence
25280:
24788:{\displaystyle D_{\mathrm {KL} }(P\parallel Q)>2}
22308:
is in fact a δ function representing certainty that
16038:
12564:
10879:. It then follows that there must exist some values
5713:), but it cannot be thought of as a distance, since
3164:
be the distributions shown in the table and figure.
3014:
in words. Often it is referred to as the divergence
455:
as "the mean information for discrimination between
96:
is different from a second probability distribution
25401:
Mathematics Stack Exchange (math.stackexchange.com)
25383:
Statistics Stack Exchange (stats.stackexchange.com)
25197:
24607:
Relationship to other probability-distance measures
21636:, and subsequently learnt the true distribution of
4893:, we can minimize the KL divergence and compute an
536:", where one is comparing two probability measures
26514:
25959:Numerical Recipes: The Art of Scientific Computing
25906:
25627:Extreme Value Methods with Applications to Finance
25452:
25235:
25214:
25139:
24912:
24886:
24787:
24734:
24650:
24564:
24534:
24473:
24363:
24334:
24308:
24269:
24099:
24030:
23874:
23825:
23681:
23564:
23525:
23498:
23471:
23444:
23394:
23359:
23321:
23262:
23202:{\displaystyle W=\Delta G=NkT_{o}\Theta (V/V_{o})}
23201:
23126:
23099:
23068:
23005:
22963:
22937:
22896:
22814:
22783:
22763:
22687:
22627:
22588:
22551:
22509:
22448:
22300:
22271:
22242:
22186:
22106:
22071:
22036:
22007:
21974:
21942:
21716:
21661:
21620:
21582:
21533:
21450:
21291:
21258:
21231:
21201:
21174:
21143:
21116:
21089:
21062:
21035:
21004:
20885:
20817:
20619:
20503:
20372:
20214:
20179:
20131:
20066:
20023:
19990:
19800:
19759:
19724:
19675:
19532:
19413:
19372:
19333:
19307:
19229:
19188:
18997:
18914:
18877:
18833:
18798:
18735:
18277:
18186:
18035:
18006:
17962:
17933:
17874:
17675:
17628:
17587:
17546:
17261:
17235:
17202:
17172:
17140:
17015:Relation to other quantities of information theory
16979:
16837:
16614:
16224:
16074:
16019:
15980:
15946:
15767:
15613:
15485:
15451:
15418:
15392:
15297:
15277:
15212:
15105:
15078:
15058:
15016:
14981:
14933:
14854:
14805:
14731:
14664:
14622:
14548:bits would be needed to identify one element of a
14540:
14507:
14468:
14377:
14339:
14301:
14256:
14111:
14060:
13859:
13538:
13266:
13210:
13159:
13104:
13049:
13002:
12949:
12659:If the two distributions have the same dimension,
12651:
12605:
12540:
12507:
12449:
12273:
12131:
12023:
11951:
11920:
11873:{\displaystyle {\frac {Q(d\theta )}{P(d\theta )}}}
11872:
11821:almost surely with respect to probability measure
11810:
11692:
11564:
11506:
11480:
11454:
11412:
11359:
11335:
11315:
11266:
11240:
11160:
11140:
11105:
11085:
11056:
11030:
11010:
10984:
10949:
10923:
10897:
10871:
10851:
10822:
10794:
10259:
10231:
9963:
9937:
9917:
9897:
9855:
9212:
8842:
8822:
8765:
8678:
8658:
8629:
8443:
8417:
8374:
8134:
8088:
8042:
8003:
7959:
7830:
7790:
7702:
7614:
7554:
7500:
7471:
7434:
7385:
7336:
6773:
6715:
6559:
6393:
6353:
6267:
6230:
6195:
6176:
6132:
6070:
6004:
5975:
5930:{\displaystyle D_{\text{KL}}(P\parallel Q)\geq 0,}
5929:
5826:
5784:
5755:
5701:
5650:
5597:
5554:
5514:
5274:
5247:
5216:
5153:
5101:
5004:
4962:
4831:
4758:
4684:
4622:
4537:
4122:
3685:
3643:
3596:
3568:
3540:
3512:
3479:
3451:
3423:
3395:
3336:
3302:
3243:
3216:
3190:
3118:
3006:
2915:
2857:
2834:
2661:
2627:
2547:
2459:
2435:
2400:
2173:
2093:
2043:
1961:
1776:
1702:
1539:
1466:
1428:
1398:
1357:
1326:
1298:
1257:
1190:
1043:
894:
814:
737:
710:
683:
648:
608:
568:
528:
501:
474:
399:
230:
77:
26052:"Information theory and statistical mechanics II"
25794:"Derivations for Linear Algebra and Optimization"
25749:Communications in Statistics - Theory and Methods
22599:
22475:that one is attempting to optimise by minimising
22360:is equivalent to minimizing the cross-entropy of
20078:can be updated further, to give a new best guess
18849:were coded according to the uniform distribution
18758:the relative entropy of the product distribution
18278:{\displaystyle \log(N)-D_{\text{KL}}(p(x)||m(x))}
17978:were coded according to the uniform distribution
14584:distance, in some cases satisfying a generalized
4924:distance, in some cases satisfying a generalized
3693:are calculated as follows. This example uses the
2819:
2615:
2579:
2535:
2499:
2385:
2354:
2331:
2291:
2268:
2161:
2125:
2028:
2002:
1946:
1915:
1892:
1690:
1562:, relative entropy is defined to be the integral
313:in information systems, randomness in continuous
26667:
26415:
24795:, since the total variation distance is at most
24284:from discovering which probability distribution
22471:that is some fixed prior reference measure, and
19241:for constructing the encoding scheme instead of
18066:
17243:is available to the receiver, not the fact that
14750:
12919:
12904:
12024:{\displaystyle {\frac {\exp h(\theta )}{E_{P}}}}
8823:{\displaystyle D_{\text{KL}}(P\parallel Q)=f(1)}
6238:thus sets a minimum value for the cross-entropy
3351:Two distributions to illustrate relative entropy
1485:
26205:
26147:"Thermal roots of correlation-based complexity"
25719:
25450:
25259:
25145:
24924:Other notable measures of distance include the
23962:
21724:, and the earlier prior distribution would be:
21583:{\displaystyle D_{\text{KL}}(f\parallel f_{0})}
21494:Principle of minimum discrimination information
20896:
16082:satisfies the following regularity conditions:
15988:vary (and dropping the subindex 0) the Hessian
8425:may be Taylor expanded about its minimum (i.e.
3119:{\displaystyle \ D_{\text{KL}}(P\parallel Q)\ }
3007:{\displaystyle \ D_{\text{KL}}(P\parallel Q)\ }
2643:are both absolutely continuous with respect to
452:
26109:
26002:"Information theory and statistical mechanics"
25883:
25881:
25809:
25722:Communications on Pure and Applied Mathematics
25699:See the section "differential entropy – 4" in
24658:. This is connected to the divergence through
21461:Either of the two quantities can be used as a
5763:is not zero. This can be fixed by subtracting
5029:
1540:{\displaystyle \lim _{x\to 0^{+}}x\log(x)=0~.}
24989:entropy serves as theoretical background for
24978:entropy serves as theoretical background for
23696:
21593:For example, if one had a prior distribution
20997:
20937:
20913:
20810:
20751:
20731:
20646:
20612:
20540:
19882:
19830:
19808:would have added an expected number of bits:
19595:
19564:
17056:, the self-information can be represented as
14572:. While metrics are symmetric and generalize
14482:times narrower uniform distribution contains
12970:since all terms apart from the last are base-
5122:Illustration of the relative entropy for two
4912:. While metrics are symmetric and generalize
822:, which had already been defined and used by
321:; and practical, such as applied statistics,
26499:: CS1 maint: multiple names: authors list (
25962:(3rd ed.). Cambridge University Press.
25693:
22809:
22778:
22348:for short. Minimising relative entropy from
18055:), which defines the continuous entropy as
17537:
17485:
17459:
17407:
17127:
17114:
17045:is defined as the negative logarithm of the
14957:Relative entropy is directly related to the
14952:
14726:
14694:
11896:For a short proof assuming integrability of
11684:
11634:
8142:are two pairs of probability measures then
7582:in much the same way as Shannon entropy. If
4920:, divergences are asymmetric and generalize
3297:
3279:
26614:(Second ed.). Oxford University Press.
26338:
26267:
25954:"Section 14.7.2. Kullback–Leibler Distance"
25887:
25878:
25669:
25292:
24611:There are many other important measures of
23875:{\displaystyle D_{\text{KL}}(P\parallel Q)}
22552:{\displaystyle D_{\text{KL}}(p\parallel m)}
22510:{\displaystyle D_{\text{KL}}(p\parallel m)}
19237:) for encoding the events because of using
16990:
14673:
14665:{\displaystyle D_{\text{KL}}(Q\parallel P)}
14623:{\displaystyle D_{\text{KL}}(P\parallel Q)}
13175:solutions to the triangular linear systems
12515:, for which equality occurs if and only if
9898:{\displaystyle D_{\text{KL}}(P\parallel Q)}
8418:{\displaystyle D_{\text{KL}}(P\parallel Q)}
8004:{\displaystyle D_{\text{KL}}(P\parallel Q)}
7791:{\displaystyle Q(dx,dy)=Q_{1}(dx)Q_{2}(dy)}
7703:{\displaystyle P(dx,dy)=P_{1}(dx)P_{2}(dy)}
6354:{\displaystyle D_{\text{KL}}(P\parallel Q)}
5976:{\displaystyle D_{\text{KL}}(P\parallel Q)}
5842:), the asymmetric form is more useful. See
5839:
5827:{\displaystyle D_{\text{KL}}(P\parallel Q)}
5651:{\displaystyle D_{\text{KL}}(P\parallel Q)}
5126:. The typical asymmetry is clearly visible.
5016:form of relative entropy, specifically its
5005:{\displaystyle D_{\text{KL}}(Q\parallel P)}
4963:{\displaystyle D_{\text{KL}}(P\parallel Q)}
4832:{\displaystyle D_{\text{KL}}(P\parallel Q)}
4759:{\displaystyle D_{\text{KL}}(P\parallel Q)}
4685:{\displaystyle D_{\text{KL}}(P\parallel Q)}
3686:{\displaystyle D_{\text{KL}}(Q\parallel P)}
3644:{\displaystyle D_{\text{KL}}(P\parallel Q)}
2966:Various conventions exist for referring to
400:{\displaystyle D_{\text{KL}}(P\parallel Q)}
340:
78:{\displaystyle D_{\text{KL}}(P\parallel Q)}
26627:Information Theoretical Estimators Toolbox
25344:
24580:. The Jensen–Shannon divergence, like all
24056:This quantity has sometimes been used for
23965:also considered the symmetrized function:
23322:{\displaystyle \Theta (x)=x-1-\ln x\geq 0}
20514:and so the combined information gain does
19056:. The cross entropy for two distributions
13160:{\displaystyle \Sigma _{1}=L_{1}L_{1}^{T}}
13105:{\displaystyle \Sigma _{0}=L_{0}L_{0}^{T}}
12978:. Dividing the entire expression above by
11565:{\displaystyle (\Theta ,{\mathcal {F}},P)}
11455:{\displaystyle (\Theta ,{\mathcal {F}},Q)}
11413:{\displaystyle (\Theta ,{\mathcal {F}},P)}
11289:Donsker and Varadhan's variational formula
5846:for more on the geometric interpretation.
26560:Information Geometry and Its Applications
26392:
26382:
26364:
26321:
26311:
26293:
26223:
26162:
26144:
25936:
25926:
25888:Cover, Thomas M.; Thomas, Joy A. (1991),
25760:
25684:
25556:
25546:
25528:
25455:Model Selection and Multi-Model Inference
25172:
25130:
25108:
25106:
24488:is the average of the two distributions,
23957:
21155:, is the expected weight of evidence for
19199:For explicit derivation of this, see the
18885:rather than the conditional distribution
18174:
16822:
16720:
16589:
16445:
16328:
15854:
15729:
15715:
15701:
15177:
14739:is a sequence of distributions such that
14672:, and while this can be symmetrized (see
14112:{\displaystyle k=\sigma _{1}/\sigma _{0}}
13549:A special case, and a common quantity in
11518:is absolutely continuous with respect to
11283:Duality formula for variational inference
7545:
7323:
7192:
7145:
7073:
6915:
6785:, the relative entropy may be rewritten:
6635:
6604:
6550:
6469:
6438:
5709:is itself such a measurement (formally a
2951:2 if information is measured in units of
2185:is absolutely continuous with respect to
26609:
26584:
26522:International Journal of Computer Vision
26468:Introduction to nonparametric estimation
26465:B.), Tsybakov, A. B. (Alexandre (2010).
25909:"Bayesian experimental design: a review"
25579:
25510:
25451:Burnham, K. P.; Anderson, D. R. (2002).
25438:
25417:Pattern recognition and machine learning
25338:
25298:
25286:
25274:
25208:
22603:
21953:i.e. the sum of the relative entropy of
21012:can also be interpreted as the expected
19725:{\displaystyle \mathrm {H} (p(x\mid I))}
16995:Another information-theoretic metric is
14855:{\displaystyle P_{n}{\xrightarrow {D}}Q}
14266:
13870:For two univariate normal distributions
13267:{\displaystyle L_{1}y=\mu _{1}-\mu _{0}}
12652:{\displaystyle \Sigma _{0},\Sigma _{1}.}
11880:denotes the Radon-Nikodym derivative of
8850:may be derived and evaluated as follows
5662:, a measure of how far the distribution
5117:
5073:
3346:
3303:{\displaystyle {\mathcal {X}}=\{0,1,2\}}
839:
827:
691:, and defined the "'divergence' between
16:Mathematical statistics distance measure
26344:
26273:
26238:
25859:
25855:
25853:
25675:
25112:
24296:, if they currently have probabilities
23898:Relationship between models and reality
22120:conditional Kullback–Leibler divergence
21510:): given new facts, a new distribution
20005:, that has been learned by discovering
16999:, which is roughly a symmetrization of
14989:. Consider then two close by values of
14732:{\displaystyle \{P_{1},P_{2},\ldots \}}
14478:Intuitively, the information gain to a
12964:in the last term must be taken to base
12460:where the last inequality follows from
5871:in terms of Kullback–Leibler divergence
5217:{\displaystyle q(x_{i})=2^{-\ell _{i}}}
4864:
443:The relative entropy was introduced by
429:expectation–maximization algorithm (EM)
345:Consider two probability distributions
26668:
26206:Burnham, K.P.; Anderson, D.R. (2001).
26049:
25999:
25982:
25650:
25350:
25103:
24920:, various inequalities may be deduced.
24535:{\displaystyle M={\frac {1}{2}}(P+Q).}
24049:in 1948; it is accordingly called the
22079:from the new conditional distribution
20886:{\displaystyle p(y_{2}\mid y_{1},x,I)}
20132:{\displaystyle p(x\mid y_{1},y_{2},I)}
19052:, rather than the "true" distribution
18288:
17694:is not already known to the receiver.
14555:
13632:
12831:
6774:{\displaystyle |{\tfrac {dy}{dx}}(x)|}
6303:, rather than the "true" distribution
305:(which applies to squared distances).
26554:
26508:
26239:Burnham, Kenneth P. (December 2010).
26091:
25907:Chaloner, K.; Verdinelli, I. (1995).
25619:
25606:
25600:
25483:
25244:
25229:
25065:Information theory and measure theory
22635:times the Kullback–Leibler divergence
22243:{\displaystyle q(x\mid a)=p(x\mid a)}
17272:
17214:if only the probability distribution
11323:be a set endowed with an appropriate
5658:can be thought of geometrically as a
2931:or a convenient variant thereof like
1310:). Otherwise, it is often defined as
656:(respectively). They denoted this by
25850:
25791:
24963:
21212:The expected weight of evidence for
19248:
11572:. Then the following equality holds
6283:required when using a code based on
5843:
815:{\displaystyle J(1,2)=I(1:2)+I(2:1)}
242:
26464:
25746:
23925:how much the model has yet to learn
23263:{\displaystyle V_{o}=NkT_{o}/P_{o}}
22315:MDI can be seen as an extension of
18049:limiting density of discrete points
17022:
15452:{\displaystyle \theta =\theta _{0}}
10859:with measure strictly greater than
7622:are independent distributions, and
4708:rather than the code optimized for
4623:{\displaystyle \log P(Y)-\log Q(Y)}
2067:almost everywhere defined function
1777:{\displaystyle \ {\mathcal {X}}\ ,}
895:{\displaystyle \ {\mathcal {X}}\ ,}
100:. Mathematically, it is defined as
13:
26132:10.1038/scientificamerican0971-179
25498:10.1002/j.1538-7305.1956.tb03809.x
25459:(2nd ed.). Springer. p.
25413:
25055:Information gain in decision trees
24856:
24853:
24758:
24755:
23643:
23602:
23582:
23556:
23377:
23277:
23172:
23147:
22567:
22402:
22379:
22342:Principle of Minimum Cross-Entropy
21805:
21502:Minimum Discrimination Information
20893:, the two sides will average out.
19694:
19558:
19308:{\displaystyle p(x)\to p(x\mid I)}
19214:
19142:
19102:
19079:
18982:
18643:
18593:
18577:
18312:
18076:
18014:rather than the true distribution
17753:
17741:
17721:
17697:
17638:marginal probability distributions
17473:
17395:
17295:
17067:
16940:
16860:
16789:
16760:
16751:
16694:
16671:
16663:
16605:
16554:
16496:
16470:
16455:
16428:
16370:
16356:
16338:
16311:
16260:
16252:
16206:
16180:
16157:
16131:
16115:
16095:
15855:
15841:
15831:
15716:
15702:
15513:
15509:
15486:{\displaystyle \Delta \theta _{j}}
15470:
15358:
15350:
15230:
15164:
14760:
14421:
14411:
14378:{\displaystyle C\leq A<B\leq D}
14250:
14247:
14244:
14241:
14155:
14145:
13919:
13909:
13706:
13580:
13319:
13302:
13120:
13065:
12923:
12908:
12838:
12772:
12754:
12708:
12691:
12637:
12624:
11548:
11540:
11438:
11430:
11396:
11388:
11352:
11310:
11261:
11141:{\displaystyle P\geq 2Q+\epsilon }
10985:{\displaystyle P\geq 2Q+\epsilon }
10944:
10743:
10673:
10628:
10497:
10452:
10344:
10294:
10145:
10100:
10030:
9769:
9562:
9409:
9268:
9056:
8898:
8547:
8502:
6246:
6215:
6133:{\displaystyle Q(dx)=q(x)\mu (dx)}
6071:{\displaystyle P(dx)=p(x)\mu (dx)}
5582:
5533:
5495:
5472:
5415:
5353:
5062:. For probabilities over a finite
4885:. In order to find a distribution
4848:posterior probability distribution
4548:
4189:
3774:
3271:
2812:
2728:
2669:), then the relative entropy from
2608:
2572:
2528:
2492:
2452:
2378:
2347:
2324:
2284:
2261:
2244:
2154:
2118:
2094:{\displaystyle \ {\mathcal {X}}\ }
2083:
2021:
1995:
1939:
1908:
1885:
1855:
1763:
1683:
1613:
1608:
1429:{\displaystyle \ {\mathcal {X}}\ }
1418:
1349:
1321:
1108:
961:
881:
856:discrete probability distributions
365:. The Kullback–Leibler divergence
151:
14:
26702:
26620:
26587:Information Theory and Statistics
25653:Concepts in statistical mechanics
25160:Annals of Mathematical Statistics
23886:can also be used as a measure of
23682:{\displaystyle \Delta I=Nk\left.}
22908:is a constrained multiplicity or
22822:so that, for instance, there are
22589:{\displaystyle \mathrm {H} (p,m)}
22463:. However, this is just as often
21209:to be expected from each sample.
19044:) measures the average number of
18806:from the true joint distribution
17041:of a signal, random variable, or
16039:Fisher information metric theorem
14385:). Then the information gain is:
12606:{\displaystyle \mu _{0},\mu _{1}}
12571:multivariate normal distributions
12565:Multivariate normal distributions
7508:are also dimensioned, since e.g.
6268:{\displaystyle \mathrm {H} (P,Q)}
5555:{\displaystyle \mathrm {H} (P,Q)}
5082:Financial risks are connected to
3078:. Another common way to refer to
1796:, then the relative entropy from
649:{\displaystyle \mu _{1},\mu _{2}}
569:{\displaystyle \mu _{1},\mu _{2}}
89:: a measure of how one reference
25860:Buchner, Johannes (2022-04-29).
25155:"On information and sufficiency"
24087:An alternative is given via the
23402:defined as the average value of
22897:{\displaystyle A\equiv -k\ln(Z)}
22321:Principle of Insufficient Reason
20180:{\displaystyle p(x\mid y_{1},I)}
19380:to a new posterior distribution
18933:
17049:of the given outcome occurring.
16027:defines a (possibly degenerate)
15059:{\displaystyle Q=P(\theta _{0})}
13725:
13717:
5030:§ Fisher information metric
4558:In the field of statistics, the
3452:{\displaystyle {\frac {12}{25}}}
3151:
269:when the actual distribution is
26458:
26409:
26232:
26199:
26138:
26103:
26085:
26043:
25993:
25976:
25945:
25900:
25803:
25785:
25740:
25713:
25655:. New York: Gordon and Breach.
25644:
25573:
25504:
25477:
25444:
25407:
25389:
25371:
23565:{\displaystyle W=T_{o}\Delta I}
23395:{\displaystyle \Delta I\geq 0,}
19767:instead of a new code based on
19230:{\displaystyle \mathrm {H} (p)}
19200:
18998:{\displaystyle \mathrm {H} (p)}
17557:is the relative entropy of the
17003:. It is a metric on the set of
16020:{\displaystyle g_{jk}(\theta )}
11367:, and two probability measures
8637:which converges if and only if
6231:{\displaystyle \mathrm {H} (P)}
5598:{\displaystyle \mathrm {H} (P)}
3480:{\displaystyle {\frac {4}{25}}}
3424:{\displaystyle {\frac {9}{25}}}
2927:for discrete distributions, or
26092:Gibbs, Josiah Willard (1871).
25890:Elements of Information Theory
25318:10.1080/00031305.1987.10475510
25035:Deviance information criterion
25020:Bayesian information criterion
25000:the source (minimum size of a
24874:
24862:
24825:
24813:
24776:
24764:
24724:
24712:
24684:
24672:
24645:
24633:
24526:
24514:
24468:
24456:
24430:
24418:
24261:
24255:
24243:
24225:
24212:
24200:
24194:
24188:
24176:
24158:
24139:
24127:
24042:
24025:
24013:
23997:
23985:
23869:
23857:
23817:
23814:
23811:
23805:
23793:
23787:
23778:
23772:
23760:
23748:
23439:
23418:
23286:
23280:
23196:
23175:
22891:
22885:
22860:availability in entropy units
22682:
22668:
22600:Relationship to available work
22583:
22571:
22546:
22534:
22504:
22492:
22440:
22428:
22412:
22406:
22395:
22383:
22295:
22289:
22266:
22260:
22237:
22225:
22216:
22204:
22181:
22178:
22166:
22157:
22145:
22139:
22101:
22089:
22066:
22054:
22031:
22025:
22002:
21996:
21986:from the updated distribution
21969:
21963:
21934:
21931:
21925:
21916:
21910:
21904:
21883:
21880:
21868:
21859:
21847:
21841:
21818:
21812:
21798:
21795:
21783:
21774:
21768:
21762:
21750:
21744:
21717:{\displaystyle q(x\mid a)u(a)}
21711:
21705:
21699:
21687:
21656:
21650:
21615:
21603:
21577:
21558:
21442:
21439:
21427:
21418:
21406:
21400:
21375:
21372:
21353:
21344:
21325:
21319:
21286:
21280:
20992:
20973:
20964:
20945:
20880:
20842:
20805:
20793:
20784:
20759:
20726:
20701:
20692:
20654:
20607:
20595:
20586:
20548:
20518:obey the triangle inequality:
20490:
20478:
20470:
20445:
20426:
20401:
20360:
20348:
20340:
20302:
20283:
20245:
20209:
20197:
20174:
20149:
20126:
20088:
19978:
19966:
19958:
19940:
19921:
19903:
19877:
19865:
19856:
19838:
19795:
19777:
19754:
19742:
19719:
19716:
19704:
19698:
19667:
19649:
19637:
19619:
19590:
19572:
19524:
19512:
19504:
19492:
19486:
19468:
19456:
19438:
19408:
19390:
19367:
19355:
19302:
19290:
19284:
19281:
19275:
19224:
19218:
19180:
19168:
19152:
19146:
19135:
19132:
19126:
19114:
19095:
19083:
18992:
18986:
18909:
18902:
18895:
18872:
18866:
18828:
18816:
18793:
18787:
18781:
18775:
18754:equally likely possibilities,
18716:
18710:
18636:
18630:
18611:
18599:
18587:
18581:
18563:
18560:
18554:
18538:
18532:
18526:
18510:
18507:
18501:
18495:
18489:
18480:
18468:
18462:
18446:
18440:
18421:
18418:
18412:
18406:
18400:
18384:
18372:
18366:
18350:
18344:
18328:
18316:
18272:
18269:
18263:
18256:
18251:
18247:
18241:
18235:
18219:
18213:
18168:
18162:
18154:
18148:
18133:
18127:
18115:
18109:
18097:
18091:
18073:
18030:
18024:
18001:
17995:
17957:
17951:
17928:
17922:
17893:equally likely possibilities,
17860:
17854:
17838:
17832:
17801:
17795:
17771:
17765:
17731:
17725:
17670:
17658:
17623:
17617:
17611:
17605:
17582:
17570:
17559:joint probability distribution
17534:
17531:
17525:
17516:
17504:
17498:
17456:
17453:
17447:
17438:
17426:
17420:
17381:
17378:
17372:
17366:
17360:
17351:
17339:
17333:
17313:
17301:
17230:
17224:
17167:
17161:
17080:
17074:
16971:
16907:
16904:
16892:
16883:
16871:
16865:
16784:
16772:
16689:
16677:
16586:
16580:
16574:
16562:
16532:
16526:
16491:
16485:
16442:
16436:
16406:
16400:
16325:
16319:
16289:
16283:
16201:
16195:
16152:
16146:
16110:
16104:
16067:
16055:
16014:
16008:
15941:
15938:
15925:
15916:
15910:
15904:
15818:
15805:
15756:
15743:
15685:
15682:
15669:
15660:
15654:
15648:
15599:
15596:
15583:
15574:
15568:
15562:
15463:order in the small parameters
15387:
15374:
15266:
15246:
15201:
15188:
15158:
15145:
15136:
15130:
15053:
15040:
15011:
15005:
14902:
14794:
14775:
14757:
14659:
14647:
14617:
14605:
14334:
14322:
14296:
14284:
14237:
14231:
14209:
14187:
14013:
13986:
13513:
13499:
13485:
13471:
13425:
13416:
13397:
13380:
12997:
12991:
12496:
12477:
12441:
12429:
12407:
12388:
12372:
12360:
12338:
12326:
12310:
12304:
12265:
12253:
12231:
12225:
12206:
12194:
12178:
12159:
12126:
12117:
12108:
12096:
12081:
12075:
12057:
12048:
12015:
12003:
11988:
11982:
11915:
11909:
11864:
11855:
11847:
11838:
11799:
11787:
11772:
11766:
11745:
11736:
11728:
11719:
11681:
11669:
11653:
11647:
11610:
11598:
11559:
11537:
11449:
11427:
11407:
11385:
11360:{\displaystyle {\mathcal {F}}}
11258:
11196:
11184:
11113:component of the series where
11064:component of the series where
10898:{\displaystyle \epsilon >0}
10769:
10757:
10709:
10703:
10694:
10688:
10654:
10642:
10579:
10573:
10565:
10559:
10533:
10527:
10518:
10512:
10478:
10466:
10406:
10399:
10385:
10381:
10375:
10366:
10360:
10354:
10325:
10313:
10207:
10200:
10186:
10182:
10176:
10167:
10161:
10155:
10126:
10114:
10060:
10054:
10049:
10043:
10004:
9992:
9892:
9880:
9831:
9824:
9810:
9806:
9800:
9791:
9785:
9779:
9750:
9738:
9729:
9719:
9709:
9703:
9698:
9692:
9661:
9655:
9643:
9637:
9631:
9619:
9603:
9599:
9593:
9584:
9578:
9572:
9543:
9531:
9522:
9512:
9502:
9496:
9491:
9485:
9470:
9464:
9450:
9446:
9440:
9431:
9425:
9419:
9386:
9380:
9362:
9356:
9344:
9338:
9332:
9320:
9309:
9305:
9299:
9290:
9284:
9278:
9245:
9239:
9193:
9187:
9165:
9159:
9151:
9145:
9133:
9127:
9121:
9109:
9093:
9090:
9084:
9075:
9069:
9063:
9012:
9006:
8998:
8992:
8980:
8974:
8968:
8956:
8935:
8932:
8926:
8917:
8911:
8905:
8875:
8869:
8817:
8811:
8802:
8790:
8760:
8739:
8727:
8724:
8708:
8702:
8609:
8602:
8588:
8584:
8578:
8569:
8563:
8557:
8528:
8516:
8480:
8468:
8412:
8400:
8349:
8323:
8310:
8298:
8292:
8266:
8247:
8234:
8222:
8190:
8178:
8159:
8129:
8103:
8083:
8057:
8037:
8025:
7998:
7986:
7951:
7925:
7909:
7883:
7867:
7855:
7798:for independent distributions
7785:
7776:
7763:
7754:
7738:
7720:
7697:
7688:
7675:
7666:
7650:
7632:
7555:{\displaystyle P(dx)=p(x)\,dx}
7542:
7536:
7527:
7518:
7495:
7489:
7466:
7460:
7435:{\displaystyle y_{b}=y(x_{b})}
7429:
7416:
7386:{\displaystyle y_{a}=y(x_{a})}
7380:
7367:
7313:
7307:
7301:
7290:
7284:
7278:
7256:
7250:
7244:
7181:
7177:
7171:
7147:
7142:
7139:
7133:
7127:
7121:
7109:
7105:
7099:
7075:
7070:
7067:
7061:
7055:
7049:
7026:
7022:
7016:
6992:
6988:
6985:
6979:
6973:
6967:
6905:
6899:
6891:
6885:
6866:
6860:
6818:
6806:
6767:
6763:
6757:
6731:
6703:
6699:
6693:
6667:
6663:
6657:
6651:
6632:
6626:
6620:
6601:
6595:
6586:
6577:
6546:
6542:
6536:
6510:
6506:
6503:
6497:
6491:
6485:
6466:
6460:
6454:
6435:
6429:
6420:
6411:
6388:
6382:
6348:
6336:
6262:
6250:
6225:
6219:
6171:
6165:
6156:
6150:
6127:
6118:
6112:
6106:
6097:
6088:
6065:
6056:
6050:
6044:
6035:
6026:
5970:
5958:
5915:
5903:
5821:
5809:
5779:
5773:
5750:
5744:
5735:
5723:
5696:
5684:
5645:
5633:
5592:
5586:
5549:
5537:
5505:
5499:
5488:
5476:
5455:
5449:
5431:
5425:
5393:
5387:
5369:
5363:
5330:
5318:
5255:is the length of the code for
5188:
5175:
5161:out of a set of possibilities
5036:(geometrically interpreted as
4999:
4987:
4957:
4945:
4841:prior probability distribution
4826:
4814:
4753:
4741:
4679:
4667:
4617:
4611:
4596:
4590:
4517:
4511:
4496:
4490:
4475:
4469:
4244:
4238:
4230:
4224:
4205:
4199:
4166:
4154:
4099:
4093:
4078:
4072:
4057:
4051:
3829:
3823:
3815:
3809:
3790:
3784:
3751:
3739:
3680:
3668:
3638:
3626:
3597:{\displaystyle {\frac {1}{3}}}
3569:{\displaystyle {\frac {1}{3}}}
3541:{\displaystyle {\frac {1}{3}}}
3507:
3501:
3390:
3384:
3126:is as the relative entropy of
3110:
3098:
2998:
2986:
2959:if information is measured in
2935:or the uniform measure on the
2823:
2808:
2792:
2786:
2775:
2769:
2744:
2738:
2709:
2697:
2619:
2604:
2598:
2592:
2583:
2568:
2539:
2524:
2518:
2512:
2503:
2488:
2460:{\displaystyle {\mathcal {X}}}
2423:. Continuing in this case, if
2389:
2374:
2358:
2343:
2335:
2320:
2295:
2280:
2272:
2257:
2225:
2213:
2165:
2150:
2144:
2138:
2129:
2114:
2032:
2017:
2006:
1991:
1950:
1935:
1919:
1904:
1896:
1881:
1836:
1824:
1669:
1663:
1655:
1649:
1627:
1621:
1594:
1582:
1522:
1516:
1492:
1458:
1452:
1399:{\displaystyle \ Q(x)\neq 0\ }
1384:
1378:
1284:
1278:
1243:
1237:
1172:
1166:
1155:
1149:
1124:
1118:
1086:
1074:
1025:
1019:
1008:
1002:
977:
971:
942:
930:
809:
797:
788:
776:
767:
755:
745:" as the symmetrized quantity
678:
666:
415:rather than one optimized for
394:
382:
301:), it satisfies a generalized
215:
209:
198:
192:
167:
161:
132:
120:
72:
60:
1:
26657:2009. One-hour video lecture.
26419:Séminaire de Probabilités XII
25771:10.1080/03610926.2021.1921214
25097:
24565:{\displaystyle D_{\text{JS}}}
23963:Kullback & Leibler (1951)
23445:{\displaystyle k\ln(p/p_{o})}
22945:) is also minimized. Thus if
18942:, we can encode them (with a
17941:, from the true distribution
16075:{\displaystyle p_{(x,\rho )}}
15115:Einstein summation convention
14674:§ Symmetrised divergence
8135:{\displaystyle (P_{2},Q_{2})}
8089:{\displaystyle (P_{1},Q_{1})}
5876:
5849:Relative entropy relates to "
5840:§ Symmetrised divergence
5113:
5102:{\displaystyle D_{\text{KL}}}
5046:maximum likelihood estimation
4793:Expressed in the language of
4553:
849:
453:Kullback & Leibler (1951)
26598:in 1968; reprinted in 1978:
25113:Csiszar, I (February 1975).
25015:Akaike information criterion
24651:{\displaystyle \delta (p,q)}
24364:{\displaystyle \lambda =0.5}
23936:Akaike information criterion
23006:{\displaystyle F\equiv U-TS}
22325:Principle of Maximum Entropy
22116:conditional relative entropy
20903:Bayesian experimental design
20897:Bayesian experimental design
20627:may be <, = or > than
20034:If a further piece of data,
19801:{\displaystyle p(x\mid y,I)}
19543:This distribution has a new
19414:{\displaystyle p(x\mid y,I)}
19068:is thus defined as follows.
18799:{\displaystyle P_{U}(X)P(Y)}
17184:representing certainty that
15017:{\displaystyle P=P(\theta )}
14560:While relative entropy is a
13211:{\displaystyle L_{1}M=L_{0}}
12552:For alternative proof using
11526:be a real-valued integrable
11267:{\displaystyle n\to \infty }
11148:. The absolute value of the
10950:{\displaystyle U<\infty }
9905:via the Taylor expansion of
5756:{\displaystyle H(P,P)=:H(P)}
5051:The relative entropy is the
4715:
1358:{\displaystyle \ +\infty \ }
438:
281:), and does not satisfy the
7:
25351:MacKay, David J.C. (2003).
25260:Kullback & Leibler 1951
25007:
24951:Kolmogorov–Smirnov distance
23838:quantum information science
22688:{\displaystyle s=k\ln(1/p)}
21982:the prior distribution for
20828:All one can say is that on
20067:{\displaystyle Y_{2}=y_{2}}
15981:{\displaystyle \theta _{0}}
15628:one has up to second order
15106:{\displaystyle \theta _{0}}
13050:{\displaystyle L_{0},L_{1}}
12559:
11296:
7831:{\displaystyle Q_{1},Q_{2}}
7615:{\displaystyle P_{1},P_{2}}
7450:is a dimensioned variable,
5882:Relative entropy is always
5863:characterization of entropy
5130:In information theory, the
4704:using a code optimized for
609:{\displaystyle H_{1},H_{2}}
433:evidence lower bound (ELBO)
411:using a code optimized for
10:
26707:
26585:Kullback, Solomon (1959),
24967:
24335:{\displaystyle 1-\lambda }
24078:Population Stability Index
23907:
23901:
23882:over all separable states
23697:Quantum information theory
22312:has one particular value.
22250:over the whole support of
22107:{\displaystyle q(x\mid a)}
22072:{\displaystyle p(x\mid a)}
21014:discrimination information
20914:Discrimination information
20215:{\displaystyle p(x\mid I)}
19760:{\displaystyle p(x\mid I)}
19373:{\displaystyle p(x\mid I)}
18966:), which will be equal to
17026:
14541:{\displaystyle \log _{2}k}
14508:{\displaystyle \log _{2}k}
12548:. The conclusion follows.
11274:, so the series diverges.
10924:{\displaystyle \rho >0}
8830:. The first derivative of
2193:), this can be written as
1560:continuous random variable
1406:everywhere, provided that
1299:{\displaystyle \ P(x)=0\ }
1258:{\displaystyle \ Q(x)=0\ }
1201:In other words, it is the
902:the relative entropy from
832:discrimination information
419:. Note that the roles of
26610:Jeffreys, Harold (1948).
26568:10.1007/978-4-431-55978-8
25367:– via Google Books.
25305:The American Statistician
25070:Jensen–Shannon divergence
24590:Fisher information metric
24373:Jensen–Shannon divergence
23572:, where relative entropy
23069:{\displaystyle G=U+PV-TS}
22938:{\displaystyle T\times A}
19034:probability distributions
18944:lossless data compression
18051:(as opposed to the usual
14959:Fisher information metric
14953:Fisher information metric
14685:probability distributions
14576:distance, satisfying the
13010:yields the divergence in
12569:Suppose that we have two
9964:{\displaystyle \alpha =1}
7580:independent distributions
6366:parameter transformations
6177:{\displaystyle p(x)=q(x)}
5666:is from the distribution
5248:{\displaystyle \ell _{i}}
5026:Fisher information metric
4916:distance, satisfying the
4774:would be used instead of
4641:
3310:), each with probability
2845:Note that such a measure
26515:Rubner, Y.; Tomasi, C.;
25828:10.1348/000711010x522227
25511:Soklakov, A. N. (2020).
25075:Quantum relative entropy
25045:Entropy power inequality
24621:total-variation distance
24309:{\displaystyle \lambda }
24100:{\displaystyle \lambda }
23718:quantum relative entropy
22971:and number of molecules
21468:On the entropy scale of
18878:{\displaystyle P_{U}(X)}
18197:which is equivalent to:
18007:{\displaystyle P_{U}(X)}
17934:{\displaystyle P_{U}(X)}
17629:{\displaystyle P(X)P(Y)}
17054:discrete random variable
16997:variation of information
16991:Variation of information
13882:the above simplifies to
12613:and with (non-singular)
11248:, which is unbounded as
11086:{\displaystyle P\leq 2Q}
10823:{\displaystyle P\leq 2Q}
10260:{\displaystyle P\leq 2Q}
8659:{\displaystyle P\leq 2Q}
7444:dimensionally consistent
5670:. Geometrically it is a
4570:based on an observation
2662:{\displaystyle \ \mu \ }
2053:Radon–Nikodym derivative
1467:{\displaystyle \ P(x)\ }
1327:{\displaystyle +\infty }
738:{\displaystyle \mu _{2}}
711:{\displaystyle \mu _{1}}
529:{\displaystyle \mu _{1}}
341:Introduction and context
279:variation of information
245:of the KL divergence of
91:probability distribution
26676:Entropy and information
26535:10.1023/A:1026543900054
26345:Nielsen, Frank (2020).
26274:Nielsen, Frank (2019).
26079:10.1103/physrev.108.171
26029:10.1103/physrev.106.620
25651:Hobson, Arthur (1971).
25414:Bishop, Christopher M.
25174:10.1214/aoms/1177729694
24940:quadratic form distance
24913:{\displaystyle \alpha }
24803:(see, also, Tsybakov):
24592:. It is similar to the
23917:distance to equilibrium
23367:by relative entropy or
22919:is fixed, free energy (
22194:) This is minimized if
15298:{\displaystyle \theta }
15079:{\displaystyle \theta }
14982:{\displaystyle \theta }
12541:{\displaystyle Q=Q^{*}}
11921:{\displaystyle \exp(h)}
11336:{\displaystyle \sigma }
11316:{\displaystyle \Theta }
11057:{\displaystyle 1-\rho }
11011:{\displaystyle Q\leq U}
10852:{\displaystyle P>2Q}
5068:statistical divergences
4857:is used to approximate
4634:is actually drawn from
1054:which is equivalent to
285:. Instead, in terms of
21:mathematical statistics
26145:Fraundorf, P. (2007).
26050:Jaynes, E. T. (1957).
26000:Jaynes, E. T. (1957).
25983:Tribus, Myron (1959).
25734:10.1002/cpa.3160360204
25132:10.1214/aop/1176996454
25092:Bhattacharyya distance
25060:Information gain ratio
25040:Entropic value at risk
24957:earth mover's distance
24930:histogram intersection
24914:
24888:
24789:
24736:
24652:
24566:
24545:We can also interpret
24536:
24475:
24365:
24336:
24310:
24271:
24101:
24032:
23958:Symmetrised divergence
23941:mean squared deviation
23876:
23827:
23683:
23566:
23527:
23500:
23473:
23446:
23396:
23361:
23323:
23264:
23203:
23128:
23101:
23070:
23007:
22965:
22939:
22898:
22816:
22785:
22765:
22689:
22636:
22629:
22590:
22553:
22511:
22450:
22302:
22273:
22244:
22188:
22108:
22073:
22038:
22009:
21976:
21944:
21718:
21663:
21622:
21621:{\displaystyle p(x,a)}
21584:
21535:
21452:
21293:
21260:
21233:
21203:
21176:
21145:
21118:
21091:
21064:
21037:
21006:
20887:
20819:
20621:
20505:
20380:may be ≤ or > than
20374:
20216:
20181:
20133:
20068:
20025:
19992:
19802:
19761:
19726:
19677:
19534:
19415:
19374:
19335:
19309:
19263:posterior distribution
19231:
19190:
18999:
18916:
18915:{\displaystyle P(X|Y)}
18879:
18835:
18834:{\displaystyle P(X,Y)}
18800:
18737:
18279:
18188:
18037:
18008:
17964:
17935:
17876:
17677:
17676:{\displaystyle P(X,Y)}
17630:
17589:
17588:{\displaystyle P(X,Y)}
17548:
17263:
17237:
17204:
17174:
17142:
16981:
16839:
16616:
16226:
16076:
16021:
15982:
15948:
15769:
15615:
15487:
15453:
15420:
15394:
15299:
15279:
15214:
15107:
15080:
15066:so that the parameter
15060:
15018:
14983:
14935:
14856:
14807:
14733:
14687:. More concretely, if
14666:
14624:
14542:
14509:
14470:
14379:
14341:
14303:
14258:
14119:, this simplifies to:
14113:
14062:
13861:
13772:
13540:
13461:
13379:
13268:
13212:
13161:
13106:
13051:
13004:
13003:{\displaystyle \ln(2)}
12951:
12653:
12607:
12542:
12509:
12451:
12275:
12133:
12025:
11953:
11922:
11874:
11812:
11694:
11566:
11508:
11507:{\displaystyle Q\ll P}
11482:
11481:{\displaystyle Q\ll P}
11456:
11414:
11375:, which formulate two
11361:
11337:
11317:
11268:
11242:
11162:
11142:
11107:
11087:
11058:
11032:
11012:
10986:
10951:
10925:
10899:
10873:
10853:
10824:
10796:
10747:
10632:
10456:
10298:
10261:
10233:
10104:
10034:
9965:
9939:
9919:
9899:
9857:
9214:
8844:
8824:
8767:
8680:
8660:
8631:
8506:
8445:
8419:
8376:
8136:
8090:
8044:
8005:
7961:
7832:
7792:
7704:
7616:
7556:
7502:
7473:
7436:
7387:
7338:
6775:
6717:
6561:
6395:
6355:
6269:
6232:
6197:
6178:
6134:
6072:
6006:
5977:
5931:
5844:§ Interpretations
5828:
5786:
5757:
5703:
5702:{\displaystyle H(P,Q)}
5652:
5599:
5556:
5516:
5276:
5249:
5218:
5155:
5132:Kraft–McMillan theorem
5127:
5103:
5042:information projection
5006:
4964:
4895:information projection
4833:
4760:
4686:
4624:
4539:
4124:
3687:
3645:
3598:
3570:
3542:
3514:
3481:
3453:
3425:
3397:
3352:
3338:
3304:
3245:
3218:
3192:
3120:
3008:
2917:
2859:
2836:
2663:
2629:
2549:
2461:
2437:
2402:
2175:
2095:
2045:
1963:
1778:
1704:
1541:
1468:
1430:
1400:
1359:
1328:
1300:
1259:
1192:
1045:
896:
816:
739:
712:
685:
684:{\displaystyle I(1:2)}
650:
610:
570:
530:
503:
476:
401:
293:, a generalization of
265:as a model instead of
232:
79:
26612:Theory of Probability
26591:John Wiley & Sons
25928:10.1214/ss/1177009939
25894:John Wiley & Sons
24935:Chi-squared statistic
24915:
24889:
24801:Bretagnolle and Huber
24790:
24737:
24653:
24567:
24537:
24476:
24366:
24337:
24311:
24272:
24102:
24033:
23908:Further information:
23877:
23828:
23684:
23567:
23528:
23526:{\displaystyle T_{o}}
23501:
23499:{\displaystyle V_{o}}
23474:
23472:{\displaystyle p_{o}}
23447:
23397:
23362:
23360:{\displaystyle T_{o}}
23324:
23265:
23204:
23129:
23127:{\displaystyle P_{o}}
23102:
23100:{\displaystyle T_{o}}
23071:
23008:
22977:Helmholtz free energy
22966:
22940:
22899:
22817:
22815:{\displaystyle J/K\}}
22786:
22771:then surprisal is in
22766:
22690:
22630:
22628:{\displaystyle T_{o}}
22607:
22591:
22554:
22512:
22451:
22303:
22274:
22245:
22189:
22109:
22074:
22039:
22010:
21977:
21945:
21719:
21664:
21623:
21585:
21536:
21534:{\displaystyle f_{0}}
21453:
21294:
21261:
21259:{\displaystyle H_{0}}
21234:
21232:{\displaystyle H_{1}}
21204:
21202:{\displaystyle H_{0}}
21177:
21175:{\displaystyle H_{1}}
21146:
21144:{\displaystyle H_{1}}
21119:
21117:{\displaystyle H_{0}}
21097:against a hypothesis
21092:
21090:{\displaystyle H_{1}}
21065:
21063:{\displaystyle H_{0}}
21038:
21036:{\displaystyle H_{1}}
21007:
20888:
20820:
20622:
20506:
20375:
20217:
20182:
20134:
20069:
20026:
19993:
19803:
19762:
19727:
19678:
19535:
19416:
19375:
19336:
19310:
19232:
19191:
19000:
18917:
18880:
18836:
18801:
18738:
18280:
18189:
18038:
18009:
17965:
17936:
17877:
17678:
17631:
17590:
17549:
17264:
17238:
17205:
17175:
17143:
16982:
16840:
16617:
16227:
16077:
16022:
15983:
15959:positive semidefinite
15949:
15770:
15616:
15488:
15459:, it changes only to
15454:
15421:
15395:
15300:
15280:
15215:
15108:
15081:
15061:
15019:
14984:
14936:
14857:
14817:then it is said that
14808:
14734:
14667:
14625:
14543:
14510:
14471:
14380:
14342:
14304:
14267:Uniform distributions
14259:
14114:
14063:
13862:
13752:
13551:variational inference
13541:
13441:
13353:
13269:
13213:
13162:
13107:
13052:
13005:
12952:
12654:
12608:
12543:
12510:
12452:
12276:
12134:
12026:
11954:
11952:{\displaystyle Q^{*}}
11923:
11875:
11813:
11695:
11567:
11509:
11483:
11457:
11415:
11362:
11338:
11318:
11269:
11243:
11163:
11143:
11108:
11106:{\displaystyle \rho }
11088:
11059:
11033:
11031:{\displaystyle \rho }
11013:
10987:
10952:
10926:
10900:
10874:
10854:
10825:
10797:
10727:
10612:
10436:
10278:
10262:
10234:
10084:
10014:
9966:
9940:
9920:
9900:
9858:
9215:
8845:
8825:
8768:
8681:
8661:
8632:
8486:
8446:
8420:
8377:
8137:
8091:
8045:
8043:{\displaystyle (P,Q)}
8006:
7962:
7833:
7793:
7705:
7617:
7557:
7503:
7474:
7437:
7388:
7339:
6776:
6718:
6562:
6396:
6356:
6270:
6233:
6198:
6179:
6135:
6073:
6007:
5978:
5932:
5829:
5787:
5758:
5704:
5653:
5616:The relative entropy
5600:
5557:
5517:
5277:
5275:{\displaystyle x_{i}}
5250:
5219:
5156:
5154:{\displaystyle x_{i}}
5121:
5104:
5074:Finance (game theory)
5038:dually flat manifolds
5007:
4965:
4834:
4761:
4687:
4625:
4540:
4125:
3688:
3646:
3599:
3571:
3543:
3515:
3482:
3454:
3426:
3398:
3350:
3339:
3337:{\displaystyle p=1/3}
3305:
3246:
3219:
3217:{\displaystyle p=0.4}
3193:
3170:binomial distribution
3121:
3035:or as the divergence
3009:
2918:
2860:
2837:
2664:
2630:
2550:
2462:
2438:
2403:
2181:which exists because
2176:
2096:
2046:
1964:
1790:absolutely continuous
1779:
1723:probability densities
1705:
1542:
1469:
1431:
1401:
1360:
1329:
1301:
1260:
1193:
1046:
897:
836:statistical distances
817:
740:
713:
686:
651:
611:
571:
531:
509:per observation from
504:
502:{\displaystyle H_{2}}
477:
475:{\displaystyle H_{1}}
402:
233:
80:
26686:Information geometry
25381:. Machine learning.
24904:
24807:
24746:
24666:
24660:Pinsker's inequality
24627:
24613:probability distance
24598:statistical manifold
24588:proportional to the
24549:
24495:
24382:
24349:
24320:
24300:
24114:
24091:
24072:are the conditional
23972:
23844:
23735:
23579:
23537:
23510:
23483:
23456:
23406:
23374:
23344:
23274:
23213:
23138:
23111:
23084:
23036:
22982:
22949:
22923:
22864:
22854:entropy maximization
22852:). This constrained
22795:
22775:
22703:
22650:
22612:
22563:
22521:
22479:
22375:
22334:differential entropy
22301:{\displaystyle u(a)}
22283:
22272:{\displaystyle u(a)}
22254:
22198:
22126:
22083:
22048:
22037:{\displaystyle u(a)}
22019:
22008:{\displaystyle u(a)}
21990:
21975:{\displaystyle p(a)}
21957:
21731:
21681:
21662:{\displaystyle u(a)}
21644:
21597:
21545:
21518:
21486:for uncertainty are
21306:
21292:{\displaystyle p(H)}
21274:
21243:
21216:
21186:
21159:
21128:
21101:
21074:
21047:
21020:
20922:
20836:
20631:
20525:
20384:
20229:
20191:
20143:
20082:
20038:
20009:
19815:
19771:
19736:
19690:
19554:
19432:
19384:
19349:
19319:
19269:
19210:
19075:
18978:
18889:
18853:
18810:
18762:
18304:
18204:
18062:
18053:differential entropy
18036:{\displaystyle P(X)}
18018:
17982:
17963:{\displaystyle P(X)}
17945:
17909:
17713:
17652:
17599:
17564:
17288:
17247:
17236:{\displaystyle P(i)}
17218:
17188:
17173:{\displaystyle P(i)}
17155:
17063:
17037:, also known as the
16855:
16639:
16238:
16089:
16047:
15992:
15965:
15789:
15635:
15500:
15467:
15430:
15404:
15313:
15289:
15227:
15124:
15090:
15070:
15028:
14993:
14973:
14876:
14867:Pinsker's inequality
14824:
14746:
14691:
14634:
14592:
14562:statistical distance
14519:
14486:
14391:
14351:
14313:
14275:
14125:
14075:
13889:
13560:
13281:
13222:
13179:
13116:
13061:
13021:
12982:
12670:
12620:
12577:
12519:
12464:
12291:
12146:
12035:
11967:
11936:
11900:
11829:
11710:
11579:
11534:
11492:
11466:
11424:
11382:
11347:
11327:
11307:
11252:
11172:
11152:
11117:
11097:
11068:
11042:
11022:
10996:
10961:
10935:
10909:
10883:
10863:
10834:
10805:
10271:
10242:
9975:
9949:
9929:
9909:
9867:
9224:
8854:
8834:
8777:
8696:
8670:
8666:almost surely w.r.t
8641:
8455:
8429:
8387:
8146:
8100:
8054:
8022:
8017:probability measures
7973:
7842:
7802:
7714:
7626:
7586:
7574:Relative entropy is
7512:
7501:{\displaystyle q(x)}
7483:
7472:{\displaystyle p(x)}
7454:
7397:
7348:
6789:
6727:
6571:
6405:
6394:{\displaystyle y(x)}
6376:
6323:
6242:
6211:
6196:{\displaystyle \mu }
6187:
6144:
6082:
6020:
5990:
5945:
5890:
5796:
5785:{\displaystyle H(P)}
5767:
5717:
5678:
5660:statistical distance
5620:
5578:
5529:
5301:
5259:
5232:
5169:
5138:
5124:normal distributions
5086:
5034:exponential families
4974:
4932:
4902:statistical distance
4865:Information geometry
4801:
4766:is often called the
4728:
4654:
4578:
4560:Neyman–Pearson lemma
4137:
3722:
3713:units of information
3655:
3613:
3581:
3553:
3525:
3513:{\displaystyle Q(x)}
3495:
3464:
3436:
3408:
3396:{\displaystyle P(x)}
3378:
3314:
3266:
3232:
3202:
3176:
3082:
3048:. This reflects the
2970:
2869:
2858:{\displaystyle \mu }
2849:
2684:
2647:
2635:exist (meaning that
2559:
2479:
2467:for which densities
2447:
2436:{\displaystyle \mu }
2427:
2200:
2105:
2075:
1976:
1811:
1755:
1569:
1481:
1443:
1410:
1369:
1365:is possible even if
1340:
1315:
1269:
1228:
1061:
917:
873:
865:defined on the same
749:
722:
695:
660:
620:
580:
540:
513:
486:
459:
369:
287:information geometry
107:
87:statistical distance
47:
26375:2020Entrp..22..221N
26304:2019Entrp..21..485N
26173:2008Cmplx..13c..18F
26124:1971SciAm.225c.179T
26112:Scientific American
26071:1957PhRv..108..171J
26021:1957PhRv..106..620J
25914:Statistical Science
25625:Novak S.Y. (2011),
25539:2020Entrp..22..860S
24051:Jeffreys divergence
23336:More generally the
22964:{\displaystyle T,V}
21299:of the hypotheses,
20024:{\displaystyle Y=y}
19334:{\displaystyle Y=y}
19315:. If some new fact
19255:Bayesian statistics
18295:conditional entropy
18289:Conditional entropy
17262:{\displaystyle i=m}
17203:{\displaystyle i=m}
17039:information content
17029:Information content
17001:conditional entropy
16755:
16667:
16558:
16432:
16315:
15419:{\displaystyle P=Q}
14926:
14897:
14846:
14586:Pythagorean theorem
14578:triangle inequality
14556:Relation to metrics
14041:
13982:
13844:
13810:
13792:
13690:
13666:
13156:
13101:
12854:
12770:
12615:covariance matrices
11301: —
8444:{\displaystyle P=Q}
7446:quantity, since if
7237:
6960:
6856:
6005:{\displaystyle P=Q}
5853:" in the theory of
4926:Pythagorean theorem
4918:triangle inequality
4889:that is closest to
3609:Relative entropies
3191:{\displaystyle N=2}
1736:More generally, if
1617:
1308:absolute continuity
844:Jeffreys divergence
303:Pythagorean theorem
283:triangle inequality
26596:Dover Publications
26428:10.1007/bfb0064610
26181:10.1002/cplx.20195
25631:Chapman & Hall
25486:Bell Syst. Tech. J
25385:. Cross validated.
25050:Hellinger distance
25025:Bregman divergence
24926:Hellinger distance
24910:
24884:
24785:
24732:
24648:
24562:
24532:
24471:
24361:
24332:
24306:
24267:
24097:
24028:
23948:maximum likelihood
23872:
23823:
23679:
23562:
23523:
23496:
23469:
23442:
23392:
23357:
23319:
23260:
23199:
23124:
23097:
23066:
23003:
22975:are constant, the
22961:
22935:
22910:partition function
22894:
22812:
22784:{\displaystyle \{}
22781:
22761:
22685:
22637:
22625:
22586:
22549:
22507:
22446:
22298:
22269:
22240:
22184:
22104:
22069:
22034:
22005:
21972:
21940:
21714:
21659:
21618:
21580:
21531:
21480:Riemann hypothesis
21448:
21289:
21256:
21229:
21199:
21172:
21141:
21124:, when hypothesis
21114:
21087:
21060:
21033:
21002:
20883:
20832:, averaging using
20815:
20617:
20501:
20500:
20397:
20370:
20241:
20212:
20177:
20129:
20064:
20021:
19988:
19899:
19798:
19757:
19722:
19673:
19615:
19530:
19411:
19370:
19331:
19305:
19259:prior distribution
19227:
19186:
18995:
18912:
18875:
18831:
18796:
18733:
18731:
18275:
18184:
18080:
18033:
18004:
17960:
17931:
17872:
17870:
17673:
17626:
17585:
17544:
17542:
17279:mutual information
17273:Mutual information
17259:
17233:
17200:
17170:
17138:
17052:When applied to a
16977:
16835:
16735:
16647:
16629:is independent of
16612:
16610:
16538:
16412:
16295:
16222:
16072:
16017:
15978:
15944:
15782:of the divergence
15765:
15611:
15483:
15449:
15416:
15390:
15295:
15285:a small change of
15275:
15210:
15103:
15076:
15056:
15014:
14979:
14931:
14852:
14803:
14764:
14729:
14662:
14620:
14538:
14505:
14466:
14375:
14340:{\displaystyle q=}
14337:
14302:{\displaystyle p=}
14299:
14254:
14109:
14058:
14027:
13968:
13857:
13830:
13796:
13778:
13676:
13652:
13536:
13264:
13208:
13157:
13142:
13102:
13087:
13047:
13000:
12947:
12837:
12753:
12649:
12603:
12538:
12505:
12447:
12271:
12129:
12021:
11949:
11918:
11894:
11870:
11808:
11690:
11562:
11504:
11478:
11452:
11410:
11377:probability spaces
11357:
11333:
11313:
11299:
11264:
11238:
11158:
11138:
11103:
11083:
11054:
11028:
11008:
10982:
10947:
10921:
10895:
10869:
10849:
10820:
10792:
10790:
10679:
10503:
10350:
10257:
10229:
10227:
10151:
9961:
9935:
9915:
9895:
9863:Hence solving for
9853:
9851:
9775:
9568:
9415:
9274:
9210:
9208:
9062:
8904:
8840:
8820:
8763:
8676:
8656:
8627:
8553:
8441:
8415:
8372:
8132:
8086:
8040:
8001:
7957:
7828:
7788:
7700:
7612:
7552:
7498:
7469:
7432:
7383:
7334:
7332:
7209:
6932:
6828:
6771:
6755:
6713:
6691:
6557:
6534:
6391:
6361:can be calculated.
6351:
6265:
6228:
6193:
6174:
6130:
6068:
6016:In particular, if
6002:
5973:
5937:a result known as
5927:
5867:mutual information
5824:
5782:
5753:
5699:
5648:
5595:
5552:
5512:
5510:
5421:
5359:
5272:
5245:
5214:
5151:
5128:
5099:
5053:Bregman divergence
5002:
4960:
4829:
4795:Bayesian inference
4756:
4720:In the context of
4682:
4646:In the context of
4620:
4535:
4533:
4195:
4120:
4118:
3780:
3707:to get results in
3683:
3641:
3594:
3566:
3538:
3510:
3477:
3449:
3421:
3393:
3353:
3334:
3300:
3244:{\displaystyle x=}
3241:
3214:
3188:
3116:
3054:Bayesian inference
3004:
2913:
2855:
2832:
2659:
2625:
2545:
2457:
2443:is any measure on
2433:
2398:
2171:
2091:
2063:, i.e. the unique
2041:
1959:
1774:
1700:
1600:
1550:For distributions
1537:
1506:
1464:
1426:
1396:
1355:
1324:
1296:
1255:
1188:
1114:
1041:
967:
892:
812:
735:
708:
681:
646:
606:
566:
526:
499:
472:
397:
299:exponential family
289:, it is a type of
228:
157:
75:
26594:. Republished by
26577:978-4-431-55977-1
26478:978-1-4419-2709-5
26437:978-3-540-08761-8
26384:10.3390/e22020221
26313:10.3390/e21050485
26252:978-1-4419-2973-0
26212:Wildlife Research
26100:footnote page 52.
25969:978-0-521-88068-8
25703:video lecture by
25639:978-1-4398-3574-6
25548:10.3390/e22080860
24970:Data differencing
24964:Data differencing
24898:Rényi divergences
24879:
24727:
24709:
24700:
24584:-divergences, is
24559:
24512:
24453:
24444:
24415:
24406:
24392:
24222:
24155:
24058:feature selection
24010:
23982:
23932:statistical model
23854:
23745:
23728:is defined to be
23665:
23641:
23624:
23031:Gibbs free energy
22915:When temperature
22835:average surprisal
22531:
22489:
22425:
22136:
22122:) and denoted by
21901:
21838:
21741:
21555:
21397:
21316:
20932:
20918:Relative entropy
20901:A common goal in
20746:
20641:
20535:
20494:
20388:
20364:
20232:
19982:
19890:
19825:
19606:
19528:
19249:Bayesian updating
19165:
19066:probability space
18968:Shannon's Entropy
18667:
18523:
18459:
18363:
18232:
18172:
18065:
17814:
17595:from the product
17495:
17417:
17330:
17108:
17093:
17009:probability space
16966:
16933:
16803:
16701:
16510:
16384:
16267:
16220:
16171:
16122:
16029:Riemannian metric
15901:
15869:
15699:
15645:
15559:
15527:
15372:
14927:
14898:
14847:
14772:
14749:
14644:
14602:
14464:
14401:
14135:
14056:
14043:
13960:
13899:
13750:
13570:
13526:
13346:
13291:
12933:
12735:
12680:
12474:
12385:
12323:
12191:
12156:
12112:
12019:
11892:
11868:
11803:
11749:
11666:
11620:
11295:
11225:
11200:
11161:{\displaystyle n}
10872:{\displaystyle 0}
10773:
10660:
10658:
10583:
10484:
10482:
10422:
10331:
10329:
10223:
10132:
10130:
10072:
9989:
9938:{\displaystyle 0}
9918:{\displaystyle f}
9877:
9847:
9756:
9681:
9549:
9474:
9396:
9366:
9255:
9169:
9043:
9016:
8885:
8843:{\displaystyle f}
8787:
8721:
8679:{\displaystyle Q}
8625:
8534:
8532:
8465:
8397:
8355:
8320:
8263:
8156:
7983:
7969:Relative entropy
7922:
7880:
7852:
7317:
7304:
7281:
7247:
7186:
7169:
7124:
7097:
7052:
7014:
6970:
6909:
6803:
6754:
6690:
6654:
6623:
6533:
6488:
6457:
6333:
6205:almost everywhere
5955:
5939:Gibbs' inequality
5900:
5806:
5630:
5459:
5402:
5397:
5340:
5315:
5096:
4984:
4942:
4869:In applications,
4811:
4738:
4664:
4450:
4426:
4386:
4369:
4329:
4312:
4272:
4248:
4176:
4151:
4035:
4011:
3971:
3954:
3914:
3897:
3857:
3833:
3761:
3736:
3665:
3623:
3607:
3606:
3592:
3564:
3536:
3475:
3447:
3419:
3115:
3095:
3087:
3003:
2983:
2975:
2912:
2889:
2874:
2828:
2818:
2804:
2796:
2780:
2765:
2749:
2694:
2658:
2652:
2624:
2614:
2578:
2564:
2544:
2534:
2498:
2484:
2394:
2384:
2370:
2362:
2353:
2330:
2303:
2299:
2290:
2267:
2210:
2170:
2160:
2124:
2110:
2090:
2080:
2039:
2037:
2027:
2011:
2001:
1987:
1981:
1955:
1945:
1931:
1923:
1914:
1891:
1864:
1821:
1770:
1760:
1696:
1689:
1681:
1673:
1632:
1579:
1533:
1484:
1463:
1448:
1425:
1415:
1395:
1374:
1354:
1345:
1295:
1274:
1254:
1233:
1184:
1176:
1160:
1145:
1129:
1095:
1071:
1037:
1029:
1013:
998:
982:
948:
927:
910:is defined to be
888:
878:
379:
311:(Shannon) entropy
219:
203:
188:
172:
138:
117:
57:
26698:
26651:Relative Entropy
26615:
26593:
26581:
26556:Amari, Shun-ichi
26547:
26546:
26512:
26506:
26504:
26498:
26490:
26462:
26456:
26454:
26453:
26452:
26413:
26407:
26406:
26396:
26386:
26368:
26342:
26336:
26335:
26325:
26315:
26297:
26271:
26265:
26264:
26236:
26230:
26229:
26227:
26203:
26197:
26196:
26191:. Archived from
26166:
26142:
26136:
26135:
26107:
26101:
26099:
26089:
26083:
26082:
26056:
26047:
26041:
26040:
26006:
25997:
25991:
25990:
25980:
25974:
25973:
25949:
25943:
25942:
25940:
25930:
25904:
25898:
25897:
25885:
25876:
25875:
25857:
25848:
25847:
25807:
25801:
25800:
25798:
25789:
25783:
25782:
25764:
25755:(6): 1549–1568.
25744:
25738:
25737:
25717:
25711:
25701:Relative Entropy
25697:
25691:
25690:
25688:
25673:
25667:
25666:
25648:
25642:
25623:
25617:
25616:
25604:
25598:
25597:
25577:
25571:
25570:
25560:
25550:
25532:
25508:
25502:
25501:
25481:
25475:
25474:
25458:
25448:
25442:
25436:
25430:
25429:
25411:
25405:
25404:
25393:
25387:
25386:
25375:
25369:
25368:
25348:
25342:
25336:
25330:
25329:
25296:
25290:
25284:
25278:
25272:
25263:
25257:
25248:
25242:
25233:
25227:
25212:
25206:
25195:
25194:
25176:
25143:
25137:
25136:
25134:
25110:
25080:Solomon Kullback
24919:
24917:
24916:
24911:
24893:
24891:
24890:
24885:
24880:
24878:
24877:
24861:
24860:
24859:
24832:
24798:
24794:
24792:
24791:
24786:
24763:
24762:
24761:
24741:
24739:
24738:
24733:
24728:
24711:
24710:
24707:
24701:
24693:
24691:
24657:
24655:
24654:
24649:
24594:Hellinger metric
24583:
24579:
24575:
24571:
24569:
24568:
24563:
24561:
24560:
24557:
24541:
24539:
24538:
24533:
24513:
24505:
24487:
24480:
24478:
24477:
24472:
24455:
24454:
24451:
24445:
24437:
24417:
24416:
24413:
24407:
24399:
24394:
24393:
24390:
24370:
24368:
24367:
24362:
24341:
24339:
24338:
24333:
24315:
24313:
24312:
24307:
24295:
24291:
24287:
24283:
24276:
24274:
24273:
24268:
24224:
24223:
24220:
24157:
24156:
24153:
24126:
24125:
24106:
24104:
24103:
24098:
24071:
24067:
24064:problems, where
24043:§ Etymology
24037:
24035:
24034:
24029:
24012:
24011:
24008:
23984:
23983:
23980:
23893:
23885:
23881:
23879:
23878:
23873:
23856:
23855:
23852:
23832:
23830:
23829:
23824:
23747:
23746:
23743:
23727:
23723:
23711:
23707:
23703:density matrices
23688:
23686:
23685:
23680:
23675:
23671:
23670:
23666:
23664:
23663:
23651:
23642:
23634:
23629:
23625:
23623:
23622:
23610:
23571:
23569:
23568:
23563:
23555:
23554:
23532:
23530:
23529:
23524:
23522:
23521:
23505:
23503:
23502:
23497:
23495:
23494:
23478:
23476:
23475:
23470:
23468:
23467:
23451:
23449:
23448:
23443:
23438:
23437:
23428:
23401:
23399:
23398:
23393:
23366:
23364:
23363:
23358:
23356:
23355:
23331:Gibbs inequality
23328:
23326:
23325:
23320:
23269:
23267:
23266:
23261:
23259:
23258:
23249:
23244:
23243:
23225:
23224:
23208:
23206:
23205:
23200:
23195:
23194:
23185:
23171:
23170:
23133:
23131:
23130:
23125:
23123:
23122:
23106:
23104:
23103:
23098:
23096:
23095:
23075:
23073:
23072:
23067:
23028:
23024:
23020:
23016:
23012:
23010:
23009:
23004:
22974:
22970:
22968:
22967:
22962:
22944:
22942:
22941:
22936:
22918:
22907:
22903:
22901:
22900:
22895:
22851:
22847:
22839:
22829:
22825:
22821:
22819:
22818:
22813:
22805:
22790:
22788:
22787:
22782:
22770:
22768:
22767:
22762:
22760:
22756:
22755:
22754:
22724:
22698:
22694:
22692:
22691:
22686:
22678:
22645:
22634:
22632:
22631:
22626:
22624:
22623:
22595:
22593:
22592:
22587:
22570:
22558:
22556:
22555:
22550:
22533:
22532:
22529:
22516:
22514:
22513:
22508:
22491:
22490:
22487:
22474:
22470:
22462:
22455:
22453:
22452:
22447:
22427:
22426:
22423:
22405:
22382:
22367:
22363:
22359:
22356:with respect to
22355:
22351:
22311:
22307:
22305:
22304:
22299:
22278:
22276:
22275:
22270:
22249:
22247:
22246:
22241:
22193:
22191:
22190:
22185:
22138:
22137:
22134:
22113:
22111:
22110:
22105:
22078:
22076:
22075:
22070:
22043:
22041:
22040:
22035:
22014:
22012:
22011:
22006:
21985:
21981:
21979:
21978:
21973:
21949:
21947:
21946:
21941:
21903:
21902:
21899:
21890:
21886:
21840:
21839:
21836:
21822:
21821:
21743:
21742:
21739:
21723:
21721:
21720:
21715:
21676:
21672:
21668:
21666:
21665:
21660:
21639:
21635:
21631:
21627:
21625:
21624:
21619:
21589:
21587:
21586:
21581:
21576:
21575:
21557:
21556:
21553:
21540:
21538:
21537:
21532:
21530:
21529:
21513:
21504:
21503:
21470:information gain
21463:utility function
21457:
21455:
21454:
21449:
21399:
21398:
21395:
21371:
21370:
21343:
21342:
21318:
21317:
21314:
21298:
21296:
21295:
21290:
21265:
21263:
21262:
21257:
21255:
21254:
21238:
21236:
21235:
21230:
21228:
21227:
21208:
21206:
21205:
21200:
21198:
21197:
21181:
21179:
21178:
21173:
21171:
21170:
21150:
21148:
21147:
21142:
21140:
21139:
21123:
21121:
21120:
21115:
21113:
21112:
21096:
21094:
21093:
21088:
21086:
21085:
21069:
21067:
21066:
21061:
21059:
21058:
21042:
21040:
21039:
21034:
21032:
21031:
21011:
21009:
21008:
21003:
21001:
21000:
20991:
20990:
20963:
20962:
20941:
20940:
20934:
20933:
20930:
20892:
20890:
20889:
20884:
20867:
20866:
20854:
20853:
20824:
20822:
20821:
20816:
20814:
20813:
20777:
20776:
20755:
20754:
20748:
20747:
20744:
20735:
20734:
20719:
20718:
20685:
20684:
20672:
20671:
20650:
20649:
20643:
20642:
20639:
20626:
20624:
20623:
20618:
20616:
20615:
20579:
20578:
20566:
20565:
20544:
20543:
20537:
20536:
20533:
20510:
20508:
20507:
20502:
20499:
20495:
20493:
20473:
20463:
20462:
20440:
20419:
20418:
20396:
20379:
20377:
20376:
20371:
20369:
20365:
20363:
20343:
20333:
20332:
20320:
20319:
20297:
20276:
20275:
20263:
20262:
20240:
20221:
20219:
20218:
20213:
20186:
20184:
20183:
20178:
20167:
20166:
20138:
20136:
20135:
20130:
20119:
20118:
20106:
20105:
20077:
20073:
20071:
20070:
20065:
20063:
20062:
20050:
20049:
20030:
20028:
20027:
20022:
20004:
19997:
19995:
19994:
19989:
19987:
19983:
19981:
19961:
19935:
19898:
19886:
19885:
19834:
19833:
19827:
19826:
19823:
19807:
19805:
19804:
19799:
19766:
19764:
19763:
19758:
19731:
19729:
19728:
19723:
19697:
19682:
19680:
19679:
19674:
19614:
19599:
19598:
19568:
19567:
19561:
19539:
19537:
19536:
19531:
19529:
19527:
19507:
19463:
19420:
19418:
19417:
19412:
19379:
19377:
19376:
19371:
19344:
19340:
19338:
19337:
19332:
19314:
19312:
19311:
19306:
19244:
19240:
19236:
19234:
19233:
19228:
19217:
19195:
19193:
19192:
19187:
19167:
19166:
19163:
19145:
19110:
19109:
19082:
19063:
19059:
19055:
19051:
19043:
19039:
19024:
19020:
19008:
19004:
19002:
19001:
18996:
18985:
18973:
18965:
18957:
18952:prefix-free code
18948:entropy encoding
18941:
18929:
18925:
18921:
18919:
18918:
18913:
18905:
18884:
18882:
18881:
18876:
18865:
18864:
18848:
18840:
18838:
18837:
18832:
18805:
18803:
18802:
18797:
18774:
18773:
18753:
18749:
18742:
18740:
18739:
18734:
18732:
18728:
18724:
18723:
18719:
18709:
18708:
18696:
18692:
18669:
18668:
18665:
18651:
18650:
18617:
18580:
18569:
18553:
18552:
18525:
18524:
18521:
18461:
18460:
18457:
18427:
18399:
18398:
18365:
18364:
18361:
18315:
18284:
18282:
18281:
18276:
18259:
18254:
18234:
18233:
18230:
18193:
18191:
18190:
18185:
18173:
18171:
18157:
18143:
18090:
18089:
18079:
18042:
18040:
18039:
18034:
18013:
18011:
18010:
18005:
17994:
17993:
17977:
17969:
17967:
17966:
17961:
17940:
17938:
17937:
17932:
17921:
17920:
17904:
17892:
17888:
17881:
17879:
17878:
17873:
17871:
17867:
17863:
17853:
17852:
17831:
17830:
17816:
17815:
17812:
17782:
17778:
17774:
17761:
17760:
17724:
17693:
17690:if the value of
17689:
17682:
17680:
17679:
17674:
17647:
17643:
17635:
17633:
17632:
17627:
17594:
17592:
17591:
17586:
17553:
17551:
17550:
17545:
17543:
17497:
17496:
17493:
17481:
17480:
17465:
17419:
17418:
17415:
17403:
17402:
17387:
17332:
17331:
17328:
17268:
17266:
17265:
17260:
17242:
17240:
17239:
17234:
17213:
17209:
17207:
17206:
17201:
17179:
17177:
17176:
17171:
17147:
17145:
17144:
17139:
17134:
17130:
17126:
17125:
17110:
17109:
17106:
17095:
17094:
17091:
17070:
17035:self-information
17023:Self-information
16986:
16984:
16983:
16978:
16967:
16964:
16962:
16958:
16957:
16944:
16943:
16934:
16929:
16928:
16927:
16914:
16864:
16863:
16844:
16842:
16841:
16836:
16821:
16820:
16809:
16805:
16804:
16802:
16801:
16800:
16787:
16768:
16767:
16757:
16754:
16749:
16719:
16718:
16707:
16703:
16702:
16700:
16692:
16669:
16666:
16661:
16632:
16628:
16621:
16619:
16618:
16613:
16611:
16557:
16552:
16515:
16511:
16509:
16508:
16507:
16494:
16478:
16477:
16467:
16431:
16426:
16389:
16385:
16383:
16382:
16381:
16368:
16364:
16363:
16353:
16314:
16309:
16272:
16268:
16266:
16258:
16250:
16231:
16229:
16228:
16223:
16221:
16219:
16218:
16217:
16204:
16188:
16187:
16177:
16172:
16170:
16169:
16168:
16155:
16139:
16138:
16128:
16123:
16121:
16113:
16093:
16081:
16079:
16078:
16073:
16071:
16070:
16034:
16026:
16024:
16023:
16018:
16007:
16006:
15987:
15985:
15984:
15979:
15977:
15976:
15953:
15951:
15950:
15945:
15937:
15936:
15903:
15902:
15899:
15893:
15892:
15891:
15890:
15874:
15870:
15868:
15867:
15866:
15853:
15852:
15839:
15838:
15829:
15817:
15816:
15804:
15803:
15774:
15772:
15771:
15766:
15755:
15754:
15742:
15741:
15728:
15727:
15714:
15713:
15700:
15692:
15681:
15680:
15647:
15646:
15643:
15626:Taylor expansion
15620:
15618:
15617:
15612:
15595:
15594:
15561:
15560:
15557:
15551:
15550:
15549:
15548:
15532:
15528:
15526:
15525:
15524:
15508:
15492:
15490:
15489:
15484:
15482:
15481:
15458:
15456:
15455:
15450:
15448:
15447:
15425:
15423:
15422:
15417:
15399:
15397:
15396:
15391:
15386:
15385:
15373:
15371:
15370:
15369:
15356:
15348:
15343:
15339:
15338:
15325:
15324:
15308:
15304:
15302:
15301:
15296:
15284:
15282:
15281:
15276:
15274:
15273:
15264:
15263:
15242:
15241:
15219:
15217:
15216:
15211:
15200:
15199:
15187:
15186:
15176:
15175:
15157:
15156:
15112:
15110:
15109:
15104:
15102:
15101:
15085:
15083:
15082:
15077:
15065:
15063:
15062:
15057:
15052:
15051:
15023:
15021:
15020:
15015:
14988:
14986:
14985:
14980:
14968:
14964:
14940:
14938:
14937:
14932:
14915:
14914:
14913:
14889:
14888:
14887:
14861:
14859:
14858:
14853:
14848:
14838:
14836:
14835:
14812:
14810:
14809:
14804:
14787:
14786:
14774:
14773:
14770:
14763:
14738:
14736:
14735:
14730:
14719:
14718:
14706:
14705:
14683:on the space of
14671:
14669:
14668:
14663:
14646:
14645:
14642:
14629:
14627:
14626:
14621:
14604:
14603:
14600:
14551:
14547:
14545:
14544:
14539:
14531:
14530:
14514:
14512:
14511:
14506:
14498:
14497:
14481:
14475:
14473:
14472:
14467:
14465:
14463:
14452:
14441:
14430:
14426:
14425:
14424:
14415:
14414:
14403:
14402:
14399:
14384:
14382:
14381:
14376:
14346:
14344:
14343:
14338:
14309:enclosed within
14308:
14306:
14305:
14300:
14263:
14261:
14260:
14255:
14253:
14224:
14216:
14202:
14201:
14177:
14176:
14164:
14160:
14159:
14158:
14149:
14148:
14137:
14136:
14133:
14118:
14116:
14115:
14110:
14108:
14107:
14098:
14093:
14092:
14067:
14065:
14064:
14059:
14057:
14049:
14044:
14042:
14040:
14035:
14022:
14021:
14020:
14011:
14010:
13998:
13997:
13981:
13976:
13966:
13961:
13959:
13958:
13949:
13948:
13939:
13928:
13924:
13923:
13922:
13913:
13912:
13901:
13900:
13897:
13881:
13875:
13866:
13864:
13863:
13858:
13853:
13849:
13848:
13843:
13838:
13809:
13804:
13791:
13786:
13771:
13766:
13751:
13743:
13738:
13734:
13733:
13729:
13728:
13720:
13710:
13709:
13700:
13696:
13695:
13691:
13689:
13684:
13665:
13660:
13637:
13636:
13635:
13629:
13625:
13624:
13623:
13605:
13604:
13584:
13583:
13572:
13571:
13568:
13545:
13543:
13542:
13537:
13532:
13528:
13527:
13525:
13524:
13523:
13511:
13510:
13497:
13496:
13495:
13483:
13482:
13469:
13460:
13455:
13434:
13433:
13428:
13419:
13405:
13404:
13395:
13394:
13378:
13373:
13347:
13339:
13334:
13330:
13329:
13328:
13323:
13322:
13312:
13311:
13306:
13305:
13293:
13292:
13289:
13273:
13271:
13270:
13265:
13263:
13262:
13250:
13249:
13234:
13233:
13217:
13215:
13214:
13209:
13207:
13206:
13191:
13190:
13174:
13170:
13166:
13164:
13163:
13158:
13155:
13150:
13141:
13140:
13128:
13127:
13111:
13109:
13108:
13103:
13100:
13095:
13086:
13085:
13073:
13072:
13056:
13054:
13053:
13048:
13046:
13045:
13033:
13032:
13009:
13007:
13006:
13001:
12973:
12968:
12956:
12954:
12953:
12948:
12943:
12939:
12938:
12934:
12932:
12931:
12930:
12917:
12916:
12915:
12902:
12887:
12883:
12882:
12881:
12869:
12868:
12853:
12845:
12836:
12835:
12834:
12828:
12824:
12823:
12822:
12810:
12809:
12785:
12781:
12780:
12779:
12769:
12761:
12736:
12728:
12723:
12719:
12718:
12717:
12712:
12711:
12701:
12700:
12695:
12694:
12682:
12681:
12678:
12662:
12658:
12656:
12655:
12650:
12645:
12644:
12632:
12631:
12612:
12610:
12609:
12604:
12602:
12601:
12589:
12588:
12547:
12545:
12544:
12539:
12537:
12536:
12514:
12512:
12511:
12506:
12495:
12494:
12476:
12475:
12472:
12456:
12454:
12453:
12448:
12428:
12427:
12406:
12405:
12387:
12386:
12383:
12359:
12358:
12325:
12324:
12321:
12303:
12302:
12280:
12278:
12277:
12272:
12252:
12251:
12224:
12223:
12193:
12192:
12189:
12177:
12176:
12158:
12157:
12154:
12138:
12136:
12135:
12130:
12113:
12111:
12095:
12094:
12084:
12064:
12047:
12046:
12030:
12028:
12027:
12022:
12020:
12018:
12002:
12001:
11991:
11971:
11962:
11958:
11956:
11955:
11950:
11948:
11947:
11931:
11928:with respect to
11927:
11925:
11924:
11919:
11887:
11884:with respect to
11883:
11879:
11877:
11876:
11871:
11869:
11867:
11850:
11833:
11824:
11817:
11815:
11814:
11809:
11804:
11802:
11786:
11785:
11775:
11755:
11750:
11748:
11731:
11714:
11699:
11697:
11696:
11691:
11668:
11667:
11664:
11646:
11645:
11633:
11632:
11621:
11618:
11597:
11596:
11571:
11569:
11568:
11563:
11552:
11551:
11525:
11521:
11517:
11513:
11511:
11510:
11505:
11487:
11485:
11484:
11479:
11461:
11459:
11458:
11453:
11442:
11441:
11419:
11417:
11416:
11411:
11400:
11399:
11374:
11370:
11366:
11364:
11363:
11358:
11356:
11355:
11342:
11340:
11339:
11334:
11322:
11320:
11319:
11314:
11302:
11298:
11273:
11271:
11270:
11265:
11247:
11245:
11244:
11239:
11237:
11236:
11231:
11227:
11226:
11218:
11201:
11199:
11176:
11167:
11165:
11164:
11159:
11147:
11145:
11144:
11139:
11112:
11110:
11109:
11104:
11092:
11090:
11089:
11084:
11063:
11061:
11060:
11055:
11037:
11035:
11034:
11029:
11017:
11015:
11014:
11009:
10991:
10989:
10988:
10983:
10956:
10954:
10953:
10948:
10930:
10928:
10927:
10922:
10904:
10902:
10901:
10896:
10878:
10876:
10875:
10870:
10858:
10856:
10855:
10850:
10829:
10827:
10826:
10821:
10801:
10799:
10798:
10793:
10791:
10778:
10774:
10772:
10749:
10746:
10741:
10720:
10716:
10712:
10678:
10677:
10676:
10659:
10657:
10634:
10631:
10626:
10605:
10601:
10600:
10589:
10585:
10584:
10582:
10568:
10554:
10540:
10536:
10502:
10501:
10500:
10483:
10481:
10458:
10455:
10450:
10428:
10424:
10423:
10421:
10420:
10419:
10394:
10393:
10392:
10352:
10349:
10348:
10347:
10330:
10328:
10305:
10297:
10292:
10266:
10264:
10263:
10258:
10238:
10236:
10235:
10230:
10228:
10224:
10222:
10221:
10220:
10195:
10194:
10193:
10153:
10150:
10149:
10148:
10131:
10129:
10106:
10103:
10098:
10077:
10073:
10071:
10063:
10053:
10052:
10036:
10033:
10028:
9991:
9990:
9987:
9970:
9968:
9967:
9962:
9944:
9942:
9941:
9936:
9924:
9922:
9921:
9916:
9904:
9902:
9901:
9896:
9879:
9878:
9875:
9862:
9860:
9859:
9854:
9852:
9848:
9846:
9845:
9844:
9819:
9818:
9817:
9777:
9774:
9773:
9772:
9737:
9736:
9702:
9701:
9682:
9680:
9679:
9668:
9664:
9612:
9611:
9610:
9570:
9567:
9566:
9565:
9530:
9529:
9495:
9494:
9475:
9473:
9459:
9458:
9457:
9417:
9414:
9413:
9412:
9379:
9367:
9365:
9318:
9317:
9316:
9276:
9273:
9272:
9271:
9238:
9219:
9217:
9216:
9211:
9209:
9186:
9174:
9170:
9168:
9154:
9107:
9061:
9060:
9059:
9036:
9032:
9028:
9021:
9017:
9015:
9001:
8954:
8903:
8902:
8901:
8868:
8849:
8847:
8846:
8841:
8829:
8827:
8826:
8821:
8789:
8788:
8785:
8772:
8770:
8769:
8764:
8723:
8722:
8719:
8685:
8683:
8682:
8677:
8665:
8663:
8662:
8657:
8636:
8634:
8633:
8628:
8626:
8624:
8623:
8622:
8597:
8596:
8595:
8555:
8552:
8551:
8550:
8533:
8531:
8508:
8505:
8500:
8467:
8466:
8463:
8450:
8448:
8447:
8442:
8424:
8422:
8421:
8416:
8399:
8398:
8395:
8381:
8379:
8378:
8373:
8356:
8353:
8348:
8347:
8335:
8334:
8322:
8321:
8318:
8291:
8290:
8278:
8277:
8265:
8264:
8261:
8246:
8245:
8218:
8217:
8202:
8201:
8174:
8173:
8158:
8157:
8154:
8141:
8139:
8138:
8133:
8128:
8127:
8115:
8114:
8095:
8093:
8092:
8087:
8082:
8081:
8069:
8068:
8049:
8047:
8046:
8041:
8010:
8008:
8007:
8002:
7985:
7984:
7981:
7966:
7964:
7963:
7958:
7950:
7949:
7937:
7936:
7924:
7923:
7920:
7908:
7907:
7895:
7894:
7882:
7881:
7878:
7854:
7853:
7850:
7837:
7835:
7834:
7829:
7827:
7826:
7814:
7813:
7797:
7795:
7794:
7789:
7775:
7774:
7753:
7752:
7710:, and likewise
7709:
7707:
7706:
7701:
7687:
7686:
7665:
7664:
7621:
7619:
7618:
7613:
7611:
7610:
7598:
7597:
7565:self-information
7561:
7559:
7558:
7553:
7507:
7505:
7504:
7499:
7478:
7476:
7475:
7470:
7449:
7441:
7439:
7438:
7433:
7428:
7427:
7409:
7408:
7392:
7390:
7389:
7384:
7379:
7378:
7360:
7359:
7343:
7341:
7340:
7335:
7333:
7322:
7318:
7316:
7306:
7305:
7297:
7293:
7283:
7282:
7274:
7270:
7249:
7248:
7240:
7236:
7235:
7234:
7224:
7223:
7222:
7202:
7191:
7187:
7185:
7184:
7170:
7168:
7160:
7152:
7150:
7126:
7125:
7117:
7113:
7112:
7098:
7096:
7088:
7080:
7078:
7054:
7053:
7045:
7041:
7029:
7015:
7013:
7005:
6997:
6995:
6972:
6971:
6963:
6959:
6958:
6957:
6947:
6946:
6945:
6925:
6914:
6910:
6908:
6894:
6880:
6855:
6854:
6853:
6843:
6842:
6841:
6805:
6804:
6801:
6780:
6778:
6777:
6772:
6770:
6756:
6753:
6745:
6737:
6734:
6722:
6720:
6719:
6714:
6706:
6692:
6689:
6681:
6673:
6670:
6656:
6655:
6647:
6625:
6624:
6616:
6566:
6564:
6563:
6558:
6549:
6535:
6532:
6524:
6516:
6513:
6490:
6489:
6481:
6459:
6458:
6450:
6400:
6398:
6397:
6392:
6371:
6360:
6358:
6357:
6352:
6335:
6334:
6331:
6318:
6314:
6306:
6302:
6298:
6294:
6290:
6286:
6274:
6272:
6271:
6266:
6249:
6237:
6235:
6234:
6229:
6218:
6202:
6200:
6199:
6194:
6183:
6181:
6180:
6175:
6139:
6137:
6136:
6131:
6077:
6075:
6074:
6069:
6011:
6009:
6008:
6003:
5982:
5980:
5979:
5974:
5957:
5956:
5953:
5936:
5934:
5933:
5928:
5902:
5901:
5898:
5865:. Consequently,
5855:large deviations
5833:
5831:
5830:
5825:
5808:
5807:
5804:
5791:
5789:
5788:
5783:
5762:
5760:
5759:
5754:
5708:
5706:
5705:
5700:
5669:
5665:
5657:
5655:
5654:
5649:
5632:
5631:
5628:
5612:
5604:
5602:
5601:
5596:
5585:
5573:
5569:
5561:
5559:
5558:
5553:
5536:
5521:
5519:
5518:
5513:
5511:
5498:
5475:
5464:
5460:
5458:
5441:
5420:
5419:
5418:
5398:
5396:
5379:
5358:
5357:
5356:
5317:
5316:
5313:
5289:
5285:
5281:
5279:
5278:
5273:
5271:
5270:
5254:
5252:
5251:
5246:
5244:
5243:
5227:
5223:
5221:
5220:
5215:
5213:
5212:
5211:
5210:
5187:
5186:
5164:
5160:
5158:
5157:
5152:
5150:
5149:
5108:
5106:
5105:
5100:
5098:
5097:
5094:
5059:
5024:that equals the
5011:
5009:
5008:
5003:
4986:
4985:
4982:
4969:
4967:
4966:
4961:
4944:
4943:
4940:
4892:
4888:
4884:
4876:
4872:
4860:
4856:
4852:
4845:
4838:
4836:
4835:
4830:
4813:
4812:
4809:
4789:
4786:with respect to
4785:
4780:relative entropy
4777:
4773:
4768:information gain
4765:
4763:
4762:
4757:
4740:
4739:
4736:
4722:machine learning
4711:
4707:
4703:
4691:
4689:
4688:
4683:
4666:
4665:
4662:
4637:
4633:
4629:
4627:
4626:
4621:
4573:
4569:
4565:
4544:
4542:
4541:
4536:
4534:
4524:
4520:
4451:
4443:
4435:
4431:
4427:
4425:
4421:
4412:
4408:
4399:
4387:
4379:
4374:
4370:
4368:
4364:
4355:
4351:
4342:
4330:
4322:
4317:
4313:
4311:
4307:
4298:
4294:
4285:
4273:
4265:
4257:
4253:
4249:
4247:
4233:
4219:
4194:
4193:
4192:
4153:
4152:
4149:
4129:
4127:
4126:
4121:
4119:
4106:
4102:
4036:
4028:
4020:
4016:
4012:
4010:
4006:
3997:
3993:
3984:
3972:
3964:
3959:
3955:
3953:
3949:
3940:
3936:
3927:
3915:
3907:
3902:
3898:
3896:
3892:
3883:
3879:
3870:
3858:
3850:
3842:
3838:
3834:
3832:
3818:
3804:
3779:
3778:
3777:
3738:
3737:
3734:
3706:
3701:
3692:
3690:
3689:
3684:
3667:
3666:
3663:
3650:
3648:
3647:
3642:
3625:
3624:
3621:
3603:
3601:
3600:
3595:
3593:
3585:
3575:
3573:
3572:
3567:
3565:
3557:
3547:
3545:
3544:
3539:
3537:
3529:
3519:
3517:
3516:
3511:
3486:
3484:
3483:
3478:
3476:
3468:
3458:
3456:
3455:
3450:
3448:
3440:
3430:
3428:
3427:
3422:
3420:
3412:
3402:
3400:
3399:
3394:
3360:
3355:
3354:
3343:
3341:
3340:
3335:
3330:
3309:
3307:
3306:
3301:
3275:
3274:
3261:
3257:
3253:
3250:
3248:
3247:
3242:
3227:
3223:
3221:
3220:
3215:
3197:
3195:
3194:
3189:
3167:
3163:
3159:
3147:
3143:
3138:information gain
3135:
3129:
3125:
3123:
3122:
3117:
3113:
3097:
3096:
3093:
3085:
3077:
3066:
3047:
3041:
3034:
3028:
3024:
3020:
3013:
3011:
3010:
3005:
3001:
2985:
2984:
2981:
2973:
2958:
2933:Gaussian measure
2929:Lebesgue measure
2925:counting measure
2922:
2920:
2919:
2914:
2910:
2909:
2905:
2890:
2882:
2872:
2864:
2862:
2861:
2856:
2841:
2839:
2838:
2833:
2826:
2816:
2815:
2802:
2801:
2797:
2795:
2781:
2778:
2763:
2761:
2747:
2734:
2733:
2732:
2731:
2696:
2695:
2692:
2676:
2672:
2668:
2666:
2665:
2660:
2656:
2650:
2642:
2638:
2634:
2632:
2631:
2626:
2622:
2612:
2611:
2576:
2575:
2562:
2554:
2552:
2551:
2546:
2542:
2532:
2531:
2496:
2495:
2482:
2474:
2470:
2466:
2464:
2463:
2458:
2456:
2455:
2442:
2440:
2439:
2434:
2422:
2418:
2407:
2405:
2404:
2399:
2392:
2382:
2381:
2368:
2367:
2363:
2361:
2351:
2350:
2338:
2328:
2327:
2315:
2301:
2300:
2298:
2288:
2287:
2275:
2265:
2264:
2252:
2250:
2249:
2248:
2247:
2212:
2211:
2208:
2188:
2184:
2180:
2178:
2177:
2172:
2168:
2158:
2157:
2122:
2121:
2108:
2100:
2098:
2097:
2092:
2088:
2087:
2086:
2078:
2070:
2066:
2062:
2059:with respect to
2058:
2050:
2048:
2047:
2042:
2040:
2038:
2035:
2025:
2024:
2012:
2009:
1999:
1998:
1985:
1983:
1979:
1968:
1966:
1965:
1960:
1953:
1943:
1942:
1929:
1928:
1924:
1922:
1912:
1911:
1899:
1889:
1888:
1876:
1862:
1861:
1860:
1859:
1858:
1823:
1822:
1819:
1803:
1799:
1795:
1792:with respect to
1787:
1783:
1781:
1780:
1775:
1768:
1767:
1766:
1758:
1750:measurable space
1744:are probability
1743:
1739:
1732:
1728:
1720:
1716:
1709:
1707:
1706:
1701:
1694:
1687:
1686:
1679:
1678:
1674:
1672:
1658:
1644:
1630:
1616:
1611:
1581:
1580:
1577:
1557:
1553:
1546:
1544:
1543:
1538:
1531:
1505:
1504:
1503:
1473:
1471:
1470:
1465:
1461:
1446:
1435:
1433:
1432:
1427:
1423:
1422:
1421:
1413:
1405:
1403:
1402:
1397:
1393:
1372:
1364:
1362:
1361:
1356:
1352:
1343:
1335:
1333:
1331:
1330:
1325:
1305:
1303:
1302:
1297:
1293:
1272:
1264:
1262:
1261:
1256:
1252:
1231:
1223:
1216:
1212:
1208:
1197:
1195:
1194:
1189:
1182:
1181:
1177:
1175:
1161:
1158:
1143:
1141:
1127:
1113:
1112:
1111:
1073:
1072:
1069:
1050:
1048:
1047:
1042:
1035:
1034:
1030:
1028:
1014:
1011:
996:
994:
980:
966:
965:
964:
929:
928:
925:
909:
905:
901:
899:
898:
893:
886:
885:
884:
876:
864:
860:
821:
819:
818:
813:
744:
742:
741:
736:
734:
733:
717:
715:
714:
709:
707:
706:
690:
688:
687:
682:
655:
653:
652:
647:
645:
644:
632:
631:
615:
613:
612:
607:
605:
604:
592:
591:
575:
573:
572:
567:
565:
564:
552:
551:
535:
533:
532:
527:
525:
524:
508:
506:
505:
500:
498:
497:
481:
479:
478:
473:
471:
470:
445:Solomon Kullback
426:
422:
418:
414:
410:
406:
404:
403:
398:
381:
380:
377:
364:
360:
356:
352:
348:
335:machine learning
295:squared distance
272:
268:
264:
252:
248:
237:
235:
234:
229:
224:
220:
218:
204:
201:
186:
184:
170:
156:
155:
154:
119:
118:
115:
99:
95:
84:
82:
81:
76:
59:
58:
55:
37:relative entropy
25:Kullback–Leibler
26706:
26705:
26701:
26700:
26699:
26697:
26696:
26695:
26666:
26665:
26623:
26618:
26578:
26550:
26513:
26509:
26492:
26491:
26479:
26463:
26459:
26450:
26448:
26438:
26414:
26410:
26343:
26339:
26272:
26268:
26253:
26237:
26233:
26225:10.1071/WR99107
26204:
26200:
26143:
26139:
26108:
26104:
26090:
26086:
26059:Physical Review
26054:
26048:
26044:
26009:Physical Review
26004:
25998:
25994:
25989:. Van Nostrand.
25981:
25977:
25970:
25950:
25946:
25905:
25901:
25886:
25879:
25858:
25851:
25808:
25804:
25796:
25790:
25786:
25745:
25741:
25718:
25714:
25698:
25694:
25674:
25670:
25663:
25649:
25645:
25624:
25620:
25605:
25601:
25578:
25574:
25509:
25505:
25482:
25478:
25471:
25449:
25445:
25437:
25433:
25412:
25408:
25399:. Integration.
25395:
25394:
25390:
25377:
25376:
25372:
25365:
25349:
25345:
25337:
25333:
25297:
25293:
25285:
25281:
25273:
25266:
25258:
25251:
25243:
25236:
25228:
25215:
25207:
25198:
25144:
25140:
25111:
25104:
25100:
25088:
25084:Richard Leibler
25010:
24972:
24966:
24905:
24902:
24901:
24852:
24851:
24847:
24843:
24839:
24831:
24808:
24805:
24804:
24796:
24754:
24753:
24749:
24747:
24744:
24743:
24706:
24702:
24692:
24690:
24667:
24664:
24663:
24628:
24625:
24624:
24609:
24581:
24577:
24573:
24556:
24552:
24550:
24547:
24546:
24504:
24496:
24493:
24492:
24485:
24450:
24446:
24436:
24412:
24408:
24398:
24389:
24385:
24383:
24380:
24379:
24350:
24347:
24346:
24342:respectively.
24321:
24318:
24317:
24301:
24298:
24297:
24293:
24289:
24288:is drawn from,
24285:
24281:
24219:
24215:
24152:
24148:
24121:
24117:
24115:
24112:
24111:
24092:
24089:
24088:
24069:
24065:
24047:Harold Jeffreys
24007:
24003:
23979:
23975:
23973:
23970:
23969:
23960:
23952:maximum spacing
23912:
23910:Model selection
23906:
23900:
23891:
23883:
23851:
23847:
23845:
23842:
23841:
23840:the minimum of
23742:
23738:
23736:
23733:
23732:
23725:
23721:
23709:
23705:
23699:
23659:
23655:
23650:
23646:
23633:
23618:
23614:
23609:
23605:
23601:
23597:
23580:
23577:
23576:
23550:
23546:
23538:
23535:
23534:
23517:
23513:
23511:
23508:
23507:
23490:
23486:
23484:
23481:
23480:
23463:
23459:
23457:
23454:
23453:
23433:
23429:
23424:
23407:
23404:
23403:
23375:
23372:
23371:
23351:
23347:
23345:
23342:
23341:
23275:
23272:
23271:
23254:
23250:
23245:
23239:
23235:
23220:
23216:
23214:
23211:
23210:
23190:
23186:
23181:
23166:
23162:
23139:
23136:
23135:
23118:
23114:
23112:
23109:
23108:
23091:
23087:
23085:
23082:
23081:
23037:
23034:
23033:
23026:
23022:
23018:
23014:
22983:
22980:
22979:
22972:
22950:
22947:
22946:
22924:
22921:
22920:
22916:
22905:
22865:
22862:
22861:
22849:
22845:
22837:
22827:
22823:
22801:
22796:
22793:
22792:
22791:nats, bits, or
22776:
22773:
22772:
22747:
22743:
22720:
22710:
22706:
22704:
22701:
22700:
22696:
22674:
22651:
22648:
22647:
22643:
22619:
22615:
22613:
22610:
22609:
22602:
22566:
22564:
22561:
22560:
22528:
22524:
22522:
22519:
22518:
22486:
22482:
22480:
22477:
22476:
22472:
22468:
22460:
22422:
22418:
22401:
22378:
22376:
22373:
22372:
22365:
22361:
22357:
22353:
22349:
22309:
22284:
22281:
22280:
22255:
22252:
22251:
22199:
22196:
22195:
22133:
22129:
22127:
22124:
22123:
22084:
22081:
22080:
22049:
22046:
22045:
22020:
22017:
22016:
21991:
21988:
21987:
21983:
21958:
21955:
21954:
21898:
21894:
21835:
21831:
21830:
21826:
21808:
21804:
21738:
21734:
21732:
21729:
21728:
21682:
21679:
21678:
21674:
21670:
21645:
21642:
21641:
21637:
21633:
21629:
21598:
21595:
21594:
21571:
21567:
21552:
21548:
21546:
21543:
21542:
21525:
21521:
21519:
21516:
21515:
21511:
21501:
21500:
21496:
21394:
21390:
21366:
21362:
21338:
21334:
21313:
21309:
21307:
21304:
21303:
21275:
21272:
21271:
21250:
21246:
21244:
21241:
21240:
21223:
21219:
21217:
21214:
21213:
21193:
21189:
21187:
21184:
21183:
21166:
21162:
21160:
21157:
21156:
21135:
21131:
21129:
21126:
21125:
21108:
21104:
21102:
21099:
21098:
21081:
21077:
21075:
21072:
21071:
21054:
21050:
21048:
21045:
21044:
21027:
21023:
21021:
21018:
21017:
20996:
20995:
20986:
20982:
20958:
20954:
20936:
20935:
20929:
20925:
20923:
20920:
20919:
20916:
20908:Bayes d-optimal
20899:
20862:
20858:
20849:
20845:
20837:
20834:
20833:
20809:
20808:
20772:
20768:
20750:
20749:
20743:
20739:
20730:
20729:
20714:
20710:
20680:
20676:
20667:
20663:
20645:
20644:
20638:
20634:
20632:
20629:
20628:
20611:
20610:
20574:
20570:
20561:
20557:
20539:
20538:
20532:
20528:
20526:
20523:
20522:
20474:
20458:
20454:
20441:
20439:
20435:
20414:
20410:
20392:
20385:
20382:
20381:
20344:
20328:
20324:
20315:
20311:
20298:
20296:
20292:
20271:
20267:
20258:
20254:
20236:
20230:
20227:
20226:
20192:
20189:
20188:
20162:
20158:
20144:
20141:
20140:
20114:
20110:
20101:
20097:
20083:
20080:
20079:
20075:
20058:
20054:
20045:
20041:
20039:
20036:
20035:
20010:
20007:
20006:
20002:
19962:
19936:
19934:
19930:
19894:
19881:
19880:
19829:
19828:
19822:
19818:
19816:
19813:
19812:
19772:
19769:
19768:
19737:
19734:
19733:
19693:
19691:
19688:
19687:
19610:
19594:
19593:
19563:
19562:
19557:
19555:
19552:
19551:
19508:
19464:
19462:
19433:
19430:
19429:
19385:
19382:
19381:
19350:
19347:
19346:
19342:
19320:
19317:
19316:
19270:
19267:
19266:
19251:
19242:
19238:
19213:
19211:
19208:
19207:
19203:section above.
19162:
19158:
19141:
19105:
19101:
19078:
19076:
19073:
19072:
19061:
19057:
19053:
19049:
19041:
19037:
19022:
19018:
19006:
18981:
18979:
18976:
18975:
18971:
18963:
18955:
18939:
18936:
18927:
18923:
18901:
18890:
18887:
18886:
18860:
18856:
18854:
18851:
18850:
18846:
18811:
18808:
18807:
18769:
18765:
18763:
18760:
18759:
18751:
18747:
18730:
18729:
18704:
18700:
18682:
18678:
18674:
18670:
18664:
18660:
18659:
18655:
18646:
18642:
18615:
18614:
18576:
18567:
18566:
18548:
18544:
18520:
18516:
18456:
18452:
18425:
18424:
18394:
18390:
18360:
18356:
18331:
18311:
18307:
18305:
18302:
18301:
18291:
18255:
18250:
18229:
18225:
18205:
18202:
18201:
18158:
18144:
18142:
18085:
18081:
18069:
18063:
18060:
18059:
18019:
18016:
18015:
17989:
17985:
17983:
17980:
17979:
17975:
17946:
17943:
17942:
17916:
17912:
17910:
17907:
17906:
17902:
17899:random variates
17890:
17886:
17869:
17868:
17848:
17844:
17826:
17822:
17821:
17817:
17811:
17807:
17780:
17779:
17756:
17752:
17751:
17747:
17734:
17720:
17716:
17714:
17711:
17710:
17704:Shannon entropy
17700:
17698:Shannon entropy
17691:
17687:
17653:
17650:
17649:
17645:
17641:
17600:
17597:
17596:
17565:
17562:
17561:
17541:
17540:
17492:
17488:
17476:
17472:
17463:
17462:
17414:
17410:
17398:
17394:
17385:
17384:
17327:
17323:
17316:
17291:
17289:
17286:
17285:
17275:
17248:
17245:
17244:
17219:
17216:
17215:
17211:
17189:
17186:
17185:
17182:Kronecker delta
17156:
17153:
17152:
17121:
17117:
17105:
17101:
17100:
17096:
17090:
17086:
17066:
17064:
17061:
17060:
17031:
17025:
17017:
16993:
16963:
16953:
16949:
16945:
16939:
16938:
16923:
16919:
16915:
16913:
16859:
16858:
16856:
16853:
16852:
16810:
16796:
16792:
16788:
16763:
16759:
16758:
16756:
16750:
16739:
16734:
16731:
16730:
16708:
16693:
16670:
16668:
16662:
16651:
16646:
16643:
16642:
16640:
16637:
16636:
16630:
16626:
16609:
16608:
16553:
16542:
16516:
16503:
16499:
16495:
16473:
16469:
16468:
16466:
16462:
16459:
16458:
16427:
16416:
16390:
16377:
16373:
16369:
16359:
16355:
16354:
16352:
16348:
16345:
16344:
16310:
16299:
16273:
16259:
16251:
16249:
16245:
16241:
16239:
16236:
16235:
16213:
16209:
16205:
16183:
16179:
16178:
16176:
16164:
16160:
16156:
16134:
16130:
16129:
16127:
16114:
16094:
16092:
16090:
16087:
16086:
16054:
16050:
16048:
16045:
16044:
16041:
16032:
15999:
15995:
15993:
15990:
15989:
15972:
15968:
15966:
15963:
15962:
15932:
15928:
15898:
15894:
15886:
15882:
15875:
15862:
15858:
15848:
15844:
15840:
15834:
15830:
15828:
15825:
15824:
15812:
15808:
15796:
15792:
15790:
15787:
15786:
15750:
15746:
15734:
15730:
15723:
15719:
15709:
15705:
15691:
15676:
15672:
15642:
15638:
15636:
15633:
15632:
15590:
15586:
15556:
15552:
15544:
15540:
15533:
15520:
15516:
15512:
15507:
15504:
15503:
15501:
15498:
15497:
15477:
15473:
15468:
15465:
15464:
15443:
15439:
15431:
15428:
15427:
15405:
15402:
15401:
15381:
15377:
15365:
15361:
15357:
15349:
15347:
15334:
15330:
15326:
15320:
15316:
15314:
15311:
15310:
15309:direction, and
15306:
15290:
15287:
15286:
15269:
15265:
15259:
15255:
15237:
15233:
15228:
15225:
15224:
15195:
15191:
15182:
15178:
15171:
15167:
15152:
15148:
15125:
15122:
15121:
15097:
15093:
15091:
15088:
15087:
15071:
15068:
15067:
15047:
15043:
15029:
15026:
15025:
14994:
14991:
14990:
14974:
14971:
14970:
14966:
14962:
14955:
14947:total variation
14909:
14905:
14883:
14879:
14877:
14874:
14873:
14837:
14831:
14827:
14825:
14822:
14821:
14782:
14778:
14769:
14765:
14753:
14747:
14744:
14743:
14714:
14710:
14701:
14697:
14692:
14689:
14688:
14679:It generates a
14641:
14637:
14635:
14632:
14631:
14630:does not equal
14599:
14595:
14593:
14590:
14589:
14558:
14549:
14526:
14522:
14520:
14517:
14516:
14493:
14489:
14487:
14484:
14483:
14479:
14453:
14442:
14440:
14420:
14419:
14410:
14409:
14408:
14404:
14398:
14394:
14392:
14389:
14388:
14352:
14349:
14348:
14314:
14311:
14310:
14276:
14273:
14272:
14269:
14240:
14220:
14212:
14194:
14190:
14172:
14168:
14154:
14153:
14144:
14143:
14142:
14138:
14132:
14128:
14126:
14123:
14122:
14103:
14099:
14094:
14088:
14084:
14076:
14073:
14072:
14048:
14036:
14031:
14023:
14016:
14012:
14006:
14002:
13993:
13989:
13977:
13972:
13967:
13965:
13954:
13950:
13944:
13940:
13938:
13918:
13917:
13908:
13907:
13906:
13902:
13896:
13892:
13890:
13887:
13886:
13877:
13871:
13839:
13834:
13826:
13805:
13800:
13787:
13782:
13777:
13773:
13767:
13756:
13742:
13724:
13716:
13715:
13711:
13705:
13704:
13685:
13680:
13661:
13656:
13651:
13647:
13631:
13630:
13619:
13615:
13600:
13596:
13595:
13591:
13590:
13589:
13585:
13579:
13578:
13577:
13573:
13567:
13563:
13561:
13558:
13557:
13516:
13512:
13506:
13502:
13498:
13488:
13484:
13478:
13474:
13470:
13468:
13456:
13445:
13429:
13424:
13423:
13415:
13400:
13396:
13387:
13383:
13374:
13357:
13352:
13348:
13338:
13324:
13318:
13317:
13316:
13307:
13301:
13300:
13299:
13298:
13294:
13288:
13284:
13282:
13279:
13278:
13258:
13254:
13245:
13241:
13229:
13225:
13223:
13220:
13219:
13202:
13198:
13186:
13182:
13180:
13177:
13176:
13172:
13168:
13151:
13146:
13136:
13132:
13123:
13119:
13117:
13114:
13113:
13096:
13091:
13081:
13077:
13068:
13064:
13062:
13059:
13058:
13041:
13037:
13028:
13024:
13022:
13019:
13018:
12983:
12980:
12979:
12971:
12966:
12926:
12922:
12918:
12911:
12907:
12903:
12901:
12897:
12877:
12873:
12864:
12860:
12859:
12855:
12846:
12841:
12830:
12829:
12818:
12814:
12805:
12801:
12800:
12796:
12795:
12775:
12771:
12762:
12757:
12752:
12748:
12741:
12737:
12727:
12713:
12707:
12706:
12705:
12696:
12690:
12689:
12688:
12687:
12683:
12677:
12673:
12671:
12668:
12667:
12660:
12640:
12636:
12627:
12623:
12621:
12618:
12617:
12597:
12593:
12584:
12580:
12578:
12575:
12574:
12567:
12562:
12550:
12532:
12528:
12520:
12517:
12516:
12490:
12486:
12471:
12467:
12465:
12462:
12461:
12423:
12419:
12401:
12397:
12382:
12378:
12354:
12350:
12320:
12316:
12298:
12294:
12292:
12289:
12288:
12247:
12243:
12219:
12215:
12188:
12184:
12172:
12168:
12153:
12149:
12147:
12144:
12143:
12090:
12086:
12085:
12065:
12063:
12042:
12038:
12036:
12033:
12032:
11997:
11993:
11992:
11972:
11970:
11968:
11965:
11964:
11960:
11943:
11939:
11937:
11934:
11933:
11929:
11901:
11898:
11897:
11890:
11885:
11881:
11851:
11834:
11832:
11830:
11827:
11826:
11822:
11781:
11777:
11776:
11756:
11754:
11732:
11715:
11713:
11711:
11708:
11707:
11663:
11659:
11641:
11637:
11622:
11617:
11616:
11592:
11588:
11580:
11577:
11576:
11547:
11546:
11535:
11532:
11531:
11528:random variable
11523:
11519:
11515:
11514:indicates that
11493:
11490:
11489:
11467:
11464:
11463:
11437:
11436:
11425:
11422:
11421:
11395:
11394:
11383:
11380:
11379:
11372:
11368:
11351:
11350:
11348:
11345:
11344:
11328:
11325:
11324:
11308:
11305:
11304:
11300:
11285:
11279:
11277:
11276:
11253:
11250:
11249:
11232:
11217:
11210:
11206:
11205:
11180:
11175:
11173:
11170:
11169:
11153:
11150:
11149:
11118:
11115:
11114:
11098:
11095:
11094:
11069:
11066:
11065:
11043:
11040:
11039:
11023:
11020:
11019:
10997:
10994:
10993:
10962:
10959:
10958:
10936:
10933:
10932:
10910:
10907:
10906:
10884:
10881:
10880:
10864:
10861:
10860:
10835:
10832:
10831:
10806:
10803:
10802:
10789:
10788:
10776:
10775:
10753:
10748:
10742:
10731:
10718:
10717:
10684:
10680:
10672:
10671:
10664:
10638:
10633:
10627:
10616:
10603:
10602:
10590:
10569:
10555:
10553:
10546:
10542:
10541:
10508:
10504:
10496:
10495:
10488:
10462:
10457:
10451:
10440:
10429:
10409:
10405:
10395:
10388:
10384:
10353:
10351:
10343:
10342:
10335:
10309:
10304:
10303:
10299:
10293:
10282:
10274:
10272:
10269:
10268:
10243:
10240:
10239:
10226:
10225:
10210:
10206:
10196:
10189:
10185:
10154:
10152:
10144:
10143:
10136:
10110:
10105:
10099:
10088:
10075:
10074:
10064:
10042:
10038:
10037:
10035:
10029:
10018:
10007:
9986:
9982:
9978:
9976:
9973:
9972:
9950:
9947:
9946:
9930:
9927:
9926:
9910:
9907:
9906:
9874:
9870:
9868:
9865:
9864:
9850:
9849:
9834:
9830:
9820:
9813:
9809:
9778:
9776:
9768:
9767:
9760:
9732:
9728:
9712:
9691:
9687:
9684:
9683:
9669:
9618:
9614:
9613:
9606:
9602:
9571:
9569:
9561:
9560:
9553:
9525:
9521:
9505:
9484:
9480:
9477:
9476:
9460:
9453:
9449:
9418:
9416:
9408:
9407:
9400:
9389:
9372:
9369:
9368:
9319:
9312:
9308:
9277:
9275:
9267:
9266:
9259:
9248:
9231:
9227:
9225:
9222:
9221:
9207:
9206:
9196:
9179:
9176:
9175:
9155:
9108:
9106:
9102:
9055:
9054:
9047:
9034:
9033:
9002:
8955:
8953:
8949:
8942:
8938:
8897:
8896:
8889:
8878:
8861:
8857:
8855:
8852:
8851:
8835:
8832:
8831:
8784:
8780:
8778:
8775:
8774:
8718:
8714:
8697:
8694:
8693:
8671:
8668:
8667:
8642:
8639:
8638:
8612:
8608:
8598:
8591:
8587:
8556:
8554:
8546:
8545:
8538:
8512:
8507:
8501:
8490:
8462:
8458:
8456:
8453:
8452:
8430:
8427:
8426:
8394:
8390:
8388:
8385:
8384:
8354: for
8352:
8343:
8339:
8330:
8326:
8317:
8313:
8286:
8282:
8273:
8269:
8260:
8256:
8241:
8237:
8213:
8209:
8197:
8193:
8169:
8165:
8153:
8149:
8147:
8144:
8143:
8123:
8119:
8110:
8106:
8101:
8098:
8097:
8077:
8073:
8064:
8060:
8055:
8052:
8051:
8023:
8020:
8019:
8015:in the pair of
7980:
7976:
7974:
7971:
7970:
7945:
7941:
7932:
7928:
7919:
7915:
7903:
7899:
7890:
7886:
7877:
7873:
7849:
7845:
7843:
7840:
7839:
7822:
7818:
7809:
7805:
7803:
7800:
7799:
7770:
7766:
7748:
7744:
7715:
7712:
7711:
7682:
7678:
7660:
7656:
7627:
7624:
7623:
7606:
7602:
7593:
7589:
7587:
7584:
7583:
7569:Shannon entropy
7513:
7510:
7509:
7484:
7481:
7480:
7455:
7452:
7451:
7447:
7423:
7419:
7404:
7400:
7398:
7395:
7394:
7374:
7370:
7355:
7351:
7349:
7346:
7345:
7331:
7330:
7296:
7295:
7294:
7273:
7272:
7271:
7269:
7265:
7239:
7238:
7230:
7226:
7225:
7218:
7214:
7213:
7200:
7199:
7180:
7161:
7153:
7151:
7146:
7116:
7115:
7114:
7108:
7089:
7081:
7079:
7074:
7044:
7043:
7042:
7040:
7036:
7025:
7006:
6998:
6996:
6991:
6962:
6961:
6953:
6949:
6948:
6941:
6937:
6936:
6923:
6922:
6895:
6881:
6879:
6875:
6849:
6845:
6844:
6837:
6833:
6832:
6821:
6800:
6796:
6792:
6790:
6787:
6786:
6766:
6746:
6738:
6735:
6730:
6728:
6725:
6724:
6702:
6682:
6674:
6671:
6666:
6646:
6645:
6615:
6614:
6572:
6569:
6568:
6545:
6525:
6517:
6514:
6509:
6480:
6479:
6449:
6448:
6406:
6403:
6402:
6377:
6374:
6373:
6369:
6330:
6326:
6324:
6321:
6320:
6316:
6312:
6304:
6300:
6296:
6292:
6288:
6284:
6245:
6243:
6240:
6239:
6214:
6212:
6209:
6208:
6188:
6185:
6184:
6145:
6142:
6141:
6083:
6080:
6079:
6021:
6018:
6017:
5991:
5988:
5987:
5952:
5948:
5946:
5943:
5942:
5897:
5893:
5891:
5888:
5887:
5879:
5803:
5799:
5797:
5794:
5793:
5768:
5765:
5764:
5718:
5715:
5714:
5679:
5676:
5675:
5667:
5663:
5627:
5623:
5621:
5618:
5617:
5610:
5581:
5579:
5576:
5575:
5571:
5567:
5532:
5530:
5527:
5526:
5509:
5508:
5494:
5471:
5462:
5461:
5445:
5440:
5414:
5413:
5406:
5383:
5378:
5352:
5351:
5344:
5333:
5312:
5308:
5304:
5302:
5299:
5298:
5287:
5283:
5266:
5262:
5260:
5257:
5256:
5239:
5235:
5233:
5230:
5229:
5225:
5206:
5202:
5198:
5194:
5182:
5178:
5170:
5167:
5166:
5162:
5145:
5141:
5139:
5136:
5135:
5116:
5093:
5089:
5087:
5084:
5083:
5076:
5057:
4981:
4977:
4975:
4972:
4971:
4970:does not equal
4939:
4935:
4933:
4930:
4929:
4890:
4886:
4882:
4874:
4870:
4867:
4858:
4854:
4850:
4843:
4808:
4804:
4802:
4799:
4798:
4787:
4783:
4775:
4771:
4735:
4731:
4729:
4726:
4725:
4718:
4709:
4705:
4701:
4661:
4657:
4655:
4652:
4651:
4644:
4635:
4631:
4579:
4576:
4575:
4571:
4567:
4563:
4556:
4551:
4549:Interpretations
4532:
4531:
4456:
4452:
4442:
4433:
4432:
4417:
4413:
4404:
4400:
4398:
4394:
4378:
4360:
4356:
4347:
4343:
4341:
4337:
4321:
4303:
4299:
4290:
4286:
4284:
4280:
4264:
4255:
4254:
4234:
4220:
4218:
4214:
4188:
4187:
4180:
4169:
4148:
4144:
4140:
4138:
4135:
4134:
4117:
4116:
4041:
4037:
4027:
4018:
4017:
4002:
3998:
3989:
3985:
3983:
3979:
3963:
3945:
3941:
3932:
3928:
3926:
3922:
3906:
3888:
3884:
3875:
3871:
3869:
3865:
3849:
3840:
3839:
3819:
3805:
3803:
3799:
3773:
3772:
3765:
3754:
3733:
3729:
3725:
3723:
3720:
3719:
3704:
3699:
3662:
3658:
3656:
3653:
3652:
3620:
3616:
3614:
3611:
3610:
3584:
3582:
3579:
3578:
3556:
3554:
3551:
3550:
3528:
3526:
3523:
3522:
3496:
3493:
3492:
3467:
3465:
3462:
3461:
3439:
3437:
3434:
3433:
3411:
3409:
3406:
3405:
3379:
3376:
3375:
3358:
3326:
3315:
3312:
3311:
3270:
3269:
3267:
3264:
3263:
3259:
3255:
3251:
3233:
3230:
3229:
3225:
3203:
3200:
3199:
3177:
3174:
3173:
3165:
3161:
3157:
3154:
3145:
3141:
3133:
3131:with respect to
3127:
3092:
3088:
3083:
3080:
3079:
3075:
3064:
3056:, which starts
3045:
3039:
3032:
3026:
3022:
3018:
2980:
2976:
2971:
2968:
2967:
2956:
2895:
2891:
2881:
2870:
2867:
2866:
2850:
2847:
2846:
2811:
2782:
2762:
2760:
2756:
2727:
2726:
2719:
2715:
2691:
2687:
2685:
2682:
2681:
2674:
2670:
2648:
2645:
2644:
2640:
2636:
2607:
2571:
2560:
2557:
2556:
2527:
2491:
2480:
2477:
2476:
2472:
2468:
2451:
2450:
2448:
2445:
2444:
2428:
2425:
2424:
2420:
2416:
2377:
2346:
2339:
2323:
2316:
2314:
2310:
2283:
2276:
2260:
2253:
2251:
2243:
2242:
2235:
2231:
2207:
2203:
2201:
2198:
2197:
2186:
2182:
2153:
2117:
2106:
2103:
2102:
2082:
2081:
2076:
2073:
2072:
2068:
2064:
2060:
2056:
2020:
2013:
1994:
1984:
1982:
1977:
1974:
1973:
1938:
1907:
1900:
1884:
1877:
1875:
1871:
1854:
1853:
1846:
1842:
1818:
1814:
1812:
1809:
1808:
1801:
1797:
1793:
1785:
1762:
1761:
1756:
1753:
1752:
1741:
1737:
1730:
1726:
1718:
1714:
1682:
1659:
1645:
1643:
1639:
1612:
1604:
1576:
1572:
1570:
1567:
1566:
1555:
1551:
1499:
1495:
1488:
1482:
1479:
1478:
1444:
1441:
1440:
1417:
1416:
1411:
1408:
1407:
1370:
1367:
1366:
1341:
1338:
1337:
1316:
1313:
1312:
1311:
1270:
1267:
1266:
1229:
1226:
1225:
1221:
1214:
1210:
1206:
1162:
1142:
1140:
1136:
1107:
1106:
1099:
1068:
1064:
1062:
1059:
1058:
1015:
995:
993:
989:
960:
959:
952:
924:
920:
918:
915:
914:
907:
903:
880:
879:
874:
871:
870:
862:
858:
852:
828:Kullback (1959)
824:Harold Jeffreys
750:
747:
746:
729:
725:
723:
720:
719:
702:
698:
696:
693:
692:
661:
658:
657:
640:
636:
627:
623:
621:
618:
617:
600:
596:
587:
583:
581:
578:
577:
560:
556:
547:
543:
541:
538:
537:
520:
516:
514:
511:
510:
493:
489:
487:
484:
483:
466:
462:
460:
457:
456:
449:Richard Leibler
441:
424:
420:
416:
412:
408:
376:
372:
370:
367:
366:
362:
358:
354:
350:
346:
343:
323:fluid mechanics
270:
266:
262:
250:
246:
205:
185:
183:
179:
150:
149:
142:
114:
110:
108:
105:
104:
97:
93:
85:, is a type of
54:
50:
48:
45:
44:
17:
12:
11:
5:
26704:
26694:
26693:
26691:Thermodynamics
26688:
26683:
26678:
26664:
26663:
26658:
26644:
26639:
26634:
26629:
26622:
26621:External links
26619:
26617:
26616:
26607:
26582:
26576:
26551:
26549:
26548:
26507:
26505:Equation 2.25.
26477:
26457:
26436:
26408:
26337:
26266:
26251:
26231:
26218:(2): 111–119.
26198:
26195:on 2011-08-13.
26137:
26118:(3): 179–186.
26102:
26098:. The Academy.
26084:
26065:(2): 171–190.
26042:
26015:(4): 620–630.
25992:
25975:
25968:
25944:
25921:(3): 273–304.
25899:
25877:
25849:
25822:(2): 291–309.
25802:
25784:
25739:
25728:(2): 183–212.
25712:
25692:
25668:
25662:978-0677032405
25661:
25643:
25618:
25599:
25572:
25503:
25492:(4): 917–926.
25476:
25469:
25443:
25431:
25420:. p. 55.
25406:
25388:
25370:
25363:
25343:
25331:
25312:(4): 340–341.
25291:
25279:
25277:, p. 158.
25264:
25249:
25234:
25213:
25196:
25138:
25125:(1): 146–158.
25101:
25099:
25096:
25095:
25094:
25087:
25086:
25077:
25072:
25067:
25062:
25057:
25052:
25047:
25042:
25037:
25032:
25027:
25022:
25017:
25011:
25009:
25006:
24968:Main article:
24965:
24962:
24945:match distance
24922:
24921:
24909:
24896:The family of
24894:
24883:
24876:
24873:
24870:
24867:
24864:
24858:
24855:
24850:
24846:
24842:
24838:
24835:
24830:
24827:
24824:
24821:
24818:
24815:
24812:
24784:
24781:
24778:
24775:
24772:
24769:
24766:
24760:
24757:
24752:
24731:
24726:
24723:
24720:
24717:
24714:
24705:
24699:
24696:
24689:
24686:
24683:
24680:
24677:
24674:
24671:
24647:
24644:
24641:
24638:
24635:
24632:
24608:
24605:
24555:
24543:
24542:
24531:
24528:
24525:
24522:
24519:
24516:
24511:
24508:
24503:
24500:
24482:
24481:
24470:
24467:
24464:
24461:
24458:
24449:
24443:
24440:
24435:
24432:
24429:
24426:
24423:
24420:
24411:
24405:
24402:
24397:
24388:
24360:
24357:
24354:
24331:
24328:
24325:
24305:
24278:
24277:
24266:
24263:
24260:
24257:
24254:
24251:
24248:
24245:
24242:
24239:
24236:
24233:
24230:
24227:
24218:
24214:
24211:
24208:
24205:
24202:
24199:
24196:
24193:
24190:
24187:
24184:
24181:
24178:
24175:
24172:
24169:
24166:
24163:
24160:
24151:
24147:
24144:
24141:
24138:
24135:
24132:
24129:
24124:
24120:
24096:
24062:classification
24039:
24038:
24027:
24024:
24021:
24018:
24015:
24006:
24002:
23999:
23996:
23993:
23990:
23987:
23978:
23959:
23956:
23921:available work
23899:
23896:
23871:
23868:
23865:
23862:
23859:
23850:
23834:
23833:
23822:
23819:
23816:
23813:
23810:
23807:
23804:
23801:
23798:
23795:
23792:
23789:
23786:
23783:
23780:
23777:
23774:
23771:
23768:
23765:
23762:
23759:
23756:
23753:
23750:
23741:
23698:
23695:
23690:
23689:
23678:
23674:
23669:
23662:
23658:
23654:
23649:
23645:
23640:
23637:
23632:
23628:
23621:
23617:
23613:
23608:
23604:
23600:
23596:
23593:
23590:
23587:
23584:
23561:
23558:
23553:
23549:
23545:
23542:
23520:
23516:
23493:
23489:
23466:
23462:
23441:
23436:
23432:
23427:
23423:
23420:
23417:
23414:
23411:
23391:
23388:
23385:
23382:
23379:
23354:
23350:
23338:work available
23318:
23315:
23312:
23309:
23306:
23303:
23300:
23297:
23294:
23291:
23288:
23285:
23282:
23279:
23257:
23253:
23248:
23242:
23238:
23234:
23231:
23228:
23223:
23219:
23198:
23193:
23189:
23184:
23180:
23177:
23174:
23169:
23165:
23161:
23158:
23155:
23152:
23149:
23146:
23143:
23121:
23117:
23094:
23090:
23065:
23062:
23059:
23056:
23053:
23050:
23047:
23044:
23041:
23017:is energy and
23002:
22999:
22996:
22993:
22990:
22987:
22960:
22957:
22954:
22934:
22931:
22928:
22893:
22890:
22887:
22884:
22881:
22878:
22875:
22872:
22869:
22811:
22808:
22804:
22800:
22780:
22759:
22753:
22750:
22746:
22742:
22739:
22736:
22733:
22730:
22727:
22723:
22719:
22716:
22713:
22709:
22684:
22681:
22677:
22673:
22670:
22667:
22664:
22661:
22658:
22655:
22646:is defined as
22622:
22618:
22601:
22598:
22585:
22582:
22579:
22576:
22573:
22569:
22559:, rather than
22548:
22545:
22542:
22539:
22536:
22527:
22506:
22503:
22500:
22497:
22494:
22485:
22457:
22456:
22445:
22442:
22439:
22436:
22433:
22430:
22421:
22417:
22414:
22411:
22408:
22404:
22400:
22397:
22394:
22391:
22388:
22385:
22381:
22297:
22294:
22291:
22288:
22268:
22265:
22262:
22259:
22239:
22236:
22233:
22230:
22227:
22224:
22221:
22218:
22215:
22212:
22209:
22206:
22203:
22183:
22180:
22177:
22174:
22171:
22168:
22165:
22162:
22159:
22156:
22153:
22150:
22147:
22144:
22141:
22132:
22103:
22100:
22097:
22094:
22091:
22088:
22068:
22065:
22062:
22059:
22056:
22053:
22033:
22030:
22027:
22024:
22004:
22001:
21998:
21995:
21971:
21968:
21965:
21962:
21951:
21950:
21939:
21936:
21933:
21930:
21927:
21924:
21921:
21918:
21915:
21912:
21909:
21906:
21897:
21893:
21889:
21885:
21882:
21879:
21876:
21873:
21870:
21867:
21864:
21861:
21858:
21855:
21852:
21849:
21846:
21843:
21834:
21829:
21825:
21820:
21817:
21814:
21811:
21807:
21803:
21800:
21797:
21794:
21791:
21788:
21785:
21782:
21779:
21776:
21773:
21770:
21767:
21764:
21761:
21758:
21755:
21752:
21749:
21746:
21737:
21713:
21710:
21707:
21704:
21701:
21698:
21695:
21692:
21689:
21686:
21658:
21655:
21652:
21649:
21617:
21614:
21611:
21608:
21605:
21602:
21579:
21574:
21570:
21566:
21563:
21560:
21551:
21528:
21524:
21495:
21492:
21459:
21458:
21447:
21444:
21441:
21438:
21435:
21432:
21429:
21426:
21423:
21420:
21417:
21414:
21411:
21408:
21405:
21402:
21393:
21389:
21386:
21383:
21380:
21377:
21374:
21369:
21365:
21361:
21358:
21355:
21352:
21349:
21346:
21341:
21337:
21333:
21330:
21327:
21324:
21321:
21312:
21288:
21285:
21282:
21279:
21253:
21249:
21226:
21222:
21196:
21192:
21169:
21165:
21138:
21134:
21111:
21107:
21084:
21080:
21057:
21053:
21030:
21026:
20999:
20994:
20989:
20985:
20981:
20978:
20975:
20972:
20969:
20966:
20961:
20957:
20953:
20950:
20947:
20944:
20939:
20928:
20915:
20912:
20898:
20895:
20882:
20879:
20876:
20873:
20870:
20865:
20861:
20857:
20852:
20848:
20844:
20841:
20826:
20825:
20812:
20807:
20804:
20801:
20798:
20795:
20792:
20789:
20786:
20783:
20780:
20775:
20771:
20767:
20764:
20761:
20758:
20753:
20742:
20738:
20733:
20728:
20725:
20722:
20717:
20713:
20709:
20706:
20703:
20700:
20697:
20694:
20691:
20688:
20683:
20679:
20675:
20670:
20666:
20662:
20659:
20656:
20653:
20648:
20637:
20614:
20609:
20606:
20603:
20600:
20597:
20594:
20591:
20588:
20585:
20582:
20577:
20573:
20569:
20564:
20560:
20556:
20553:
20550:
20547:
20542:
20531:
20512:
20511:
20498:
20492:
20489:
20486:
20483:
20480:
20477:
20472:
20469:
20466:
20461:
20457:
20453:
20450:
20447:
20444:
20438:
20434:
20431:
20428:
20425:
20422:
20417:
20413:
20409:
20406:
20403:
20400:
20395:
20391:
20368:
20362:
20359:
20356:
20353:
20350:
20347:
20342:
20339:
20336:
20331:
20327:
20323:
20318:
20314:
20310:
20307:
20304:
20301:
20295:
20291:
20288:
20285:
20282:
20279:
20274:
20270:
20266:
20261:
20257:
20253:
20250:
20247:
20244:
20239:
20235:
20211:
20208:
20205:
20202:
20199:
20196:
20176:
20173:
20170:
20165:
20161:
20157:
20154:
20151:
20148:
20128:
20125:
20122:
20117:
20113:
20109:
20104:
20100:
20096:
20093:
20090:
20087:
20061:
20057:
20053:
20048:
20044:
20020:
20017:
20014:
19999:
19998:
19986:
19980:
19977:
19974:
19971:
19968:
19965:
19960:
19957:
19954:
19951:
19948:
19945:
19942:
19939:
19933:
19929:
19926:
19923:
19920:
19917:
19914:
19911:
19908:
19905:
19902:
19897:
19893:
19889:
19884:
19879:
19876:
19873:
19870:
19867:
19864:
19861:
19858:
19855:
19852:
19849:
19846:
19843:
19840:
19837:
19832:
19821:
19797:
19794:
19791:
19788:
19785:
19782:
19779:
19776:
19756:
19753:
19750:
19747:
19744:
19741:
19721:
19718:
19715:
19712:
19709:
19706:
19703:
19700:
19696:
19684:
19683:
19672:
19669:
19666:
19663:
19660:
19657:
19654:
19651:
19648:
19645:
19642:
19639:
19636:
19633:
19630:
19627:
19624:
19621:
19618:
19613:
19609:
19605:
19602:
19597:
19592:
19589:
19586:
19583:
19580:
19577:
19574:
19571:
19566:
19560:
19541:
19540:
19526:
19523:
19520:
19517:
19514:
19511:
19506:
19503:
19500:
19497:
19494:
19491:
19488:
19485:
19482:
19479:
19476:
19473:
19470:
19467:
19461:
19458:
19455:
19452:
19449:
19446:
19443:
19440:
19437:
19423:Bayes' theorem
19410:
19407:
19404:
19401:
19398:
19395:
19392:
19389:
19369:
19366:
19363:
19360:
19357:
19354:
19330:
19327:
19324:
19304:
19301:
19298:
19295:
19292:
19289:
19286:
19283:
19280:
19277:
19274:
19250:
19247:
19226:
19223:
19220:
19216:
19197:
19196:
19185:
19182:
19179:
19176:
19173:
19170:
19161:
19157:
19154:
19151:
19148:
19144:
19140:
19137:
19134:
19131:
19128:
19125:
19122:
19119:
19116:
19113:
19108:
19104:
19100:
19097:
19094:
19091:
19088:
19085:
19081:
19064:over the same
18994:
18991:
18988:
18984:
18960:Huffman coding
18935:
18932:
18911:
18908:
18904:
18900:
18897:
18894:
18874:
18871:
18868:
18863:
18859:
18830:
18827:
18824:
18821:
18818:
18815:
18795:
18792:
18789:
18786:
18783:
18780:
18777:
18772:
18768:
18744:
18743:
18727:
18722:
18718:
18715:
18712:
18707:
18703:
18699:
18695:
18691:
18688:
18685:
18681:
18677:
18673:
18663:
18658:
18654:
18649:
18645:
18641:
18638:
18635:
18632:
18629:
18626:
18623:
18620:
18618:
18616:
18613:
18610:
18607:
18604:
18601:
18598:
18595:
18592:
18589:
18586:
18583:
18579:
18575:
18572:
18570:
18568:
18565:
18562:
18559:
18556:
18551:
18547:
18543:
18540:
18537:
18534:
18531:
18528:
18519:
18515:
18512:
18509:
18506:
18503:
18500:
18497:
18494:
18491:
18488:
18485:
18482:
18479:
18476:
18473:
18470:
18467:
18464:
18455:
18451:
18448:
18445:
18442:
18439:
18436:
18433:
18430:
18428:
18426:
18423:
18420:
18417:
18414:
18411:
18408:
18405:
18402:
18397:
18393:
18389:
18386:
18383:
18380:
18377:
18374:
18371:
18368:
18359:
18355:
18352:
18349:
18346:
18343:
18340:
18337:
18334:
18332:
18330:
18327:
18324:
18321:
18318:
18314:
18310:
18309:
18290:
18287:
18286:
18285:
18274:
18271:
18268:
18265:
18262:
18258:
18253:
18249:
18246:
18243:
18240:
18237:
18228:
18224:
18221:
18218:
18215:
18212:
18209:
18195:
18194:
18183:
18180:
18177:
18170:
18167:
18164:
18161:
18156:
18153:
18150:
18147:
18141:
18138:
18135:
18132:
18129:
18126:
18123:
18120:
18117:
18114:
18111:
18108:
18105:
18102:
18099:
18096:
18093:
18088:
18084:
18078:
18075:
18072:
18068:
18032:
18029:
18026:
18023:
18003:
18000:
17997:
17992:
17988:
17959:
17956:
17953:
17950:
17930:
17927:
17924:
17919:
17915:
17883:
17882:
17866:
17862:
17859:
17856:
17851:
17847:
17843:
17840:
17837:
17834:
17829:
17825:
17820:
17810:
17806:
17803:
17800:
17797:
17794:
17791:
17788:
17785:
17783:
17781:
17777:
17773:
17770:
17767:
17764:
17759:
17755:
17750:
17746:
17743:
17740:
17737:
17735:
17733:
17730:
17727:
17723:
17719:
17718:
17699:
17696:
17672:
17669:
17666:
17663:
17660:
17657:
17625:
17622:
17619:
17616:
17613:
17610:
17607:
17604:
17584:
17581:
17578:
17575:
17572:
17569:
17555:
17554:
17539:
17536:
17533:
17530:
17527:
17524:
17521:
17518:
17515:
17512:
17509:
17506:
17503:
17500:
17491:
17487:
17484:
17479:
17475:
17471:
17468:
17466:
17464:
17461:
17458:
17455:
17452:
17449:
17446:
17443:
17440:
17437:
17434:
17431:
17428:
17425:
17422:
17413:
17409:
17406:
17401:
17397:
17393:
17390:
17388:
17386:
17383:
17380:
17377:
17374:
17371:
17368:
17365:
17362:
17359:
17356:
17353:
17350:
17347:
17344:
17341:
17338:
17335:
17326:
17322:
17319:
17317:
17315:
17312:
17309:
17306:
17303:
17300:
17297:
17294:
17293:
17274:
17271:
17258:
17255:
17252:
17232:
17229:
17226:
17223:
17199:
17196:
17193:
17169:
17166:
17163:
17160:
17149:
17148:
17137:
17133:
17129:
17124:
17120:
17116:
17113:
17104:
17099:
17089:
17085:
17082:
17079:
17076:
17073:
17069:
17027:Main article:
17024:
17021:
17016:
17013:
17007:of a discrete
16992:
16989:
16988:
16987:
16976:
16973:
16970:
16965: as
16961:
16956:
16952:
16948:
16942:
16937:
16932:
16926:
16922:
16918:
16912:
16909:
16906:
16903:
16900:
16897:
16894:
16891:
16888:
16885:
16882:
16879:
16876:
16873:
16870:
16867:
16862:
16846:
16845:
16834:
16831:
16828:
16825:
16819:
16816:
16813:
16808:
16799:
16795:
16791:
16786:
16783:
16780:
16777:
16774:
16771:
16766:
16762:
16753:
16748:
16745:
16742:
16738:
16733:
16729:
16726:
16723:
16717:
16714:
16711:
16706:
16699:
16696:
16691:
16688:
16685:
16682:
16679:
16676:
16673:
16665:
16660:
16657:
16654:
16650:
16645:
16623:
16622:
16607:
16604:
16601:
16598:
16595:
16592:
16588:
16585:
16582:
16579:
16576:
16573:
16570:
16567:
16564:
16561:
16556:
16551:
16548:
16545:
16541:
16537:
16534:
16531:
16528:
16525:
16522:
16519:
16517:
16514:
16506:
16502:
16498:
16493:
16490:
16487:
16484:
16481:
16476:
16472:
16465:
16461:
16460:
16457:
16454:
16451:
16448:
16444:
16441:
16438:
16435:
16430:
16425:
16422:
16419:
16415:
16411:
16408:
16405:
16402:
16399:
16396:
16393:
16391:
16388:
16380:
16376:
16372:
16367:
16362:
16358:
16351:
16347:
16346:
16343:
16340:
16337:
16334:
16331:
16327:
16324:
16321:
16318:
16313:
16308:
16305:
16302:
16298:
16294:
16291:
16288:
16285:
16282:
16279:
16276:
16274:
16271:
16265:
16262:
16257:
16254:
16248:
16244:
16243:
16233:
16216:
16212:
16208:
16203:
16200:
16197:
16194:
16191:
16186:
16182:
16175:
16167:
16163:
16159:
16154:
16151:
16148:
16145:
16142:
16137:
16133:
16126:
16120:
16117:
16112:
16109:
16106:
16103:
16100:
16097:
16069:
16066:
16063:
16060:
16057:
16053:
16040:
16037:
16016:
16013:
16010:
16005:
16002:
15998:
15975:
15971:
15955:
15954:
15943:
15940:
15935:
15931:
15927:
15924:
15921:
15918:
15915:
15912:
15909:
15906:
15897:
15889:
15885:
15881:
15878:
15873:
15865:
15861:
15857:
15851:
15847:
15843:
15837:
15833:
15827:
15823:
15820:
15815:
15811:
15807:
15802:
15799:
15795:
15780:Hessian matrix
15776:
15775:
15764:
15761:
15758:
15753:
15749:
15745:
15740:
15737:
15733:
15726:
15722:
15718:
15712:
15708:
15704:
15698:
15695:
15690:
15687:
15684:
15679:
15675:
15671:
15668:
15665:
15662:
15659:
15656:
15653:
15650:
15641:
15622:
15621:
15610:
15607:
15604:
15601:
15598:
15593:
15589:
15585:
15582:
15579:
15576:
15573:
15570:
15567:
15564:
15555:
15547:
15543:
15539:
15536:
15531:
15523:
15519:
15515:
15511:
15506:
15480:
15476:
15472:
15446:
15442:
15438:
15435:
15415:
15412:
15409:
15389:
15384:
15380:
15376:
15368:
15364:
15360:
15355:
15352:
15346:
15342:
15337:
15333:
15329:
15323:
15319:
15294:
15272:
15268:
15262:
15258:
15254:
15251:
15248:
15245:
15240:
15236:
15232:
15221:
15220:
15209:
15206:
15203:
15198:
15194:
15190:
15185:
15181:
15174:
15170:
15166:
15163:
15160:
15155:
15151:
15147:
15144:
15141:
15138:
15135:
15132:
15129:
15100:
15096:
15075:
15055:
15050:
15046:
15042:
15039:
15036:
15033:
15013:
15010:
15007:
15004:
15001:
14998:
14978:
14954:
14951:
14943:
14942:
14930:
14925:
14922:
14918:
14912:
14908:
14904:
14901:
14896:
14892:
14886:
14882:
14864:
14863:
14851:
14845:
14841:
14834:
14830:
14815:
14814:
14802:
14799:
14796:
14793:
14790:
14785:
14781:
14777:
14768:
14762:
14759:
14756:
14752:
14728:
14725:
14722:
14717:
14713:
14709:
14704:
14700:
14696:
14661:
14658:
14655:
14652:
14649:
14640:
14619:
14616:
14613:
14610:
14607:
14598:
14564:, it is not a
14557:
14554:
14537:
14534:
14529:
14525:
14504:
14501:
14496:
14492:
14462:
14459:
14456:
14451:
14448:
14445:
14439:
14436:
14433:
14429:
14423:
14418:
14413:
14407:
14397:
14374:
14371:
14368:
14365:
14362:
14359:
14356:
14336:
14333:
14330:
14327:
14324:
14321:
14318:
14298:
14295:
14292:
14289:
14286:
14283:
14280:
14268:
14265:
14252:
14249:
14246:
14243:
14239:
14236:
14233:
14230:
14227:
14223:
14219:
14215:
14211:
14208:
14205:
14200:
14197:
14193:
14189:
14186:
14183:
14180:
14175:
14171:
14167:
14163:
14157:
14152:
14147:
14141:
14131:
14106:
14102:
14097:
14091:
14087:
14083:
14080:
14069:
14068:
14055:
14052:
14047:
14039:
14034:
14030:
14026:
14019:
14015:
14009:
14005:
14001:
13996:
13992:
13988:
13985:
13980:
13975:
13971:
13964:
13957:
13953:
13947:
13943:
13937:
13934:
13931:
13927:
13921:
13916:
13911:
13905:
13895:
13868:
13867:
13856:
13852:
13847:
13842:
13837:
13833:
13829:
13825:
13822:
13819:
13816:
13813:
13808:
13803:
13799:
13795:
13790:
13785:
13781:
13776:
13770:
13765:
13762:
13759:
13755:
13749:
13746:
13741:
13737:
13732:
13727:
13723:
13719:
13714:
13708:
13703:
13699:
13694:
13688:
13683:
13679:
13675:
13672:
13669:
13664:
13659:
13655:
13650:
13646:
13643:
13640:
13634:
13628:
13622:
13618:
13614:
13611:
13608:
13603:
13599:
13594:
13588:
13582:
13576:
13566:
13547:
13546:
13535:
13531:
13522:
13519:
13515:
13509:
13505:
13501:
13494:
13491:
13487:
13481:
13477:
13473:
13467:
13464:
13459:
13454:
13451:
13448:
13444:
13440:
13437:
13432:
13427:
13422:
13418:
13414:
13411:
13408:
13403:
13399:
13393:
13390:
13386:
13382:
13377:
13372:
13369:
13366:
13363:
13360:
13356:
13351:
13345:
13342:
13337:
13333:
13327:
13321:
13315:
13310:
13304:
13297:
13287:
13261:
13257:
13253:
13248:
13244:
13240:
13237:
13232:
13228:
13205:
13201:
13197:
13194:
13189:
13185:
13154:
13149:
13145:
13139:
13135:
13131:
13126:
13122:
13099:
13094:
13090:
13084:
13080:
13076:
13071:
13067:
13044:
13040:
13036:
13031:
13027:
12999:
12996:
12993:
12990:
12987:
12958:
12957:
12946:
12942:
12937:
12929:
12925:
12921:
12914:
12910:
12906:
12900:
12896:
12893:
12890:
12886:
12880:
12876:
12872:
12867:
12863:
12858:
12852:
12849:
12844:
12840:
12833:
12827:
12821:
12817:
12813:
12808:
12804:
12799:
12794:
12791:
12788:
12784:
12778:
12774:
12768:
12765:
12760:
12756:
12751:
12747:
12744:
12740:
12734:
12731:
12726:
12722:
12716:
12710:
12704:
12699:
12693:
12686:
12676:
12648:
12643:
12639:
12635:
12630:
12626:
12600:
12596:
12592:
12587:
12583:
12566:
12563:
12561:
12558:
12554:measure theory
12535:
12531:
12527:
12524:
12504:
12501:
12498:
12493:
12489:
12485:
12482:
12479:
12470:
12458:
12457:
12446:
12443:
12440:
12437:
12434:
12431:
12426:
12422:
12418:
12415:
12412:
12409:
12404:
12400:
12396:
12393:
12390:
12381:
12377:
12374:
12371:
12368:
12365:
12362:
12357:
12353:
12349:
12346:
12343:
12340:
12337:
12334:
12331:
12328:
12319:
12315:
12312:
12309:
12306:
12301:
12297:
12282:
12281:
12270:
12267:
12264:
12261:
12258:
12255:
12250:
12246:
12242:
12239:
12236:
12233:
12230:
12227:
12222:
12218:
12214:
12211:
12208:
12205:
12202:
12199:
12196:
12187:
12183:
12180:
12175:
12171:
12167:
12164:
12161:
12152:
12128:
12125:
12122:
12119:
12116:
12110:
12107:
12104:
12101:
12098:
12093:
12089:
12083:
12080:
12077:
12074:
12071:
12068:
12062:
12059:
12056:
12053:
12050:
12045:
12041:
12017:
12014:
12011:
12008:
12005:
12000:
11996:
11990:
11987:
11984:
11981:
11978:
11975:
11946:
11942:
11917:
11914:
11911:
11908:
11905:
11891:
11866:
11863:
11860:
11857:
11854:
11849:
11846:
11843:
11840:
11837:
11819:
11818:
11807:
11801:
11798:
11795:
11792:
11789:
11784:
11780:
11774:
11771:
11768:
11765:
11762:
11759:
11753:
11747:
11744:
11741:
11738:
11735:
11730:
11727:
11724:
11721:
11718:
11701:
11700:
11689:
11686:
11683:
11680:
11677:
11674:
11671:
11662:
11658:
11655:
11652:
11649:
11644:
11640:
11636:
11631:
11628:
11625:
11615:
11612:
11609:
11606:
11603:
11600:
11595:
11591:
11587:
11584:
11561:
11558:
11555:
11550:
11545:
11542:
11539:
11503:
11500:
11497:
11477:
11474:
11471:
11451:
11448:
11445:
11440:
11435:
11432:
11429:
11409:
11406:
11403:
11398:
11393:
11390:
11387:
11354:
11332:
11312:
11293:
11284:
11281:
11263:
11260:
11257:
11235:
11230:
11224:
11221:
11216:
11213:
11209:
11204:
11198:
11195:
11192:
11189:
11186:
11183:
11179:
11157:
11137:
11134:
11131:
11128:
11125:
11122:
11102:
11082:
11079:
11076:
11073:
11053:
11050:
11047:
11027:
11007:
11004:
11001:
10981:
10978:
10975:
10972:
10969:
10966:
10946:
10943:
10940:
10920:
10917:
10914:
10894:
10891:
10888:
10868:
10848:
10845:
10842:
10839:
10819:
10816:
10813:
10810:
10787:
10784:
10781:
10779:
10777:
10771:
10768:
10765:
10762:
10759:
10756:
10752:
10745:
10740:
10737:
10734:
10730:
10726:
10723:
10721:
10719:
10715:
10711:
10708:
10705:
10702:
10699:
10696:
10693:
10690:
10687:
10683:
10675:
10670:
10667:
10663:
10656:
10653:
10650:
10647:
10644:
10641:
10637:
10630:
10625:
10622:
10619:
10615:
10611:
10608:
10606:
10604:
10599:
10596:
10593:
10588:
10581:
10578:
10575:
10572:
10567:
10564:
10561:
10558:
10552:
10549:
10545:
10539:
10535:
10532:
10529:
10526:
10523:
10520:
10517:
10514:
10511:
10507:
10499:
10494:
10491:
10487:
10480:
10477:
10474:
10471:
10468:
10465:
10461:
10454:
10449:
10446:
10443:
10439:
10435:
10432:
10430:
10427:
10418:
10415:
10412:
10408:
10404:
10401:
10398:
10391:
10387:
10383:
10380:
10377:
10374:
10371:
10368:
10365:
10362:
10359:
10356:
10346:
10341:
10338:
10334:
10327:
10324:
10321:
10318:
10315:
10312:
10308:
10302:
10296:
10291:
10288:
10285:
10281:
10277:
10276:
10256:
10253:
10250:
10247:
10219:
10216:
10213:
10209:
10205:
10202:
10199:
10192:
10188:
10184:
10181:
10178:
10175:
10172:
10169:
10166:
10163:
10160:
10157:
10147:
10142:
10139:
10135:
10128:
10125:
10122:
10119:
10116:
10113:
10109:
10102:
10097:
10094:
10091:
10087:
10083:
10080:
10078:
10076:
10070:
10067:
10062:
10059:
10056:
10051:
10048:
10045:
10041:
10032:
10027:
10024:
10021:
10017:
10013:
10010:
10008:
10006:
10003:
10000:
9997:
9994:
9985:
9981:
9980:
9960:
9957:
9954:
9934:
9914:
9894:
9891:
9888:
9885:
9882:
9873:
9843:
9840:
9837:
9833:
9829:
9826:
9823:
9816:
9812:
9808:
9805:
9802:
9799:
9796:
9793:
9790:
9787:
9784:
9781:
9771:
9766:
9763:
9759:
9755:
9752:
9749:
9746:
9743:
9740:
9735:
9731:
9727:
9724:
9721:
9718:
9715:
9713:
9711:
9708:
9705:
9700:
9697:
9694:
9690:
9686:
9685:
9678:
9675:
9672:
9667:
9663:
9660:
9657:
9654:
9651:
9648:
9645:
9642:
9639:
9636:
9633:
9630:
9627:
9624:
9621:
9617:
9609:
9605:
9601:
9598:
9595:
9592:
9589:
9586:
9583:
9580:
9577:
9574:
9564:
9559:
9556:
9552:
9548:
9545:
9542:
9539:
9536:
9533:
9528:
9524:
9520:
9517:
9514:
9511:
9508:
9506:
9504:
9501:
9498:
9493:
9490:
9487:
9483:
9479:
9478:
9472:
9469:
9466:
9463:
9456:
9452:
9448:
9445:
9442:
9439:
9436:
9433:
9430:
9427:
9424:
9421:
9411:
9406:
9403:
9399:
9395:
9392:
9390:
9388:
9385:
9382:
9378:
9375:
9371:
9370:
9364:
9361:
9358:
9355:
9352:
9349:
9346:
9343:
9340:
9337:
9334:
9331:
9328:
9325:
9322:
9315:
9311:
9307:
9304:
9301:
9298:
9295:
9292:
9289:
9286:
9283:
9280:
9270:
9265:
9262:
9258:
9254:
9251:
9249:
9247:
9244:
9241:
9237:
9234:
9230:
9229:
9205:
9202:
9199:
9197:
9195:
9192:
9189:
9185:
9182:
9178:
9177:
9173:
9167:
9164:
9161:
9158:
9153:
9150:
9147:
9144:
9141:
9138:
9135:
9132:
9129:
9126:
9123:
9120:
9117:
9114:
9111:
9105:
9101:
9098:
9095:
9092:
9089:
9086:
9083:
9080:
9077:
9074:
9071:
9068:
9065:
9058:
9053:
9050:
9046:
9042:
9039:
9037:
9035:
9031:
9027:
9024:
9020:
9014:
9011:
9008:
9005:
9000:
8997:
8994:
8991:
8988:
8985:
8982:
8979:
8976:
8973:
8970:
8967:
8964:
8961:
8958:
8952:
8948:
8945:
8941:
8937:
8934:
8931:
8928:
8925:
8922:
8919:
8916:
8913:
8910:
8907:
8900:
8895:
8892:
8888:
8884:
8881:
8879:
8877:
8874:
8871:
8867:
8864:
8860:
8859:
8839:
8819:
8816:
8813:
8810:
8807:
8804:
8801:
8798:
8795:
8792:
8783:
8773:and note that
8762:
8759:
8756:
8753:
8750:
8747:
8744:
8741:
8738:
8735:
8732:
8729:
8726:
8717:
8713:
8710:
8707:
8704:
8701:
8690:
8689:
8688:
8687:
8675:
8655:
8652:
8649:
8646:
8621:
8618:
8615:
8611:
8607:
8604:
8601:
8594:
8590:
8586:
8583:
8580:
8577:
8574:
8571:
8568:
8565:
8562:
8559:
8549:
8544:
8541:
8537:
8530:
8527:
8524:
8521:
8518:
8515:
8511:
8504:
8499:
8496:
8493:
8489:
8485:
8482:
8479:
8476:
8473:
8470:
8461:
8440:
8437:
8434:
8414:
8411:
8408:
8405:
8402:
8393:
8382:
8371:
8368:
8365:
8362:
8359:
8351:
8346:
8342:
8338:
8333:
8329:
8325:
8316:
8312:
8309:
8306:
8303:
8300:
8297:
8294:
8289:
8285:
8281:
8276:
8272:
8268:
8259:
8255:
8252:
8249:
8244:
8240:
8236:
8233:
8230:
8227:
8224:
8221:
8216:
8212:
8208:
8205:
8200:
8196:
8192:
8189:
8186:
8183:
8180:
8177:
8172:
8168:
8164:
8161:
8152:
8131:
8126:
8122:
8118:
8113:
8109:
8105:
8085:
8080:
8076:
8072:
8067:
8063:
8059:
8039:
8036:
8033:
8030:
8027:
8000:
7997:
7994:
7991:
7988:
7979:
7967:
7956:
7953:
7948:
7944:
7940:
7935:
7931:
7927:
7918:
7914:
7911:
7906:
7902:
7898:
7893:
7889:
7885:
7876:
7872:
7869:
7866:
7863:
7860:
7857:
7848:
7825:
7821:
7817:
7812:
7808:
7787:
7784:
7781:
7778:
7773:
7769:
7765:
7762:
7759:
7756:
7751:
7747:
7743:
7740:
7737:
7734:
7731:
7728:
7725:
7722:
7719:
7699:
7696:
7693:
7690:
7685:
7681:
7677:
7674:
7671:
7668:
7663:
7659:
7655:
7652:
7649:
7646:
7643:
7640:
7637:
7634:
7631:
7609:
7605:
7601:
7596:
7592:
7572:
7551:
7548:
7544:
7541:
7538:
7535:
7532:
7529:
7526:
7523:
7520:
7517:
7497:
7494:
7491:
7488:
7468:
7465:
7462:
7459:
7431:
7426:
7422:
7418:
7415:
7412:
7407:
7403:
7382:
7377:
7373:
7369:
7366:
7363:
7358:
7354:
7329:
7326:
7321:
7315:
7312:
7309:
7303:
7300:
7292:
7289:
7286:
7280:
7277:
7268:
7264:
7261:
7258:
7255:
7252:
7246:
7243:
7233:
7229:
7221:
7217:
7212:
7208:
7205:
7203:
7201:
7198:
7195:
7190:
7183:
7179:
7176:
7173:
7167:
7164:
7159:
7156:
7149:
7144:
7141:
7138:
7135:
7132:
7129:
7123:
7120:
7111:
7107:
7104:
7101:
7095:
7092:
7087:
7084:
7077:
7072:
7069:
7066:
7063:
7060:
7057:
7051:
7048:
7039:
7035:
7032:
7028:
7024:
7021:
7018:
7012:
7009:
7004:
7001:
6994:
6990:
6987:
6984:
6981:
6978:
6975:
6969:
6966:
6956:
6952:
6944:
6940:
6935:
6931:
6928:
6926:
6924:
6921:
6918:
6913:
6907:
6904:
6901:
6898:
6893:
6890:
6887:
6884:
6878:
6874:
6871:
6868:
6865:
6862:
6859:
6852:
6848:
6840:
6836:
6831:
6827:
6824:
6822:
6820:
6817:
6814:
6811:
6808:
6799:
6795:
6794:
6769:
6765:
6762:
6759:
6752:
6749:
6744:
6741:
6733:
6712:
6709:
6705:
6701:
6698:
6695:
6688:
6685:
6680:
6677:
6669:
6665:
6662:
6659:
6653:
6650:
6644:
6641:
6638:
6634:
6631:
6628:
6622:
6619:
6613:
6610:
6607:
6603:
6600:
6597:
6594:
6591:
6588:
6585:
6582:
6579:
6576:
6556:
6553:
6548:
6544:
6541:
6538:
6531:
6528:
6523:
6520:
6512:
6508:
6505:
6502:
6499:
6496:
6493:
6487:
6484:
6478:
6475:
6472:
6468:
6465:
6462:
6456:
6453:
6447:
6444:
6441:
6437:
6434:
6431:
6428:
6425:
6422:
6419:
6416:
6413:
6410:
6401:, then, since
6390:
6387:
6384:
6381:
6362:
6350:
6347:
6344:
6341:
6338:
6329:
6264:
6261:
6258:
6255:
6252:
6248:
6227:
6224:
6221:
6217:
6207:. The entropy
6192:
6173:
6170:
6167:
6164:
6161:
6158:
6155:
6152:
6149:
6129:
6126:
6123:
6120:
6117:
6114:
6111:
6108:
6105:
6102:
6099:
6096:
6093:
6090:
6087:
6067:
6064:
6061:
6058:
6055:
6052:
6049:
6046:
6043:
6040:
6037:
6034:
6031:
6028:
6025:
6014:
6013:
6001:
5998:
5995:
5985:if and only if
5972:
5969:
5966:
5963:
5960:
5951:
5926:
5923:
5920:
5917:
5914:
5911:
5908:
5905:
5896:
5878:
5875:
5823:
5820:
5817:
5814:
5811:
5802:
5781:
5778:
5775:
5772:
5752:
5749:
5746:
5743:
5740:
5737:
5734:
5731:
5728:
5725:
5722:
5698:
5695:
5692:
5689:
5686:
5683:
5647:
5644:
5641:
5638:
5635:
5626:
5594:
5591:
5588:
5584:
5551:
5548:
5545:
5542:
5539:
5535:
5523:
5522:
5507:
5504:
5501:
5497:
5493:
5490:
5487:
5484:
5481:
5478:
5474:
5470:
5467:
5465:
5463:
5457:
5454:
5451:
5448:
5444:
5439:
5436:
5433:
5430:
5427:
5424:
5417:
5412:
5409:
5405:
5401:
5395:
5392:
5389:
5386:
5382:
5377:
5374:
5371:
5368:
5365:
5362:
5355:
5350:
5347:
5343:
5339:
5336:
5334:
5332:
5329:
5326:
5323:
5320:
5311:
5307:
5306:
5269:
5265:
5242:
5238:
5209:
5205:
5201:
5197:
5193:
5190:
5185:
5181:
5177:
5174:
5148:
5144:
5115:
5112:
5092:
5075:
5072:
5001:
4998:
4995:
4992:
4989:
4980:
4959:
4956:
4953:
4950:
4947:
4938:
4904:, it is not a
4900:While it is a
4866:
4863:
4828:
4825:
4822:
4819:
4816:
4807:
4755:
4752:
4749:
4746:
4743:
4734:
4717:
4714:
4681:
4678:
4675:
4672:
4669:
4660:
4643:
4640:
4619:
4616:
4613:
4610:
4607:
4604:
4601:
4598:
4595:
4592:
4589:
4586:
4583:
4555:
4552:
4550:
4547:
4546:
4545:
4530:
4527:
4523:
4519:
4516:
4513:
4510:
4507:
4504:
4501:
4498:
4495:
4492:
4489:
4486:
4483:
4480:
4477:
4474:
4471:
4468:
4465:
4462:
4459:
4455:
4449:
4446:
4441:
4438:
4436:
4434:
4430:
4424:
4420:
4416:
4411:
4407:
4403:
4397:
4393:
4390:
4385:
4382:
4377:
4373:
4367:
4363:
4359:
4354:
4350:
4346:
4340:
4336:
4333:
4328:
4325:
4320:
4316:
4310:
4306:
4302:
4297:
4293:
4289:
4283:
4279:
4276:
4271:
4268:
4263:
4260:
4258:
4256:
4252:
4246:
4243:
4240:
4237:
4232:
4229:
4226:
4223:
4217:
4213:
4210:
4207:
4204:
4201:
4198:
4191:
4186:
4183:
4179:
4175:
4172:
4170:
4168:
4165:
4162:
4159:
4156:
4147:
4143:
4142:
4131:
4130:
4115:
4112:
4109:
4105:
4101:
4098:
4095:
4092:
4089:
4086:
4083:
4080:
4077:
4074:
4071:
4068:
4065:
4062:
4059:
4056:
4053:
4050:
4047:
4044:
4040:
4034:
4031:
4026:
4023:
4021:
4019:
4015:
4009:
4005:
4001:
3996:
3992:
3988:
3982:
3978:
3975:
3970:
3967:
3962:
3958:
3952:
3948:
3944:
3939:
3935:
3931:
3925:
3921:
3918:
3913:
3910:
3905:
3901:
3895:
3891:
3887:
3882:
3878:
3874:
3868:
3864:
3861:
3856:
3853:
3848:
3845:
3843:
3841:
3837:
3831:
3828:
3825:
3822:
3817:
3814:
3811:
3808:
3802:
3798:
3795:
3792:
3789:
3786:
3783:
3776:
3771:
3768:
3764:
3760:
3757:
3755:
3753:
3750:
3747:
3744:
3741:
3732:
3728:
3727:
3682:
3679:
3676:
3673:
3670:
3661:
3640:
3637:
3634:
3631:
3628:
3619:
3605:
3604:
3591:
3588:
3576:
3563:
3560:
3548:
3535:
3532:
3520:
3509:
3506:
3503:
3500:
3488:
3487:
3474:
3471:
3459:
3446:
3443:
3431:
3418:
3415:
3403:
3392:
3389:
3386:
3383:
3371:
3370:
3367:
3364:
3361:
3333:
3329:
3325:
3322:
3319:
3299:
3296:
3293:
3290:
3287:
3284:
3281:
3278:
3273:
3240:
3237:
3213:
3210:
3207:
3187:
3184:
3181:
3153:
3150:
3112:
3109:
3106:
3103:
3100:
3091:
3000:
2997:
2994:
2991:
2988:
2979:
2908:
2904:
2901:
2898:
2894:
2888:
2885:
2880:
2877:
2854:
2843:
2842:
2831:
2825:
2822:
2814:
2810:
2807:
2800:
2794:
2791:
2788:
2785:
2777:
2774:
2771:
2768:
2759:
2755:
2752:
2746:
2743:
2740:
2737:
2730:
2725:
2722:
2718:
2714:
2711:
2708:
2705:
2702:
2699:
2690:
2655:
2621:
2618:
2610:
2606:
2603:
2600:
2597:
2594:
2591:
2588:
2585:
2582:
2574:
2570:
2567:
2541:
2538:
2530:
2526:
2523:
2520:
2517:
2514:
2511:
2508:
2505:
2502:
2494:
2490:
2487:
2454:
2432:
2409:
2408:
2397:
2391:
2388:
2380:
2376:
2373:
2366:
2360:
2357:
2349:
2345:
2342:
2337:
2334:
2326:
2322:
2319:
2313:
2309:
2306:
2297:
2294:
2286:
2282:
2279:
2274:
2271:
2263:
2259:
2256:
2246:
2241:
2238:
2234:
2230:
2227:
2224:
2221:
2218:
2215:
2206:
2167:
2164:
2156:
2152:
2149:
2146:
2143:
2140:
2137:
2134:
2131:
2128:
2120:
2116:
2113:
2085:
2034:
2031:
2023:
2019:
2016:
2008:
2005:
1997:
1993:
1990:
1970:
1969:
1958:
1952:
1949:
1941:
1937:
1934:
1927:
1921:
1918:
1910:
1906:
1903:
1898:
1895:
1887:
1883:
1880:
1874:
1870:
1867:
1857:
1852:
1849:
1845:
1841:
1838:
1835:
1832:
1829:
1826:
1817:
1804:is defined as
1773:
1765:
1711:
1710:
1699:
1693:
1685:
1677:
1671:
1668:
1665:
1662:
1657:
1654:
1651:
1648:
1642:
1638:
1635:
1629:
1626:
1623:
1620:
1615:
1610:
1607:
1603:
1599:
1596:
1593:
1590:
1587:
1584:
1575:
1548:
1547:
1536:
1530:
1527:
1524:
1521:
1518:
1515:
1512:
1509:
1502:
1498:
1494:
1491:
1487:
1460:
1457:
1454:
1451:
1420:
1392:
1389:
1386:
1383:
1380:
1377:
1351:
1348:
1336:but the value
1323:
1320:
1292:
1289:
1286:
1283:
1280:
1277:
1251:
1248:
1245:
1242:
1239:
1236:
1199:
1198:
1187:
1180:
1174:
1171:
1168:
1165:
1157:
1154:
1151:
1148:
1139:
1135:
1132:
1126:
1123:
1120:
1117:
1110:
1105:
1102:
1098:
1094:
1091:
1088:
1085:
1082:
1079:
1076:
1067:
1052:
1051:
1040:
1033:
1027:
1024:
1021:
1018:
1010:
1007:
1004:
1001:
992:
988:
985:
979:
976:
973:
970:
963:
958:
955:
951:
947:
944:
941:
938:
935:
932:
923:
891:
883:
851:
848:
840:Kullback (1959
811:
808:
805:
802:
799:
796:
793:
790:
787:
784:
781:
778:
775:
772:
769:
766:
763:
760:
757:
754:
732:
728:
705:
701:
680:
677:
674:
671:
668:
665:
643:
639:
635:
630:
626:
603:
599:
595:
590:
586:
563:
559:
555:
550:
546:
523:
519:
496:
492:
469:
465:
440:
437:
435:computations.
396:
393:
390:
387:
384:
375:
342:
339:
331:bioinformatics
243:interpretation
239:
238:
227:
223:
217:
214:
211:
208:
200:
197:
194:
191:
182:
178:
175:
169:
166:
163:
160:
153:
148:
145:
141:
137:
134:
131:
128:
125:
122:
113:
74:
71:
68:
65:
62:
53:
15:
9:
6:
4:
3:
2:
26703:
26692:
26689:
26687:
26684:
26682:
26681:F-divergences
26679:
26677:
26674:
26673:
26671:
26662:
26659:
26656:
26652:
26648:
26645:
26643:
26640:
26638:
26635:
26633:
26630:
26628:
26625:
26624:
26613:
26608:
26605:
26604:0-8446-5625-9
26601:
26597:
26592:
26588:
26583:
26579:
26573:
26569:
26565:
26561:
26557:
26553:
26552:
26544:
26540:
26536:
26532:
26529:(2): 99–121.
26528:
26524:
26523:
26518:
26517:Guibas, L. J.
26511:
26502:
26496:
26488:
26484:
26480:
26474:
26470:
26469:
26461:
26447:
26443:
26439:
26433:
26429:
26425:
26421:
26420:
26412:
26404:
26400:
26395:
26390:
26385:
26380:
26376:
26372:
26367:
26362:
26358:
26354:
26353:
26348:
26341:
26333:
26329:
26324:
26319:
26314:
26309:
26305:
26301:
26296:
26291:
26287:
26283:
26282:
26277:
26270:
26262:
26258:
26254:
26248:
26244:
26243:
26235:
26226:
26221:
26217:
26213:
26209:
26202:
26194:
26190:
26186:
26182:
26178:
26174:
26170:
26165:
26160:
26156:
26152:
26148:
26141:
26133:
26129:
26125:
26121:
26117:
26113:
26106:
26097:
26096:
26088:
26080:
26076:
26072:
26068:
26064:
26060:
26053:
26046:
26038:
26034:
26030:
26026:
26022:
26018:
26014:
26010:
26003:
25996:
25988:
25987:
25979:
25971:
25965:
25961:
25960:
25955:
25948:
25939:
25934:
25929:
25924:
25920:
25916:
25915:
25910:
25903:
25895:
25891:
25884:
25882:
25873:
25869:
25865:
25864:
25856:
25854:
25845:
25841:
25837:
25833:
25829:
25825:
25821:
25817:
25813:
25806:
25799:. p. 13.
25795:
25788:
25780:
25776:
25772:
25768:
25763:
25758:
25754:
25750:
25743:
25735:
25731:
25727:
25723:
25716:
25709:
25706:
25702:
25696:
25687:
25682:
25678:
25672:
25664:
25658:
25654:
25647:
25640:
25636:
25632:
25628:
25622:
25614:
25610:
25603:
25595:
25591:
25587:
25583:
25576:
25568:
25564:
25559:
25554:
25549:
25544:
25540:
25536:
25531:
25526:
25522:
25518:
25514:
25507:
25499:
25495:
25491:
25487:
25480:
25472:
25470:9780387953649
25466:
25462:
25457:
25456:
25447:
25440:
25439:Kullback 1959
25435:
25427:
25423:
25419:
25418:
25410:
25402:
25398:
25392:
25384:
25380:
25374:
25366:
25364:9780521642989
25360:
25356:
25355:
25347:
25340:
25339:Kullback 1959
25335:
25327:
25323:
25319:
25315:
25311:
25307:
25306:
25301:
25295:
25288:
25287:Kullback 1959
25283:
25276:
25275:Jeffreys 1948
25271:
25269:
25262:, p. 80.
25261:
25256:
25254:
25247:, p. 28.
25246:
25241:
25239:
25232:, p. 11.
25231:
25226:
25224:
25222:
25220:
25218:
25210:
25209:Kullback 1959
25205:
25203:
25201:
25192:
25188:
25184:
25180:
25175:
25170:
25166:
25162:
25161:
25156:
25152:
25151:Leibler, R.A.
25148:
25142:
25133:
25128:
25124:
25120:
25116:
25109:
25107:
25102:
25093:
25090:
25089:
25085:
25081:
25078:
25076:
25073:
25071:
25068:
25066:
25063:
25061:
25058:
25056:
25053:
25051:
25048:
25046:
25043:
25041:
25038:
25036:
25033:
25031:
25030:Cross-entropy
25028:
25026:
25023:
25021:
25018:
25016:
25013:
25012:
25005:
25003:
24999:
24995:
24994:
24988:
24984:
24983:
24977:
24971:
24961:
24959:
24958:
24953:
24952:
24947:
24946:
24941:
24937:
24936:
24931:
24927:
24907:
24899:
24895:
24881:
24871:
24868:
24865:
24848:
24844:
24840:
24836:
24833:
24828:
24822:
24819:
24816:
24810:
24802:
24782:
24779:
24773:
24770:
24767:
24750:
24729:
24721:
24718:
24715:
24703:
24697:
24694:
24687:
24681:
24678:
24675:
24669:
24661:
24642:
24639:
24636:
24630:
24622:
24618:
24617:
24616:
24614:
24604:
24601:
24599:
24595:
24591:
24587:
24553:
24529:
24523:
24520:
24517:
24509:
24506:
24501:
24498:
24491:
24490:
24489:
24465:
24462:
24459:
24447:
24441:
24438:
24433:
24427:
24424:
24421:
24409:
24403:
24400:
24395:
24386:
24378:
24377:
24376:
24375:, defined by
24374:
24358:
24355:
24352:
24343:
24329:
24326:
24323:
24303:
24264:
24258:
24252:
24249:
24246:
24240:
24237:
24234:
24231:
24228:
24216:
24209:
24206:
24203:
24197:
24191:
24185:
24182:
24179:
24173:
24170:
24167:
24164:
24161:
24149:
24145:
24142:
24136:
24133:
24130:
24122:
24118:
24110:
24109:
24108:
24107:-divergence,
24094:
24085:
24083:
24079:
24075:
24063:
24059:
24054:
24052:
24048:
24044:
24022:
24019:
24016:
24004:
24000:
23994:
23991:
23988:
23976:
23968:
23967:
23966:
23964:
23955:
23953:
23949:
23944:
23942:
23937:
23933:
23928:
23926:
23922:
23918:
23911:
23905:
23895:
23890:in the state
23889:
23866:
23863:
23860:
23848:
23839:
23820:
23808:
23802:
23799:
23796:
23790:
23784:
23781:
23775:
23769:
23766:
23763:
23757:
23754:
23751:
23739:
23731:
23730:
23729:
23719:
23715:
23714:Hilbert space
23704:
23694:
23676:
23672:
23667:
23660:
23656:
23652:
23647:
23638:
23635:
23630:
23626:
23619:
23615:
23611:
23606:
23598:
23594:
23591:
23588:
23585:
23575:
23574:
23573:
23559:
23551:
23547:
23543:
23540:
23518:
23514:
23491:
23487:
23464:
23460:
23434:
23430:
23425:
23421:
23415:
23412:
23409:
23389:
23386:
23383:
23380:
23370:
23369:net surprisal
23352:
23348:
23339:
23334:
23332:
23316:
23313:
23310:
23307:
23304:
23301:
23298:
23295:
23292:
23289:
23283:
23255:
23251:
23246:
23240:
23236:
23232:
23229:
23226:
23221:
23217:
23191:
23187:
23182:
23178:
23167:
23163:
23159:
23156:
23153:
23150:
23144:
23141:
23119:
23115:
23107:and pressure
23092:
23088:
23079:
23063:
23060:
23057:
23054:
23051:
23048:
23045:
23042:
23039:
23032:
23000:
22997:
22994:
22991:
22988:
22985:
22978:
22958:
22955:
22952:
22932:
22929:
22926:
22913:
22911:
22888:
22882:
22879:
22876:
22873:
22870:
22867:
22859:
22855:
22843:
22836:
22831:
22806:
22802:
22798:
22757:
22751:
22748:
22744:
22740:
22737:
22734:
22731:
22728:
22725:
22721:
22717:
22714:
22711:
22707:
22679:
22675:
22671:
22665:
22662:
22659:
22656:
22653:
22641:
22620:
22616:
22606:
22597:
22580:
22577:
22574:
22543:
22540:
22537:
22525:
22501:
22498:
22495:
22483:
22466:
22443:
22437:
22434:
22431:
22419:
22415:
22409:
22398:
22392:
22389:
22386:
22371:
22370:
22369:
22347:
22343:
22338:
22336:
22335:
22330:
22326:
22322:
22318:
22313:
22292:
22286:
22263:
22257:
22234:
22231:
22228:
22222:
22219:
22213:
22210:
22207:
22201:
22175:
22172:
22169:
22163:
22160:
22154:
22151:
22148:
22142:
22130:
22121:
22117:
22098:
22095:
22092:
22086:
22063:
22060:
22057:
22051:
22028:
22022:
21999:
21993:
21966:
21960:
21937:
21928:
21922:
21919:
21913:
21907:
21895:
21891:
21887:
21877:
21874:
21871:
21865:
21862:
21856:
21853:
21850:
21844:
21832:
21827:
21823:
21815:
21809:
21801:
21792:
21789:
21786:
21780:
21777:
21771:
21765:
21759:
21756:
21753:
21747:
21735:
21727:
21726:
21725:
21708:
21702:
21696:
21693:
21690:
21684:
21653:
21647:
21612:
21609:
21606:
21600:
21591:
21590:as possible.
21572:
21568:
21564:
21561:
21549:
21526:
21522:
21509:
21505:
21491:
21489:
21485:
21484:loss function
21481:
21476:
21471:
21466:
21464:
21445:
21436:
21433:
21430:
21424:
21421:
21415:
21412:
21409:
21403:
21391:
21387:
21384:
21381:
21378:
21367:
21363:
21359:
21356:
21350:
21347:
21339:
21335:
21331:
21328:
21322:
21310:
21302:
21301:
21300:
21283:
21277:
21269:
21251:
21247:
21224:
21220:
21210:
21194:
21190:
21167:
21163:
21154:
21136:
21132:
21109:
21105:
21082:
21078:
21055:
21051:
21028:
21024:
21015:
20987:
20983:
20979:
20976:
20970:
20967:
20959:
20955:
20951:
20948:
20942:
20926:
20911:
20909:
20904:
20894:
20877:
20874:
20871:
20868:
20863:
20859:
20855:
20850:
20846:
20839:
20831:
20802:
20799:
20796:
20790:
20787:
20781:
20778:
20773:
20769:
20765:
20762:
20756:
20740:
20736:
20723:
20720:
20715:
20711:
20707:
20704:
20698:
20695:
20689:
20686:
20681:
20677:
20673:
20668:
20664:
20660:
20657:
20651:
20635:
20604:
20601:
20598:
20592:
20589:
20583:
20580:
20575:
20571:
20567:
20562:
20558:
20554:
20551:
20545:
20529:
20521:
20520:
20519:
20517:
20496:
20487:
20484:
20481:
20475:
20467:
20464:
20459:
20455:
20451:
20448:
20442:
20436:
20432:
20429:
20423:
20420:
20415:
20411:
20407:
20404:
20398:
20393:
20389:
20366:
20357:
20354:
20351:
20345:
20337:
20334:
20329:
20325:
20321:
20316:
20312:
20308:
20305:
20299:
20293:
20289:
20286:
20280:
20277:
20272:
20268:
20264:
20259:
20255:
20251:
20248:
20242:
20237:
20233:
20225:
20224:
20223:
20206:
20203:
20200:
20194:
20171:
20168:
20163:
20159:
20155:
20152:
20146:
20123:
20120:
20115:
20111:
20107:
20102:
20098:
20094:
20091:
20085:
20059:
20055:
20051:
20046:
20042:
20032:
20018:
20015:
20012:
19984:
19975:
19972:
19969:
19963:
19955:
19952:
19949:
19946:
19943:
19937:
19931:
19927:
19924:
19918:
19915:
19912:
19909:
19906:
19900:
19895:
19891:
19887:
19874:
19871:
19868:
19862:
19859:
19853:
19850:
19847:
19844:
19841:
19835:
19819:
19811:
19810:
19809:
19792:
19789:
19786:
19783:
19780:
19774:
19751:
19748:
19745:
19739:
19713:
19710:
19707:
19701:
19670:
19664:
19661:
19658:
19655:
19652:
19646:
19643:
19640:
19634:
19631:
19628:
19625:
19622:
19616:
19611:
19607:
19603:
19600:
19587:
19584:
19581:
19578:
19575:
19569:
19550:
19549:
19548:
19546:
19521:
19518:
19515:
19509:
19501:
19498:
19495:
19489:
19483:
19480:
19477:
19474:
19471:
19465:
19459:
19453:
19450:
19447:
19444:
19441:
19435:
19428:
19427:
19426:
19424:
19405:
19402:
19399:
19396:
19393:
19387:
19364:
19361:
19358:
19352:
19328:
19325:
19322:
19299:
19296:
19293:
19287:
19278:
19272:
19264:
19260:
19256:
19246:
19221:
19204:
19202:
19183:
19177:
19174:
19171:
19159:
19155:
19149:
19138:
19129:
19123:
19120:
19117:
19111:
19106:
19098:
19092:
19089:
19086:
19071:
19070:
19069:
19067:
19047:
19035:
19031:
19030:cross entropy
19026:
19016:
19015:cross entropy
19012:
18989:
18969:
18961:
18953:
18949:
18945:
18934:Cross entropy
18931:
18906:
18898:
18892:
18869:
18861:
18857:
18844:
18825:
18822:
18819:
18813:
18790:
18784:
18778:
18770:
18766:
18757:
18725:
18720:
18713:
18705:
18701:
18697:
18693:
18689:
18686:
18683:
18679:
18675:
18671:
18661:
18656:
18652:
18647:
18639:
18633:
18627:
18624:
18621:
18619:
18608:
18605:
18602:
18596:
18590:
18584:
18573:
18571:
18557:
18549:
18545:
18541:
18535:
18529:
18517:
18513:
18504:
18498:
18492:
18486:
18483:
18477:
18474:
18471:
18465:
18453:
18449:
18443:
18437:
18434:
18431:
18429:
18415:
18409:
18403:
18395:
18391:
18387:
18381:
18378:
18375:
18369:
18357:
18353:
18347:
18341:
18338:
18335:
18333:
18325:
18322:
18319:
18300:
18299:
18298:
18296:
18266:
18260:
18244:
18238:
18226:
18222:
18216:
18210:
18207:
18200:
18199:
18198:
18181:
18178:
18175:
18165:
18159:
18151:
18145:
18139:
18136:
18130:
18124:
18121:
18118:
18112:
18106:
18103:
18100:
18094:
18086:
18082:
18070:
18058:
18057:
18056:
18054:
18050:
18046:
18027:
18021:
17998:
17990:
17986:
17973:
17954:
17948:
17925:
17917:
17913:
17900:
17896:
17864:
17857:
17849:
17845:
17841:
17835:
17827:
17823:
17818:
17808:
17804:
17798:
17792:
17789:
17786:
17784:
17775:
17768:
17762:
17757:
17748:
17744:
17738:
17736:
17728:
17709:
17708:
17707:
17705:
17695:
17685:
17667:
17664:
17661:
17655:
17639:
17620:
17614:
17608:
17602:
17579:
17576:
17573:
17567:
17560:
17528:
17522:
17519:
17513:
17510:
17507:
17501:
17489:
17482:
17477:
17469:
17467:
17450:
17444:
17441:
17435:
17432:
17429:
17423:
17411:
17404:
17399:
17391:
17389:
17375:
17369:
17363:
17357:
17354:
17348:
17345:
17342:
17336:
17324:
17320:
17318:
17310:
17307:
17304:
17298:
17284:
17283:
17282:
17280:
17270:
17256:
17253:
17250:
17227:
17221:
17197:
17194:
17191:
17183:
17164:
17158:
17135:
17131:
17122:
17118:
17111:
17102:
17097:
17087:
17083:
17077:
17071:
17059:
17058:
17057:
17055:
17050:
17048:
17044:
17040:
17036:
17030:
17020:
17012:
17010:
17006:
17002:
16998:
16974:
16968:
16959:
16954:
16950:
16946:
16935:
16930:
16924:
16920:
16916:
16910:
16901:
16898:
16895:
16889:
16886:
16880:
16877:
16874:
16868:
16851:
16850:
16849:
16832:
16829:
16826:
16823:
16817:
16814:
16811:
16806:
16797:
16793:
16781:
16778:
16775:
16769:
16764:
16746:
16743:
16740:
16736:
16727:
16724:
16721:
16715:
16712:
16709:
16704:
16697:
16686:
16683:
16680:
16674:
16658:
16655:
16652:
16648:
16635:
16634:
16633:
16602:
16599:
16596:
16593:
16590:
16583:
16577:
16571:
16568:
16565:
16559:
16549:
16546:
16543:
16539:
16535:
16529:
16523:
16520:
16518:
16512:
16504:
16500:
16488:
16482:
16479:
16474:
16463:
16452:
16449:
16446:
16439:
16433:
16423:
16420:
16417:
16413:
16409:
16403:
16397:
16394:
16392:
16386:
16378:
16374:
16365:
16360:
16349:
16341:
16335:
16332:
16329:
16322:
16316:
16306:
16303:
16300:
16296:
16292:
16286:
16280:
16277:
16275:
16269:
16263:
16255:
16246:
16234:
16214:
16210:
16198:
16192:
16189:
16184:
16173:
16165:
16161:
16149:
16143:
16140:
16135:
16124:
16118:
16107:
16101:
16098:
16085:
16084:
16083:
16064:
16061:
16058:
16051:
16036:
16030:
16011:
16003:
16000:
15996:
15973:
15969:
15960:
15933:
15929:
15922:
15919:
15913:
15907:
15895:
15887:
15883:
15879:
15876:
15871:
15863:
15859:
15849:
15845:
15835:
15821:
15813:
15809:
15800:
15797:
15793:
15785:
15784:
15783:
15781:
15762:
15759:
15751:
15747:
15738:
15735:
15731:
15724:
15720:
15710:
15706:
15696:
15693:
15688:
15677:
15673:
15666:
15663:
15657:
15651:
15639:
15631:
15630:
15629:
15627:
15608:
15605:
15602:
15591:
15587:
15580:
15577:
15571:
15565:
15553:
15545:
15541:
15537:
15534:
15529:
15521:
15517:
15496:
15495:
15494:
15478:
15474:
15462:
15444:
15440:
15436:
15433:
15413:
15410:
15407:
15382:
15378:
15366:
15362:
15353:
15344:
15340:
15335:
15331:
15327:
15321:
15317:
15292:
15270:
15260:
15256:
15252:
15249:
15243:
15238:
15234:
15207:
15204:
15196:
15192:
15183:
15179:
15172:
15168:
15161:
15153:
15149:
15142:
15139:
15133:
15127:
15120:
15119:
15118:
15116:
15098:
15094:
15073:
15048:
15044:
15037:
15034:
15031:
15008:
15002:
14999:
14996:
14976:
14960:
14950:
14948:
14928:
14923:
14920:
14916:
14910:
14906:
14899:
14894:
14890:
14884:
14880:
14872:
14871:
14870:
14869:entails that
14868:
14849:
14843:
14839:
14832:
14828:
14820:
14819:
14818:
14800:
14797:
14791:
14788:
14783:
14779:
14766:
14754:
14742:
14741:
14740:
14723:
14720:
14715:
14711:
14707:
14702:
14698:
14686:
14682:
14677:
14675:
14656:
14653:
14650:
14638:
14614:
14611:
14608:
14596:
14588:. In general
14587:
14583:
14579:
14575:
14571:
14567:
14563:
14553:
14552:long stream.
14535:
14532:
14527:
14523:
14502:
14499:
14494:
14490:
14476:
14460:
14457:
14454:
14449:
14446:
14443:
14437:
14434:
14431:
14427:
14416:
14405:
14395:
14386:
14372:
14369:
14366:
14363:
14360:
14357:
14354:
14331:
14328:
14325:
14319:
14316:
14293:
14290:
14287:
14281:
14278:
14264:
14234:
14228:
14225:
14221:
14217:
14213:
14206:
14203:
14198:
14195:
14191:
14184:
14181:
14178:
14173:
14169:
14165:
14161:
14150:
14139:
14129:
14120:
14104:
14100:
14095:
14089:
14085:
14081:
14078:
14053:
14050:
14045:
14037:
14032:
14028:
14024:
14017:
14007:
14003:
13999:
13994:
13990:
13983:
13978:
13973:
13969:
13962:
13955:
13951:
13945:
13941:
13935:
13932:
13929:
13925:
13914:
13903:
13893:
13885:
13884:
13883:
13880:
13874:
13854:
13850:
13845:
13840:
13835:
13831:
13827:
13823:
13820:
13817:
13814:
13811:
13806:
13801:
13797:
13793:
13788:
13783:
13779:
13774:
13768:
13763:
13760:
13757:
13753:
13747:
13744:
13739:
13735:
13730:
13721:
13712:
13701:
13697:
13692:
13686:
13681:
13677:
13673:
13670:
13667:
13662:
13657:
13653:
13648:
13644:
13641:
13638:
13626:
13620:
13616:
13612:
13609:
13606:
13601:
13597:
13592:
13586:
13574:
13564:
13556:
13555:
13554:
13552:
13533:
13529:
13520:
13517:
13507:
13503:
13492:
13489:
13479:
13475:
13465:
13462:
13457:
13452:
13449:
13446:
13442:
13438:
13435:
13430:
13420:
13412:
13409:
13406:
13401:
13391:
13388:
13384:
13375:
13370:
13367:
13364:
13361:
13358:
13354:
13349:
13343:
13340:
13335:
13331:
13325:
13313:
13308:
13295:
13285:
13277:
13276:
13275:
13259:
13255:
13251:
13246:
13242:
13238:
13235:
13230:
13226:
13203:
13199:
13195:
13192:
13187:
13183:
13152:
13147:
13143:
13137:
13133:
13129:
13124:
13097:
13092:
13088:
13082:
13078:
13074:
13069:
13042:
13038:
13034:
13029:
13025:
13015:
13013:
12994:
12988:
12985:
12977:
12969:
12963:
12944:
12940:
12935:
12927:
12912:
12898:
12894:
12891:
12888:
12884:
12878:
12874:
12870:
12865:
12861:
12856:
12850:
12847:
12842:
12825:
12819:
12815:
12811:
12806:
12802:
12797:
12792:
12789:
12786:
12782:
12776:
12766:
12763:
12758:
12749:
12745:
12742:
12738:
12732:
12729:
12724:
12720:
12714:
12702:
12697:
12684:
12674:
12666:
12665:
12664:
12646:
12641:
12633:
12628:
12616:
12598:
12594:
12590:
12585:
12581:
12573:, with means
12572:
12557:
12555:
12549:
12533:
12529:
12525:
12522:
12502:
12499:
12491:
12487:
12483:
12480:
12468:
12444:
12438:
12435:
12432:
12424:
12420:
12416:
12413:
12410:
12402:
12398:
12394:
12391:
12379:
12375:
12369:
12366:
12363:
12355:
12351:
12347:
12344:
12341:
12335:
12332:
12329:
12317:
12313:
12307:
12299:
12295:
12287:
12286:
12285:
12268:
12262:
12259:
12256:
12248:
12244:
12240:
12237:
12234:
12228:
12220:
12216:
12212:
12209:
12203:
12200:
12197:
12185:
12181:
12173:
12169:
12165:
12162:
12150:
12142:
12141:
12140:
12123:
12120:
12114:
12105:
12102:
12099:
12091:
12087:
12078:
12072:
12069:
12066:
12060:
12054:
12051:
12043:
12039:
12012:
12009:
12006:
11998:
11994:
11985:
11979:
11976:
11973:
11944:
11940:
11912:
11906:
11903:
11889:
11861:
11858:
11852:
11844:
11841:
11835:
11805:
11796:
11793:
11790:
11782:
11778:
11769:
11763:
11760:
11757:
11751:
11742:
11739:
11733:
11725:
11722:
11716:
11706:
11705:
11704:
11687:
11678:
11675:
11672:
11660:
11656:
11650:
11642:
11638:
11629:
11626:
11623:
11613:
11607:
11604:
11601:
11593:
11589:
11585:
11582:
11575:
11574:
11573:
11556:
11553:
11543:
11529:
11501:
11498:
11495:
11475:
11472:
11469:
11446:
11443:
11433:
11404:
11401:
11391:
11378:
11330:
11292:
11290:
11280:
11275:
11255:
11233:
11228:
11222:
11219:
11214:
11211:
11207:
11202:
11193:
11190:
11187:
11181:
11177:
11155:
11135:
11132:
11129:
11126:
11123:
11120:
11100:
11080:
11077:
11074:
11071:
11051:
11048:
11045:
11025:
11018:with measure
11005:
11002:
10999:
10979:
10976:
10973:
10970:
10967:
10964:
10941:
10938:
10918:
10915:
10912:
10892:
10889:
10886:
10866:
10846:
10843:
10840:
10837:
10817:
10814:
10811:
10808:
10785:
10782:
10780:
10766:
10763:
10760:
10754:
10750:
10738:
10735:
10732:
10728:
10724:
10722:
10713:
10706:
10700:
10697:
10691:
10685:
10681:
10668:
10665:
10661:
10651:
10648:
10645:
10639:
10635:
10623:
10620:
10617:
10613:
10609:
10607:
10597:
10594:
10591:
10586:
10576:
10570:
10562:
10556:
10550:
10547:
10543:
10537:
10530:
10524:
10521:
10515:
10509:
10505:
10492:
10489:
10485:
10475:
10472:
10469:
10463:
10459:
10447:
10444:
10441:
10437:
10433:
10431:
10425:
10416:
10413:
10410:
10402:
10396:
10389:
10378:
10372:
10369:
10363:
10357:
10339:
10336:
10332:
10322:
10319:
10316:
10310:
10306:
10300:
10289:
10286:
10283:
10279:
10254:
10251:
10248:
10245:
10217:
10214:
10211:
10203:
10197:
10190:
10179:
10173:
10170:
10164:
10158:
10140:
10137:
10133:
10123:
10120:
10117:
10111:
10107:
10095:
10092:
10089:
10085:
10081:
10079:
10068:
10065:
10057:
10046:
10039:
10025:
10022:
10019:
10015:
10011:
10009:
10001:
9998:
9995:
9983:
9958:
9955:
9952:
9945:evaluated at
9932:
9912:
9889:
9886:
9883:
9871:
9841:
9838:
9835:
9827:
9821:
9814:
9803:
9797:
9794:
9788:
9782:
9764:
9761:
9757:
9753:
9747:
9744:
9741:
9733:
9725:
9722:
9716:
9714:
9706:
9695:
9688:
9676:
9673:
9670:
9665:
9658:
9652:
9649:
9646:
9640:
9634:
9628:
9625:
9622:
9615:
9607:
9596:
9590:
9587:
9581:
9575:
9557:
9554:
9550:
9546:
9540:
9537:
9534:
9526:
9518:
9515:
9509:
9507:
9499:
9488:
9481:
9467:
9461:
9454:
9443:
9437:
9434:
9428:
9422:
9404:
9401:
9397:
9393:
9391:
9383:
9376:
9373:
9359:
9353:
9350:
9347:
9341:
9335:
9329:
9326:
9323:
9313:
9302:
9296:
9293:
9287:
9281:
9263:
9260:
9256:
9252:
9250:
9242:
9235:
9232:
9203:
9200:
9198:
9190:
9183:
9180:
9171:
9162:
9156:
9148:
9142:
9139:
9136:
9130:
9124:
9118:
9115:
9112:
9103:
9099:
9096:
9087:
9081:
9078:
9072:
9066:
9051:
9048:
9044:
9040:
9038:
9029:
9025:
9022:
9018:
9009:
9003:
8995:
8989:
8986:
8983:
8977:
8971:
8965:
8962:
8959:
8950:
8946:
8943:
8939:
8929:
8923:
8920:
8914:
8908:
8893:
8890:
8886:
8882:
8880:
8872:
8865:
8862:
8837:
8814:
8808:
8805:
8799:
8796:
8793:
8781:
8757:
8754:
8751:
8748:
8745:
8742:
8736:
8733:
8730:
8715:
8711:
8705:
8699:
8673:
8653:
8650:
8647:
8644:
8619:
8616:
8613:
8605:
8599:
8592:
8581:
8575:
8572:
8566:
8560:
8542:
8539:
8535:
8525:
8522:
8519:
8513:
8509:
8497:
8494:
8491:
8487:
8483:
8477:
8474:
8471:
8459:
8438:
8435:
8432:
8409:
8406:
8403:
8391:
8383:
8369:
8366:
8363:
8360:
8357:
8344:
8340:
8336:
8331:
8327:
8314:
8307:
8304:
8301:
8295:
8287:
8283:
8279:
8274:
8270:
8257:
8253:
8250:
8242:
8238:
8231:
8228:
8225:
8219:
8214:
8210:
8206:
8203:
8198:
8194:
8187:
8184:
8181:
8175:
8170:
8166:
8162:
8150:
8124:
8120:
8116:
8111:
8107:
8078:
8074:
8070:
8065:
8061:
8034:
8031:
8028:
8018:
8014:
7995:
7992:
7989:
7977:
7968:
7954:
7946:
7942:
7938:
7933:
7929:
7916:
7912:
7904:
7900:
7896:
7891:
7887:
7874:
7870:
7864:
7861:
7858:
7846:
7823:
7819:
7815:
7810:
7806:
7782:
7779:
7771:
7767:
7760:
7757:
7749:
7745:
7741:
7735:
7732:
7729:
7726:
7723:
7717:
7694:
7691:
7683:
7679:
7672:
7669:
7661:
7657:
7653:
7647:
7644:
7641:
7638:
7635:
7629:
7607:
7603:
7599:
7594:
7590:
7581:
7577:
7573:
7570:
7566:
7549:
7546:
7539:
7533:
7530:
7524:
7521:
7515:
7492:
7486:
7463:
7457:
7445:
7424:
7420:
7413:
7410:
7405:
7401:
7375:
7371:
7364:
7361:
7356:
7352:
7327:
7324:
7319:
7310:
7298:
7287:
7275:
7266:
7262:
7259:
7253:
7241:
7231:
7227:
7219:
7215:
7210:
7206:
7204:
7196:
7193:
7188:
7174:
7165:
7162:
7157:
7154:
7136:
7130:
7118:
7102:
7093:
7090:
7085:
7082:
7064:
7058:
7046:
7037:
7033:
7030:
7019:
7010:
7007:
7002:
6999:
6982:
6976:
6964:
6954:
6950:
6942:
6938:
6933:
6929:
6927:
6919:
6916:
6911:
6902:
6896:
6888:
6882:
6876:
6872:
6869:
6863:
6857:
6850:
6846:
6838:
6834:
6829:
6825:
6823:
6815:
6812:
6809:
6797:
6784:
6760:
6750:
6747:
6742:
6739:
6710:
6707:
6696:
6686:
6683:
6678:
6675:
6660:
6648:
6642:
6639:
6636:
6629:
6617:
6611:
6608:
6605:
6598:
6592:
6589:
6583:
6580:
6574:
6554:
6551:
6539:
6529:
6526:
6521:
6518:
6500:
6494:
6482:
6476:
6473:
6470:
6463:
6451:
6445:
6442:
6439:
6432:
6426:
6423:
6417:
6414:
6408:
6385:
6379:
6367:
6363:
6345:
6342:
6339:
6327:
6310:
6309:
6308:
6282:
6278:
6259:
6256:
6253:
6222:
6206:
6190:
6168:
6162:
6159:
6153:
6147:
6124:
6121:
6115:
6109:
6103:
6100:
6094:
6091:
6085:
6062:
6059:
6053:
6047:
6041:
6038:
6032:
6029:
6023:
5999:
5996:
5993:
5986:
5967:
5964:
5961:
5949:
5940:
5924:
5921:
5918:
5912:
5909:
5906:
5894:
5885:
5881:
5880:
5874:
5872:
5868:
5864:
5858:
5856:
5852:
5851:rate function
5847:
5845:
5841:
5837:
5818:
5815:
5812:
5800:
5776:
5770:
5747:
5741:
5738:
5732:
5729:
5726:
5720:
5712:
5711:loss function
5693:
5690:
5687:
5681:
5673:
5661:
5642:
5639:
5636:
5624:
5614:
5608:
5589:
5565:
5564:cross entropy
5546:
5543:
5540:
5502:
5491:
5485:
5482:
5479:
5468:
5466:
5452:
5446:
5442:
5437:
5434:
5428:
5422:
5410:
5407:
5403:
5399:
5390:
5384:
5380:
5375:
5372:
5366:
5360:
5348:
5345:
5341:
5337:
5335:
5327:
5324:
5321:
5309:
5297:
5296:
5295:
5293:
5267:
5263:
5240:
5236:
5207:
5203:
5199:
5195:
5191:
5183:
5179:
5172:
5146:
5142:
5133:
5125:
5120:
5111:
5090:
5080:
5071:
5069:
5065:
5061:
5054:
5049:
5047:
5043:
5039:
5035:
5031:
5027:
5023:
5022:metric tensor
5019:
5015:
5014:infinitesimal
4996:
4993:
4990:
4978:
4954:
4951:
4948:
4936:
4928:. In general
4927:
4923:
4919:
4915:
4911:
4907:
4903:
4898:
4896:
4880:
4879:approximation
4862:
4849:
4842:
4823:
4820:
4817:
4805:
4796:
4791:
4781:
4769:
4750:
4747:
4744:
4732:
4723:
4713:
4700:samples from
4699:
4695:
4676:
4673:
4670:
4658:
4649:
4648:coding theory
4639:
4614:
4608:
4605:
4602:
4599:
4593:
4587:
4584:
4581:
4561:
4528:
4525:
4521:
4514:
4508:
4505:
4502:
4499:
4493:
4487:
4484:
4481:
4478:
4472:
4466:
4463:
4460:
4457:
4453:
4447:
4444:
4439:
4437:
4428:
4422:
4418:
4414:
4409:
4405:
4401:
4395:
4391:
4388:
4383:
4380:
4375:
4371:
4365:
4361:
4357:
4352:
4348:
4344:
4338:
4334:
4331:
4326:
4323:
4318:
4314:
4308:
4304:
4300:
4295:
4291:
4287:
4281:
4277:
4274:
4269:
4266:
4261:
4259:
4250:
4241:
4235:
4227:
4221:
4215:
4211:
4208:
4202:
4196:
4184:
4181:
4177:
4173:
4171:
4163:
4160:
4157:
4145:
4133:
4132:
4113:
4110:
4107:
4103:
4096:
4090:
4087:
4084:
4081:
4075:
4069:
4066:
4063:
4060:
4054:
4048:
4045:
4042:
4038:
4032:
4029:
4024:
4022:
4013:
4007:
4003:
3999:
3994:
3990:
3986:
3980:
3976:
3973:
3968:
3965:
3960:
3956:
3950:
3946:
3942:
3937:
3933:
3929:
3923:
3919:
3916:
3911:
3908:
3903:
3899:
3893:
3889:
3885:
3880:
3876:
3872:
3866:
3862:
3859:
3854:
3851:
3846:
3844:
3835:
3826:
3820:
3812:
3806:
3800:
3796:
3793:
3787:
3781:
3769:
3766:
3762:
3758:
3756:
3748:
3745:
3742:
3730:
3718:
3717:
3716:
3714:
3710:
3703:, designated
3702:
3696:
3677:
3674:
3671:
3659:
3635:
3632:
3629:
3617:
3589:
3586:
3577:
3561:
3558:
3549:
3533:
3530:
3521:
3504:
3498:
3491:Distribution
3490:
3489:
3472:
3469:
3460:
3444:
3441:
3432:
3416:
3413:
3404:
3387:
3381:
3374:Distribution
3373:
3372:
3368:
3365:
3362:
3357:
3356:
3349:
3345:
3331:
3327:
3323:
3320:
3317:
3294:
3291:
3288:
3285:
3282:
3276:
3238:
3235:
3211:
3208:
3205:
3185:
3182:
3179:
3171:
3152:Basic example
3149:
3139:
3132:
3107:
3104:
3101:
3089:
3074:
3070:
3063:
3059:
3055:
3051:
3044:
3038:
3031:
3017:
2995:
2992:
2989:
2977:
2964:
2962:
2955:, or to base
2954:
2950:
2946:
2942:
2938:
2934:
2930:
2926:
2906:
2902:
2899:
2896:
2892:
2886:
2883:
2878:
2875:
2852:
2829:
2820:
2805:
2798:
2789:
2783:
2772:
2766:
2757:
2753:
2750:
2741:
2735:
2723:
2720:
2716:
2712:
2706:
2703:
2700:
2688:
2680:
2679:
2678:
2653:
2616:
2601:
2595:
2589:
2586:
2580:
2565:
2536:
2521:
2515:
2509:
2506:
2500:
2485:
2430:
2414:
2411:which is the
2395:
2386:
2371:
2364:
2355:
2340:
2332:
2317:
2311:
2307:
2304:
2292:
2277:
2269:
2254:
2239:
2236:
2232:
2228:
2222:
2219:
2216:
2204:
2196:
2195:
2194:
2192:
2162:
2147:
2141:
2135:
2132:
2126:
2111:
2054:
2029:
2014:
2003:
1988:
1956:
1947:
1932:
1925:
1916:
1901:
1893:
1878:
1872:
1868:
1865:
1850:
1847:
1843:
1839:
1833:
1830:
1827:
1815:
1807:
1806:
1805:
1791:
1771:
1751:
1747:
1734:
1724:
1697:
1691:
1675:
1666:
1660:
1652:
1646:
1640:
1636:
1633:
1624:
1618:
1605:
1601:
1597:
1591:
1588:
1585:
1573:
1565:
1564:
1563:
1561:
1534:
1528:
1525:
1519:
1513:
1510:
1507:
1500:
1496:
1489:
1477:
1476:
1475:
1455:
1449:
1437:
1390:
1387:
1381:
1375:
1346:
1318:
1309:
1290:
1287:
1281:
1275:
1249:
1246:
1240:
1234:
1218:
1204:
1185:
1178:
1169:
1163:
1152:
1146:
1137:
1133:
1130:
1121:
1115:
1103:
1100:
1096:
1092:
1089:
1083:
1080:
1077:
1065:
1057:
1056:
1055:
1038:
1031:
1022:
1016:
1005:
999:
990:
986:
983:
974:
968:
956:
953:
949:
945:
939:
936:
933:
921:
913:
912:
911:
889:
868:
857:
847:
845:
841:
838:are given in
837:
833:
829:
825:
806:
803:
800:
794:
791:
785:
782:
779:
773:
770:
764:
761:
758:
752:
730:
726:
703:
699:
675:
672:
669:
663:
641:
637:
633:
628:
624:
601:
597:
593:
588:
584:
561:
557:
553:
548:
544:
521:
517:
494:
490:
467:
463:
454:
450:
446:
436:
434:
430:
391:
388:
385:
373:
338:
336:
332:
328:
324:
320:
316:
312:
306:
304:
300:
296:
292:
288:
284:
280:
276:
260:
256:
244:
225:
221:
212:
206:
195:
189:
180:
176:
173:
164:
158:
146:
143:
139:
135:
129:
126:
123:
111:
103:
102:
101:
92:
88:
69:
66:
63:
51:
42:
38:
35:(also called
34:
30:
26:
22:
26647:Sergio Verdú
26611:
26586:
26559:
26526:
26520:
26510:
26471:. Springer.
26467:
26460:
26449:, retrieved
26418:
26411:
26356:
26350:
26340:
26285:
26279:
26269:
26245:. Springer.
26241:
26234:
26215:
26211:
26201:
26193:the original
26157:(3): 18–26.
26154:
26150:
26140:
26115:
26111:
26105:
26094:
26087:
26062:
26058:
26045:
26012:
26008:
25995:
25985:
25978:
25958:
25947:
25938:11299/199630
25918:
25912:
25902:
25896:, p. 22
25889:
25862:
25819:
25815:
25805:
25787:
25752:
25748:
25742:
25725:
25721:
25715:
25705:Sergio Verdú
25695:
25671:
25652:
25646:
25626:
25621:
25615:(84): 11–44.
25612:
25609:Mat. Sbornik
25608:
25602:
25585:
25581:
25575:
25520:
25516:
25506:
25489:
25485:
25479:
25454:
25446:
25441:, p. 5.
25434:
25416:
25409:
25400:
25391:
25382:
25373:
25353:
25346:
25341:, p. 6.
25334:
25309:
25303:
25300:Kullback, S.
25294:
25289:, p. 7.
25282:
25167:(1): 79–86.
25164:
25158:
25147:Kullback, S.
25141:
25122:
25118:
24997:
24993:differencing
24992:
24986:
24981:
24975:
24973:
24955:
24949:
24943:
24939:
24933:
24929:
24923:
24610:
24602:
24585:
24544:
24483:
24344:
24279:
24086:
24081:
24077:
24055:
24050:
24040:
23961:
23954:estimators.
23945:
23929:
23924:
23920:
23916:
23913:
23888:entanglement
23835:
23700:
23691:
23368:
23335:
22914:
22834:
22832:
22638:
22464:
22458:
22345:
22341:
22339:
22332:
22314:
22119:
22115:
21952:
21592:
21507:
21499:
21497:
21487:
21469:
21467:
21460:
21267:
21211:
21013:
20917:
20900:
20829:
20827:
20515:
20513:
20187:rather than
20033:
20000:
19685:
19542:
19252:
19205:
19198:
19032:between two
19027:
18974:(denoted as
18937:
18842:
18755:
18745:
18292:
18196:
17971:
17894:
17884:
17701:
17683:
17556:
17276:
17150:
17051:
17032:
17018:
16994:
16847:
16624:
16042:
15956:
15777:
15623:
15460:
15222:
14956:
14944:
14865:
14816:
14678:
14581:
14573:
14559:
14477:
14387:
14270:
14121:
14070:
13878:
13872:
13869:
13548:
13167:. Then with
13016:
12959:
12568:
12551:
12459:
12283:
11895:
11820:
11702:
11294:
11288:
11286:
11278:
8691:
6372:to variable
6287:rather than
6015:
6012:as measures.
5983:equals zero
5884:non-negative
5859:
5848:
5835:
5615:
5524:
5291:
5290:: it is the
5129:
5081:
5077:
5050:
4921:
4913:
4899:
4868:
4792:
4779:
4770:achieved if
4719:
4696:required to
4645:
4557:
3608:
3155:
3130:
3068:
3067:and updates
3057:
3042:
3036:
3029:
3015:
2965:
2941:Haar measure
2844:
2677:is given as
2419:relative to
2410:
1971:
1735:
1712:
1549:
1438:
1219:
1200:
1053:
867:sample space
853:
843:
831:
826:in 1948. In
442:
344:
327:neuroscience
307:
240:
41:I-divergence
40:
36:
32:
28:
24:
18:
25677:Bonnici, V.
25119:Ann. Probab
24982:compression
22329:E.T. Jaynes
18045:E.T. Jaynes
17636:of the two
17047:probability
15624:and by the
12284:Therefore,
6295:drawn from
5060:-divergence
3695:natural log
1721:denote the
1203:expectation
353:. Usually,
315:time-series
261:from using
43:), denoted
26670:Categories
26451:2023-02-14
26366:1912.00610
26359:(2): 221.
26295:1904.04017
26288:(5): 485.
26151:Complexity
25872:1363563215
25762:2008.01006
25686:2008.05932
25629:ch. 14.5 (
25530:1811.08308
25523:(8): 860.
25426:1334664824
25245:Amari 2016
25230:Amari 2016
25098:References
24371:gives the
24345:The value
23902:See also:
23329:(see also
22848:or volume
22640:Surprisals
22323:, and the
21153:I. J. Good
19201:Motivation
17005:partitions
15961:. Letting
15778:where the
14570:divergence
13057:such that
10957:such that
8050:, i.e. if
6279:number of
5877:Properties
5672:divergence
5114:Motivation
5020:, gives a
4910:divergence
4554:Statistics
3697:with base
2191:chain rule
2101:such that
850:Definition
291:divergence
33:divergence
26495:cite book
26487:757859245
26455:Lemma 2.1
26446:122597694
26261:878132909
26164:1103.2481
25836:0007-1102
25792:Duchi J.
25779:220935477
24908:α
24869:∥
24845:−
24837:−
24829:≤
24811:δ
24771:∥
24719:∥
24688:≤
24670:δ
24631:δ
24463:∥
24425:∥
24353:λ
24330:λ
24327:−
24304:λ
24253:λ
24250:−
24235:λ
24232:∥
24210:λ
24207:−
24186:λ
24183:−
24168:λ
24165:∥
24146:λ
24134:∥
24123:λ
24095:λ
24020:∥
23992:∥
23864:∥
23803:
23797:−
23785:
23770:
23755:∥
23644:Θ
23603:Θ
23583:Δ
23557:Δ
23416:
23384:≥
23378:Δ
23314:≥
23308:
23302:−
23296:−
23278:Θ
23173:Θ
23148:Δ
23058:−
22995:−
22989:≡
22930:×
22883:
22874:−
22871:≡
22749:−
22741:×
22729:
22666:
22541:∥
22499:∥
22435:∥
22344:(MCE) or
22232:∣
22211:∣
22173:∣
22161:∥
22152:∣
22096:∣
22061:∣
21920:∥
21875:∣
21863:∥
21854:∣
21824:
21778:∥
21757:∣
21694:∣
21565:∥
21434:∣
21422:∥
21413:∣
21379:≠
21360:∣
21348:∥
21332:∣
20980:∣
20968:∥
20952:∣
20856:∣
20800:∣
20788:∥
20766:∣
20708:∣
20696:∥
20661:∣
20602:∣
20590:∥
20555:∣
20485:∣
20452:∣
20433:
20408:∣
20390:∑
20355:∣
20309:∣
20290:
20252:∣
20234:∑
20204:∣
20156:∣
20095:∣
19973:∣
19947:∣
19928:
19910:∣
19892:∑
19872:∣
19860:∥
19845:∣
19784:∣
19749:∣
19711:∣
19656:∣
19644:
19626:∣
19608:∑
19604:−
19579:∣
19519:∣
19499:∣
19475:∣
19445:∣
19397:∣
19362:∣
19297:∣
19285:→
19175:∥
19124:
19118:−
19112:
18698:∥
18687:∣
18653:
18640:−
18628:
18597:
18591:−
18542:∥
18514:−
18484:∥
18450:−
18438:
18388:∥
18354:−
18342:
18323:∣
18223:−
18211:
18140:
18122:∫
18119:−
18107:
18077:∞
18074:→
17842:∥
17805:−
17793:
17763:
17745:
17520:∥
17511:∣
17483:
17442:∥
17433:∣
17405:
17355:∥
17299:
17112:∥
17103:δ
17072:
16972:→
16969:ρ
16951:ρ
16921:ρ
16902:ρ
16887:∥
16812:ρ
16794:ρ
16790:∂
16782:ρ
16761:∂
16752:∞
16737:∫
16710:ρ
16698:ρ
16695:∂
16687:ρ
16672:∂
16664:∞
16649:∫
16606:∞
16600:ξ
16555:∞
16540:∫
16501:ρ
16497:∂
16483:
16471:∂
16456:∞
16429:∞
16414:∫
16375:ρ
16371:∂
16357:∂
16339:∞
16312:∞
16297:∫
16264:ρ
16261:∂
16253:∂
16211:ρ
16207:∂
16193:
16181:∂
16162:ρ
16158:∂
16144:
16132:∂
16119:ρ
16116:∂
16102:
16096:∂
16065:ρ
16012:θ
15970:θ
15930:θ
15920:∥
15914:θ
15884:θ
15877:θ
15860:θ
15856:∂
15846:θ
15842:∂
15832:∂
15810:θ
15763:⋯
15748:θ
15721:θ
15717:Δ
15707:θ
15703:Δ
15674:θ
15664:∥
15658:θ
15588:θ
15578:∥
15572:θ
15542:θ
15535:θ
15518:θ
15514:∂
15510:∂
15475:θ
15471:Δ
15441:θ
15434:θ
15379:θ
15363:θ
15359:∂
15351:∂
15332:θ
15293:θ
15257:θ
15253:−
15250:θ
15235:θ
15231:Δ
15208:⋯
15193:θ
15169:θ
15165:Δ
15150:θ
15134:θ
15095:θ
15074:θ
15045:θ
15009:θ
14977:θ
14903:⇒
14789:∥
14761:∞
14758:→
14724:…
14654:∥
14612:∥
14533:
14500:
14458:−
14447:−
14438:
14417:∥
14370:≤
14358:≤
14229:
14204:−
14196:−
14179:
14151:∥
14101:σ
14086:σ
14046:−
14029:σ
14004:μ
14000:−
13991:μ
13970:σ
13952:σ
13942:σ
13936:
13915:∥
13832:σ
13824:
13818:−
13812:−
13798:μ
13780:σ
13754:∑
13702:∥
13678:σ
13671:…
13654:σ
13645:
13617:μ
13610:…
13598:μ
13466:
13443:∑
13407:−
13355:∑
13314:∥
13256:μ
13252:−
13243:μ
13121:Σ
13066:Σ
12989:
12962:logarithm
12924:Σ
12909:Σ
12895:
12875:μ
12871:−
12862:μ
12848:−
12839:Σ
12816:μ
12812:−
12803:μ
12787:−
12773:Σ
12764:−
12755:Σ
12746:
12703:∥
12638:Σ
12625:Σ
12595:μ
12582:μ
12534:∗
12500:≥
12492:∗
12484:∥
12436:
12417:
12411:≤
12403:∗
12395:∥
12376:−
12367:
12348:
12333:∥
12314:−
12260:
12241:
12213:−
12201:∥
12182:−
12174:∗
12166:∥
12124:θ
12103:
12079:θ
12070:
12055:θ
12044:∗
12010:
11986:θ
11977:
11963:-density
11945:∗
11907:
11862:θ
11845:θ
11794:
11770:θ
11761:
11743:θ
11726:θ
11676:∥
11657:−
11627:≪
11605:
11586:
11541:Θ
11499:≪
11473:≪
11431:Θ
11389:Θ
11331:σ
11311:Θ
11262:∞
11259:→
11220:ϵ
11203:ρ
11191:−
11136:ϵ
11124:≥
11101:ρ
11075:≤
11052:ρ
11049:−
11026:ρ
11003:≤
10980:ϵ
10968:≥
10945:∞
10913:ρ
10887:ϵ
10812:≤
10764:−
10744:∞
10729:∑
10725:≤
10698:−
10669:∈
10662:∑
10649:−
10629:∞
10614:∑
10610:≤
10595:−
10551:−
10522:−
10493:∈
10486:∑
10473:−
10453:∞
10438:∑
10414:−
10370:−
10340:∈
10333:∑
10320:−
10295:∞
10280:∑
10249:≤
10215:−
10171:−
10141:∈
10134:∑
10121:−
10101:∞
10086:∑
10031:∞
10016:∑
9999:∥
9953:α
9887:∥
9839:−
9795:−
9765:∈
9758:∑
9745:−
9723:−
9674:−
9650:α
9629:α
9626:−
9588:−
9558:∈
9551:∑
9538:−
9516:−
9500:α
9435:−
9405:∈
9398:∑
9351:α
9330:α
9327:−
9294:−
9264:∈
9257:∑
9243:α
9140:α
9119:α
9116:−
9100:
9079:−
9052:∈
9045:∑
8987:α
8966:α
8963:−
8947:
8921:−
8894:∈
8887:∑
8873:α
8797:∥
8755:∥
8749:α
8737:α
8734:−
8706:α
8648:≤
8617:−
8573:−
8543:∈
8536:∑
8523:−
8503:∞
8488:∑
8475:∥
8407:∥
8367:≤
8364:λ
8361:≤
8337:∥
8308:λ
8305:−
8280:∥
8254:λ
8251:≤
8232:λ
8229:−
8207:λ
8204:∥
8188:λ
8185:−
8163:λ
7993:∥
7939:∥
7897:∥
7862:∥
7302:~
7279:~
7263:
7245:~
7211:∫
7122:~
7050:~
7034:
6968:~
6934:∫
6873:
6830:∫
6813:∥
6652:~
6621:~
6486:~
6455:~
6343:∥
6191:μ
6116:μ
6054:μ
5965:∥
5919:≥
5910:∥
5816:∥
5640:∥
5492:−
5438:
5411:∈
5404:∑
5400:−
5376:
5349:∈
5342:∑
5325:∥
5294:entropy.
5237:ℓ
5204:ℓ
5200:−
4994:∥
4952:∥
4821:∥
4748:∥
4716:Inference
4674:∥
4606:
4600:−
4585:
4529:0.097455.
4526:≈
4509:
4488:
4479:−
4467:
4458:−
4392:
4335:
4278:
4212:
4185:∈
4178:∑
4161:∥
4111:0.0852996
4108:≈
4091:
4082:−
4070:
4049:
3977:
3920:
3863:
3797:
3770:∈
3763:∑
3746:∥
3675:∥
3633:∥
3105:∥
3073:posterior
3050:asymmetry
2993:∥
2945:Lie group
2876:μ
2853:μ
2806:μ
2754:
2724:∈
2717:∫
2704:∥
2654:μ
2602:μ
2522:μ
2431:μ
2308:
2240:∈
2233:∫
2220:∥
1869:
1851:∈
1844:∫
1831:∥
1637:
1614:∞
1609:∞
1606:−
1602:∫
1589:∥
1514:
1493:→
1439:Whenever
1388:≠
1350:∞
1322:∞
1134:
1104:∈
1097:∑
1093:−
1081:∥
987:
957:∈
950:∑
937:∥
727:μ
700:μ
638:μ
625:μ
558:μ
545:μ
518:μ
439:Etymology
389:∥
319:inference
241:A simple
177:
147:∈
140:∑
127:∥
67:∥
26558:(2016).
26543:14106275
26403:33285995
26332:33267199
26189:20794688
26037:17870175
25844:21492134
25567:33286632
25153:(1951).
25008:See also
24987:relative
24976:absolute
24974:Just as
23533:is thus
22368:, since
19017:between
18946:) using
15957:must be
14917:→
14891:→
14840:→
14681:topology
12560:Examples
11825:, where
11297:Theorem
9377:″
9236:″
9184:′
8866:′
7576:additive
6783:Jacobian
6277:expected
5792:to make
5228:, where
5064:alphabet
1746:measures
1265:implies
259:surprise
255:expected
26394:7516653
26371:Bibcode
26352:Entropy
26323:7514974
26300:Bibcode
26281:Entropy
26169:Bibcode
26120:Bibcode
26067:Bibcode
26017:Bibcode
25594:4134885
25558:7517462
25535:Bibcode
25517:Entropy
25326:2684769
25191:0039968
25183:2236703
24586:locally
23013:(where
22842:entropy
22830:coins.
22346:Minxent
22317:Laplace
20830:average
19545:entropy
18841:— i.e.
17970:— i.e.
17180:from a
16031:on the
15426:, i.e.
15305:in the
14582:squared
12556:, see.
12031:, i.e.
11522:.) Let
11462:, with
11343:-field
9971:yields
8692:Denote
6140:, then
5941:, with
5607:entropy
5605:is the
5562:is the
5044:and in
5018:Hessian
4922:squared
4846:to the
3136:or the
3016:between
2413:entropy
2051:is the
257:excess
253:is the
23:, the
26602:
26574:
26541:
26485:
26475:
26444:
26434:
26401:
26391:
26330:
26320:
26259:
26249:
26187:
26035:
25966:
25870:
25842:
25834:
25777:
25659:
25637:
25592:
25565:
25555:
25467:
25424:
25361:
25324:
25189:
25181:
24954:, and
24484:where
23716:, the
23452:where
23209:where
22904:where
19421:using
18926:given
16848:then:
16625:where
16232:exist,
15461:second
14574:linear
14566:metric
13218:, and
11932:, let
10931:, and
9925:about
8013:convex
7344:where
6723:where
6275:, the
5836:excess
5574:, and
5525:where
5292:excess
5028:; see
4914:linear
4906:metric
4642:Coding
3262:(i.e.
3114:
3086:
3002:
2974:
2937:sphere
2911:
2873:
2827:
2817:
2803:
2779:
2764:
2748:
2657:
2651:
2623:
2613:
2577:
2563:
2543:
2533:
2497:
2483:
2393:
2383:
2369:
2352:
2329:
2302:
2289:
2266:
2169:
2159:
2123:
2109:
2089:
2079:
2036:
2026:
2010:
2000:
1986:
1980:
1972:where
1954:
1944:
1930:
1913:
1890:
1863:
1769:
1759:
1713:where
1695:
1688:
1680:
1631:
1532:
1462:
1447:
1424:
1414:
1394:
1373:
1353:
1344:
1294:
1273:
1253:
1232:
1183:
1159:
1144:
1128:
1036:
1012:
997:
981:
887:
877:
576:, and
333:, and
275:metric
202:
187:
171:
26539:S2CID
26442:S2CID
26361:arXiv
26290:arXiv
26185:S2CID
26159:arXiv
26055:(PDF)
26033:S2CID
26005:(PDF)
25797:(PDF)
25775:S2CID
25757:arXiv
25681:arXiv
25525:arXiv
25322:JSTOR
25179:JSTOR
25002:patch
24998:given
24991:data
24980:data
23720:from
23712:on a
22858:Gibbs
22695:. If
21628:over
21475:logit
21239:over
21182:over
21043:over
19345:from
19261:to a
18750:from
17889:from
17043:event
16043:When
15223:with
12139:Then
11959:have
11893:Proof
8451:) as
7838:then
5224:over
3711:(see
3172:with
3144:over
3140:from
3062:prior
2943:on a
2475:with
1748:on a
1558:of a
249:from
26655:NIPS
26600:ISBN
26572:ISBN
26501:link
26483:OCLC
26473:ISBN
26432:ISBN
26399:PMID
26328:PMID
26257:OCLC
26247:ISBN
25964:ISBN
25868:OCLC
25840:PMID
25832:ISSN
25710:2009
25708:NIPS
25657:ISBN
25635:ISBN
25590:SSRN
25586:June
25582:Risk
25563:PMID
25465:ISBN
25422:OCLC
25359:ISBN
25082:and
24780:>
24619:The
24576:and
24316:and
24074:pdfs
24068:and
23950:and
23934:via
23708:and
23701:For
23506:and
23270:and
23078:work
23025:and
22738:1.38
22364:and
22118:(or
21673:and
21640:was
21632:and
21488:both
21016:for
19060:and
19046:bits
19040:and
19028:The
19021:and
19011:bits
18843:less
18756:less
18293:The
17972:less
17895:less
17702:The
17644:and
17277:The
17033:The
16603:<
16597:<
16521:<
16453:<
16395:<
16336:<
16278:<
15024:and
14965:and
14364:<
13876:and
13642:diag
13171:and
13112:and
13012:bits
12976:nats
12960:The
11420:and
11371:and
11303:Let
10992:and
10942:<
10916:>
10890:>
10841:>
8096:and
7578:for
7479:and
7393:and
6567:and
6315:and
6281:bits
6078:and
5570:and
4698:code
4694:bits
4566:and
3709:nats
3651:and
3198:and
3160:and
3071:the
3058:from
3037:from
3030:from
3021:and
2961:nats
2953:bits
2949:base
2639:and
2555:and
2471:and
1784:and
1740:and
1729:and
1717:and
1554:and
1209:and
861:and
854:For
718:and
482:and
447:and
431:and
423:and
349:and
39:and
26564:doi
26531:doi
26424:doi
26389:PMC
26379:doi
26318:PMC
26308:doi
26220:doi
26177:doi
26128:doi
26116:224
26075:doi
26063:108
26025:doi
26013:106
25933:hdl
25923:doi
25824:doi
25767:doi
25730:doi
25633:).
25553:PMC
25543:doi
25494:doi
25314:doi
25169:doi
25127:doi
25004:).
24600:).
24359:0.5
24292:or
24082:PSI
24060:in
23836:In
23800:log
23782:log
23724:to
23333:).
23134:is
22699:is
22465:not
22352:to
22327:of
22319:'s
21508:MDI
21268:not
21266:is
20516:not
20430:log
20287:log
19925:log
19641:log
19253:In
19121:log
18970:of
18922:of
18625:log
18435:log
18339:log
18208:log
18137:log
18104:log
18067:lim
17901:of
17790:log
16480:log
16190:log
16141:log
16099:log
14751:lim
14524:log
14491:log
14435:log
14170:log
13933:log
12920:det
12905:det
12433:exp
12414:log
12364:exp
12345:log
12257:exp
12238:log
12100:exp
12067:exp
12007:exp
11974:exp
11904:exp
11791:exp
11758:exp
11619:sup
11602:exp
11583:log
11530:on
11488:. (
9097:log
8944:log
8011:is
7567:or
7260:log
7031:log
6870:log
5609:of
5566:of
5435:log
5373:log
4881:of
4782:of
4603:log
4582:log
3715:):
3212:0.4
3052:in
2751:log
2673:to
2415:of
2305:log
2071:on
2055:of
1866:log
1800:to
1788:is
1725:of
1634:log
1511:log
1486:lim
1131:log
984:log
906:to
451:in
174:log
19:In
26672::
26653:,
26649:,
26589:,
26570:.
26537:.
26527:40
26525:.
26497:}}
26493:{{
26481:.
26440:,
26430:,
26397:.
26387:.
26377:.
26369:.
26357:22
26355:.
26349:.
26326:.
26316:.
26306:.
26298:.
26286:21
26284:.
26278:.
26255:.
26216:28
26214:.
26210:.
26183:.
26175:.
26167:.
26155:13
26153:.
26149:.
26126:.
26114:.
26073:.
26061:.
26057:.
26031:.
26023:.
26011:.
26007:.
25956:.
25931:.
25919:10
25917:.
25911:.
25892:,
25880:^
25866:.
25852:^
25838:.
25830:.
25820:64
25818:.
25814:.
25773:.
25765:.
25753:51
25751:.
25726:36
25724:.
25613:42
25611:.
25588:.
25584:.
25561:.
25551:.
25541:.
25533:.
25521:22
25519:.
25515:.
25488:.
25463:.
25461:51
25320:.
25310:41
25308:.
25267:^
25252:^
25237:^
25216:^
25199:^
25187:MR
25185:.
25177:.
25165:22
25163:.
25157:.
25149:;
25121:.
25117:.
25105:^
24985:,
24960:.
24948:,
24942:,
24938:,
24932:,
24928:,
24708:KL
24662::
24623:,
24558:JS
24452:KL
24414:KL
24391:JS
24221:KL
24154:KL
24053:.
24009:KL
23981:KL
23927:.
23894:.
23853:KL
23767:Tr
23744:KL
23413:ln
23305:ln
22912:.
22880:ln
22752:23
22745:10
22726:ln
22663:ln
22596:.
22530:KL
22488:KL
22424:KL
22135:KL
21900:KL
21837:KL
21740:KL
21677:,
21554:KL
21396:KL
21315:KL
20931:KL
20910:.
20745:KL
20640:KL
20534:KL
20031:.
19824:KL
19547::
19425::
19265::
19245:.
19164:KL
19025:.
18930:.
18666:KL
18522:KL
18458:KL
18362:KL
18297:,
18231:KL
17905:,
17813:KL
17706:,
17684:is
17494:KL
17416:KL
17329:KL
17281:,
17269:.
17107:im
17092:KL
17011:.
16975:0.
15900:KL
15644:KL
15558:KL
15117:)
14949:.
14771:KL
14643:KL
14601:KL
14400:KL
14226:ln
14134:KL
13898:KL
13821:ln
13569:KL
13463:ln
13290:KL
13274:,
13014:.
12986:ln
12892:ln
12743:tr
12679:KL
12473:KL
12384:KL
12322:KL
12190:KL
12155:KL
11888:.
11665:KL
11291:.
10905:,
9988:KL
9876:KL
8786:KL
8720:KL
8712::=
8464:KL
8396:KL
8370:1.
8319:KL
8262:KL
8155:KL
7982:KL
7921:KL
7879:KL
7851:KL
6802:KL
6332:KL
6307:.
5954:KL
5899:KL
5886:,
5873:.
5857:.
5805:KL
5739:=:
5629:KL
5314:KL
5095:KL
5070:.
5048:.
4983:KL
4941:KL
4897:.
4861:.
4810:KL
4797:,
4790:.
4737:KL
4724:,
4712:.
4663:KL
4650:,
4506:ln
4485:ln
4464:ln
4423:25
4389:ln
4366:25
4358:12
4332:ln
4309:25
4275:ln
4209:ln
4150:KL
4088:ln
4085:50
4067:ln
4064:55
4046:ln
4043:32
4033:25
3995:25
3974:ln
3969:25
3938:25
3930:12
3917:ln
3912:25
3909:12
3881:25
3860:ln
3855:25
3794:ln
3735:KL
3705:ln
3664:KL
3622:KL
3473:25
3445:25
3442:12
3417:25
3369:2
3344:.
3258:,
3254:,
3224:.
3148:.
3094:KL
3069:to
3060:a
3043:to
2982:KL
2939:,
2693:KL
2209:KL
1820:KL
1733:.
1578:KL
1224:,
1217:.
1070:KL
926:KL
869:,
846:.
378:KL
337:.
329:,
325:,
116:KL
56:KL
31:)
29:KL
26606:.
26580:.
26566::
26545:.
26533::
26503:)
26489:.
26426::
26405:.
26381::
26373::
26363::
26334:.
26310::
26302::
26292::
26263:.
26228:.
26222::
26179::
26171::
26161::
26134:.
26130::
26122::
26081:.
26077::
26069::
26039:.
26027::
26019::
25972:.
25941:.
25935::
25925::
25874:.
25846:.
25826::
25781:.
25769::
25759::
25736:.
25732::
25689:.
25683::
25665:.
25641:.
25596:.
25569:.
25545::
25537::
25527::
25500:.
25496::
25490:2
25473:.
25428:.
25403:.
25328:.
25316::
25211:.
25193:.
25171::
25135:.
25129::
25123:3
24882:.
24875:)
24872:Q
24866:P
24863:(
24857:L
24854:K
24849:D
24841:e
24834:1
24826:)
24823:Q
24820:,
24817:P
24814:(
24797:1
24783:2
24777:)
24774:Q
24768:P
24765:(
24759:L
24756:K
24751:D
24730:.
24725:)
24722:Q
24716:P
24713:(
24704:D
24698:2
24695:1
24685:)
24682:Q
24679:,
24676:P
24673:(
24646:)
24643:q
24640:,
24637:p
24634:(
24582:f
24578:Q
24574:P
24554:D
24530:.
24527:)
24524:Q
24521:+
24518:P
24515:(
24510:2
24507:1
24502:=
24499:M
24486:M
24469:)
24466:M
24460:Q
24457:(
24448:D
24442:2
24439:1
24434:+
24431:)
24428:M
24422:P
24419:(
24410:D
24404:2
24401:1
24396:=
24387:D
24356:=
24324:1
24294:Q
24290:P
24286:X
24282:X
24265:,
24262:)
24259:Q
24256:)
24247:1
24244:(
24241:+
24238:P
24229:Q
24226:(
24217:D
24213:)
24204:1
24201:(
24198:+
24195:)
24192:Q
24189:)
24180:1
24177:(
24174:+
24171:P
24162:P
24159:(
24150:D
24143:=
24140:)
24137:Q
24131:P
24128:(
24119:D
24080:(
24070:Q
24066:P
24026:)
24023:P
24017:Q
24014:(
24005:D
24001:+
23998:)
23995:Q
23989:P
23986:(
23977:D
23892:P
23884:Q
23870:)
23867:Q
23861:P
23858:(
23849:D
23821:.
23818:)
23815:)
23812:)
23809:Q
23806:(
23794:)
23791:P
23788:(
23779:(
23776:P
23773:(
23764:=
23761:)
23758:Q
23752:P
23749:(
23740:D
23726:P
23722:Q
23710:Q
23706:P
23677:.
23673:]
23668:)
23661:o
23657:T
23653:T
23648:(
23639:2
23636:3
23631:+
23627:)
23620:o
23616:V
23612:V
23607:(
23599:[
23595:k
23592:N
23589:=
23586:I
23560:I
23552:o
23548:T
23544:=
23541:W
23519:o
23515:T
23492:o
23488:V
23465:o
23461:p
23440:)
23435:o
23431:p
23426:/
23422:p
23419:(
23410:k
23390:,
23387:0
23381:I
23353:o
23349:T
23317:0
23311:x
23299:1
23293:x
23290:=
23287:)
23284:x
23281:(
23256:o
23252:P
23247:/
23241:o
23237:T
23233:k
23230:N
23227:=
23222:o
23218:V
23197:)
23192:o
23188:V
23183:/
23179:V
23176:(
23168:o
23164:T
23160:k
23157:N
23154:=
23151:G
23145:=
23142:W
23120:o
23116:P
23093:o
23089:T
23064:S
23061:T
23055:V
23052:P
23049:+
23046:U
23043:=
23040:G
23027:P
23023:T
23019:S
23015:U
23001:S
22998:T
22992:U
22986:F
22973:N
22959:V
22956:,
22953:T
22933:A
22927:T
22917:T
22906:Z
22892:)
22889:Z
22886:(
22877:k
22868:A
22850:V
22846:P
22840:(
22838:S
22828:N
22824:N
22810:}
22807:K
22803:/
22799:J
22779:{
22758:}
22735:,
22732:2
22722:/
22718:1
22715:,
22712:1
22708:{
22697:k
22683:)
22680:p
22676:/
22672:1
22669:(
22660:k
22657:=
22654:s
22644:p
22621:o
22617:T
22584:)
22581:m
22578:,
22575:p
22572:(
22568:H
22547:)
22544:m
22538:p
22535:(
22526:D
22505:)
22502:m
22496:p
22493:(
22484:D
22473:p
22469:m
22461:p
22444:,
22441:)
22438:m
22432:p
22429:(
22420:D
22416:+
22413:)
22410:p
22407:(
22403:H
22399:=
22396:)
22393:m
22390:,
22387:p
22384:(
22380:H
22366:m
22362:p
22358:m
22354:p
22350:m
22310:a
22296:)
22293:a
22290:(
22287:u
22267:)
22264:a
22261:(
22258:u
22238:)
22235:a
22229:x
22226:(
22223:p
22220:=
22217:)
22214:a
22208:x
22205:(
22202:q
22182:)
22179:)
22176:a
22170:x
22167:(
22164:p
22158:)
22155:a
22149:x
22146:(
22143:q
22140:(
22131:D
22102:)
22099:a
22093:x
22090:(
22087:q
22067:)
22064:a
22058:x
22055:(
22052:p
22032:)
22029:a
22026:(
22023:u
22003:)
22000:a
21997:(
21994:u
21984:a
21970:)
21967:a
21964:(
21961:p
21938:,
21935:)
21932:)
21929:a
21926:(
21923:p
21917:)
21914:a
21911:(
21908:u
21905:(
21896:D
21892:+
21888:}
21884:)
21881:)
21878:a
21872:x
21869:(
21866:p
21860:)
21857:a
21851:x
21848:(
21845:q
21842:(
21833:D
21828:{
21819:)
21816:a
21813:(
21810:u
21806:E
21802:=
21799:)
21796:)
21793:a
21790:,
21787:x
21784:(
21781:p
21775:)
21772:a
21769:(
21766:u
21763:)
21760:a
21754:x
21751:(
21748:q
21745:(
21736:D
21712:)
21709:a
21706:(
21703:u
21700:)
21697:a
21691:x
21688:(
21685:q
21675:a
21671:x
21657:)
21654:a
21651:(
21648:u
21638:a
21634:a
21630:x
21616:)
21613:a
21610:,
21607:x
21604:(
21601:p
21578:)
21573:0
21569:f
21562:f
21559:(
21550:D
21527:0
21523:f
21512:f
21506:(
21446:.
21443:)
21440:)
21437:I
21431:H
21428:(
21425:p
21419:)
21416:x
21410:H
21407:(
21404:p
21401:(
21392:D
21388:=
21385:G
21382:I
21376:)
21373:)
21368:0
21364:H
21357:x
21354:(
21351:p
21345:)
21340:1
21336:H
21329:x
21326:(
21323:p
21320:(
21311:D
21287:)
21284:H
21281:(
21278:p
21252:0
21248:H
21225:1
21221:H
21195:0
21191:H
21168:1
21164:H
21137:1
21133:H
21110:0
21106:H
21083:1
21079:H
21056:0
21052:H
21029:1
21025:H
20998:)
20993:)
20988:0
20984:H
20977:x
20974:(
20971:p
20965:)
20960:1
20956:H
20949:x
20946:(
20943:p
20938:(
20927:D
20881:)
20878:I
20875:,
20872:x
20869:,
20864:1
20860:y
20851:2
20847:y
20843:(
20840:p
20811:)
20806:)
20803:I
20797:x
20794:(
20791:p
20785:)
20782:I
20779:,
20774:1
20770:y
20763:x
20760:(
20757:p
20752:(
20741:D
20737:+
20732:)
20727:)
20724:I
20721:,
20716:1
20712:y
20705:x
20702:(
20699:p
20693:)
20690:I
20687:,
20682:2
20678:y
20674:,
20669:1
20665:y
20658:x
20655:(
20652:p
20647:(
20636:D
20613:)
20608:)
20605:I
20599:x
20596:(
20593:p
20587:)
20584:I
20581:,
20576:2
20572:y
20568:,
20563:1
20559:y
20552:x
20549:(
20546:p
20541:(
20530:D
20497:)
20491:)
20488:I
20482:x
20479:(
20476:p
20471:)
20468:I
20465:,
20460:1
20456:y
20449:x
20446:(
20443:p
20437:(
20427:)
20424:I
20421:,
20416:1
20412:y
20405:x
20402:(
20399:p
20394:x
20367:)
20361:)
20358:I
20352:x
20349:(
20346:p
20341:)
20338:I
20335:,
20330:2
20326:y
20322:,
20317:1
20313:y
20306:x
20303:(
20300:p
20294:(
20284:)
20281:I
20278:,
20273:2
20269:y
20265:,
20260:1
20256:y
20249:x
20246:(
20243:p
20238:x
20210:)
20207:I
20201:x
20198:(
20195:p
20175:)
20172:I
20169:,
20164:1
20160:y
20153:x
20150:(
20147:p
20127:)
20124:I
20121:,
20116:2
20112:y
20108:,
20103:1
20099:y
20092:x
20089:(
20086:p
20076:x
20060:2
20056:y
20052:=
20047:2
20043:Y
20019:y
20016:=
20013:Y
20003:X
19985:)
19979:)
19976:I
19970:x
19967:(
19964:p
19959:)
19956:I
19953:,
19950:y
19944:x
19941:(
19938:p
19932:(
19922:)
19919:I
19916:,
19913:y
19907:x
19904:(
19901:p
19896:x
19888:=
19883:)
19878:)
19875:I
19869:x
19866:(
19863:p
19857:)
19854:I
19851:,
19848:y
19842:x
19839:(
19836:p
19831:(
19820:D
19796:)
19793:I
19790:,
19787:y
19781:x
19778:(
19775:p
19755:)
19752:I
19746:x
19743:(
19740:p
19720:)
19717:)
19714:I
19708:x
19705:(
19702:p
19699:(
19695:H
19671:,
19668:)
19665:I
19662:,
19659:y
19653:x
19650:(
19647:p
19638:)
19635:I
19632:,
19629:y
19623:x
19620:(
19617:p
19612:x
19601:=
19596:)
19591:)
19588:I
19585:,
19582:y
19576:x
19573:(
19570:p
19565:(
19559:H
19525:)
19522:I
19516:y
19513:(
19510:p
19505:)
19502:I
19496:x
19493:(
19490:p
19487:)
19484:I
19481:,
19478:x
19472:y
19469:(
19466:p
19460:=
19457:)
19454:I
19451:,
19448:y
19442:x
19439:(
19436:p
19409:)
19406:I
19403:,
19400:y
19394:x
19391:(
19388:p
19368:)
19365:I
19359:x
19356:(
19353:p
19343:X
19329:y
19326:=
19323:Y
19303:)
19300:I
19294:x
19291:(
19288:p
19282:)
19279:x
19276:(
19273:p
19243:p
19239:q
19225:)
19222:p
19219:(
19215:H
19184:.
19181:)
19178:q
19172:p
19169:(
19160:D
19156:+
19153:)
19150:p
19147:(
19143:H
19139:=
19136:]
19133:)
19130:q
19127:(
19115:[
19107:p
19103:E
19099:=
19096:)
19093:q
19090:,
19087:p
19084:(
19080:H
19062:q
19058:p
19054:p
19050:q
19042:q
19038:p
19036:(
19023:q
19019:p
19007:q
18993:)
18990:p
18987:(
18983:H
18972:p
18964:p
18956:p
18940:p
18928:Y
18924:X
18910:)
18907:Y
18903:|
18899:X
18896:(
18893:P
18873:)
18870:X
18867:(
18862:U
18858:P
18847:X
18829:)
18826:Y
18823:,
18820:X
18817:(
18814:P
18794:)
18791:Y
18788:(
18785:P
18782:)
18779:X
18776:(
18771:U
18767:P
18752:N
18748:X
18726:]
18721:)
18717:)
18714:X
18711:(
18706:U
18702:P
18694:)
18690:Y
18684:X
18680:(
18676:P
18672:(
18662:D
18657:[
18648:Y
18644:E
18637:)
18634:N
18631:(
18622:=
18612:)
18609:Y
18606:;
18603:X
18600:(
18594:I
18588:)
18585:X
18582:(
18578:H
18574:=
18564:)
18561:)
18558:X
18555:(
18550:U
18546:P
18539:)
18536:X
18533:(
18530:P
18527:(
18518:D
18511:)
18508:)
18505:Y
18502:(
18499:P
18496:)
18493:X
18490:(
18487:P
18481:)
18478:Y
18475:,
18472:X
18469:(
18466:P
18463:(
18454:D
18447:)
18444:N
18441:(
18432:=
18422:)
18419:)
18416:Y
18413:(
18410:P
18407:)
18404:X
18401:(
18396:U
18392:P
18385:)
18382:Y
18379:,
18376:X
18373:(
18370:P
18367:(
18358:D
18351:)
18348:N
18345:(
18336:=
18329:)
18326:Y
18320:X
18317:(
18313:H
18273:)
18270:)
18267:x
18264:(
18261:m
18257:|
18252:|
18248:)
18245:x
18242:(
18239:p
18236:(
18227:D
18220:)
18217:N
18214:(
18182:,
18179:x
18176:d
18169:)
18166:x
18163:(
18160:m
18155:)
18152:x
18149:(
18146:p
18134:)
18131:x
18128:(
18125:p
18116:)
18113:N
18110:(
18101:=
18098:)
18095:X
18092:(
18087:N
18083:H
18071:N
18031:)
18028:X
18025:(
18022:P
18002:)
17999:X
17996:(
17991:U
17987:P
17976:X
17958:)
17955:X
17952:(
17949:P
17929:)
17926:X
17923:(
17918:U
17914:P
17903:X
17891:N
17887:X
17865:)
17861:)
17858:X
17855:(
17850:U
17846:P
17839:)
17836:x
17833:(
17828:X
17824:p
17819:(
17809:D
17802:)
17799:N
17796:(
17787:=
17776:]
17772:)
17769:x
17766:(
17758:X
17754:I
17749:[
17742:E
17739:=
17732:)
17729:X
17726:(
17722:H
17692:X
17688:Y
17671:)
17668:Y
17665:,
17662:X
17659:(
17656:P
17646:Y
17642:X
17624:)
17621:Y
17618:(
17615:P
17612:)
17609:X
17606:(
17603:P
17583:)
17580:Y
17577:,
17574:X
17571:(
17568:P
17538:}
17535:)
17532:)
17529:X
17526:(
17523:P
17517:)
17514:Y
17508:X
17505:(
17502:P
17499:(
17490:D
17486:{
17478:Y
17474:E
17470:=
17460:}
17457:)
17454:)
17451:Y
17448:(
17445:P
17439:)
17436:X
17430:Y
17427:(
17424:P
17421:(
17412:D
17408:{
17400:X
17396:E
17392:=
17382:)
17379:)
17376:Y
17373:(
17370:P
17367:)
17364:X
17361:(
17358:P
17352:)
17349:Y
17346:,
17343:X
17340:(
17337:P
17334:(
17325:D
17321:=
17314:)
17311:Y
17308:;
17305:X
17302:(
17296:I
17257:m
17254:=
17251:i
17231:)
17228:i
17225:(
17222:P
17212:i
17198:m
17195:=
17192:i
17168:)
17165:i
17162:(
17159:P
17136:,
17132:)
17128:}
17123:i
17119:p
17115:{
17098:(
17088:D
17084:=
17081:)
17078:m
17075:(
17068:I
16960:)
16955:3
16947:(
16941:O
16936:+
16931:2
16925:2
16917:c
16911:=
16908:)
16905:)
16899:,
16896:x
16893:(
16890:p
16884:)
16881:0
16878:,
16875:x
16872:(
16869:p
16866:(
16861:D
16833:0
16830:=
16827:x
16824:d
16818:0
16815:=
16807:|
16798:2
16785:)
16779:,
16776:x
16773:(
16770:p
16765:2
16747:0
16744:=
16741:x
16728:=
16725:x
16722:d
16716:0
16713:=
16705:|
16690:)
16684:,
16681:x
16678:(
16675:p
16659:0
16656:=
16653:x
16631:ρ
16627:ξ
16594:x
16591:d
16587:)
16584:x
16581:(
16578:H
16575:)
16572:0
16569:,
16566:x
16563:(
16560:p
16550:0
16547:=
16544:x
16536::
16533:)
16530:x
16527:(
16524:H
16513:|
16505:3
16492:)
16489:p
16486:(
16475:3
16464:|
16450:x
16447:d
16443:)
16440:x
16437:(
16434:G
16424:0
16421:=
16418:x
16410::
16407:)
16404:x
16401:(
16398:G
16387:|
16379:2
16366:p
16361:2
16350:|
16342:,
16333:x
16330:d
16326:)
16323:x
16320:(
16317:F
16307:0
16304:=
16301:x
16293::
16290:)
16287:x
16284:(
16281:F
16270:|
16256:p
16247:|
16215:3
16202:)
16199:p
16196:(
16185:3
16174:,
16166:2
16153:)
16150:p
16147:(
16136:2
16125:,
16111:)
16108:p
16105:(
16068:)
16062:,
16059:x
16056:(
16052:p
16033:θ
16015:)
16009:(
16004:k
16001:j
15997:g
15974:0
15942:)
15939:)
15934:0
15926:(
15923:P
15917:)
15911:(
15908:P
15905:(
15896:D
15888:0
15880:=
15872:|
15864:k
15850:j
15836:2
15822:=
15819:)
15814:0
15806:(
15801:k
15798:j
15794:g
15760:+
15757:)
15752:0
15744:(
15739:k
15736:j
15732:g
15725:k
15711:j
15697:2
15694:1
15689:=
15686:)
15683:)
15678:0
15670:(
15667:P
15661:)
15655:(
15652:P
15649:(
15640:D
15609:,
15606:0
15603:=
15600:)
15597:)
15592:0
15584:(
15581:P
15575:)
15569:(
15566:P
15563:(
15554:D
15546:0
15538:=
15530:|
15522:j
15479:j
15445:0
15437:=
15414:Q
15411:=
15408:P
15388:)
15383:0
15375:(
15367:j
15354:P
15345:=
15341:)
15336:0
15328:(
15322:j
15318:P
15307:j
15271:j
15267:)
15261:0
15247:(
15244:=
15239:j
15205:+
15202:)
15197:0
15189:(
15184:j
15180:P
15173:j
15162:+
15159:)
15154:0
15146:(
15143:P
15140:=
15137:)
15131:(
15128:P
15099:0
15054:)
15049:0
15041:(
15038:P
15035:=
15032:Q
15012:)
15006:(
15003:P
15000:=
14997:P
14967:Q
14963:P
14941:,
14929:P
14924:V
14921:T
14911:n
14907:P
14900:P
14895:D
14885:n
14881:P
14862:.
14850:Q
14844:D
14833:n
14829:P
14813:,
14801:0
14798:=
14795:)
14792:Q
14784:n
14780:P
14776:(
14767:D
14755:n
14727:}
14721:,
14716:2
14712:P
14708:,
14703:1
14699:P
14695:{
14660:)
14657:P
14651:Q
14648:(
14639:D
14618:)
14615:Q
14609:P
14606:(
14597:D
14550:k
14536:k
14528:2
14503:k
14495:2
14480:k
14461:A
14455:B
14450:C
14444:D
14432:=
14428:)
14422:q
14412:p
14406:(
14396:D
14373:D
14367:B
14361:A
14355:C
14347:(
14335:]
14332:D
14329:,
14326:C
14323:[
14320:=
14317:q
14297:]
14294:B
14291:,
14288:A
14285:[
14282:=
14279:p
14251:s
14248:t
14245:i
14242:b
14238:)
14235:2
14232:(
14222:/
14218:2
14214:/
14210:)
14207:1
14199:2
14192:k
14188:(
14185:+
14182:k
14174:2
14166:=
14162:)
14156:q
14146:p
14140:(
14130:D
14105:0
14096:/
14090:1
14082:=
14079:k
14054:2
14051:1
14038:2
14033:1
14025:2
14018:2
14014:)
14008:1
13995:0
13987:(
13984:+
13979:2
13974:0
13963:+
13956:0
13946:1
13930:=
13926:)
13920:q
13910:p
13904:(
13894:D
13879:q
13873:p
13855:.
13851:)
13846:)
13841:2
13836:i
13828:(
13815:1
13807:2
13802:i
13794:+
13789:2
13784:i
13775:(
13769:k
13764:1
13761:=
13758:i
13748:2
13745:1
13740:=
13736:)
13731:)
13726:I
13722:,
13718:0
13713:(
13707:N
13698:)
13693:)
13687:2
13682:k
13674:,
13668:,
13663:2
13658:1
13649:(
13639:,
13633:T
13627:)
13621:k
13613:,
13607:,
13602:1
13593:(
13587:(
13581:N
13575:(
13565:D
13534:.
13530:)
13521:i
13518:i
13514:)
13508:0
13504:L
13500:(
13493:i
13490:i
13486:)
13480:1
13476:L
13472:(
13458:k
13453:1
13450:=
13447:i
13439:2
13436:+
13431:2
13426:|
13421:y
13417:|
13413:+
13410:k
13402:2
13398:)
13392:j
13389:i
13385:M
13381:(
13376:k
13371:1
13368:=
13365:j
13362:,
13359:i
13350:(
13344:2
13341:1
13336:=
13332:)
13326:1
13320:N
13309:0
13303:N
13296:(
13286:D
13260:0
13247:1
13239:=
13236:y
13231:1
13227:L
13204:0
13200:L
13196:=
13193:M
13188:1
13184:L
13173:y
13169:M
13153:T
13148:1
13144:L
13138:1
13134:L
13130:=
13125:1
13098:T
13093:0
13089:L
13083:0
13079:L
13075:=
13070:0
13043:1
13039:L
13035:,
13030:0
13026:L
12998:)
12995:2
12992:(
12972:e
12967:e
12945:.
12941:)
12936:)
12928:0
12913:1
12899:(
12889:+
12885:)
12879:0
12866:1
12857:(
12851:1
12843:1
12832:T
12826:)
12820:0
12807:1
12798:(
12793:+
12790:k
12783:)
12777:0
12767:1
12759:1
12750:(
12739:(
12733:2
12730:1
12725:=
12721:)
12715:1
12709:N
12698:0
12692:N
12685:(
12675:D
12661:k
12647:.
12642:1
12634:,
12629:0
12599:1
12591:,
12586:0
12530:Q
12526:=
12523:Q
12503:0
12497:)
12488:Q
12481:Q
12478:(
12469:D
12445:,
12442:]
12439:h
12430:[
12425:P
12421:E
12408:)
12399:Q
12392:Q
12389:(
12380:D
12373:]
12370:h
12361:[
12356:P
12352:E
12342:=
12339:)
12336:P
12330:Q
12327:(
12318:D
12311:]
12308:h
12305:[
12300:Q
12296:E
12269:.
12266:]
12263:h
12254:[
12249:P
12245:E
12235:+
12232:]
12229:h
12226:[
12221:Q
12217:E
12210:=
12207:)
12204:P
12198:Q
12195:(
12186:D
12179:)
12170:Q
12163:Q
12160:(
12151:D
12127:)
12121:d
12118:(
12115:P
12109:]
12106:h
12097:[
12092:P
12088:E
12082:)
12076:(
12073:h
12061:=
12058:)
12052:d
12049:(
12040:Q
12016:]
12013:h
12004:[
11999:P
11995:E
11989:)
11983:(
11980:h
11961:P
11941:Q
11930:P
11916:)
11913:h
11910:(
11886:P
11882:Q
11865:)
11859:d
11856:(
11853:P
11848:)
11842:d
11839:(
11836:Q
11823:P
11806:,
11800:]
11797:h
11788:[
11783:P
11779:E
11773:)
11767:(
11764:h
11752:=
11746:)
11740:d
11737:(
11734:P
11729:)
11723:d
11720:(
11717:Q
11688:.
11685:}
11682:)
11679:P
11673:Q
11670:(
11661:D
11654:]
11651:h
11648:[
11643:Q
11639:E
11635:{
11630:P
11624:Q
11614:=
11611:]
11608:h
11599:[
11594:P
11590:E
11560:)
11557:P
11554:,
11549:F
11544:,
11538:(
11524:h
11520:P
11516:Q
11502:P
11496:Q
11476:P
11470:Q
11450:)
11447:Q
11444:,
11439:F
11434:,
11428:(
11408:)
11405:P
11402:,
11397:F
11392:,
11386:(
11373:Q
11369:P
11353:F
11256:n
11234:n
11229:)
11223:U
11215:+
11212:1
11208:(
11197:)
11194:1
11188:n
11185:(
11182:n
11178:1
11156:n
11133:+
11130:Q
11127:2
11121:P
11081:Q
11078:2
11072:P
11046:1
11006:U
11000:Q
10977:+
10974:Q
10971:2
10965:P
10939:U
10919:0
10893:0
10867:0
10847:Q
10844:2
10838:P
10818:Q
10815:2
10809:P
10786:1
10783:=
10770:)
10767:1
10761:n
10758:(
10755:n
10751:1
10739:2
10736:=
10733:n
10714:|
10710:)
10707:x
10704:(
10701:P
10695:)
10692:x
10689:(
10686:Q
10682:|
10674:X
10666:x
10655:)
10652:1
10646:n
10643:(
10640:n
10636:1
10624:2
10621:=
10618:n
10598:1
10592:n
10587:|
10580:)
10577:x
10574:(
10571:Q
10566:)
10563:x
10560:(
10557:P
10548:1
10544:|
10538:|
10534:)
10531:x
10528:(
10525:P
10519:)
10516:x
10513:(
10510:Q
10506:|
10498:X
10490:x
10479:)
10476:1
10470:n
10467:(
10464:n
10460:1
10448:2
10445:=
10442:n
10434:=
10426:|
10417:1
10411:n
10407:)
10403:x
10400:(
10397:Q
10390:n
10386:)
10382:)
10379:x
10376:(
10373:P
10367:)
10364:x
10361:(
10358:Q
10355:(
10345:X
10337:x
10326:)
10323:1
10317:n
10314:(
10311:n
10307:1
10301:|
10290:2
10287:=
10284:n
10255:Q
10252:2
10246:P
10218:1
10212:n
10208:)
10204:x
10201:(
10198:Q
10191:n
10187:)
10183:)
10180:x
10177:(
10174:P
10168:)
10165:x
10162:(
10159:Q
10156:(
10146:X
10138:x
10127:)
10124:1
10118:n
10115:(
10112:n
10108:1
10096:2
10093:=
10090:n
10082:=
10069:!
10066:n
10061:)
10058:0
10055:(
10050:)
10047:n
10044:(
10040:f
10026:0
10023:=
10020:n
10012:=
10005:)
10002:Q
9996:P
9993:(
9984:D
9959:1
9956:=
9933:0
9913:f
9893:)
9890:Q
9884:P
9881:(
9872:D
9842:1
9836:n
9832:)
9828:x
9825:(
9822:Q
9815:n
9811:)
9807:)
9804:x
9801:(
9798:Q
9792:)
9789:x
9786:(
9783:P
9780:(
9770:X
9762:x
9754:!
9751:)
9748:2
9742:n
9739:(
9734:n
9730:)
9726:1
9720:(
9717:=
9710:)
9707:0
9704:(
9699:)
9696:n
9693:(
9689:f
9677:1
9671:n
9666:)
9662:)
9659:x
9656:(
9653:P
9647:+
9644:)
9641:x
9638:(
9635:Q
9632:)
9623:1
9620:(
9616:(
9608:n
9604:)
9600:)
9597:x
9594:(
9591:Q
9585:)
9582:x
9579:(
9576:P
9573:(
9563:X
9555:x
9547:!
9544:)
9541:2
9535:n
9532:(
9527:n
9523:)
9519:1
9513:(
9510:=
9503:)
9497:(
9492:)
9489:n
9486:(
9482:f
9471:)
9468:x
9465:(
9462:Q
9455:2
9451:)
9447:)
9444:x
9441:(
9438:Q
9432:)
9429:x
9426:(
9423:P
9420:(
9410:X
9402:x
9394:=
9387:)
9384:0
9381:(
9374:f
9363:)
9360:x
9357:(
9354:P
9348:+
9345:)
9342:x
9339:(
9336:Q
9333:)
9324:1
9321:(
9314:2
9310:)
9306:)
9303:x
9300:(
9297:Q
9291:)
9288:x
9285:(
9282:P
9279:(
9269:X
9261:x
9253:=
9246:)
9240:(
9233:f
9204:0
9201:=
9194:)
9191:0
9188:(
9181:f
9172:)
9166:)
9163:x
9160:(
9157:Q
9152:)
9149:x
9146:(
9143:P
9137:+
9134:)
9131:x
9128:(
9125:Q
9122:)
9113:1
9110:(
9104:(
9094:)
9091:)
9088:x
9085:(
9082:Q
9076:)
9073:x
9070:(
9067:P
9064:(
9057:X
9049:x
9041:=
9030:)
9026:1
9023:+
9019:)
9013:)
9010:x
9007:(
9004:Q
8999:)
8996:x
8993:(
8990:P
8984:+
8981:)
8978:x
8975:(
8972:Q
8969:)
8960:1
8957:(
8951:(
8940:(
8936:)
8933:)
8930:x
8927:(
8924:Q
8918:)
8915:x
8912:(
8909:P
8906:(
8899:X
8891:x
8883:=
8876:)
8870:(
8863:f
8838:f
8818:)
8815:1
8812:(
8809:f
8806:=
8803:)
8800:Q
8794:P
8791:(
8782:D
8761:)
8758:Q
8752:P
8746:+
8743:Q
8740:)
8731:1
8728:(
8725:(
8716:D
8709:)
8703:(
8700:f
8686:.
8674:Q
8654:Q
8651:2
8645:P
8620:1
8614:n
8610:)
8606:x
8603:(
8600:Q
8593:n
8589:)
8585:)
8582:x
8579:(
8576:P
8570:)
8567:x
8564:(
8561:Q
8558:(
8548:X
8540:x
8529:)
8526:1
8520:n
8517:(
8514:n
8510:1
8498:2
8495:=
8492:n
8484:=
8481:)
8478:Q
8472:P
8469:(
8460:D
8439:Q
8436:=
8433:P
8413:)
8410:Q
8404:P
8401:(
8392:D
8358:0
8350:)
8345:2
8341:Q
8332:2
8328:P
8324:(
8315:D
8311:)
8302:1
8299:(
8296:+
8293:)
8288:1
8284:Q
8275:1
8271:P
8267:(
8258:D
8248:)
8243:2
8239:Q
8235:)
8226:1
8223:(
8220:+
8215:1
8211:Q
8199:2
8195:P
8191:)
8182:1
8179:(
8176:+
8171:1
8167:P
8160:(
8151:D
8130:)
8125:2
8121:Q
8117:,
8112:2
8108:P
8104:(
8084:)
8079:1
8075:Q
8071:,
8066:1
8062:P
8058:(
8038:)
8035:Q
8032:,
8029:P
8026:(
7999:)
7996:Q
7990:P
7987:(
7978:D
7955:.
7952:)
7947:2
7943:Q
7934:2
7930:P
7926:(
7917:D
7913:+
7910:)
7905:1
7901:Q
7892:1
7888:P
7884:(
7875:D
7871:=
7868:)
7865:Q
7859:P
7856:(
7847:D
7824:2
7820:Q
7816:,
7811:1
7807:Q
7786:)
7783:y
7780:d
7777:(
7772:2
7768:Q
7764:)
7761:x
7758:d
7755:(
7750:1
7746:Q
7742:=
7739:)
7736:y
7733:d
7730:,
7727:x
7724:d
7721:(
7718:Q
7698:)
7695:y
7692:d
7689:(
7684:2
7680:P
7676:)
7673:x
7670:d
7667:(
7662:1
7658:P
7654:=
7651:)
7648:y
7645:d
7642:,
7639:x
7636:d
7633:(
7630:P
7608:2
7604:P
7600:,
7595:1
7591:P
7550:x
7547:d
7543:)
7540:x
7537:(
7534:p
7531:=
7528:)
7525:x
7522:d
7519:(
7516:P
7496:)
7493:x
7490:(
7487:q
7467:)
7464:x
7461:(
7458:p
7448:x
7430:)
7425:b
7421:x
7417:(
7414:y
7411:=
7406:b
7402:y
7381:)
7376:a
7372:x
7368:(
7365:y
7362:=
7357:a
7353:y
7328:y
7325:d
7320:)
7314:)
7311:y
7308:(
7299:q
7291:)
7288:y
7285:(
7276:p
7267:(
7257:)
7254:y
7251:(
7242:p
7232:b
7228:y
7220:a
7216:y
7207:=
7197:x
7194:d
7189:)
7182:|
7178:)
7175:x
7172:(
7166:x
7163:d
7158:y
7155:d
7148:|
7143:)
7140:)
7137:x
7134:(
7131:y
7128:(
7119:q
7110:|
7106:)
7103:x
7100:(
7094:x
7091:d
7086:y
7083:d
7076:|
7071:)
7068:)
7065:x
7062:(
7059:y
7056:(
7047:p
7038:(
7027:|
7023:)
7020:x
7017:(
7011:x
7008:d
7003:y
7000:d
6993:|
6989:)
6986:)
6983:x
6980:(
6977:y
6974:(
6965:p
6955:b
6951:x
6943:a
6939:x
6930:=
6920:x
6917:d
6912:)
6906:)
6903:x
6900:(
6897:q
6892:)
6889:x
6886:(
6883:p
6877:(
6867:)
6864:x
6861:(
6858:p
6851:b
6847:x
6839:a
6835:x
6826:=
6819:)
6816:Q
6810:P
6807:(
6798:D
6768:|
6764:)
6761:x
6758:(
6751:x
6748:d
6743:y
6740:d
6732:|
6711:x
6708:d
6704:|
6700:)
6697:x
6694:(
6687:x
6684:d
6679:y
6676:d
6668:|
6664:)
6661:y
6658:(
6649:q
6643:=
6640:y
6637:d
6633:)
6630:y
6627:(
6618:q
6612:=
6609:x
6606:d
6602:)
6599:x
6596:(
6593:q
6590:=
6587:)
6584:x
6581:d
6578:(
6575:Q
6555:x
6552:d
6547:|
6543:)
6540:x
6537:(
6530:x
6527:d
6522:y
6519:d
6511:|
6507:)
6504:)
6501:x
6498:(
6495:y
6492:(
6483:p
6477:=
6474:y
6471:d
6467:)
6464:y
6461:(
6452:p
6446:=
6443:x
6440:d
6436:)
6433:x
6430:(
6427:p
6424:=
6421:)
6418:x
6415:d
6412:(
6409:P
6389:)
6386:x
6383:(
6380:y
6370:x
6349:)
6346:Q
6340:P
6337:(
6328:D
6317:Q
6313:P
6305:P
6301:Q
6297:X
6293:x
6289:P
6285:Q
6263:)
6260:Q
6257:,
6254:P
6251:(
6247:H
6226:)
6223:P
6220:(
6216:H
6203:-
6172:)
6169:x
6166:(
6163:q
6160:=
6157:)
6154:x
6151:(
6148:p
6128:)
6125:x
6122:d
6119:(
6113:)
6110:x
6107:(
6104:q
6101:=
6098:)
6095:x
6092:d
6089:(
6086:Q
6066:)
6063:x
6060:d
6057:(
6051:)
6048:x
6045:(
6042:p
6039:=
6036:)
6033:x
6030:d
6027:(
6024:P
6000:Q
5997:=
5994:P
5971:)
5968:Q
5962:P
5959:(
5950:D
5925:,
5922:0
5916:)
5913:Q
5907:P
5904:(
5895:D
5822:)
5819:Q
5813:P
5810:(
5801:D
5780:)
5777:P
5774:(
5771:H
5751:)
5748:P
5745:(
5742:H
5736:)
5733:P
5730:,
5727:P
5724:(
5721:H
5697:)
5694:Q
5691:,
5688:P
5685:(
5682:H
5668:P
5664:Q
5646:)
5643:Q
5637:P
5634:(
5625:D
5611:P
5593:)
5590:P
5587:(
5583:H
5572:Q
5568:P
5550:)
5547:Q
5544:,
5541:P
5538:(
5534:H
5506:)
5503:P
5500:(
5496:H
5489:)
5486:Q
5483:,
5480:P
5477:(
5473:H
5469:=
5456:)
5453:x
5450:(
5447:p
5443:1
5432:)
5429:x
5426:(
5423:p
5416:X
5408:x
5394:)
5391:x
5388:(
5385:q
5381:1
5370:)
5367:x
5364:(
5361:p
5354:X
5346:x
5338:=
5331:)
5328:Q
5322:P
5319:(
5310:D
5288:P
5284:Q
5268:i
5264:x
5241:i
5226:X
5208:i
5196:2
5192:=
5189:)
5184:i
5180:x
5176:(
5173:q
5163:X
5147:i
5143:x
5091:D
5058:f
5000:)
4997:P
4991:Q
4988:(
4979:D
4958:)
4955:Q
4949:P
4946:(
4937:D
4891:P
4887:Q
4883:P
4875:Q
4871:P
4859:P
4855:Q
4851:P
4844:Q
4827:)
4824:Q
4818:P
4815:(
4806:D
4788:Q
4784:P
4776:Q
4772:P
4754:)
4751:Q
4745:P
4742:(
4733:D
4710:P
4706:Q
4702:P
4680:)
4677:Q
4671:P
4668:(
4659:D
4636:P
4632:Y
4618:)
4615:Y
4612:(
4609:Q
4597:)
4594:Y
4591:(
4588:P
4572:Y
4568:Q
4564:P
4522:)
4518:)
4515:5
4512:(
4503:6
4500:+
4497:)
4494:3
4491:(
4482:6
4476:)
4473:2
4470:(
4461:4
4454:(
4448:3
4445:1
4440:=
4429:)
4419:/
4415:4
4410:3
4406:/
4402:1
4396:(
4384:3
4381:1
4376:+
4372:)
4362:/
4353:3
4349:/
4345:1
4339:(
4327:3
4324:1
4319:+
4315:)
4305:/
4301:9
4296:3
4292:/
4288:1
4282:(
4270:3
4267:1
4262:=
4251:)
4245:)
4242:x
4239:(
4236:P
4231:)
4228:x
4225:(
4222:Q
4216:(
4206:)
4203:x
4200:(
4197:Q
4190:X
4182:x
4174:=
4167:)
4164:P
4158:Q
4155:(
4146:D
4114:,
4104:)
4100:)
4097:5
4094:(
4079:)
4076:3
4073:(
4061:+
4058:)
4055:2
4052:(
4039:(
4030:1
4025:=
4014:)
4008:3
4004:/
4000:1
3991:/
3987:4
3981:(
3966:4
3961:+
3957:)
3951:3
3947:/
3943:1
3934:/
3924:(
3904:+
3900:)
3894:3
3890:/
3886:1
3877:/
3873:9
3867:(
3852:9
3847:=
3836:)
3830:)
3827:x
3824:(
3821:Q
3816:)
3813:x
3810:(
3807:P
3801:(
3791:)
3788:x
3785:(
3782:P
3775:X
3767:x
3759:=
3752:)
3749:Q
3743:P
3740:(
3731:D
3700:e
3681:)
3678:P
3672:Q
3669:(
3660:D
3639:)
3636:Q
3630:P
3627:(
3618:D
3590:3
3587:1
3562:3
3559:1
3534:3
3531:1
3508:)
3505:x
3502:(
3499:Q
3470:4
3414:9
3391:)
3388:x
3385:(
3382:P
3366:1
3363:0
3359:x
3332:3
3328:/
3324:1
3321:=
3318:p
3298:}
3295:2
3292:,
3289:1
3286:,
3283:0
3280:{
3277:=
3272:X
3260:2
3256:1
3252:0
3239:=
3236:x
3226:Q
3209:=
3206:p
3186:2
3183:=
3180:N
3166:P
3162:Q
3158:P
3146:Q
3142:P
3134:Q
3128:P
3111:)
3108:Q
3102:P
3099:(
3090:D
3076:P
3065:Q
3046:P
3040:Q
3033:Q
3027:P
3023:Q
3019:P
2999:)
2996:Q
2990:P
2987:(
2978:D
2957:e
2907:)
2903:Q
2900:+
2897:P
2893:(
2887:2
2884:1
2879:=
2830:.
2824:)
2821:x
2813:d
2809:(
2799:)
2793:)
2790:x
2787:(
2784:q
2776:)
2773:x
2770:(
2767:p
2758:(
2745:)
2742:x
2739:(
2736:p
2729:X
2721:x
2713:=
2710:)
2707:Q
2701:P
2698:(
2689:D
2675:P
2671:Q
2641:Q
2637:P
2620:)
2617:x
2609:d
2605:(
2599:)
2596:x
2593:(
2590:q
2587:=
2584:)
2581:x
2573:d
2569:(
2566:Q
2540:)
2537:x
2529:d
2525:(
2519:)
2516:x
2513:(
2510:p
2507:=
2504:)
2501:x
2493:d
2489:(
2486:P
2473:q
2469:p
2453:X
2421:Q
2417:P
2396:,
2390:)
2387:x
2379:d
2375:(
2372:Q
2365:)
2359:)
2356:x
2348:d
2344:(
2341:Q
2336:)
2333:x
2325:d
2321:(
2318:P
2312:(
2296:)
2293:x
2285:d
2281:(
2278:Q
2273:)
2270:x
2262:d
2258:(
2255:P
2245:X
2237:x
2229:=
2226:)
2223:Q
2217:P
2214:(
2205:D
2187:Q
2183:P
2166:)
2163:x
2155:d
2151:(
2148:Q
2145:)
2142:x
2139:(
2136:r
2133:=
2130:)
2127:x
2119:d
2115:(
2112:P
2084:X
2069:r
2065:Q
2061:Q
2057:P
2033:)
2030:x
2022:d
2018:(
2015:Q
2007:)
2004:x
1996:d
1992:(
1989:P
1957:,
1951:)
1948:x
1940:d
1936:(
1933:P
1926:)
1920:)
1917:x
1909:d
1905:(
1902:Q
1897:)
1894:x
1886:d
1882:(
1879:P
1873:(
1856:X
1848:x
1840:=
1837:)
1834:Q
1828:P
1825:(
1816:D
1802:P
1798:Q
1794:Q
1786:P
1772:,
1764:X
1742:Q
1738:P
1731:Q
1727:P
1719:q
1715:p
1698:,
1692:x
1684:d
1676:)
1670:)
1667:x
1664:(
1661:q
1656:)
1653:x
1650:(
1647:p
1641:(
1628:)
1625:x
1622:(
1619:p
1598:=
1595:)
1592:Q
1586:P
1583:(
1574:D
1556:Q
1552:P
1535:.
1529:0
1526:=
1523:)
1520:x
1517:(
1508:x
1501:+
1497:0
1490:x
1459:)
1456:x
1453:(
1450:P
1419:X
1391:0
1385:)
1382:x
1379:(
1376:Q
1347:+
1334:,
1319:+
1306:(
1291:0
1288:=
1285:)
1282:x
1279:(
1276:P
1250:0
1247:=
1244:)
1241:x
1238:(
1235:Q
1222:x
1215:P
1211:Q
1207:P
1186:.
1179:)
1173:)
1170:x
1167:(
1164:P
1156:)
1153:x
1150:(
1147:Q
1138:(
1125:)
1122:x
1119:(
1116:P
1109:X
1101:x
1090:=
1087:)
1084:Q
1078:P
1075:(
1066:D
1039:,
1032:)
1026:)
1023:x
1020:(
1017:Q
1009:)
1006:x
1003:(
1000:P
991:(
978:)
975:x
972:(
969:P
962:X
954:x
946:=
943:)
940:Q
934:P
931:(
922:D
908:P
904:Q
890:,
882:X
863:Q
859:P
810:)
807:1
804::
801:2
798:(
795:I
792:+
789:)
786:2
783::
780:1
777:(
774:I
771:=
768:)
765:2
762:,
759:1
756:(
753:J
731:2
704:1
679:)
676:2
673::
670:1
667:(
664:I
642:2
634:,
629:1
602:2
598:H
594:,
589:1
585:H
562:2
554:,
549:1
522:1
495:2
491:H
468:1
464:H
425:Q
421:P
417:P
413:Q
409:P
395:)
392:Q
386:P
383:(
374:D
363:P
359:Q
355:P
351:Q
347:P
271:P
267:P
263:Q
251:Q
247:P
226:.
222:)
216:)
213:x
210:(
207:Q
199:)
196:x
193:(
190:P
181:(
168:)
165:x
162:(
159:P
152:X
144:x
136:=
133:)
130:Q
124:P
121:(
112:D
98:Q
94:P
73:)
70:Q
64:P
61:(
52:D
27:(
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.