119:
22:
3648:
1078:
An example: The weather forecast broadcast is: "Tonight's forecast: Dark. Continued darkness until widely scattered light in the morning." This message contains almost no information. However, a forecast of a snowstorm would certainly contain information since such does not happen every evening.
2897:
828:
It can also be shown that a compound message of two (or more) unrelated messages would have a quantity of information that is the sum of the measures of information of each message individually. That can be derived using this definition by considering a compound message
1308:
3969:
3250:
5002:
4357:
2100:
1839:
1613:
1073:
2406:
704:
3403:
4852:
804:. The base of the logarithm only affects a scaling factor and, consequently, the units in which the measured information content is expressed. If the logarithm is base 2, the measure of information is expressed in units of
2632:
3738:
2993:
824:
with certainty, for instance, through a back-channel) provide no information, as the above equation indicates. Infrequently occurring messages contain more information than more frequently occurring messages.
4608:
4193:
819:
Information from a source is gained by a recipient only if the recipient did not already have that information to begin with. Messages that convey information over a certain (P=1) event (or one which is
4715:
4490:
562:
2487:
1151:
3851:
1459:
3799:
762:
395:
1700:
925:
893:
999:
3110:
354:
284:
208:
315:
242:
4858:
1484:
1345:
1646:
1406:
488:
853:
5135:
5040:
5098:
5069:
4244:
2228:
4236:
4216:
4045:
4021:
4001:
3839:
3819:
3761:
3395:
3375:
3355:
3335:
3293:
3273:
3102:
3082:
3062:
3042:
2624:
2604:
2580:
2560:
2531:
2511:
2202:
2182:
2143:
2123:
1937:
1917:
1893:
1873:
1143:
1115:
802:
782:
600:
508:
168:
148:
3643:{\displaystyle \operatorname {I} (X;Y)=\sum _{y\in Y}p(y)\sum _{x\in X}{p(x|y)\log {\frac {p(x|y)}{p(x)}}}=\sum _{x,y}p(x,y)\log {\frac {p(x,y)}{p(x)\,p(y)}}.}
1945:
1717:
1492:
1004:
4374:: mutual information can be considered a statistic for assessing independence between a pair of variables, and has a well-specified asymptotic distribution.
2236:
608:
3104:
is the correct distribution, Kullback–Leibler divergence is the number of average additional bits per datum necessary for compression, or, mathematically,
4721:
2892:{\displaystyle \mathrm {H} (X|Y)=\mathbb {E} _{Y}\left=-\sum _{y\in Y}p(y)\sum _{x\in X}p(x|y)\log p(x|y)=\sum _{x,y}p(x,y)\log {\frac {p(y)}{p(x,y)}}.}
4400:. Although, in both cases, mutual information expresses the number of bits of information common to the two sources in question, the analogy does
3659:
855:
providing information regarding the values of two random variables M and N using a message which is the concatenation of the elementary messages
2915:
4496:
441:
as used in data-processing to refer to a binary value or stream regardless of its entropy (information content) Other units include the
86:
4389:
4053:
39:
3317:. This is a measure of how much information can be obtained about one random variable by observing another. The mutual information of
58:
1083:. The amount of information in a forecast of snow for a location where it never snows (impossible event) is the highest (infinity).
4238:. This is often recalculated as the divergence from the product of the marginal distributions to the actual joint distribution:
65:
4614:
1355:
An important property of entropy is that it is maximized when all the messages in the message space are equiprobable (e.g.
72:
4407:
The differential analogies of entropy, joint entropy, conditional entropy, and mutual information are defined as follows:
4413:
1303:{\displaystyle \mathrm {H} (M)=\mathbb {E} \left=\sum _{m\in M}p(m)\operatorname {I} (m)=-\sum _{m\in M}p(m)\log p(m)}
513:
54:
105:
3964:{\displaystyle \operatorname {I} (X;Y)=\operatorname {I} (Y;X)=\mathrm {H} (X)+\mathrm {H} (Y)-\mathrm {H} (X,Y).\,}
2414:
1079:
There would be an even greater amount of information in an accurate forecast of snow for a warm location, such as
3976:
3005:
118:
43:
1411:
4371:
3766:
712:
362:
1651:
4397:
898:
866:
211:
5219:
3245:{\displaystyle D_{\mathrm {KL} }{\bigl (}p(X)\|q(X){\bigr )}=\sum _{x\in X}p(x)\log {\frac {p(x)}{q(x)}}.}
942:
5241:
4997:{\displaystyle \operatorname {I} (X;Y)=\int _{Y}\int _{X}f(x,y)\log {\frac {f(x,y)}{f(x)f(y)}}\,dx\,dy}
437:. Although "bit" is more frequently used in place of "shannon", its name is not distinguished from the
320:
250:
176:
79:
5189:
4393:
289:
216:
4367:
3022:
2146:
936:
1467:
1319:
2903:
2490:
1707:
1618:
813:
32:
1358:
4198:
In other words, this is a measure of how much, on the average, the probability distribution on
463:
832:
4363:
3980:
5103:
5010:
4352:{\displaystyle \operatorname {I} (X;Y)=D_{\mathrm {KL} }{\bigl (}p(X,Y)\|p(X)p(Y){\bigr )}.}
5170:
4383:
3296:
418:
5074:
5045:
8:
2538:
2207:
2095:{\displaystyle \mathrm {H} (X,Y)=\mathbb {E} _{X,Y}\left=-\sum _{x,y}p(x,y)\log p(x,y)\,}
1093:
422:
245:
123:
5174:
1834:{\displaystyle \mathrm {H} _{\mbox{b}}(p)=\mathrm {H} (p,1-p)=-p\log p-(1-p)\log(1-p)\,}
1608:{\displaystyle \mathrm {H} (p_{1},p_{2},\ldots ,p_{k})=-\sum _{i=1}^{k}p_{i}\log p_{i},}
1068:{\displaystyle \operatorname {I} (m\&n)=\operatorname {I} (m)+\operatorname {I} (n)}
1001:(the definition of statistical independence) it is clear from the above definition that
5146:
4221:
4201:
4030:
4006:
3986:
3842:
3824:
3804:
3746:
3380:
3360:
3340:
3320:
3309:
3278:
3258:
3087:
3067:
3047:
3027:
2609:
2589:
2565:
2545:
2516:
2496:
2401:{\displaystyle \mathrm {H} (X|y)=\mathbb {E} _{\left}=-\sum _{x\in X}p(x|y)\log p(x|y)}
2187:
2167:
2128:
2108:
1922:
1902:
1896:
1878:
1858:
1128:
1100:
787:
767:
699:{\displaystyle \operatorname {I} (m)=\log \left({\frac {1}{p(m)}}\right)=-\log(p(m))\,}
585:
493:
406:
402:
357:
153:
133:
5205:
3307:
It turns out that one of the most useful and important measures of information is the
4024:
446:
5190:"Some inequalities satisfied by the quantities of information of Fisher and Shannon"
5201:
574:
454:
434:
4847:{\displaystyle h(X|Y)=\int _{Y}\int _{X}f(x,y)\log {\frac {f(y)}{f(x,y)}}\,dx\,dy}
4404:
imply identical properties; for example, differential entropy may be negative.
1348:
1122:
805:
430:
127:
5235:
5220:"Three approaches to the definition of the concept "quantity of information""
2153:
1851:
171:
3733:{\displaystyle \operatorname {I} (X;Y)=\mathrm {H} (X)-\mathrm {H} (X|Y).\,}
417:. The choice of logarithmic base in the following formulae determines the
2149:, then the joint entropy is simply the sum of their individual entropies.
4388:
The basic measures of discrete entropy have been extended by analogy to
3357:(which represents conceptually the average amount of information about
442:
410:
2988:{\displaystyle \mathrm {H} (X|Y)=\mathrm {H} (X,Y)-\mathrm {H} (Y).\,}
21:
4603:{\displaystyle h(X,Y)=-\int _{Y}\int _{X}f(x,y)\log f(x,y)\,dx\,dy}
4188:{\displaystyle \operatorname {I} (X;Y)=\mathbb {E} _{p(y)}\left.}
2998:
1121:
one has about which message will be chosen. It is defined as the
450:
1486:
is expressed in terms of the probabilities of the distribution:
812:
in other contexts is rather defined as a "binary digit", whose
1080:
3084:
is the distribution underlying some data, when, in reality,
572:
Shannon derived a measure of information content called the
130:'s basic quantities of information for correlated variables
935:
each depend only on M and N, and the processes M and N are
425:
that is used. The most common unit of information is the
2152:(Note: The joint entropy should not be confused with the
809:
784:
is chosen from all possible choices in the message space
490:
is considered by convention to be equal to zero whenever
438:
3302:
5167:
Information theory, inferences, and learning algorithms
4710:{\displaystyle h(X|y)=-\int _{X}f(x|y)\log f(x|y)\,dx}
1728:
5106:
5077:
5048:
5013:
4861:
4724:
4617:
4499:
4416:
4247:
4224:
4204:
4056:
4033:
4009:
3989:
3854:
3827:
3807:
3769:
3749:
3662:
3406:
3383:
3363:
3343:
3323:
3281:
3261:
3113:
3090:
3070:
3050:
3030:
2918:
2909:
A basic property of the conditional entropy is that:
2635:
2612:
2592:
2568:
2548:
2519:
2499:
2417:
2239:
2210:
2190:
2170:
2159:
2131:
2111:
1948:
1925:
1905:
1881:
1861:
1720:
1654:
1621:
1495:
1470:
1414:
1361:
1322:
1154:
1131:
1103:
1007:
945:
901:
869:
835:
790:
770:
715:
611:
588:
516:
496:
466:
365:
323:
292:
253:
219:
179:
156:
136:
126:
showing additive and subtractive relationships among
3653:
A basic property of the mutual information is that:
3021:) is a way of comparing two distributions, a "true"
3975:Mutual information can be expressed as the average
46:. Unsourced material may be challenged and removed.
5129:
5092:
5063:
5034:
4996:
4846:
4709:
4602:
4484:
4351:
4230:
4210:
4187:
4039:
4015:
3995:
3963:
3833:
3813:
3793:
3755:
3732:
3642:
3389:
3369:
3349:
3329:
3287:
3267:
3244:
3096:
3076:
3056:
3036:
2987:
2891:
2618:
2598:
2574:
2554:
2525:
2505:
2481:
2400:
2222:
2196:
2176:
2137:
2117:
2094:
1931:
1911:
1887:
1867:
1833:
1694:
1640:
1607:
1478:
1453:
1400:
1339:
1302:
1137:
1109:
1067:
993:
919:
887:
847:
796:
776:
756:
698:
594:
556:
502:
482:
389:
348:
309:
278:
236:
202:
162:
142:
863:, each of whose information content are given by
210:. The circle on the left (red and violet) is the
5233:
4485:{\displaystyle h(X)=-\int _{X}f(x)\log f(x)\,dx}
3064:. If we compress data in a manner that assumes
557:{\displaystyle \lim _{p\rightarrow 0+}p\log p=0}
518:
286:. The circle on the right (blue and violet) is
2999:Kullback–Leibler divergence (information gain)
2164:Given a particular value of a random variable
5164:
4366:in the context of contingency tables and the
4362:Mutual information is closely related to the
4341:
4289:
4172:
4124:
3165:
3131:
2482:{\displaystyle p(x|y)={\frac {p(x,y)}{p(y)}}}
4392:spaces by replacing sums with integrals and
4312:
4155:
3148:
3044:, and an arbitrary probability distribution
170:. The area contained by both circles is the
460:In what follows, an expression of the form
4987:
4980:
4837:
4830:
4700:
4593:
4586:
4475:
4218:will change if we are given the value of
4083:
3960:
3729:
3621:
2984:
2663:
2267:
2091:
1974:
1830:
1705:An important special case of this is the
1324:
1173:
695:
479:
106:Learn how and when to remove this message
3255:It is in some sense the "distance" from
1454:{\displaystyle \mathrm {H} (M)=\log |M|}
413:, and measures information with several
117:
4377:
3794:{\displaystyle \operatorname {I} (X;Y)}
757:{\displaystyle p(m)=\mathrm {Pr} (M=m)}
390:{\displaystyle \operatorname {I} (X;Y)}
5234:
1695:{\displaystyle \sum _{i=1}^{k}p_{i}=1}
3303:Mutual information (transinformation)
920:{\displaystyle \operatorname {I} (n)}
888:{\displaystyle \operatorname {I} (m)}
5187:
5100:are the marginal distributions, and
994:{\displaystyle P(m\&n)=P(m)P(n)}
510:is zero. This is justified because
44:adding citations to reliable sources
15:
567:
13:
4862:
4281:
4278:
4248:
4116:
4113:
4057:
3981:posterior probability distribution
3938:
3921:
3904:
3879:
3855:
3770:
3705:
3688:
3663:
3407:
3123:
3120:
2968:
2945:
2920:
2679:
2637:
2241:
2160:Conditional entropy (equivocation)
1950:
1748:
1723:
1497:
1472:
1416:
1233:
1182:
1156:
1050:
1032:
1020:
1008:
955:
902:
870:
839:
735:
732:
612:
403:mathematical theory of information
366:
325:
294:
255:
221:
181:
14:
5253:
5137:is the conditional distribution.
1895:is defined as the entropy of the
1855:of two discrete random variables
349:{\displaystyle \mathrm {H} (Y|X)}
279:{\displaystyle \mathrm {H} (X|Y)}
203:{\displaystyle \mathrm {H} (X,Y)}
3377:that can be gained by observing
3299:due to its not being symmetric.
1844:
764:is the probability that message
20:
5042:is the joint density function,
808:or more often simply "bits" (a
310:{\displaystyle \mathrm {H} (Y)}
237:{\displaystyle \mathrm {H} (X)}
31:needs additional citations for
5212:
5181:
5158:
5124:
5117:
5110:
5087:
5081:
5058:
5052:
5029:
5017:
4974:
4968:
4962:
4956:
4948:
4936:
4921:
4909:
4880:
4868:
4824:
4812:
4804:
4798:
4783:
4771:
4742:
4735:
4728:
4697:
4690:
4683:
4671:
4664:
4657:
4635:
4628:
4621:
4583:
4571:
4559:
4547:
4515:
4503:
4472:
4466:
4454:
4448:
4426:
4420:
4336:
4330:
4324:
4318:
4309:
4297:
4266:
4254:
4167:
4161:
4152:
4139:
4132:
4097:
4091:
4075:
4063:
3954:
3942:
3931:
3925:
3914:
3908:
3897:
3885:
3873:
3861:
3788:
3776:
3723:
3716:
3709:
3698:
3692:
3681:
3669:
3631:
3625:
3618:
3612:
3604:
3592:
3577:
3565:
3536:
3530:
3522:
3515:
3508:
3493:
3486:
3479:
3456:
3450:
3425:
3413:
3233:
3227:
3219:
3213:
3198:
3192:
3160:
3154:
3145:
3139:
2978:
2972:
2961:
2949:
2938:
2931:
2924:
2880:
2868:
2860:
2854:
2839:
2827:
2802:
2795:
2788:
2776:
2769:
2762:
2740:
2734:
2692:
2655:
2648:
2641:
2473:
2467:
2459:
2447:
2435:
2428:
2421:
2395:
2388:
2381:
2369:
2362:
2355:
2327:
2324:
2317:
2310:
2295:
2281:
2259:
2252:
2245:
2156:, despite similar notations.)
2088:
2076:
2064:
2052:
2019:
2007:
1966:
1954:
1827:
1815:
1806:
1794:
1770:
1752:
1741:
1735:
1546:
1501:
1447:
1439:
1426:
1420:
1394:
1386:
1371:
1365:
1334:
1328:
1297:
1291:
1279:
1273:
1245:
1239:
1230:
1224:
1194:
1188:
1166:
1160:
1125:self-information of a message
1117:is a measure of the amount of
1062:
1056:
1044:
1038:
1026:
1014:
988:
982:
976:
970:
961:
949:
927:respectively. If the messages
914:
908:
882:
876:
751:
739:
725:
719:
692:
689:
683:
677:
655:
649:
624:
618:
525:
384:
372:
343:
336:
329:
304:
298:
273:
266:
259:
231:
225:
197:
185:
1:
5206:10.1016/S0019-9958(59)90348-1
5152:
4398:probability density functions
2184:, the conditional entropy of
3763:, we can save an average of
3295:, although it is not a true
1479:{\displaystyle \mathrm {H} }
1340:{\displaystyle \mathbb {E} }
1097:of a discrete message space
7:
5140:
3977:Kullback–Leibler divergence
3006:Kullback–Leibler divergence
1641:{\displaystyle p_{i}\geq 0}
814:average information content
55:"Quantities of information"
10:
5258:
4394:probability mass functions
4381:
3979:(information gain) of the
1401:{\displaystyle p(m)=1/|M|}
1086:
564:for any logarithmic base.
453:, based on the base 10 or
4364:log-likelihood ratio test
3841:. Mutual information is
2906:from probability theory.
1145:from that message space:
483:{\displaystyle p\log p\,}
415:quantities of information
244:, with the red being the
4368:multinomial distribution
3821:compared to not knowing
3023:probability distribution
848:{\displaystyle m\&n}
429:, or more correctly the
5194:Information and Control
2904:conditional expectation
2491:conditional probability
1708:binary entropy function
1464:Sometimes the function
816:is at most 1 shannon).
5165:D.J.C. Mackay (2003).
5131:
5130:{\displaystyle f(x|y)}
5094:
5065:
5036:
5035:{\displaystyle f(x,y)}
4998:
4848:
4711:
4604:
4486:
4353:
4232:
4212:
4189:
4041:
4017:
3997:
3965:
3835:
3815:
3795:
3757:
3734:
3644:
3391:
3371:
3351:
3331:
3289:
3269:
3246:
3098:
3078:
3058:
3038:
3011:information divergence
2989:
2893:
2620:
2600:
2576:
2556:
2527:
2507:
2483:
2402:
2224:
2198:
2178:
2139:
2119:
2096:
1933:
1913:
1889:
1869:
1835:
1696:
1675:
1642:
1609:
1575:
1480:
1455:
1402:
1341:
1304:
1139:
1111:
1069:
995:
921:
889:
849:
798:
778:
758:
700:
596:
558:
504:
484:
398:
391:
350:
317:, with the blue being
311:
280:
238:
204:
164:
144:
5132:
5095:
5066:
5037:
4999:
4849:
4712:
4605:
4487:
4354:
4233:
4213:
4190:
4042:
4018:
3998:
3966:
3836:
3816:
3796:
3758:
3735:
3645:
3392:
3372:
3352:
3332:
3290:
3270:
3247:
3099:
3079:
3059:
3039:
2990:
2894:
2621:
2601:
2577:
2557:
2528:
2508:
2484:
2403:
2225:
2199:
2179:
2140:
2120:
2097:
1934:
1914:
1890:
1870:
1836:
1697:
1655:
1643:
1610:
1555:
1481:
1456:
1403:
1342:
1305:
1140:
1112:
1070:
996:
922:
890:
850:
799:
779:
759:
701:
597:
559:
505:
485:
392:
351:
312:
281:
239:
205:
165:
145:
121:
5104:
5093:{\displaystyle f(y)}
5075:
5064:{\displaystyle f(x)}
5046:
5011:
4859:
4722:
4615:
4497:
4414:
4384:Differential entropy
4378:Differential entropy
4245:
4222:
4202:
4054:
4031:
4007:
3987:
3852:
3825:
3805:
3767:
3747:
3660:
3404:
3381:
3361:
3341:
3321:
3279:
3259:
3111:
3088:
3068:
3048:
3028:
2916:
2633:
2610:
2590:
2566:
2546:
2517:
2497:
2415:
2237:
2208:
2188:
2168:
2129:
2109:
1946:
1923:
1903:
1879:
1859:
1718:
1652:
1619:
1493:
1468:
1412:
1359:
1320:
1152:
1129:
1101:
1005:
943:
899:
867:
833:
788:
768:
713:
609:
586:
514:
494:
464:
363:
356:. The violet is the
321:
290:
251:
217:
177:
154:
134:
40:improve this article
5188:Stam, A.J. (1959).
5175:2003itil.book.....M
4003:given the value of
2539:conditional entropy
2223:{\displaystyle Y=y}
423:information entropy
246:conditional entropy
124:information diagram
5242:Information theory
5147:Information theory
5127:
5090:
5061:
5032:
4994:
4844:
4707:
4600:
4482:
4349:
4228:
4208:
4185:
4037:
4025:prior distribution
4013:
3993:
3961:
3831:
3811:
3791:
3753:
3730:
3640:
3561:
3474:
3446:
3387:
3367:
3347:
3327:
3310:mutual information
3285:
3265:
3242:
3188:
3094:
3074:
3054:
3034:
2985:
2889:
2823:
2758:
2730:
2626:is then given by:
2616:
2596:
2582:, also called the
2572:
2552:
2523:
2503:
2479:
2398:
2351:
2220:
2194:
2174:
2135:
2115:
2092:
2048:
1929:
1909:
1897:joint distribution
1885:
1865:
1831:
1732:
1692:
1638:
1605:
1476:
1451:
1398:
1337:
1300:
1269:
1220:
1135:
1107:
1065:
991:
917:
885:
845:
794:
774:
754:
696:
592:
554:
535:
500:
480:
407:probability theory
399:
387:
358:mutual information
346:
307:
276:
234:
212:individual entropy
200:
160:
140:
4978:
4828:
4231:{\displaystyle Y}
4211:{\displaystyle X}
4040:{\displaystyle X}
4016:{\displaystyle Y}
3996:{\displaystyle X}
3834:{\displaystyle Y}
3814:{\displaystyle X}
3801:bits in encoding
3756:{\displaystyle Y}
3743:That is, knowing
3635:
3546:
3540:
3459:
3431:
3390:{\displaystyle Y}
3370:{\displaystyle X}
3350:{\displaystyle Y}
3330:{\displaystyle X}
3288:{\displaystyle p}
3268:{\displaystyle q}
3237:
3173:
3097:{\displaystyle p}
3077:{\displaystyle q}
3057:{\displaystyle q}
3037:{\displaystyle p}
2884:
2808:
2743:
2715:
2619:{\displaystyle Y}
2599:{\displaystyle X}
2575:{\displaystyle Y}
2555:{\displaystyle X}
2526:{\displaystyle y}
2506:{\displaystyle x}
2477:
2336:
2197:{\displaystyle X}
2177:{\displaystyle Y}
2138:{\displaystyle Y}
2118:{\displaystyle X}
2033:
1932:{\displaystyle Y}
1912:{\displaystyle X}
1888:{\displaystyle Y}
1868:{\displaystyle X}
1731:
1254:
1205:
1138:{\displaystyle m}
1110:{\displaystyle M}
797:{\displaystyle M}
777:{\displaystyle m}
659:
595:{\displaystyle m}
517:
503:{\displaystyle p}
447:natural logarithm
163:{\displaystyle Y}
143:{\displaystyle X}
116:
115:
108:
90:
5249:
5227:
5226:
5224:
5216:
5210:
5209:
5185:
5179:
5178:
5162:
5136:
5134:
5133:
5128:
5120:
5099:
5097:
5096:
5091:
5070:
5068:
5067:
5062:
5041:
5039:
5038:
5033:
5003:
5001:
5000:
4995:
4979:
4977:
4951:
4931:
4905:
4904:
4895:
4894:
4853:
4851:
4850:
4845:
4829:
4827:
4807:
4793:
4767:
4766:
4757:
4756:
4738:
4716:
4714:
4713:
4708:
4693:
4667:
4653:
4652:
4631:
4609:
4607:
4606:
4601:
4543:
4542:
4533:
4532:
4491:
4489:
4488:
4483:
4444:
4443:
4372:Pearson's χ test
4358:
4356:
4355:
4350:
4345:
4344:
4293:
4292:
4286:
4285:
4284:
4237:
4235:
4234:
4229:
4217:
4215:
4214:
4209:
4194:
4192:
4191:
4186:
4181:
4177:
4176:
4175:
4142:
4128:
4127:
4121:
4120:
4119:
4101:
4100:
4086:
4046:
4044:
4043:
4038:
4022:
4020:
4019:
4014:
4002:
4000:
3999:
3994:
3970:
3968:
3967:
3962:
3941:
3924:
3907:
3840:
3838:
3837:
3832:
3820:
3818:
3817:
3812:
3800:
3798:
3797:
3792:
3762:
3760:
3759:
3754:
3739:
3737:
3736:
3731:
3719:
3708:
3691:
3649:
3647:
3646:
3641:
3636:
3634:
3607:
3587:
3560:
3542:
3541:
3539:
3525:
3518:
3503:
3489:
3473:
3445:
3396:
3394:
3393:
3388:
3376:
3374:
3373:
3368:
3356:
3354:
3353:
3348:
3336:
3334:
3333:
3328:
3315:transinformation
3294:
3292:
3291:
3286:
3274:
3272:
3271:
3266:
3251:
3249:
3248:
3243:
3238:
3236:
3222:
3208:
3187:
3169:
3168:
3135:
3134:
3128:
3127:
3126:
3103:
3101:
3100:
3095:
3083:
3081:
3080:
3075:
3063:
3061:
3060:
3055:
3043:
3041:
3040:
3035:
3019:relative entropy
3015:information gain
2994:
2992:
2991:
2986:
2971:
2948:
2934:
2923:
2898:
2896:
2895:
2890:
2885:
2883:
2863:
2849:
2822:
2798:
2772:
2757:
2729:
2708:
2704:
2703:
2699:
2695:
2682:
2672:
2671:
2666:
2651:
2640:
2625:
2623:
2622:
2617:
2605:
2603:
2602:
2597:
2581:
2579:
2578:
2573:
2561:
2559:
2558:
2553:
2532:
2530:
2529:
2524:
2512:
2510:
2509:
2504:
2488:
2486:
2485:
2480:
2478:
2476:
2462:
2442:
2431:
2407:
2405:
2404:
2399:
2391:
2365:
2350:
2320:
2294:
2293:
2292:
2288:
2284:
2270:
2255:
2244:
2229:
2227:
2226:
2221:
2203:
2201:
2200:
2195:
2183:
2181:
2180:
2175:
2144:
2142:
2141:
2136:
2124:
2122:
2121:
2116:
2101:
2099:
2098:
2093:
2047:
2026:
2022:
1989:
1988:
1977:
1953:
1938:
1936:
1935:
1930:
1918:
1916:
1915:
1910:
1894:
1892:
1891:
1886:
1874:
1872:
1871:
1866:
1840:
1838:
1837:
1832:
1751:
1734:
1733:
1729:
1726:
1701:
1699:
1698:
1693:
1685:
1684:
1674:
1669:
1647:
1645:
1644:
1639:
1631:
1630:
1614:
1612:
1611:
1606:
1601:
1600:
1585:
1584:
1574:
1569:
1545:
1544:
1526:
1525:
1513:
1512:
1500:
1485:
1483:
1482:
1477:
1475:
1460:
1458:
1457:
1452:
1450:
1442:
1419:
1408:). In this case
1407:
1405:
1404:
1399:
1397:
1389:
1384:
1346:
1344:
1343:
1338:
1327:
1309:
1307:
1306:
1301:
1268:
1219:
1201:
1197:
1176:
1159:
1144:
1142:
1141:
1136:
1116:
1114:
1113:
1108:
1074:
1072:
1071:
1066:
1000:
998:
997:
992:
926:
924:
923:
918:
894:
892:
891:
886:
854:
852:
851:
846:
803:
801:
800:
795:
783:
781:
780:
775:
763:
761:
760:
755:
738:
705:
703:
702:
697:
664:
660:
658:
641:
601:
599:
598:
593:
575:self-information
568:Self-information
563:
561:
560:
555:
534:
509:
507:
506:
501:
489:
487:
486:
481:
455:common logarithm
435:binary logarithm
396:
394:
393:
388:
355:
353:
352:
347:
339:
328:
316:
314:
313:
308:
297:
285:
283:
282:
277:
269:
258:
243:
241:
240:
235:
224:
209:
207:
206:
201:
184:
169:
167:
166:
161:
149:
147:
146:
141:
111:
104:
100:
97:
91:
89:
48:
24:
16:
5257:
5256:
5252:
5251:
5250:
5248:
5247:
5246:
5232:
5231:
5230:
5222:
5218:
5217:
5213:
5186:
5182:
5163:
5159:
5155:
5143:
5116:
5105:
5102:
5101:
5076:
5073:
5072:
5047:
5044:
5043:
5012:
5009:
5008:
4952:
4932:
4930:
4900:
4896:
4890:
4886:
4860:
4857:
4856:
4808:
4794:
4792:
4762:
4758:
4752:
4748:
4734:
4723:
4720:
4719:
4689:
4663:
4648:
4644:
4627:
4616:
4613:
4612:
4538:
4534:
4528:
4524:
4498:
4495:
4494:
4439:
4435:
4415:
4412:
4411:
4386:
4380:
4340:
4339:
4288:
4287:
4277:
4276:
4272:
4246:
4243:
4242:
4223:
4220:
4219:
4203:
4200:
4199:
4171:
4170:
4138:
4123:
4122:
4112:
4111:
4107:
4106:
4102:
4087:
4082:
4081:
4055:
4052:
4051:
4032:
4029:
4028:
4008:
4005:
4004:
3988:
3985:
3984:
3974:
3937:
3920:
3903:
3853:
3850:
3849:
3826:
3823:
3822:
3806:
3803:
3802:
3768:
3765:
3764:
3748:
3745:
3744:
3715:
3704:
3687:
3661:
3658:
3657:
3608:
3588:
3586:
3550:
3526:
3514:
3504:
3502:
3485:
3475:
3463:
3435:
3405:
3402:
3401:
3397:) is given by:
3382:
3379:
3378:
3362:
3359:
3358:
3342:
3339:
3338:
3322:
3319:
3318:
3305:
3280:
3277:
3276:
3260:
3257:
3256:
3223:
3209:
3207:
3177:
3164:
3163:
3130:
3129:
3119:
3118:
3114:
3112:
3109:
3108:
3089:
3086:
3085:
3069:
3066:
3065:
3049:
3046:
3045:
3029:
3026:
3025:
3001:
2967:
2944:
2930:
2919:
2917:
2914:
2913:
2864:
2850:
2848:
2812:
2794:
2768:
2747:
2719:
2691:
2687:
2683:
2678:
2677:
2673:
2667:
2662:
2661:
2647:
2636:
2634:
2631:
2630:
2611:
2608:
2607:
2591:
2588:
2587:
2567:
2564:
2563:
2547:
2544:
2543:
2518:
2515:
2514:
2498:
2495:
2494:
2463:
2443:
2441:
2427:
2416:
2413:
2412:
2387:
2361:
2340:
2316:
2280:
2276:
2272:
2271:
2266:
2265:
2251:
2240:
2238:
2235:
2234:
2230:is defined as:
2209:
2206:
2205:
2189:
2186:
2185:
2169:
2166:
2165:
2162:
2130:
2127:
2126:
2110:
2107:
2106:
2037:
1994:
1990:
1978:
1973:
1972:
1949:
1947:
1944:
1943:
1924:
1921:
1920:
1904:
1901:
1900:
1880:
1877:
1876:
1860:
1857:
1856:
1847:
1747:
1727:
1722:
1721:
1719:
1716:
1715:
1680:
1676:
1670:
1659:
1653:
1650:
1649:
1626:
1622:
1620:
1617:
1616:
1596:
1592:
1580:
1576:
1570:
1559:
1540:
1536:
1521:
1517:
1508:
1504:
1496:
1494:
1491:
1490:
1471:
1469:
1466:
1465:
1446:
1438:
1415:
1413:
1410:
1409:
1393:
1385:
1380:
1360:
1357:
1356:
1323:
1321:
1318:
1317:
1258:
1209:
1181:
1177:
1172:
1155:
1153:
1150:
1149:
1130:
1127:
1126:
1102:
1099:
1098:
1089:
1006:
1003:
1002:
944:
941:
940:
900:
897:
896:
868:
865:
864:
834:
831:
830:
789:
786:
785:
769:
766:
765:
731:
714:
711:
710:
645:
640:
636:
610:
607:
606:
587:
584:
583:
570:
521:
515:
512:
511:
495:
492:
491:
465:
462:
461:
445:, based on the
433:, based on the
364:
361:
360:
335:
324:
322:
319:
318:
293:
291:
288:
287:
265:
254:
252:
249:
248:
220:
218:
215:
214:
180:
178:
175:
174:
155:
152:
151:
135:
132:
131:
112:
101:
95:
92:
49:
47:
37:
25:
12:
11:
5:
5255:
5245:
5244:
5229:
5228:
5211:
5200:(2): 101–112.
5180:
5156:
5154:
5151:
5150:
5149:
5142:
5139:
5126:
5123:
5119:
5115:
5112:
5109:
5089:
5086:
5083:
5080:
5060:
5057:
5054:
5051:
5031:
5028:
5025:
5022:
5019:
5016:
5005:
5004:
4993:
4990:
4986:
4983:
4976:
4973:
4970:
4967:
4964:
4961:
4958:
4955:
4950:
4947:
4944:
4941:
4938:
4935:
4929:
4926:
4923:
4920:
4917:
4914:
4911:
4908:
4903:
4899:
4893:
4889:
4885:
4882:
4879:
4876:
4873:
4870:
4867:
4864:
4854:
4843:
4840:
4836:
4833:
4826:
4823:
4820:
4817:
4814:
4811:
4806:
4803:
4800:
4797:
4791:
4788:
4785:
4782:
4779:
4776:
4773:
4770:
4765:
4761:
4755:
4751:
4747:
4744:
4741:
4737:
4733:
4730:
4727:
4717:
4706:
4703:
4699:
4696:
4692:
4688:
4685:
4682:
4679:
4676:
4673:
4670:
4666:
4662:
4659:
4656:
4651:
4647:
4643:
4640:
4637:
4634:
4630:
4626:
4623:
4620:
4610:
4599:
4596:
4592:
4589:
4585:
4582:
4579:
4576:
4573:
4570:
4567:
4564:
4561:
4558:
4555:
4552:
4549:
4546:
4541:
4537:
4531:
4527:
4523:
4520:
4517:
4514:
4511:
4508:
4505:
4502:
4492:
4481:
4478:
4474:
4471:
4468:
4465:
4462:
4459:
4456:
4453:
4450:
4447:
4442:
4438:
4434:
4431:
4428:
4425:
4422:
4419:
4382:Main article:
4379:
4376:
4360:
4359:
4348:
4343:
4338:
4335:
4332:
4329:
4326:
4323:
4320:
4317:
4314:
4311:
4308:
4305:
4302:
4299:
4296:
4291:
4283:
4280:
4275:
4271:
4268:
4265:
4262:
4259:
4256:
4253:
4250:
4227:
4207:
4196:
4195:
4184:
4180:
4174:
4169:
4166:
4163:
4160:
4157:
4154:
4151:
4148:
4145:
4141:
4137:
4134:
4131:
4126:
4118:
4115:
4110:
4105:
4099:
4096:
4093:
4090:
4085:
4080:
4077:
4074:
4071:
4068:
4065:
4062:
4059:
4036:
4012:
3992:
3972:
3971:
3959:
3956:
3953:
3950:
3947:
3944:
3940:
3936:
3933:
3930:
3927:
3923:
3919:
3916:
3913:
3910:
3906:
3902:
3899:
3896:
3893:
3890:
3887:
3884:
3881:
3878:
3875:
3872:
3869:
3866:
3863:
3860:
3857:
3830:
3810:
3790:
3787:
3784:
3781:
3778:
3775:
3772:
3752:
3741:
3740:
3728:
3725:
3722:
3718:
3714:
3711:
3707:
3703:
3700:
3697:
3694:
3690:
3686:
3683:
3680:
3677:
3674:
3671:
3668:
3665:
3651:
3650:
3639:
3633:
3630:
3627:
3624:
3620:
3617:
3614:
3611:
3606:
3603:
3600:
3597:
3594:
3591:
3585:
3582:
3579:
3576:
3573:
3570:
3567:
3564:
3559:
3556:
3553:
3549:
3545:
3538:
3535:
3532:
3529:
3524:
3521:
3517:
3513:
3510:
3507:
3501:
3498:
3495:
3492:
3488:
3484:
3481:
3478:
3472:
3469:
3466:
3462:
3458:
3455:
3452:
3449:
3444:
3441:
3438:
3434:
3430:
3427:
3424:
3421:
3418:
3415:
3412:
3409:
3386:
3366:
3346:
3326:
3304:
3301:
3284:
3264:
3253:
3252:
3241:
3235:
3232:
3229:
3226:
3221:
3218:
3215:
3212:
3206:
3203:
3200:
3197:
3194:
3191:
3186:
3183:
3180:
3176:
3172:
3167:
3162:
3159:
3156:
3153:
3150:
3147:
3144:
3141:
3138:
3133:
3125:
3122:
3117:
3093:
3073:
3053:
3033:
3000:
2997:
2996:
2995:
2983:
2980:
2977:
2974:
2970:
2966:
2963:
2960:
2957:
2954:
2951:
2947:
2943:
2940:
2937:
2933:
2929:
2926:
2922:
2902:This uses the
2900:
2899:
2888:
2882:
2879:
2876:
2873:
2870:
2867:
2862:
2859:
2856:
2853:
2847:
2844:
2841:
2838:
2835:
2832:
2829:
2826:
2821:
2818:
2815:
2811:
2807:
2804:
2801:
2797:
2793:
2790:
2787:
2784:
2781:
2778:
2775:
2771:
2767:
2764:
2761:
2756:
2753:
2750:
2746:
2742:
2739:
2736:
2733:
2728:
2725:
2722:
2718:
2714:
2711:
2707:
2702:
2698:
2694:
2690:
2686:
2681:
2676:
2670:
2665:
2660:
2657:
2654:
2650:
2646:
2643:
2639:
2615:
2595:
2571:
2551:
2522:
2502:
2475:
2472:
2469:
2466:
2461:
2458:
2455:
2452:
2449:
2446:
2440:
2437:
2434:
2430:
2426:
2423:
2420:
2409:
2408:
2397:
2394:
2390:
2386:
2383:
2380:
2377:
2374:
2371:
2368:
2364:
2360:
2357:
2354:
2349:
2346:
2343:
2339:
2335:
2332:
2329:
2326:
2323:
2319:
2315:
2312:
2309:
2306:
2303:
2300:
2297:
2291:
2287:
2283:
2279:
2275:
2269:
2264:
2261:
2258:
2254:
2250:
2247:
2243:
2219:
2216:
2213:
2193:
2173:
2161:
2158:
2134:
2114:
2103:
2102:
2090:
2087:
2084:
2081:
2078:
2075:
2072:
2069:
2066:
2063:
2060:
2057:
2054:
2051:
2046:
2043:
2040:
2036:
2032:
2029:
2025:
2021:
2018:
2015:
2012:
2009:
2006:
2003:
2000:
1997:
1993:
1987:
1984:
1981:
1976:
1971:
1968:
1965:
1962:
1959:
1956:
1952:
1928:
1908:
1884:
1864:
1846:
1843:
1842:
1841:
1829:
1826:
1823:
1820:
1817:
1814:
1811:
1808:
1805:
1802:
1799:
1796:
1793:
1790:
1787:
1784:
1781:
1778:
1775:
1772:
1769:
1766:
1763:
1760:
1757:
1754:
1750:
1746:
1743:
1740:
1737:
1725:
1703:
1702:
1691:
1688:
1683:
1679:
1673:
1668:
1665:
1662:
1658:
1637:
1634:
1629:
1625:
1604:
1599:
1595:
1591:
1588:
1583:
1579:
1573:
1568:
1565:
1562:
1558:
1554:
1551:
1548:
1543:
1539:
1535:
1532:
1529:
1524:
1520:
1516:
1511:
1507:
1503:
1499:
1474:
1449:
1445:
1441:
1437:
1434:
1431:
1428:
1425:
1422:
1418:
1396:
1392:
1388:
1383:
1379:
1376:
1373:
1370:
1367:
1364:
1353:
1352:
1349:expected value
1336:
1333:
1330:
1326:
1311:
1310:
1299:
1296:
1293:
1290:
1287:
1284:
1281:
1278:
1275:
1272:
1267:
1264:
1261:
1257:
1253:
1250:
1247:
1244:
1241:
1238:
1235:
1232:
1229:
1226:
1223:
1218:
1215:
1212:
1208:
1204:
1200:
1196:
1193:
1190:
1187:
1184:
1180:
1175:
1171:
1168:
1165:
1162:
1158:
1134:
1106:
1088:
1085:
1064:
1061:
1058:
1055:
1052:
1049:
1046:
1043:
1040:
1037:
1034:
1031:
1028:
1025:
1022:
1019:
1016:
1013:
1010:
990:
987:
984:
981:
978:
975:
972:
969:
966:
963:
960:
957:
954:
951:
948:
916:
913:
910:
907:
904:
884:
881:
878:
875:
872:
844:
841:
838:
793:
773:
753:
750:
747:
744:
741:
737:
734:
730:
727:
724:
721:
718:
707:
706:
694:
691:
688:
685:
682:
679:
676:
673:
670:
667:
663:
657:
654:
651:
648:
644:
639:
635:
632:
629:
626:
623:
620:
617:
614:
591:
569:
566:
553:
550:
547:
544:
541:
538:
533:
530:
527:
524:
520:
499:
478:
475:
472:
469:
386:
383:
380:
377:
374:
371:
368:
345:
342:
338:
334:
331:
327:
306:
303:
300:
296:
275:
272:
268:
264:
261:
257:
233:
230:
227:
223:
199:
196:
193:
190:
187:
183:
159:
139:
114:
113:
96:September 2012
28:
26:
19:
9:
6:
4:
3:
2:
5254:
5243:
5240:
5239:
5237:
5221:
5215:
5207:
5203:
5199:
5195:
5191:
5184:
5176:
5172:
5168:
5161:
5157:
5148:
5145:
5144:
5138:
5121:
5113:
5107:
5084:
5078:
5055:
5049:
5026:
5023:
5020:
5014:
4991:
4988:
4984:
4981:
4971:
4965:
4959:
4953:
4945:
4942:
4939:
4933:
4927:
4924:
4918:
4915:
4912:
4906:
4901:
4897:
4891:
4887:
4883:
4877:
4874:
4871:
4865:
4855:
4841:
4838:
4834:
4831:
4821:
4818:
4815:
4809:
4801:
4795:
4789:
4786:
4780:
4777:
4774:
4768:
4763:
4759:
4753:
4749:
4745:
4739:
4731:
4725:
4718:
4704:
4701:
4694:
4686:
4680:
4677:
4674:
4668:
4660:
4654:
4649:
4645:
4641:
4638:
4632:
4624:
4618:
4611:
4597:
4594:
4590:
4587:
4580:
4577:
4574:
4568:
4565:
4562:
4556:
4553:
4550:
4544:
4539:
4535:
4529:
4525:
4521:
4518:
4512:
4509:
4506:
4500:
4493:
4479:
4476:
4469:
4463:
4460:
4457:
4451:
4445:
4440:
4436:
4432:
4429:
4423:
4417:
4410:
4409:
4408:
4405:
4403:
4399:
4395:
4391:
4385:
4375:
4373:
4369:
4365:
4346:
4333:
4327:
4321:
4315:
4306:
4303:
4300:
4294:
4273:
4269:
4263:
4260:
4257:
4251:
4241:
4240:
4239:
4225:
4205:
4182:
4178:
4164:
4158:
4149:
4146:
4143:
4135:
4129:
4108:
4103:
4094:
4088:
4078:
4072:
4069:
4066:
4060:
4050:
4049:
4048:
4034:
4026:
4010:
3990:
3982:
3978:
3957:
3951:
3948:
3945:
3934:
3928:
3917:
3911:
3900:
3894:
3891:
3888:
3882:
3876:
3870:
3867:
3864:
3858:
3848:
3847:
3846:
3844:
3828:
3808:
3785:
3782:
3779:
3773:
3750:
3726:
3720:
3712:
3701:
3695:
3684:
3678:
3675:
3672:
3666:
3656:
3655:
3654:
3637:
3628:
3622:
3615:
3609:
3601:
3598:
3595:
3589:
3583:
3580:
3574:
3571:
3568:
3562:
3557:
3554:
3551:
3547:
3543:
3533:
3527:
3519:
3511:
3505:
3499:
3496:
3490:
3482:
3476:
3470:
3467:
3464:
3460:
3453:
3447:
3442:
3439:
3436:
3432:
3428:
3422:
3419:
3416:
3410:
3400:
3399:
3398:
3384:
3364:
3344:
3324:
3316:
3312:
3311:
3300:
3298:
3282:
3262:
3239:
3230:
3224:
3216:
3210:
3204:
3201:
3195:
3189:
3184:
3181:
3178:
3174:
3170:
3157:
3151:
3142:
3136:
3115:
3107:
3106:
3105:
3091:
3071:
3051:
3031:
3024:
3020:
3016:
3012:
3008:
3007:
2981:
2975:
2964:
2958:
2955:
2952:
2941:
2935:
2927:
2912:
2911:
2910:
2907:
2905:
2886:
2877:
2874:
2871:
2865:
2857:
2851:
2845:
2842:
2836:
2833:
2830:
2824:
2819:
2816:
2813:
2809:
2805:
2799:
2791:
2785:
2782:
2779:
2773:
2765:
2759:
2754:
2751:
2748:
2744:
2737:
2731:
2726:
2723:
2720:
2716:
2712:
2709:
2705:
2700:
2696:
2688:
2684:
2674:
2668:
2658:
2652:
2644:
2629:
2628:
2627:
2613:
2593:
2585:
2569:
2549:
2541:
2540:
2534:
2520:
2500:
2492:
2470:
2464:
2456:
2453:
2450:
2444:
2438:
2432:
2424:
2418:
2392:
2384:
2378:
2375:
2372:
2366:
2358:
2352:
2347:
2344:
2341:
2337:
2333:
2330:
2321:
2313:
2307:
2304:
2301:
2298:
2289:
2285:
2277:
2273:
2262:
2256:
2248:
2233:
2232:
2231:
2217:
2214:
2211:
2191:
2171:
2157:
2155:
2154:cross entropy
2150:
2148:
2132:
2112:
2085:
2082:
2079:
2073:
2070:
2067:
2061:
2058:
2055:
2049:
2044:
2041:
2038:
2034:
2030:
2027:
2023:
2016:
2013:
2010:
2004:
2001:
1998:
1995:
1991:
1985:
1982:
1979:
1969:
1963:
1960:
1957:
1942:
1941:
1940:
1926:
1906:
1898:
1882:
1862:
1854:
1853:
1852:joint entropy
1845:Joint entropy
1824:
1821:
1818:
1812:
1809:
1803:
1800:
1797:
1791:
1788:
1785:
1782:
1779:
1776:
1773:
1767:
1764:
1761:
1758:
1755:
1744:
1738:
1714:
1713:
1712:
1710:
1709:
1689:
1686:
1681:
1677:
1671:
1666:
1663:
1660:
1656:
1635:
1632:
1627:
1623:
1602:
1597:
1593:
1589:
1586:
1581:
1577:
1571:
1566:
1563:
1560:
1556:
1552:
1549:
1541:
1537:
1533:
1530:
1527:
1522:
1518:
1514:
1509:
1505:
1489:
1488:
1487:
1462:
1443:
1435:
1432:
1429:
1423:
1390:
1381:
1377:
1374:
1368:
1362:
1350:
1331:
1316:
1315:
1314:
1294:
1288:
1285:
1282:
1276:
1270:
1265:
1262:
1259:
1255:
1251:
1248:
1242:
1236:
1227:
1221:
1216:
1213:
1210:
1206:
1202:
1198:
1191:
1185:
1178:
1169:
1163:
1148:
1147:
1146:
1132:
1124:
1120:
1104:
1096:
1095:
1084:
1082:
1076:
1059:
1053:
1047:
1041:
1035:
1029:
1023:
1017:
1011:
985:
979:
973:
967:
964:
958:
952:
946:
939:, then since
938:
934:
930:
911:
905:
879:
873:
862:
858:
842:
836:
826:
823:
817:
815:
811:
807:
791:
771:
748:
745:
742:
728:
722:
716:
686:
680:
674:
671:
668:
665:
661:
652:
646:
642:
637:
633:
630:
627:
621:
615:
605:
604:
603:
589:
582:of a message
581:
577:
576:
565:
551:
548:
545:
542:
539:
536:
531:
528:
522:
497:
476:
473:
470:
467:
458:
456:
452:
448:
444:
440:
436:
432:
428:
424:
420:
416:
412:
408:
404:
381:
378:
375:
369:
359:
340:
332:
301:
270:
262:
247:
228:
213:
194:
191:
188:
173:
172:joint entropy
157:
137:
129:
125:
122:A misleading
120:
110:
107:
99:
88:
85:
81:
78:
74:
71:
67:
64:
60:
57: –
56:
52:
51:Find sources:
45:
41:
35:
34:
29:This article
27:
23:
18:
17:
5214:
5197:
5193:
5183:
5166:
5160:
5006:
4406:
4401:
4387:
4361:
4197:
3973:
3742:
3652:
3337:relative to
3314:
3308:
3306:
3254:
3018:
3014:
3010:
3004:
3002:
2908:
2901:
2584:equivocation
2583:
2537:
2535:
2410:
2163:
2151:
2104:
1850:
1848:
1706:
1704:
1463:
1354:
1347:denotes the
1312:
1118:
1092:
1090:
1077:
932:
928:
860:
856:
827:
821:
818:
708:
579:
573:
571:
459:
426:
414:
405:is based on
400:
102:
93:
83:
76:
69:
62:
50:
38:Please help
33:verification
30:
2147:independent
1615:where each
1119:uncertainty
937:independent
580:"surprisal"
5153:References
4390:continuous
1351:operation.
449:, and the
411:statistics
66:newspapers
4928:
4898:∫
4888:∫
4866:
4790:
4760:∫
4750:∫
4678:
4646:∫
4642:−
4566:
4536:∫
4526:∫
4522:−
4461:
4437:∫
4433:−
4313:‖
4252:
4156:‖
4061:
3935:−
3883:
3859:
3843:symmetric
3774:
3702:−
3667:
3584:
3548:∑
3500:
3468:∈
3461:∑
3440:∈
3433:∑
3411:
3205:
3182:∈
3175:∑
3149:‖
2965:−
2846:
2810:∑
2783:
2752:∈
2745:∑
2724:∈
2717:∑
2713:−
2376:
2345:∈
2338:∑
2334:−
2305:
2299:−
2071:
2035:∑
2031:−
2002:
1996:−
1822:−
1813:
1801:−
1792:−
1786:
1777:−
1765:−
1657:∑
1633:≥
1590:
1557:∑
1553:−
1531:…
1436:
1332:−
1286:
1263:∈
1256:∑
1252:−
1237:
1214:∈
1207:∑
1186:
1054:
1036:
1021:&
1012:
956:&
906:
874:
840:&
675:
669:−
634:
616:
543:
526:→
474:
370:
5236:Category
5141:See also
806:shannons
5171:Bibcode
4370:and to
4023:to the
2489:is the
1123:average
1094:entropy
1087:Entropy
451:hartley
431:shannon
128:Shannon
80:scholar
5007:where
3297:metric
2606:about
2562:given
2513:given
2411:where
2204:given
1313:where
709:where
82:
75:
68:
61:
53:
5223:(PDF)
4396:with
3313:, or
3017:, or
1081:Miami
822:known
87:JSTOR
73:books
5071:and
3009:(or
3003:The
2536:The
2145:are
2125:and
1919:and
1875:and
1849:The
1648:and
1091:The
931:and
895:and
859:and
419:unit
409:and
401:The
150:and
59:news
5202:doi
4925:log
4787:log
4675:log
4563:log
4458:log
4402:not
4027:on
3983:of
3581:log
3497:log
3275:to
3202:log
2843:log
2780:log
2586:of
2542:of
2493:of
2373:log
2302:log
2105:If
2068:log
1999:log
1899:of
1810:log
1783:log
1587:log
1433:log
1283:log
810:bit
672:log
631:log
578:or
540:log
519:lim
471:log
443:nat
439:bit
427:bit
421:of
42:by
5238::
5196:.
5192:.
5169:.
4047::
3845::
3013:,
2533:.
1939::
1711::
1461:.
1075:.
602::
457:.
5225:.
5208:.
5204::
5198:2
5177:.
5173::
5125:)
5122:y
5118:|
5114:x
5111:(
5108:f
5088:)
5085:y
5082:(
5079:f
5059:)
5056:x
5053:(
5050:f
5030:)
5027:y
5024:,
5021:x
5018:(
5015:f
4992:y
4989:d
4985:x
4982:d
4975:)
4972:y
4969:(
4966:f
4963:)
4960:x
4957:(
4954:f
4949:)
4946:y
4943:,
4940:x
4937:(
4934:f
4922:)
4919:y
4916:,
4913:x
4910:(
4907:f
4902:X
4892:Y
4884:=
4881:)
4878:Y
4875:;
4872:X
4869:(
4863:I
4842:y
4839:d
4835:x
4832:d
4825:)
4822:y
4819:,
4816:x
4813:(
4810:f
4805:)
4802:y
4799:(
4796:f
4784:)
4781:y
4778:,
4775:x
4772:(
4769:f
4764:X
4754:Y
4746:=
4743:)
4740:Y
4736:|
4732:X
4729:(
4726:h
4705:x
4702:d
4698:)
4695:y
4691:|
4687:x
4684:(
4681:f
4672:)
4669:y
4665:|
4661:x
4658:(
4655:f
4650:X
4639:=
4636:)
4633:y
4629:|
4625:X
4622:(
4619:h
4598:y
4595:d
4591:x
4588:d
4584:)
4581:y
4578:,
4575:x
4572:(
4569:f
4560:)
4557:y
4554:,
4551:x
4548:(
4545:f
4540:X
4530:Y
4519:=
4516:)
4513:Y
4510:,
4507:X
4504:(
4501:h
4480:x
4477:d
4473:)
4470:x
4467:(
4464:f
4455:)
4452:x
4449:(
4446:f
4441:X
4430:=
4427:)
4424:X
4421:(
4418:h
4347:.
4342:)
4337:)
4334:Y
4331:(
4328:p
4325:)
4322:X
4319:(
4316:p
4310:)
4307:Y
4304:,
4301:X
4298:(
4295:p
4290:(
4282:L
4279:K
4274:D
4270:=
4267:)
4264:Y
4261:;
4258:X
4255:(
4249:I
4226:Y
4206:X
4183:.
4179:]
4173:)
4168:)
4165:X
4162:(
4159:p
4153:)
4150:y
4147:=
4144:Y
4140:|
4136:X
4133:(
4130:p
4125:(
4117:L
4114:K
4109:D
4104:[
4098:)
4095:y
4092:(
4089:p
4084:E
4079:=
4076:)
4073:Y
4070:;
4067:X
4064:(
4058:I
4035:X
4011:Y
3991:X
3958:.
3955:)
3952:Y
3949:,
3946:X
3943:(
3939:H
3932:)
3929:Y
3926:(
3922:H
3918:+
3915:)
3912:X
3909:(
3905:H
3901:=
3898:)
3895:X
3892:;
3889:Y
3886:(
3880:I
3877:=
3874:)
3871:Y
3868:;
3865:X
3862:(
3856:I
3829:Y
3809:X
3789:)
3786:Y
3783:;
3780:X
3777:(
3771:I
3751:Y
3727:.
3724:)
3721:Y
3717:|
3713:X
3710:(
3706:H
3699:)
3696:X
3693:(
3689:H
3685:=
3682:)
3679:Y
3676:;
3673:X
3670:(
3664:I
3638:.
3632:)
3629:y
3626:(
3623:p
3619:)
3616:x
3613:(
3610:p
3605:)
3602:y
3599:,
3596:x
3593:(
3590:p
3578:)
3575:y
3572:,
3569:x
3566:(
3563:p
3558:y
3555:,
3552:x
3544:=
3537:)
3534:x
3531:(
3528:p
3523:)
3520:y
3516:|
3512:x
3509:(
3506:p
3494:)
3491:y
3487:|
3483:x
3480:(
3477:p
3471:X
3465:x
3457:)
3454:y
3451:(
3448:p
3443:Y
3437:y
3429:=
3426:)
3423:Y
3420:;
3417:X
3414:(
3408:I
3385:Y
3365:X
3345:Y
3325:X
3283:p
3263:q
3240:.
3234:)
3231:x
3228:(
3225:q
3220:)
3217:x
3214:(
3211:p
3199:)
3196:x
3193:(
3190:p
3185:X
3179:x
3171:=
3166:)
3161:)
3158:X
3155:(
3152:q
3146:)
3143:X
3140:(
3137:p
3132:(
3124:L
3121:K
3116:D
3092:p
3072:q
3052:q
3032:p
2982:.
2979:)
2976:Y
2973:(
2969:H
2962:)
2959:Y
2956:,
2953:X
2950:(
2946:H
2942:=
2939:)
2936:Y
2932:|
2928:X
2925:(
2921:H
2887:.
2881:)
2878:y
2875:,
2872:x
2869:(
2866:p
2861:)
2858:y
2855:(
2852:p
2840:)
2837:y
2834:,
2831:x
2828:(
2825:p
2820:y
2817:,
2814:x
2806:=
2803:)
2800:y
2796:|
2792:x
2789:(
2786:p
2777:)
2774:y
2770:|
2766:x
2763:(
2760:p
2755:X
2749:x
2741:)
2738:y
2735:(
2732:p
2727:Y
2721:y
2710:=
2706:]
2701:)
2697:y
2693:|
2689:X
2685:(
2680:H
2675:[
2669:Y
2664:E
2659:=
2656:)
2653:Y
2649:|
2645:X
2642:(
2638:H
2614:Y
2594:X
2570:Y
2550:X
2521:y
2501:x
2474:)
2471:y
2468:(
2465:p
2460:)
2457:y
2454:,
2451:x
2448:(
2445:p
2439:=
2436:)
2433:y
2429:|
2425:x
2422:(
2419:p
2396:)
2393:y
2389:|
2385:x
2382:(
2379:p
2370:)
2367:y
2363:|
2359:x
2356:(
2353:p
2348:X
2342:x
2331:=
2328:]
2325:)
2322:y
2318:|
2314:x
2311:(
2308:p
2296:[
2290:]
2286:Y
2282:|
2278:X
2274:[
2268:E
2263:=
2260:)
2257:y
2253:|
2249:X
2246:(
2242:H
2218:y
2215:=
2212:Y
2192:X
2172:Y
2133:Y
2113:X
2089:)
2086:y
2083:,
2080:x
2077:(
2074:p
2065:)
2062:y
2059:,
2056:x
2053:(
2050:p
2045:y
2042:,
2039:x
2028:=
2024:]
2020:)
2017:y
2014:,
2011:x
2008:(
2005:p
1992:[
1986:Y
1983:,
1980:X
1975:E
1970:=
1967:)
1964:Y
1961:,
1958:X
1955:(
1951:H
1927:Y
1907:X
1883:Y
1863:X
1828:)
1825:p
1819:1
1816:(
1807:)
1804:p
1798:1
1795:(
1789:p
1780:p
1774:=
1771:)
1768:p
1762:1
1759:,
1756:p
1753:(
1749:H
1745:=
1742:)
1739:p
1736:(
1730:b
1724:H
1690:1
1687:=
1682:i
1678:p
1672:k
1667:1
1664:=
1661:i
1636:0
1628:i
1624:p
1603:,
1598:i
1594:p
1582:i
1578:p
1572:k
1567:1
1564:=
1561:i
1550:=
1547:)
1542:k
1538:p
1534:,
1528:,
1523:2
1519:p
1515:,
1510:1
1506:p
1502:(
1498:H
1473:H
1448:|
1444:M
1440:|
1430:=
1427:)
1424:M
1421:(
1417:H
1395:|
1391:M
1387:|
1382:/
1378:1
1375:=
1372:)
1369:m
1366:(
1363:p
1335:]
1329:[
1325:E
1298:)
1295:m
1292:(
1289:p
1280:)
1277:m
1274:(
1271:p
1266:M
1260:m
1249:=
1246:)
1243:m
1240:(
1234:I
1231:)
1228:m
1225:(
1222:p
1217:M
1211:m
1203:=
1199:]
1195:)
1192:M
1189:(
1183:I
1179:[
1174:E
1170:=
1167:)
1164:M
1161:(
1157:H
1133:m
1105:M
1063:)
1060:n
1057:(
1051:I
1048:+
1045:)
1042:m
1039:(
1033:I
1030:=
1027:)
1024:n
1018:m
1015:(
1009:I
989:)
986:n
983:(
980:P
977:)
974:m
971:(
968:P
965:=
962:)
959:n
953:m
950:(
947:P
933:n
929:m
915:)
912:n
909:(
903:I
883:)
880:m
877:(
871:I
861:n
857:m
843:n
837:m
792:M
772:m
752:)
749:m
746:=
743:M
740:(
736:r
733:P
729:=
726:)
723:m
720:(
717:p
693:)
690:)
687:m
684:(
681:p
678:(
666:=
662:)
656:)
653:m
650:(
647:p
643:1
638:(
628:=
625:)
622:m
619:(
613:I
590:m
552:0
549:=
546:p
537:p
532:+
529:0
523:p
498:p
477:p
468:p
397:.
385:)
382:Y
379:;
376:X
373:(
367:I
344:)
341:X
337:|
333:Y
330:(
326:H
305:)
302:Y
299:(
295:H
274:)
271:Y
267:|
263:X
260:(
256:H
232:)
229:X
226:(
222:H
198:)
195:Y
192:,
189:X
186:(
182:H
158:Y
138:X
109:)
103:(
98:)
94:(
84:·
77:·
70:·
63:·
36:.
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.
