5074:
4663:
5069:{\displaystyle {\begin{aligned}f(c_{1},\ldots ,c_{6})&{}=\Pr(C_{1}=c_{1}{\text{ and }}\dots {\text{ and }}C_{6}=c_{6})\\&{}={\begin{cases}{\displaystyle {1 \over {18}}{1 \over c_{1}!\cdots c_{k}!}},\ &{\text{when }}\sum _{i=1}^{6}c_{i}=2\\0&{\text{otherwise,}}\end{cases}}\\&{}={\begin{cases}{1 \over 18},\ &{\text{when 2 }}c_{k}{\text{ are }}1\\{1 \over 36},\ &{\text{when exactly one }}c_{k}=2\\0,\ &{\text{otherwise.}}\end{cases}}\end{aligned}}}
826:
2414:
8907:
36:
3808:
2163:
8686:
2105:
4326:
7859:. If the receiving entity had previously known the content of a message with certainty before receiving the message, the amount of information of the message received is zero. Only when the advance knowledge of the content of the message by the receiver is less than 100% certain does the message actually convey information.
3572:
4537:
4053:
2124:
of independent events is the sum of the log-likelihoods of each event. Interpreting log-likelihood as "support" or negative surprisal (the degree to which an event supports a given model: a model is supported by an event to the extent that the event is unsurprising, given the model), this states that
9495:
As a quick illustration, the information content associated with an outcome of 4 heads (or any specific outcome) in 4 consecutive tosses of a coin would be 4 shannons (probability 1/16), and the information content associated with getting a result other than the one specified would be ~0.09 shannons
1757:
9389:
4061:
2409:{\displaystyle {\begin{alignedat}{2}\mathrm {H} (X)&=\sum _{x}{-p_{X}{\left(x\right)}\log {p_{X}{\left(x\right)}}}\\&=\sum _{x}{p_{X}{\left(x\right)}\operatorname {I} _{X}(x)}\\&{\overset {\underset {\mathrm {def} }{}}{=}}\ \operatorname {E} {\left},\end{alignedat}}}
8902:{\displaystyle {\begin{aligned}\operatorname {I} (C)&=\operatorname {I} (A)+\operatorname {I} (B)\\f(\operatorname {P} (C))&=f(\operatorname {P} (A))+f(\operatorname {P} (B))\\&=f{\big (}\operatorname {P} (A)\cdot \operatorname {P} (B){\big )}\\\end{aligned}}}
6943:
152:, which is the expected value of the self-information of a random variable, quantifying how surprising the random variable is "on average". This is the average amount of self-information an observer would expect to gain about a random variable when measuring it.
3282:
3455:
8650:
7561:
4342:
3088:
3869:
2936:
5456:
Unsurprisingly, the information content of learning that both dice were rolled as the same particular number is more than the information content of learning that one dice was one number and the other was a different number. Take for examples the events
669:
8493:
6683:
5959:
5272:
6296:
6220:
3803:{\displaystyle {\begin{aligned}p_{X,Y}\!\left(x,y\right)&{}=\Pr(X=x,\,Y=y)=p_{X}\!(x)\,p_{Y}\!(y)\\&{}={\begin{cases}\displaystyle {1 \over 36},\ &x,y\in \cap \mathbb {N} \\0&{\text{otherwise.}}\end{cases}}\end{aligned}}}
138:
can be interpreted as quantifying the level of "surprise" of a particular outcome. As it is such a basic quantity, it also appears in several other settings, such as the length of a message needed to transmit the event given an optimal
9267:
7074:
5873:
7990:
1307:
5327:. These are the only events that are faithfully preserved with identity of which dice rolled which outcome because the outcomes are the same. Without knowledge to distinguish the dice rolling the other numbers, the other
1653:
374:
1377:
6783:
2809:
2100:{\displaystyle {\begin{aligned}\operatorname {I} _{X,Y}(x,y)&=-\log _{2}\left=-\log _{2}\left\\&=-\log _{2}\left-\log _{2}\left\\&=\operatorname {I} _{X}(x)+\operatorname {I} _{Y}(y)\end{aligned}}}
9645:
R. B. Bernstein and R. D. Levine (1972) "Entropy and
Chemical Change. I. Characterization of Product (and Reactant) Energy Distributions in Reactive Molecular Collisions: Information and Entropy Deficiency",
1762:
6830:
7828:
7440:
5451:
3156:
3336:
8560:
7451:
7353:
6463:
4321:{\displaystyle {\begin{aligned}\operatorname {I} _{X,Y}{(2,4)}&=-\log _{2}\!{\left}=-\log _{2}\!{\left}-\log _{2}\!{\left}\\&=2\log _{2}\!{6}\\&\approx 5.169925{\text{ Sh}}.\end{aligned}}}
2965:
191:
The detailed derivation is below, but it can be shown that there is a unique function of probability that meets these three axioms, up to a multiplicative scaling factor. Broadly, given a real number
2818:
6054:
2504:
534:
8403:
8199:
8154:
6583:
6019:
8691:
5607:
4668:
4066:
3874:
3577:
8109:
8064:
6578:
9262:
9196:
6146:
6100:
5877:
5716:
6224:
6151:
7703:
5791:
5657:
2542:
9007:
5524:
4655:
486:
7221:
4599:
3519:
3331:
7593:
6982:
3148:
5799:
1204:
1027:
5370:
9454:
9102:
5081:
3864:
9050:
7855:
By definition, information is transferred from an originating entity possessing the information to a receiving entity only when the receiver had not known the information
4532:{\displaystyle C_{k}:=\delta _{k}(X)+\delta _{k}(Y)={\begin{cases}0,&\neg \,(X=k\vee Y=k)\\1,&\quad X=k\,\veebar \,Y=k\\2,&\quad X=k\,\wedge \,Y=k\end{cases}}}
9418:
9223:
7915:
4048:{\displaystyle {\begin{aligned}\operatorname {I} _{X,Y}{(2,4)}&=-\log _{2}\!{\left}=\log _{2}\!{36}=2\log _{2}\!{6}\\&\approx 5.169925{\text{ Sh}},\end{aligned}}}
1752:
2710:
2688:
983:
8942:
8681:
8019:
7922:
1516:
1480:
1229:
782:
707:
5305:
9154:
1527:
742:
215:
9124:
7883:, the amount of information conveyed in that forecast is zero because it is known, in advance of receiving the forecast, that darkness always comes with the night.
6821:
6346:
266:
3466:
1131:
811:
7744:
7160:
7107:
6977:
6516:
1441:
9482:
8555:
8535:
8515:
8398:
8378:
8354:
8334:
8314:
8294:
8274:
8254:
8226:
7639:
7284:
7264:
7241:
7127:
6483:
3567:
3547:
2960:
2658:
2566:
2438:
2155:
1314:
1224:
1151:
1102:
529:
506:
443:
261:
238:
938:
906:
1695:
1675:
2125:
independent events add support: the information that the two events together provide for statistical inference is the sum of their independent information.
9420:, the larger the quantity of self-information associated with the message that the event indeed occurred. If the above logarithm is base 2, the unit of
9130:. Since the probabilities of events are always between 0 and 1 and the information associated with these events must be nonnegative, that requires that
9384:{\displaystyle \operatorname {I} (\omega _{n})=-\log(\operatorname {P} (\omega _{n}))=\log \left({\frac {1}{\operatorname {P} (\omega _{n})}}\right)}
7751:
9690:
7289:
187:
If two independent events are measured separately, the total amount of information is the sum of the self-informations of the individual events.
6703:
2727:
9743:
7376:
5394:
9052:. The only operational difference between logarithms of different bases is that of different scaling constants, so we may assume
2595:" of seeing the outcome (a highly improbable outcome is very surprising). This term (as a log-probability measure) was coined by
166:
The term 'perplexity' has been used in language modelling to quantify the uncertainty inherent in a set of prospective events.
9721:
6351:
2451:
Sometimes, the entropy itself is called the "self-information" of the random variable, possibly because the entropy satisfies
9621:
9594:
6024:
2454:
869:
are intuitively more "surprising", and yield more information content, than more common values. Thus, self-information is a
8947:
60:
8159:
8114:
2168:
6938:{\displaystyle p_{X}(k)={\begin{cases}{\frac {1}{N}},&k\in \cap \mathbb {Z} \\0,&{\text{otherwise}}.\end{cases}}}
6310:
5966:
5529:
8069:
8024:
140:
6521:
3277:{\displaystyle p_{X}(k)={\begin{cases}{\frac {1}{6}},&k\in \{1,2,3,4,5,6\}\\0,&{\text{otherwise}}\end{cases}}}
9559:
9228:
9162:
1698:
78:
6105:
6059:
5662:
3450:{\displaystyle \operatorname {I} _{X}(4)=-\log _{2}{p_{X}{(4)}}=-\log _{2}{\tfrac {1}{6}}\approx 2.585\;{\text{Sh}}}
9768:
9057:
8645:{\displaystyle \operatorname {P} (C)=\operatorname {P} (A\cap B)=\operatorname {P} (A)\cdot \operatorname {P} (B).}
7556:{\displaystyle p_{X}(k)={\begin{cases}p_{i},&k=s_{i}\in {\mathcal {S}}\\0,&{\text{otherwise}}.\end{cases}}}
6697:
17:
3083:{\displaystyle \operatorname {I} _{X}(T)=-\log _{2}{p_{X}{({\text{T}})}}=-\log _{2}{\tfrac {1}{2}}=1{\text{ Sh}}.}
8912:
2931:{\displaystyle \operatorname {I} _{X}({\text{H}})=-\log _{2}{p_{X}{({\text{H}})}}=-\log _{2}\!{\tfrac {1}{2}}=1,}
1519:
7886:
Accordingly, the amount of self-information contained in a message conveying content informing an occurrence of
664:{\displaystyle \operatorname {I} _{X}(x):=-\log {\left}=\log {\left({\frac {1}{p_{X}{\left(x\right)}}}\right)}.}
5375:
correspond to one die rolling one number and the other die rolling a different number, each having probability
3526:
8488:{\displaystyle \operatorname {I} (C)=\operatorname {I} (A\cap B)=\operatorname {I} (A)+\operatorname {I} (B).}
7648:
2606:
When the event is a random realization (of a variable) the self-information of the variable is defined as the
6678:{\displaystyle \operatorname {I} _{Z}(5)=-\log _{2}{\tfrac {1}{9}}=\log _{2}{9}\approx 3.169925{\text{ Sh}}.}
5721:
5612:
3102:
2509:
7833:
5460:
3522:
2812:
1414:
451:
148:
4604:
4542:
837:
9763:
7568:
7168:
3471:
3287:
3107:
2572:
5954:{\displaystyle \operatorname {I} \left(B_{3,4}\right)=-\log _{2}\!{\tfrac {1}{18}}=4.169925{\text{ Sh}}}
7887:
7642:
7615:
7603:
5267:{\textstyle (X,Y)\in \left\{(k,k)\right\}_{k=1}^{6}=\left\{(1,1),(2,2),(3,3),(4,4),(5,5),(6,6)\right\}}
2665:
1156:
866:
218:
135:
116:
991:
7443:
6824:
3151:
1444:
446:
51:
9423:
7482:
6861:
6291:{\displaystyle \operatorname {I} ({\text{Diff}})=-\log _{2}\!{\tfrac {5}{6}}=0.2630344{\text{ Sh}}.}
5330:
4948:
4803:
4408:
3717:
3187:
9676:(D. Van Nostrand, 24 West 40 Street, New York 18, New York, U.S.A) Tribus, Myron (1961), pp. 64–66
8233:
7840:
7836:
7600:
7367:
7133:
7129:
6215:{\displaystyle \operatorname {I} ({\text{Same}})=-\log _{2}\!{\tfrac {1}{6}}=2.5849625{\text{ Sh}}}
4658:
3818:
2441:
9016:
27:
Basic quantity derived from the probability of a particular event occurring from a random variable
9396:
9201:
7893:
6313:
1707:
2693:
2671:
947:
8918:
8657:
7995:
7880:
1037:
Intuitively, more information is gained from observing an unexpected event—it is "surprising".
4339:
of which die had which value, we can formalize the approach with so-called counting variables
1485:
1449:
751:
676:
9611:
9584:
9505:
8205:
7371:
6490:
5277:
2445:
1702:
1410:
509:
9133:
7069:{\displaystyle \operatorname {I} _{X}(k)=-\log _{2}{\frac {1}{N}}=\log _{2}{N}{\text{ Sh}}.}
2158:
1041:
712:
194:
184:
The less probable an event is, the more surprising it is and the more information it yields.
9109:
7364:
7244:
6486:
6319:
5868:{\displaystyle \operatorname {I} (A_{2})=-\log _{2}\!{\tfrac {1}{36}}=5.169925{\text{ Sh}}}
2576:
156:
9674:
An
Introduction to Energy, Information and States of Matter, with Engineering Applications
6788:
1107:
787:
8:
7723:
7619:
7139:
7086:
6956:
6495:
6306:
2717:
1054:
1049:
131:, but which has particular mathematical advantages in the setting of information theory.
1381:
In other words, the log-odds can be interpreted as the level of surprise when the event
944:
If a particular event has a 100% probability of occurring, then its self-information is
9510:
9467:
8540:
8520:
8500:
8383:
8363:
8339:
8319:
8299:
8279:
8259:
8239:
8211:
7856:
7714:
7710:
7624:
7269:
7249:
7226:
7112:
6468:
4336:
3552:
3532:
2945:
2643:
2592:
2551:
2545:
2423:
2140:
1419:
1398:
1209:
1136:
1087:
870:
745:
514:
491:
428:
246:
223:
92:
9654:
988:
If a particular event has a 0% probability of occurring, then its self-information is
911:
879:
9677:
9617:
9590:
9565:
9555:
9461:
9127:
6949:, or for the purposes of information theory even uniformly spaced; they need only be
5308:
862:
403:
46:
7844:
1680:
1660:
1406:
181:
An event with probability 100% is perfectly unsurprising and yields no information.
7985:{\displaystyle \operatorname {I} (\omega _{n})=f(\operatorname {P} (\omega _{n}))}
123:. It can be thought of as an alternative way of expressing probability, much like
8208:
of self-information is nonnegative and additive. If a message informing of event
2634:
2134:
1302:{\displaystyle {\text{log-odds}}(x)=\log \left({\frac {p(x)}{p(\lnot x)}}\right)}
1064:
1048:, her friend Bob will gain significantly more information from learning that she
120:
9717:
9489:
9457:
7832:
From these examples, it is possible to calculate the information of any set of
7611:
3813:
2939:
2607:
2417:
2121:
1648:{\displaystyle p_{X,Y}\!\left(x,y\right)=\Pr(X=x,\,Y=y)=p_{X}\!(x)\,p_{Y}\!(y)}
419:
394:
174:
9738:
369:{\displaystyle \mathrm {I} (x):=-\log _{b}{\left}=-\log _{b}{\left(P\right)}.}
9757:
9569:
7863:
7607:
7163:
3333:, as for any other valid roll. The information content of rolling a 4 is thus
2638:
908:, self-informations are represented by extended real numbers in the interval
876:
While standard probabilities are represented by real numbers in the interval
9636:
Thomas M. Cover, Joy A. Thomas; Elements of
Information Theory; p. 20; 1991.
1063:
This establishes an implicit relationship between the self-information of a
940:. In particular, we have the following, for any choice of logarithmic base:
6950:
6316:. In the case of independent fair 6-sided dice rolls, the random variable
2617:
2596:
1402:
7862:
For example, quoting a character (the Hippy Dippy
Weatherman) of comedian
1401:
is the sum of each event's information content. This property is known as
1372:{\displaystyle {\text{log-odds}}(x)=\mathrm {I} (\lnot x)-\mathrm {I} (x)}
9666:
7706:
7355:
In general, there is no information gained from measuring a known value.
5372:
2661:
985:: its occurrence is "perfectly non-surprising" and yields no information.
241:
112:
6778:{\displaystyle X\sim \mathrm {DU} ;\quad a,b\in \mathbb {Z} ,\ b\geq a.}
2804:{\textstyle p_{X}{({\text{H}})}=p_{X}{({\text{T}})}={\tfrac {1}{2}}=0.5}
825:
9485:
2117:
408:
1226:
not occurring. Then we have the following definition of the log-odds:
9010:
8296:
occurring is that of the compound message of both independent events
7645:, we can assume the categorical distribution is supported on the set
3098:
2713:
177:'s definition of self-information was chosen to meet several axioms:
8360:
of the amounts of information of the individual component messages
2721:
1311:
This can be expressed as a difference of two
Shannon informations:
1081:
1068:
856:
128:
2109:
159:, of which the most common is the "bit" (more formally called the
7874:
Continued dark overnight, with widely scattered light by morning.
6946:
1045:
873:
of the probability, or sometimes called an "antitonic" function.
9748:
3460:
709:
for self-information above is not universal. Since the notation
9744:"Surprisal" entry in a glossary of molecular information theory
7823:{\displaystyle \operatorname {I} _{X}(x)=-\log _{2}{p_{X}(x)}.}
7596:
7435:{\textstyle {\mathcal {S}}={\bigl \{}s_{i}{\bigr \}}_{i=1}^{N}}
5446:{\textstyle 6\cdot {\tfrac {1}{36}}+15\cdot {\tfrac {1}{18}}=1}
382:
corresponds to the scaling factor above. Different choices of
2938:
so the information gain of a fair coin landing as heads is 1
9488:. For the base 10 logarithm, the unit of information is the
8336:
occurring. The quantity of information of compound message
9609:
9159:
Taking into account these properties, the self-information
7549:
6931:
5058:
4927:
4525:
3792:
3270:
784:
for self-entropy instead, mirroring the use of the capital
124:
7348:{\displaystyle \operatorname {I} _{X}(b)=-\log _{2}{1}=0.}
6458:{\textstyle p_{Z}(z)=p_{X}(x)*p_{Y}(y)={6-|z-7| \over 36}}
2815:
the variable as heads, the associated information gain is
7136:
with probability distribution deterministically given by
7078:
6687:
6305:
The probability mass or density function (collectively
6049:{\displaystyle {\text{Diff}}={\overline {\text{Same}}}}
2499:{\displaystyle \mathrm {H} (X)=\operatorname {I} (X;X)}
9496:(probability 15/16). See above for detailed examples.
7651:
7379:
7171:
6791:
6629:
6524:
6354:
6263:
6190:
6127:
6108:
6081:
6062:
6021:
be the event that both dice rolled the same value and
5969:
5929:
5843:
5426:
5405:
5397:
5333:
5084:
4607:
4335:
If we receive information about the value of the dice
4330:
3474:
3424:
3290:
3055:
2908:
2784:
2730:
1683:
1663:
1422:
9470:
9460:. This is the most common practice. When using the
9426:
9399:
9270:
9231:
9204:
9165:
9136:
9112:
9060:
9019:
8950:
8921:
8689:
8660:
8563:
8543:
8523:
8503:
8406:
8386:
8366:
8342:
8322:
8302:
8282:
8262:
8242:
8214:
8162:
8117:
8072:
8027:
7998:
7925:
7896:
7754:
7726:
7627:
7571:
7454:
7292:
7272:
7252:
7229:
7142:
7115:
7089:
6985:
6959:
6833:
6706:
6586:
6498:
6471:
6322:
6227:
6154:
6027:
5880:
5802:
5724:
5665:
5615:
5532:
5463:
5280:
4807:
4666:
4545:
4345:
4064:
4056:
3872:
3821:
3720:
3575:
3555:
3535:
3467:
independent, identically distributed random variables
3339:
3159:
3110:
2968:
2948:
2821:
2696:
2674:
2646:
2554:
2512:
2457:
2426:
2166:
2143:
1760:
1710:
1530:
1488:
1452:
1317:
1232:
1212:
1159:
1139:
1110:
1090:
994:
950:
914:
882:
790:
754:
715:
679:
537:
517:
494:
454:
431:
269:
249:
226:
197:
8194:{\displaystyle \operatorname {I} (\omega _{n})>0}
8149:{\displaystyle \operatorname {P} (\omega _{n})<1}
6014:{\textstyle {\text{Same}}=\bigcup _{i=1}^{6}{A_{i}}}
2942:. Likewise, the information gain of measuring tails
2110:§ Two independent, identically distributed dice
155:
The information content can be expressed in various
7565:For the purposes of information theory, the values
7266:, so the information content of any measurement of
5602:{\displaystyle B_{j,k}=\{c_{j}=1\}\cap \{c_{k}=1\}}
1392:
1385:happen, minus the level of surprise when the event
386:correspond to different units of information: when
9476:
9448:
9412:
9383:
9256:
9217:
9190:
9148:
9118:
9096:
9044:
9001:
8936:
8901:
8675:
8644:
8549:
8529:
8509:
8487:
8392:
8372:
8348:
8328:
8308:
8288:
8268:
8248:
8220:
8193:
8148:
8103:
8058:
8013:
7984:
7909:
7822:
7738:
7697:
7633:
7587:
7555:
7434:
7347:
7278:
7258:
7235:
7215:
7154:
7121:
7101:
7068:
6971:
6937:
6815:
6777:
6677:
6572:
6510:
6477:
6457:
6340:
6290:
6214:
6140:
6094:
6048:
6013:
5953:
5867:
5785:
5710:
5651:
5601:
5518:
5445:
5364:
5299:
5266:
5068:
4649:
4593:
4531:
4320:
4047:
3858:
3802:
3561:
3541:
3513:
3449:
3325:
3276:
3142:
3082:
2954:
2930:
2803:
2704:
2682:
2652:
2560:
2536:
2498:
2432:
2408:
2149:
2099:
1746:
1689:
1669:
1647:
1510:
1474:
1435:
1371:
1301:
1218:
1198:
1145:
1125:
1096:
1080:The Shannon information is closely related to the
1021:
977:
932:
900:
805:
776:
736:
701:
663:
523:
500:
480:
437:
368:
255:
232:
209:
8104:{\displaystyle \operatorname {I} (\omega _{n})=0}
8059:{\displaystyle \operatorname {P} (\omega _{n})=1}
7917:, depends only on the probability of that event.
6573:{\textstyle p_{Z}(5)={\frac {4}{36}}={1 \over 9}}
6261:
6188:
5927:
5841:
4287:
4232:
4187:
4125:
4014:
3992:
3933:
3690:
3669:
3596:
2906:
1917:
1897:
1635:
1614:
1547:
263:, the information content is defined as follows:
9755:
7363:Generalizing all of the above cases, consider a
6109:
6063:
4718:
3625:
1570:
1033:From this, we can get a few general properties:
857:Monotonically decreasing function of probability
309:
9257:{\displaystyle \operatorname {P} (\omega _{n})}
9191:{\displaystyle \operatorname {I} (\omega _{n})}
6945:In general, the values of the DURV need not be
6141:{\textstyle \Pr({\text{Diff}})={\tfrac {5}{6}}}
6095:{\textstyle \Pr({\text{Same}})={\tfrac {1}{6}}}
744:is also often used for the related quantity of
6314:is the convolution of each probability measure
6300:
5711:{\displaystyle A_{2}=\{X=2{\text{ and }}Y=2\}}
146:The Shannon information is closely related to
9586:Fundamentals in Information Theory and Coding
9582:
9533:, Vol., Clarendon Press, Oxford pp 11–15 1979
8890:
8850:
7410:
7392:
6148:. The information contents of the events are
5350:
5337:
3461:Two independent, identically distributed dice
1052:than that she lost on a given day. (See also
9639:
5780:
5744:
5705:
5679:
5596:
5577:
5571:
5552:
5513:
5477:
4588:
4552:
3247:
3211:
2610:of the self-information of the realization.
1029:: its occurrence is "infinitely surprising".
7879:Assuming that one does not reside near the
7358:
1075:
425:Formally, given a discrete random variable
6953:. The information gain of any observation
6056:be the event that the dice differed. Then
3441:
2128:
9660:
7698:{\textstyle =\left\{1,2,\dots ,N\right\}}
6904:
6753:
4512:
4508:
4476:
4472:
4422:
3849:
3771:
3679:
3643:
3481:
1624:
1588:
1429:
79:Learn how and when to remove this message
6580:. Therefore, the information asserted is
397:(symbol Sh), often called a 'bit'; when
9549:
6311:sum of two independent random variables
5786:{\displaystyle B_{3,4}=\{(3,4),(4,3)\}}
5652:{\displaystyle j\neq k,1\leq j,k\leq 6}
2537:{\displaystyle \operatorname {I} (X;X)}
14:
9756:
9728:, Vol. 27, pp 379–423, (Part I), 1948.
9722:A Mathematical Theory of Communication
9610:Han, Te Sun; Kobayashi, Kingo (2002).
8497:Because of the independence of events
7079:Special case: constant random variable
2420:information content of measurement of
871:strictly decreasing monotonic function
9613:Mathematics of Information and Coding
9393:The smaller the probability of event
9002:{\displaystyle f(x\cdot y)=f(x)+f(y)}
6688:General discrete uniform distribution
5519:{\displaystyle A_{k}=\{(X,Y)=(k,k)\}}
4650:{\textstyle \sum _{k=1}^{6}{C_{k}}=2}
1413:and probability theory. Consider two
481:{\displaystyle p_{X}{\left(x\right)}}
111:is a basic quantity derived from the
9545:
9543:
9541:
9539:
7216:{\textstyle p_{X}(k)=\delta _{b}(k)}
4594:{\displaystyle k\in \{1,2,3,4,5,6\}}
3514:{\textstyle X,\,Y\sim \mathrm {DU} }
3326:{\textstyle p_{X}(4)={\frac {1}{6}}}
2440:. The expectation is taken over the
820:
488:, the self-information of measuring
29:
9554:. Hoboken, NJ: Wiley-Interscience.
7871:Weather forecast for tonight: dark.
7588:{\displaystyle s\in {\mathcal {S}}}
6693:
4331:Information from frequency of rolls
3143:{\displaystyle X\sim \mathrm {DU} }
24:
9711:
9352:
9308:
9271:
9232:
9166:
8873:
8855:
8817:
8790:
8759:
8734:
8716:
8694:
8624:
8606:
8582:
8564:
8467:
8449:
8425:
8407:
8163:
8118:
8073:
8028:
7957:
7926:
7756:
7580:
7521:
7382:
7294:
6987:
6717:
6714:
6696:example above, consider a general
6588:
6228:
6155:
5881:
5803:
5341:
4419:
4070:
3878:
3492:
3489:
3341:
3284:The probability of rolling a 4 is
3121:
3118:
2970:
2823:
2587:This measure has also been called
2513:
2475:
2459:
2372:
2359:
2349:
2346:
2343:
2306:
2172:
2072:
2047:
1766:
1356:
1343:
1336:
1283:
1166:
1084:. In particular, given some event
1016:
924:
539:
271:
25:
9780:
9732:
9671:Thermodynamics and Thermostatics:
9616:. American Mathematical Society.
9536:
3101:. The value of a dice roll is a
2628:
1701:. The information content of the
1199:{\displaystyle p(\lnot x)=1-p(x)}
8276:, then the information of event
7705:; the mathematical structure is
6698:discrete uniform random variable
3103:discrete uniform random variable
3092:
2601:Thermostatics and Thermodynamics
1393:Additivity of independent events
1022:{\displaystyle -\log(0)=\infty }
824:
34:
9648:The Journal of Chemical Physics
8356:would be expected to equal the
7720:The information of the outcome
6739:
5365:{\textstyle {\binom {6}{2}}=15}
5078:To verify this, the 6 outcomes
4498:
4462:
3812:The information content of the
2116:The corresponding property for
1520:joint probability mass function
1397:The information content of two
748:, many authors use a lowercase
9739:Examples of surprisal measures
9726:Bell Systems Technical Journal
9683:
9630:
9603:
9576:
9523:
9449:{\displaystyle I(\omega _{n})}
9443:
9430:
9371:
9358:
9330:
9327:
9314:
9305:
9290:
9277:
9251:
9238:
9185:
9172:
9091:
9085:
9070:
9064:
9039:
9033:
8996:
8990:
8981:
8975:
8966:
8954:
8944:having the property such that
8931:
8925:
8915:, the only monotone functions
8885:
8879:
8867:
8861:
8832:
8829:
8823:
8814:
8805:
8802:
8796:
8787:
8774:
8771:
8765:
8756:
8746:
8740:
8728:
8722:
8706:
8700:
8670:
8664:
8636:
8630:
8618:
8612:
8600:
8588:
8576:
8570:
8479:
8473:
8461:
8455:
8443:
8431:
8419:
8413:
8182:
8169:
8137:
8124:
8092:
8079:
8047:
8034:
8008:
8002:
7979:
7976:
7963:
7954:
7945:
7932:
7813:
7807:
7774:
7768:
7658:
7652:
7471:
7465:
7312:
7306:
7210:
7204:
7188:
7182:
7005:
6999:
6897:
6885:
6850:
6844:
6733:
6721:
6606:
6600:
6541:
6535:
6445:
6431:
6415:
6409:
6393:
6387:
6371:
6365:
6348:has probability mass function
6242:
6234:
6169:
6161:
6120:
6112:
6074:
6066:
5822:
5809:
5777:
5765:
5759:
5747:
5510:
5498:
5492:
5480:
5256:
5244:
5238:
5226:
5220:
5208:
5202:
5190:
5184:
5172:
5166:
5154:
5121:
5109:
5097:
5085:
4783:
4721:
4706:
4674:
4447:
4423:
4397:
4391:
4375:
4369:
4255:
4249:
4210:
4204:
4161:
4149:
4101:
4089:
4055:and can also be calculated by
3969:
3957:
3909:
3897:
3853:
3840:
3834:
3822:
3764:
3752:
3697:
3691:
3676:
3670:
3653:
3628:
3508:
3496:
3399:
3393:
3359:
3353:
3307:
3301:
3176:
3170:
3137:
3125:
3030:
3022:
2988:
2982:
2885:
2877:
2843:
2835:
2776:
2768:
2750:
2742:
2531:
2519:
2493:
2481:
2469:
2463:
2390:
2384:
2324:
2318:
2182:
2176:
2090:
2084:
2065:
2059:
2027:
2021:
1980:
1974:
1924:
1918:
1904:
1898:
1855:
1843:
1796:
1784:
1741:
1729:
1723:
1711:
1642:
1636:
1621:
1615:
1598:
1573:
1505:
1499:
1469:
1463:
1366:
1360:
1349:
1340:
1329:
1323:
1289:
1280:
1272:
1266:
1244:
1238:
1193:
1187:
1172:
1163:
1120:
1114:
1010:
1004:
966:
960:
927:
915:
895:
883:
800:
794:
771:
765:
731:
719:
696:
690:
557:
551:
281:
275:
13:
1:
9531:Elementary Information Theory
9516:
9097:{\displaystyle f(x)=K\log(x)}
7850:
5796:The information contents are
3859:{\displaystyle (X,Y)=(2,\,4)}
3525:fair 6-sided dice roll. The
2668:of the coin landing as heads
2575:the corresponding concept is
816:
169:
9045:{\displaystyle \log _{b}(x)}
8913:Cauchy's functional equation
8204:Further, by definition, the
8021:to be determined below. If
7162:and probability measure the
6041:
1415:independent random variables
1044:chance of Alice winning the
7:
9749:Bayesian Theory of Surprise
9552:Quantum Computing Explained
9499:
9413:{\displaystyle \omega _{n}}
9218:{\displaystyle \omega _{n}}
8654:However, applying function
8537:, the probability of event
7910:{\displaystyle \omega _{n}}
6301:Information from sum of die
2623:
2573:continuous random variables
2416:by definition equal to the
1747:{\displaystyle (X,Y)=(x,y)}
1040:For example, if there is a
865:, the measurement of rarer
409:natural unit of information
54:. The specific problem is:
10:
9785:
9691:"A quote by George Carlin"
9550:McMahon, David M. (2008).
7643:Without loss of generality
2705:{\displaystyle {\text{T}}}
2683:{\displaystyle {\text{H}}}
1445:probability mass functions
978:{\displaystyle -\log(1)=0}
8937:{\displaystyle f(\cdot )}
8676:{\displaystyle f(\cdot )}
8014:{\displaystyle f(\cdot )}
7444:probability mass function
6825:probability mass function
3521:each corresponding to an
3152:probability mass function
447:probability mass function
9198:associated with outcome
7368:discrete random variable
7359:Categorical distribution
7134:constant random variable
6785:For convenience, define
5274:correspond to the event
4659:multinomial distribution
4657:and the counts have the
2591:, as it represents the "
2582:
1511:{\displaystyle p_{Y}(y)}
1475:{\displaystyle p_{X}(x)}
1076:Relationship to log-odds
777:{\displaystyle h_{X}(x)}
702:{\displaystyle I_{X}(x)}
673:The use of the notation
143:of the random variable.
9769:Entropy and information
9484:, the unit will be the
5300:{\displaystyle C_{k}=2}
2137:of the random variable
2129:Relationship to entropy
411:(symbol nat); and when
163:), as explained below.
9583:Borda, Monica (2011).
9478:
9450:
9414:
9385:
9258:
9219:
9192:
9150:
9149:{\displaystyle K<0}
9120:
9098:
9046:
9003:
8938:
8903:
8677:
8646:
8551:
8531:
8511:
8489:
8394:
8374:
8350:
8330:
8310:
8290:
8270:
8250:
8222:
8195:
8150:
8105:
8060:
8015:
7986:
7911:
7877:
7824:
7740:
7699:
7635:
7589:
7557:
7436:
7349:
7280:
7260:
7237:
7217:
7156:
7123:
7103:
7070:
6973:
6939:
6817:
6779:
6679:
6574:
6512:
6479:
6459:
6342:
6292:
6216:
6142:
6096:
6050:
6015:
5998:
5955:
5869:
5787:
5712:
5653:
5603:
5520:
5447:
5366:
5301:
5268:
5070:
5016:when exactly one
4893:
4651:
4628:
4595:
4533:
4322:
4049:
3860:
3804:
3563:
3543:
3515:
3451:
3327:
3278:
3144:
3084:
2956:
2932:
2805:
2706:
2684:
2654:
2562:
2538:
2500:
2434:
2410:
2151:
2113:below for an example.
2101:
1748:
1691:
1671:
1649:
1512:
1476:
1437:
1373:
1303:
1220:
1206:is the probability of
1200:
1147:
1133:is the probability of
1127:
1098:
1023:
979:
934:
902:
807:
778:
738:
737:{\displaystyle I(X;Y)}
703:
665:
525:
502:
482:
439:
370:
257:
234:
211:
210:{\displaystyle b>1}
9506:Kolmogorov complexity
9479:
9451:
9415:
9386:
9259:
9220:
9193:
9151:
9121:
9119:{\displaystyle \log }
9099:
9047:
9004:
8939:
8904:
8678:
8647:
8552:
8532:
8512:
8490:
8395:
8375:
8351:
8331:
8311:
8291:
8271:
8251:
8223:
8196:
8151:
8106:
8061:
8016:
7987:
7912:
7868:
7825:
7741:
7700:
7636:
7590:
7558:
7437:
7350:
7281:
7261:
7238:
7218:
7157:
7124:
7104:
7071:
6974:
6940:
6818:
6816:{\textstyle N:=b-a+1}
6780:
6694:§ Fair dice roll
6680:
6575:
6513:
6480:
6460:
6343:
6341:{\displaystyle Z=X+Y}
6293:
6217:
6143:
6097:
6051:
6016:
5978:
5956:
5870:
5788:
5713:
5654:
5604:
5521:
5448:
5367:
5302:
5269:
5071:
4873:
4652:
4608:
4596:
4534:
4323:
4050:
3861:
3805:
3564:
3544:
3516:
3452:
3328:
3279:
3145:
3085:
2957:
2933:
2806:
2707:
2685:
2655:
2563:
2539:
2501:
2435:
2411:
2152:
2102:
1749:
1692:
1672:
1650:
1513:
1477:
1438:
1374:
1304:
1221:
1201:
1148:
1128:
1099:
1024:
980:
935:
903:
808:
779:
739:
704:
666:
526:
503:
483:
440:
371:
258:
235:
212:
9468:
9424:
9397:
9268:
9229:
9202:
9163:
9134:
9110:
9058:
9017:
8948:
8919:
8687:
8658:
8561:
8541:
8521:
8501:
8404:
8384:
8364:
8340:
8320:
8300:
8280:
8260:
8240:
8212:
8160:
8115:
8070:
8025:
7996:
7923:
7894:
7752:
7724:
7649:
7625:
7569:
7452:
7377:
7290:
7270:
7250:
7227:
7169:
7140:
7113:
7087:
6983:
6957:
6831:
6789:
6704:
6584:
6522:
6496:
6487:discrete convolution
6469:
6352:
6320:
6225:
6152:
6106:
6060:
6025:
5967:
5878:
5800:
5722:
5663:
5613:
5530:
5461:
5395:
5331:
5278:
5082:
4664:
4605:
4543:
4343:
4062:
4057:additivity of events
3870:
3819:
3573:
3553:
3533:
3472:
3465:Suppose we have two
3337:
3288:
3157:
3108:
2966:
2946:
2819:
2728:
2694:
2672:
2644:
2577:differential entropy
2552:
2510:
2455:
2424:
2164:
2141:
1758:
1708:
1681:
1661:
1528:
1486:
1450:
1420:
1405:in mathematics, and
1315:
1230:
1210:
1157:
1153:occurring, and that
1137:
1126:{\displaystyle p(x)}
1108:
1088:
992:
948:
912:
880:
806:{\displaystyle H(X)}
788:
752:
713:
677:
535:
515:
492:
452:
429:
267:
247:
224:
195:
157:units of information
61:improve this article
56:unclear terminology.
50:to meet Knowledge's
7739:{\displaystyle X=x}
7620:probability measure
7431:
7155:{\displaystyle X=b}
7102:{\displaystyle b=a}
6972:{\displaystyle X=k}
6511:{\displaystyle Z=5}
6307:probability measure
5145:
2718:obverse and reverse
2639:tossing a fair coin
2618:proper scoring rule
2616:is an example of a
1055:Lottery mathematics
136:Shannon information
109:Shannon information
97:information content
9764:Information theory
9511:Surprisal analysis
9474:
9446:
9410:
9381:
9254:
9215:
9188:
9146:
9116:
9094:
9042:
8999:
8934:
8911:Thanks to work on
8899:
8897:
8673:
8642:
8547:
8527:
8507:
8485:
8390:
8370:
8346:
8326:
8306:
8286:
8266:
8246:
8218:
8191:
8146:
8101:
8056:
8011:
7992:for some function
7982:
7907:
7820:
7736:
7715:information theory
7711:probability theory
7695:
7631:
7601:mutually exclusive
7599:; they can be any
7595:do not have to be
7585:
7553:
7548:
7432:
7407:
7345:
7276:
7256:
7233:
7223:. The only value
7213:
7152:
7119:
7099:
7066:
6969:
6935:
6930:
6813:
6775:
6675:
6638:
6570:
6508:
6475:
6455:
6338:
6288:
6272:
6212:
6199:
6138:
6136:
6092:
6090:
6046:
6011:
5951:
5938:
5865:
5852:
5783:
5708:
5649:
5599:
5516:
5443:
5435:
5414:
5362:
5297:
5264:
5103:
5066:
5064:
5057:
4926:
4858:
4647:
4591:
4529:
4524:
4318:
4316:
4045:
4043:
3856:
3800:
3798:
3791:
3737:
3559:
3539:
3527:joint distribution
3511:
3447:
3433:
3323:
3274:
3269:
3140:
3099:fair six-sided die
3097:Suppose we have a
3080:
3064:
2952:
2928:
2917:
2801:
2793:
2702:
2680:
2650:
2558:
2546:mutual information
2534:
2496:
2430:
2406:
2404:
2353:
2280:
2201:
2147:
2097:
2095:
1744:
1687:
1667:
1645:
1518:respectively. The
1508:
1472:
1436:{\textstyle X,\,Y}
1433:
1399:independent events
1369:
1299:
1216:
1196:
1143:
1123:
1094:
1019:
975:
930:
898:
836:. You can help by
803:
774:
746:mutual information
734:
699:
661:
521:
498:
478:
435:
418:, the unit is the
407:, the unit is the
393:, the unit is the
366:
253:
230:
207:
93:information theory
9695:www.goodreads.com
9623:978-0-8218-4256-0
9596:978-3-642-20346-6
9477:{\displaystyle e}
9462:natural logarithm
9375:
9225:with probability
9128:natural logarithm
8550:{\displaystyle C}
8530:{\displaystyle B}
8510:{\displaystyle A}
8393:{\displaystyle B}
8373:{\displaystyle A}
8349:{\displaystyle C}
8329:{\displaystyle B}
8309:{\displaystyle A}
8289:{\displaystyle C}
8269:{\displaystyle B}
8249:{\displaystyle A}
8221:{\displaystyle C}
7634:{\displaystyle p}
7541:
7279:{\displaystyle X}
7259:{\displaystyle b}
7245:deterministically
7236:{\displaystyle X}
7122:{\displaystyle X}
7061:
7035:
6923:
6872:
6762:
6692:Generalizing the
6670:
6637:
6568:
6555:
6478:{\displaystyle *}
6453:
6283:
6271:
6240:
6210:
6198:
6167:
6135:
6118:
6089:
6072:
6044:
6040:
6031:
5973:
5949:
5937:
5863:
5851:
5694:
5434:
5413:
5348:
5309:total probability
5053:
5047:
5017:
5011:
5004:
4987:
4972:
4966:
4959:
4922:
4871:
4865:
4856:
4818:
4758:
4750:
4337:without knowledge
4309:
4036:
3787:
3736:
3729:
3562:{\displaystyle Y}
3542:{\displaystyle X}
3445:
3432:
3321:
3265:
3198:
3075:
3063:
3028:
2955:{\displaystyle T}
2916:
2883:
2841:
2792:
2774:
2748:
2700:
2678:
2653:{\displaystyle X}
2599:in his 1961 book
2561:{\displaystyle X}
2433:{\displaystyle X}
2358:
2354:
2341:
2340:
2271:
2192:
2150:{\displaystyle X}
1409:in particular in
1321:
1293:
1236:
1219:{\displaystyle x}
1146:{\displaystyle x}
1097:{\displaystyle x}
863:probability space
854:
853:
813:for the entropy.
651:
524:{\displaystyle x}
501:{\displaystyle X}
438:{\displaystyle X}
256:{\displaystyle P}
233:{\displaystyle x}
119:occurring from a
89:
88:
81:
52:quality standards
43:This article may
16:(Redirected from
9776:
9705:
9704:
9702:
9701:
9687:
9681:
9664:
9658:
9643:
9637:
9634:
9628:
9627:
9607:
9601:
9600:
9580:
9574:
9573:
9547:
9534:
9527:
9483:
9481:
9480:
9475:
9455:
9453:
9452:
9447:
9442:
9441:
9419:
9417:
9416:
9411:
9409:
9408:
9390:
9388:
9387:
9382:
9380:
9376:
9374:
9370:
9369:
9347:
9326:
9325:
9289:
9288:
9263:
9261:
9260:
9255:
9250:
9249:
9224:
9222:
9221:
9216:
9214:
9213:
9197:
9195:
9194:
9189:
9184:
9183:
9155:
9153:
9152:
9147:
9125:
9123:
9122:
9117:
9103:
9101:
9100:
9095:
9051:
9049:
9048:
9043:
9029:
9028:
9008:
9006:
9005:
9000:
8943:
8941:
8940:
8935:
8908:
8906:
8905:
8900:
8898:
8894:
8893:
8854:
8853:
8838:
8682:
8680:
8679:
8674:
8651:
8649:
8648:
8643:
8556:
8554:
8553:
8548:
8536:
8534:
8533:
8528:
8516:
8514:
8513:
8508:
8494:
8492:
8491:
8486:
8399:
8397:
8396:
8391:
8379:
8377:
8376:
8371:
8355:
8353:
8352:
8347:
8335:
8333:
8332:
8327:
8315:
8313:
8312:
8307:
8295:
8293:
8292:
8287:
8275:
8273:
8272:
8267:
8255:
8253:
8252:
8247:
8227:
8225:
8224:
8219:
8200:
8198:
8197:
8192:
8181:
8180:
8155:
8153:
8152:
8147:
8136:
8135:
8110:
8108:
8107:
8102:
8091:
8090:
8065:
8063:
8062:
8057:
8046:
8045:
8020:
8018:
8017:
8012:
7991:
7989:
7988:
7983:
7975:
7974:
7944:
7943:
7916:
7914:
7913:
7908:
7906:
7905:
7829:
7827:
7826:
7821:
7816:
7806:
7805:
7792:
7791:
7764:
7763:
7745:
7743:
7742:
7737:
7704:
7702:
7701:
7696:
7694:
7690:
7640:
7638:
7637:
7632:
7594:
7592:
7591:
7586:
7584:
7583:
7562:
7560:
7559:
7554:
7552:
7551:
7542:
7539:
7525:
7524:
7515:
7514:
7494:
7493:
7464:
7463:
7441:
7439:
7438:
7433:
7430:
7425:
7414:
7413:
7406:
7405:
7396:
7395:
7386:
7385:
7354:
7352:
7351:
7346:
7338:
7330:
7329:
7302:
7301:
7285:
7283:
7282:
7277:
7265:
7263:
7262:
7257:
7242:
7240:
7239:
7234:
7222:
7220:
7219:
7214:
7203:
7202:
7181:
7180:
7161:
7159:
7158:
7153:
7128:
7126:
7125:
7120:
7108:
7106:
7105:
7100:
7075:
7073:
7072:
7067:
7062:
7059:
7057:
7049:
7048:
7036:
7028:
7023:
7022:
6995:
6994:
6978:
6976:
6975:
6970:
6944:
6942:
6941:
6936:
6934:
6933:
6924:
6921:
6907:
6873:
6865:
6843:
6842:
6822:
6820:
6819:
6814:
6784:
6782:
6781:
6776:
6760:
6756:
6720:
6684:
6682:
6681:
6676:
6671:
6668:
6660:
6652:
6651:
6639:
6630:
6624:
6623:
6596:
6595:
6579:
6577:
6576:
6571:
6569:
6561:
6556:
6548:
6534:
6533:
6518:has probability
6517:
6515:
6514:
6509:
6484:
6482:
6481:
6476:
6464:
6462:
6461:
6456:
6454:
6449:
6448:
6434:
6422:
6408:
6407:
6386:
6385:
6364:
6363:
6347:
6345:
6344:
6339:
6297:
6295:
6294:
6289:
6284:
6281:
6273:
6264:
6260:
6259:
6241:
6238:
6221:
6219:
6218:
6213:
6211:
6208:
6200:
6191:
6187:
6186:
6168:
6165:
6147:
6145:
6144:
6139:
6137:
6128:
6119:
6116:
6101:
6099:
6098:
6093:
6091:
6082:
6073:
6070:
6055:
6053:
6052:
6047:
6045:
6038:
6037:
6032:
6029:
6020:
6018:
6017:
6012:
6010:
6009:
6008:
5997:
5992:
5974:
5971:
5960:
5958:
5957:
5952:
5950:
5947:
5939:
5930:
5926:
5925:
5910:
5906:
5905:
5874:
5872:
5871:
5866:
5864:
5861:
5853:
5844:
5840:
5839:
5821:
5820:
5792:
5790:
5789:
5784:
5740:
5739:
5717:
5715:
5714:
5709:
5695:
5692:
5675:
5674:
5658:
5656:
5655:
5650:
5608:
5606:
5605:
5600:
5589:
5588:
5564:
5563:
5548:
5547:
5525:
5523:
5522:
5517:
5473:
5472:
5452:
5450:
5449:
5444:
5436:
5427:
5415:
5406:
5390:
5388:
5387:
5384:
5381:
5371:
5369:
5368:
5363:
5355:
5354:
5353:
5340:
5326:
5324:
5323:
5320:
5317:
5306:
5304:
5303:
5298:
5290:
5289:
5273:
5271:
5270:
5265:
5263:
5259:
5144:
5139:
5128:
5124:
5075:
5073:
5072:
5067:
5065:
5061:
5060:
5054:
5051:
5045:
5028:
5027:
5018:
5015:
5009:
5005:
4997:
4988:
4985:
4983:
4982:
4973:
4970:
4964:
4960:
4952:
4939:
4934:
4930:
4929:
4923:
4920:
4903:
4902:
4892:
4887:
4872:
4869:
4863:
4859:
4857:
4855:
4851:
4850:
4835:
4834:
4821:
4819:
4817:
4809:
4794:
4789:
4782:
4781:
4769:
4768:
4759:
4756:
4751:
4748:
4746:
4745:
4733:
4732:
4714:
4705:
4704:
4686:
4685:
4656:
4654:
4653:
4648:
4640:
4639:
4638:
4627:
4622:
4600:
4598:
4597:
4592:
4538:
4536:
4535:
4530:
4528:
4527:
4390:
4389:
4368:
4367:
4355:
4354:
4327:
4325:
4324:
4319:
4317:
4310:
4307:
4296:
4292:
4286:
4285:
4267:
4263:
4262:
4258:
4248:
4247:
4231:
4230:
4218:
4217:
4213:
4203:
4202:
4186:
4185:
4170:
4169:
4165:
4164:
4147:
4146:
4124:
4123:
4104:
4084:
4083:
4054:
4052:
4051:
4046:
4044:
4037:
4034:
4023:
4019:
4013:
4012:
3997:
3991:
3990:
3978:
3977:
3973:
3972:
3955:
3954:
3932:
3931:
3912:
3892:
3891:
3865:
3863:
3862:
3857:
3809:
3807:
3806:
3801:
3799:
3795:
3794:
3788:
3785:
3774:
3734:
3730:
3722:
3708:
3703:
3689:
3688:
3668:
3667:
3621:
3615:
3611:
3595:
3594:
3568:
3566:
3565:
3560:
3548:
3546:
3545:
3540:
3520:
3518:
3517:
3512:
3495:
3457:of information.
3456:
3454:
3453:
3448:
3446:
3443:
3434:
3425:
3419:
3418:
3403:
3402:
3391:
3390:
3377:
3376:
3349:
3348:
3332:
3330:
3329:
3324:
3322:
3314:
3300:
3299:
3283:
3281:
3280:
3275:
3273:
3272:
3266:
3263:
3199:
3191:
3169:
3168:
3149:
3147:
3146:
3141:
3124:
3089:
3087:
3086:
3081:
3076:
3073:
3065:
3056:
3050:
3049:
3034:
3033:
3029:
3026:
3020:
3019:
3006:
3005:
2978:
2977:
2961:
2959:
2958:
2953:
2937:
2935:
2934:
2929:
2918:
2909:
2905:
2904:
2889:
2888:
2884:
2881:
2875:
2874:
2861:
2860:
2842:
2839:
2831:
2830:
2810:
2808:
2807:
2802:
2794:
2785:
2779:
2775:
2772:
2766:
2765:
2753:
2749:
2746:
2740:
2739:
2711:
2709:
2708:
2703:
2701:
2698:
2689:
2687:
2686:
2681:
2679:
2676:
2659:
2657:
2656:
2651:
2614:Self-information
2567:
2565:
2564:
2559:
2543:
2541:
2540:
2535:
2505:
2503:
2502:
2497:
2462:
2439:
2437:
2436:
2431:
2415:
2413:
2412:
2407:
2405:
2398:
2397:
2393:
2380:
2379:
2356:
2355:
2352:
2336:
2331:
2327:
2314:
2313:
2304:
2303:
2291:
2290:
2279:
2264:
2260:
2259:
2258:
2257:
2245:
2244:
2228:
2227:
2215:
2214:
2200:
2175:
2156:
2154:
2153:
2148:
2106:
2104:
2103:
2098:
2096:
2080:
2079:
2055:
2054:
2039:
2035:
2031:
2030:
2019:
2018:
2001:
2000:
1988:
1984:
1983:
1972:
1971:
1954:
1953:
1935:
1931:
1927:
1916:
1915:
1896:
1895:
1878:
1877:
1862:
1858:
1842:
1841:
1818:
1817:
1780:
1779:
1753:
1751:
1750:
1745:
1696:
1694:
1693:
1688:
1676:
1674:
1673:
1668:
1654:
1652:
1651:
1646:
1634:
1633:
1613:
1612:
1566:
1562:
1546:
1545:
1517:
1515:
1514:
1509:
1498:
1497:
1481:
1479:
1478:
1473:
1462:
1461:
1442:
1440:
1439:
1434:
1407:sigma additivity
1378:
1376:
1375:
1370:
1359:
1339:
1322:
1319:
1308:
1306:
1305:
1300:
1298:
1294:
1292:
1275:
1261:
1237:
1234:
1225:
1223:
1222:
1217:
1205:
1203:
1202:
1197:
1152:
1150:
1149:
1144:
1132:
1130:
1129:
1124:
1103:
1101:
1100:
1095:
1042:one-in-a-million
1028:
1026:
1025:
1020:
984:
982:
981:
976:
939:
937:
936:
933:{\displaystyle }
931:
907:
905:
904:
901:{\displaystyle }
899:
849:
846:
828:
821:
812:
810:
809:
804:
783:
781:
780:
775:
764:
763:
743:
741:
740:
735:
708:
706:
705:
700:
689:
688:
670:
668:
667:
662:
657:
656:
652:
650:
649:
648:
636:
635:
622:
606:
605:
601:
600:
599:
587:
586:
547:
546:
530:
528:
527:
522:
507:
505:
504:
499:
487:
485:
484:
479:
477:
476:
464:
463:
444:
442:
441:
436:
417:
406:
392:
375:
373:
372:
367:
362:
361:
346:
345:
330:
329:
325:
324:
323:
299:
298:
274:
262:
260:
259:
254:
239:
237:
236:
231:
216:
214:
213:
208:
115:of a particular
101:self-information
84:
77:
73:
70:
64:
38:
37:
30:
21:
18:Self-information
9784:
9783:
9779:
9778:
9777:
9775:
9774:
9773:
9754:
9753:
9735:
9714:
9712:Further reading
9709:
9708:
9699:
9697:
9689:
9688:
9684:
9665:
9661:
9644:
9640:
9635:
9631:
9624:
9608:
9604:
9597:
9581:
9577:
9562:
9548:
9537:
9528:
9524:
9519:
9502:
9469:
9466:
9465:
9437:
9433:
9425:
9422:
9421:
9404:
9400:
9398:
9395:
9394:
9365:
9361:
9351:
9346:
9342:
9321:
9317:
9284:
9280:
9269:
9266:
9265:
9264:is defined as:
9245:
9241:
9230:
9227:
9226:
9209:
9205:
9203:
9200:
9199:
9179:
9175:
9164:
9161:
9160:
9135:
9132:
9131:
9111:
9108:
9107:
9059:
9056:
9055:
9024:
9020:
9018:
9015:
9014:
8949:
8946:
8945:
8920:
8917:
8916:
8896:
8895:
8889:
8888:
8849:
8848:
8836:
8835:
8777:
8750:
8749:
8709:
8690:
8688:
8685:
8684:
8659:
8656:
8655:
8562:
8559:
8558:
8542:
8539:
8538:
8522:
8519:
8518:
8502:
8499:
8498:
8405:
8402:
8401:
8385:
8382:
8381:
8365:
8362:
8361:
8341:
8338:
8337:
8321:
8318:
8317:
8301:
8298:
8297:
8281:
8278:
8277:
8261:
8258:
8257:
8241:
8238:
8237:
8213:
8210:
8209:
8176:
8172:
8161:
8158:
8157:
8131:
8127:
8116:
8113:
8112:
8086:
8082:
8071:
8068:
8067:
8041:
8037:
8026:
8023:
8022:
7997:
7994:
7993:
7970:
7966:
7939:
7935:
7924:
7921:
7920:
7901:
7897:
7895:
7892:
7891:
7853:
7801:
7797:
7796:
7787:
7783:
7759:
7755:
7753:
7750:
7749:
7725:
7722:
7721:
7668:
7664:
7650:
7647:
7646:
7626:
7623:
7622:
7579:
7578:
7570:
7567:
7566:
7547:
7546:
7538:
7536:
7527:
7526:
7520:
7519:
7510:
7506:
7498:
7489:
7485:
7478:
7477:
7459:
7455:
7453:
7450:
7449:
7426:
7415:
7409:
7408:
7401:
7397:
7391:
7390:
7381:
7380:
7378:
7375:
7374:
7361:
7334:
7325:
7321:
7297:
7293:
7291:
7288:
7287:
7271:
7268:
7267:
7251:
7248:
7247:
7228:
7225:
7224:
7198:
7194:
7176:
7172:
7170:
7167:
7166:
7141:
7138:
7137:
7114:
7111:
7110:
7088:
7085:
7084:
7081:
7058:
7053:
7044:
7040:
7027:
7018:
7014:
6990:
6986:
6984:
6981:
6980:
6958:
6955:
6954:
6929:
6928:
6920:
6918:
6909:
6908:
6903:
6877:
6864:
6857:
6856:
6838:
6834:
6832:
6829:
6828:
6790:
6787:
6786:
6752:
6713:
6705:
6702:
6701:
6690:
6667:
6656:
6647:
6643:
6628:
6619:
6615:
6591:
6587:
6585:
6582:
6581:
6560:
6547:
6529:
6525:
6523:
6520:
6519:
6497:
6494:
6493:
6485:represents the
6470:
6467:
6466:
6444:
6430:
6423:
6421:
6403:
6399:
6381:
6377:
6359:
6355:
6353:
6350:
6349:
6321:
6318:
6317:
6303:
6280:
6262:
6255:
6251:
6237:
6226:
6223:
6222:
6207:
6189:
6182:
6178:
6164:
6153:
6150:
6149:
6126:
6115:
6107:
6104:
6103:
6080:
6069:
6061:
6058:
6057:
6036:
6028:
6026:
6023:
6022:
6004:
6000:
5999:
5993:
5982:
5970:
5968:
5965:
5964:
5946:
5928:
5921:
5917:
5895:
5891:
5887:
5879:
5876:
5875:
5860:
5842:
5835:
5831:
5816:
5812:
5801:
5798:
5797:
5729:
5725:
5723:
5720:
5719:
5693: and
5691:
5670:
5666:
5664:
5661:
5660:
5659:. For example,
5614:
5611:
5610:
5584:
5580:
5559:
5555:
5537:
5533:
5531:
5528:
5527:
5468:
5464:
5462:
5459:
5458:
5453:, as required.
5425:
5404:
5396:
5393:
5392:
5385:
5382:
5379:
5378:
5376:
5349:
5336:
5335:
5334:
5332:
5329:
5328:
5321:
5318:
5315:
5314:
5312:
5285:
5281:
5279:
5276:
5275:
5153:
5149:
5140:
5129:
5108:
5104:
5083:
5080:
5079:
5063:
5062:
5056:
5055:
5050:
5048:
5036:
5035:
5023:
5019:
5014:
5012:
4996:
4993:
4992:
4986: are
4984:
4978:
4974:
4969:
4967:
4951:
4944:
4943:
4938:
4932:
4931:
4925:
4924:
4919:
4917:
4911:
4910:
4898:
4894:
4888:
4877:
4868:
4866:
4846:
4842:
4830:
4826:
4825:
4820:
4813:
4808:
4806:
4799:
4798:
4793:
4787:
4786:
4777:
4773:
4764:
4760:
4757: and
4755:
4749: and
4747:
4741:
4737:
4728:
4724:
4713:
4709:
4700:
4696:
4681:
4677:
4667:
4665:
4662:
4661:
4634:
4630:
4629:
4623:
4612:
4606:
4603:
4602:
4544:
4541:
4540:
4523:
4522:
4496:
4487:
4486:
4460:
4451:
4450:
4417:
4404:
4403:
4385:
4381:
4363:
4359:
4350:
4346:
4344:
4341:
4340:
4333:
4315:
4314:
4306:
4294:
4293:
4288:
4281:
4277:
4265:
4264:
4243:
4239:
4238:
4234:
4233:
4226:
4222:
4198:
4194:
4193:
4189:
4188:
4181:
4177:
4148:
4136:
4132:
4131:
4127:
4126:
4119:
4115:
4105:
4088:
4073:
4069:
4065:
4063:
4060:
4059:
4042:
4041:
4033:
4021:
4020:
4015:
4008:
4004:
3993:
3986:
3982:
3956:
3944:
3940:
3939:
3935:
3934:
3927:
3923:
3913:
3896:
3881:
3877:
3873:
3871:
3868:
3867:
3820:
3817:
3816:
3797:
3796:
3790:
3789:
3784:
3782:
3776:
3775:
3770:
3738:
3721:
3713:
3712:
3707:
3701:
3700:
3684:
3680:
3663:
3659:
3620:
3616:
3601:
3597:
3584:
3580:
3576:
3574:
3571:
3570:
3554:
3551:
3550:
3534:
3531:
3530:
3488:
3473:
3470:
3469:
3463:
3442:
3423:
3414:
3410:
3392:
3386:
3382:
3381:
3372:
3368:
3344:
3340:
3338:
3335:
3334:
3313:
3295:
3291:
3289:
3286:
3285:
3268:
3267:
3262:
3260:
3251:
3250:
3203:
3190:
3183:
3182:
3164:
3160:
3158:
3155:
3154:
3117:
3109:
3106:
3105:
3095:
3072:
3054:
3045:
3041:
3025:
3021:
3015:
3011:
3010:
3001:
2997:
2973:
2969:
2967:
2964:
2963:
2947:
2944:
2943:
2907:
2900:
2896:
2880:
2876:
2870:
2866:
2865:
2856:
2852:
2838:
2826:
2822:
2820:
2817:
2816:
2783:
2771:
2767:
2761:
2757:
2745:
2741:
2735:
2731:
2729:
2726:
2725:
2697:
2695:
2692:
2691:
2675:
2673:
2670:
2669:
2645:
2642:
2641:
2635:Bernoulli trial
2631:
2626:
2585:
2553:
2550:
2549:
2511:
2508:
2507:
2458:
2456:
2453:
2452:
2442:discrete values
2425:
2422:
2421:
2403:
2402:
2375:
2371:
2370:
2366:
2365:
2342:
2335:
2329:
2328:
2309:
2305:
2293:
2292:
2286:
2282:
2281:
2275:
2262:
2261:
2247:
2246:
2240:
2236:
2235:
2217:
2216:
2210:
2206:
2202:
2196:
2185:
2171:
2167:
2165:
2162:
2161:
2142:
2139:
2138:
2135:Shannon entropy
2131:
2094:
2093:
2075:
2071:
2050:
2046:
2037:
2036:
2020:
2014:
2010:
2009:
2005:
1996:
1992:
1973:
1967:
1963:
1962:
1958:
1949:
1945:
1933:
1932:
1911:
1907:
1891:
1887:
1886:
1882:
1873:
1869:
1831:
1827:
1826:
1822:
1813:
1809:
1799:
1769:
1765:
1761:
1759:
1756:
1755:
1709:
1706:
1705:
1682:
1679:
1678:
1662:
1659:
1658:
1629:
1625:
1608:
1604:
1552:
1548:
1535:
1531:
1529:
1526:
1525:
1493:
1489:
1487:
1484:
1483:
1457:
1453:
1451:
1448:
1447:
1421:
1418:
1417:
1395:
1355:
1335:
1318:
1316:
1313:
1312:
1276:
1262:
1260:
1256:
1233:
1231:
1228:
1227:
1211:
1208:
1207:
1158:
1155:
1154:
1138:
1135:
1134:
1109:
1106:
1105:
1104:, suppose that
1089:
1086:
1085:
1078:
1065:random variable
993:
990:
989:
949:
946:
945:
913:
910:
909:
881:
878:
877:
859:
850:
844:
841:
834:needs expansion
819:
789:
786:
785:
759:
755:
753:
750:
749:
714:
711:
710:
684:
680:
678:
675:
674:
638:
637:
631:
627:
626:
621:
617:
616:
589:
588:
582:
578:
577:
573:
572:
542:
538:
536:
533:
532:
516:
513:
512:
493:
490:
489:
466:
465:
459:
455:
453:
450:
449:
430:
427:
426:
422:(symbol Hart).
412:
398:
387:
351:
350:
341:
337:
313:
312:
308:
304:
303:
294:
290:
270:
268:
265:
264:
248:
245:
244:
225:
222:
221:
196:
193:
192:
172:
121:random variable
85:
74:
68:
65:
58:
39:
35:
28:
23:
22:
15:
12:
11:
5:
9782:
9772:
9771:
9766:
9752:
9751:
9746:
9741:
9734:
9733:External links
9731:
9730:
9729:
9713:
9710:
9707:
9706:
9682:
9659:
9638:
9629:
9622:
9602:
9595:
9575:
9560:
9535:
9521:
9520:
9518:
9515:
9514:
9513:
9508:
9501:
9498:
9473:
9445:
9440:
9436:
9432:
9429:
9407:
9403:
9379:
9373:
9368:
9364:
9360:
9357:
9354:
9350:
9345:
9341:
9338:
9335:
9332:
9329:
9324:
9320:
9316:
9313:
9310:
9307:
9304:
9301:
9298:
9295:
9292:
9287:
9283:
9279:
9276:
9273:
9253:
9248:
9244:
9240:
9237:
9234:
9212:
9208:
9187:
9182:
9178:
9174:
9171:
9168:
9145:
9142:
9139:
9115:
9093:
9090:
9087:
9084:
9081:
9078:
9075:
9072:
9069:
9066:
9063:
9041:
9038:
9035:
9032:
9027:
9023:
8998:
8995:
8992:
8989:
8986:
8983:
8980:
8977:
8974:
8971:
8968:
8965:
8962:
8959:
8956:
8953:
8933:
8930:
8927:
8924:
8892:
8887:
8884:
8881:
8878:
8875:
8872:
8869:
8866:
8863:
8860:
8857:
8852:
8847:
8844:
8841:
8839:
8837:
8834:
8831:
8828:
8825:
8822:
8819:
8816:
8813:
8810:
8807:
8804:
8801:
8798:
8795:
8792:
8789:
8786:
8783:
8780:
8778:
8776:
8773:
8770:
8767:
8764:
8761:
8758:
8755:
8752:
8751:
8748:
8745:
8742:
8739:
8736:
8733:
8730:
8727:
8724:
8721:
8718:
8715:
8712:
8710:
8708:
8705:
8702:
8699:
8696:
8693:
8692:
8672:
8669:
8666:
8663:
8641:
8638:
8635:
8632:
8629:
8626:
8623:
8620:
8617:
8614:
8611:
8608:
8605:
8602:
8599:
8596:
8593:
8590:
8587:
8584:
8581:
8578:
8575:
8572:
8569:
8566:
8546:
8526:
8506:
8484:
8481:
8478:
8475:
8472:
8469:
8466:
8463:
8460:
8457:
8454:
8451:
8448:
8445:
8442:
8439:
8436:
8433:
8430:
8427:
8424:
8421:
8418:
8415:
8412:
8409:
8400:respectively:
8389:
8369:
8345:
8325:
8305:
8285:
8265:
8245:
8217:
8190:
8187:
8184:
8179:
8175:
8171:
8168:
8165:
8145:
8142:
8139:
8134:
8130:
8126:
8123:
8120:
8100:
8097:
8094:
8089:
8085:
8081:
8078:
8075:
8055:
8052:
8049:
8044:
8040:
8036:
8033:
8030:
8010:
8007:
8004:
8001:
7981:
7978:
7973:
7969:
7965:
7962:
7959:
7956:
7953:
7950:
7947:
7942:
7938:
7934:
7931:
7928:
7904:
7900:
7852:
7849:
7819:
7815:
7812:
7809:
7804:
7800:
7795:
7790:
7786:
7782:
7779:
7776:
7773:
7770:
7767:
7762:
7758:
7735:
7732:
7729:
7713:and therefore
7693:
7689:
7686:
7683:
7680:
7677:
7674:
7671:
7667:
7663:
7660:
7657:
7654:
7630:
7614:that has been
7612:finite measure
7582:
7577:
7574:
7550:
7545:
7537:
7535:
7532:
7529:
7528:
7523:
7518:
7513:
7509:
7505:
7502:
7499:
7497:
7492:
7488:
7484:
7483:
7481:
7476:
7473:
7470:
7467:
7462:
7458:
7429:
7424:
7421:
7418:
7412:
7404:
7400:
7394:
7389:
7384:
7360:
7357:
7344:
7341:
7337:
7333:
7328:
7324:
7320:
7317:
7314:
7311:
7308:
7305:
7300:
7296:
7275:
7255:
7232:
7212:
7209:
7206:
7201:
7197:
7193:
7190:
7187:
7184:
7179:
7175:
7151:
7148:
7145:
7118:
7098:
7095:
7092:
7080:
7077:
7065:
7056:
7052:
7047:
7043:
7039:
7034:
7031:
7026:
7021:
7017:
7013:
7010:
7007:
7004:
7001:
6998:
6993:
6989:
6968:
6965:
6962:
6932:
6927:
6919:
6917:
6914:
6911:
6910:
6906:
6902:
6899:
6896:
6893:
6890:
6887:
6884:
6881:
6878:
6876:
6871:
6868:
6863:
6862:
6860:
6855:
6852:
6849:
6846:
6841:
6837:
6812:
6809:
6806:
6803:
6800:
6797:
6794:
6774:
6771:
6768:
6765:
6759:
6755:
6751:
6748:
6745:
6742:
6738:
6735:
6732:
6729:
6726:
6723:
6719:
6716:
6712:
6709:
6689:
6686:
6674:
6666:
6663:
6659:
6655:
6650:
6646:
6642:
6636:
6633:
6627:
6622:
6618:
6614:
6611:
6608:
6605:
6602:
6599:
6594:
6590:
6567:
6564:
6559:
6554:
6551:
6546:
6543:
6540:
6537:
6532:
6528:
6507:
6504:
6501:
6474:
6452:
6447:
6443:
6440:
6437:
6433:
6429:
6426:
6420:
6417:
6414:
6411:
6406:
6402:
6398:
6395:
6392:
6389:
6384:
6380:
6376:
6373:
6370:
6367:
6362:
6358:
6337:
6334:
6331:
6328:
6325:
6302:
6299:
6287:
6279:
6276:
6270:
6267:
6258:
6254:
6250:
6247:
6244:
6236:
6233:
6230:
6206:
6203:
6197:
6194:
6185:
6181:
6177:
6174:
6171:
6163:
6160:
6157:
6134:
6131:
6125:
6122:
6114:
6111:
6088:
6085:
6079:
6076:
6068:
6065:
6043:
6035:
6007:
6003:
5996:
5991:
5988:
5985:
5981:
5977:
5945:
5942:
5936:
5933:
5924:
5920:
5916:
5913:
5909:
5904:
5901:
5898:
5894:
5890:
5886:
5883:
5859:
5856:
5850:
5847:
5838:
5834:
5830:
5827:
5824:
5819:
5815:
5811:
5808:
5805:
5782:
5779:
5776:
5773:
5770:
5767:
5764:
5761:
5758:
5755:
5752:
5749:
5746:
5743:
5738:
5735:
5732:
5728:
5707:
5704:
5701:
5698:
5690:
5687:
5684:
5681:
5678:
5673:
5669:
5648:
5645:
5642:
5639:
5636:
5633:
5630:
5627:
5624:
5621:
5618:
5598:
5595:
5592:
5587:
5583:
5579:
5576:
5573:
5570:
5567:
5562:
5558:
5554:
5551:
5546:
5543:
5540:
5536:
5515:
5512:
5509:
5506:
5503:
5500:
5497:
5494:
5491:
5488:
5485:
5482:
5479:
5476:
5471:
5467:
5442:
5439:
5433:
5430:
5424:
5421:
5418:
5412:
5409:
5403:
5400:
5361:
5358:
5352:
5347:
5344:
5339:
5296:
5293:
5288:
5284:
5262:
5258:
5255:
5252:
5249:
5246:
5243:
5240:
5237:
5234:
5231:
5228:
5225:
5222:
5219:
5216:
5213:
5210:
5207:
5204:
5201:
5198:
5195:
5192:
5189:
5186:
5183:
5180:
5177:
5174:
5171:
5168:
5165:
5162:
5159:
5156:
5152:
5148:
5143:
5138:
5135:
5132:
5127:
5123:
5120:
5117:
5114:
5111:
5107:
5102:
5099:
5096:
5093:
5090:
5087:
5059:
5049:
5044:
5041:
5038:
5037:
5034:
5031:
5026:
5022:
5013:
5008:
5003:
5000:
4995:
4994:
4991:
4981:
4977:
4968:
4963:
4958:
4955:
4950:
4949:
4947:
4942:
4937:
4935:
4933:
4928:
4918:
4916:
4913:
4912:
4909:
4906:
4901:
4897:
4891:
4886:
4883:
4880:
4876:
4867:
4862:
4854:
4849:
4845:
4841:
4838:
4833:
4829:
4824:
4816:
4812:
4805:
4804:
4802:
4797:
4792:
4790:
4788:
4785:
4780:
4776:
4772:
4767:
4763:
4754:
4744:
4740:
4736:
4731:
4727:
4723:
4720:
4717:
4712:
4710:
4708:
4703:
4699:
4695:
4692:
4689:
4684:
4680:
4676:
4673:
4670:
4669:
4646:
4643:
4637:
4633:
4626:
4621:
4618:
4615:
4611:
4590:
4587:
4584:
4581:
4578:
4575:
4572:
4569:
4566:
4563:
4560:
4557:
4554:
4551:
4548:
4526:
4521:
4518:
4515:
4511:
4507:
4504:
4501:
4497:
4495:
4492:
4489:
4488:
4485:
4482:
4479:
4475:
4471:
4468:
4465:
4461:
4459:
4456:
4453:
4452:
4449:
4446:
4443:
4440:
4437:
4434:
4431:
4428:
4425:
4421:
4418:
4416:
4413:
4410:
4409:
4407:
4402:
4399:
4396:
4393:
4388:
4384:
4380:
4377:
4374:
4371:
4366:
4362:
4358:
4353:
4349:
4332:
4329:
4313:
4305:
4302:
4299:
4297:
4295:
4291:
4284:
4280:
4276:
4273:
4270:
4268:
4266:
4261:
4257:
4254:
4251:
4246:
4242:
4237:
4229:
4225:
4221:
4216:
4212:
4209:
4206:
4201:
4197:
4192:
4184:
4180:
4176:
4173:
4168:
4163:
4160:
4157:
4154:
4151:
4145:
4142:
4139:
4135:
4130:
4122:
4118:
4114:
4111:
4108:
4106:
4103:
4100:
4097:
4094:
4091:
4087:
4082:
4079:
4076:
4072:
4068:
4067:
4040:
4032:
4029:
4026:
4024:
4022:
4018:
4011:
4007:
4003:
4000:
3996:
3989:
3985:
3981:
3976:
3971:
3968:
3965:
3962:
3959:
3953:
3950:
3947:
3943:
3938:
3930:
3926:
3922:
3919:
3916:
3914:
3911:
3908:
3905:
3902:
3899:
3895:
3890:
3887:
3884:
3880:
3876:
3875:
3855:
3852:
3848:
3845:
3842:
3839:
3836:
3833:
3830:
3827:
3824:
3814:random variate
3793:
3783:
3781:
3778:
3777:
3773:
3769:
3766:
3763:
3760:
3757:
3754:
3751:
3748:
3745:
3742:
3739:
3733:
3728:
3725:
3719:
3718:
3716:
3711:
3706:
3704:
3702:
3699:
3696:
3693:
3687:
3683:
3678:
3675:
3672:
3666:
3662:
3658:
3655:
3652:
3649:
3646:
3642:
3639:
3636:
3633:
3630:
3627:
3624:
3619:
3617:
3614:
3610:
3607:
3604:
3600:
3593:
3590:
3587:
3583:
3579:
3578:
3558:
3538:
3510:
3507:
3504:
3501:
3498:
3494:
3491:
3487:
3484:
3480:
3477:
3462:
3459:
3440:
3437:
3431:
3428:
3422:
3417:
3413:
3409:
3406:
3401:
3398:
3395:
3389:
3385:
3380:
3375:
3371:
3367:
3364:
3361:
3358:
3355:
3352:
3347:
3343:
3320:
3317:
3312:
3309:
3306:
3303:
3298:
3294:
3271:
3261:
3259:
3256:
3253:
3252:
3249:
3246:
3243:
3240:
3237:
3234:
3231:
3228:
3225:
3222:
3219:
3216:
3213:
3210:
3207:
3204:
3202:
3197:
3194:
3189:
3188:
3186:
3181:
3178:
3175:
3172:
3167:
3163:
3139:
3136:
3133:
3130:
3127:
3123:
3120:
3116:
3113:
3094:
3091:
3079:
3071:
3068:
3062:
3059:
3053:
3048:
3044:
3040:
3037:
3032:
3024:
3018:
3014:
3009:
3004:
3000:
2996:
2993:
2990:
2987:
2984:
2981:
2976:
2972:
2951:
2927:
2924:
2921:
2915:
2912:
2903:
2899:
2895:
2892:
2887:
2879:
2873:
2869:
2864:
2859:
2855:
2851:
2848:
2845:
2837:
2834:
2829:
2825:
2800:
2797:
2791:
2788:
2782:
2778:
2770:
2764:
2760:
2756:
2752:
2744:
2738:
2734:
2649:
2630:
2629:Fair coin toss
2627:
2625:
2622:
2608:expected value
2584:
2581:
2557:
2533:
2530:
2527:
2524:
2521:
2518:
2515:
2495:
2492:
2489:
2486:
2483:
2480:
2477:
2474:
2471:
2468:
2465:
2461:
2429:
2401:
2396:
2392:
2389:
2386:
2383:
2378:
2374:
2369:
2364:
2361:
2351:
2348:
2345:
2339:
2334:
2332:
2330:
2326:
2323:
2320:
2317:
2312:
2308:
2302:
2299:
2296:
2289:
2285:
2278:
2274:
2270:
2267:
2265:
2263:
2256:
2253:
2250:
2243:
2239:
2234:
2231:
2226:
2223:
2220:
2213:
2209:
2205:
2199:
2195:
2191:
2188:
2186:
2184:
2181:
2178:
2174:
2170:
2169:
2146:
2130:
2127:
2122:log-likelihood
2092:
2089:
2086:
2083:
2078:
2074:
2070:
2067:
2064:
2061:
2058:
2053:
2049:
2045:
2042:
2040:
2038:
2034:
2029:
2026:
2023:
2017:
2013:
2008:
2004:
1999:
1995:
1991:
1987:
1982:
1979:
1976:
1970:
1966:
1961:
1957:
1952:
1948:
1944:
1941:
1938:
1936:
1934:
1930:
1926:
1923:
1920:
1914:
1910:
1906:
1903:
1900:
1894:
1890:
1885:
1881:
1876:
1872:
1868:
1865:
1861:
1857:
1854:
1851:
1848:
1845:
1840:
1837:
1834:
1830:
1825:
1821:
1816:
1812:
1808:
1805:
1802:
1800:
1798:
1795:
1792:
1789:
1786:
1783:
1778:
1775:
1772:
1768:
1764:
1763:
1743:
1740:
1737:
1734:
1731:
1728:
1725:
1722:
1719:
1716:
1713:
1690:{\textstyle Y}
1686:
1670:{\textstyle X}
1666:
1644:
1641:
1638:
1632:
1628:
1623:
1620:
1617:
1611:
1607:
1603:
1600:
1597:
1594:
1591:
1587:
1584:
1581:
1578:
1575:
1572:
1569:
1565:
1561:
1558:
1555:
1551:
1544:
1541:
1538:
1534:
1507:
1504:
1501:
1496:
1492:
1471:
1468:
1465:
1460:
1456:
1432:
1428:
1425:
1394:
1391:
1368:
1365:
1362:
1358:
1354:
1351:
1348:
1345:
1342:
1338:
1334:
1331:
1328:
1325:
1297:
1291:
1288:
1285:
1282:
1279:
1274:
1271:
1268:
1265:
1259:
1255:
1252:
1249:
1246:
1243:
1240:
1215:
1195:
1192:
1189:
1186:
1183:
1180:
1177:
1174:
1171:
1168:
1165:
1162:
1142:
1122:
1119:
1116:
1113:
1093:
1077:
1074:
1073:
1072:
1061:
1060:
1059:
1031:
1030:
1018:
1015:
1012:
1009:
1006:
1003:
1000:
997:
986:
974:
971:
968:
965:
962:
959:
956:
953:
929:
926:
923:
920:
917:
897:
894:
891:
888:
885:
858:
855:
852:
851:
831:
829:
818:
815:
802:
799:
796:
793:
773:
770:
767:
762:
758:
733:
730:
727:
724:
721:
718:
698:
695:
692:
687:
683:
660:
655:
647:
644:
641:
634:
630:
625:
620:
615:
612:
609:
604:
598:
595:
592:
585:
581:
576:
571:
568:
565:
562:
559:
556:
553:
550:
545:
541:
531:is defined as
520:
497:
475:
472:
469:
462:
458:
434:
365:
360:
357:
354:
349:
344:
340:
336:
333:
328:
322:
319:
316:
311:
307:
302:
297:
293:
289:
286:
283:
280:
277:
273:
252:
229:
206:
203:
200:
189:
188:
185:
182:
175:Claude Shannon
171:
168:
87:
86:
42:
40:
33:
26:
9:
6:
4:
3:
2:
9781:
9770:
9767:
9765:
9762:
9761:
9759:
9750:
9747:
9745:
9742:
9740:
9737:
9736:
9727:
9723:
9719:
9716:
9715:
9696:
9692:
9686:
9679:
9675:
9672:
9668:
9663:
9656:
9652:
9649:
9642:
9633:
9625:
9619:
9615:
9614:
9606:
9598:
9592:
9588:
9587:
9579:
9571:
9567:
9563:
9561:9780470181386
9557:
9553:
9546:
9544:
9542:
9540:
9532:
9529:Jones, D.S.,
9526:
9522:
9512:
9509:
9507:
9504:
9503:
9497:
9493:
9491:
9487:
9471:
9463:
9459:
9438:
9434:
9427:
9405:
9401:
9391:
9377:
9366:
9362:
9355:
9348:
9343:
9339:
9336:
9333:
9322:
9318:
9311:
9302:
9299:
9296:
9293:
9285:
9281:
9274:
9246:
9242:
9235:
9210:
9206:
9180:
9176:
9169:
9157:
9143:
9140:
9137:
9129:
9113:
9104:
9088:
9082:
9079:
9076:
9073:
9067:
9061:
9053:
9036:
9030:
9025:
9021:
9012:
8993:
8987:
8984:
8978:
8972:
8969:
8963:
8960:
8957:
8951:
8928:
8922:
8914:
8909:
8882:
8876:
8870:
8864:
8858:
8845:
8842:
8840:
8826:
8820:
8811:
8808:
8799:
8793:
8784:
8781:
8779:
8768:
8762:
8753:
8743:
8737:
8731:
8725:
8719:
8713:
8711:
8703:
8697:
8667:
8661:
8652:
8639:
8633:
8627:
8621:
8615:
8609:
8603:
8597:
8594:
8591:
8585:
8579:
8573:
8567:
8544:
8524:
8504:
8495:
8482:
8476:
8470:
8464:
8458:
8452:
8446:
8440:
8437:
8434:
8428:
8422:
8416:
8410:
8387:
8367:
8359:
8343:
8323:
8303:
8283:
8263:
8243:
8235:
8231:
8215:
8207:
8202:
8188:
8185:
8177:
8173:
8166:
8143:
8140:
8132:
8128:
8121:
8098:
8095:
8087:
8083:
8076:
8053:
8050:
8042:
8038:
8031:
8005:
7999:
7971:
7967:
7960:
7951:
7948:
7940:
7936:
7929:
7918:
7902:
7898:
7889:
7884:
7882:
7881:polar regions
7876:
7875:
7872:
7867:
7865:
7864:George Carlin
7860:
7858:
7848:
7846:
7842:
7841:distributions
7838:
7835:
7830:
7817:
7810:
7802:
7798:
7793:
7788:
7784:
7780:
7777:
7771:
7765:
7760:
7747:
7733:
7730:
7727:
7718:
7716:
7712:
7708:
7691:
7687:
7684:
7681:
7678:
7675:
7672:
7669:
7665:
7661:
7655:
7644:
7628:
7621:
7617:
7613:
7609:
7608:measure space
7605:
7602:
7598:
7575:
7572:
7563:
7543:
7533:
7530:
7516:
7511:
7507:
7503:
7500:
7495:
7490:
7486:
7479:
7474:
7468:
7460:
7456:
7447:
7445:
7427:
7422:
7419:
7416:
7402:
7398:
7387:
7373:
7369:
7366:
7356:
7342:
7339:
7335:
7331:
7326:
7322:
7318:
7315:
7309:
7303:
7298:
7273:
7253:
7246:
7230:
7207:
7199:
7195:
7191:
7185:
7177:
7173:
7165:
7164:Dirac measure
7149:
7146:
7143:
7135:
7131:
7116:
7096:
7093:
7090:
7076:
7063:
7054:
7050:
7045:
7041:
7037:
7032:
7029:
7024:
7019:
7015:
7011:
7008:
7002:
6996:
6991:
6966:
6963:
6960:
6952:
6948:
6925:
6915:
6912:
6900:
6894:
6891:
6888:
6882:
6879:
6874:
6869:
6866:
6858:
6853:
6847:
6839:
6835:
6826:
6810:
6807:
6804:
6801:
6798:
6795:
6792:
6772:
6769:
6766:
6763:
6757:
6749:
6746:
6743:
6740:
6736:
6730:
6727:
6724:
6710:
6707:
6699:
6695:
6685:
6672:
6664:
6661:
6657:
6653:
6648:
6644:
6640:
6634:
6631:
6625:
6620:
6616:
6612:
6609:
6603:
6597:
6592:
6565:
6562:
6557:
6552:
6549:
6544:
6538:
6530:
6526:
6505:
6502:
6499:
6492:
6488:
6472:
6450:
6441:
6438:
6435:
6427:
6424:
6418:
6412:
6404:
6400:
6396:
6390:
6382:
6378:
6374:
6368:
6360:
6356:
6335:
6332:
6329:
6326:
6323:
6315:
6312:
6308:
6298:
6285:
6277:
6274:
6268:
6265:
6256:
6252:
6248:
6245:
6231:
6204:
6201:
6195:
6192:
6183:
6179:
6175:
6172:
6158:
6132:
6129:
6123:
6086:
6083:
6077:
6033:
6005:
6001:
5994:
5989:
5986:
5983:
5979:
5975:
5961:
5943:
5940:
5934:
5931:
5922:
5918:
5914:
5911:
5907:
5902:
5899:
5896:
5892:
5888:
5884:
5857:
5854:
5848:
5845:
5836:
5832:
5828:
5825:
5817:
5813:
5806:
5794:
5774:
5771:
5768:
5762:
5756:
5753:
5750:
5741:
5736:
5733:
5730:
5726:
5702:
5699:
5696:
5688:
5685:
5682:
5676:
5671:
5667:
5646:
5643:
5640:
5637:
5634:
5631:
5628:
5625:
5622:
5619:
5616:
5593:
5590:
5585:
5581:
5574:
5568:
5565:
5560:
5556:
5549:
5544:
5541:
5538:
5534:
5507:
5504:
5501:
5495:
5489:
5486:
5483:
5474:
5469:
5465:
5454:
5440:
5437:
5431:
5428:
5422:
5419:
5416:
5410:
5407:
5401:
5398:
5374:
5359:
5356:
5345:
5342:
5310:
5294:
5291:
5286:
5282:
5260:
5253:
5250:
5247:
5241:
5235:
5232:
5229:
5223:
5217:
5214:
5211:
5205:
5199:
5196:
5193:
5187:
5181:
5178:
5175:
5169:
5163:
5160:
5157:
5150:
5146:
5141:
5136:
5133:
5130:
5125:
5118:
5115:
5112:
5105:
5100:
5094:
5091:
5088:
5076:
5042:
5039:
5032:
5029:
5024:
5020:
5006:
5001:
4998:
4989:
4979:
4975:
4961:
4956:
4953:
4945:
4940:
4936:
4914:
4907:
4904:
4899:
4895:
4889:
4884:
4881:
4878:
4874:
4860:
4852:
4847:
4843:
4839:
4836:
4831:
4827:
4822:
4814:
4810:
4800:
4795:
4791:
4778:
4774:
4770:
4765:
4761:
4752:
4742:
4738:
4734:
4729:
4725:
4715:
4711:
4701:
4697:
4693:
4690:
4687:
4682:
4678:
4671:
4660:
4644:
4641:
4635:
4631:
4624:
4619:
4616:
4613:
4609:
4585:
4582:
4579:
4576:
4573:
4570:
4567:
4564:
4561:
4558:
4555:
4549:
4546:
4519:
4516:
4513:
4509:
4505:
4502:
4499:
4493:
4490:
4483:
4480:
4477:
4473:
4469:
4466:
4463:
4457:
4454:
4444:
4441:
4438:
4435:
4432:
4429:
4426:
4414:
4411:
4405:
4400:
4394:
4386:
4382:
4378:
4372:
4364:
4360:
4356:
4351:
4347:
4338:
4328:
4311:
4303:
4300:
4298:
4289:
4282:
4278:
4274:
4271:
4269:
4259:
4252:
4244:
4240:
4235:
4227:
4223:
4219:
4214:
4207:
4199:
4195:
4190:
4182:
4178:
4174:
4171:
4166:
4158:
4155:
4152:
4143:
4140:
4137:
4133:
4128:
4120:
4116:
4112:
4109:
4107:
4098:
4095:
4092:
4085:
4080:
4077:
4074:
4058:
4038:
4030:
4027:
4025:
4016:
4009:
4005:
4001:
3998:
3994:
3987:
3983:
3979:
3974:
3966:
3963:
3960:
3951:
3948:
3945:
3941:
3936:
3928:
3924:
3920:
3917:
3915:
3906:
3903:
3900:
3893:
3888:
3885:
3882:
3850:
3846:
3843:
3837:
3831:
3828:
3825:
3815:
3810:
3779:
3767:
3761:
3758:
3755:
3749:
3746:
3743:
3740:
3731:
3726:
3723:
3714:
3709:
3705:
3694:
3685:
3681:
3673:
3664:
3660:
3656:
3650:
3647:
3644:
3640:
3637:
3634:
3631:
3622:
3618:
3612:
3608:
3605:
3602:
3598:
3591:
3588:
3585:
3581:
3556:
3536:
3528:
3524:
3505:
3502:
3499:
3485:
3482:
3478:
3475:
3468:
3458:
3438:
3435:
3429:
3426:
3420:
3415:
3411:
3407:
3404:
3396:
3387:
3383:
3378:
3373:
3369:
3365:
3362:
3356:
3350:
3345:
3318:
3315:
3310:
3304:
3296:
3292:
3257:
3254:
3244:
3241:
3238:
3235:
3232:
3229:
3226:
3223:
3220:
3217:
3214:
3208:
3205:
3200:
3195:
3192:
3184:
3179:
3173:
3165:
3161:
3153:
3134:
3131:
3128:
3114:
3111:
3104:
3100:
3093:Fair die roll
3090:
3077:
3069:
3066:
3060:
3057:
3051:
3046:
3042:
3038:
3035:
3016:
3012:
3007:
3002:
2998:
2994:
2991:
2985:
2979:
2974:
2949:
2941:
2925:
2922:
2919:
2913:
2910:
2901:
2897:
2893:
2890:
2871:
2867:
2862:
2857:
2853:
2849:
2846:
2832:
2827:
2814:
2798:
2795:
2789:
2786:
2780:
2762:
2758:
2754:
2736:
2732:
2723:
2719:
2715:
2667:
2663:
2662:probabilities
2647:
2640:
2636:
2633:Consider the
2621:
2619:
2615:
2611:
2609:
2604:
2602:
2598:
2594:
2590:
2580:
2578:
2574:
2569:
2568:with itself.
2555:
2547:
2528:
2525:
2522:
2516:
2490:
2487:
2484:
2478:
2472:
2466:
2449:
2447:
2443:
2427:
2419:
2399:
2394:
2387:
2381:
2376:
2367:
2362:
2337:
2333:
2321:
2315:
2310:
2300:
2297:
2294:
2287:
2283:
2276:
2272:
2268:
2266:
2254:
2251:
2248:
2241:
2237:
2232:
2229:
2224:
2221:
2218:
2211:
2207:
2203:
2197:
2193:
2189:
2187:
2179:
2160:
2144:
2136:
2126:
2123:
2119:
2114:
2112:
2111:
2087:
2081:
2076:
2068:
2062:
2056:
2051:
2043:
2041:
2032:
2024:
2015:
2011:
2006:
2002:
1997:
1993:
1989:
1985:
1977:
1968:
1964:
1959:
1955:
1950:
1946:
1942:
1939:
1937:
1928:
1921:
1912:
1908:
1901:
1892:
1888:
1883:
1879:
1874:
1870:
1866:
1863:
1859:
1852:
1849:
1846:
1838:
1835:
1832:
1828:
1823:
1819:
1814:
1810:
1806:
1803:
1801:
1793:
1790:
1787:
1781:
1776:
1773:
1770:
1738:
1735:
1732:
1726:
1720:
1717:
1714:
1704:
1700:
1684:
1664:
1655:
1639:
1630:
1626:
1618:
1609:
1605:
1601:
1595:
1592:
1589:
1585:
1582:
1579:
1576:
1567:
1563:
1559:
1556:
1553:
1549:
1542:
1539:
1536:
1532:
1523:
1521:
1502:
1494:
1490:
1466:
1458:
1454:
1446:
1430:
1426:
1423:
1416:
1412:
1408:
1404:
1400:
1390:
1388:
1384:
1379:
1363:
1352:
1346:
1332:
1326:
1309:
1295:
1286:
1277:
1269:
1263:
1257:
1253:
1250:
1247:
1241:
1213:
1190:
1184:
1181:
1178:
1175:
1169:
1160:
1140:
1117:
1111:
1091:
1083:
1070:
1066:
1062:
1057:
1056:
1051:
1047:
1043:
1039:
1038:
1036:
1035:
1034:
1013:
1007:
1001:
998:
995:
987:
972:
969:
963:
957:
954:
951:
943:
942:
941:
921:
918:
892:
889:
886:
874:
872:
868:
864:
848:
839:
835:
832:This section
830:
827:
823:
822:
814:
797:
791:
768:
760:
756:
747:
728:
725:
722:
716:
693:
685:
681:
671:
658:
653:
645:
642:
639:
632:
628:
623:
618:
613:
610:
607:
602:
596:
593:
590:
583:
579:
574:
569:
566:
563:
560:
554:
548:
543:
518:
511:
495:
473:
470:
467:
460:
456:
448:
432:
423:
421:
415:
410:
405:
401:
396:
390:
385:
381:
376:
363:
358:
355:
352:
347:
342:
338:
334:
331:
326:
320:
317:
314:
305:
300:
295:
291:
287:
284:
278:
250:
243:
227:
220:
204:
201:
198:
186:
183:
180:
179:
178:
176:
167:
164:
162:
158:
153:
151:
150:
144:
142:
141:source coding
137:
132:
130:
126:
122:
118:
114:
110:
106:
102:
98:
94:
83:
80:
72:
62:
57:
53:
49:
48:
41:
32:
31:
19:
9725:
9718:C.E. Shannon
9698:. Retrieved
9694:
9685:
9673:
9670:
9667:Myron Tribus
9662:
9650:
9647:
9641:
9632:
9612:
9605:
9589:. Springer.
9585:
9578:
9551:
9530:
9525:
9494:
9392:
9158:
9105:
9054:
8910:
8653:
8496:
8357:
8230:intersection
8229:
8203:
7919:
7885:
7878:
7873:
7870:
7869:
7861:
7854:
7831:
7748:
7719:
7709:in terms of
7564:
7448:
7362:
7243:can take is
7082:
6951:equiprobable
6691:
6304:
5962:
5795:
5455:
5373:combinations
5077:
4971:when 2
4334:
3811:
3464:
3096:
2632:
2613:
2612:
2605:
2600:
2597:Myron Tribus
2588:
2586:
2570:
2450:
2132:
2120:is that the
2115:
2108:
1656:
1524:
1396:
1386:
1382:
1380:
1310:
1079:
1053:
1032:
875:
861:For a given
860:
845:October 2018
842:
838:adding to it
833:
672:
424:
413:
399:
388:
383:
379:
377:
190:
173:
165:
160:
154:
147:
145:
133:
108:
104:
100:
96:
90:
75:
66:
59:Please help
55:
44:
8683:results in
8234:independent
7839:with known
7834:independent
7365:categorical
7130:degenerates
3523:independent
2118:likelihoods
1699:independent
242:probability
113:probability
63:if you can.
9758:Categories
9700:2021-04-01
9653:, 434–449
9517:References
9013:functions
7851:Derivation
7845:additivity
7707:isomorphic
7616:normalized
5391:. Indeed,
5052:otherwise.
4921:otherwise,
4870:when
3786:otherwise.
2690:and tails
2159:defined as
1403:additivity
817:Properties
170:Definition
9570:608622533
9435:ω
9402:ω
9363:ω
9356:
9340:
9319:ω
9312:
9303:
9297:−
9282:ω
9275:
9243:ω
9236:
9207:ω
9177:ω
9170:
9083:
9031:
9011:logarithm
8961:⋅
8929:⋅
8877:
8871:⋅
8859:
8821:
8794:
8763:
8738:
8720:
8698:
8668:⋅
8628:
8622:⋅
8610:
8595:∩
8586:
8568:
8471:
8453:
8438:∩
8429:
8411:
8174:ω
8167:
8129:ω
8122:
8084:ω
8077:
8039:ω
8032:
8006:⋅
7968:ω
7961:
7937:ω
7930:
7899:ω
7794:
7781:−
7766:
7746:is given
7717:as well.
7682:…
7576:∈
7540:otherwise
7517:∈
7446:given by
7332:
7319:−
7304:
7196:δ
7051:
7025:
7012:−
6997:
6922:otherwise
6901:∩
6883:∈
6802:−
6767:≥
6750:∈
6711:∼
6662:≈
6654:
6626:
6613:−
6598:
6473:∗
6439:−
6428:−
6397:∗
6309:) of the
6278:0.2630344
6249:−
6232:
6205:2.5849625
6176:−
6159:
6042:¯
5980:⋃
5915:−
5885:
5829:−
5807:
5644:≤
5632:≤
5620:≠
5575:∩
5423:⋅
5402:⋅
5101:∈
4875:∑
4840:⋯
4753:…
4691:…
4610:∑
4550:∈
4510:∧
4474:⊻
4436:∨
4420:¬
4383:δ
4361:δ
4301:≈
4220:−
4175:−
4113:−
4086:
4028:≈
3921:−
3894:
3768:∩
3750:∈
3486:∼
3436:≈
3421:
3408:−
3379:
3366:−
3351:
3264:otherwise
3209:∈
3115:∼
3052:
3039:−
3008:
2995:−
2980:
2894:−
2863:
2850:−
2833:
2813:measuring
2714:fair coin
2589:surprisal
2517:
2479:
2444:over its
2382:
2363:
2316:
2273:∑
2233:
2204:−
2194:∑
2157:above is
2082:
2057:
2003:
1990:−
1956:
1943:−
1880:
1867:−
1820:
1807:−
1782:
1353:−
1344:¬
1284:¬
1254:
1182:−
1167:¬
1017:∞
1002:
996:−
958:
952:−
925:∞
614:
570:
564:−
549:
378:The base
348:
335:−
301:
288:−
105:surprisal
69:June 2017
9500:See also
9464:of base
9009:are the
7857:a priori
7060: Sh
6947:integers
6669: Sh
6665:3.169925
6465:, where
6282: Sh
6209: Sh
5948: Sh
5944:4.169925
5862: Sh
5858:5.169925
4308: Sh
4304:5.169925
4035: Sh
4031:5.169925
3074: Sh
2811:. Upon
2722:one half
2624:Examples
2593:surprise
2506:, where
2418:expected
1657:because
1389:happen.
1320:log-odds
1235:log-odds
1082:log-odds
1069:variance
1067:and its
129:log-odds
45:require
9669:(1961)
9490:hartley
9458:shannon
9126:is the
8236:events
8232:of two
8228:is the
8206:measure
8156:, then
8066:, then
7597:numbers
7372:support
7109:above,
6700:(DURV)
6491:outcome
6489:. The
5389:
5377:
5325:
5313:
4601:, then
2940:shannon
2664:of the
2544:is the
2446:support
1703:outcome
1411:measure
1383:doesn't
1046:lottery
510:outcome
420:hartley
395:shannon
217:and an
161:shannon
149:entropy
47:cleanup
9678:borrow
9620:
9593:
9568:
9558:
9106:where
8111:. If
7604:events
6823:. The
6761:
5307:and a
5046:
5010:
4965:
4864:
3735:
2724:each,
2720:) are
2666:events
2660:. The
2357:
867:events
95:, the
7888:event
7618:to a
7606:on a
7370:with
7132:to a
3439:2.585
3150:with
2712:(see
2583:Notes
1443:with
445:with
240:with
219:event
117:event
107:, or
9655:link
9618:ISBN
9591:ISBN
9566:OCLC
9556:ISBN
9141:<
8517:and
8380:and
8316:and
8256:and
8186:>
8141:<
7837:DRVs
7442:and
6239:Diff
6166:Same
6117:Diff
6102:and
6071:Same
6039:Same
6030:Diff
5972:Same
5963:Let
5718:and
5609:for
5526:and
4539:for
3549:and
2716:and
2571:For
2448:.
2133:The
2107:See
1697:are
1677:and
1482:and
1387:does
416:= 10
202:>
134:The
125:odds
9486:nat
9456:is
9337:log
9300:log
9114:log
9080:log
9022:log
8557:is
8358:sum
7843:by
7785:log
7641:.
7610:of
7323:log
7083:If
7042:log
7016:log
6827:is
6645:log
6617:log
6253:log
6180:log
5919:log
5833:log
5311:of
4279:log
4224:log
4179:log
4117:log
4006:log
3984:log
3925:log
3866:is
3529:of
3412:log
3370:log
3043:log
2999:log
2898:log
2854:log
2799:0.5
2637:of
2548:of
2230:log
1994:log
1947:log
1871:log
1811:log
1522:is
1251:log
1050:won
999:log
955:log
840:.
611:log
567:log
508:as
391:= 2
339:log
292:log
127:or
91:In
9760::
9724:,
9720:,
9693:.
9651:57
9564:.
9538:^
9492:.
9156:.
8201:.
7890:,
7847:.
7343:0.
7286:is
6979:is
6796::=
6553:36
6451:36
6110:Pr
6064:Pr
5935:18
5849:36
5793:.
5432:18
5420:15
5411:36
5386:18
5360:15
5002:36
4957:18
4815:18
4719:Pr
4357::=
3995:36
3727:36
3626:Pr
3569:is
3444:Sh
2962:is
2620:.
2603:.
2579:.
1754:is
1571:Pr
1058:.)
561::=
402:=
310:Pr
285::=
103:,
99:,
9703:.
9680:.
9657:.
9626:.
9599:.
9572:.
9472:e
9444:)
9439:n
9431:(
9428:I
9406:n
9378:)
9372:)
9367:n
9359:(
9353:P
9349:1
9344:(
9334:=
9331:)
9328:)
9323:n
9315:(
9309:P
9306:(
9294:=
9291:)
9286:n
9278:(
9272:I
9252:)
9247:n
9239:(
9233:P
9211:n
9186:)
9181:n
9173:(
9167:I
9144:0
9138:K
9092:)
9089:x
9086:(
9077:K
9074:=
9071:)
9068:x
9065:(
9062:f
9040:)
9037:x
9034:(
9026:b
8997:)
8994:y
8991:(
8988:f
8985:+
8982:)
8979:x
8976:(
8973:f
8970:=
8967:)
8964:y
8958:x
8955:(
8952:f
8932:)
8926:(
8923:f
8891:)
8886:)
8883:B
8880:(
8874:P
8868:)
8865:A
8862:(
8856:P
8851:(
8846:f
8843:=
8833:)
8830:)
8827:B
8824:(
8818:P
8815:(
8812:f
8809:+
8806:)
8803:)
8800:A
8797:(
8791:P
8788:(
8785:f
8782:=
8775:)
8772:)
8769:C
8766:(
8760:P
8757:(
8754:f
8747:)
8744:B
8741:(
8735:I
8732:+
8729:)
8726:A
8723:(
8717:I
8714:=
8707:)
8704:C
8701:(
8695:I
8671:)
8665:(
8662:f
8640:.
8637:)
8634:B
8631:(
8625:P
8619:)
8616:A
8613:(
8607:P
8604:=
8601:)
8598:B
8592:A
8589:(
8583:P
8580:=
8577:)
8574:C
8571:(
8565:P
8545:C
8525:B
8505:A
8483:.
8480:)
8477:B
8474:(
8468:I
8465:+
8462:)
8459:A
8456:(
8450:I
8447:=
8444:)
8441:B
8435:A
8432:(
8426:I
8423:=
8420:)
8417:C
8414:(
8408:I
8388:B
8368:A
8344:C
8324:B
8304:A
8284:C
8264:B
8244:A
8216:C
8189:0
8183:)
8178:n
8170:(
8164:I
8144:1
8138:)
8133:n
8125:(
8119:P
8099:0
8096:=
8093:)
8088:n
8080:(
8074:I
8054:1
8051:=
8048:)
8043:n
8035:(
8029:P
8009:)
8003:(
8000:f
7980:)
7977:)
7972:n
7964:(
7958:P
7955:(
7952:f
7949:=
7946:)
7941:n
7933:(
7927:I
7903:n
7866::
7818:.
7814:)
7811:x
7808:(
7803:X
7799:p
7789:2
7778:=
7775:)
7772:x
7769:(
7761:X
7757:I
7734:x
7731:=
7728:X
7692:}
7688:N
7685:,
7679:,
7676:2
7673:,
7670:1
7666:{
7662:=
7659:]
7656:N
7653:[
7629:p
7581:S
7573:s
7544:.
7534:,
7531:0
7522:S
7512:i
7508:s
7504:=
7501:k
7496:,
7491:i
7487:p
7480:{
7475:=
7472:)
7469:k
7466:(
7461:X
7457:p
7428:N
7423:1
7420:=
7417:i
7411:}
7403:i
7399:s
7393:{
7388:=
7383:S
7340:=
7336:1
7327:2
7316:=
7313:)
7310:b
7307:(
7299:X
7295:I
7274:X
7254:b
7231:X
7211:)
7208:k
7205:(
7200:b
7192:=
7189:)
7186:k
7183:(
7178:X
7174:p
7150:b
7147:=
7144:X
7117:X
7097:a
7094:=
7091:b
7064:.
7055:N
7046:2
7038:=
7033:N
7030:1
7020:2
7009:=
7006:)
7003:k
7000:(
6992:X
6988:I
6967:k
6964:=
6961:X
6926:.
6916:,
6913:0
6905:Z
6898:]
6895:b
6892:,
6889:a
6886:[
6880:k
6875:,
6870:N
6867:1
6859:{
6854:=
6851:)
6848:k
6845:(
6840:X
6836:p
6811:1
6808:+
6805:a
6799:b
6793:N
6773:.
6770:a
6764:b
6758:,
6754:Z
6747:b
6744:,
6741:a
6737:;
6734:]
6731:b
6728:,
6725:a
6722:[
6718:U
6715:D
6708:X
6673:.
6658:9
6649:2
6641:=
6635:9
6632:1
6621:2
6610:=
6607:)
6604:5
6601:(
6593:Z
6589:I
6566:9
6563:1
6558:=
6550:4
6545:=
6542:)
6539:5
6536:(
6531:Z
6527:p
6506:5
6503:=
6500:Z
6446:|
6442:7
6436:z
6432:|
6425:6
6419:=
6416:)
6413:y
6410:(
6405:Y
6401:p
6394:)
6391:x
6388:(
6383:X
6379:p
6375:=
6372:)
6369:z
6366:(
6361:Z
6357:p
6336:Y
6333:+
6330:X
6327:=
6324:Z
6286:.
6275:=
6269:6
6266:5
6257:2
6246:=
6243:)
6235:(
6229:I
6202:=
6196:6
6193:1
6184:2
6173:=
6170:)
6162:(
6156:I
6133:6
6130:5
6124:=
6121:)
6113:(
6087:6
6084:1
6078:=
6075:)
6067:(
6034:=
6006:i
6002:A
5995:6
5990:1
5987:=
5984:i
5976:=
5941:=
5932:1
5923:2
5912:=
5908:)
5903:4
5900:,
5897:3
5893:B
5889:(
5882:I
5855:=
5846:1
5837:2
5826:=
5823:)
5818:2
5814:A
5810:(
5804:I
5781:}
5778:)
5775:3
5772:,
5769:4
5766:(
5763:,
5760:)
5757:4
5754:,
5751:3
5748:(
5745:{
5742:=
5737:4
5734:,
5731:3
5727:B
5706:}
5703:2
5700:=
5697:Y
5689:2
5686:=
5683:X
5680:{
5677:=
5672:2
5668:A
5647:6
5641:k
5638:,
5635:j
5629:1
5626:,
5623:k
5617:j
5597:}
5594:1
5591:=
5586:k
5582:c
5578:{
5572:}
5569:1
5566:=
5561:j
5557:c
5553:{
5550:=
5545:k
5542:,
5539:j
5535:B
5514:}
5511:)
5508:k
5505:,
5502:k
5499:(
5496:=
5493:)
5490:Y
5487:,
5484:X
5481:(
5478:{
5475:=
5470:k
5466:A
5441:1
5438:=
5429:1
5417:+
5408:1
5399:6
5383:/
5380:1
5357:=
5351:)
5346:2
5343:6
5338:(
5322:6
5319:/
5316:1
5295:2
5292:=
5287:k
5283:C
5261:}
5257:)
5254:6
5251:,
5248:6
5245:(
5242:,
5239:)
5236:5
5233:,
5230:5
5227:(
5224:,
5221:)
5218:4
5215:,
5212:4
5209:(
5206:,
5203:)
5200:3
5197:,
5194:3
5191:(
5188:,
5185:)
5182:2
5179:,
5176:2
5173:(
5170:,
5167:)
5164:1
5161:,
5158:1
5155:(
5151:{
5147:=
5142:6
5137:1
5134:=
5131:k
5126:}
5122:)
5119:k
5116:,
5113:k
5110:(
5106:{
5098:)
5095:Y
5092:,
5089:X
5086:(
5043:,
5040:0
5033:2
5030:=
5025:k
5021:c
5007:,
4999:1
4990:1
4980:k
4976:c
4962:,
4954:1
4946:{
4941:=
4915:0
4908:2
4905:=
4900:i
4896:c
4890:6
4885:1
4882:=
4879:i
4861:,
4853:!
4848:k
4844:c
4837:!
4832:1
4828:c
4823:1
4811:1
4801:{
4796:=
4784:)
4779:6
4775:c
4771:=
4766:6
4762:C
4743:1
4739:c
4735:=
4730:1
4726:C
4722:(
4716:=
4707:)
4702:6
4698:c
4694:,
4688:,
4683:1
4679:c
4675:(
4672:f
4645:2
4642:=
4636:k
4632:C
4625:6
4620:1
4617:=
4614:k
4589:}
4586:6
4583:,
4580:5
4577:,
4574:4
4571:,
4568:3
4565:,
4562:2
4559:,
4556:1
4553:{
4547:k
4520:k
4517:=
4514:Y
4506:k
4503:=
4500:X
4494:,
4491:2
4484:k
4481:=
4478:Y
4470:k
4467:=
4464:X
4458:,
4455:1
4448:)
4445:k
4442:=
4439:Y
4433:k
4430:=
4427:X
4424:(
4415:,
4412:0
4406:{
4401:=
4398:)
4395:Y
4392:(
4387:k
4379:+
4376:)
4373:X
4370:(
4365:k
4352:k
4348:C
4312:.
4290:6
4283:2
4275:2
4272:=
4260:]
4256:)
4253:4
4250:(
4245:Y
4241:p
4236:[
4228:2
4215:]
4211:)
4208:2
4205:(
4200:X
4196:p
4191:[
4183:2
4172:=
4167:]
4162:)
4159:4
4156:,
4153:2
4150:(
4144:Y
4141:,
4138:X
4134:p
4129:[
4121:2
4110:=
4102:)
4099:4
4096:,
4093:2
4090:(
4081:Y
4078:,
4075:X
4071:I
4039:,
4017:6
4010:2
4002:2
3999:=
3988:2
3980:=
3975:]
3970:)
3967:4
3964:,
3961:2
3958:(
3952:Y
3949:,
3946:X
3942:p
3937:[
3929:2
3918:=
3910:)
3907:4
3904:,
3901:2
3898:(
3889:Y
3886:,
3883:X
3879:I
3854:)
3851:4
3847:,
3844:2
3841:(
3838:=
3835:)
3832:Y
3829:,
3826:X
3823:(
3780:0
3772:N
3765:]
3762:6
3759:,
3756:1
3753:[
3747:y
3744:,
3741:x
3732:,
3724:1
3715:{
3710:=
3698:)
3695:y
3692:(
3686:Y
3682:p
3677:)
3674:x
3671:(
3665:X
3661:p
3657:=
3654:)
3651:y
3648:=
3645:Y
3641:,
3638:x
3635:=
3632:X
3629:(
3623:=
3613:)
3609:y
3606:,
3603:x
3599:(
3592:Y
3589:,
3586:X
3582:p
3557:Y
3537:X
3509:]
3506:6
3503:,
3500:1
3497:[
3493:U
3490:D
3483:Y
3479:,
3476:X
3430:6
3427:1
3416:2
3405:=
3400:)
3397:4
3394:(
3388:X
3384:p
3374:2
3363:=
3360:)
3357:4
3354:(
3346:X
3342:I
3319:6
3316:1
3311:=
3308:)
3305:4
3302:(
3297:X
3293:p
3258:,
3255:0
3248:}
3245:6
3242:,
3239:5
3236:,
3233:4
3230:,
3227:3
3224:,
3221:2
3218:,
3215:1
3212:{
3206:k
3201:,
3196:6
3193:1
3185:{
3180:=
3177:)
3174:k
3171:(
3166:X
3162:p
3138:]
3135:6
3132:,
3129:1
3126:[
3122:U
3119:D
3112:X
3078:.
3070:1
3067:=
3061:2
3058:1
3047:2
3036:=
3031:)
3027:T
3023:(
3017:X
3013:p
3003:2
2992:=
2989:)
2986:T
2983:(
2975:X
2971:I
2950:T
2926:,
2923:1
2920:=
2914:2
2911:1
2902:2
2891:=
2886:)
2882:H
2878:(
2872:X
2868:p
2858:2
2847:=
2844:)
2840:H
2836:(
2828:X
2824:I
2796:=
2790:2
2787:1
2781:=
2777:)
2773:T
2769:(
2763:X
2759:p
2755:=
2751:)
2747:H
2743:(
2737:X
2733:p
2699:T
2677:H
2648:X
2556:X
2532:)
2529:X
2526:;
2523:X
2520:(
2514:I
2494:)
2491:X
2488:;
2485:X
2482:(
2476:I
2473:=
2470:)
2467:X
2464:(
2460:H
2428:X
2400:,
2395:]
2391:)
2388:X
2385:(
2377:X
2373:I
2368:[
2360:E
2350:f
2347:e
2344:d
2338:=
2325:)
2322:x
2319:(
2311:X
2307:I
2301:)
2298:x
2295:(
2288:X
2284:p
2277:x
2269:=
2255:)
2252:x
2249:(
2242:X
2238:p
2225:)
2222:x
2219:(
2212:X
2208:p
2198:x
2190:=
2183:)
2180:X
2177:(
2173:H
2145:X
2091:)
2088:y
2085:(
2077:Y
2073:I
2069:+
2066:)
2063:x
2060:(
2052:X
2048:I
2044:=
2033:]
2028:)
2025:y
2022:(
2016:Y
2012:p
2007:[
1998:2
1986:]
1981:)
1978:x
1975:(
1969:X
1965:p
1960:[
1951:2
1940:=
1929:]
1925:)
1922:y
1919:(
1913:Y
1909:p
1905:)
1902:x
1899:(
1893:X
1889:p
1884:[
1875:2
1864:=
1860:]
1856:)
1853:y
1850:,
1847:x
1844:(
1839:Y
1836:,
1833:X
1829:p
1824:[
1815:2
1804:=
1797:)
1794:y
1791:,
1788:x
1785:(
1777:Y
1774:,
1771:X
1767:I
1742:)
1739:y
1736:,
1733:x
1730:(
1727:=
1724:)
1721:Y
1718:,
1715:X
1712:(
1685:Y
1665:X
1643:)
1640:y
1637:(
1631:Y
1627:p
1622:)
1619:x
1616:(
1610:X
1606:p
1602:=
1599:)
1596:y
1593:=
1590:Y
1586:,
1583:x
1580:=
1577:X
1574:(
1568:=
1564:)
1560:y
1557:,
1554:x
1550:(
1543:Y
1540:,
1537:X
1533:p
1506:)
1503:y
1500:(
1495:Y
1491:p
1470:)
1467:x
1464:(
1459:X
1455:p
1431:Y
1427:,
1424:X
1367:)
1364:x
1361:(
1357:I
1350:)
1347:x
1341:(
1337:I
1333:=
1330:)
1327:x
1324:(
1296:)
1290:)
1287:x
1281:(
1278:p
1273:)
1270:x
1267:(
1264:p
1258:(
1248:=
1245:)
1242:x
1239:(
1214:x
1194:)
1191:x
1188:(
1185:p
1179:1
1176:=
1173:)
1170:x
1164:(
1161:p
1141:x
1121:)
1118:x
1115:(
1112:p
1092:x
1071:.
1014:=
1011:)
1008:0
1005:(
973:0
970:=
967:)
964:1
961:(
928:]
922:,
919:0
916:[
896:]
893:1
890:,
887:0
884:[
847:)
843:(
801:)
798:X
795:(
792:H
772:)
769:x
766:(
761:X
757:h
732:)
729:Y
726:;
723:X
720:(
717:I
697:)
694:x
691:(
686:X
682:I
659:.
654:)
646:)
643:x
640:(
633:X
629:p
624:1
619:(
608:=
603:]
597:)
594:x
591:(
584:X
580:p
575:[
558:)
555:x
552:(
544:X
540:I
519:x
496:X
474:)
471:x
468:(
461:X
457:p
433:X
414:b
404:e
400:b
389:b
384:b
380:b
364:.
359:)
356:P
353:(
343:b
332:=
327:]
321:)
318:x
315:(
306:[
296:b
282:)
279:x
276:(
272:I
251:P
228:x
205:1
199:b
82:)
76:(
71:)
67:(
20:)
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.
