2061:
2081:
38:
443:
1021:
164:
732:
73:
of the test, where the precision is the number of true positive results divided by the number of all samples predicted to be positive, including those not identified correctly, and the recall is the number of true positive results divided by the number of all samples that should have been identified
2339:
ignores the True
Negatives and thus is misleading for unbalanced classes, while kappa and correlation measures are symmetric and assess both directions of predictability - the classifier predicting the true class and the true class predicting the classifier prediction, proposing separate multiclass
2403:
Macro F1 is a macro-averaged F1 score. To calculate macro F1, two different averaging-formulas have been used: the F-score of (arithmetic) class-wise precision and recall means or the arithmetic mean of class-wise F-scores, where the latter exhibits more desirable properties.
748:
438:{\displaystyle F_{1}={\frac {2}{\mathrm {recall} ^{-1}+\mathrm {precision} ^{-1}}}=2{\frac {\mathrm {precision} \cdot \mathrm {recall} }{\mathrm {precision} +\mathrm {recall} }}={\frac {2\mathrm {tp} }{2\mathrm {tp} +\mathrm {fp} +\mathrm {fn} }}}
2140:
of positive to negative test cases. This means that comparison of the F-score across different problems with differing class ratios is problematic. One way to address this issue (see e.g., Siblini et al., 2020 ) is to use a standard class ratio
2287:-score of a classifier which always predicts the positive class is equal to 2 * proportion_of_positive_class / ( 1 + proportion_of_positive_class ), since the recall is 1, and the precision is equal to the proportion of the positive class.
2317:
score since it gives equal importance to precision and recall. In practice, different types of mis-classifications incur different costs. In other words, the relative importance of precision and recall is an aspect of the problem.
551:
1182:
1016:{\displaystyle F_{\beta }={\frac {(1+\beta ^{2})\cdot \mathrm {true\ positive} }{(1+\beta ^{2})\cdot \mathrm {true\ positive} +\beta ^{2}\cdot \mathrm {false\ negative} +\mathrm {false\ positive} }}\,}
1273:
2355:
is its lack of symmetry. It means it may change its value when dataset labeling is changed - the "positive" samples are named "negative" and vice versa. This criticism is met by the
2192:
performance. It is particularly relevant in applications which are primarily concerned with the positive class and where the positive class is rare relative to the negative class.
2290:
If the scoring model is uninformative (cannot distinguish between the positive and negative class) then the optimal threshold is 0 so that the positive class is always predicted.
2486:
1225:
2224:
2118:
1075:
483:
121:
3167:"The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation"
1095:
1045:
543:
523:
503:
2166:
2816:
2138:
126:
The highest possible value of an F-score is 1.0, indicating perfect precision and recall, and the lowest possible value is 0, if precision and recall are zero.
727:{\displaystyle F_{\beta }=(1+\beta ^{2})\cdot {\frac {\mathrm {precision} \cdot \mathrm {recall} }{(\beta ^{2}\cdot \mathrm {precision} )+\mathrm {recall} }}}
3284:
Siblini, W.; Fréry, J.; He-Guelton, L.; Oblé, F.; Wang, Y. Q. (2020). "Master your metrics with calibration". In M. Berthold; A. Feelders; G. Krempl (eds.).
3774:
2332:
2199:
score, but with the proliferation of large scale search engines, performance goals changed to place more emphasis on either precision or recall and so
1107:
2479:
3759:
1525:
2472:
2615:
3764:
1973:
3088:
Brooks, Harold; Brown, Barb; Ebert, Beth; Ferro, Chris; Jolliffe, Ian; Koh, Tieh-Yong; Roebber, Paul; Stephenson, David (2015-01-26).
3498:
Lipton, Z.C., Elkan, C.P., & Narayanaswamy, B. (2014). F1-Optimal
Thresholding in the Multi-Label Setting. ArXiv, abs/1402.1892.
2522:
2423:
2280:-score of a classifier which always predicts the positive class converges to 1 as the probability of the positive class increases.
2630:
134:
The name F-measure is believed to be named after a different F function in Van
Rijsbergen's book, when introduced to the Fourth
3610:
Powers, David M W (2011). "Evaluation: From
Precision, Recall and F-Score to ROC, Informedness, Markedness & Correlation".
3249:
Brabec, Jan; Komárek, Tomáš; Franc, Vojtěch; Machlica, Lukáš (2020). "On model evaluation under non-constant class imbalance".
2084:
Precision-Recall Curve: points from different thresholds are color coded, the point with optimal F-score is highlighted in red
3561:"The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation"
3116:"The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation"
3072:
96:
of the precision and recall. It thus symmetrically represents both precision and recall in one metric. The more generic
2882:
1230:
3365:
2620:
2438:
1631:
66:
2326:
2234:
1947:
135:
2842:
2703:
1739:
2527:
2395:). A common method is to average the F-score over each class, aiming at a balanced measurement of performance.
1411:
70:
2537:
2248:
The F-score has been widely used in the natural language processing literature, such as in the evaluation of
3693:"A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice"
2793:
3779:
3471:
2552:
155:
3769:
2759:
2688:
2625:
79:
2572:
2547:
2532:
2053:
Type I error: A test result which wrongly indicates that a particular condition or attribute is present
2026:
Type II error: A test result which wrongly indicates that a particular condition or attribute is absent
3034:"Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation"
2562:
2557:
2392:
2376:
2310:
1931:
1792:
1712:
1687:
1625:
1047:
are 2, which weighs recall higher than precision, and 0.5, which weighs recall lower than precision.
739:
75:
1191:
3329:
2658:
2249:
17:
2749:
2653:
2648:
2448:
2185:
1325:
2202:
2096:
1053:
461:
99:
3509:
3444:
3324:
3015:"Data Science for Business: What You Need to Know about Data Mining and Data-Analytic Thinking"
2590:
1306:
2233:. However, the F-measures do not take true negatives into account, hence measures such as the
2177:
1833:
1284:
54:
50:
2391:
The F-score is also used for evaluating classification problems with more than two classes (
1080:
1030:
528:
508:
488:
2811:
2711:
2600:
2595:
2189:
2144:
1764:
1338:
1098:
123:
score applies additional weights, valuing one of precision or recall more than the other.
8:
2517:
2504:
1659:
1567:
1518:
1449:
1415:
3733:
3704:
3638:
3587:
3560:
3535:
3465:
3402:
3371:
3289:
3254:
3193:
3166:
3142:
3115:
2995:
2931:
2904:
2837:
2803:
2683:
2512:
2123:
1561:
1407:
2464:
2060:
3592:
3510:"A note on using the F-measure for evaluating record linkage algorithms - Dimensions"
3361:
3198:
3147:
3068:
3033:
3014:
2936:
2905:"Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool"
2832:
2721:
2668:
2253:
1421:
3539:
2017:
A test result that correctly indicates the presence of a condition or characteristic
3714:
3671:
3619:
3582:
3572:
3525:
3517:
3375:
3353:
3299:
3264:
3229:
3188:
3178:
3137:
3127:
3060:
2999:
2987:
2926:
2916:
2855:
2640:
2605:
2542:
2495:
2418:
2298:
2269:
2242:
2230:
2044:
A test result that correctly indicates the absence of a condition or characteristic
3089:
3304:
2991:
2972:
2788:
2731:
2610:
2454:
1177:{\displaystyle E=1-\left({\frac {\alpha }{p}}+{\frac {1-\alpha }{r}}\right)^{-1}}
3430:
Proceedings of the
International Conference on Language Resources and Evaluation
3269:
3183:
2443:
2380:
2080:
1495:
1396:
3676:
3659:
3577:
3521:
3234:
3217:
3132:
3064:
2955:
2921:
3753:
3389:
2673:
2433:
2372:
1507:
1459:
1384:
1077:"measures the effectiveness of retrieval with respect to a user who attaches
93:
3423:
3357:
3596:
3530:
3485:
3202:
3151:
2940:
2341:
2238:
1533:
3718:
2348:
for the two directions, noting that their geometric mean is correlation.
2359:
definition, which is sometimes indicated as a symmetrical extension of F
3692:
3637:
Sitarz, Mikolaj (2022). "Extending F1 metric, probabilistic approach".
2767:
2678:
2663:
2345:
2181:
1820:
1600:
46:
3623:
2585:
2356:
3738:
3709:
3643:
3407:
3294:
3259:
2245:
may be preferred to assess the performance of a binary classifier.
1885:
65:
is a measure of predictive performance. It is calculated from the
37:
3401:
Powers, David M. W (2015). "What the F-measure doesn't measure".
3090:"WWRP/WGNE Joint Working Group on Forecast Verification Research"
3055:
Ting, Kai Ming (2011). Sammut, Claude; Webb, Geoffrey I. (eds.).
2693:
1097:
times as much importance to recall as precision". It is based on
3486:"What is the baseline of the F1 score for a binary classifier?"
2428:
2088:
31:
3697:
Transactions of the
Association for Computational Linguistics
2459:
2272:
of the set of retrieved items and the set of relevant items.
2772:
2744:
2739:
2716:
2413:
3283:
3248:
2386:
3094:
Collaboration for
Australian Weather and Climate Research
2366:
3452:. Exercise 8.7: Cambridge University Press. p. 200
3343:
2494:
3087:
2321:
According to Davide Chicco and
Giuseppe Jurman, the F
2205:
2147:
2126:
2099:
1233:
1194:
1110:
1083:
1056:
1033:
751:
554:
531:
511:
491:
464:
167:
102:
3732:
J. Opitz; S. Burst (2019). "Macro F1 and Macro F1".
3346:
Learning query intent from regularized click graphs
3164:
1420:probability of detection, hit rate,
3012:
2218:
2160:
2132:
2112:
1267:
1219:
1176:
1089:
1069:
1039:
1015:
726:
537:
517:
497:
477:
437:
115:
3251:International Conference on Computational Science
1972:Threat score (TS), critical success index (CSI),
1268:{\displaystyle \alpha ={\frac {1}{1+\beta ^{2}}}}
3751:
3731:
3242:
2325:score is less truthful and informative than the
2313:and others criticize the widespread use of the F
3165:Chicco D, Toetsch N, Jurman G (February 2021).
146:The traditional F-measure or balanced F-score (
3415:
2953:
2480:
2035:the number of real negative cases in the data
2008:the number of real positive cases in the data
1287:where recall is often termed "sensitivity".
3558:
3425:Complementarity, F-score, and NLP Evaluation
3321:On Understanding and Classifying Web Queries
3113:
2089:Dependence of the F-score on class imbalance
3286:Advances in Intelligent Data Analysis XVIII
3013:Provost, Foster; Tom Fawcett (2013-08-01).
1524:probability of false alarm,
3603:
3421:
3277:
3215:
2487:
2473:
2176:The F-score is often used in the field of
3775:Summary statistics for contingency tables
3737:
3708:
3675:
3642:
3586:
3576:
3529:
3406:
3344:X. Li; Y.-Y. Wang; A. Acero (July 2008).
3328:
3303:
3293:
3268:
3258:
3233:
3192:
3182:
3141:
3131:
2930:
2920:
2902:
1012:
525:is chosen such that recall is considered
3612:Journal of Machine Learning Technologies
3446:An Introduction to Information Retrieval
3350:Proceedings of the 31st SIGIR Conference
3318:
3038:Journal of Machine Learning Technologies
2892:. Vol. 1, no. 5. pp. 1–5.
2195:Earlier works focused primarily on the F
2120:score, explicitly depends on the ratio
2079:
2059:
74:as positive. Precision is also known as
36:
27:Statistical measure of a test's accuracy
3760:Statistical natural language processing
3657:
3442:
2970:
2387:Extension to multi-class classification
14:
3752:
3636:
3609:
3443:Manning, Christopher (April 1, 2009).
3400:
3031:
2960:(2nd ed.). Butterworth-Heinemann.
2880:
2327:Matthews correlation coefficient (MCC)
3690:
2468:
2367:Difference from Fowlkes–Mallows index
2329:in binary evaluation classification.
2093:Precision-recall curve, and thus the
1278:
545:times as important as precision, is:
82:in diagnostic binary classification.
3054:
2072:is recall and the vertical axis is F
2064:Normalised harmonic mean plot where
3660:"Classification assessment methods"
3559:Chicco D, Jurman G (January 2020).
3218:"Classification assessment methods"
3114:Chicco D, Jurman G (January 2020).
3096:. World Meteorological Organisation
485:, that uses a positive real factor
24:
1050:The F-measure was derived so that
1005:
1002:
999:
996:
993:
990:
987:
984:
978:
975:
972:
969:
966:
958:
955:
952:
949:
946:
943:
940:
937:
931:
928:
925:
922:
919:
898:
895:
892:
889:
886:
883:
880:
877:
871:
868:
865:
862:
830:
827:
824:
821:
818:
815:
812:
809:
803:
800:
797:
794:
717:
714:
711:
708:
705:
702:
691:
688:
685:
682:
679:
676:
673:
670:
667:
644:
641:
638:
635:
632:
629:
621:
618:
615:
612:
609:
606:
603:
600:
597:
428:
425:
417:
414:
406:
403:
393:
390:
373:
370:
367:
364:
361:
358:
350:
347:
344:
341:
338:
335:
332:
329:
326:
319:
316:
313:
310:
307:
304:
296:
293:
290:
287:
284:
281:
278:
275:
272:
246:
243:
240:
237:
234:
231:
228:
225:
222:
204:
201:
198:
195:
192:
189:
25:
3791:
3765:Evaluation of machine translation
3664:Applied Computing and Informatics
3388:See, e.g., the evaluation of the
3222:Applied Computing and Informatics
2973:"An Introduction to ROC Analysis"
2439:Receiver operating characteristic
3507:
3057:Encyclopedia of machine learning
2235:Matthews correlation coefficient
1948:Matthews correlation coefficient
1283:This is related to the field of
136:Message Understanding Conference
3725:
3684:
3651:
3630:
3552:
3501:
3492:
3478:
3436:
3394:
3382:
3337:
3312:
3209:
3158:
2843:Pearson correlation coefficient
2351:Another source of critique of F
2171:
2047:
2038:
2029:
2020:
2011:
2002:
1289:
30:For the significance test, see
3288:. Springer. pp. 457–469.
3107:
3081:
3048:
3025:
3006:
2964:
2954:Van Rijsbergen, C. J. (1979).
2947:
2896:
2874:
2168:when making such comparisons.
1220:{\displaystyle F_{\beta }=1-E}
855:
836:
787:
768:
695:
650:
587:
568:
78:, and recall is also known as
13:
1:
2867:
2782:Deep Learning Related Metrics
2424:Hypothesis tests for accuracy
2375:of recall and precision, the
2259:
2226:is seen in wide application.
1027:Two commonly used values for
141:
3319:Beitzel., Steven M. (2006).
3305:10.1007/978-3-030-44584-3_36
3253:. Springer. pp. 74–87.
3032:Powers, David M. W. (2011).
2992:10.1016/j.patrec.2005.10.010
2883:"The truth of the F-measure"
2305:
2229:The F-score is also used in
129:
7:
3270:10.1007/978-3-030-50423-6_6
2980:Pattern Recognition Letters
2626:Sensitivity and specificity
2407:
2398:
2371:While the F-measure is the
2076:score, in percentage points
1502:false alarm, overestimation
1330:bookmaker informedness (BM)
449:
10:
3796:
3216:Tharwat A. (August 2018).
3184:10.1186/s13040-021-00244-z
2301:in the true positive rate.
2219:{\displaystyle F_{\beta }}
2113:{\displaystyle F_{\beta }}
1070:{\displaystyle F_{\beta }}
478:{\displaystyle F_{\beta }}
116:{\displaystyle F_{\beta }}
29:
3677:10.1016/j.aci.2018.08.003
3658:Tharwat A (August 2018).
3578:10.1186/s12864-019-6413-7
3522:10.1007/s11222-017-9746-6
3235:10.1016/j.aci.2018.08.003
3133:10.1186/s12864-019-6413-7
3065:10.1007/978-0-387-30164-8
2922:10.1186/s12880-015-0068-x
2903:Aziz Taha, Abdel (2015).
2851:
2825:
2802:
2781:
2758:
2730:
2702:
2639:
2571:
2503:
2393:Multiclass classification
1971:
1793:Negative predictive value
1713:Negative likelihood ratio
1688:Positive likelihood ratio
1626:Positive predictive value
1597:
1370:
1301:
1296:
1292:
1101:'s effectiveness measure
740:Type I and type II errors
158:of precision and recall:
76:positive predictive value
3470:: CS1 maint: location (
2250:named entity recognition
458:A more general F score,
3422:Derczynski, L. (2016).
3358:10.1145/1390334.1390393
2654:Calinski-Harabasz index
2449:Uncertainty coefficient
2186:document classification
1863:Balanced accuracy (BA)
1460:type II error
1321:Predicted negative (PN)
1316:Predicted positive (PP)
2335:has pointed out that F
2220:
2162:
2134:
2114:
2085:
2077:
1534:type I error
1269:
1221:
1188:Their relationship is
1178:
1091:
1090:{\displaystyle \beta }
1071:
1041:
1040:{\displaystyle \beta }
1017:
728:
539:
538:{\displaystyle \beta }
519:
518:{\displaystyle \beta }
499:
498:{\displaystyle \beta }
479:
439:
117:
42:
3323:(Ph.D. thesis). IIT.
2971:Fawcett, Tom (2006).
2957:Information Retrieval
2817:Intra-list Similarity
2377:Fowlkes–Mallows index
2221:
2178:information retrieval
2163:
2161:{\displaystyle r_{0}}
2135:
2115:
2083:
2063:
1966:FNR × FPR × FOR × FDR
1957:TPR × TNR × PPV × NPV
1932:Fowlkes–Mallows index
1403:miss, underestimation
1285:binary classification
1270:
1222:
1179:
1092:
1072:
1042:
1018:
729:
540:
520:
500:
480:
440:
118:
55:information retrieval
51:binary classification
40:
3719:10.1162/tacl_a_00675
3691:Opitz, Juri (2024).
2203:
2190:query classification
2145:
2124:
2097:
1765:False discovery rate
1339:Prevalence threshold
1231:
1192:
1108:
1081:
1054:
1031:
749:
552:
529:
509:
489:
462:
165:
100:
41:Precision and recall
3780:Clustering criteria
3019:O'Reilly Media, Inc
2909:BMC Medical Imaging
2881:Sasaki, Y. (2007).
1660:False omission rate
1519:False positive rate
1450:False negative rate
1298:Predicted condition
3770:Statistical ratios
2838:Euclidean distance
2804:Recommender system
2684:Similarity measure
2498:evaluation metrics
2216:
2158:
2130:
2110:
2086:
2078:
1570:(SPC), selectivity
1562:True negative rate
1408:True positive rate
1279:Diagnostic testing
1265:
1217:
1174:
1087:
1067:
1037:
1013:
724:
535:
515:
495:
475:
435:
113:
43:
3514:app.dimensions.ai
3074:978-0-387-30164-8
2890:Teach tutor mater
2864:
2863:
2833:Cosine similarity
2669:Hopkins statistic
2451:, aka Proficiency
2254:word segmentation
2133:{\displaystyle r}
1998:
1997:
1514:correct rejection
1263:
1158:
1137:
1010:
983:
936:
876:
808:
722:
433:
378:
260:
16:(Redirected from
3787:
3744:
3743:
3741:
3729:
3723:
3722:
3712:
3688:
3682:
3681:
3679:
3655:
3649:
3648:
3646:
3634:
3628:
3627:
3607:
3601:
3600:
3590:
3580:
3556:
3550:
3549:
3547:
3546:
3533:
3505:
3499:
3496:
3490:
3489:
3482:
3476:
3475:
3469:
3461:
3459:
3457:
3451:
3440:
3434:
3433:
3419:
3413:
3412:
3410:
3398:
3392:
3386:
3380:
3379:
3341:
3335:
3334:
3332:
3316:
3310:
3309:
3307:
3297:
3281:
3275:
3274:
3272:
3262:
3246:
3240:
3239:
3237:
3213:
3207:
3206:
3196:
3186:
3162:
3156:
3155:
3145:
3135:
3111:
3105:
3104:
3102:
3101:
3085:
3079:
3078:
3052:
3046:
3045:
3029:
3023:
3022:
3010:
3004:
3003:
2977:
2968:
2962:
2961:
2951:
2945:
2944:
2934:
2924:
2900:
2894:
2893:
2887:
2878:
2856:Confusion matrix
2631:Logarithmic Loss
2496:Machine learning
2489:
2482:
2475:
2466:
2465:
2419:Confusion matrix
2333:David M W Powers
2270:Dice coefficient
2231:machine learning
2225:
2223:
2222:
2217:
2215:
2214:
2167:
2165:
2164:
2159:
2157:
2156:
2139:
2137:
2136:
2131:
2119:
2117:
2116:
2111:
2109:
2108:
2054:
2051:
2045:
2042:
2036:
2033:
2027:
2024:
2018:
2015:
2009:
2006:
1994:
1993:
1991:
1990:
1987:
1984:
1969:
1968:
1967:
1960:
1959:
1958:
1944:
1943:
1942:
1928:
1927:
1925:
1924:
1921:
1918:
1910:
1909:
1907:
1906:
1903:
1900:
1882:
1881:
1879:
1878:
1875:
1872:
1858:
1857:
1855:
1854:
1851:
1848:
1837:
1830:
1826:
1825:deltaP (Δp)
1817:
1814:
1813:
1811:
1810:
1807:
1804:
1789:
1786:
1785:
1783:
1782:
1779:
1776:
1761:
1760:
1758:
1757:
1754:
1751:
1734:
1733:
1731:
1730:
1727:
1724:
1709:
1708:
1706:
1705:
1702:
1699:
1684:
1681:
1680:
1678:
1677:
1674:
1671:
1656:
1653:
1652:
1650:
1649:
1646:
1643:
1634:
1629:
1621:
1620:
1618:
1617:
1614:
1611:
1593:
1590:
1589:
1587:
1586:
1583:
1580:
1571:
1558:
1555:
1554:
1552:
1551:
1548:
1545:
1536:
1530:
1528:
1515:
1503:
1484:
1481:
1480:
1478:
1477:
1474:
1471:
1462:
1456:
1446:
1443:
1442:
1440:
1439:
1436:
1433:
1424:
1404:
1392:
1373:Actual condition
1366:
1365:
1363:
1362:
1359:
1356:
1354:
1353:
1335:
1331:
1312:
1307:Total population
1290:
1274:
1272:
1271:
1266:
1264:
1262:
1261:
1260:
1241:
1226:
1224:
1223:
1218:
1204:
1203:
1183:
1181:
1180:
1175:
1173:
1172:
1164:
1160:
1159:
1154:
1143:
1138:
1130:
1096:
1094:
1093:
1088:
1076:
1074:
1073:
1068:
1066:
1065:
1046:
1044:
1043:
1038:
1022:
1020:
1019:
1014:
1011:
1009:
1008:
981:
961:
934:
914:
913:
901:
874:
854:
853:
834:
833:
806:
786:
785:
766:
761:
760:
733:
731:
730:
725:
723:
721:
720:
694:
662:
661:
648:
647:
624:
594:
586:
585:
564:
563:
544:
542:
541:
536:
524:
522:
521:
516:
504:
502:
501:
496:
484:
482:
481:
476:
474:
473:
444:
442:
441:
436:
434:
432:
431:
420:
409:
397:
396:
384:
379:
377:
376:
353:
323:
322:
299:
269:
261:
259:
258:
257:
249:
216:
215:
207:
182:
177:
176:
122:
120:
119:
114:
112:
111:
21:
3795:
3794:
3790:
3789:
3788:
3786:
3785:
3784:
3750:
3749:
3748:
3747:
3730:
3726:
3689:
3685:
3656:
3652:
3635:
3631:
3608:
3604:
3557:
3553:
3544:
3542:
3506:
3502:
3497:
3493:
3484:
3483:
3479:
3463:
3462:
3455:
3453:
3449:
3441:
3437:
3420:
3416:
3399:
3395:
3387:
3383:
3368:
3352:. p. 339.
3342:
3338:
3317:
3313:
3282:
3278:
3247:
3243:
3214:
3210:
3163:
3159:
3126:(1): 6-1–6-13.
3112:
3108:
3099:
3097:
3086:
3082:
3075:
3053:
3049:
3030:
3026:
3011:
3007:
2975:
2969:
2965:
2952:
2948:
2901:
2897:
2885:
2879:
2875:
2870:
2865:
2860:
2847:
2821:
2798:
2789:Inception score
2777:
2754:
2732:Computer Vision
2726:
2698:
2635:
2567:
2499:
2493:
2455:Word error rate
2410:
2401:
2389:
2369:
2362:
2354:
2338:
2324:
2316:
2308:
2296:
2286:
2279:
2267:
2262:
2210:
2206:
2204:
2201:
2200:
2198:
2174:
2152:
2148:
2146:
2143:
2142:
2125:
2122:
2121:
2104:
2100:
2098:
2095:
2094:
2091:
2075:
2058:
2057:
2052:
2048:
2043:
2039:
2034:
2030:
2025:
2021:
2016:
2012:
2007:
2003:
1988:
1985:
1982:
1981:
1979:
1977:
1976:
1965:
1963:
1961:
1956:
1954:
1952:
1951:
1940:
1938:
1936:
1935:
1922:
1919:
1916:
1915:
1913:
1911:
1904:
1901:
1898:
1897:
1895:
1893:
1892:
1889:
1876:
1873:
1870:
1869:
1867:
1865:
1864:
1852:
1849:
1846:
1845:
1843:
1841:
1840:
1835:
1829:= PPV + NPV − 1
1828:
1827:
1824:
1815:
1808:
1805:
1802:
1801:
1799:
1797:
1796:
1787:
1780:
1777:
1774:
1773:
1771:
1769:
1768:
1755:
1752:
1749:
1748:
1746:
1744:
1743:
1728:
1725:
1722:
1721:
1719:
1717:
1716:
1703:
1700:
1697:
1696:
1694:
1692:
1691:
1682:
1675:
1672:
1669:
1668:
1666:
1664:
1663:
1654:
1647:
1644:
1641:
1640:
1638:
1636:
1635:
1630:
1624:
1615:
1612:
1609:
1608:
1606:
1604:
1603:
1591:
1584:
1581:
1578:
1577:
1575:
1573:
1572:
1566:
1565:
1556:
1549:
1546:
1543:
1542:
1540:
1538:
1537:
1532:
1531:
1526:
1523:
1522:
1513:
1512:
1501:
1500:
1482:
1475:
1472:
1469:
1468:
1466:
1464:
1463:
1458:
1457:
1454:
1453:
1444:
1437:
1434:
1431:
1430:
1428:
1426:
1425:
1419:
1402:
1401:
1390:
1389:
1375:
1360:
1357:
1351:
1349:
1348:
1347:
1345:
1343:
1342:
1334:= TPR + TNR − 1
1333:
1332:
1329:
1310:
1309:
1281:
1256:
1252:
1245:
1240:
1232:
1229:
1228:
1199:
1195:
1193:
1190:
1189:
1165:
1144:
1142:
1129:
1128:
1124:
1123:
1109:
1106:
1105:
1082:
1079:
1078:
1061:
1057:
1055:
1052:
1051:
1032:
1029:
1028:
965:
918:
909:
905:
861:
849:
845:
835:
793:
781:
777:
767:
765:
756:
752:
750:
747:
746:
701:
666:
657:
653:
649:
628:
596:
595:
593:
581:
577:
559:
555:
553:
550:
549:
530:
527:
526:
510:
507:
506:
490:
487:
486:
469:
465:
463:
460:
459:
456:
453:
424:
413:
402:
398:
389:
385:
383:
357:
325:
324:
303:
271:
270:
268:
250:
221:
220:
208:
188:
187:
186:
181:
172:
168:
166:
163:
162:
151:
144:
138:(MUC-4, 1992).
132:
107:
103:
101:
98:
97:
90:
35:
28:
23:
22:
15:
12:
11:
5:
3793:
3783:
3782:
3777:
3772:
3767:
3762:
3746:
3745:
3724:
3683:
3650:
3629:
3602:
3551:
3500:
3491:
3477:
3435:
3414:
3393:
3381:
3366:
3336:
3330:10.1.1.127.634
3311:
3276:
3241:
3208:
3171:BioData Mining
3157:
3106:
3080:
3073:
3047:
3024:
3005:
2986:(8): 861–874.
2963:
2946:
2895:
2872:
2871:
2869:
2866:
2862:
2861:
2859:
2858:
2852:
2849:
2848:
2846:
2845:
2840:
2835:
2829:
2827:
2823:
2822:
2820:
2819:
2814:
2808:
2806:
2800:
2799:
2797:
2796:
2791:
2785:
2783:
2779:
2778:
2776:
2775:
2770:
2764:
2762:
2756:
2755:
2753:
2752:
2747:
2742:
2736:
2734:
2728:
2727:
2725:
2724:
2719:
2714:
2708:
2706:
2700:
2699:
2697:
2696:
2691:
2686:
2681:
2676:
2671:
2666:
2661:
2659:Davies-Bouldin
2656:
2651:
2645:
2643:
2637:
2636:
2634:
2633:
2628:
2623:
2618:
2613:
2608:
2603:
2598:
2593:
2588:
2583:
2577:
2575:
2573:Classification
2569:
2568:
2566:
2565:
2560:
2555:
2550:
2545:
2540:
2535:
2530:
2525:
2520:
2515:
2509:
2507:
2501:
2500:
2492:
2491:
2484:
2477:
2469:
2463:
2462:
2457:
2452:
2446:
2444:ROUGE (metric)
2441:
2436:
2431:
2426:
2421:
2416:
2409:
2406:
2400:
2397:
2388:
2385:
2381:geometric mean
2368:
2365:
2360:
2352:
2336:
2322:
2314:
2307:
2304:
2303:
2302:
2294:
2291:
2288:
2284:
2281:
2277:
2265:
2261:
2258:
2213:
2209:
2196:
2180:for measuring
2173:
2170:
2155:
2151:
2129:
2107:
2103:
2090:
2087:
2073:
2068:is precision,
2056:
2055:
2046:
2037:
2028:
2019:
2010:
2000:
1999:
1996:
1995:
1970:
1945:
1929:
1923:2 TP + FP + FN
1887:
1883:
1860:
1859:
1831:
1818:
1790:
1762:
1736:
1735:
1710:
1685:
1657:
1622:
1598:
1595:
1594:
1559:
1516:
1504:
1496:False positive
1492:
1486:
1485:
1455:miss rate
1447:
1405:
1397:False negative
1393:
1381:
1376:
1371:
1368:
1367:
1336:
1323:
1318:
1313:
1303:
1302:
1300:
1295:
1293:
1280:
1277:
1259:
1255:
1251:
1248:
1244:
1239:
1236:
1216:
1213:
1210:
1207:
1202:
1198:
1186:
1185:
1171:
1168:
1163:
1157:
1153:
1150:
1147:
1141:
1136:
1133:
1127:
1122:
1119:
1116:
1113:
1099:Van Rijsbergen
1086:
1064:
1060:
1036:
1025:
1024:
1007:
1004:
1001:
998:
995:
992:
989:
986:
980:
977:
974:
971:
968:
964:
960:
957:
954:
951:
948:
945:
942:
939:
933:
930:
927:
924:
921:
917:
912:
908:
904:
900:
897:
894:
891:
888:
885:
882:
879:
873:
870:
867:
864:
860:
857:
852:
848:
844:
841:
838:
832:
829:
826:
823:
820:
817:
814:
811:
805:
802:
799:
796:
792:
789:
784:
780:
776:
773:
770:
764:
759:
755:
742:this becomes:
736:
735:
719:
716:
713:
710:
707:
704:
700:
697:
693:
690:
687:
684:
681:
678:
675:
672:
669:
665:
660:
656:
652:
646:
643:
640:
637:
634:
631:
627:
623:
620:
617:
614:
611:
608:
605:
602:
599:
592:
589:
584:
580:
576:
573:
570:
567:
562:
558:
534:
514:
494:
472:
468:
455:
451:
448:
447:
446:
430:
427:
423:
419:
416:
412:
408:
405:
401:
395:
392:
388:
382:
375:
372:
369:
366:
363:
360:
356:
352:
349:
346:
343:
340:
337:
334:
331:
328:
321:
318:
315:
312:
309:
306:
302:
298:
295:
292:
289:
286:
283:
280:
277:
274:
267:
264:
256:
253:
248:
245:
242:
239:
236:
233:
230:
227:
224:
219:
214:
211:
206:
203:
200:
197:
194:
191:
185:
180:
175:
171:
149:
143:
140:
131:
128:
110:
106:
88:
26:
9:
6:
4:
3:
2:
3792:
3781:
3778:
3776:
3773:
3771:
3768:
3766:
3763:
3761:
3758:
3757:
3755:
3740:
3735:
3728:
3720:
3716:
3711:
3706:
3702:
3698:
3694:
3687:
3678:
3673:
3669:
3665:
3661:
3654:
3645:
3640:
3633:
3625:
3621:
3617:
3613:
3606:
3598:
3594:
3589:
3584:
3579:
3574:
3570:
3566:
3562:
3555:
3541:
3537:
3532:
3531:10044/1/46235
3527:
3523:
3519:
3515:
3511:
3508:Hand, David.
3504:
3495:
3487:
3481:
3473:
3467:
3448:
3447:
3439:
3431:
3427:
3426:
3418:
3409:
3404:
3397:
3390:
3385:
3377:
3373:
3369:
3367:9781605581644
3363:
3359:
3355:
3351:
3347:
3340:
3331:
3326:
3322:
3315:
3306:
3301:
3296:
3291:
3287:
3280:
3271:
3266:
3261:
3256:
3252:
3245:
3236:
3231:
3227:
3223:
3219:
3212:
3204:
3200:
3195:
3190:
3185:
3180:
3176:
3172:
3168:
3161:
3153:
3149:
3144:
3139:
3134:
3129:
3125:
3121:
3117:
3110:
3095:
3091:
3084:
3076:
3070:
3066:
3062:
3058:
3051:
3043:
3039:
3035:
3028:
3020:
3016:
3009:
3001:
2997:
2993:
2989:
2985:
2981:
2974:
2967:
2959:
2958:
2950:
2942:
2938:
2933:
2928:
2923:
2918:
2914:
2910:
2906:
2899:
2891:
2884:
2877:
2873:
2857:
2854:
2853:
2850:
2844:
2841:
2839:
2836:
2834:
2831:
2830:
2828:
2824:
2818:
2815:
2813:
2810:
2809:
2807:
2805:
2801:
2795:
2792:
2790:
2787:
2786:
2784:
2780:
2774:
2771:
2769:
2766:
2765:
2763:
2761:
2757:
2751:
2748:
2746:
2743:
2741:
2738:
2737:
2735:
2733:
2729:
2723:
2720:
2718:
2715:
2713:
2710:
2709:
2707:
2705:
2701:
2695:
2692:
2690:
2687:
2685:
2682:
2680:
2677:
2675:
2674:Jaccard index
2672:
2670:
2667:
2665:
2662:
2660:
2657:
2655:
2652:
2650:
2647:
2646:
2644:
2642:
2638:
2632:
2629:
2627:
2624:
2622:
2619:
2617:
2614:
2612:
2609:
2607:
2604:
2602:
2599:
2597:
2594:
2592:
2589:
2587:
2584:
2582:
2579:
2578:
2576:
2574:
2570:
2564:
2561:
2559:
2556:
2554:
2551:
2549:
2546:
2544:
2541:
2539:
2536:
2534:
2531:
2529:
2526:
2524:
2521:
2519:
2516:
2514:
2511:
2510:
2508:
2506:
2502:
2497:
2490:
2485:
2483:
2478:
2476:
2471:
2470:
2467:
2461:
2458:
2456:
2453:
2450:
2447:
2445:
2442:
2440:
2437:
2435:
2434:NIST (metric)
2432:
2430:
2427:
2425:
2422:
2420:
2417:
2415:
2412:
2411:
2405:
2396:
2394:
2384:
2382:
2378:
2374:
2373:harmonic mean
2364:
2358:
2349:
2347:
2343:
2334:
2330:
2328:
2319:
2312:
2300:
2292:
2289:
2282:
2275:
2274:
2273:
2271:
2268:score is the
2257:
2255:
2251:
2246:
2244:
2243:Cohen's kappa
2240:
2236:
2232:
2227:
2211:
2207:
2193:
2191:
2187:
2183:
2179:
2169:
2153:
2149:
2127:
2105:
2101:
2082:
2071:
2067:
2062:
2050:
2041:
2032:
2023:
2014:
2005:
2001:
1975:
1974:Jaccard index
1949:
1946:
1933:
1930:
1891:
1884:
1862:
1861:
1838:
1832:
1822:
1819:
1794:
1791:
1766:
1763:
1741:
1738:
1737:
1714:
1711:
1689:
1686:
1661:
1658:
1633:
1627:
1623:
1602:
1599:
1596:
1569:
1563:
1560:
1535:
1529:
1520:
1517:
1510:
1509:
1508:True negative
1505:
1498:
1497:
1493:
1491:
1488:
1487:
1461:
1451:
1448:
1423:
1417:
1413:
1409:
1406:
1399:
1398:
1394:
1387:
1386:
1385:True positive
1382:
1380:
1377:
1374:
1369:
1340:
1337:
1327:
1324:
1322:
1319:
1317:
1314:
1308:
1305:
1304:
1299:
1294:
1291:
1288:
1286:
1276:
1257:
1253:
1249:
1246:
1242:
1237:
1234:
1214:
1211:
1208:
1205:
1200:
1196:
1169:
1166:
1161:
1155:
1151:
1148:
1145:
1139:
1134:
1131:
1125:
1120:
1117:
1114:
1111:
1104:
1103:
1102:
1100:
1084:
1062:
1058:
1048:
1034:
962:
915:
910:
906:
902:
858:
850:
846:
842:
839:
790:
782:
778:
774:
771:
762:
757:
753:
745:
744:
743:
741:
698:
663:
658:
654:
625:
590:
582:
578:
574:
571:
565:
560:
556:
548:
547:
546:
532:
512:
492:
470:
466:
421:
410:
399:
386:
380:
354:
300:
265:
262:
254:
251:
217:
212:
209:
183:
178:
173:
169:
161:
160:
159:
157:
156:harmonic mean
153:
139:
137:
127:
124:
108:
104:
95:
94:harmonic mean
92:score is the
91:
83:
81:
77:
72:
68:
64:
60:
57:systems, the
56:
52:
48:
39:
33:
19:
3727:
3700:
3696:
3686:
3667:
3663:
3653:
3632:
3618:(1): 37–63.
3615:
3611:
3605:
3568:
3565:BMC Genomics
3564:
3554:
3543:. Retrieved
3513:
3503:
3494:
3480:
3454:. Retrieved
3445:
3438:
3429:
3424:
3417:
3396:
3384:
3349:
3345:
3339:
3320:
3314:
3285:
3279:
3250:
3244:
3225:
3221:
3211:
3174:
3170:
3160:
3123:
3120:BMC Genomics
3119:
3109:
3098:. Retrieved
3093:
3083:
3059:. Springer.
3056:
3050:
3041:
3037:
3027:
3018:
3008:
2983:
2979:
2966:
2956:
2949:
2915:(29): 1–28.
2912:
2908:
2898:
2889:
2876:
2580:
2402:
2390:
2370:
2350:
2342:Informedness
2331:
2320:
2309:
2263:
2247:
2239:Informedness
2228:
2194:
2175:
2172:Applications
2092:
2069:
2065:
2049:
2040:
2031:
2022:
2013:
2004:
1989:TP + FN + FP
1506:
1494:
1490:Negative (N)
1489:
1395:
1383:
1379:Positive (P)
1378:
1372:
1326:Informedness
1320:
1315:
1297:
1282:
1187:
1049:
1026:
738:In terms of
737:
457:
147:
145:
133:
125:
86:
84:
62:
58:
49:analysis of
44:
3703:: 820–836.
3670:: 168–192.
3228:: 168–192.
3044:(1): 37–63.
1899:2 PPV × TPR
1834:Diagnostic
1568:specificity
1416:sensitivity
80:sensitivity
47:statistical
3754:Categories
3739:1911.03347
3710:2404.16958
3644:2210.11997
3624:2328/27165
3545:2018-12-08
3408:1503.06410
3295:1909.02827
3260:2001.05571
3177:(13): 13.
3100:2019-07-17
2868:References
2826:Similarity
2768:Perplexity
2679:Rand index
2664:Dunn index
2649:Silhouette
2641:Clustering
2505:Regression
2346:Markedness
2311:David Hand
2260:Properties
1836:odds ratio
1821:Markedness
1601:Prevalence
142:Definition
3466:cite book
3325:CiteSeerX
2596:Precision
2548:RMSE/RMSD
2379:is their
2357:P4 metric
2340:measures
2306:Criticism
2297:score is
2212:β
2106:β
1941:PPV × TPR
1905:PPV + TPR
1871:TPR + TNR
1816:= 1 − FOR
1788:= 1 − PPV
1683:= 1 − NPV
1655:= 1 − FDR
1632:precision
1592:= 1 − FPR
1557:= 1 − TNR
1483:= 1 − TPR
1445:= 1 − FNR
1361:TPR - FPR
1352:TPR × FPR
1254:β
1235:α
1212:−
1201:β
1167:−
1152:α
1149:−
1132:α
1121:−
1085:β
1063:β
1035:β
916:⋅
907:β
859:⋅
847:β
791:⋅
779:β
758:β
664:⋅
655:β
626:⋅
591:⋅
579:β
561:β
533:β
513:β
493:β
471:β
301:⋅
252:−
210:−
154:) is the
130:Etymology
109:β
67:precision
63:F-measure
3597:31898477
3571:(6): 6.
3540:38782128
3203:33541410
3152:31898477
2941:26263899
2812:Coverage
2591:Accuracy
2408:See also
2399:Macro F1
1740:Accuracy
1527:fall-out
505:, where
18:F1 score
3588:6941312
3456:18 July
3376:8482989
3194:7863449
3143:6941312
3000:2027090
2932:4533825
2704:Ranking
2694:SimHash
2581:F-score
2299:concave
1992:
1980:
1964:√
1955:√
1939:√
1926:
1914:
1908:
1896:
1880:
1868:
1856:
1844:
1812:
1800:
1784:
1772:
1759:
1750:TP + TN
1747:
1732:
1720:
1707:
1695:
1679:
1667:
1651:
1639:
1619:
1607:
1588:
1576:
1564:(TNR),
1553:
1541:
1521:(FPR),
1479:
1467:
1452:(FNR),
1441:
1429:
1418:(SEN),
1410:(TPR),
1364:
1350:√
1346:
1311:= P + N
59:F-score
3595:
3585:
3538:
3374:
3364:
3327:
3201:
3191:
3150:
3140:
3071:
2998:
2939:
2929:
2601:Recall
2429:METEOR
2188:, and
2182:search
1950:(MCC)
1839:(DOR)
1823:(MK),
1795:(NPV)
1767:(FDR)
1742:(ACC)
1715:(LR−)
1690:(LR+)
1662:(FOR)
1628:(PPV),
1511:(TN),
1499:(FP),
1412:recall
1400:(FN),
1388:(TP),
1227:where
982:
935:
875:
807:
71:recall
32:F-test
3734:arXiv
3705:arXiv
3639:arXiv
3536:S2CID
3450:(PDF)
3403:arXiv
3372:S2CID
3290:arXiv
3255:arXiv
2996:S2CID
2976:(PDF)
2886:(PDF)
2606:Kappa
2523:sMAPE
2460:LEPOR
2283:The F
2276:The F
2264:The F
1934:(FM)
1890:score
1756:P + N
1616:P + N
1422:power
1355:- FPR
1341:(PT)
454:score
152:score
3593:PMID
3472:link
3458:2022
3362:ISBN
3199:PMID
3148:PMID
3069:ISBN
2937:PMID
2773:BLEU
2745:SSIM
2740:PSNR
2717:NDCG
2538:MSPE
2533:MASE
2528:MAPE
2414:BLEU
2344:and
2252:and
2241:or
1917:2 TP
85:The
69:and
53:and
3715:doi
3672:doi
3620:hdl
3583:PMC
3573:doi
3526:hdl
3518:doi
3354:doi
3300:doi
3265:doi
3230:doi
3189:PMC
3179:doi
3138:PMC
3128:doi
3061:doi
2988:doi
2927:PMC
2917:doi
2794:FID
2760:NLP
2750:IoU
2712:MRR
2689:SMC
2621:ROC
2616:AUC
2611:MCC
2563:MAD
2558:MDA
2543:RMS
2518:MAE
2513:MSE
1853:LR−
1847:LR+
1729:TNR
1723:FNR
1704:FPR
1698:TPR
1391:hit
61:or
45:In
3756::
3713:.
3701:12
3699:.
3695:.
3668:17
3666:.
3662:.
3614:.
3591:.
3581:.
3569:21
3567:.
3563:.
3534:.
3524:.
3516:.
3512:.
3468:}}
3464:{{
3428:.
3370:.
3360:.
3348:.
3298:.
3263:.
3226:17
3224:.
3220:.
3197:.
3187:.
3175:14
3173:.
3169:.
3146:.
3136:.
3124:21
3122:.
3118:.
3092:.
3067:.
3040:.
3036:.
3017:.
2994:.
2984:27
2982:.
2978:.
2935:.
2925:.
2913:15
2911:.
2907:.
2888:.
2722:AP
2586:P4
2383:.
2363:.
2256:.
2237:,
2184:,
1983:TP
1978:=
1962:-
1953:=
1937:=
1912:=
1894:=
1866:=
1842:=
1809:PN
1803:TN
1798:=
1781:PP
1775:FP
1770:=
1745:=
1718:=
1693:=
1676:PN
1670:FN
1665:=
1648:PP
1642:TP
1637:=
1605:=
1579:TN
1574:=
1544:FP
1539:=
1470:FN
1465:=
1432:TP
1427:=
1414:,
1344:=
1328:,
1275:.
3742:.
3736::
3721:.
3717::
3707::
3680:.
3674::
3647:.
3641::
3626:.
3622::
3616:2
3599:.
3575::
3548:.
3528::
3520::
3488:.
3474:)
3460:.
3432:.
3411:.
3405::
3391:.
3378:.
3356::
3333:.
3308:.
3302::
3292::
3273:.
3267::
3257::
3238:.
3232::
3205:.
3181::
3154:.
3130::
3103:.
3077:.
3063::
3042:2
3021:.
3002:.
2990::
2943:.
2919::
2553:R
2488:e
2481:t
2474:v
2361:1
2353:1
2337:1
2323:1
2315:1
2295:1
2293:F
2285:1
2278:1
2266:1
2208:F
2197:1
2154:0
2150:r
2128:r
2102:F
2074:1
2070:y
2066:x
1986:/
1920:/
1902:/
1888:1
1886:F
1877:2
1874:/
1850:/
1806:/
1778:/
1753:/
1726:/
1701:/
1673:/
1645:/
1613:/
1610:P
1585:N
1582:/
1550:N
1547:/
1476:P
1473:/
1438:P
1435:/
1358:/
1258:2
1250:+
1247:1
1243:1
1238:=
1215:E
1209:1
1206:=
1197:F
1184:.
1170:1
1162:)
1156:r
1146:1
1140:+
1135:p
1126:(
1118:1
1115:=
1112:E
1059:F
1023:.
1006:e
1003:v
1000:i
997:t
994:i
991:s
988:o
985:p
979:e
976:s
973:l
970:a
967:f
963:+
959:e
956:v
953:i
950:t
947:a
944:g
941:e
938:n
932:e
929:s
926:l
923:a
920:f
911:2
903:+
899:e
896:v
893:i
890:t
887:i
884:s
881:o
878:p
872:e
869:u
866:r
863:t
856:)
851:2
843:+
840:1
837:(
831:e
828:v
825:i
822:t
819:i
816:s
813:o
810:p
804:e
801:u
798:r
795:t
788:)
783:2
775:+
772:1
769:(
763:=
754:F
734:.
718:l
715:l
712:a
709:c
706:e
703:r
699:+
696:)
692:n
689:o
686:i
683:s
680:i
677:c
674:e
671:r
668:p
659:2
651:(
645:l
642:l
639:a
636:c
633:e
630:r
622:n
619:o
616:i
613:s
610:i
607:c
604:e
601:r
598:p
588:)
583:2
575:+
572:1
569:(
566:=
557:F
467:F
452:β
450:F
445:.
429:n
426:f
422:+
418:p
415:f
411:+
407:p
404:t
400:2
394:p
391:t
387:2
381:=
374:l
371:l
368:a
365:c
362:e
359:r
355:+
351:n
348:o
345:i
342:s
339:i
336:c
333:e
330:r
327:p
320:l
317:l
314:a
311:c
308:e
305:r
297:n
294:o
291:i
288:s
285:i
282:c
279:e
276:r
273:p
266:2
263:=
255:1
247:n
244:o
241:i
238:s
235:i
232:c
229:e
226:r
223:p
218:+
213:1
205:l
202:l
199:a
196:c
193:e
190:r
184:2
179:=
174:1
170:F
150:1
148:F
105:F
89:1
87:F
34:.
20:)
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.
