465:
and
Partners Scotland calculated the 80th percentile value (i.e., 80% likelihood of staying within budget) for total capital costs to be £400 million, which equaled 57% contingency. Similarly, they calculated the 50th percentile value (i.e., 50% likelihood of staying within budget) to be £357 million, which equaled 40% contingency. The review further acknowledged that the reference class forecasts were likely to be too low because the guidelines recommended that the uplifts should be applied at the time of decision to build, which the project had not yet reached, and that the risks therefore would be substantially higher at this early business case stage. On this basis, the review concluded that the forecasted costs could have been underestimated. The Edinburgh Tram Line 2 opened three years late in May 2014 with a final outturn cost of £776 million, which equals £628 million in 2004-prices.
31:
501:
479:(the Association for the Advancement of Cost Engineering) included Estimate Validation as a distinct step in the recommended practice of Cost Estimating (Estimate Validation is equivalent to Reference class forecasting in that it calls for separate empirical-based evaluations to benchmark the base estimate):
488:
In the process industries (e.g., oil and gas, chemicals, mining, energy, etc. which tend to dominate AACE's membership), benchmarking (i.e., "outside view") of project cost estimates against the historical costs of completed projects of similar types, including probabilistic information, has a long
436:
Suppose someone were trying to predict how long it would take to write a psychology textbook. Reference class tennis would involve debating whether we should take the average of all books (closest to an outside view), just all textbooks, or just all psychology textbooks (closest to an inside view).
464:
business case, which was carried out in
October 2004 by Ove Arup and Partners Scotland. At the time, the project was forecast to cost a total of £320 million, of which £64 million – or 25% – was allocated for contingency. Using the newly implemented reference class forecasting guidelines, Ove Arup
394:
Kahneman and
Tversky concluded that disregard of distributional information, i.e. risk, is perhaps the major source of error in forecasting. On that basis they recommended that forecasters "should therefore make every effort to frame the forecasting problem so as to facilitate utilizing all the
483:
The estimate should be benchmarked or validated against or compared to historical experience and/or past estimates of the enterprise and of competitive enterprises to check its appropriateness, competitiveness, and to identify improvement opportunities...Validation examines the estimate from a
489:
history. A method combining reference class forecasting and competitive crowdsourcing, Human Forest, has also been used in the life sciences, to estimate the likelihood that vaccines and treatments will successfully progress through clinical trial phases.
781:
387:
People tend to underestimate the costs, completion times, and risks of planned actions, whereas they tend to overestimate the benefits of those same actions. Such error is caused by actors taking an "
319:
871:
346:
is a method of predicting the future by looking at similar past situations and their outcomes. The theories behind reference class forecasting were developed by
395:
distributional information that is available". Using distributional information from previous ventures similar to the one being forecast is called taking an "
391:", where focus is on the constituents of the specific planned action instead of on the actual outcomes of similar ventures that have already been completed.
694:
Kahneman, Daniel; Tversky, Amos (1982). "Intuitive prediction: Biases and corrective procedures". In
Kahneman, Daniel; Slovic, Paul; Tversky, Amos (eds.).
468:
Since the
Edinburgh forecast, reference class forecasting has been applied to numerous other projects in the UK, including the £15 (US$ 29) billion
365:
is so named as it predicts the outcome of a planned action based on actual outcomes in a reference class of similar actions to that being forecast.
759:
453:(2004) developed the method for its practical use in policy and planning, which was published as an official Guidance Document in June 2004 by the
472:
project in London. After 2004, The
Netherlands, Denmark, and Switzerland have also implemented various types of reference class forecasting.
420:
Compare the specific project with the reference class distribution, in order to establish the most likely outcome for the specific project.
460:
The first instance of reference class forecasting in practice is described in
Flyvbjerg (2006). This forecast was part of a review of the
895:
326:
663:
433:, also known as reference class tennis, is the discussion of which reference class to use when forecasting a given situation.
1009:
275:
1089:
1054:
711:
920:
Atanasov, Pavel D.; Joseph, Regina; Feijoo, Felipe; Marshall, Max; Conway, Amanda; Siddiqui, Sauleh (2022-07-27).
601:
896:
AACE International, "Total Cost
Management Framework, Section 7.3, Cost Estimating and Budgeting", 2011. p. 147
1113:
907:
506:
90:
1118:
1108:
285:
192:
85:
295:
38:
132:
966:
687:
621:
454:
410:
384:
due to overconfidence and insufficient consideration of distributional information about outcomes.
355:
312:
57:
727:
578:
539:
430:
369:
270:
160:
80:
616:
524: – When the actual benefits of a venture are less than the projected or estimated benefits
251:
97:
1081:
1065:
1034:
945:
857:
368:
Discussion of which reference class to use when forecasting a given situation is known as the
981:"NSF Award Search: Award # 1919333 - Human Forests versus Random Forest Models in Prediction"
953:
674:
247:
175:
70:
980:
829:
399:". Reference class forecasting is a method for taking an outside view on planned actions.
8:
1035:"Public Planning of Mega-Projects: Overestimation of Demand and Underestimation of Costs"
557:
484:
different perspective and using different metrics than are used in estimate preparation.
300:
47:
833:
933:
845:
819:
810:
Flyvbjerg, Bent (2006). "From Nobel Prize to
Project Management: Getting Risks Right".
656:
634:
527:
476:
402:
Reference class forecasting for a specific project involves the following three steps:
290:
280:
170:
165:
146:
127:
75:
1069:
1085:
1050:
1021:
941:
937:
922:"Human Forest vs. Random Forest in Time-Sensitive COVID-19 Clinical Trial Prediction"
853:
707:
521:
515:
222:
849:
445:
Whereas
Kahneman and Tversky developed the theories of reference class forecasting,
1077:
1042:
925:
837:
699:
626:
572:
227:
212:
185:
137:
122:
52:
703:
461:
347:
260:
207:
107:
102:
65:
22:
1046:
530: – Prediction of the future created by combining several separate forecasts
30:
841:
545:
265:
237:
112:
1102:
566:
381:
232:
202:
180:
518: – Error in thinking which involves under-valuing base rate information
1025:
921:
732:
533:
351:
197:
754:
929:
551:
362:
155:
638:
1066:"Over Budget, Over Time, Over and Over Again: Managing Major Projects"
1010:"Delusions of success. How optimism undermines executives' decisions"
469:
446:
414:
630:
117:
824:
782:"Procedures for Dealing with Optimism Bias in Transport Planning"
536: – Unexpected incurred costs in excess of budgeted amounts
500:
548: – Any of various types of risk associated with financing
919:
380:
Kahneman and Tversky found that human judgment is generally
1037:. In Priemus, Hugo; Flyvbjerg, Bent; van Wee, Bert (eds.).
650:
648:
450:
560: – Self-referential adage referring to time estimates
657:"Intuitive prediction: Biases and corrective procedures"
645:
593:
562:
Pages displaying short descriptions of redirect targets
440:
692:
Decision Research Technical Report PTR-1042-77-6. In
602:"Prospect Theory: An Analysis of Decision under Risk"
575: – Cognitive bias of underestimating time needed
406:
Identify a reference class of past, similar projects.
496:
554: – Making predictions based on available data
696:Judgment Under Uncertainty: Heuristics and Biases
1100:
1007:
693:
654:
599:
354:. The theoretical work helped Kahneman win the
320:
803:
1068:. In Morris, Peter W. G; Pinto, Jeffrey K;
720:
581: – Issue when estimating a probability
327:
313:
1074:The Oxford Handbook of Project Management
1063:
1032:
823:
809:
620:
424:
413:for the selected reference class for the
908:Assessing Project Cost and Schedule Risk
655:Kahneman, Daniel; Tversky, Amos (1977).
600:Kahneman, Daniel; Tversky, Amos (1979).
475:Before this, in 2001 (updated in 2011),
1082:10.1093/oxfordhb/9780199563142.003.0014
872:"Report for the Edinburgh Tram Inquiry"
669:from the original on September 8, 2013.
1101:
728:"Week 10: Reference Class Forecasting"
910:", 1990 AACE Transactions. pp H.6.1-7
789:The British Department for Transport
441:Practical use in policy and planning
13:
542: – Network analysis technique
14:
1130:
747:
1039:Decision-making on Mega-Projects
1008:Lovallo, D; Kahneman, D (2003).
499:
29:
1001:
973:
913:
16:Method of predicting the future
900:
889:
864:
774:
569: – Type of cognitive bias
1:
906:Merrow, E. and Yarossi, M., "
586:
507:Business and economics portal
704:10.1017/CBO9780511809477.031
344:comparison class forecasting
7:
1047:10.4337/9781848440173.00014
492:
455:UK Department for Transport
375:
340:Reference class forecasting
286:Science fiction prototyping
218:Reference class forecasting
10:
1135:
842:10.1177/875697280603700302
812:Project Management Journal
1064:Flyvbjerg, Bent (2011).
1033:Flyvbjerg, Bent (2008).
411:probability distribution
356:Nobel Prize in Economics
58:Global catastrophic risk
1014:Harvard Business Review
579:Reference class problem
540:Event chain methodology
431:reference class problem
417:that is being forecast.
370:reference class problem
271:Exploratory engineering
161:Causal layered analysis
961:Cite journal requires
682:Cite journal requires
486:
425:Reference class tennis
98:Historical materialism
481:
462:Edinburgh Tram Line 2
176:Cross impact analysis
1114:Economic forecasting
930:10.2139/ssrn.3981732
698:. pp. 414–421.
834:2013arXiv1302.3642F
301:Technology scouting
48:Accelerating change
1119:Futures techniques
1109:Evaluation methods
528:Consensus forecast
477:AACE International
291:Speculative design
171:Consensus forecast
166:Chain-linked model
128:Resource depletion
924:. Rochester, NY.
522:Benefit shortfall
516:Base rate fallacy
337:
336:
223:Scenario planning
86:Space exploration
1126:
1095:
1070:Söderlund, Jonas
1060:
1029:
995:
994:
992:
991:
977:
971:
970:
964:
959:
957:
949:
917:
911:
904:
898:
893:
887:
886:
884:
882:
876:
868:
862:
861:
827:
807:
801:
800:
798:
796:
786:
778:
772:
771:
769:
767:
751:
745:
744:
742:
740:
724:
718:
717:
691:
685:
680:
678:
670:
668:
661:
652:
643:
642:
624:
606:
597:
573:Planning fallacy
563:
558:Hofstadter's Law
509:
504:
503:
361:Reference class
329:
322:
315:
228:Systems analysis
213:Horizon scanning
186:Real-time Delphi
123:Population cycle
53:Cashless society
33:
19:
18:
1134:
1133:
1129:
1128:
1127:
1125:
1124:
1123:
1099:
1098:
1092:
1057:
1004:
999:
998:
989:
987:
979:
978:
974:
962:
960:
951:
950:
918:
914:
905:
901:
894:
890:
880:
878:
877:. February 2018
874:
870:
869:
865:
808:
804:
794:
792:
784:
780:
779:
775:
765:
763:
753:
752:
748:
738:
736:
726:
725:
721:
714:
683:
681:
672:
671:
666:
659:
653:
646:
631:10.2307/1914185
622:10.1.1.407.1910
604:
598:
594:
589:
584:
561:
505:
498:
495:
443:
427:
378:
348:Daniel Kahneman
333:
261:Critical design
208:Future workshop
108:Kardashev scale
103:Kondratiev wave
23:Futures studies
17:
12:
11:
5:
1132:
1122:
1121:
1116:
1111:
1097:
1096:
1090:
1061:
1055:
1030:
1003:
1000:
997:
996:
972:
963:|journal=
912:
899:
888:
863:
802:
773:
760:LessWrong Wiki
755:"Outside view"
746:
719:
712:
684:|journal=
644:
615:(2): 263–291.
591:
590:
588:
585:
583:
582:
576:
570:
564:
555:
549:
546:Financial risk
543:
537:
531:
525:
519:
512:
511:
510:
494:
491:
442:
439:
426:
423:
422:
421:
418:
407:
377:
374:
335:
334:
332:
331:
324:
317:
309:
306:
305:
304:
303:
298:
293:
288:
283:
278:
273:
268:
266:Design fiction
263:
255:
254:
243:
242:
241:
240:
238:Trend analysis
235:
230:
225:
220:
215:
210:
205:
200:
195:
190:
189:
188:
178:
173:
168:
163:
158:
150:
149:
143:
142:
141:
140:
135:
130:
125:
120:
115:
110:
105:
100:
95:
94:
93:
88:
83:
78:
73:
68:
60:
55:
50:
42:
41:
35:
34:
26:
25:
15:
9:
6:
4:
3:
2:
1131:
1120:
1117:
1115:
1112:
1110:
1107:
1106:
1104:
1093:
1091:9780199563142
1087:
1083:
1079:
1075:
1071:
1067:
1062:
1058:
1056:9781848440173
1052:
1048:
1044:
1040:
1036:
1031:
1027:
1023:
1019:
1015:
1011:
1006:
1005:
986:
982:
976:
968:
955:
947:
943:
939:
935:
931:
927:
923:
916:
909:
903:
897:
892:
873:
867:
859:
855:
851:
847:
843:
839:
835:
831:
826:
821:
817:
813:
806:
790:
783:
777:
762:
761:
756:
750:
735:
734:
729:
723:
715:
713:9780511809477
709:
705:
701:
697:
689:
676:
665:
658:
651:
649:
640:
636:
632:
628:
623:
618:
614:
610:
603:
596:
592:
580:
577:
574:
571:
568:
567:Optimism bias
565:
559:
556:
553:
550:
547:
544:
541:
538:
535:
532:
529:
526:
523:
520:
517:
514:
513:
508:
502:
497:
490:
485:
480:
478:
473:
471:
466:
463:
458:
456:
452:
448:
438:
434:
432:
419:
416:
412:
408:
405:
404:
403:
400:
398:
392:
390:
385:
383:
373:
371:
366:
364:
359:
357:
353:
349:
345:
341:
330:
325:
323:
318:
316:
311:
310:
308:
307:
302:
299:
297:
294:
292:
289:
287:
284:
282:
279:
277:
274:
272:
269:
267:
264:
262:
259:
258:
257:
256:
253:
249:
245:
244:
239:
236:
234:
233:Threatcasting
231:
229:
226:
224:
221:
219:
216:
214:
211:
209:
206:
204:
203:Futures wheel
201:
199:
196:
194:
191:
187:
184:
183:
182:
179:
177:
174:
172:
169:
167:
164:
162:
159:
157:
154:
153:
152:
151:
148:
145:
144:
139:
138:Swanson's law
136:
134:
131:
129:
126:
124:
121:
119:
116:
114:
111:
109:
106:
104:
101:
99:
96:
92:
89:
87:
84:
82:
79:
77:
74:
72:
69:
67:
64:
63:
61:
59:
56:
54:
51:
49:
46:
45:
44:
43:
40:
37:
36:
32:
28:
27:
24:
21:
20:
1073:
1038:
1020:(7): 56–63.
1017:
1013:
1002:Bibliography
988:. Retrieved
984:
975:
954:cite journal
915:
902:
891:
879:. Retrieved
866:
815:
811:
805:
793:. Retrieved
788:
776:
764:. Retrieved
758:
749:
737:. Retrieved
733:Conceptually
731:
722:
695:
675:cite journal
612:
609:Econometrica
608:
595:
534:Cost overrun
487:
482:
474:
467:
459:
444:
435:
428:
409:Establish a
401:
397:outside view
396:
393:
388:
386:
379:
367:
360:
352:Amos Tversky
343:
339:
338:
217:
198:Future-proof
985:www.nsf.gov
818:(3): 5–15.
791:. June 2004
552:Forecasting
389:inside view
363:forecasting
252:forecasting
246:Technology
156:Backcasting
133:Singularity
113:Moore's law
71:Mathematics
1103:Categories
990:2022-09-20
587:References
382:optimistic
281:Hype cycle
248:assessment
147:Techniques
938:245174381
825:1302.3642
617:CiteSeerX
470:Crossrail
447:Flyvbjerg
415:parameter
193:Foresight
1072:(eds.).
1026:12858711
850:13203075
766:20 April
739:20 April
664:Archived
493:See also
376:Overview
118:Peak oil
91:Universe
39:Concepts
946:3981732
881:23 July
858:2238013
830:Bibcode
795:23 July
639:1914185
81:Climate
62:Future
1088:
1053:
1024:
944:
936:
856:
848:
710:
637:
619:
181:Delphi
934:S2CID
875:(PDF)
846:S2CID
820:arXiv
785:(PDF)
667:(PDF)
660:(PDF)
635:JSTOR
605:(PDF)
66:Earth
1086:ISBN
1051:ISBN
1022:PMID
967:help
942:SSRN
883:2021
854:SSRN
797:2021
768:2017
741:2017
708:ISBN
688:help
451:COWI
449:and
429:The
350:and
250:and
76:Race
1078:doi
1043:doi
926:doi
838:doi
700:doi
627:doi
342:or
296:TRL
276:FTA
1105::
1084:.
1076:.
1049:.
1041:.
1018:81
1016:.
1012:.
983:.
958::
956:}}
952:{{
940:.
932:.
852:.
844:.
836:.
828:.
816:37
814:.
787:.
757:.
730:.
706:.
679::
677:}}
673:{{
662:.
647:^
633:.
625:.
613:47
611:.
607:.
457:.
372:.
358:.
1094:.
1080::
1059:.
1045::
1028:.
993:.
969:)
965:(
948:.
928::
885:.
860:.
840::
832::
822::
799:.
770:.
743:.
716:.
702::
690:)
686:(
641:.
629::
328:e
321:t
314:v
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.
