43:. Capital One has been aggressive about testing since the firm was founded, testing everything from product design to marketing to customer selection to collection policies. In a single year, the company performs tens of thousands of tests, allowing it to offer thousands of different types of credit cards to customers, based on the knowledge gained from their tests. Richard Fairbank,
216:
Companies wanting to conduct Test and Learn must mobilize, track and analyze site and customer data at very granular levels. This can often be a massive and expensive undertaking. Increasingly however, an exponential increase in computing power, and corresponding decrease in its cost, have made
19:
is a set of practices followed by retailers, banks and other consumer-focused companies to test ideas in a small number of locations or customers to predict impact. The process is often designed to answer three questions about any tested program before rollout:
168:
tools available today to support systematic testing within an organization. However, there is currently no one software package that covers all types of tests, and in some cases significant knowledge of statistics is still required for effective analysis.
119:
ran a promotion that lowered the price of its foot-long sandwiches to $ 5. Before deciding to launch the promotion nationally, Subway ran a test that compared locations offering the $ 5 foot-long sandwiches to a group of similar
221:
February 2009 article "How to Design Smart
Business Experiments", Thomas Davenport discusses the Test and Learn Process, and describes how several companies have overcome these and other barriers to testing.
440:
354:
77:
frequently employs the Test and Learn methodology to determine the effectiveness of different homepage layouts, product recommendation algorithms, and pricing strategies.
463:
517:
192:, Trial Run and MarketDial. There are also a variety of more narrowly tailored products, focused on particular problems or fields. In
130:, a manufacturer and distributor of packaged foods, uses testing to optimize the placement of its products on grocery store shelves:
494:
144:, a recently bankrupt retail chain specializing in home decor and furnishings, uses testing to measure the effectiveness of
151:
Test and learn is also methodically used online to test impact of various web page layouts on user behavior. Tools such as
71:
The UK government, through research projects such as the
College for Teaching and Learning's Close the Gap: Test and learn
355:"Using A/B testing to measure the efficacy of recommendations generated by Amazon Personalize | AWS Machine Learning Blog"
180:, have developed experiment management software within the company. Alternatives to internal solutions include broad
155:
Website
Analyzer provide every site with the ability to test various elements and their effect on user engagement.
189:
24:
What impact will the program have on key performance indicators if executed across the network or customer base?
256:
340:
93:
83:
often employs the Test and Learn methodology for its algorithms, user interface, and even content creation.
47:
of
Capital One, called test and learn "a marketing revolution that can be applied to many businesses".
380:"PFLB Platform - Exploring Test and Learn: A Scientific Examination of Predictive Business Strategies"
102:, a Mid-Atlantic convenience store chain, uses Test and Learn across several aspects of its business:
246:
136:
Measuring the impact on product sales of a change to in-store sales representatives' responsibilities
294:
251:
548:
471:
319:
299:
218:
59:
Fast food restaurants launching a new menu item or product for a limited time in select locations
241:
404:
181:
8:
290:
116:
145:
121:
422:
315:
236:
165:
99:
96:
uses a test-and-learn approach to identify which stores should undergo renovations.
553:
412:
111:
Understanding how the opening a new location would impact existing nearby locations
74:
231:
558:
405:"Netflix & Big Data: The Strategic Ambivalence of an Entertainment Company"
141:
417:
133:
Determining the effect of a coffee aisle re-design on sales of premium coffees
542:
426:
193:
185:
27:
Will the program have a larger impact on some stores/customers than others?
261:
173:
127:
40:
39:
Test and learn has been systematically applied as far back as 1988 by
521:
201:
197:
80:
56:
Retail banks running different commercials in different markets.
205:
152:
68:
Retailers testing merchandising layouts across their door base
441:"High-tech secret behind the future of your local Woolworths"
379:
177:
62:
Big-box retailers evaluating “store of the future” programs
44:
518:"Case Study: How Search Ads Helped Pier 1 Stay Afloat"
30:Which components of the idea are actually working?
124:restaurants that were not offering the promotion.
108:Determining the impact of adding additional staff
105:Measuring the effect of introducing a new product
540:
87:
295:"How to Design Smart Business Experiments"
159:
492:
416:
289:
455:
188:or test and learn focused software from
461:
313:
65:Companies evaluating pricing strategies
541:
493:Angrisani, Carol (December 22, 2008).
402:
285:
283:
281:
279:
277:
34:
515:
208:have created specialized software.
13:
274:
14:
570:
495:"Coffee Aisle Revamp Lifts Sales"
341:"Closing the gap: Test and learn"
316:"This is a Marketing Revolution"
217:testing more accessible. In the
509:
314:Fishman, Charles (1999-04-30).
190:Applied Predictive Technologies
516:Wong, Elaine (June 24, 2009).
486:
462:Fleenor, D. Gail (June 2009).
433:
396:
372:
347:
333:
307:
257:Business process reengineering
1:
267:
196:, for example, firms such as
403:van Es, Karin (2022-09-26).
211:
7:
225:
50:
10:
575:
464:"These Tests Are Positive"
409:Television & New Media
172:Many companies, including
88:Test and learn in practice
443:. 9News. January 23, 2020
418:10.1177/15274764221125745
247:Evidence-based management
252:Evidence-based practices
300:Harvard Business Review
219:Harvard Business Review
184:software tools such as
164:There are a variety of
160:Test and learn software
242:Design of experiments
291:Davenport, Thomas H.
182:statistical analysis
411:: 152747642211257.
497:. Supermarket News
474:on 12 October 2009
146:search advertising
35:Historical context
293:(February 2009).
237:Scientific method
100:Wawa Food Markets
566:
533:
532:
530:
528:
513:
507:
506:
504:
502:
490:
484:
483:
481:
479:
470:. Archived from
459:
453:
452:
450:
448:
437:
431:
430:
420:
400:
394:
393:
391:
390:
384:platform.pflb.us
376:
370:
369:
367:
366:
351:
345:
344:
337:
331:
330:
328:
326:
311:
305:
304:
287:
574:
573:
569:
568:
567:
565:
564:
563:
539:
538:
537:
536:
526:
524:
514:
510:
500:
498:
491:
487:
477:
475:
468:STORES Magazine
460:
456:
446:
444:
439:
438:
434:
401:
397:
388:
386:
378:
377:
373:
364:
362:
353:
352:
348:
339:
338:
334:
324:
322:
312:
308:
288:
275:
270:
232:Clinical trials
228:
214:
162:
90:
53:
37:
12:
11:
5:
572:
562:
561:
556:
551:
549:Business terms
535:
534:
508:
485:
454:
432:
395:
371:
359:aws.amazon.com
346:
332:
306:
272:
271:
269:
266:
265:
264:
259:
254:
249:
244:
239:
234:
227:
224:
213:
210:
161:
158:
157:
156:
149:
142:Pier 1 Imports
139:
138:
137:
134:
125:
114:
113:
112:
109:
106:
97:
89:
86:
85:
84:
78:
72:
69:
66:
63:
60:
57:
52:
49:
36:
33:
32:
31:
28:
25:
17:Test and learn
9:
6:
4:
3:
2:
571:
560:
557:
555:
552:
550:
547:
546:
544:
527:September 22,
523:
519:
512:
501:September 22,
496:
489:
473:
469:
465:
458:
442:
436:
428:
424:
419:
414:
410:
406:
399:
385:
381:
375:
360:
356:
350:
342:
336:
325:September 22,
321:
317:
310:
302:
301:
296:
292:
286:
284:
282:
280:
278:
273:
263:
260:
258:
255:
253:
250:
248:
245:
243:
240:
238:
235:
233:
230:
229:
223:
220:
209:
207:
203:
199:
195:
194:web analytics
191:
187:
183:
179:
175:
170:
167:
154:
150:
147:
143:
140:
135:
132:
131:
129:
126:
123:
118:
115:
110:
107:
104:
103:
101:
98:
95:
92:
91:
82:
79:
76:
73:
70:
67:
64:
61:
58:
55:
54:
48:
46:
42:
29:
26:
23:
22:
21:
18:
525:. Retrieved
511:
499:. Retrieved
488:
478:18 September
476:. Retrieved
472:the original
467:
457:
445:. Retrieved
435:
408:
398:
387:. Retrieved
383:
374:
363:. Retrieved
361:. 2020-08-20
358:
349:
335:
323:. Retrieved
320:Fast Company
309:
298:
215:
171:
163:
38:
16:
15:
262:A/B testing
174:Capital One
128:Kraft Foods
41:Capital One
543:Categories
389:2023-07-26
365:2023-07-26
268:References
148:campaigns.
94:Woolworths
522:Brandweek
427:1527-4764
212:Criticism
202:WebTrends
226:See also
198:Omniture
166:software
51:Examples
554:Quality
447:July 6,
122:control
81:Netflix
425:
206:Google
204:, and
153:Google
117:Subway
75:Amazon
559:Tests
529:2009
503:2009
480:2009
449:2020
423:ISSN
327:2009
178:eBay
176:and
413:doi
186:SAS
45:CEO
545::
520:.
466:.
421:.
407:.
382:.
357:.
318:.
297:.
276:^
200:,
531:.
505:.
482:.
451:.
429:.
415::
392:.
368:.
343:.
329:.
303:.
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.