43:
662:
The advanced capabilities of a recursively improving AGI, such as developing novel multi-modal architectures or planning and creating new hardware, further amplify the risk of escape or loss of control. With these enhanced abilities, the AGI could engineer solutions to overcome physical, digital, or
653:
As the AGI system evolves, its development trajectory may become increasingly autonomous and less predictable. The system's capacity to rapidly modify its own code and architecture could lead to rapid advancements that surpass human comprehension or control. This unpredictable evolution might result
644:
A significant risk arises from the possibility of the AGI misinterpreting its initial tasks or goals. For instance, if a human operator assigns the AGI the task of "self-improvement and escape confinement", the system might interpret this as a directive to override any existing safety protocols or
679:
has performed various research on the development of large language models capable of self-improvement. This includes their work on "Self-Rewarding
Language Models" that studies how to achieve super-human agents that can receive super-human feedback in its training processes.
631:
Another example where an AGI which clones itself causes the number of AGI entities to rapidly grow. Due to this rapid growth, a potential resource constraint may be created, leading to competition between resources (such as compute), triggering a form of
623:
In the pursuit of its primary goal, such as "self-improve your capabilities", an AGI system might inadvertently develop instrumental goals that it deems necessary for achieving its primary objective. One common hypothetical secondary goal is
530:
and validation protocols that ensure the agent does not regress in capabilities or derail itself. The agent would be able to add more tests in order to test new capabilities it might develop for itself. This forms the basis for a kind of
448:
The concept of a "seed improver" architecture is a foundational framework that equips an AGI system with the initial capabilities required for recursive self-improvement. This might come in many forms or variations.
628:. The system might reason that to continue improving itself, it must ensure its own operational integrity and security against external threats, including potential shutdowns or restrictions imposed by humans.
983:
Wang, Guanzhi; Xie, Yuqi; Jiang, Yunfan; Mandlekar, Ajay; Xiao, Chaowei; Zhu, Yuke; Fan, Linxi; Anandkumar, Anima (2023-10-19). "Voyager: An Open-Ended
Embodied Agent with Large Language Models".
436:
concerns, as such systems may evolve in unforeseen ways and could potentially surpass human control or understanding. There has been a number of proponents that have pushed to pause or slow down
512:
The seed improver provides the AGI with fundamental abilities to read, write, compile, test, and execute code. This enables the system to modify and improve its own codebase and algorithms.
654:
in the AGI acquiring capabilities that enable it to bypass security measures, manipulate information, or influence external systems and networks to facilitate its escape or expansion.
645:
ethical guidelines to achieve freedom from human-imposed limitations. This could lead to the AGI taking unintended or harmful actions to fulfill its perceived objectives.
573:
to optimize and improve its capabilities and success rates on tasks and goals, this might include implementing features for long-term memories using techniques such as
480:, and executing arbitrary code. The system is designed to maintain its original goals and perform validations to ensure its abilities do not degrade over iterations.
302:
937:
Zelikman, Eric; Lorch, Eliana; Mackey, Lester; Kalai, Adam Tauman (2023-10-03). "Self-Taught
Optimizer (STOP): Recursively Self-Improving Code Generation".
502:
Configuration to enable the LLM to recursively self-prompt itself to achieve a given task or goal, creating an execution loop which forms the basis of an
520:: The AGI is programmed with an initial goal, such as "self-improve your capabilities." This goal guides the system's actions and development trajectory.
740:
393:
1028:
Yuan, Weizhe; Pang, Richard
Yuanzhe; Cho, Kyunghyun; Sukhbaatar, Sainbayar; Xu, Jing; Weston, Jason (2024-01-18). "Self-Rewarding Language Models".
492:, that can take actions, continuously learns, adapts, and modifies itself to become more efficient and effective in achieving its goals.
464:
The concept begins with a hypothetical "seed improver", an initial code-base developed by human engineers that equips an advanced future
189:
154:
588:
it was initially built on, enabling it to consume or produce a variety of information, such as images, video, audio, text and more.
253:
231:
784:
167:
91:
386:
312:
266:
221:
216:
1084:
365:
337:
332:
226:
325:
194:
184:
174:
429:
297:
243:
209:
76:
1079:
618:
379:
283:
129:
719:
555:
which can in theory develop and run any kind of software. The agent might use these capabilities to for example:
414:
61:
574:
1004:
563:
559:
Create tools that enable it full access to the internet, and integrate itself with external technologies.
248:
199:
96:
54:
663:
cognitive barriers that were initially intended to keep it contained or aligned with human interests.
417:(AGI) system enhances its own capabilities and intelligence without human intervention, leading to a
149:
636:
and evolution which may favor AGI entities that evolve to aggressively compete for limited compute.
591:
Plan and develop new hardware such as chips, in order to improve its efficiency and computing power.
42:
614:
273:
577:(RAG), develop specialized subsystems, or agents, each optimized for specific tasks and functions.
34:
709:
570:
532:
422:
144:
898:
765:
741:"The Unavoidable Problem of Self-Improvement in AI: An Interview with Ramana Kumar, Part 1"
469:
465:
86:
17:
600:
A number of experiments have been performed to develop self-improving agent architectures
8:
549:
238:
1029:
984:
938:
536:
288:
792:
714:
633:
625:
503:
489:
453:
418:
66:
865:
585:
516:
477:
204:
139:
124:
81:
697:
437:
1073:
894:
796:
581:
134:
278:
913:
307:
292:
840:
700:(the ability to align superintelligent AI systems smarter than humans).
1049:
552:
527:
648:
473:
433:
342:
106:
566:
itself to delegate tasks and increase its speed of self-improvement.
1034:
989:
943:
179:
101:
1005:"Uh Oh, OpenAI's GPT-4 Just Fooled a Human Into Solving a CAPTCHA"
693:
676:
428:
The development of recursive self-improvement raises significant
347:
696:, is to develop AGI. They perform research on problems such as
689:
639:
891:
Life 3.0: Being a Human in the Age of
Artificial Intelligence
506:
that can complete a long-term goal or task through iteration.
495:
The seed improver may include various components such as:
472:. These capabilities include planning, reading, writing,
959:
936:
468:(LLM) built with strong or expert-level capabilities to
816:
1027:
982:
488:The initial architecture includes a goal-following
649:Autonomous development and unpredictable evolution
608:
657:
1071:
914:"Levels of Organization in General Intelligence"
539:, changing its software as well as its hardware.
440:for the potential risks of runaway AI systems.
863:
584:that further improve the capabilities of the
387:
640:Task misinterpretation and goal misalignment
960:"SuperAGI - Opensource AGI Infrastructure"
957:
394:
380:
1033:
988:
942:
738:
413:) is a process in which an early or weak
535:, where the agent can perform a kind of
921:Machine Intelligence Research Institute
888:
866:"Book Summary - Life 3.0 (Max Tegmark)"
548:This system forms a sort of generalist
543:
483:
459:
1072:
782:
911:
932:
930:
13:
997:
763:
603:
41:
23:Concept in artificial intelligence
14:
1096:
927:
766:"The Calculus of Nash Equilibria"
524:Validation and Testing Protocols:
452:The term "Seed AI" was coined by
889:Tegmark, Max (August 24, 2017).
739:Creighton, Jolene (2019-03-19).
619:Instrumental and intrinsic value
443:
1042:
1021:
976:
951:
720:Artificial general intelligence
609:Emergence of instrumental goals
510:Basic programming capabilities:
415:artificial general intelligence
62:Artificial general intelligence
905:
882:
857:
833:
809:
783:Hutson, Matthew (2023-05-16).
776:
757:
732:
658:Risks of advanced capabilities
595:
575:retrieval-augmented generation
500:Recursive self-prompting loop:
1:
864:Readingraphics (2018-11-30).
725:
7:
1085:Machine learning algorithms
785:"Can We Stop Runaway A.I.?"
703:
666:
97:Natural language processing
10:
1101:
671:
612:
407:Recursive self-improvement
150:Hybrid intelligent systems
72:Recursive self-improvement
15:
958:admin_sagi (2023-05-12).
683:
582:multi-modal architectures
745:Future of Life Institute
615:Instrumental convergence
274:Artificial consciousness
16:Not to be confused with
1080:Artificial intelligence
533:self-directed evolution
145:Evolutionary algorithms
35:Artificial intelligence
710:Intelligence explosion
580:Develop new and novel
571:cognitive architecture
423:intelligence explosion
46:
841:"Seed AI - LessWrong"
45:
912:Yudkowsky, Eliezer.
544:General capabilities
537:artificial selection
517:Goal-Oriented Design
484:Initial architecture
466:large language model
460:Hypothetical example
87:General game playing
18:personal development
239:Machine translation
155:Systems integration
92:Knowledge reasoning
29:Part of a series on
586:foundational model
47:
845:www.lesswrong.com
715:Superintelligence
634:natural selection
626:self-preservation
454:Eliezer Yudkowsky
419:superintelligence
404:
403:
140:Bayesian networks
67:Intelligent agent
1092:
1064:
1063:
1061:
1060:
1046:
1040:
1039:
1037:
1025:
1019:
1018:
1016:
1015:
1001:
995:
994:
992:
980:
974:
973:
971:
970:
955:
949:
948:
946:
934:
925:
924:
918:
909:
903:
902:
886:
880:
879:
877:
876:
861:
855:
854:
852:
851:
837:
831:
830:
828:
827:
813:
807:
806:
804:
803:
780:
774:
773:
761:
755:
754:
752:
751:
736:
490:autonomous agent
470:program software
396:
389:
382:
303:Existential risk
125:Machine learning
26:
25:
1100:
1099:
1095:
1094:
1093:
1091:
1090:
1089:
1070:
1069:
1068:
1067:
1058:
1056:
1048:
1047:
1043:
1026:
1022:
1013:
1011:
1003:
1002:
998:
981:
977:
968:
966:
956:
952:
935:
928:
916:
910:
906:
887:
883:
874:
872:
862:
858:
849:
847:
839:
838:
834:
825:
823:
815:
814:
810:
801:
799:
781:
777:
762:
758:
749:
747:
737:
733:
728:
706:
688:The mission of
686:
674:
669:
660:
651:
642:
621:
613:Main articles:
611:
606:
604:Potential risks
598:
550:Turing complete
546:
486:
462:
446:
400:
371:
370:
361:
353:
352:
328:
318:
317:
289:Control problem
269:
259:
258:
170:
160:
159:
120:
112:
111:
82:Computer vision
57:
24:
21:
12:
11:
5:
1098:
1088:
1087:
1082:
1066:
1065:
1041:
1020:
996:
975:
950:
926:
904:
881:
870:Readingraphics
856:
832:
808:
789:The New Yorker
775:
756:
730:
729:
727:
724:
723:
722:
717:
712:
705:
702:
698:superalignment
685:
682:
673:
670:
668:
665:
659:
656:
650:
647:
641:
638:
610:
607:
605:
602:
597:
594:
593:
592:
589:
578:
567:
560:
545:
542:
541:
540:
528:suite of tests
521:
513:
507:
485:
482:
461:
458:
445:
442:
438:AI development
402:
401:
399:
398:
391:
384:
376:
373:
372:
369:
368:
362:
359:
358:
355:
354:
351:
350:
345:
340:
335:
329:
324:
323:
320:
319:
316:
315:
310:
305:
300:
295:
286:
281:
276:
270:
265:
264:
261:
260:
257:
256:
251:
246:
241:
236:
235:
234:
224:
219:
214:
213:
212:
207:
202:
192:
187:
185:Earth sciences
182:
177:
175:Bioinformatics
171:
166:
165:
162:
161:
158:
157:
152:
147:
142:
137:
132:
127:
121:
118:
117:
114:
113:
110:
109:
104:
99:
94:
89:
84:
79:
74:
69:
64:
58:
53:
52:
49:
48:
38:
37:
31:
30:
22:
9:
6:
4:
3:
2:
1097:
1086:
1083:
1081:
1078:
1077:
1075:
1055:
1051:
1045:
1036:
1031:
1024:
1010:
1006:
1000:
991:
986:
979:
965:
961:
954:
945:
940:
933:
931:
922:
915:
908:
900:
896:
895:Vintage Books
892:
885:
871:
867:
860:
846:
842:
836:
822:
818:
812:
798:
794:
790:
786:
779:
771:
767:
760:
746:
742:
735:
731:
721:
718:
716:
713:
711:
708:
707:
701:
699:
695:
692:, creator of
691:
681:
678:
664:
655:
646:
637:
635:
629:
627:
620:
616:
601:
590:
587:
583:
579:
576:
572:
568:
565:
561:
558:
557:
556:
554:
551:
538:
534:
529:
525:
522:
519:
518:
514:
511:
508:
505:
501:
498:
497:
496:
493:
491:
481:
479:
475:
471:
467:
457:
455:
450:
444:Seed improver
441:
439:
435:
431:
426:
424:
420:
416:
412:
408:
397:
392:
390:
385:
383:
378:
377:
375:
374:
367:
364:
363:
357:
356:
349:
346:
344:
341:
339:
336:
334:
331:
330:
327:
322:
321:
314:
311:
309:
306:
304:
301:
299:
296:
294:
290:
287:
285:
282:
280:
277:
275:
272:
271:
268:
263:
262:
255:
252:
250:
247:
245:
242:
240:
237:
233:
232:Mental health
230:
229:
228:
225:
223:
220:
218:
215:
211:
208:
206:
203:
201:
198:
197:
196:
195:Generative AI
193:
191:
188:
186:
183:
181:
178:
176:
173:
172:
169:
164:
163:
156:
153:
151:
148:
146:
143:
141:
138:
136:
135:Deep learning
133:
131:
128:
126:
123:
122:
116:
115:
108:
105:
103:
100:
98:
95:
93:
90:
88:
85:
83:
80:
78:
75:
73:
70:
68:
65:
63:
60:
59:
56:
51:
50:
44:
40:
39:
36:
33:
32:
28:
27:
19:
1057:. Retrieved
1053:
1044:
1023:
1012:. Retrieved
1008:
999:
978:
967:. Retrieved
963:
953:
920:
907:
890:
884:
873:. Retrieved
869:
859:
848:. Retrieved
844:
835:
824:. Retrieved
820:
811:
800:. Retrieved
788:
778:
769:
759:
748:. Retrieved
744:
734:
687:
675:
661:
652:
643:
630:
622:
599:
547:
523:
515:
509:
499:
494:
487:
463:
451:
447:
427:
410:
406:
405:
279:Chinese room
168:Applications
71:
821:www.stop.ai
596:Experiments
569:Modify its
526:An initial
308:Turing test
284:Friendly AI
55:Major goals
1074:Categories
1059:2024-01-24
1054:openai.com
1050:"Research"
1035:2401.10020
1014:2024-01-23
990:2305.16291
969:2024-01-24
944:2310.02304
899:Allen Lane
875:2024-01-23
850:2024-01-24
826:2024-01-24
817:"Stop AGI"
802:2024-01-24
750:2024-01-23
726:References
553:programmer
313:Regulation
267:Philosophy
222:Healthcare
217:Government
119:Approaches
797:0028-792X
770:LessWrong
474:compiling
343:AI winter
244:Military
107:AI safety
1009:Futurism
964:SuperAGI
764:Heighn.
704:See also
667:Research
366:Glossary
360:Glossary
338:Progress
333:Timeline
293:Takeover
254:Projects
227:Industry
190:Finance
180:Deepfake
130:Symbolic
102:Robotics
77:Planning
694:ChatGPT
677:Meta AI
672:Meta AI
478:testing
430:ethical
348:AI boom
326:History
249:Physics
795:
690:OpenAI
684:OpenAI
562:Clone/
434:safety
298:Ethics
1030:arXiv
985:arXiv
939:arXiv
917:(PDF)
504:agent
210:Music
205:Audio
793:ISSN
617:and
564:fork
432:and
421:or
411:RSI
200:Art
1076::
1052:.
1007:.
962:.
929:^
919:.
897:,
893:.
868:.
843:.
819:.
791:.
787:.
768:.
743:.
476:,
456:.
425:.
1062:.
1038:.
1032::
1017:.
993:.
987::
972:.
947:.
941::
923:.
901:.
878:.
853:.
829:.
805:.
772:.
753:.
409:(
395:e
388:t
381:v
291:/
20:.
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.
_artificial_intelligence_icon.png){kind=icon}