465:), it is possible to concatenate disks, but also volumes such as RAID sets. With larger drive capacities, write delay and rebuilding time increase dramatically (especially, as described above, with RAID 5 and RAID 6). By splitting a larger RAID N set into smaller subsets and concatenating them with linear JBOD, write and rebuilding time will be reduced. If a hardware RAID controller is not capable of nesting linear JBOD with RAID N, then linear JBOD can be achieved with OS-level software RAID in combination with separate RAID N subset volumes created within one, or more, hardware RAID controller(s). Besides a drastic speed increase, this also provides a substantial advantage: the possibility to start a linear JBOD with a small set of disks and to be able to expand the total set with disks of different size, later on (in time, disks of bigger size become available on the market). There is another advantage in the form of disaster recovery (if a RAID N subset happens to fail, then the data on the other RAID N subsets is not lost, reducing restore time).
857:
906:, which limits the error recovery time to seven seconds. Around September 2009, Western Digital disabled this feature in their desktop drives (such as the Caviar Black line), making such drives unsuitable for use in RAID configurations. However, Western Digital enterprise class drives are shipped from the factory with TLER enabled. Similar technologies are used by Seagate, Samsung, and Hitachi. For non-RAID usage, an enterprise class drive with a short error recovery timeout that cannot be changed is therefore less suitable than a desktop drive. In late 2010, the
1084:, which is a caching system that reports the data as written as soon as it is written to cache, as opposed to when it is written to the non-volatile medium. If the system experiences a power loss or other major failure, the data may be irrevocably lost from the cache before reaching the non-volatile storage. For this reason good write-back cache implementations include mechanisms, such as redundant battery power, to preserve cache contents across system failures (including power failures) and to flush the cache at system restart time.
256:
966:, as a background process, can be used to detect and recover from UREs, effectively reducing the risk of them happening during RAID rebuilds and causing double-drive failures. The recovery of UREs involves remapping of affected underlying disk sectors, utilizing the drive's sector remapping pool; in case of UREs detected during background scrubbing, data redundancy provided by a fully operational RAID set allows the missing data to be reconstructed and rewritten to a remapped sector.
404:
four disks. As with RAID 5, a single drive failure results in reduced performance of the entire array until the failed drive has been replaced. With a RAID 6 array, using drives from multiple sources and manufacturers, it is possible to mitigate most of the problems associated with RAID 5. The larger the drive capacities and the larger the array size, the more important it becomes to choose RAID 6 instead of RAID 5. RAID 10 also minimizes these problems.
329:), improving performance. Sustained read throughput, if the controller or software is optimized for it, approaches the sum of throughputs of every drive in the set, just as for RAID 0. Actual read throughput of most RAID 1 implementations is slower than the fastest drive. Write throughput is always slower because every drive must be updated, and the slowest drive limits the write performance. The array continues to operate as long as at least one drive is functioning.
875:
dedicated RAID controller chip, but simply a standard drive controller chip, or the chipset built-in RAID function, with proprietary firmware and drivers. During early bootup, the RAID is implemented by the firmware and, once the operating system has been more completely loaded, the drivers take over control. Consequently, such controllers may not work when driver support is not available for the host operating system. An example is
304:, the capacity of a RAID 0 volume is the same; it is the sum of the capacities of the drives in the set. But because striping distributes the contents of each file among all drives in the set, the failure of any drive causes the entire RAID 0 volume and all files to be lost. In comparison, a spanned volume preserves the files on the unfailing drives. The benefit of RAID 0 is that the
898:) involves periodic reading and checking by the RAID controller of all the blocks in an array, including those not otherwise accessed. This detects bad blocks before use. Data scrubbing checks for bad blocks on each storage device in an array, but also uses the redundancy of the array to recover bad blocks on a single drive and to reassign the recovered data to spare blocks elsewhere on the drive.
382:. The main advantage of RAID 4 over RAID 2 and 3 is I/O parallelism: in RAID 2 and 3, a single read I/O operation requires reading the whole group of data drives, while in RAID 4 one I/O read operation does not have to spread across all data drives. As a result, more I/O operations can be executed in parallel, improving the performance of small transfers.
902:
complete its internal error recovery procedure. Consequently, using consumer-marketed drives with RAID can be risky, and so-called "enterprise class" drives limit this error recovery time to reduce risk. Western
Digital's desktop drives used to have a specific fix. A utility called WDTLER.exe limited a drive's error recovery time. The utility enabled
1179:"Originally referred to as Redundant Array of Inexpensive Disks, the term RAID was first published in the late 1980s by Patterson, Gibson, and Katz of the University of California at Berkeley. (The RAID Advisory Board has since substituted the term Inexpensive with Independent.)" Storage Area Network Fundamentals; Meeta Gupta; Cisco Press;
932:
in fact statistically correlated. In practice, the chances for a second failure before the first has been recovered (causing data loss) are higher than the chances for random failures. In a study of about 100,000 drives, the probability of two drives in the same cluster failing within one hour was four times larger than predicted by the
1381:
Since a large number of bits are handled in parallel, it is practical to use error checking and correction (ECC) bits, and each 39 bit byte is composed of 32 data bits and seven ECC bits. The ECC bits accompany all data transferred to or from the high-speed disks, and, on reading, are used to correct
961:
Double-protection parity-based schemes, such as RAID 6, attempt to address this issue by providing redundancy that allows double-drive failures; as a downside, such schemes suffer from elevated write penalty—the number of times the storage medium must be accessed during a single write operation.
957:
or SATA), and less than one bit in 10 for desktop-class drives (IDE/ATA/PATA or SATA). Increasing drive capacities and large RAID 5 instances have led to the maximum error rates being insufficient to guarantee a successful recovery, due to the high likelihood of such an error occurring on one or
320:
consists of data mirroring, without parity or striping. Data is written identically to two or more drives, thereby producing a "mirrored set" of drives. Thus, any read request can be serviced by any drive in the set. If a request is broadcast to every drive in the set, it can be serviced by the drive
990:
Mirroring schemes such as RAID 10 have a bounded recovery time as they require the copy of a single failed drive, compared with parity schemes such as RAID 6, which require the copy of all blocks of the drives in an array set. Triple parity schemes, or triple mirroring, have been suggested
975:
also limited if the entire array is still in operation at reduced capacity. Given an array with only one redundant drive (which applies to RAID levels 3, 4 and 5, and to "classic" two-drive RAID 1), a second drive failure would cause complete failure of the array. Even though individual drives'
974:
Drive capacity has grown at a much faster rate than transfer speed, and error rates have only fallen a little in comparison. Therefore, larger-capacity drives may take hours if not days to rebuild, during which time other drives may fail or yet undetected read errors may surface. The rebuild time is
931:
In practice, the drives are often the same age (with similar wear) and subject to the same environment. Since many drive failures are due to mechanical issues (which are more likely on older drives), this violates the assumptions of independent, identical rate of failure amongst drives; failures are
874:
Software-implemented RAID is not always compatible with the system's boot process, and it is generally impractical for desktop versions of
Windows. However, hardware RAID controllers are expensive and proprietary. To fill this gap, inexpensive "RAID controllers" were introduced that do not contain a
515:
scatters dual (or more) copies of the data across all disks (possibly hundreds) in a storage subsystem, while holding back enough spare capacity to allow for a few disks to fail. The scattering is based on algorithms which give the appearance of arbitrariness. When one or more disks fail the missing
441:
creates two stripes and mirrors them. If a single drive failure occurs then one of the mirrors has failed, at this point it is running effectively as RAID 0 with no redundancy. Significantly higher risk is introduced during a rebuild than RAID 1+0 as all the data from all the drives in the remaining
390:
consists of block-level striping with distributed parity. Unlike RAID 4, parity information is distributed among the drives, requiring all drives but one to be present to operate. Upon failure of a single drive, subsequent reads can be calculated from the distributed parity such that no data is
882:
Because some minimal hardware support is involved, this implementation is also called "hardware-assisted software RAID", "hybrid model" RAID, or even "fake RAID". If RAID 5 is supported, the hardware may provide a hardware XOR accelerator. An advantage of this model over the pure software RAID
403:
consists of block-level striping with double distributed parity. Double parity provides fault tolerance up to two failed drives. This makes larger RAID groups more practical, especially for high-availability systems, as large-capacity drives take longer to restore. RAID 6 requires a minimum of
488:
provides a general RAID driver that in its "near" layout defaults to a standard RAID 1 with two drives, and a standard RAID 1+0 with four drives; however, it can include any number of drives, including odd numbers. With its "far" layout, MD RAID 10 can run both striped and mirrored,
1220:
We were not the first to think of the idea of replacing what
Patterson described as a slow large expensive disk (SLED) with an array of inexpensive disks. For example, the concept of disk mirroring, pioneered by Tandem, was well known, and some storage products had already been constructed around
986:
study of
Berriman et al., the chance of failure decreases by a factor of about 3,800 (relative to RAID 5) for a proper implementation of RAID 6, even when using commodity drives. Nevertheless, if the currently observed technology trends remain unchanged, in 2019 a RAID 6 array will
917:
An array can be overwhelmed by catastrophic failure that exceeds its recovery capacity and the entire array is at risk of physical damage by fire, natural disaster, and human forces, however backups can be stored off site. An array is also vulnerable to controller failure because it is not always
698:
supports the equivalents of RAID 0, RAID 1, RAID 5 (RAID-Z1) single-parity, RAID 6 (RAID-Z2) double-parity, and a triple-parity version (RAID-Z3) also referred to as RAID 7. As it always stripes over top-level vdevs, it supports equivalents of the 1+0, 5+0, and 6+0 nested
913:
While RAID may protect against physical drive failure, the data is still exposed to operator, software, hardware, and virus destruction. Many studies cite operator fault as a common source of malfunction, such as a server operator replacing the incorrect drive in a faulty RAID, and disabling the
901:
Frequently, a RAID controller is configured to "drop" a component drive (that is, to assume a component drive has failed) if the drive has been unresponsive for eight seconds or so; this might cause the array controller to drop a good drive because that drive has not been given enough time to
480:
Many configurations other than the basic numbered RAID levels are possible, and many companies, organizations, and groups have created their own non-standard configurations, in many cases designed to meet the specialized needs of a small niche group. Such configurations include the following:
509:, the parallel file system, has internal striping (comparable to file-based RAID0) and replication (comparable to file-based RAID10) options to aggregate throughput and capacity of multiple servers and is typically based on top of an underlying RAID to make disk failures transparent.
1410:
A typical IBM 7030 Data
Processing System might have been comprised of the following units: IBM 353 Disk Storage Unit – similar to IBM 1301 Disk File, but much faster. 2,097,152 (2^21) 72-bit words (64 data bits and 8 ECC bits), 125,000 words per
958:
more remaining drives during a RAID set rebuild. When rebuilding, parity-based schemes such as RAID 5 are particularly prone to the effects of UREs as they affect not only the sector where they occur, but also reconstructed blocks using that sector for parity computation.
999:
A system crash or other interruption of a write operation can result in states where the parity is inconsistent with the data due to non-atomicity of the write process, such that the parity cannot be used for recovery in the case of a disk failure. This is commonly termed the
721:
protection schemes up to n+3. A particularity is the dynamic rebuilding priority which runs with low impact in the background until a data chunk hits n+0 redundancy, in which case this chunk is quickly rebuilt to at least n+1. On top, Spectrum Scale supports metro-distance
1340:
944:
Unrecoverable read errors (URE) present as sector read failures, also known as latent sector errors (LSE). The associated media assessment measure, unrecoverable bit error (UBE) rate, is typically guaranteed to be less than one bit in 10 for enterprise-class drives
237:(SSDs) without the expense of an all-SSD system. For example, a fast SSD can be mirrored with a mechanical drive. For this configuration to provide a significant speed advantage, an appropriate controller is needed that uses the fast SSD for all read operations.
1267:
979:(MTBF) have increased over time, this increase has not kept pace with the increased storage capacity of the drives. The time to rebuild the array after a single drive failure, as well as the chance of a second failure during a rebuild, have increased over time.
601:
Some other operating systems have implemented their own generic frameworks for interfacing with any RAID controller, and provide tools for monitoring RAID volume status, as well as facilitation of drive identification through LED blinking, alarm management and
835:
If a boot drive fails, the system has to be sophisticated enough to be able to boot from the remaining drive or drives. For instance, consider a computer whose disk is configured as RAID 1 (mirrored drives); if the first drive in the array fails, then a
1321:
391:
lost. RAID 5 requires at least three disks. Like all single-parity concepts, large RAID 5 implementations are susceptible to system failures because of trends regarding array rebuild time and the chance of drive failure during rebuild (see "
533:. A software solution may be part of the operating system, part of the firmware and drivers supplied with a standard drive controller (so-called "hardware-assisted software RAID"), or it may reside entirely within the hardware RAID controller.
516:
copies are rebuilt into that spare capacity, again arbitrarily. Because the rebuild is done from and to all the remaining disks, it operates much faster than with traditional RAID, reducing the overall impact on clients of the storage system.
91:
and performance. The different schemes, or data distribution layouts, are named by the word "RAID" followed by a number, for example RAID 0 or RAID 1. Each scheme, or RAID level, provides a different balance among the key goals:
936:—which characterizes processes in which events occur continuously and independently at a constant average rate. The probability of two failures in the same 10-hour period was twice as large as predicted by an exponential distribution.
349:
is on a different drive. Hamming-code parity is calculated across corresponding bits and stored on at least one parity drive. This level is of historical significance only; although it was used on some early machines (for example, the
734:
was originally designed to provide an integrated volume manager that supports concatenating, mirroring and striping of multiple physical storage devices. However, the implementation of XFS in Linux kernel lacks the integrated volume
312:. The cost is increased vulnerability to drive failures—since any drive in a RAID 0 setup failing causes the entire volume to be lost, the average failure rate of the volume rises with the number of attached drives.
2978:
Ulf
Troppens, Wolfgang Mueller-Friedt, Rainer Erkens, Rainer Wolafka, Nils Haustein. Storage Networks Explained: Basics and Application of Fibre Channel SAN, NAS, ISCSI, InfiniBand and FCoE. John Wiley and Sons, 2009.
819:, which also allows users to specify mirroring, parity, or no redundancy on a folder-by-folder basis. These options are similar to RAID 1 and RAID 5, but are implemented at a higher abstraction level.
1038:
uses a "write-intent-bitmap". If it finds any location marked as incompletely written at startup, it resyncs them. It closes the write hole but does not protect against loss of in-transit data, unlike a full
579:, without a need for any third-party tools, each manufacturer of each RAID controller usually provides their own proprietary software tooling for each operating system that they deem to support, ensuring a
151:
Although not yet using that terminology, the technologies of the five levels of RAID named in the June 1988 paper were used in various products prior to the paper's publication, including the following:
962:
Schemes that duplicate (mirror) data in a drive-to-drive manner, such as RAID 1 and RAID 10, have a lower risk from UREs than those using parity computation or mirroring between striped sets.
366:
is on a different drive. Parity is calculated across corresponding bytes and stored on a dedicated parity drive. Although implementations exist, RAID 3 is not commonly used in practice.
1046:
can save a "partial parity" that, when combined with modified chunks, recovers the original parity. This closes the write hole, but again does not protect against loss of in-transit data.
2688:
1250:
Patterson recalled the beginnings of his RAID project in 1987. 1988: David A. Patterson leads a team that defines RAID standards for improved performance, reliability and scalability.
982:
Some commentators have declared that RAID 6 is only a "band aid" in this respect, because it only kicks the problem a little further down the road. However, according to the 2006
148:
market. Although failures would rise in proportion to the number of drives, by configuring for redundancy, the reliability of an array could far exceed that of any large single drive.
883:
is that—if using a redundancy mode—the boot drive is protected from failure (due to the firmware) during the boot process even before the operating system's drivers take over.
84:, performance improvement, or both. This is in contrast to the previous concept of highly reliable mainframe disk drives referred to as "single large expensive disk" (SLED).
3252:
Baker, M.; Shah, M.; Rosenthal, D.S.H.; Roussopoulos, M.; Maniatis, P.; Giuli, T.; Bungale, P (April 2006). "A fresh look at the reliability of long-term digital storage".
2809:
1920:
598:, and use the Linux tooling from Adaptec, potentially compromising the stability, reliability and security of their setup, especially when taking the long-term view.
1004:
which is a known data corruption issue in older and low-end RAIDs, caused by interrupted destaging of writes to disk. The write hole can be addressed in a few ways:
1068:
Write hole is a little understood and rarely mentioned failure mode for redundant storage systems that do not utilize transactional features. Database researcher
442:
stripe has to be read rather than just from one drive, increasing the chance of an unrecoverable read error (URE) and significantly extending the rebuild window.
910:
program began supporting the configuration of ATA Error
Recovery Control, allowing the tool to configure many desktop class hard drives for use in RAID setups.
1547:
781:
supports RAID 0, RAID 1, RAID 4, RAID 5, RAID 6, and all nestings. Certain reshaping/resizing/expanding operations are also supported.
430:
The final array is known as the top array. When the top array is RAID 0 (such as in RAID 1+0 and RAID 5+0), most vendors omit the "+" (yielding
2988:
Dell
Computers, Background Patrol Read for Dell PowerEdge RAID Controllers, By Drew Habas and John Sieber, Reprinted from Dell Power Solutions, February 2006
395:" section, below). Rebuilding an array requires reading all data from all disks, opening a chance for a second drive failure and the loss of the entire array.
1783:
2232:
3033:
17:
1281:
3182:
2384:
1114:
1755:
3879:
452:) creates a striped set from a series of mirrored drives. The array can sustain multiple drive losses so long as no mirror loses all its drives.
750:
operating system supports RAID 1. The mirrored disks, called a "shadow set", can be in different locations to assist in disaster recovery.
362:
consists of byte-level striping with dedicated parity. All disk spindle rotation is synchronized and data is striped such that each sequential
3566:
2692:
2128:
2182:
691:
are designed to organize data across multiple storage devices directly, without needing the help of a third-party logical volume manager:
308:
of read and write operations to any file is multiplied by the number of drives because, unlike spanned volumes, reads and writes are done
190:
used error correction codes (now known as RAID 2) in an array of disk drives. A similar approach was used in the early 1960s on the
1440:
Chen, Peter; Lee, Edward; Gibson, Garth; Katz, Randy; Patterson, David (1994). "RAID: High-Performance, Reliable
Secondary Storage".
144:
disk drives of the time could be beaten on performance by an array of the inexpensive drives that had been developed for the growing
3222:
2923:
606:
designations from within the operating system without having to reboot into card BIOS. For example, this was the approach taken by
3503:
276:
493:
layout; this runs mirroring with striped reads, giving the read performance of RAID 0. Regular RAID 1, as provided by
3914:
2632:"You cannot select or format a hard disk partition when you try to install Windows Vista, Windows 7 or Windows Server 2008 R2"
2959:
2898:
2838:
1824:
1184:
1109:
87:
Data is distributed across the drives in one of several ways, referred to as RAID levels, depending on the required level of
3872:
3081:
Murphy, Brendan; Gent, Ted (1995). "Measuring system and software reliability using an automated data collection process".
1139:
121:
3006:
2639:
2178:
699:
RAID levels (as well as striped triple-parity sets) but not other nested combinations. ZFS is the native file system on
1668:
1338:, David Potter et al., "Method and Apparatus for Operating Multi-Unit Array of Memories", issued 1990-02-06
133:
3317:
2602:
227:
3624:
1544:
1358:
718:
512:
714:
259:
Storage servers with 24 hard disk drives each and built-in hardware RAID controllers supporting various RAID levels
3607:
614:
utility, which provide volume status, and allow LED/alarm/hotspare control, as well as the sensors (including the
3865:
3183:"Matrix methods for lost data reconstruction in erasure codes. USENIX Conference on File and Storage Technologies
2240:
1234:
378:, but has now been largely replaced by a proprietary implementation of RAID 4 with two parity disks, called
3064:
1731:
3657:
3302:
Proceedings of the 2007 ACM SIGMETRICS international conference on
Measurement and modeling of computer systems
3269:
1979:
1292:
876:
707:, and is also available on FreeBSD and Linux. Open-source ZFS implementations are actively developed under the
662:
309:
222:, but RAID 6 uses two separate parities based respectively on addition and multiplication in a particular
2392:
136:
in 1987. In their June 1988 paper "A Case for Redundant Arrays of Inexpensive Disks (RAID)", presented at the
2764:
2285:
2261:
1288:
1069:
351:
184:
177:
began shipping subsystem mirrored RA8X disk drives (now known as RAID 1) as part of its HSC50 subsystem.
174:
2661:
1265:, Norman Ken Ouchi, "System for Recovering Data Stored in Failed Memory Unit", issued 1978-05-30
1064:, which uses a copying garbage collector, chooses this option. COW again protect references to striped data.
3848:
1104:
571:, which can usually be configured and serviced entirely through the common operating system paradigms like
564:
3574:
1765:
1706:"RAID-DP:NetApp Implementation of Double Parity RAID for Data Protection. NetApp Technical Report TR-3298"
1072:
wrote "Update in Place is a Poison Apple" during the early days of relational database commercialization.
856:
3117:
2439:
2095:
Murenin, Constantine A. (2010-05-21). "1.1. Motivation; 4. Sensor Drivers; 7.1. NetBSD envsys / sysmon".
1954:
1804:
903:
427:
may be either individual drives or arrays themselves. Arrays are rarely nested more than one level deep.
263:
Originally, there were five standard levels of RAID, but many variations have evolved, including several
3630:
3294:
2517:
3157:
2989:
700:
160:
3484:
2924:"Hardware RAID vs. Software RAID: Which Implementation is Best for my Application? Adaptec Whitepaper"
1335:
1316:
1262:
844:
from the second drive as a fallback. The second-stage boot loader for FreeBSD is capable of loading a
105:
1892:
1319:, Brian E. Clark, et al., "Parity Spreading to Enhance Storage Access", issued 1988-08-02
991:
as one approach to improve resilience to an additional drive failure during this large rebuild time.
976:
869:
845:
31:
2310:
2102:
1594:
1530:
683:
A layer that sits above any file system and provides parity protection to user data (such as RAID-F)
3936:
3746:
3563:
2132:
1454:
1151:
1098:
1057:) transactional semantics guard metadata associated with stripes. The downside is IO fragmentation.
933:
503:
has a RAID system that generates a parity file by xor-ing a stripe of blocks in a single HDFS file.
475:
268:
88:
2477:
1705:
3909:
3792:
1201:
1080:
There are concerns about write-cache reliability, specifically regarding devices equipped with a
93:
799:, but this was limited only to professional and server editions of Windows until the release of
3888:
3613:
2207:
2106:
1449:
180:
In 1986, Clark et al. at IBM filed a patent disclosing what was subsequently named RAID 5.
73:
2884:
2053:
3814:
3708:
2945:
2828:
2341:
728:
supports RAID 0, RAID 1 and RAID 10 (RAID 5 and 6 are under development).
35:
3540:
2715:
1933:
1394:
3736:
3380:
2890:
954:
841:
788:
787:
supports RAID 0, RAID 1, and RAID 5 using various software implementations.
563:
configuration utilities are available from the manufacturer of each controller. Unlike the
560:
272:
250:
101:
3137:
2073:
1784:"How to protect yourself from RAID-related Unrecoverable Read Errors (UREs). Techrepublic"
374:
consists of block-level striping with dedicated parity. This level was previously used by
8:
3946:
3921:
3899:
3230:
1022:
1008:
812:
658:
595:
494:
198:
Industry manufacturers later redefined the RAID acronym to stand for "redundant array of
2413:
1868:
795:, allows for the creation of RAID 0, RAID 1, and RAID 5 volumes by using
3741:
3650:
3511:
3323:
3293:
Bairavasundaram, L.N.; Goodson, G.R.; Pasupathy, S.; Schindler, J. (June 12–16, 2007).
3275:
3056:
2289:
2265:
2211:
1518:
1467:
673:
414:
326:
264:
141:
3515:
3425:"ANNOUNCE: mdadm 3.4 - A tool for managing md Soft RAID under Linux [LWN.net]"
3254:
Proceedings of the 1st ACM SIGOPS/EuroSys European Conference on Computer Systems 2006
2859:
2788:
1684:
1053:
ensures that each block is its own stripe, so every block is complete. Copy-on-write (
1017:
mdadm can use a dedicated journaling device (to avoid performance penalty, typically,
767:
supports RAID 0, RAID 1, RAID 3, and RAID 5, and all nestings via
3587:
3313:
3265:
3196:
2955:
2894:
2834:
2665:
2208:"Creating and Destroying ZFS Storage Pools – Oracle Solaris ZFS Administration Guide"
1820:
1664:
1471:
1180:
784:
618:) for health monitoring; this approach has subsequently been adopted and extended by
530:
526:
234:
145:
3698:
3327:
2362:
1566:
864:
controller that provides RAID functionality through proprietary firmware and drivers
3904:
3305:
3279:
3257:
3140:
3090:
3060:
3048:
2951:
2153:
2110:
1459:
1093:
1081:
825:
supports RAID 0, 1, 4, and 5 via its software implementation, named RAIDframe.
631:
556:
525:
The distribution of data across multiple drives can be managed either by dedicated
485:
157:
47:
3138:
Disk Failures in the Real World: What Does an MTTF of 1,000,000 Hours Mean to You?
1382:
a single bit error in a byte and detect double and most multiple errors in a byte.
717:, initially developed by IBM for media streaming and scalable analytics, supports
211:
3772:
3767:
3682:
3675:
3570:
3144:
2558:
1809:
Proceedings of the 1995 International Conference on Parallel Processing: Volume 1
1551:
1143:
743:
542:
215:
81:
2497:
108:. RAID levels greater than RAID 0 provide protection against unrecoverable
3713:
3617:
3361:"Triple-Parity RAID and Beyond. ACM Queue, Association for Computing Machinery"
3010:
2631:
2538:
891:
816:
654:
639:
603:
580:
301:
293:
3360:
2186:
1346:
831:
supports RAID 0, 1 and 5 via its software implementation, named softraid.
739:
Many operating systems provide RAID implementations, including the following:
3967:
3961:
3777:
3703:
3643:
2069:
1054:
950:
907:
796:
615:
423:, many storage controllers allow RAID levels to be nested. The elements of a
342:
289:
156:
Mirroring (RAID 1) was well established in the 1970s including, for example,
3309:
3261:
1685:"FreeBSD Handbook, Chapter 20.5 GEOM: Modular Disk Transformation Framework"
987:
have the same chance of failure as its RAID 5 counterpart had in 2010.
297:
255:
30:
This article is about the data storage technology. For the police unit, see
3941:
3926:
3762:
3718:
3094:
2965:
2904:
2844:
1830:
1619:
792:
758:
643:
338:
223:
219:
125:
97:
2610:
1463:
3857:
3799:
3728:
918:
possible to migrate it to a new, different controller without data loss.
688:
275:). RAID levels and their associated data formats are standardized by the
109:
3292:
2003:
1805:"Dual-Crosshatch Disk Array: A Highly Reliable Hybrid-RAID Architecture"
1014:
Hardware RAID systems use an onboard nonvolatile cache for this purpose.
3837:
3804:
3787:
3690:
3678:
3223:"RAID 5 versus RAID 10 (or even RAID 3, or RAID 4)"
2037:
1147:
1031:
837:
804:
552:
305:
129:
77:
2457:
2114:
354:
CM-2), as of 2014 it is not used by any commercially available system.
3052:
2810:"Red Hat Enterprise Linux – Storage Administrator Guide – RAID Types"
2740:
2635:
1812:
808:
800:
322:
187:
3406:
2990:
http://www.dell.com/downloads/global/power/ps1q06-20050212-Habas.pdf
638:
A layer that abstracts multiple devices, thereby providing a single
392:
3782:
3471:
3442:
3424:
2578:
2098:
OpenBSD Hardware Sensors — Environmental Monitoring and Fan Control
1061:
861:
572:
568:
112:
read errors, as well as against failures of whole physical drives.
2262:"Double Parity RAID-Z (raidz2) (Solaris ZFS Administration Guide)"
1869:"RAID 10 Vs RAID 01 (RAID 1+0 Vs RAID 0+1) Explained with Diagram"
1844:
969:
170:
filed a patent disclosing what was subsequently named RAID 4.
2286:"Triple Parity RAIDZ (raidz3) (Solaris ZFS Administration Guide)"
2081:
2045:
2033:
2011:
828:
764:
747:
708:
704:
607:
591:
587:
449:
431:
379:
238:
191:
3822:
1050:
983:
822:
619:
611:
506:
500:
399:
386:
375:
370:
358:
333:
316:
284:
137:
3608:"Empirical Measurements of Disk Failure Rates and Error Rates"
3251:
3034:"A census of Tandem system availability between 1985 and 1990"
1958:
1556:
Colorado Springs, 28 July 2006. Retrieved on 22 February 2011.
1202:"RAID: A Personal Recollection of How Storage Became a System"
341:
parity. All disk spindle rotation is synchronized and data is
80:
components into one or more logical units for the purposes of
3832:
3467:
1760:
1043:
1035:
807:
can be modified to unlock support for RAID 0, 1, and 5.
778:
774:
754:
725:
677:
647:
214:", a widely used method in information technology to provide
2689:"Virtualizing storage for scale, resiliency, and efficiency"
279:(SNIA) in the Common RAID Disk Drive Format (DDF) standard:
3827:
946:
939:
768:
657:(provided with most server-class operating systems such as
576:
548:
497:, does not stripe reads, but can perform reads in parallel.
456:
363:
210:
Many RAID levels employ an error protection scheme called "
53:
3564:
Jim Gray: The Transaction Concept: Virtues and Limitations
3461:
2830:
Working with Windows Small Business Server 2011 Essentials
630:
Software RAID implementations are provided by many modern
3181:
J.L. Hafner, V. Dheenadhayalan, K. Rao, and J.A. Tomlin.
2827:
Russel, Charlie; Crawford, Sharon; Edney, Andrew (2011).
2603:"Windows Vista support for large-sector hard disk drives"
2321:
2096:
1501:
1369:
1018:
840:
might not be sophisticated enough to attempt loading the
731:
695:
669:
547:
Hardware RAID controllers can be configured through card
346:
167:
1934:"Performance, Tools & General Bone-Headed Questions"
3635:
3381:""Write Hole" in RAID5, RAID6, RAID1, and Other Arrays"
1802:
1153:
A Case for Redundant Arrays of Inexpensive Disks (RAID)
2769:
Chapter 19 GEOM: Modular Disk Transformation Framework
2437:
1661:
Computer Architecture: A Quantitative Approach, 4th ed
559:
is booted, and after the operating system is booted,
3610:, by Jim Gray and Catharine van Ingen, December 2005
3295:"An analysis of latent sector errors in disk drives"
2518:"freebsd-geom mailing list – new class / geom_raid5"
2458:"Mac OS X: How to combine RAID sets in Disk Utility"
2385:"Scalability and Performance in Modern File Systems"
1487:
MCSA/MCSE 2006 JumpStart Computer and Network Basics
761:
support RAID 0, RAID 1, and RAID 1+0.
56:
50:
3588:"Definition of write-back cache at SNIA dictionary"
3344:. New York: Morgan Kaufmann Publishers. pp 604–605.
2826:
1395:"IBM Stretch (aka IBM 7030 Data Processing System)"
1359:"IBM 7030 Data Processing System: Reference Manual"
1138:
879:, implemented on many consumer-level motherboards.
3197:"Understanding RAID Performance at Various Levels"
2820:
2498:"FreeBSD System Manager's Manual page for GEOM(8)"
1115:Self-Monitoring, Analysis and Reporting Technology
3083:Quality and Reliability Engineering International
2680:
2311:"General Parallel File System (GPFS) Native RAID"
2052:"aac -- Adaptec AdvancedRAID Controller driver".
2038:"aac(4) — Adaptec AdvancedRAID Controller driver"
1658:
1503:Redundant Arrays of Independent Disks from FOLDOC
1439:
1435:
1433:
1431:
1429:
1427:
1425:
1423:
1421:
1419:
3959:
3001:
2999:
2997:
2117:. Document ID: ab71498b6b1a60ff817b29d56997a418.
2031:
1893:"Comparing RAID 10 and RAID 01 | SMB IT Journal"
140:Conference, they argued that the top-performing
2876:
2154:"ZFS Raidz Performance, Capacity and Integrity"
2074:"RAID management support coming in OpenBSD 3.8"
1803:Vijayan, S.; Selvamani, S.; Vijayan, S (1995).
1725:
1723:
1721:
970:Increasing rebuild time and failure probability
594:RAID controllers, users are required to enable
393:Increasing rebuild time and failure probability
321:that accesses the data first (depending on its
1595:"Common RAID Disk Drive Format (DDF) standard"
1416:
610:in 2005 with its bio(4) pseudo-device and the
337:consists of bit-level striping with dedicated
3873:
3651:
3407:"write hole: which RAID levels are affected?"
2994:
2662:"Using Windows XP to Make RAID 5 Happen"
2539:"FreeBSD Kernel Interfaces Manual for CCD(4)"
2308:
2088:
851:
2918:
2916:
2914:
2886:Wiley Pathways Network Security Fundamentals
2062:
1998:
1996:
1915:
1913:
1718:
1545:"Common RAID Disk Data Format Specification"
1291:. July 1986. pp. 29, 32. Archived from
3007:"Error Recovery Control with Smartmontools"
2882:
2440:"HPE Support document - HPE Support Center"
1921:"Intro to Nested-RAID: RAID-01 and RAID-10"
1134:
1132:
1130:
76:technology that combines multiple physical
3887:
3880:
3866:
3658:
3644:
3627:(RAID 3, 4 and 5 versus RAID 10)
3538:
3443:"A journal for MD/RAID5 [LWN.net]"
3080:
2382:
1703:
1506:. Imperial College Department of Computing
1075:
914:system (even temporarily) in the process.
590:, in order to access the configuration of
583:, and contributing to reliability issues.
3358:
3131:
2937:
2911:
2691:. Building Windows 8 blog. Archived from
1993:
1910:
1659:Hennessy, John; Patterson, David (2006).
1453:
233:RAID can also provide data security with
3534:
3532:
3400:
3398:
3149:
3107:Patterson, D., Hennessy, J. (2009), 574.
2883:Krutz, Ronald L.; Conley, James (2007).
2686:
1127:
940:Unrecoverable read errors during rebuild
855:
815:introduced a RAID-like feature known as
668:A component of the file system (such as
254:
218:in a given set of data. Most use simple
3501:
3354:
3352:
3350:
2713:
2094:
1845:"Why is RAID 1+0 better than RAID 0+1?"
1796:
1777:
1775:
1756:"Why RAID 6 stops working in 2019"
1646:Structured Computer Organization 6th ed
634:. Software RAID can be implemented as:
408:
277:Storage Networking Industry Association
14:
3960:
3495:
3194:
3155:
1957:. osdl.org. 2010-08-20. Archived from
1729:
1697:
1484:
1195:
1193:
926:
469:
27:Data storage virtualization technology
3861:
3639:
3529:
3395:
3220:
2943:
2068:
2025:
1982:. Hadoopblog.blogspot.com. 2009-08-28
1748:
1704:White, Jay; Lueth, Chris (May 2010).
1643:
1232:
1175:
1173:
1110:Redundant array of independent memory
894:(referred to in some environments as
3631:A Clean-Slate Look at Disk Scrubbing
3347:
3340:Patterson, D., Hennessy, J. (2009).
3031:
2687:Sinofsky, Steven (January 5, 2012).
2478:"Apple Mac OS X Server File Systems"
1781:
1772:
1282:"HSC50/70 Hardware Technical Manual"
1199:
934:exponential statistical distribution
70:redundant array of independent disks
66:redundant array of inexpensive disks
18:Redundant array of independent disks
3625:BAARF: Battle Against Any Raid Five
3541:"bcachefs: Principles of Operation"
3373:
3158:"Does RAID 6 stop working in 2019?"
2833:. O'Reilly Media, Inc. p. 90.
1190:
1060:Avoiding overwriting used stripes.
24:
1764:. 22 February 2010. Archived from
1233:Hayes, Frank (November 17, 2003).
1170:
904:TLER (time limited error recovery)
520:
244:
134:University of California, Berkeley
25:
3979:
3601:
3404:
3195:Miller, Scott Alan (2016-01-05).
2857:
2791:. Ata.wiki.kernel.org. 2011-04-08
2716:"NetBSD 1.4 Release Announcement"
2642:from the original on 3 March 2011
2004:"3.8: "Hackers of the Lost RAID""
1923:, Linux Magazine, January 6, 2011
1543:Dawkins, Bill and Jones, Arnold.
625:
536:
434:and RAID 50, respectively).
3539:Overstreet, Kent (18 Dec 2021).
3342:Computer Organization and Design
3041:IEEE Transactions on Reliability
2947:PostgreSQL 9.0: High Performance
1499:
1025:are preferred) for this purpose.
120:The term "RAID" was invented by
46:
3580:
3557:
3477:
3453:
3435:
3417:
3334:
3286:
3245:
3221:Kagel, Art S. (March 2, 2011).
3214:
3188:
3175:
3110:
3101:
3074:
3025:
2982:
2972:
2851:
2802:
2781:
2757:
2733:
2707:
2654:
2624:
2595:
2571:
2551:
2531:
2510:
2490:
2470:
2450:
2431:
2406:
2383:Trautman, Philip; Mostek, Jim.
2376:
2355:
2334:
2302:
2278:
2254:
2233:"20.2. The Z File System (ZFS)"
2225:
2200:
2171:
2146:
2121:
1972:
1947:
1926:
1885:
1861:
1837:
1677:
1652:
1637:
1612:
1587:
1567:"Adaptec Hybrid RAID Solutions"
1559:
1537:
1493:
1489:(2nd ed.). Glasgow: SYBEX.
1478:
1387:
1200:Katz, Randy H. (October 2010).
3489:The Linux Kernel documentation
3359:Leventhal, Adam (2009-12-01).
3118:"The RAID Migration Adventure"
1351:
1328:
1309:
1274:
1255:
1226:
963:
877:Intel Rapid Storage Technology
419:In what was originally termed
13:
1:
2860:"19.5. Software RAID Devices"
2714:Metzger, Perry (1999-05-12).
2309:Deenadhayalan, Veera (2011).
1732:"RAID's Days May Be Numbered"
1347:The Connection Machine (1988)
1121:
921:
565:network interface controllers
489:even with only two drives in
228:Reed–Solomon error correction
166:In 1977, Norman Ken Ouchi at
3849:Non-RAID drive architectures
3502:Bonwick, Jeff (2005-11-17).
3156:Harris, Robin (2010-02-27).
2609:. 2007-05-29. Archived from
2438:Hewlett Packard Enterprise.
2185:. 2014-09-15. Archived from
1730:Newman, Henry (2009-09-17).
1105:Non-RAID drive architectures
994:
886:
7:
2741:"OpenBSD softraid man page"
2579:"mdadm(8) – Linux man page"
1366:bitsavers.trailing-edge.com
1087:
205:
10:
3984:
2342:"Btrfs Wiki: Feature List"
1782:Lowe, Scott (2009-11-16).
867:
852:Firmware- and driver-based
540:
473:
412:
345:such that each sequential
248:
241:calls this "hybrid RAID".
115:
29:
3895:
3846:
3813:
3755:
3727:
3689:
3673:
3614:The Mathematics of RAID-6
2559:"The Software-RAID HowTo"
1334:
1315:
1261:
977:mean time between failure
870:MD RAID external metadata
596:Linux compatibility layer
32:RAID (French Police unit)
3937:Software-defined storage
3910:Distributed file systems
3665:
2414:"Linux RAID Setup – XFS"
1955:"Main Page – Linux-raid"
1099:Network-attached storage
842:second-stage boot loader
476:Non-standard RAID levels
288:consists of block-level
3310:10.1145/1254882.1254917
3262:10.1145/1217935.1217957
2944:Smith, Gregory (2010).
2722:. The NetBSD Foundation
2363:"Btrfs Wiki: Changelog"
2129:"RAID over File System"
1211:. IEEE Computer Society
1076:Write-cache reliability
838:first-stage boot loader
653:A more generic logical
650:and OpenBSD's softraid)
3889:Storage virtualization
3095:10.1002/qre.4680110505
3032:Gray, Jim (Oct 1990).
3009:. 2009. Archived from
2107:University of Waterloo
1736:EnterpriseStorageForum
1221:arrays of small disks.
865:
260:
74:storage virtualization
34:. For other uses, see
3709:Disk array controller
3514:Blogs. Archived from
3013:on September 28, 2011
2891:John Wiley & Sons
2638:. 14 September 2011.
2008:OpenBSD Release Songs
1644:Tanenbaum, Andrew S.
1464:10.1145/176979.176981
1442:ACM Computing Surveys
1336:US patent 4899342
1317:US patent 4761785
1263:US patent 4092732
1049:Dynamic stripe size.
859:
463:just a bunch of disks
258:
36:Raid (disambiguation)
3485:"Partial Parity Log"
3304:. pp. 289–300.
3256:. pp. 221–234.
3047:(4). IEEE: 409–418.
2055:FreeBSD Manual Pages
1897:www.smbitjournal.com
1873:www.thegeekstuff.com
1159:. SIGMOD Conferences
848:from such an array.
789:Logical Disk Manager
409:Nested (hybrid) RAID
251:Standard RAID levels
3947:Virtual file system
3922:File virtualization
3900:Block-level storage
3508:Jeff Bonwick's Blog
3233:on November 3, 2014
3185:, Dec. 13–16, 2005.
2042:BSD Cross Reference
1919:Jeffrey B. Layton:
1768:on August 15, 2010.
1500:Howe, Denis (ed.).
1485:Donald, L. (2003).
1372:. 1960. p. 157
1009:Write-ahead logging
927:Correlated failures
813:Windows Server 2012
495:Linux software RAID
470:Non-standard levels
269:non-standard levels
3569:2008-06-11 at the
2765:"FreeBSD Handbook"
2668:. 19 November 2004
2290:Oracle Corporation
2266:Oracle Corporation
2212:Oracle Corporation
1550:2009-08-24 at the
1235:"The Story So Far"
866:
791:, introduced with
415:Nested RAID levels
327:rotational latency
261:
235:solid-state drives
185:Thinking Machines'
3955:
3954:
3855:
3854:
3405:Danti, Gionatan.
2961:978-1-84951-031-8
2954:Ltd. p. 31.
2900:978-0-470-10192-6
2840:978-0-7356-5670-3
2389:linux-xfs.sgi.com
1826:978-0-8493-2615-8
1620:"SNIA Dictionary"
1185:978-1-58705-065-7
785:Microsoft Windows
711:umbrella project.
632:operating systems
622:in 2007 as well.
527:computer hardware
352:Thinking Machines
183:Around 1988, the
146:personal computer
16:(Redirected from
3975:
3905:Disk aggregation
3882:
3875:
3868:
3859:
3858:
3660:
3653:
3646:
3637:
3636:
3596:
3595:
3584:
3578:
3573:(Invited Paper)
3561:
3555:
3554:
3552:
3550:
3545:
3536:
3527:
3526:
3524:
3523:
3499:
3493:
3492:
3481:
3475:
3465:
3464:
3457:
3451:
3450:
3439:
3433:
3432:
3421:
3415:
3414:
3402:
3393:
3392:
3390:
3388:
3377:
3371:
3370:
3368:
3367:
3356:
3345:
3338:
3332:
3331:
3299:
3290:
3284:
3283:
3249:
3243:
3242:
3240:
3238:
3229:. Archived from
3218:
3212:
3211:
3209:
3208:
3192:
3186:
3179:
3173:
3172:
3170:
3169:
3153:
3147:
3141:Bianca Schroeder
3135:
3129:
3128:
3126:
3125:
3114:
3108:
3105:
3099:
3098:
3078:
3072:
3071:
3069:
3063:. Archived from
3053:10.1109/24.58719
3038:
3029:
3023:
3022:
3020:
3018:
3003:
2992:
2986:
2980:
2976:
2970:
2969:
2952:Packt Publishing
2941:
2935:
2934:
2928:
2920:
2909:
2908:
2880:
2874:
2873:
2871:
2870:
2855:
2849:
2848:
2824:
2818:
2817:
2806:
2800:
2799:
2797:
2796:
2785:
2779:
2778:
2776:
2775:
2761:
2755:
2754:
2752:
2751:
2737:
2731:
2730:
2728:
2727:
2711:
2705:
2704:
2702:
2700:
2684:
2678:
2677:
2675:
2673:
2658:
2652:
2651:
2649:
2647:
2628:
2622:
2621:
2619:
2618:
2599:
2593:
2592:
2590:
2589:
2575:
2569:
2568:
2566:
2565:
2555:
2549:
2548:
2546:
2545:
2535:
2529:
2528:
2526:
2525:
2514:
2508:
2507:
2505:
2504:
2494:
2488:
2487:
2485:
2484:
2474:
2468:
2467:
2465:
2464:
2454:
2448:
2447:
2435:
2429:
2428:
2426:
2425:
2410:
2404:
2403:
2401:
2400:
2391:. Archived from
2380:
2374:
2373:
2371:
2370:
2359:
2353:
2352:
2350:
2349:
2338:
2332:
2331:
2329:
2328:
2315:
2306:
2300:
2299:
2297:
2296:
2282:
2276:
2275:
2273:
2272:
2258:
2252:
2251:
2249:
2248:
2239:. Archived from
2229:
2223:
2222:
2220:
2219:
2204:
2198:
2197:
2195:
2194:
2175:
2169:
2168:
2166:
2164:
2150:
2144:
2143:
2141:
2140:
2131:. Archived from
2125:
2119:
2118:
2092:
2086:
2085:
2080:(Mailing list).
2066:
2060:
2059:
2049:
2029:
2023:
2022:
2020:
2019:
2000:
1991:
1990:
1988:
1987:
1976:
1970:
1969:
1967:
1966:
1951:
1945:
1944:
1942:
1941:
1930:
1924:
1917:
1908:
1907:
1905:
1904:
1889:
1883:
1882:
1880:
1879:
1865:
1859:
1858:
1856:
1855:
1841:
1835:
1834:
1815:. pp. I–146
1800:
1794:
1793:
1791:
1790:
1779:
1770:
1769:
1752:
1746:
1745:
1743:
1742:
1727:
1716:
1715:
1713:
1712:
1701:
1695:
1694:
1692:
1691:
1681:
1675:
1674:
1656:
1650:
1649:
1641:
1635:
1634:
1632:
1631:
1616:
1610:
1609:
1607:
1606:
1591:
1585:
1584:
1582:
1581:
1571:
1563:
1557:
1541:
1535:
1534:
1528:
1524:
1522:
1514:
1512:
1511:
1497:
1491:
1490:
1482:
1476:
1475:
1457:
1437:
1414:
1413:
1407:
1406:
1391:
1385:
1384:
1378:
1377:
1363:
1355:
1349:
1344:
1343:
1339:
1332:
1326:
1325:
1324:
1320:
1313:
1307:
1306:
1304:
1303:
1297:
1286:
1278:
1272:
1271:
1270:
1266:
1259:
1253:
1252:
1247:
1245:
1230:
1224:
1223:
1217:
1216:
1206:
1197:
1188:
1177:
1168:
1167:
1165:
1164:
1158:
1144:Gibson, Garth A.
1140:Patterson, David
1136:
1094:Disk data format
1082:write-back cache
1042:Partial parity.
771:modules and ccd.
719:declustered RAID
586:For example, in
557:operating system
513:Declustered RAID
492:
486:Linux MD RAID 10
300:. Compared to a
63:
62:
59:
58:
55:
52:
21:
3983:
3982:
3978:
3977:
3976:
3974:
3973:
3972:
3958:
3957:
3956:
3951:
3891:
3886:
3856:
3851:
3842:
3809:
3773:Data redundancy
3768:Fault tolerance
3751:
3723:
3685:
3681:of independent
3669:
3664:
3604:
3599:
3586:
3585:
3581:
3571:Wayback Machine
3562:
3558:
3548:
3546:
3543:
3537:
3530:
3521:
3519:
3500:
3496:
3483:
3482:
3478:
3474:– Special Files
3460:
3459:
3458:
3454:
3441:
3440:
3436:
3423:
3422:
3418:
3403:
3396:
3386:
3384:
3379:
3378:
3374:
3365:
3363:
3357:
3348:
3339:
3335:
3320:
3297:
3291:
3287:
3272:
3250:
3246:
3236:
3234:
3219:
3215:
3206:
3204:
3193:
3189:
3180:
3176:
3167:
3165:
3162:StorageMojo.com
3154:
3150:
3145:Garth A. Gibson
3136:
3132:
3123:
3121:
3116:
3115:
3111:
3106:
3102:
3079:
3075:
3067:
3036:
3030:
3026:
3016:
3014:
3005:
3004:
2995:
2987:
2983:
2977:
2973:
2962:
2942:
2938:
2926:
2922:
2921:
2912:
2901:
2893:. p. 422.
2881:
2877:
2868:
2866:
2858:Block, Warren.
2856:
2852:
2841:
2825:
2821:
2808:
2807:
2803:
2794:
2792:
2789:"SATA RAID FAQ"
2787:
2786:
2782:
2773:
2771:
2763:
2762:
2758:
2749:
2747:
2739:
2738:
2734:
2725:
2723:
2712:
2708:
2698:
2696:
2685:
2681:
2671:
2669:
2660:
2659:
2655:
2645:
2643:
2630:
2629:
2625:
2616:
2614:
2601:
2600:
2596:
2587:
2585:
2577:
2576:
2572:
2563:
2561:
2557:
2556:
2552:
2543:
2541:
2537:
2536:
2532:
2523:
2521:
2516:
2515:
2511:
2502:
2500:
2496:
2495:
2491:
2482:
2480:
2476:
2475:
2471:
2462:
2460:
2456:
2455:
2451:
2444:support.hpe.com
2436:
2432:
2423:
2421:
2412:
2411:
2407:
2398:
2396:
2381:
2377:
2368:
2366:
2361:
2360:
2356:
2347:
2345:
2340:
2339:
2335:
2326:
2324:
2313:
2307:
2303:
2294:
2292:
2284:
2283:
2279:
2270:
2268:
2260:
2259:
2255:
2246:
2244:
2231:
2230:
2226:
2217:
2215:
2206:
2205:
2201:
2192:
2190:
2177:
2176:
2172:
2162:
2160:
2152:
2151:
2147:
2138:
2136:
2127:
2126:
2122:
2093:
2089:
2067:
2063:
2051:
2030:
2026:
2017:
2015:
2002:
2001:
1994:
1985:
1983:
1978:
1977:
1973:
1964:
1962:
1953:
1952:
1948:
1939:
1937:
1932:
1931:
1927:
1918:
1911:
1902:
1900:
1891:
1890:
1886:
1877:
1875:
1867:
1866:
1862:
1853:
1851:
1843:
1842:
1838:
1827:
1801:
1797:
1788:
1786:
1780:
1773:
1754:
1753:
1749:
1740:
1738:
1728:
1719:
1710:
1708:
1702:
1698:
1689:
1687:
1683:
1682:
1678:
1671:
1663:. p. 362.
1657:
1653:
1642:
1638:
1629:
1627:
1618:
1617:
1613:
1604:
1602:
1593:
1592:
1588:
1579:
1577:
1576:. Adaptec. 2012
1569:
1565:
1564:
1560:
1552:Wayback Machine
1542:
1538:
1526:
1525:
1516:
1515:
1509:
1507:
1498:
1494:
1483:
1479:
1438:
1417:
1404:
1402:
1393:
1392:
1388:
1375:
1373:
1361:
1357:
1356:
1352:
1341:
1333:
1329:
1322:
1314:
1310:
1301:
1299:
1295:
1284:
1280:
1279:
1275:
1268:
1260:
1256:
1243:
1241:
1231:
1227:
1214:
1212:
1204:
1198:
1191:
1178:
1171:
1162:
1160:
1156:
1137:
1128:
1124:
1090:
1078:
997:
972:
942:
929:
924:
889:
872:
854:
744:Hewlett-Packard
628:
545:
543:RAID controller
539:
523:
521:Implementations
490:
478:
472:
417:
411:
253:
247:
245:Standard levels
216:fault tolerance
208:
161:NonStop Systems
122:David Patterson
118:
82:data redundancy
49:
45:
39:
28:
23:
22:
15:
12:
11:
5:
3981:
3971:
3970:
3953:
3952:
3950:
3949:
3944:
3939:
3934:
3929:
3924:
3919:
3918:
3917:
3907:
3902:
3896:
3893:
3892:
3885:
3884:
3877:
3870:
3862:
3853:
3852:
3847:
3844:
3843:
3841:
3840:
3835:
3830:
3825:
3819:
3817:
3811:
3810:
3808:
3807:
3802:
3797:
3796:
3795:
3790:
3785:
3780:
3775:
3770:
3759:
3757:
3753:
3752:
3750:
3749:
3744:
3739:
3733:
3731:
3725:
3724:
3722:
3721:
3716:
3714:Disk mirroring
3711:
3706:
3701:
3699:Data scrubbing
3695:
3693:
3687:
3686:
3674:
3671:
3670:
3663:
3662:
3655:
3648:
3640:
3634:
3633:
3628:
3621:
3620:
3618:H. Peter Anvin
3611:
3603:
3602:External links
3600:
3598:
3597:
3579:
3556:
3528:
3494:
3476:
3452:
3434:
3416:
3394:
3372:
3346:
3333:
3318:
3285:
3270:
3244:
3213:
3203:. StorageCraft
3187:
3174:
3148:
3130:
3120:. 10 July 2007
3109:
3100:
3089:(5): 341–353.
3073:
3070:on 2019-02-20.
3024:
2993:
2981:
2971:
2960:
2936:
2910:
2899:
2875:
2850:
2839:
2819:
2801:
2780:
2756:
2732:
2706:
2695:on May 9, 2013
2679:
2666:Tom's Hardware
2653:
2623:
2594:
2570:
2550:
2530:
2509:
2489:
2469:
2449:
2430:
2405:
2375:
2354:
2333:
2301:
2277:
2253:
2224:
2199:
2179:"ZFS -illumos"
2170:
2145:
2120:
2087:
2072:(2005-09-09).
2070:Raadt, Theo de
2061:
2024:
1992:
1971:
1946:
1925:
1909:
1899:. 30 July 2014
1884:
1860:
1836:
1825:
1795:
1771:
1747:
1717:
1696:
1676:
1670:978-0123704900
1669:
1651:
1636:
1611:
1586:
1558:
1536:
1492:
1477:
1455:10.1.1.41.3889
1448:(2): 145–185.
1415:
1386:
1350:
1327:
1308:
1273:
1254:
1225:
1209:eecs.umich.edu
1189:
1169:
1125:
1123:
1120:
1119:
1118:
1112:
1107:
1102:
1096:
1089:
1086:
1077:
1074:
1066:
1065:
1058:
1047:
1040:
1032:intent logging
1028:
1027:
1026:
1015:
996:
993:
971:
968:
964:Data scrubbing
941:
938:
928:
925:
923:
920:
892:Data scrubbing
888:
885:
853:
850:
833:
832:
826:
820:
817:Storage Spaces
782:
772:
762:
751:
737:
736:
729:
723:
715:Spectrum Scale
712:
687:Some advanced
685:
684:
681:
674:Spectrum Scale
666:
655:volume manager
651:
640:virtual device
627:
626:Software-based
624:
604:hot spare disk
581:vendor lock-in
541:Main article:
538:
537:Hardware-based
535:
522:
519:
518:
517:
510:
504:
498:
474:Main article:
471:
468:
467:
466:
453:
446:RAID 1+0:
443:
439:RAID 0+1:
413:Main article:
410:
407:
406:
405:
396:
383:
367:
355:
330:
313:
302:spanned volume
249:Main article:
246:
243:
207:
204:
196:
195:
181:
178:
171:
164:
117:
114:
26:
9:
6:
4:
3:
2:
3980:
3969:
3966:
3965:
3963:
3948:
3945:
3943:
3940:
3938:
3935:
3933:
3930:
3928:
3925:
3923:
3920:
3916:
3913:
3912:
3911:
3908:
3906:
3903:
3901:
3898:
3897:
3894:
3890:
3883:
3878:
3876:
3871:
3869:
3864:
3863:
3860:
3850:
3845:
3839:
3836:
3834:
3831:
3829:
3826:
3824:
3821:
3820:
3818:
3816:
3812:
3806:
3803:
3801:
3798:
3794:
3791:
3789:
3786:
3784:
3781:
3779:
3778:Degraded mode
3776:
3774:
3771:
3769:
3766:
3765:
3764:
3761:
3760:
3758:
3754:
3748:
3745:
3743:
3740:
3738:
3735:
3734:
3732:
3730:
3726:
3720:
3717:
3715:
3712:
3710:
3707:
3705:
3704:Data striping
3702:
3700:
3697:
3696:
3694:
3692:
3688:
3684:
3680:
3677:
3672:
3668:
3661:
3656:
3654:
3649:
3647:
3642:
3641:
3638:
3632:
3629:
3626:
3623:
3622:
3619:
3615:
3612:
3609:
3606:
3605:
3593:
3589:
3583:
3576:
3572:
3568:
3565:
3560:
3542:
3535:
3533:
3518:on 2014-12-16
3517:
3513:
3509:
3505:
3498:
3490:
3486:
3480:
3473:
3470:Programmer's
3469:
3466: –
3463:
3456:
3448:
3444:
3438:
3430:
3426:
3420:
3412:
3408:
3401:
3399:
3382:
3376:
3362:
3355:
3353:
3351:
3343:
3337:
3329:
3325:
3321:
3319:9781595936394
3315:
3311:
3307:
3303:
3296:
3289:
3281:
3277:
3273:
3267:
3263:
3259:
3255:
3248:
3232:
3228:
3227:miracleas.com
3224:
3217:
3202:
3201:Recovery Zone
3198:
3191:
3184:
3178:
3163:
3159:
3152:
3146:
3142:
3139:
3134:
3119:
3113:
3104:
3096:
3092:
3088:
3084:
3077:
3066:
3062:
3058:
3054:
3050:
3046:
3042:
3035:
3028:
3017:September 29,
3012:
3008:
3002:
3000:
2998:
2991:
2985:
2975:
2967:
2963:
2957:
2953:
2949:
2948:
2940:
2932:
2925:
2919:
2917:
2915:
2906:
2902:
2896:
2892:
2888:
2887:
2879:
2865:
2861:
2854:
2846:
2842:
2836:
2832:
2831:
2823:
2815:
2811:
2805:
2790:
2784:
2770:
2766:
2760:
2746:
2742:
2736:
2721:
2717:
2710:
2694:
2690:
2683:
2667:
2663:
2657:
2641:
2637:
2633:
2627:
2613:on 2007-07-03
2612:
2608:
2604:
2598:
2584:
2583:Linux.Die.net
2580:
2574:
2560:
2554:
2540:
2534:
2520:. 6 July 2006
2519:
2513:
2499:
2493:
2479:
2473:
2459:
2453:
2445:
2441:
2434:
2419:
2415:
2409:
2395:on 2015-04-22
2394:
2390:
2386:
2379:
2364:
2358:
2343:
2337:
2323:
2319:
2312:
2305:
2291:
2287:
2281:
2267:
2263:
2257:
2243:on 2014-07-03
2242:
2238:
2234:
2228:
2213:
2209:
2203:
2189:on 2019-03-15
2188:
2184:
2180:
2174:
2159:
2155:
2149:
2135:on 2013-11-09
2134:
2130:
2124:
2116:
2112:
2108:
2104:
2100:
2099:
2091:
2083:
2079:
2075:
2071:
2065:
2057:
2056:
2047:
2043:
2039:
2035:
2032:Long, Scott;
2028:
2013:
2009:
2005:
1999:
1997:
1981:
1975:
1961:on 2008-07-05
1960:
1956:
1950:
1935:
1929:
1922:
1916:
1914:
1898:
1894:
1888:
1874:
1870:
1864:
1850:
1846:
1840:
1832:
1828:
1822:
1818:
1814:
1810:
1806:
1799:
1785:
1778:
1776:
1767:
1763:
1762:
1757:
1751:
1737:
1733:
1726:
1724:
1722:
1707:
1700:
1686:
1680:
1672:
1666:
1662:
1655:
1648:. p. 95.
1647:
1640:
1625:
1621:
1615:
1600:
1596:
1590:
1575:
1568:
1562:
1555:
1553:
1549:
1546:
1540:
1532:
1520:
1505:
1504:
1496:
1488:
1481:
1473:
1469:
1465:
1461:
1456:
1451:
1447:
1443:
1436:
1434:
1432:
1430:
1428:
1426:
1424:
1422:
1420:
1412:
1400:
1396:
1390:
1383:
1371:
1367:
1360:
1354:
1348:
1337:
1331:
1318:
1312:
1298:on 2016-03-04
1294:
1290:
1283:
1277:
1264:
1258:
1251:
1240:
1239:Computerworld
1236:
1229:
1222:
1210:
1203:
1196:
1194:
1187:; Appendix A.
1186:
1182:
1176:
1174:
1155:
1154:
1149:
1145:
1141:
1135:
1133:
1131:
1126:
1116:
1113:
1111:
1108:
1106:
1103:
1100:
1097:
1095:
1092:
1091:
1085:
1083:
1073:
1071:
1063:
1059:
1056:
1052:
1048:
1045:
1041:
1037:
1033:
1029:
1024:
1020:
1016:
1013:
1012:
1010:
1007:
1006:
1005:
1003:
992:
988:
985:
980:
978:
967:
965:
959:
956:
952:
948:
937:
935:
919:
915:
911:
909:
908:Smartmontools
905:
899:
897:
893:
884:
880:
878:
871:
863:
862:SATA 3.0
858:
849:
847:
843:
839:
830:
827:
824:
821:
818:
814:
810:
806:
802:
798:
797:dynamic disks
794:
790:
786:
783:
780:
776:
773:
770:
766:
763:
760:
756:
752:
749:
745:
742:
741:
740:
733:
730:
727:
724:
720:
716:
713:
710:
706:
702:
697:
694:
693:
692:
690:
682:
679:
675:
671:
667:
664:
660:
656:
652:
649:
645:
641:
637:
636:
635:
633:
623:
621:
617:
613:
609:
605:
599:
597:
593:
589:
584:
582:
578:
574:
570:
566:
562:
558:
554:
550:
544:
534:
532:
528:
514:
511:
508:
505:
502:
499:
496:
487:
484:
483:
482:
477:
464:
460:
458:
454:
451:
447:
444:
440:
437:
436:
435:
433:
428:
426:
422:
416:
402:
401:
397:
394:
389:
388:
384:
381:
377:
373:
372:
368:
365:
361:
360:
356:
353:
348:
344:
340:
336:
335:
331:
328:
324:
319:
318:
314:
311:
307:
303:
299:
295:
291:
287:
286:
282:
281:
280:
278:
274:
270:
266:
265:nested levels
257:
252:
242:
240:
236:
231:
229:
225:
221:
217:
213:
203:
201:
193:
189:
186:
182:
179:
176:
173:Around 1983,
172:
169:
165:
162:
159:
155:
154:
153:
149:
147:
143:
139:
135:
131:
127:
123:
113:
111:
107:
103:
99:
95:
90:
85:
83:
79:
75:
72:") is a data
71:
67:
61:
43:
37:
33:
19:
3942:Virtual disk
3931:
3927:Logical disk
3763:Availability
3747:Non-standard
3719:Parity drive
3666:
3592:www.snia.org
3591:
3582:
3559:
3547:. Retrieved
3520:. Retrieved
3516:the original
3507:
3497:
3488:
3479:
3455:
3446:
3437:
3428:
3419:
3411:Server Fault
3410:
3385:. Retrieved
3375:
3364:. Retrieved
3341:
3336:
3301:
3288:
3253:
3247:
3235:. Retrieved
3231:the original
3226:
3216:
3205:. Retrieved
3200:
3190:
3177:
3166:. Retrieved
3164:. TechnoQWAN
3161:
3151:
3133:
3122:. Retrieved
3112:
3103:
3086:
3082:
3076:
3065:the original
3044:
3040:
3027:
3015:. Retrieved
3011:the original
2984:
2974:
2966:Google Books
2964:– via
2946:
2939:
2930:
2905:Google Books
2903:– via
2885:
2878:
2867:. Retrieved
2863:
2853:
2845:Google Books
2843:– via
2829:
2822:
2813:
2804:
2793:. Retrieved
2783:
2772:. Retrieved
2768:
2759:
2748:. Retrieved
2744:
2735:
2724:. Retrieved
2719:
2709:
2697:. Retrieved
2693:the original
2682:
2670:. Retrieved
2656:
2644:. Retrieved
2626:
2615:. Retrieved
2611:the original
2606:
2597:
2586:. Retrieved
2582:
2573:
2562:. Retrieved
2553:
2542:. Retrieved
2533:
2522:. Retrieved
2512:
2501:. Retrieved
2492:
2481:. Retrieved
2472:
2461:. Retrieved
2452:
2443:
2433:
2422:. Retrieved
2420:. 2013-10-05
2417:
2408:
2397:. Retrieved
2393:the original
2388:
2378:
2367:. Retrieved
2365:. 2012-10-01
2357:
2346:. Retrieved
2344:. 2012-11-07
2336:
2325:. Retrieved
2317:
2304:
2293:. Retrieved
2280:
2269:. Retrieved
2256:
2245:. Retrieved
2241:the original
2236:
2227:
2216:. Retrieved
2214:. 2012-04-01
2202:
2191:. Retrieved
2187:the original
2173:
2161:. Retrieved
2157:
2148:
2137:. Retrieved
2133:the original
2123:
2097:
2090:
2077:
2064:
2054:
2041:
2034:Adaptec, Inc
2027:
2016:. Retrieved
2014:. 2005-11-01
2007:
1984:. Retrieved
1974:
1963:. Retrieved
1959:the original
1949:
1938:. Retrieved
1928:
1901:. Retrieved
1896:
1887:
1876:. Retrieved
1872:
1863:
1852:. Retrieved
1848:
1839:
1831:Google Books
1829:– via
1816:
1808:
1798:
1787:. Retrieved
1766:the original
1759:
1750:
1739:. Retrieved
1735:
1709:. Retrieved
1699:
1688:. Retrieved
1679:
1660:
1654:
1645:
1639:
1628:. Retrieved
1623:
1614:
1603:. Retrieved
1598:
1589:
1578:. Retrieved
1573:
1561:
1554:
1539:
1508:. Retrieved
1502:
1495:
1486:
1480:
1445:
1441:
1409:
1403:. Retrieved
1401:. 2009-06-18
1399:brouhaha.com
1398:
1389:
1380:
1374:. Retrieved
1365:
1353:
1330:
1311:
1300:. Retrieved
1293:the original
1276:
1257:
1249:
1244:November 18,
1242:. Retrieved
1238:
1228:
1219:
1213:. Retrieved
1208:
1161:. Retrieved
1152:
1117:(S.M.A.R.T.)
1079:
1067:
1001:
998:
989:
981:
973:
960:
943:
930:
916:
912:
900:
895:
890:
881:
873:
834:
793:Windows 2000
759:macOS Server
738:
722:RAID 1.
689:file systems
686:
644:Linux kernel
629:
616:drive sensor
600:
585:
546:
524:
479:
462:
455:
450:RAID 10
445:
438:
432:RAID 10
429:
424:
420:
418:
398:
385:
369:
357:
339:Hamming-code
332:
315:
310:concurrently
283:
262:
232:
224:Galois field
209:
199:
197:
192:IBM 353
150:
126:Garth Gibson
119:
98:availability
86:
69:
65:
41:
40:
3800:Scalability
3793:Replication
3729:RAID levels
3691:Disk arrays
3387:15 February
3237:October 30,
2931:adaptec.com
2864:freebsd.org
2745:OpenBSD.org
2646:17 December
2237:freebsd.org
2183:illumos.org
2158:calomel.org
2109:: UWSpace.
1980:"Hdfs Raid"
1574:Adaptec.com
1527:|work=
1148:Katz, Randy
896:patrol read
561:proprietary
461:With JBOD (
421:hybrid RAID
400:RAID 6
387:RAID 5
371:RAID 4
359:RAID 3
334:RAID 2
317:RAID 1
285:RAID 0
273:proprietary
200:independent
102:performance
94:reliability
3915:Comparison
3838:Oracle ZFS
3815:Interfaces
3805:Throughput
3788:Parity bit
3756:Principles
3522:2015-02-01
3383:. ZAR team
3366:2012-11-30
3271:1595933220
3207:2016-07-22
3168:2013-12-17
3124:2010-03-10
2869:2014-07-27
2814:redhat.com
2795:2012-08-26
2774:2009-03-19
2750:2018-02-03
2726:2013-01-30
2720:NetBSD.org
2699:January 6,
2617:2007-10-08
2588:2014-11-20
2564:2008-11-10
2544:2009-03-19
2524:2009-03-19
2503:2009-03-19
2483:2008-04-23
2463:2010-01-04
2424:2015-08-17
2418:kernel.org
2399:2015-08-17
2369:2012-11-14
2348:2012-11-16
2327:2014-09-28
2318:UseNix.org
2295:2014-07-27
2271:2014-07-27
2247:2014-07-27
2218:2014-07-27
2193:2016-05-23
2139:2014-07-22
2115:10012/5234
2058:. FreeBSD.
2018:2019-03-23
1986:2010-08-24
1965:2010-08-24
1940:2013-12-25
1936:. tldp.org
1903:2016-05-23
1878:2016-05-23
1854:2016-05-23
1789:2012-12-01
1741:2010-09-07
1711:2013-03-02
1690:2012-12-20
1630:2010-08-24
1605:2012-08-26
1580:2013-09-07
1510:2011-11-10
1405:2015-01-17
1376:2015-01-17
1302:2014-01-03
1215:2015-01-18
1163:2024-01-03
1122:References
1002:write hole
922:Weaknesses
868:See also:
805:Windows XP
555:before an
553:Option ROM
306:throughput
130:Randy Katz
89:redundancy
78:disk drive
3676:Redundant
3577:: 144–154
3575:VLDB 1981
2672:24 August
2636:Microsoft
2607:Microsoft
2105:thesis).
1813:CRC Press
1529:ignored (
1519:cite book
1472:207178693
1450:CiteSeerX
1345:See also
995:Atomicity
887:Integrity
809:Windows 8
801:Windows 8
642:(such as
459:RAID N+N:
323:seek time
294:mirroring
292:, but no
267:and many
188:DataVault
142:mainframe
3962:Category
3783:Failover
3737:Standard
3567:Archived
3504:"RAID-Z"
3328:14164251
2640:Archived
2036:(2000).
1849:aput.net
1624:SNIA.org
1599:SNIA.org
1548:Archived
1150:(1988).
1088:See also
1070:Jim Gray
1062:bcachefs
753:Apple's
735:manager.
573:ifconfig
569:Ethernet
531:software
290:striping
271:(mostly
206:Overview
202:disks".
106:capacity
3447:lwn.net
3429:lwn.net
3280:7655425
3061:2955525
2163:26 June
2082:OpenBSD
2046:FreeBSD
2012:OpenBSD
829:OpenBSD
765:FreeBSD
748:OpenVMS
709:OpenZFS
705:illumos
701:Solaris
659:Veritas
608:OpenBSD
592:Adaptec
588:FreeBSD
380:RAID-DP
343:striped
239:Adaptec
132:at the
116:History
3823:bioctl
3742:Nested
3549:10 May
3512:Oracle
3472:Manual
3326:
3316:
3278:
3268:
3059:
2958:
2897:
2837:
1823:
1667:
1626:. SNIA
1601:. SNIA
1470:
1452:
1411:second
1342:
1323:
1269:
1183:
1051:RAID-Z
1030:Write
984:NetApp
846:kernel
823:NetBSD
620:NetBSD
612:bioctl
529:or by
507:BeeGFS
501:Hadoop
448:(see:
376:NetApp
298:parity
212:parity
158:Tandem
138:SIGMOD
128:, and
110:sector
104:, and
68:" or "
3833:mdadm
3683:disks
3679:array
3616:, by
3544:(PDF)
3468:Linux
3462:md(4)
3324:S2CID
3298:(PDF)
3276:S2CID
3068:(PDF)
3057:S2CID
3037:(PDF)
2927:(PDF)
2314:(PDF)
2103:MMath
2078:misc@
1761:ZDNet
1570:(PDF)
1468:S2CID
1362:(PDF)
1296:(PDF)
1285:(PDF)
1205:(PDF)
1157:(PDF)
1101:(NAS)
1044:mdadm
1036:mdadm
775:Linux
755:macOS
726:Btrfs
678:Btrfs
3968:RAID
3932:RAID
3828:geom
3667:RAID
3551:2023
3389:2012
3314:ISBN
3266:ISBN
3239:2014
3143:and
3019:2017
2979:p.39
2956:ISBN
2895:ISBN
2835:ISBN
2701:2012
2674:2010
2648:2009
2165:2017
1821:ISBN
1665:ISBN
1531:help
1246:2016
1181:ISBN
1039:WAL.
1023:NVMs
1021:and
1019:SSDs
947:SCSI
811:and
769:GEOM
757:and
703:and
577:Unix
567:for
549:BIOS
457:JBOD
425:RAID
364:byte
325:and
42:RAID
3306:doi
3258:doi
3091:doi
3049:doi
2322:IBM
2111:hdl
2050:,
1460:doi
1370:IBM
1289:DEC
1055:COW
955:SAS
777:'s
746:'s
732:XFS
696:ZFS
676:or
670:ZFS
663:LVM
661:or
646:'s
575:in
551:or
347:bit
296:or
226:or
220:XOR
175:DEC
168:IBM
64:; "
3964::
3590:.
3531:^
3510:.
3506:.
3487:.
3445:.
3427:.
3409:.
3397:^
3349:^
3322:.
3312:.
3300:.
3274:.
3264:.
3225:.
3199:.
3160:.
3087:11
3085:.
3055:.
3045:39
3043:.
3039:.
2996:^
2950:.
2929:.
2913:^
2889:.
2862:.
2812:.
2767:.
2743:.
2718:.
2664:.
2634:.
2605:.
2581:.
2442:.
2416:.
2387:.
2320:.
2316:.
2288:.
2264:.
2235:.
2210:.
2181:.
2156:.
2076:.
2044:.
2040:.
2010:.
2006:.
1995:^
1912:^
1895:.
1871:.
1847:.
1819:.
1817:ff
1811:.
1807:.
1774:^
1758:.
1734:.
1720:^
1622:.
1597:.
1572:.
1523::
1521:}}
1517:{{
1466:.
1458:.
1446:26
1444:.
1418:^
1408:.
1397:.
1379:.
1368:.
1364:.
1287:.
1248:.
1237:.
1218:.
1207:.
1192:^
1172:^
1146:;
1142:;
1129:^
1034:.
1011:.
953:,
951:FC
949:,
860:A
803:.
779:md
672:,
648:md
491:f2
230:.
124:,
100:,
96:,
54:eɪ
3881:e
3874:t
3867:v
3659:e
3652:t
3645:v
3594:.
3553:.
3525:.
3491:.
3449:.
3431:.
3413:.
3391:.
3369:.
3330:.
3308::
3282:.
3260::
3241:.
3210:.
3171:.
3127:.
3097:.
3093::
3051::
3021:.
2968:.
2933:.
2907:.
2872:.
2847:.
2816:.
2798:.
2777:.
2753:.
2729:.
2703:.
2676:.
2650:.
2620:.
2591:.
2567:.
2547:.
2527:.
2506:.
2486:.
2466:.
2446:.
2427:.
2402:.
2372:.
2351:.
2330:.
2298:.
2274:.
2250:.
2221:.
2196:.
2167:.
2142:.
2113::
2101:(
2084:.
2048:.
2021:.
1989:.
1968:.
1943:.
1906:.
1881:.
1857:.
1833:.
1792:.
1744:.
1714:.
1693:.
1673:.
1633:.
1608:.
1583:.
1533:)
1513:.
1474:.
1462::
1305:.
1166:.
945:(
680:)
665:)
194:.
163:.
60:/
57:d
51:r
48:/
44:(
38:.
20:)
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.