A comprehensive guide to preparing for and executing a RAID resilver — covering ZFS, hardware RAID, and mdadm — so you can replace a failed drive safely and with minimal downtime.
~18 min read
Intermediate-Advanced
ZFS · Hardware RAID · mdadm
What Is Resilvering?
Resilvering is the process of reconstructing data onto a replacement drive after a drive failure in a RAID array. When a disk fails in a redundant configuration (RAID 1, 5, 6, 10, or any ZFS vdev with redundancy), the array enters a degraded state. Until the failed drive is replaced, your array is operating with reduced or — in the case of RAID 0 or a stripe — zero fault tolerance.
Resilvering reads data from the surviving disks, computes the missing data via parity or mirroring, and writes it to the new drive. The term originated in the Solaris/SPARC storage world and is most commonly used with ZFS, but the concept applies across all RAID implementations. In mdadm it is called a recovery or resync; in hardware RAID it is called a rebuild. For this guide we use "resilver" generically.
Your array is at-risk during a resilver
While a resilver is running, the array is operating without one level of redundancy. If a second drive fails during this window, you will experience data loss. This is called the URE risk window — the longer the resilver takes, the larger the chance of encountering an uncorrectable read error on another drive.
Understanding how long a resilver will take is critical for planning maintenance windows. See our RAID Rebuild Time Guide for detailed estimates based on drive size, RAID level, and hardware.
Pre-Resilver Checklist
Before you pull any drive, work through this checklist. Skipping any of these steps can turn a routine drive replacement into a data recovery emergency.
Verify the Failing Drive — Don't Pull the Wrong One
This is the #1 mistake. Double-triple-quadruple check that you are replacing the correct drive. Use the disk serial number, not the device name (which can shuffle after a reboot). On Linux:
# Check serials for every disk
lsblk -o NAME,SERIAL,SIZE,MODEL,MOUNTPOINT
# Or with smartctl
smartctl -i /dev/sdX | grep "Serial Number"
# For ZFS
zpool status -v -L
Use the -L flag with ZFS to show drive links by physical location rather than logical device names. Physically label your drives with their serial numbers before installation.
Confirm the Correct Replacement Is Ready
The replacement drive must be:
At least as large as the failed drive (same size or larger — see FAQ for "slightly larger")
Same or faster interface — replacing SATA with SAS won't work on most consumer controllers
Same rotational type — mixing 5400 RPM and 7200 RPM in the same vdev/RAID group causes performance penalties
Matching sector size — 512n vs 512e vs 4Kn mismatches can cause performance issues or incompatibility
If replacing with a larger drive, see the FAQ on what to expect.
Check That Backups Exist (and Are Recent)
RAID is not a backup. A resilver is a stress event — bugs in the RAID implementation, a flaky controller, or a URE on a surviving disk can corrupt data or cause an unrecoverable failure. Before you start:
Verify your last backup is recent and restorable
Take a fresh backup if possible
For critical data, consider a zfs send / receive or filesystem-level snapshot before proceeding
Document the current pool/array configuration so you can rebuild from scratch if needed
Check Drive Health and System Cooling
A resilver can run for hours or days, pushing every drive and controller to its thermal limit:
Check all drive temperatures — smartctl -A /dev/sdX | grep Temperature
Ensure chassis fans and cooling are working correctly
Clean dust filters and vents
Check the RAID/HBA controller temperature if reported
Consider scheduling the resilver for a cooler time of day if the server room has ambient temperature swings
Preparing the Replacement Drive
Never install a brand-new drive directly into a RAID array without preparation. New drives can have infant mortality — early-life failures that manifest in the first 48 hours of use. A few hours of burn-in testing can save you from a failed resilver and data loss.
1. Burn-In Testing (badblocks)
Run a destructive write pattern test to verify every sector on the drive:
# Destructive write test (4 passes: 0xaa, 0x55, 0xff, 0x00)
# WARNING: Destroys all data on /dev/sdX!
sudo badblocks -wsv -b 4096 /dev/sdX
# For drives > 4 TB, consider btrfs or f3 instead
# (badblocks is slow on huge HDDs)
For drives larger than 4 TB, badblocks can take days. An alternative is to run a SMART extended offline self-test, which is faster though less thorough.
2. SMART Baseline
Capture a SMART baseline for the replacement drive before it enters the array:
# Full SMART info
sudo smartctl -a /dev/sdX > /root/pre-install-smart-$(date +%F).txt
# Run a short self-test
sudo smartctl -t short /dev/sdX
# Run an extended (long) self-test
sudo smartctl -t long /dev/sdX
# Check test results
sudo smartctl -l selftest /dev/sdX
Record the Reallocated_Sector_Ct, Current_Pending_Sector, and Raw_Read_Error_Rate before installation.
3. Firmware Check
Check the drive manufacturer's website for firmware updates. Known firmware bugs have caused widespread drive failures in RAID environments:
Seagate: check for the latest firmware on their support site
WD/HGST: verify firmware via their dashboard tool
SSDs: NVMe firmware updates can fix critical reliability issues
Apply firmware before installing the drive — it's much harder to update a drive once it's part of a live array.
4. Format / Prep
Once testing is complete, prepare the drive for the array:
Partition the drive if your RAID implementation expects partitions (e.g., mdadm + GRUB)
For ZFS, use the whole disk (sudo zpool replace ... /dev/sdX) or a partition
Ensure 4K alignment: use parted with alignment to optimal I/O
Wipe any existing filesystem signatures to avoid confusion: sudo wipefs -a /dev/sdX
Parallel burn-in for multiple drives
If you're replacing several drives, burn them in in parallel on separate SATA/SAS ports. A single burn-in test on a 20 TB drive can take 48+ hours — running 4 in parallel takes the same wall time. Just monitor temperatures closely to avoid overheating during testing.
ZFS-Specific Steps for Resilvering
ZFS has several unique behaviors and tunables that affect how a resilver proceeds. Understanding these will help you avoid common pitfalls.
Verify Replacement Drive Size
ZFS requires the replacement drive to be at least as large as the drive it is replacing. It checks the available sector count, not the advertised capacity. A drive sold as "4 TB" may have slightly fewer usable sectors than another "4 TB" drive from a different manufacturer. Always verify:
# Check available sectors on the replacement
sudo blockdev --getsize64 /dev/sdX
# Compare with the failed drive's output
sudo blockdev --getsize64 /dev/sdY
# Or using smartctl
sudo smartctl -i /dev/sdX | grep "Total"
Check Pool Health Before Starting
Before you initiate the replacement, verify that the pool is otherwise healthy. A pool with existing checksum errors, a degraded second vdev, or a pending device issue will complicate the resilver:
# Check overall pool status
sudo zpool status -v
# Check for checksum errors across all vdevs
sudo zpool status -v | grep -E "(cksum|errors)"
# Check recent scrub data
sudo zpool status -v | grep -A 5 "scan:"
Verify the pool's last scrub result
If the last scrub found errors that have not been resolved, fix those before starting a resilver. Resilvering onto a replacement drive will faithfully copy any existing corruption.
The zfs_resilver_disable_defer Tunable
ZFS can defer resilvers for devices that are temporarily offline but expected to return (e.g., a drive that disconnected due to a loose cable). When you issue a zpool replace, however, you want the resilver to start immediately. Check the current setting:
# Check current setting
cat /sys/module/zfs/parameters/zfs_resilver_disable_defer
# To disable deferral (resilver starts immediately):
echo 0 | sudo tee /sys/module/zfs/parameters/zfs_resilver_disable_defer
# Or set via sysctl:
sudo sysctl -w zfs_resilver_disable_defer=0
Setting this to 0 ensures that the resilver begins as soon as the replacement drive is attached, without waiting for a deferred-resilver timeout.
Initiating the Replace
# Online replace (if the failed drive is still visible)
sudo zpool replace pool_name OLD_DEVICE NEW_DEVICE
# If the drive has been physically removed:
sudo zpool replace pool_name OLD_GUID NEW_DEVICE
# Find OLD_GUID from zpool status output
# Check resilver progress
sudo zpool status pool_name
Use ashift alignment on the replacement
When replacing in ZFS, the ashift property of the vdev is already set. However, if you're replacing spinning rust (512e, ashift=12) with an SSD (4Kn, optimally ashift=13), you may want to recreate the vdev or accept the sub-optimal alignment. The ashift cannot be changed on an existing vdev.
Hardware RAID-Specific Steps
Hardware RAID controllers (LSI/Avago/Broadcom, Adaptec, Areca, and motherboard-integrated RAID) manage the rebuild process internally. While this shields you from some low-level details, there are still critical steps to follow.
1. Check Controller Firmware
RAID controller firmware is the single most impactful component in a hardware RAID rebuild. Buggy firmware can cause rebuilds to hang, corrupt data, or fail silently. Before any rebuild:
Note the current firmware version — storcli64 /c0 show (Broadcom) or vendor tool
Check the vendor's site for known firmware issues related to rebuilds
Apply firmware updates if the release notes mention rebuild reliability fixes
Reboot after firmware update before starting the rebuild
2. Verify Hot Spare Status
If you have a global or dedicated hot spare assigned:
Verify the hot spare is in Ready or Online Spare state
Check that the hot spare is at least as large as the failed drive
If using a global hot spare, confirm it's compatible with the RAID level and stripe size
For controllers that support it, verify the hot spare is the same type (SAS vs SATA, HDD vs SSD)
Most controllers will automatically start a rebuild when a hot spare is present. If not, you may need to initiate it manually.
3. Understand and Set Rebuild Priority
Hardware RAID controllers allow you to configure the rebuild priority, which governs how much I/O bandwidth the rebuild consumes relative to host I/O. Common options:
Low (Background): Rebuild runs only when the array is idle. Takes longest but minimizes impact on production workloads.
Medium: Balanced — the rebuild and host I/O share bandwidth evenly.
High (Foreground): Rebuild takes priority. Fastest rebuild, but production workloads will see degraded performance.
For LSI/Broadcom controllers via storcli64:
# Check current rebuild rate
storcli64 /c0 show rebuildRate
# Set rate (0-100%, default is usually 30%)
storcli64 /c0 set rebuildRate=30
mdadm-Specific Steps
mdadm (Linux Software RAID) gives you fine-grained control over the recovery process. Understanding the different operational modes and how to monitor them is essential.
Check /proc/mdstat
This pseudo-file is your primary interface for monitoring md RAID status. It shows whether the array is active, degraded, and whether a recovery/resync is in progress:
[UU_U] — the third drive is missing/degraded (U = up, _ = down)
recovery = 4.2% — a full reconstruction is in progress
finish=186.2min — estimated time remaining
speed=125000K/sec — current rebuild throughput
Verify Spare Count
mdadm allows you to designate spare drives that are automatically used when a member fails. Check your spare configuration:
# Show mdadm detail
sudo mdadm --detail /dev/md0
# Look for "Spare Devices" line
sudo mdadm --detail /dev/md0 | grep -i spare
Recovery vs. Resync vs. Reshape
mdadm uses different terms for different operations:
Operation
Trigger
What It Does
recovery
Drive replacement
Full data reconstruction onto a new drive. Reads all data from surviving drives, computes parity, writes to replacement.
resync
Array assembly / reboot
Checksums and re-synchronizes all stripes. Ensures every drive's data matches the computed parity. Does not imply a failure.
reshape
RAID level change or capacity expansion
Restripes data across a different number of devices. Used when changing RAID levels (e.g., RAID 5 → RAID 6) or adding drives.
When replacing a failed drive, you will see recovery in /proc/mdstat. To add the replacement and start recovery:
# Add the new drive to the array
sudo mdadm --add /dev/md0 /dev/sdX
# Check progress
watch -n 5 cat /proc/mdstat
mdadm sync speed limits
mdadm allows you to control recovery speed via sysfs to balance rebuild time against production workload impact. See the Performance Considerations section below.
Performance Considerations
A RAID resilver is an I/O-intensive operation that reads every sector from every surviving drive and writes every sector to the replacement. This creates a massive I/O load that can severely impact production workloads. Understanding and controlling this impact is essential.
Rebuild IOPS Impact on Production Workloads
During a resilver:
Read IOPS dominance: The rebuild reads from every surviving drive at near-maximum sequential throughput. For spinning HDDs (~200 IOPS per drive), this consumes virtually all available IOPS. For SSDs, the impact is less severe but still significant.
Write amplification: On the replacement drive, the rebuild writes sequentially at high throughput. For SSDs, this can accelerate wear and increase write latency for competing workloads.
Controller bottleneck: The HBA or RAID controller's PCIe bandwidth, CPU, and cache are consumed by rebuild traffic. Parity-based RAID (5/6) requires read-modify-write cycles that tax the controller or CPU (for software RAID).
Setting Rebuild Priority / Rate Limits
Each RAID implementation provides controls for throttling the rebuild:
ZFS
# Check current scrub/resilver I/O limit
cat /sys/module/zfs/parameters/
zfs_resilver_min_time_ms
# Control I/O during resilver:
# zfs_vdev_scrub_min_active
# zfs_vdev_scrub_max_active
ZFS dynamically throttles resilver I/O based on the min/max active I/O tunables. Defaults are usually reasonable for mixed workloads.
mdadm
# Check current speed limit
cat /proc/sys/dev/raid/
speed_limit_min
cat /proc/sys/dev/raid/
speed_limit_max
# Set limits (KB/sec)
echo 50000 | sudo tee /proc/sys/dev/raid/speed_limit_min
echo 200000 | sudo tee /proc/sys/dev/raid/speed_limit_max
mdadm's speed_limit_min and speed_limit_max control the rebuild pace directly in KB/sec.
The rebuild rate is a percentage (0-100) of controller I/O bandwidth dedicated to the rebuild. 30% is a common conservative starting point.
Schedule the resilver during low-traffic hours
If you cannot afford the performance hit during peak hours, schedule the resilver to start during a maintenance window. For ZFS, you can set a deferred resilver and manually initiate it at the desired time. For mdadm, set speed_limit_min to a very low value during peak hours and raise it at night.
For detailed calculations on how long your specific array will take to rebuild at different I/O limits, see our RAID Rebuild Time Guide .
Post-Resilver Verification
The resilver is complete when the RAID software reports the array as healthy again. But don't stop there — a successful resilver means the data was written to the new drive, not that it was written correctly. These verification steps catch silent corruption and hardware issues.
1. Scrub After Rebuild
Always run a scrub immediately after the resilver completes. This verifies every block on the replacement drive and ensures the parity checksums match across the array.
Compare the replacement drive's SMART data against the baseline you captured during preparation:
Check Reallocated_Sector_Ct has not increased during the resilver
Verify Current_Pending_Sector is zero
Check Raw_Read_Error_Rate is within normal range
Review UDMA_CRC_Error_Count — high values may indicate a cable or backplane issue
Check Temperature to ensure the drive wasn't thermally stressed
3. Check for UREs (Uncorrectable Read Errors)
A URE during a resilver can cause partial data corruption or, in the worst case, a complete array failure. After the resilver and scrub:
For ZFS: check zpool status -v for any cksum (checksum) errors. Zero means no UREs were encountered.
For mdadm: check cat /sys/block/md0/md/mismatch_cnt. Zero means all stripes matched.
For hardware RAID: check the controller event log via storcli64 /c0 show events or the management UI for media errors.
4. Document and Monitor
The first 48 hours after a resilver are critical. Monitor the replacement drive closely:
Schedule regular SMART checks (cron job every 6 hours for the first 2 days)
Log the drive serial number, install date, and any SMART anomalies
Keep the old (failed) drive for 30 days — it may be needed for forensic analysis or RMA return
Schedule a second scrub within 30 days to catch latent issues
Frequently Asked Questions
How long will a resilver take?
Resilver time depends on drive capacity, drive speed, RAID level, controller/host CPU, and the rebuild I/O limits you've set. As a rough estimate:
4 TB HDD (mirror): ~4–8 hours at default speeds
12 TB HDD (RAID 5/6): ~12–36 hours
20 TB HDD (RAID 6): ~24–72+ hours
SSD (any capacity): ~30 min – 4 hours
For precise estimates based on your specific hardware, RAID level, and I/O limits, see our RAID Rebuild Time Guide.
Should I scrub before or after replacing a drive?
After is mandatory. Before is helpful but optional.
Scrubbing before replacement identifies any latent checksum or parity errors across the array. If the pre-replacement scrub finds errors, fix them first — otherwise those errors will be faithfully copied onto the replacement drive during the resilver.
However, if the array is already degraded (one drive failed), scrubbing will be slower and will report errors for the missing drive. In that case, go straight to replacement and scrub afterward.
Can I use the array while it's resilvering?
Yes, in most cases. All major RAID implementations allow read and write access during a resilver:
ZFS: Fully read/write accessible during resilver. Writes during a resilver are handled via a "resilver DDT" — ZFS captures new data and ensures it's written to the replacement drive as well.
mdadm: Full read/write access, though writes will slow both the recovery and the application workload.
Hardware RAID: Fully accessible. The controller handles consistency transparently.
However, performance will be degraded, especially for write-heavy workloads on parity RAID (5/6). Plan for higher latency and lower throughput.
What if the replacement drive is slightly larger than the original?
This depends on your RAID implementation:
ZFS: The extra capacity is not automatically usable. ZFS replaces the drive at the same size as the original vdev member. To reclaim the extra space, you would need to expand the vdev — which is only possible if you replace all drives in the vdev with larger ones (and even then requires special handling).
mdadm: The extra space is also not automatically used after a replacement. You must run mdadm --grow to expand the array, or replace all drives and then expand.
Hardware RAID: Most controllers will use the full capacity of the new drive only if the RAID group is configured to expand. Otherwise, the extra space is wasted until you grow the virtual disk.
Bottom line: replacing with a larger drive is fine, but don't expect to use the extra space immediately. When in doubt, match the drive size exactly.
Do I need to do anything special for SMR vs CMR drives during resilver?
Yes, absolutely. SMR (Shingled Magnetic Recording) drives have fundamentally different write behavior than CMR (Conventional Magnetic Recording) drives, and this matters enormously during a resilver.
During a resilver, the replacement drive receives a sustained sequential write stream. SMR drives handle sequential writes reasonably well due to their onboard DM-SMR (Device-Managed SMR) shingling. However:
If the SMR drive needs to perform garbage collection orread-modify-write cycles on overlapping shingles, write performance can drop catastrophically — from ~200 MB/s to ~10–20 MB/s
This can extend a 12-hour resilver to 3–5 days
Never mix SMR and CMR drives in the same RAID group or ZFS vdev. The SMR drive will become a permanent performance bottleneck
If you must use SMR, use them only in RAID 1 (mirror) configurations, where write patterns are sequential and predictable
For ZFS specifically, SMR drives can cause the pool to lock up during a resilver if write latency spikes exceed ZFS's transaction group timeout. This is a known issue and a strong argument against using SMR drives in any ZFS pool.
What does 'degraded' actually mean for my data?
A degraded array means one or more drives have failed, but the array is still operational due to redundancy. With RAID 1/10, you lose mirror protection. With RAID 5/6, you lose one or two parity stripes. Your data is still accessible and intact — as long as no additional drives fail. However, any read error on a surviving drive during this period cannot be recovered from parity (since the failed drive would have provided that data). This is why minimizing the resilver window is critical.
Need to Estimate Your Rebuild Time?
Use our detailed RAID Rebuild Time Calculator to estimate how long your specific array will take to resilver at different I/O limits and RAID levels.