RAID Monitoring & Scheduled Maintenance
A comprehensive guide to keeping your RAID arrays healthy — from data scrubbing and SMART monitoring to alerting, preventive maintenance, and common pitfalls.
RAID (Redundant Array of Independent Disks) gives us redundancy, performance, or both — but it is not a backup. A RAID array that isn't monitored is a time bomb. Here's why monitoring is critical.
⚠️Silent Failures
A disk can fail partially — developing bad sectors or stuck heads — without the RAID controller raising an alarm. In RAID 5 or RAID 6, a degraded drive may keep passing read requests while silently reporting incorrect data. By the time an OS-level error surfaces, the array may already be in a degraded state or irrecoverable.
🦠Bit Rot (Data Degradation)
Magnetic media degrades over time. Individual bits can flip due to cosmic rays, magnetic interference, or simple age — a phenomenon known as bit rot or data degradation. Without regular scrubbing that reads every block and recomputes parity, these silent errors accumulate. ZFS and Btrfs offer checksumming to detect bit rot automatically; mdadm and hardware RAID do not.
📉Unrecoverable Read Errors (UREs)
Every hard drive spec sheet includes a URE (Unrecoverable Read Error) rate — typically 1 in 1014 to 1016 bits read for consumer drives. During a RAID rebuild, the controller readsevery block of every remaining drive to reconstruct the failed drive's data. A single URE on a large consumer-grade array (e.g., 8 TB+ RAID 5) during rebuild can abort the entire rebuild and result in permanent data loss.
The Monitoring Mindset
A scrub reads every block on every disk in the array and verifies that the computed parity (or checksum) matches what's on disk. Any mismatch is corrected, and the bad block is remapped. This is the single most important maintenance task you can perform.
How Scrubbing Works by RAID Type
| RAID Type | Scrub Mechanism | Bit Rot Detection | Typical Command |
|---|---|---|---|
| ZFS | Full block checksum verification; repairs from parity / mirror | Yes (native checksums) | zpool scrub <pool> |
| mdadm (Linux) | raid-check script reads all blocks and verifies parity | Partial (parity check only) | echo check > /sys/.../md/sync_action |
| Btrfs | Checksum-based scrub across all data and metadata | Yes (native checksums) | btrfs scrub start /mount/point |
| Hardware RAID | Vendor-specific patrol read / consistency check | Depends on controller | perccli / storcli / MegaRAID |
Scheduling Recommendations
How often you scrub depends on array size, workload, and risk tolerance:
- 1Weekly — Enterprise / mission-critical arrays with frequent writes. ZFS recommends scrubbing every 1–2 weeks for production data.
- 2Monthly — Home lab / small business / moderate usage. Most common recommendation for mdadm and Btrfs.
- 3Quarterly — Large cold-storage arrays (60 TB+) where scrubbing takes multiple days. Balance thoroughness against I/O impact.
Can I scrub while the array is in use?
/sys/block/mdX/md/sync_speed_max. Best practice: run scrubs during low-traffic windows.How Long Should a Scrub Take?
Scrub duration depends on disk count, size, speed, and array utilization. A rough rule of thumb:
- • 4× 4 TB HDDs in RAID 5: ~8–12 hours
- • 8× 8 TB HDDs in RAID 6: ~24–36 hours
- • All-SSD arrays: typically 1–4 hours depending on controller and interface
If a scrub takes longer than your scheduled window, split the array into smaller vdevs (ZFS) or reduce the sync speed ceiling (mdadm).
S.M.A.R.T. (Self-Monitoring, Analysis, and Reporting Technology) is built into virtually every modern HDD and SSD. It tracks dozens of internal attributes that can predict failure weeks in advance.
Critical SMART Attributes to Watch
| Attribute | ID | What It Measures | Warning Threshold | Action |
|---|---|---|---|---|
| Reallocated_Sector_Ct | 05 | Bad sectors remapped to spare area | Any non-zero value | Monitor trend; replace if increasing rapidly |
| Current_Pending_Sector | C5 | Unstable sectors awaiting reallocation | > 0 | Immediate investigation; likely failing |
| Offline_Uncorrectable | BB | Sectors that could not be read after offline scan | > 0 | Replace drive immediately |
| Raw_Read_Error_Rate | 01 | Frequency of read errors | Declining normalized value | Check cable/backplane; monitor trend |
| Spin_Retry_Count | 0A | Number of spindle start/stop retries | > 0 | Mechanical issue; replace drive |
| Temperature_Celsius | C2 | Drive temperature | > 50°C (HDD) / > 65°C (SSD) | Improve airflow; check ambient temp |
Trending Over Time
A single static SMART reading is less useful than a trend. A drive that grows from 10 to 100 reallocated sectors over a week is in distress. Use tools like smartd (part of smartmontools) to log attributes to a database, or use a monitoring platform (Prometheus + node_exporter, Grafana, Zabbix) to chart values over time. The Backblaze SMART stats — published quarterly — provide excellent reference baselines for consumer and enterprise drives.
SMART ≠ Crystal Ball
Drives can run built-in diagnostic scans — called SMART self-tests — that exercise the read/write heads, media, and onboard electronics. These are independent of the OS and RAID layer.
Short vs. Long Self-Tests
| Feature | Short Test | Long / Extended Test |
|---|---|---|
| Duration | 1–2 minutes (HDD) / ~30 seconds (SSD) | Hours (depends on drive size) |
| What is tested | Electrical, servo, read/write head, first ~8 GB of media | Full media scan — every LBA on the drive |
| Detection capability | Catches ~90% of imminent failures | Catches latent media defects and weak sectors |
| Frequency recommendation | Weekly to monthly | Monthly to quarterly |
| I/O impact | Negligible | Moderate — can slow scrubs and normal I/O |
Running SMART Tests
All tests run via smartctl:
# Run a short self-test on /dev/sda smartctl -t short /dev/sda # Run a long self-test smartctl -t long /dev/sda # Check test results (look at "Self-test execution status") smartctl -l selftest /dev/sda # Automate with smartd (config: /etc/smartd.conf) # Schedule: short test Mon-Fri at 2am, long test Saturday at 3am DEVICESCAN -a -o on -S on -s (S/../.././02|L/../../6/03) -m [email protected]
SSD Considerations
A scrub report or SMART warning is useless if nobody reads it. Automated alerting is what turns monitoring into a working safety net.
TrueNAS (Core & Scale)
TrueNAS has built-in alerting via the web UI. System → Alert Settings lets you configure email (SMTP) and multiple alert types:
- • S.M.A.R.T. test failure → immediate email warning
- • ZFS pool scrub errors → alert when scrub completes with errors
- • Disk temperature warnings → configurable threshold (recommend 50°C)
- • Pool capacity thresholds → warn at 80%, critical at 95%
TrueNAS also supports Slack, PagerDuty, and Discord webhooks under System → Alert Services.
unRAID
unRAID's web UI includes a Settings → Notification Settings panel. Supported channels:
- • Email via SMTP (Gmail, SendGrid, custom)
- • Pushover — mobile push notifications
- • Slack / Discord / Telegram webhooks
- • Custom Agent script for arbitrary webhooks
unRAID also automatically runs a monthly parity check (scrub) and emails the result — make sure this is enabled.
Synology DSM
Synology DSM's Storage Manager has built-in SMART Test scheduling and Data Scrubbing for SHR/RAID arrays. Alerts can route to:
- • Email (via Synology's SMTP or custom server)
- • SMS (via Synology's SMS gateway partners)
- • Push notifications (DS finder app)
- • Webhook (HTTP POST to custom endpoint)
Go to Control Panel → Notification → Advanced to set up rules like "SMART test failed" or "Storage space degraded" and route them to multiple channels simultaneously.
Linux / mdadm (Manual Setup)
For custom Linux servers, you can wire up alerts with three core tools:
# 1. mdadm monitoring daemon mdadm --monitor --daemonise [email protected] /dev/md0 # 2. smartd (smartmontools daemon) - /etc/smartd.conf DEVICESCAN -a -o on -S on -m [email protected] \ -M diminishing -M test # 3. ZFS event daemon (zpool events) # Pipe critical events to a script zpool events -f | while read line; do if echo "$line" | grep -q "checksum|io_failure"; then /usr/local/bin/send-alert.sh "$line" fi done
For more advanced setups, use zabbix-agent, prometheus-node-exporter with Alertmanager, or netdata — all three can scrape mdadm and SMART data and push alerts via email, Slack, or webhook.
A well-equipped monitoring toolkit spans SMART, RAID, and filesystem layers. Here are the essential tools and what they're used for.
smartctl (smartmontools)
The Swiss Army knife of drive health. Read smartctl -a /dev/sda for a full dump of SMART attributes. Schedule tests with smartd, the system daemon.
smartctl -H /dev/sda # quick health checkmdadm --monitor
Built-in event monitor for Linux software RAID. Detects degraded devices, failed drives, and scrub completion. Can email alerts directly.
mdadm --detail /dev/md0 # array statuszpool events
ZFS event log with real-time monitoring. Use zpool status -v for scrub status and recent errors. Pipe events to a notification script.
zpool status -x # show degraded pools onlyScrubbing Scripts
Automate scrubs via cron/systemd. mdadm ships /etc/cron.weekly/raid-check; ZFS needs a small shell script wrapper.
zpool scrub tank && zpool status tankPrometheus + node_exporter
Export mdadm, ZFS, and SMART metrics to Prometheus. Disk statistics, temperature, and SMART attributes available via standard exporters.
node_exporter --collector.diskstatsGrafana Dashboards
Visualise drive health, temperatures, scrub progress, and SMART trends over time. Pre-built dashboards for mdadm and ZFS available on Grafana Labs.
# Dashboard ID 14032 for mdadmEssential One-Liners
# Check RAID status cat /proc/mdstat mdadm --detail /dev/md0 # Check ZFS pool health zpool status -v zpool list # Scrub an array echo check > /sys/block/md0/md/sync_action # mdadm zpool scrub tank # ZFS btrfs scrub start /mnt # Btrfs # SMART health summary for d in /dev/sd[a-z]; do smartctl -H "$d"; done # Check for UREs / disk errors grep -i "error|fail|defect" /var/log/syslog | grep -i "sd|md|zfs"
Even experienced admins make these mistakes. Avoid them and your array will thank you.
❌ Monitoring Only One Drive Type
Many home-builders monitor the OS/boot drive (often an NVMe SSD) but ignore the data drives. Set up monitoring for every physical drive in the system. With smartd, theDEVICESCAN directive auto-discovers all ATA/SAS drives connected to the system.
❌ Ignoring URE (Unrecoverable Read Error) Logs
A single URE on a consumer drive during a RAID 5 rebuild can bring the entire array down. Check /var/log/syslog or dmesg for SCSI errors, and review SMART attribute 01 (Raw_Read_Error_Rate) and attribute BB (Offline_Uncorrectable). If these climb, schedule a drive replacement before the next rebuild.
❌ Not Checking After Rebuilds
A RAID rebuild completes? Great — but don't assume data integrity. Every time a drive is replaced and the array rebuilds, run a full scrub immediately afterward. The rebuild reconstructs data from parity or mirrors, but it doesn't verify correctness. A post-rebuild scrub catches any latent errors that might have accumulated.
❌ Relying Solely on RAID for Backups
RAID protects against drive failure — not fire, theft, ransomware, or accidental deletion. The "3-2-1" backup rule (3 copies, 2 media types, 1 off-site) still applies. Monitoring is about knowing the health of your primary storage; backups are about recoverability of your data regardless of that health.
❌ Ignoring SSD Wear-Leveling
SSDs wear out from writes, not spin-ups. Monitor Wear_Leveling_Count (or Percentage Used for NVMe) — when it approaches 100%, the drive is nearing end of life. An SSD can go from healthy to read-only or dead in a matter of days once spare blocks are exhausted.
❌ Scrubbing Too Aggressively (or Not at All)
Scrubbing every day on large HDD arrays wears out disks prematurely and burns system I/O. Scrubbing once a year risks missing failures that compound over time. Find a rhythm that matches your risk profile (monthly is the industry median) and stick to it.
Beyond monitoring, a proactive maintenance schedule extends drive life and prevents surprises. Here's a checklist to run through on a regular cadence.
Firmware Updates
- • Check drive vendor site for firmware updates quarterly
- • Update RAID controller (HBA) firmware before it's needed
- • Read changelogs — some firmware fixes address data corruption bugs
- • Always backup before a firmware flash
Cable & Connection Inspection
- • Reseat SATA/SAS cables every 6–12 months
- • Check for bent pins in SATA connectors
- • Inspect power cables for melted/brittle insulation
- • Replace any cable that shows wear
Temperature Logging
- • Log drive temps daily; alert if above 50°C (HDD) / 65°C (SSD)
- • Max recommended ambient: 22–27°C (72–80°F)
- • Ensure front-to-back airflow through drive bays
- • Replace case fans at first sign of bearing noise
Backup Testing
- • Test a full restore from backup at least quarterly
- • Verify backup integrity with checksums
- • Test off-site restore (cloud / remote location)
- • Document restore procedures while everything is healthy
Monthly Maintenance Checklist
- Check
/proc/mdstatorzpool status— verify all drives are online - Review SMART attributes for all drives — focus on reallocated, pending, and uncorrectable sectors
- Verify last scrub completion date — ensure it finished without errors
- Check system logs for disk I/O errors or timeouts
- Confirm backup jobs ran and completed successfully
- Check drive temperatures — compare against last month's baseline
Quarterly Maintenance Checklist
- Run a full SMART long self-test on every drive
- Check for firmware updates for drives and controller
- Inspect and reseat cables and drive connections
- Perform a test restore from backup
- Clean dust from filters, fans, and drive bays
- Review alerting configuration — test email/webhook delivery