RAID Calculator | StashRaid

How Hardware RAID Cache Works

A hardware RAID controller is a dedicated processor on an expansion card (or integrated on a server motherboard) that manages an array of physical disks independently from the host CPU. At its heart is cache memory — DRAM (DDR3 or DDR4) or, on newer controllers, NVMe-backed cache — that sits between the host bus (PCIe) and the disk interface (SAS / SATA / NVMe).

Every I/O operation — every read and every write — transits through this cache. Without cache, every transaction must wait for the disk platters to spin to the right sector (rotational latency) or for NAND flash to complete a program/erase cycle (SSD write latency). Cache DRAM responds in nanoseconds (50–100 ns) vs. milliseconds for disks (5–15 ms for HDDs, 0.1–0.5 ms for SSDs). That 3–5 order-of-magnitude gap is what hardware cache exploits.

How Cache Works Under the Hood

RAID controllers implement a write-back cache engine with a dedicated DMA engine that moves data between the host memory and cache DRAM without CPU involvement. A parity computation ASIC (on hardware RAID cards) computes XOR parity for RAID 5/6 in-cache before destaging to disk. Cache is divided into segments(typically 64 KB or 128 KB per slot), and a segment table tracks dirty/clean state, LRU ordering, and stripe alignment.

Write-Back vs Write-Through Cache

The two primary cache writing policies define the safety-performance tradeoff for any RAID controller:

Write-Back (Fast, Risky)

Host write lands in DRAM cache — acknowledged immediately (microsecond latency)
Data is dirty in cache until destaged to disk by the controller's background flush engine
Benchmark improvement: 10–50× random write IOPS vs. write-through
Requires functional BBU / CacheVault — without it, power loss = data loss

Write-Through (Slow, Safe)

Host write must be committed to physical disk before acknowledgment
Cache only accelerates reads — write latency = disk latency (5–15 ms HDD)
Zero risk of data loss on power failure — data is always on stable media
Safe without BBU; controller forces this mode if BBU is failed or absent

Adaptive Write-Back Policies

Modern controllers (Broadcom MegaRAID, Microchip Adaptec) implement adaptive write-back: the controller dynamically switches between write-back and write-through based on workload characteristics, BBU charge level, and temperature. Some controllers throttle the write-back window when approaching cache capacity — forcing a synchronous flush before accepting more writes — rather than exhausting the cache and stalling the host.

BBU / CacheVault: What Happens When Power Fails

The Achilles' heel of write-back cache is volatility. DRAM loses its contents within milliseconds of power removal. If the server loses AC power while dirty data exists only in cache, that data is gone. RAID BBU (Battery Backup Unit) and CacheVault technology solve this — in two different ways.

Traditional BBU (Battery Backup Unit)

A Li-Ion or NiMH battery pack connected to the RAID controller
On power loss: battery keeps the cache DRAM refreshed (72 hours typical)
When power returns: controller drains dirty cache to disk, resumes normal ops
Batteries degrade over time (~3 years service life), require supervised replacement
Learning cycle: controller periodically reconditions battery (deep discharge + recharge)

CacheVault (Supercapacitor + NAND Flash)

Supercapacitors charge during normal operation — no electrochemical wear
On power loss: capacitors power an emergency copy of DRAM → dedicated NAND flash
NAND retains data indefinitely with zero power — no 72-hour clock ticking
On power restore: controller copies NAND → DRAM, then flushes to disk
Supercaps last 10+ years — effectively fit-and-forget for most deployments

The Backup Timeline

When AC power fails, the following sequence executes in hardware (typically within 30–60 seconds):

Power-good signal de-asserts — controller detects imminent power loss
Controller stops accepting new I/O from host, flushes in-flight transactions
BBU/CacheVault circuit activates — DRAM refresh begins (BBU) or supercaps charge the NAND copy engine (CacheVault)
For CacheVault: all dirty cache lines are encrypted+checksummed and written to dedicated NAND (~3–10 seconds)
Controller shuts down gracefully — metadata recorded for recovery on next boot

NVMe Cache vs DDR3/DDR4 Cache

RAID cache memory technology has evolved from dedicated DRAM modules to controller-embedded DRAM and, most recently, to NVMe-backed caching on modern Broadcom Tri-Mode controllers.

DDR3 Cache (Legacy)

Found on older controllers (LSI SAS 9260, Adaptec 6805). Speeds up to 1333 MHz. Small capacities (256 MB–1 GB). High latency by modern standards. BBU required for write-back. Discontinued.

DDR4 Cache (Current)

Standard on Broadcom MegaRAID 95xx, Microchip Adaptec 8xxx. Speeds up to 3200 MHz. Capacities 2–8 GB. Lower power draw. CacheVault compatible. Widely deployed in enterprise.

NVMe Cache (Emerging)

Broadcom Tri-Mode (SAS/SATA/NVMe) controllers like the 9600-24i use a dedicated NVMe SSD for cache. Capacities up to 800 GB. Non-volatile — no BBU needed. Acts as both read cache AND write-back journal.

NVMe as Cache — How Broadcom Tri-Mode Works

The Broadcom Tri-Mode (9600 series) architecture replaces traditional DRAM cache with a dedicated NVMe device called the Cache Acceleration Module. Here's how it differs:

Persistent write cache: because NVMe is non-volatile, write-back mode is always safe — no BBU needed. Power loss leaves data intact on the NVMe device.
Massive capacity: NVMe cache SSDs range from 400 GB to 1.6 TB vs. 2–8 GB for DRAM — can cache entire active datasets.
Lower latency than spinning disk (~20 µs NVMe read, vs. ~100 ns DRAM, vs. ~5 ms HDD). DRAM is still faster for small random I/O, but NVMe cache closes the gap for sequential workloads.
Endurance consideration: NVMe cache SSDs experience heavy write traffic — requires high-DWPD (Drive Writes Per Day) enterprise NVMe devices. Broadcom certifies specific drives.

Read Cache vs Write Cache — Separate Pools

RAID controllers partition their cache DRAM into logically separate pools for reads and writes, each optimized for different access patterns. The split is configurable (e.g., 50/50, 75/25 read/write) and can be adjusted via the controller's management utility.

Read Cache

Stores copies of recently/frequently read disk blocks
LRU eviction: least-recently-used blocks are discarded when pool is full
Read-ahead prefetch: controller predicts sequential access and pre-loads next stripe
Always safe — cache is discardable, no risk on power loss
Hit ratio target: >90% for workloads with temporal locality (databases, file servers)

Write Cache

Buffers host writes — coalesces small random writes into larger sequential stripe writes
Write coalescing: four 4 KB random writes to adjacent LBAs become a single 16 KB stripe write
Dirty until destaged: data is at risk if power fails before flush
Background flush: controller destages dirty blocks at a rate matching disk write throughput
High water mark: when dirty cache > 60%, controller throttles new writes to prevent saturation

Stripe Alignment Optimization

A key advantage of write cache is stripe alignment. RAID 5/6 writes suffer from the read-modify-write penalty: a small write requires reading old data + old parity, computing new parity, then writing new data + new parity (4 I/Os for 1 logical write). Write cache batches many small writes into full-stripe writes, avoiding the read-modify-write penalty entirely. This is why write-back improves RAID 5/6 random write performance by 10–50× compared to write-through.

Performance Impact — Benchmark Data

The performance delta between write-back and write-through is one of the largest single-configuration changes you can make on a RAID controller. Below are representative benchmarks from a Broadcom MegaRAID 9560-8i with 4 GB DDR4 cache and 8× Samsung PM1653 enterprise SSDs (RAID 10):

Workload	Write-Through	Write-Back	Improvement
4 KB Random Write (100% write)	42,000 IOPS	1,280,000 IOPS	30.5×
8 KB Random Write (70/30 R/W)	89,000 IOPS	1,050,000 IOPS	11.8×
64 KB Sequential Write	3,800 MB/s	6,400 MB/s	1.7×
4 KB Random Write (RAID 6)	8,500 IOPS	420,000 IOPS	49.4×
Database OLTP (8 KB, 70/30)	1.2 ms avg latency	0.09 ms avg latency	13.3× lower

Source: Internal Broadcom lab testing, MegaRAID 9560-8i, firmware 52.28.2-5954, 8× Samsung PM1653 3.84 TB SSDs, RAID 10 stripe 256 KB, Ubuntu 24.04, fio 3.36. Results will vary with hardware and workload.

Why the Huge Gap?

Write-back eliminates the disk's rotational/NAND latency from the I/O path. The host sees DRAM latency (~100 ns) instead of disk latency (~5,000,000 ns for HDD). For RAID 5/6, write-back additionally eliminates the read-modify-write penalty by coalescing small writes into full-stripe writes, explaining the ~50× improvement for random writes on parity RAID levels.

The “Don't Mix RAID Cache with ZFS” Rule

The Rule: Hardware RAID cache + ZFS = Do Not Do This

ZFS is itself a full-volume manager with its own caching and transaction semantics. Placing a hardware RAID controller with write-back cache in front of ZFS creates two competing caching layers, which can cause data integrity issues, performance degradation, and unpredictable behavior.

Why They Conflict

1. Double Cache, Double Trouble

ZFS has its own Adaptive Replacement Cache (ARC) in RAM and a separate ZFS Intent Log (ZIL) for synchronous writes. A hardware RAID write-back cache adds another DRAM cache between ZFS and the disks. This means:

ZFS thinks data is on stable storage (because the controller acknowledged it) but it's actually still in volatile cache
Both caches evict independently — you lose the predictability that ZFS's ARC is designed to provide
Memory pressure doubles: ZFS wants lots of ARC, the RAID controller's DRAM is wasted on data ZFS already caches

2. Power Loss = Corruption Risk

ZFS guarantees that a synchronous write (fsync) has landed on stable media before returning to the application. But a hardware RAID controller in write-back mode acknowledges the write when it hits volatile DRAM — ZFS's guarantee is violated. If power fails between the RAIDs ack and the actual disk write, ZFS sees a transaction that was supposedly committed but is actually lost.

3. Double Parity Overhead

If you use hardware RAID 5/6 and then stripe ZFS on top (e.g., a zpool of multiple RAID 6 LUNs), you pay the parity computation cost twice — once in the RAID controller and once in ZFS. This is both wasteful and complex to debug when performance is poor.

What to Do Instead

Use an HBA (Host Bus Adapter) in IT/JBOD mode — pass raw disks directly to ZFS with zero RAID cache involvement. Broadcom HBAs (9300, 9400, 9500 series) are the gold standard.
If you must use a RAID controller, set it to write-through mode and disable read cache. This at least prevents the power-loss integrity hole, though you still waste the RAID's DRAM.
ZFS on NVMe: skip hardware RAID entirely and use direct-attached NVMe drives with ZFS — no controller in the path, maximum performance.

When You Need BBU (and When You Don't)

A battery backup unit or CacheVault module is mandatory for write-back cache. Running write-back without a functional BBU is equivalent to running without a safety net — a single power outage corrupts whatever dirty data was in flight. Here's the decision framework:

BBU Strongly Recommended

Transactional databases (MySQL, PostgreSQL, SQL Server, Oracle) — high write rates, fsync-dependent, write performance directly impacts TPS
Virtualization hosts (VMware vSphere, Hyper-V) — many VMs sharing the same RAID volume, bursty random I/O with mixed workloads
High-frequency trading / real-time analytics — sub-millisecond write latency is required, write-back is non-negotiable
Servers without UPS protection — ironically, the less infrastructure you have, the more you need a BBU

BBU Less Critical

Read-heavy file servers (media streaming, document repositories) — writes are infrequent and throughput is less critical
Backup targets (Veeam, Bacula) — sequential writes at tape/disk speed, write-back provides little benefit
Servers with dual-redundant UPS and generator — power failure is practically impossible, BBU still protects against PSU failure
All-NVMe arrays — NVMe drives are fast enough that write-through may be sufficient; or use NVMe-cache controllers which don't need BBU

What Happens When the BBU Fails

The Immediate Impact: Forced Write-Through Mode

The moment a RAID controller detects a BBU fault (failed battery, failed supercapacitor, or communication error), it immediately forces the cache policy to write-through. This is a safety mechanism — without a functional BBU, write-back is too dangerous. But the performance impact is immediate and severe.

Symptoms of a Failed BBU

Visible Indicators

Flashing amber LED on the RAID controller (specific blink pattern per vendor)
Storage management software shows "BBU Failed" or "Battery Capacity Low" alert
System Event Log (SEL) entries: "BBU communication lost" or "Battery failure predicted"
SNMP traps sent to monitoring systems (Nagios, Zabbix, PRTG)

Performance Symptoms

Random write IOPS drops 10–50× overnight (from 1M+ IOPS to 40K IOPS)
Application timeouts: database queries that took 5 ms now take 50+ ms
Increased CPU load from I/O wait (iowait spikes in top/htop)
Users report "system feels slow" — especially for write-heavy operations (saves, commits, log writes)

Urgent Replacement Required

A failed BBU is not a "schedule next maintenance window" issue — it's a restore-write-performance-nowissue. Every hour the array runs in write-through mode, the organization loses productivity. Steps:

Verify BBU failure in the RAID management utility (storcli, arcconf, MegaRAID Storage Manager)
Order replacement BBU module (and consider upgrading to CacheVault if the controller supports it)
Temporarily restore write-back by setting the cache policy to "force write-back with no BBU" (NOT recommended — risk of data loss)
Hot-swap the BBU module on capable controllers (Broadcom 95xx+ supports hot-swap BBU)
After replacement, run BBU learn cycle to calibrate the new battery

Note: Some controllers allow you to force write-back without a BBU. Never do this unless you have a UPS with confirmed runtime and you accept the data-loss risk on dual-power failure.

Storage Parameters

Array Resilience

RAID 5 (Parity)

Pros & Cons

RAID Controller Cache & Battery Backup

In This Guide