Deduplication and compression are two of the most powerful — and most misunderstood — features in modern RAID-capable filesystems. Compression is essentially free on today's hardware and should almost always be enabled. Deduplication, on the other hand, can either save you terabytes or catastrophically degrade performance depending on your workload. This guide breaks down exactly how both technologies work at the block and file level, what their real-world resource requirements are, and when you should — and should not — use them.
How Deduplication Works at the Block Level vs. File Level
Deduplication is the process of identifying duplicate data and storing only a single copy, replacing subsequent duplicates with pointers to the original. The granularity at which this happens determines nearly everything about its performance characteristics.
Block-Level Deduplication
Block-level dedup operates on fixed-size (typically 4 KiB–128 KiB) or variable-size blocks within a storage pool. Every block written to disk is hashed — usually with a cryptographic hash like SHA-256 or a faster non-crypto hash like xxHash or Fletcher-4 — and the hash is looked up in a Deduplication Table (DDT). If the hash already exists, the filesystem concludes the block is identical and increments a reference count instead of writing the data again.
ZFS deduplication is the canonical example of block-level inline dedup. Every write passes through the DDT lookup before the data ever touches disk. This means the DDT must live in RAM (or at least in ARC) for the lookup to be fast enough not to stall every write. The result is extremely space-efficient for workloads with many identical blocks — VM disk images, container layers, database snapshots — but brutally RAM-hungry.
File-Level Deduplication
File-level dedup (sometimes called "single-instance storage") identifies entire files that are identical and stores only one copy. It's simpler, requires far less memory, and imposes minimal write-time overhead because the hash comparison happens once per file rather than once per block. The obvious downside: partial matches are missed. If two VM disk images share 90% of their blocks but differ in the last 10%, file-level dedup stores two full copies.
Windows Server Data Deduplication and Apple's APFS (in certain configurations) use file-level or chunk-level approaches. Linux VDO(Virtual Data Optimizer) operates at the block layer and can be placed under any filesystem, but imposes its own memory overhead. Btrfsoffers both file-level (via cp --reflink) and block-level dedup through out-of-band tools like duperemove.
Key Insight
Block-level dedup catches far more duplicates but requires dramatically more RAM. File-level dedup is memory-cheap but misses partial matches. Most home-lab users overestimate how many duplicate blocks their data actually contains — run azdb -S estimate before enabling ZFS dedup.
ZFS Dedup: The RAM Requirement Formula
The most frequently cited — and most frequently ignored — rule in ZFS dedup is that the Deduplication Table requires roughly 5× the size of the data being deduplicated in RAM. More precisely, each entry in the DDT consumes about 320 bytes (for a single-block entry with the full hash chain). If you have 10 TB of unique data after dedup, and your average block size is 4 KiB, that's roughly 2.6 billion DDT entries — requiring about 800 GB of RAM for the table alone.
Data Set Size
Avg Block Size
DDT Entries (est.)
RAM Required (est.)
Feasible?
1 TB
4 KiB
~268 M
~80 GB
Expensive
10 TB
4 KiB
~2.68 B
~800 GB
Impractical
1 TB
128 KiB (recordsize)
~8.4 M
~2.5 GB
Reasonable
100 TB
128 KiB
~838 M
~250 GB
Specialized
The ZFS Special Device (also called a "dedup table device" or "metadata vdev") can mitigate some of this pressure by offloading the DDT to a fast NVMe or Optane device. With a special device, the DDT is stored on low-latency flash instead of system RAM, freeing memory for ARC. However, lookup latency increases from ~100 ns (RAM) to ~10 µs (NVMe), which can still add up under heavy write loads. The practical formula becomes: DDT storage ≈ 5× the post-dedup unique data, placed on the fastest available device.
Warning
Running ZFS with dedup enabled and insufficient RAM or a slow DDT device is one of the most common causes of catastrophic pool performance. Write throughput can drop from gigabytes-per-second to kilobytes-per-second as the pool spends all its time thrashing the DDT. Always runzdb -S on a representative sample of your data before enabling dedup.
LZ4 & ZSTD Compression in ZFS — Essentially Free on Modern CPUs
Compression in ZFS is often described as "free" — and that's closer to the truth than a marketing exaggeration. Modern compression algorithms running on modern CPUs can compress data faster than the storage subsystem can write it, meaning the CPU finishes compressing before the I/O completes, resulting in a net performance gain because fewer physical blocks are written to disk.
LZ4: The Default Goldilocks Algorithm
LZ4 is a compression algorithm designed for extreme speed. On a single modern x86-64 core, LZ4 can compress data at 500–1,000 MB/s and decompress at 2–4 GB/s. On a typical 12-core server, LZ4 compression bandwidth far exceeds what even a PCIe 4.0 NVMe array can consume. The algorithm's simplicity means it requires almost no CPU cache pressure and introduces negligible latency — typically under 1 microsecond per 4 KiB block.
ZFS defaults to lz4 for good reason: it provides excellent compression on text, logs, source code, databases, and VM images (typically 1.5×–3× compression ratios) while imposing zero measurable overhead on mixed workloads. Even incompressible data — compressed video, audio, archives — passes through LZ4 with only a few CPU cycles wasted detecting the incompressibility and storing the block uncompressed.
ZSTD: Higher Ratios When You Need Them
ZSTD (Zstandard) is a newer algorithm from Facebook/Meta that offers a wide range of compression levels (1–19, plus ZSTD-fast variants). At its lowest level (1), ZSTD roughly matches LZ4 in speed while offering 5–15% better compression ratios. At higher levels (3–6), ZSTD can achieve 2×–4× compression on text-heavy data, though at proportionally higher CPU cost.
OpenZFS 2.0+ includes native ZSTD support. The sweet spot for most workloads is ZSTD-3 — noticeably better ratios than LZ4 with only a modest CPU increase. For archival datasets where write-once-read-rarely is the pattern, ZSTD-6 or even ZSTD-9 can be worthwhile. For already-compressed media libraries, stick with LZ4 (or just disable compression — LZ4's fast-detect path handles incompressible data gracefully anyway).
Algorithm
Compress Speed
Decompress Speed
Ratio (text)
Ratio (binaries)
CPU Cost
LZ4
500–1,000 MB/s
2–4 GB/s
2.0× – 3.0×
1.2× – 1.5×
Negligible
ZSTD-3
200–500 MB/s
1–2 GB/s
2.5× – 4.0×
1.4× – 1.8×
Low
ZSTD-9
50–150 MB/s
1–2 GB/s
3.0× – 5.0×
1.5× – 2.0×
Moderate
GZIP-9
10–30 MB/s
100–300 MB/s
3.5× – 6.0×
1.6× – 2.2×
High
The key insight: compression reduces physical I/O. If your data compresses at 2:1, the storage subsystem only needs to write half as many blocks — which means faster writes, longer SSD lifespan, and more effective use of your RAID bandwidth. In most cases, enabling compression is strictly better than leaving it off, even on low-power CPUs like Intel N100 or ARM-based NAS units.
Btrfs Compression and Dedup Tools
Btrfs takes a different philosophy from ZFS. Where ZFS offers inline dedup as an integrated feature (with all the RAM implications that come with it), Btrfs treats dedup as an out-of-band, post-process operation. This is both a strength and a limitation.
Btrfs Compression
Btrfs supports zlib,lzo, andzstd compression. It can be enabled per-subvolume or per-file with mount options (compress=zstd:3). Like ZFS, Btrfs detects incompressible data and stores it uncompressed. The implementation is per-file extent-based: each file extent is compressed individually, which means random access to compressed files is efficient — you only decompress the extents you need, not the whole file.
One notable difference from ZFS: Btrfs compression can be appliedretroactively usingbtrfs filesystem defrag -r -c zstd /mountpoint. This recompresses existing data without rewriting the entire filesystem. ZFS compression, by contrast, only applies to newly written data unless you rewrite existing files.
A caveat: aggressive defrag/recompression on Btrfs can fragment the free space and increase metadata overhead. It's best used on specific subvolumes (e.g.,@log, @docker) rather than the entire pool.
Duperemove and Btrfs Dedup
Duperemove is the primary dedup tool for Btrfs. It scans a filesystem, computes hashes (xxHash by default, or SHA-256) for each block, finds extents with identical hashes, and issuesFIDEDUPERANGE ioctls to merge them. It operates entirely in userspace and can be run on live, mounted filesystems without downtime.
Key duperemove parameters:
-d — deduplicate (the actual merge step; without this it only scans)
-h — hash algorithm (xxhash is fastest; sha256 is cryptographically safe)
--lookup-extents — use Btrfs extent tree instead of full file scan (much faster)
The hash table that duperemove builds is stored on disk as a hashfile (specified with--hashfile), which can be reused across runs. This is a crucial difference from ZFS: Btrfs dedup memory scales with the size of the data you're scanning in a single pass, not with the entire pool's unique data. You can dedup a 100 TB array with 8 GB of RAM by scanning in smaller batches.
Btrfs vs. ZFS Dedup
Btrfs + duperemove trades real-time dedup for vastly lower memory requirements. However, dedup is not instantaneous — data written between runs will be duplicated until the next scan. For workloads with frequent writes of duplicate data (e.g., CI/CD build artifacts), ZFS inline dedup may still win. For relatively static data, Btrfs post-process dedup is far more resource-efficient.
Hardware RAID Controller Dedup — Why It Doesn't Really Exist
A common question from users coming from the enterprise storage world: "Does my hardware RAID controller support deduplication?" The short answer is no, and for fundamental architectural reasons.
Hardware RAID controllers (Broadcom/Avago MegaRAID, Adaptec, Microchip, etc.) operate at the block virtualization layer. Their job is to present a logical block device (the RAID volume) to the operating system by striping, mirroring, or parity-protecting physical disk blocks. They have no understanding of files, volumes, or data content — they see only logical block addresses (LBAs) and raw data buffers.
Deduplication requires a content-aware layer that can hash incoming data, consult a hash table, and redirect writes to existing blocks. This is fundamentally a filesystem-level concern. Some "enterprise storage arrays" from NetApp, Dell EMC, and Pure Storage offer dedup — but this is implemented in the storage controller's software at the volume or filesystem layer, not in the RAID engine itself.
Hardware RAID + ZFS Dedup — A Bad Combination
If your hardware RAID controller exposes a single logical disk to ZFS, ZFS has no visibility into the underlying disk layout. It cannot issue TRIM/DISCARD commands efficiently, the controller's write-back cache may interfere with ZFS transaction groups, and if the controller reorders writes, ZFS's transactional integrity guarantees can be compromised. For ZFS dedup (or any ZFS feature), use HBA mode (IT mode) so the OS sees each disk directly — leave the "RAID" to ZFS itself.
Real-World Use Cases: When Dedup Pays Off vs. When It's a Trap
The decision to enable deduplication should be driven by a single, hard question: how many identical blocks does your data actually contain? The answer determines whether dedup saves you money or destroys performance.
✅ When Dedup Pays Off
VM Templates and Golden Images.A hypervisor cluster storing 50 copies of the same Windows Server or Ubuntu template — even with slight customization — will have enormous block-level overlap. Dedup ratios of 10:1 to 50:1 are common.
Container Registries and CI/CD Caches.Docker/OCI image layers are inherently block-identical across builds. A GitLab or Jenkins CI cache storing build artifacts benefits dramatically from both dedup and compression.
VM Snapshots and Backups.Incremental snapshots of virtual machines share the vast majority of blocks with their parent. ZFS snapshots combined with dedup can reduce backup storage by 5×–20×.
Database Development/Test Environments.Multiple cloned databases differ by tiny fractions of their total block count. Dedup makes clones effectively free.
❌ When Dedup Is a Trap
Media Libraries (Video, Audio, Photos).Already-compressed formats — H.264/H.265 video, MP3/AAC audio, JPEG/WebP images — contain almost no duplicate blocks. Enabling dedup on a Plex or Jellyfin library will consume terabytes of DDT RAM for essentially zero space savings.
Already-Compressed Archives.ZIP, RAR, 7z, and tar.gz files are already compressed. They're also typically unique enough that blocks rarely repeat. Dedup on an archive server is usually counterproductive.
Encrypted Data.Full-disk encryption randomizes block content. Two identical 1 GiB files encrypted with different keys (or even the same key at different offsets) produce completely different ciphertext blocks. Dedup and encryption are fundamentally incompatible — if you need encryption, dedup will save nothing.
Low-RAM Systems.A home NAS with 8–16 GB of RAM and 20+ TB of storage cannot safely run ZFS dedup. The DDT will evict from ARC, every write becomes a DDT cache miss, and pool throughput drops to unusable levels.
Compression: Always On (With Rare Exceptions)
Unlike dedup, compression is almost always a net win. The only scenarios where you might disable it:
CPU-bound workloads on very slow processors(e.g., ancient ARM SoCs). Even here, LZ4 is so fast that the I/O reduction usually outweighs the CPU cost.
Data that is already highly compressed(media server, archive storage). LZ4 detects incompressibility quickly, but ZSTD at higher levels may waste CPU trying to compress it.
Benchmarking — you may want compression off to measure raw device throughput.
Performance Impact of Dedup on Writes
The performance cost of dedup is concentrated entirely on the write path. Reads are unaffected (or even slightly faster, since dedup reduces the working set size). Writes, however, incur the full cost of:
Hash computation. Every block must be hashed. SHA-256 computes at roughly 500–800 MB/s per core on modern hardware. xxHash (available in newer OpenZFS) is 10× faster but is not cryptographically secure.
DDT lookup. The DDT is typically a hash table or AVL tree. A lookup involves traversing the data structure and comparing hashes. In RAM, this takes ~1–5 µs per block. On a special device (NVMe), it's ~10–50 µs. On a hard disk DDT, it's measured in milliseconds — catastrophic.
DDT update. If the block is new, the DDT must be updated with the new hash → block pointer mapping. This is a write to the DDT itself, doubling the metadata write load.
Reference count update. If the block exists, the reference count is incremented. This is a read-modify-write on the DDT entry.
Under ideal conditions (DDT in RAM, fast CPU, large recordsize), ZFS dedup adds roughly 30–50% overhead to write latency. Under poor conditions (DDT on HDD, small blocks, many concurrent writers), write throughput collapses by 90–99%.
Scenario
Write Throughput
CPU Utilization
RAM Pressure
Verdict
No dedup, LZ4
Baseline
~5%
None
Optimal
Dedup + LZ4, DDT in RAM
60–80% baseline
~15–25%
High
Use-case dependent
Dedup + LZ4, DDT on NVMe
40–60% baseline
~15–25%
Moderate
Acceptable with care
Dedup + LZ4, DDT on HDD
1–5% baseline
~30–50%
Low
Do not do this
Dedup & Compression Across Storage Platforms
Platform
Dedup Type
RAM Requirement
Compression
Best For
Dedup Verdict
ZFS (OpenZFS)
Block-level, inline
~5× unique data (in RAM)
LZ4, ZSTD, GZIP, LZJB
Enterprise VM clusters, NAS
Powerful but RAM-hungry
Btrfs
Post-process (duperemove), reflink
Per-scan, low (8 GB sufficient)
ZSTD, LZO, ZLIB
Desktop Linux, Synology, home lab
Resource-efficient, not real-time
Synology DSM (Btrfs)
Post-process (shared folder dedup)
Moderate (DSM manages scheduling)
ZSTD on Btrfs volumes
SMB/NFS home directories, backups
Good for NAS appliances
TrueNAS (SCALE/Core)
ZFS inline or post-process (plugins)
Same as ZFS: 5× unique data
LZ4 (default), ZSTD, GZIP
Enterprise/ZFS-native NAS
Only with sufficient RAM
unRAID
Plug-in based (CA Deduplicator)
Moderate (file-level, user-initiated)
None natively; ZFS plugin adds ZSTD
Mixed HDD/SSD home servers
Limited; best with ZFS plugin
Frequently Asked Questions
Is ZFS dedup worth it?
It depends entirely on your data and hardware. For VM templates, container layers, and database snapshots — where dedup ratios of 5:1 to 50:1 are common — ZFS dedup can save enormous amounts of storage and is worth the RAM investment. For media libraries, encrypted data, or already-compressed archives, dedup will consume RAM with zero benefit. Always run zdb -S on a representative sample first. If you see a dedup ratio below 1.5:1, don't enable it.
How much RAM does dedup need?
The rule of thumb: 5× the size of your unique (post-dedup) data. For a pool with 10 TB of unique data and 4 KiB blocks, the DDT alone requires ~800 GB of RAM. Using a larger recordsize (e.g., 128 KiB) reduces the entry count proportionally, bringing the requirement down to ~25 GB for the same 10 TB. A ZFS special device (fast NVMe) can store the DDT off-RAM, but at the cost of higher lookup latency. Minimum recommended RAM for any ZFS system with dedup: 64 GB, and that's for modest datasets.
Does compression affect RAID performance?
Yes — positively. Compression reduces the number of physical blocks written to disk, which means less data traverses the RAID parity calculation pipeline, fewer disk I/Os are needed, and SSD wear is reduced. On RAID-5/6 arrays, compression can improve write throughput by 20–60% because the parity calculations operate on fewer blocks. On RAID-0 or RAID-10, the improvement is proportional to the compression ratio. The CPU cost of LZ4 compression is negligible compared to the I/O latency it saves. ZSTD at level 3 represents a small CPU trade-off for significantly better ratios.
Can I dedup on hardware RAID?
Hardware RAID controllers do not support deduplication — they operate at the block-virtualization layer and have no awareness of data content. If your hardware RAID controller is in "RAID mode" (exposing a single virtual disk), you can still run a filesystem with dedup on top of it (e.g., ZFS on a hardware RAID volume), but this is strongly discouraged. The RAID controller's write-back cache, write reordering, and lack of TRIM pass-through can corrupt ZFS's transactional integrity. Always use HBA/IT mode for ZFS. For Btrfs, hardware RAID is less problematic but still non-optimal — the controller may hide disk health information needed by Btrfs's self-healing features.
What's the difference between inline and post-process dedup?
Inline dedup (ZFS) checks for duplicates during every write, before data reaches disk. Duplicate data is never written. This gives instantaneous space savings but requires the DDT to be fast enough to keep up with every write — hence the enormous RAM requirement. Post-process dedup (Btrfs + duperemove, VDO, or Windows Server Dedup) writes data normally first, then scans and merges duplicates in the background. This imposes no write-path latency, uses far less memory, and can be scheduled during off-peak hours. The trade-off: duplicates consume space temporarily, and post-process dedup may miss blocks that change between scan cycles. For most home-lab and SMB workloads, post-process dedup is the safer, more practical choice.
Find Your Optimal RAID Configuration
Not sure whether dedup or compression is right for your setup? Use the StashRaid RAID calculator to model different configurations, compare space efficiency, and see exactly how compression and dedup affect your usable capacity.