How do RAID and ZFS work?

Beyond Traditional RAID – Introducing ZFS RAIDZ

If you're building a homelab or need robust data storage, you've likely heard of RAID. I've been running various RAID setups since 2012/2013 professionally and at home and had all kinds of "fun". While traditional RAID offers some redundancy, ZFS takes data protection to the next level with RAIDZ. RAIDZ is a key component of the ZFS file system, providing data redundancy with built-in checksums to detect and correct errors. Unlike standard RAID, ZFS RAIDZ provides a more resilient solution, preventing silent data corruption and offering more granular control over your storage. Choosing the right RAIDZ level is crucial for balancing redundancy, performance, and cost. This post will break down the different RAIDZ levels and provide guidance on selecting the best option for your needs. We'll focus on the core concepts, so you can confidently configure your ZFS storage.

What about ECC RAM and ZFS?

IMO - You don't need ECC for a selfhosted setup. In all my years being a nerd I've not seen a single instance of bitrot that I could detect.

While ZFS RAIDZ provides excellent data protection against drive failures, silent data corruption – errors that occur at the bit level and aren't immediately apparent – remains a potential threat. This is where Error-Correcting Code (ECC) RAM becomes a vital component of a robust ZFS storage solution. ECC RAM is designed to detect and correct single-bit errors, and in some cases, even multi-bit errors, that can occur within the RAM itself. These errors, though rare, can propagate to the ZFS file system during read/write operations, potentially corrupting your data. ZFS relies heavily on RAM during its scrubbing process – the periodic verification of data integrity across the pool – and if that RAM is unreliable, the scrub itself can become compromised, masking potential problems.

Using ECC RAM significantly reduces the risk of these silent data corruption events and enhances the reliability of ZFS scrubs. During a scrub, ZFS reads each block of data and compares its checksum against the stored checksum. If an error is detected, ZFS attempts to correct it using the redundant data stored within the RAIDZ configuration. With ECC RAM, you're minimizing the chance that an error in RAM will falsely indicate data integrity or, worse, introduce a new error during the correction process. While not a mandatory requirement for ZFS to function, ECC RAM is strongly recommended, especially for critical data storage and long-term archival. It’ve become a standard practice for serious ZFS users, providing an extra layer of defense against data loss and ensuring the integrity of your valuable data.

Rough Chance of Bitrot in Selfhosted NAS

• Best-Case Scenario (Enterprise Drives, Frequent Scrubbing, Data Integrity Tools): 0.01% - 0.1% per year
• Typical Homelab (NAS Drives, Monthly Scrubbing): 0.2% - 1% per year

🪄Understanding RAIDZ1, RAIDZ2, and RAIDZ3

ZFS RAIDZ levels (RAIDZ1, RAIDZ2, and RAIDZ3) dictate the level of data redundancy. RAIDZ1 tolerates one drive failure, RAIDZ2 tolerates two, and RAIDZ3 tolerates three. The higher the level, the greater the protection but also the longer rebuild times and the more storage space consumed by redundancy. For most homelab and small business scenarios, RAIDZ2 is the sweet spot, providing a good balance of redundancy and performance. RAIDZ1 is generally not recommended due to its limited redundancy. RAIDZ3, while offering the highest level of protection, suffers from significantly longer rebuild times, making it less practical for larger pools. Consider your data criticality and budget when making your selection.

Choosing the Right Level – Quick Guide

Here's a quick guide to help you decide:

• 2-3 Drives: Avoid. Consider alternatives.
• 4-10 Drives: RAIDZ2 (Recommended)
• 11-18 Drives: Two sets of RAIDZ2 or RAIDZ3 (Consider rebuild times)
• 19+ Drives: Multiple sets of RAIDZ2 (Scalable and resilient)

Remember that ZFS RAIDZ is not a backup solution. It protects against drive failure, but you still need a separate backup strategy. Prioritize regular scrubbing, monitor SMART data, and choose identical drives for optimal performance. Consult with experts for critical data and always research the latest best practices.

How I Setup ZFS in my Lab

Always start with use case

When I was building my current NAS/server back in late 2023, I already knew the number of drives I had since I got an incredible deal on unused, brand new in box spare Enterprise SAS drives. I got 12X for $35 USD each! They are however, only 6TB but that's plenty for what I need. It's the reliability & MTBF that matter for me. I want my data protected, this is why I'm using ZFS in the first place.

USE CASES

1. Goal: Reliability, resiliency, max mean-time-between-failures
2. UC-1: Media server, storing all my Linux ISOs, family pictures (thanks Immich!), etc...
3. UC-2: Container host (Docker compose)
4. UC-3: One day, maybe a webserver (Via locked down VLAN & reverse proxy, all in docker!)

DRIVES

• 12X Enterprise 3.5 Inch SAS Drives (Seagate ST6000NM14)
• 1X 128GB SSD (Already owned)

My Choice: 12 Disks with ZFS RAIDZ2

But you said max 10 disks for RAIDZ2⁉️

This is the beauty of running your own selfhosted lab! You can do whatever you want. For my use case, this fit perfectly and allowed me to sqeeze out a bit more storage whilst still having a lot of redundancy in case of drive failure.

1. Limited IOPs in this config compare to other configs (2X6disk RAIDZ2)
2. Longer rebuild times in event of failure

Why RAIDZ2 Then?

1. My media storage would all go onto the spinning rust disks, and large files like tv/movies/ISOs, backups, etc, don't require a lot of IOPs.
2. If there were performance issues with any dockers or service, I would move that to the SSD where the OS is.
3. ZFS Adaptive Read Cache mitigates 90% of any performance issues I would have...

ZFS RAM Usage (ARC) you say, tell me moar!

One of the often-misunderstood aspects of ZFS is its RAM usage. ZFS is known to be ‘RAM hungry,’ but this isn’t necessarily a drawback. Right out of the box, ZFS utilizes RAM for its Adaptive Replacement Cache (ARC), which acts as a highly efficient read cache. This means frequently accessed data is stored in RAM, significantly reducing the need to read from slower storage devices. This immediate use of RAM for caching drastically & wildly improves performance, often mitigating many of the common performance concerns associated with file systems. While more RAM generally leads to better ARC performance, ZFS is designed to be functional even with relatively modest amounts of RAM, making it a surprisingly efficient solution even on systems with limited resources.

ARC is 1st cache, default. L2 is secondary, not enabled by default.

%%{init: {"theme":"base", "themeVariables": {
  "primaryColor": "#1e293b",
  "primaryTextColor": "#e2e8f0",
  "primaryBorderColor": "#94a3b8",
  "lineColor": "#94a3b8",
  "secondaryColor": "#0f172a",
  "tertiaryColor": "#1e293b"
}}}%%

flowchart TD
    subgraph SYSTEM["🖥️ System (CPU + RAM)"]
        CPU["🧠 CPU"]
        ARC["💾 ARC (Adaptive Replacement Cache)"]
        L1["⚡ L1 (RAM Cache)"]
    end

    subgraph STORAGE["📦 Storage Layer"]
        L2["🔋 L2 (SSD Cache)"]
        DISK["📀 ZFS Pool (HDDs/SSDs)"]
    end

    %% Read path
    CPU -- "📖 Read request" --> ARC
    ARC -- "✅ Cache hit (fast)" --> L1
    L1 -- "📤 Return data" --> CPU
    ARC -- "❌ Cache miss" --> L2
    L2 -- "If cached → Return data" --> ARC
    L2 -- "If miss" --> DISK
    DISK -- "📤 Return data" --> L2
    L2 --> ARC
    ARC --> CPU

    %% Write path
    CPU -- "✍️ Write request" --> ARC
    ARC --> DISK

If you'd like to see your ZFS ARC usage, simply run arcstat for a quick look, better yet arc_summary and checkout the first chunk, noting if it's HEALTHY or not, and the size info, first 3 lines:

ZFS Subsystem Report                            Fri Aug 22 11:56:18 2025
Linux 6.12.30+bpo-amd64                                  2.2.7-1~bpo12+1
Machine: codex (x86_64)                                  2.2.7-1~bpo12+1

ARC status:                                                      HEALTHY
        Memory throttle count:                                         0

ARC size (current):                                    87.9 %   27.6 GiB
        Target size (adaptive):                        87.8 %   27.5 GiB
        Min size (hard limit):                          6.2 %    2.0 GiB
        Max size (high water):                           16:1   31.4 GiB
        Anonymous data size:                            0.0 %    0 Bytes
        Anonymous metadata size:                      < 0.1 %    1.5 MiB
        MFU data target:                               69.1 %   18.7 GiB
        MFU data size:                                 69.0 %   18.6 GiB
        MFU ghost data size:                                     4.0 GiB
        MFU metadata target:                            2.5 %  697.7 MiB
        MFU metadata size:                              1.4 %  400.6 MiB
        MFU ghost metadata size:                                11.6 GiB
        MRU data target:                               25.6 %    6.9 GiB
        MRU data size:                                 27.0 %    7.3 GiB
        MRU ghost data size:                                     9.9 GiB
        MRU metadata target:                            2.7 %  749.2 MiB
        MRU metadata size:                              2.6 %  721.8 MiB
        MRU ghost metadata size:                                12.8 GiB
        Uncached data size:                             0.0 %    0 Bytes
        Uncached metadata size:                         0.0 %    0 Bytes
        Bonus size:                                     0.1 %   37.9 MiB
        Dnode cache target:                            10.0 %    3.1 GiB
        Dnode cache size:                               9.9 %  317.0 MiB
        Dbuf size:                                      0.2 %   55.7 MiB
        Header size:                                    0.6 %  172.7 MiB
        L2 header size:                                 0.0 %    0 Bytes
        ABD chunk waste size:                         < 0.1 %  853.5 KiB

Note: ZFS by default sets a max size of 50% of your RAM. Mine is 31.4GB for half.

Next, look for "ARC total accesses",you should see something like:

ARC total accesses:                                                11.5G
        Total hits:                                    99.7 %      11.4G
        Total I/O hits:                               < 0.1 %     577.0k
        Total misses:                                   0.3 %      32.0M

This is showing that my current hits are at 99.7% which is great, you want 90% or above, this means your ARC is being utilized properly and the data isn't having to be pulled from the slower disks. It's in RAM!

ZFS CLI Guide💻

Advanced ZFS CLI Guide: Monitoring & Performance

💫Be Careful❗

• Root Privileges: Most of these commands require root privileges (use sudo before the command if you're not logged in as root).
• Dataset vs. Pool: Understand the difference between a pool (the overall storage container) and a dataset (a filesystem or volume within the pool).
• Read the Manual Pages: Use man zpool and man zfs to access the official manual pages for complete details and options for each command.
• Experiment Carefully: Before running any commands on a production system, test them in a non-critical environment.
• zfs receive -F is Destructive: The -F flag on zfs receive forces the receive operation, potentially overwriting existing data. Use with extreme caution!
• Scrub Frequency: The frequency of scrubs depends on your environment and risk tolerance. A good starting point is monthly. Some opt for quarterly.

ZFS RAIDZ Deep Dive🤿 (Drive Count & Best Practices)

Key Industry Best Practices (as of August 2025):

• Prioritize Redundancy: ZFS RAIDZ2 is generally considered the minimum acceptable level of redundancy for data preservation.
• Avoid RAIDZ3 for Large Pools: Rebuild times for RAIDZ3 pools with 10+ drives are prohibitively long, leading to performance bottlenecks and increased risk of data loss during rebuilds.
• Use Identical Drives: Employ drives of the same model, capacity, and speed for optimal performance and compatibility.
• Monitor SMART Data: Regularly monitor SMART data for early signs of drive failure.
• Periodic Scrubbing: Schedule regular ZFS scrubs to verify data integrity and detect silent data corruption.
• Separate Cache and Log Devices (if possible): If your hardware allows, dedicate separate SSDs for ZFS ARC (Adaptive Replacement Cache) and ZIL (ZFS Intent Log) to improve performance.
• Avoid Over-Provisioning: Don't allocate more RAM to ZFS than necessary. The optimal amount depends on your workload and pool size.
• Consider a Hot Spare: Having a hot spare drive automatically replaces a failed drive, minimizing downtime.

IMPORTANT DISCLAIMERS:

• This is a guideline, not a guarantee. Your specific needs and budget will influence the best configuration.
• Rebuild Times: Rebuild times increase dramatically with larger pools and higher RAIDZ levels. Factor this into your risk assessment.
• Silent Data Corruption: While ZFS checksums help detect data corruption, they don't prevent it. Regular scrubbing is essential, and ECC ram helps if you can afford it.
• Hardware Compatibility: Ensure your hardware (motherboard, RAID controller) is fully compatible with ZFS and supports the desired RAIDZ level.
• Data Backup: ZFS is not a backup solution. It provides data protection and redundancy, but you still need a separate backup strategy (e.g., offsite cloud backup, another NAS).
• Consult with Experts: For critical data, consult with experienced ZFS administrators before making any decisions. Reddit and other forums are fantastic for this, or drop me a comment below!

📖Learn more about amazing ZFS!

I. Official Documentation & Core References:

• OpenZFS Website: https://openzfsonline.com/ - The central hub for all things OpenZFS. Includes downloads, community forums, and general information.
• ZFS on Linux - Official Documentation: https://zfsonlinux.org/ - More technical documentation specifically for ZFS on Linux.

II. In-Depth Articles & Tutorials:

• Learn ZFS (Greg Kurtzer's Blog): https://www.gregkurtzer.com/blog/ - Greg Kurtzer is a significant contributor to OpenZFS. His blog provides excellent insights and explanations.
• ZFS Explained (Server Density): https://serverdensity.com/blog/zfs-explained/ - A well-written overview of ZFS concepts.

III. Community Resources & Forums:

• OpenZFS Forums: https://forum.openzfsonline.com/ - A great place to ask questions and engage with the ZFS community.
• Reddit - r/zfs: https://www.reddit.com/r/zfs/ - A community-driven discussion forum for all things ZFS

Now that you've got your drives & storage all figured out, head on over to some other guides to get your lab moving!

• Best NAS OS of 2026?
• Install Guide for OMV8
• Install Guide for Docker Compose