NAS for VM storage

[eppo120]

I'm currently looking for recommendations for hardware to use for a new NAS for my homelab.
I'm planning to set up 2 servers that will run XCP-NG. I would like for this NAS to be the shared storage for it.
Not only would I be running the VM's disks from this, I would also be setting up some docker volumes to run from it via NFS.
I would like to run 10GB SFP, and be able to max out the bandwidth with the storage in there. My thoughts were to use an SSD for TrueNAS, and use 2-M.2's in a mirror for the pool.
I really dont need a ton of storage on this, so I might use 2-2TB M.2's. I have a separate instance of Truenas that I use for archival type stuff.
So basically I want to ensure that my VM disk access is as fast as possible, and would not limit the speed of my VM's.
Please let me know if there are any extra details that I may have left out, or anything I would need to consider.
Thank you,

Joe

[Fanny Moen]

Choosing the right hardware for a high-performance NAS, especially when you want to optimize bandwidth for running virtual machines and Docker over 10GbE, requires careful calculation and consideration of every detail. From choosing a CPU, a motherboard with enough PCIe slots for network cards and NVMe SSDs, to deciding on the right RAM capacity and type... Everything needs to work together for the system to run smoothly and efficiently.

This reminds me of the game Geometry Arrow. To achieve a high score, you also need to accurately calculate the shooting angle, the force of the shot, and predict the trajectory of the arrow. Just a small mistake in choosing the angle or timing, your arrow can miss the target. Both building a powerful NAS system and playing Geometry Arrow require precision, meticulousness, and a bit of strategy to achieve the best results."

[ed]

Here are some thoughts on how to achieve your goals for high-performance NAS storage for your XCP-NG and Docker volumes.

Key Considerations for Your NAS Setup:
10Gb SFP+ Connectivity:

Since you want to run 10Gb SFP+ to maximize bandwidth, make sure your NAS has support for SFP+ ports or the option to add an SFP+ network card (either built-in or via PCIe expansion).

For high-speed access, ensure that the network interface on both the NAS and the XCP-NG hosts supports 10Gb SFP+ and that your switches also support 10Gb if they are part of the setup.

Storage Pool Setup with SSDs:

2 x 2TB M.2 SSDs in a mirror for the storage pool is a solid choice. M.2 NVMe SSDs will give you excellent IOPS and low latency, which is critical for running VM disks and Docker volumes.

TrueNAS is a great choice for managing ZFS and provides a good balance of redundancy (RAID-Z or mirror) and performance. ZFS also has compression and deduplication options, which might benefit your VM storage depending on the workload.

RAID Configuration:

A RAID-1 mirror setup will give you redundancy, but if you want to maximize read and write speeds and don't need redundancy (since you're using two SSDs), you could use a RAID-0 (striped) setup for higher performance. However, you lose redundancy with RAID-0, so consider your data protection strategy.

ZFS Mirror: In TrueNAS, a ZFS mirror is similar to RAID-1 and provides redundancy while still giving you excellent read speeds. It’s a safe option for VM storage.

VM Disk Storage:

Make sure your VMs are stored on fast storage, so using SSD-backed storage will greatly benefit VM disk I/O performance. ZFS also supports ZIL (ZFS Intent Log), which can improve write performance for workloads like databases, so consider enabling that if your VMs are write-intensive.

You can set up your VMs via NFS or iSCSI as shared storage. NFS tends to be easier to configure and manage, especially with XCP-NG.

Caching and Tuning:

If you're aiming to run high-performance workloads and use TrueNAS, consider adding SSD caching. TrueNAS supports both read and write caching for SSDs, which can significantly improve performance.

You can also tune the ARC (Adaptive Replacement Cache) in ZFS to make the most out of your RAM for caching frequently accessed data.

Potential NAS Hardware:
QNAP TVS-x82T Series:

QNAP offers 10Gb SFP+ support, and their TVS-x82T series could be a good match. You get high-speed connectivity and a strong set of features for VM storage.

You can also add M.2 NVMe SSDs to speed up access. The QNAP QTS system offers a variety of storage pools with both RAID and ZFS.

Synology DS923+:

While not as expandable as some other models, Synology DS923+ offers support for NVMe SSD caching and can easily handle 10GbE via a PCIe expansion card.

This model would work well with TrueNAS or even Synology's own DSM if you don’t mind running DSM as a hypervisor.

TerraMaster F4-423:

TerraMaster F4-423 is another 4-bay option with a strong focus on storage performance, and it supports 10GbE connectivity. This might be a great budget-friendly option for you if you’re looking to stay cost-effective while still getting good performance for VMs.

DIY NAS with a Dedicated Server:

If you're open to building a custom solution, a DIY NAS server with TrueNAS could be perfect for your needs. You could use a Supermicro or ASRock Rack server, equip it with 10Gb SFP+ NIC, and 2 x 2TB M.2 SSDs in a mirrored pool. This would give you full control over the hardware and a powerful solution for VM and Docker storage.

Additional Thoughts:
ZFS is one of the best file systems for running VMs due to its reliability and high performance. Given that you’re running XCP-NG, you’ll benefit from ZFS for snapshots and volume management.

You might also want to consider dedicated caching drives (such as NVMe SSDs) to ensure you're optimizing read/write performance, especially if your workloads are read-heavy.