Large Data Solution Discussion

[Enquiries]

Hello There,

I am unsure what to do, hence why I am reaching out. I currently have a 160TB (16TB*10) rack-mounted RAID 5 array, 50TB Terramaster NAS (10TB*5), along with a cupboard filled with External Hard Drives (2TB - 20TB in size over 50 on last count). I am drowning in Data that I need to keep but unsure of what solution to go for. From watching your videos, I think ZFS is a definite and noticed TrueNAS has its Rack options, but not sure what configuration or layout I should go for. Whatever the solution is, it must be able to run without requiring updates as it will have no internet connection.

I can provide further information as required.

[ed]

Hello,

Thank you for reaching out, and I completely understand your dilemma—it sounds like you're managing a massive amount of data! Based on what you've described, you're certainly in need of a robust, scalable, and secure solution that can consolidate your current setup, especially given the combination of a 160TB RAID array, 50TB NAS, and over 50 external hard drives.

From what you’ve shared, here are some considerations and recommendations for your setup:

1. ZFS and TrueNAS:
ZFS is indeed a solid choice, especially for large datasets and long-term retention, due to its excellent data integrity features, snapshot capabilities, and flexibility.
TrueNAS (or TrueNAS Enterprise) is a great platform with strong ZFS support, and its rack-mounted solutions are particularly well-suited for large-scale storage needs like yours.
Given your budget of £20,000, you could likely afford one of TrueNAS's enterprise-level setups that provides significant storage capacity with plenty of room to scale up.
2. Configuration:
Since you're looking to manage hundreds of terabytes, I'd recommend considering the following:

RAIDZ2 or RAIDZ3 on ZFS: These configurations offer strong redundancy with 2 or 3 drive failure tolerance respectively, which is vital when dealing with such large volumes.
Pool Layout: You can create a pool that combines SSDs for caching and metadata, with HDDs for storage. This can accelerate read/write speeds, especially for the large files you're handling.
Storage Tiering: With ZFS, you could leverage SSDs or NVMe drives as a L2ARC (cache) and ZIL (write log) to speed up access for frequently used data, leaving the HDDs for mass storage.
3. Scaling & Consolidation:
Consider combining your existing storage into a unified system with ZFS replication or migration strategies to bring all your data into the TrueNAS environment.
You could look at a multi-pool setup within the system to differentiate your most active data from archives, ensuring that performance-critical data gets the attention it needs, while archival data is safely stored.
TrueNAS can also handle off-network setups, so the lack of an internet connection won’t be an issue. You can manage updates manually, or just stick with a stable version once deployed.
4. Hardware Recommendations:
TrueNAS M40 or M50: These models should fit well within your budget and provide 4U rackmount chassis with up to 3PB (petabyte) storage capacities depending on your drives.
Consider SAS HDDs for the bulk of your storage needs for reliability, and SSD/NVMe for cache and accelerated performance on high I/O tasks.
5. Maintenance-Free Operation:
Since you won’t have an internet connection, you can configure the system to run with minimal maintenance:

TrueNAS allows you to schedule tasks like scrubs, snapshots, and replication at intervals that suit your needs, ensuring that your data integrity is always maintained.
I'd also suggest configuring SMART monitoring and other self-healing properties of ZFS, so the system can detect and repair minor disk issues without manual intervention.
6. Future Growth & Budget Consideration:
With a budget of £20,000, you could comfortably cover the hardware, drives, and any potential setup services if needed. Given the amount of data you currently manage and your growth potential, a scalable system is key. You may not need to purchase all drives upfront, as long as your system allows for easy expansion when needed.