RDMA is one of the fastest ways to share files from your NAS, these are 6 reasons why you don't need it

You may have dreaded watching the progress bar crawl as you share huge files from your NAS to your computer. You may have wondered if there was a faster way besides using SMB or NFS. There is, but not a straightforward one. Remote Direct Memory Access (RDMA) can deliver higher transfer speeds with low latency, provided you configure it correctly.

It is significantly faster than FTP, HTTP, SMB, or NFS, since it is designed for high-speed transfers with low latency and low CPU overhead between machines. That said, RDMA isn’t merely a checkbox in an OS or app setting. It’s an ecosystem that requires specific compatibility of hardware, software, and network (at least 10GbE) to work in harmony. Here are some reasons that work against RDMA being the fastest way to share files from a NAS, but you probably won't need it just for regular usage.

RDMA is a two-way street for high-speed file sharing

Both devices and the network must support RDMA

RDMA enables one device to access another device’s memory directly, bypassing the CPU, operating system, and kernel. That’s why it is so fast. However, to make it work, both your NAS and the computer require RDMA-capable network cards. Without that, you wouldn’t experience the high-speed transfer boost even after tweaking your setup. These components are necessary, along with a managed network switch that supports RDMA over Converged Ethernet (RoCE v2) transport protocol, which enables low-latency transfers at high speeds for RDMA to function effectively.

RDMA-ready hardware doesn’t come cheap

Most consumer NAS don’t support it

Most consumer-grade NAS software doesn’t support RDMA out of the box. To make one work, you’ll need to install a PCIe RDMA-capable NIC expansion card in your NAS. There are limited options for NICs. To take the full advantage of RDMA, you’ll probably need an enterprise-grade NAS that supports RDMA-specific storage protocols (iSER or SMB Direct). Such NAS units come with SmartNIC or PCIe expansion options, but they’re pricey. A basic enterprise-level enclosure, such as an Enterprise-grade NAS, would cost you roughly $2800 or more. Since you’ll need NVMe drives for faster storage throughput over HDDs, the costs will further increase significantly.

Your computer also needs an RDMA-supporting NIC. For a smooth networking experience, you’ll want a managed network switch that supports lossless internet with features such as Priority-based Flow Control or Data Center Bridging. Not only that, you’ll need to use appropriate cables for your setup, such as DAC (Direct Attach Copper) for shorter runs and fiber optic for longer runs.

RDMA setup is not plug-and-play

Needs tech know-how for setup, tuning, and troubleshooting

RDMA is designed for enterprise environments. Hence, setting up a NAS on a home network is a complex process. Some high-end NAS devices, such as those from QNAP, Synology, Dell, and NetApp, feature RDMA support at the OS level, but only in premium or enterprise-tier models. You can opt for a DIY route with TrueNAS SCALE or another Linux server with NAS-specific services. However, you'll need to ensure the compatibility of kernel modules, firmware, and drivers. For networking, a managed switch with a Priority-based Flow Control (PFC) or Data Center Bridging (DCB) feature is crucial for avoiding packet loss when using RDMA protocols. Misconfiguration can lead to corrupted or incomplete files. With the DIY route, you’ll need to tune, troubleshoot (without beginner-friendly tools), and maintain the software side of things, such as firmware, drivers, and kernel modules, to keep everything running efficiently.

Bottlenecks are tough to avoid

Storage, OS, or network can slow things down

RDMA is fast, but it takes a lot to make it work. Enabling RDMA won’t make your HDDs or SATA SSDs blazing fast. To reap the benefits of an RDMA setup, you’ll need a NAS with NVMe drive support and a high-throughput handling system like ZFS or XFS. Another common bottleneck is NIC compatibility, as obtaining the correct drivers and software to work with your NAS OS can get challenging. Besides open-source tools to get every component talking nickel with others, you might need custom drivers and kernel modules. And to top it all, regular testing and tuning can also become tricky because of limited RDMA debugging tools, which aren't beginner-friendly and often tend to be vendor-specific.

Ignoring packet drops and congestion hurts performance

RDMA fails silently, without warning you

After setting up RDMA on your NAS and computers, you’ll need to configure the network switch to alert you when congestion occurs and pause the traffic. For that, using the Priority-based Flow Control or Data Center Bridging feature in the network switch is essential. Still, you’ll need to configure them because RDMA doesn’t retransmit dropped or corrupted data packets. That could break large file transfers. If your switch supports it, you can enable Explicit Congestion Notification, which signals slowdowns to the sender machine during congestion to prevent file corruption or packet loss. That said, these features are common in enterprise-level products and not commonly offered in consumer gear.

Proper power, space, and cooling matter

High throughput emits heat, shoots up electricity bills

Setting up RDMA gear results in heat and higher power draw. For starters, the network switches can typically draw 100W to 250W of power, entirely depending on how you’ve set it up. Meanwhile, the power consumption of a NAS could range from 20W to 200W. The entire RDMA setup could easily pull 400W or more. As a result, your electricity bills will increase noticeably. That kind of power draw also generates a significant amount of heat, which means you’ll need proper cooling and ventilation. For a home lab setup, a dedicated space or server room can help keep unpleasantly hot and noisy gear away.

RDMA is blazing fast, but overkill for most home NAS users

RDMA offers the fastest way to share files from your NAS, but it comes at a significant cost and requires more effort to set up for home users. It’s an absolute overkill unless you engage in large file transfers, such as 4K-8K video editing or high-performance workloads like scientific data crunching. It’s not just about buying all RDMA-compatible NICs, managed switches, and cables, but also about ensuring the software works seamlessly with all that hardware. However, if you’re willing to pay through the nose and can afford to set up RDMA, it can be fun to tinker and a stepping stone to your serious home NAS usage.