RoCE – Configuration and Use Case Examples
NetXtreme-E User’s Manual
February 26, 2018 • NetXtreme-E-UG100
Page 68
To prevent mismatching a combination of
bnxt_en
and
bnxt_re
from being loaded, the following is required:
•
If RedHat/CentOS 7.2 OS was installed to the target system using PXEboot with
bnxt_en
DUD or a kernel
module RPM, delete the file
bnxt_en.ko
found in
/lib/modules/$(uname -r)/extra/bnxt_en/bnxt_en.ko
or
edit
/etc/depmod.d
/.
•
bnxt_en.conf to override to use updated version. Users can also erase the current BCM5741X Linux kernel
driver using the rpm -e kmod-bnxt_en command. RHEL 7.3/SLES 12 Sp2 has bnxt_en inbox driver (older
than v1.7.x). This driver must be removed and the latest bnxt_en be added before applying the bnxt_re
(RoCE drivers).
Windows
Kernel Mode
Windows Server 2012 and beyond invokes the RDMA capability in the NIC for SMB file traffic if both ends are
enabled for RDMA. Broadcom NDIS miniport bnxtnd.sys v20.6.2 and beyond support RoCEv1 and RoCEv2 via
the NDKPI interface. The default setting is RoCEv1.
To enable RDMA:
1.
Upgrade the NIC NVRAM using the appropriate board packages. In CCM or in UEFI HII, enable support for
RDMA.
2.
Go to the adapter
Advanced Properties
page and set
NetworkDirect Functionality
to
Enabled
for each
BCM5741X miniport, or using PowerShell window, run the following command:
Set-NetAdapterAdvancedProperty -RegistryKeyword *NetworkDirect -RegistryValue 1
3.
The following PowerShell commands returns true if
NetworkDriect
is enabled.
a. Get-NetOffLoadGlobalSetting
b. Get-NetAdapterRDMA
Verifying RDMA
To verify RDMA:
1.
Create a file share on the remote system and open that share using Windows Explorer or "net use ….". To
avoid hard disk read/write speed bottleneck, a RAM disk is recommended as the network share under test.
2.
From PowerShell run the following commands:
Get-SmbMultichannelConnection | fl *RDMA*
ClientRdmaCapable : True
ServerRdmaCapable : True
If both Client and Server show True, then any file transfers over this SMB connection use SMB.
3.
The following commands can be used to enable/disable SMB Multichannel:
Server Side:
•
Enable: Set-SmbServerConfiguration -EnableMultiChannel $true
•
Disable: Set-SmbServerConfiguration -EnableMultiChannel $false
