6–RoCE Configuration
Configuring RoCE on the Adapter for Linux
69
AH0054602-00 A
Verify the RoCE connection by performing an RDMA ping:
On the server, issue the following command:
ibv_rc_pingpong -d <ib-dev> -g 0
On the client, issue the following command:
ibv_rc_pingpong -d <ib-dev> -g 0 <server L2 IP address>
The following are examples of successful ping pong tests on the server and the
client.
Server Ping:
root@captain:~#
ibv_rc_pingpong -d qedr0 -g 0
local address: LID 0x0000, QPN 0xff0000, PSN 0xb3e07e, GID fe80::20e:1eff:fe50:c7c0
remote address: LID 0x0000, QPN 0xff0000, PSN 0x934d28, GID fe80::20e:1eff:fe50:c570
8192000 bytes in 0.05 seconds = 1436.97 Mbit/sec
1000 iters in 0.05 seconds = 45.61 usec/iter
Client Ping:
root@lambodar:~#
ibv_rc_pingpong -d qedr0 -g 0 192.168.10.165
local address: LID 0x0000, QPN 0xff0000, PSN 0x934d28, GID fe80::20e:1eff:fe50:c570
remote address: LID 0x0000, QPN 0xff0000, PSN 0xb3e07e, GID fe80::20e:1eff:fe50:c7c0
8192000 bytes in 0.02 seconds = 4211.28 Mbit/sec
1000 iters in 0.02 seconds = 15.56 usec/iter
To display RoCE statistics, issue the following commands, where
X
is the
device number:
>
mount -t debugfs nodev /sys/kernel/debug
>
cat /sys/kernel/debug/qedr/qedr
X
/stats
VLAN Interfaces and GID Index Values
If you are using VLAN interfaces on both the server and the client, you must also
configure the same VLAN ID on the switch. If you are running traffic through a
switch, the InfiniBand applications must use the correct GID value, which is based
on the VLAN ID and VLAN IP address.
Based on the following results, the GID value (-x 4 / -x 5) should be used for any
perftest applications.
#
ibv_devinfo -d qedr0 -v|grep GID
GID[ 0]: fe80:0000:0000:0000:020e:1eff:fe50:c5b0
GID[ 1]: 0000:0000:0000:0000:0000:ffff:c0a8:0103
GID[ 2]: 2001:0db1:0000:0000:020e:1eff:fe50:c5b0
GID[ 3]: 2001:0db2:0000:0000:020e:1eff:fe50:c5b0
GID[ 4]: 0000:0000:0000:0000:0000:ffff:c0a8:0b03
IP address for VLAN interface
GID[ 5]: fe80:0000:0000:0000:020e:1e00:0350:c5b0
VLAN ID 3
