71
Perform a Latency Test
To demonstrate the latency advantage of InfiniBand compared to Ethernet, use a netperf test called TCP
request/response. This test will send a 1 byte request to the remote machine and the remote machine will
issue a 1 byte response.
6. Develop a base case for comparison on
Server 1
. Add the
-t TCP_RR
option to the
netperf
command to specify this test.
Example
7. Read the results. The sample results show performance of about 5800 request/response transactions
per second.
Example
8. Run the test over the InfiniBand interface on
Server 1
.
Example
9. Compare the results. The IB interface shows about 11600 request/response transactions per second,
which is approximately double the performance of the gigabit Ethernet interface.
SDP Performance vs IPoIB Using netperf
About SDP
If you performed the steps in
“IPoIB Performance vs Ethernet Using netperf” on page 70
, you saw that
it's difficult to take advantage of the high bandwidth of InfiniBand using IPoIB without sacrificing the
CPU overhead associated with TCP/IP.
To solve the CPU overhead problem, the SDP (Sockets Direct Protocol) can be used over the fabric. The
SDP protocol sets up a reliable connection over the InfiniBand fabric, and TCP socket connections can
be made without the overhead of TCP. RDMA (Remote Direct Memory Access) semantics are used in
the protocol, which essentially transmits data between the two host's buffers without CPU intervention.
Configuring SDP
The decision to use this protocol rather than setting up a normal TCP socket is made at the kernel level.
Applications do not have to be re-written or re-compiled to take advantage of this capability. The
decision to use this protocol rather than setting up a normal TCP socket is made at the kernel level.
There are a variety of methods to control how connections are configured to use SDP, as documented in
/usr/local/topspin/etc/libsdp.conf.
# netperf -c -C -f g -H 192.168.10.21 -t TCP_RR
TCP REQUEST/RESPONSE TEST to 192.168.10.21
Local /Remote
Socket Size Request Resp. Elapsed Trans. CPU CPU S.dem S.dem
Send Recv Size Size Time Rate local remote local remote
bytes bytes bytes bytes secs. per sec % T % T us/Tr us/Tr
16384 87380 1 1 10.00 5787.80 4.50 7.30 7.775 12.619
netperf -c -C -f g -H 192.168.0.2 -t TCP_RR
TCP REQUEST/RESPONSE TEST to 192.168.0.2
Local /Remote
Socket Size Request Resp. Elapsed Trans. CPU CPU S.dem S.dem
Send Recv Size Size Time Rate local remote local remote
bytes bytes bytes bytes secs. per sec % T % T us/Tr us/Tr
16384 87380 1 1 10.00 11629.08 18.30 19.51 15.733 16.777
