12–NVMe-oF Configuration with RDMA
Testing the NVMe-oF Devices
193
AH0054601-00 B
Testing the NVMe-oF Devices
Compare the latency of the local NVMe device on the target server with that of the
NVMe-oF device on the initiator server to show the latency that NVMe adds to the
system.
1.
Update the Repository (Repo) source and install the Flexible Input/Output
(FIO) benchmark utility on both the target and initiator servers as follows:
#
yum install fio
Figure 12-4. FIO Utility Installation
2.
Run the FIO utility to measure the latency of the initiator NVMe-Of device.
Issue the following command:
#
fio --filename=/dev/nvme0n1 --direct=1 --time_based
--rw=randread --refill_buffers --norandommap --randrepeat=0
--ioengine=libaio --bs=4k --iodepth=1 --numjobs=1
--runtime=60 --group_reporting --name=temp.out
FIO reports two latency types: submission and completion. Submission
latency (slat) measures application-to-kernel latency. Completion latency
(clat), measures end-to-end kernel latency. The industry-accepted method is
to read
clat percentiles
in the 99.00th range.
In this example, the initiator device NVMe-oF latency is 30usec.
3.
Run FIO to measure the latency of the local NVMe device on the target
server. Issue the following command:
#
fio --filename=/dev/nvme0n1 --direct=1 --time_based
--rw=randread --refill_buffers --norandommap --randrepeat=0
--ioengine=libaio --bs=4k --iodepth=1 --numjobs=1
--runtime=60 --group_reporting --name=temp.out
In this example, the target NVMe device latency is 8µsec. The total latency
that results from the use of NVMe-oF is the difference between the initiator
device NVMe-oF latency (30µsec) and the target device NVMe-oF latency
(8µsec), or 22µsec.