3
–
InfiniBand
®
Cluster Setup and Administration
Performance Settings and Management Tips
IB0054606-02 A
3-27
Increasing the number of kernel receive queues allows more CPU cores to be
involved in the processing of verbs traffic. This is important when using parallel file
systems such as Lustre or IBM's GPFS (General Parallel File System). The
module parameter that sets this number is
krcvqs
. Each additional kernel
receive queue (beyond the one default queue for each port) takes user contexts
away from PSM and from the support of MPI or compute traffic. The formula
which illustrates this trade-off is:
PSM Contexts = 16 - (krcvqs-1)x
num_ports
Where
number_ports
is the number of ports on the HCA
For example, on a single-port card with
krcvqs=4
set in
modprobe.conf
:
PSM Contexts = 16 - (4-1)x 1 = 16 - 3 = 13
If this were a 12-core node, then 13 is more than enough PSM contexts to run an
MPI process on each core without making use of context-sharing. An example ,
ib_qib options
line in the
modprobe.conf
file, for this 12-core node case is:
options ib_qib singleport=1 krcvqs=4
can be used as a guide for setting the
krcvqs
parameter for the
number of cores in the system supporting PSM processes and the number of
ports in the HCA. The table applies most readily to nodes with 1 HCA being used
to support PSM (for example, MPI or SHMEM) processes. For nodes with multiple
HCAs that are being used for PSM, the table decide the maximum number of
cores that will be assigned on each HCA to support PSM (MPI or SHMEM)
processes, then apply the table to each HCA in turn.
Table 3-2.
krcvqs
Parameter Settings
Cores per Node (to
be used for
MPI/PSM on 1
HCA):
1-port, Set krcvqs=
2 active ports in the HCA,
Set krcvqs=
61-64
1
a
1
a
57-60
2
1
a
53-56
3
2,1 (2 for port 1, 1 for one
port)
12-52
4
2
8-11
3
2,1 (2 for port 1, 1 for one
port)
4-7
2
1
a
1-3
1
a
1
a
Summary of Contents for OFED+ Host
Page 1: ...IB0054606 02 A OFED Host Software Release 1 5 4 User Guide...
Page 14: ...xiv IB0054606 02 A OFED Host Software Release 1 5 4 User Guide...
Page 22: ...1 Introduction Interoperability 1 4 IB0054606 02 A...
Page 96: ...4 Running MPI on QLogic Adapters Debugging MPI Programs 4 24 IB0054606 02 A...
Page 140: ...6 SHMEM Description and Configuration SHMEM Benchmark Programs 6 32 IB0054606 02 A...
Page 148: ...8 Dispersive Routing 8 4 IB0054606 02 A...
Page 164: ...9 gPXE HTTP Boot Setup 9 16 IB0054606 02 A...
Page 176: ...A Benchmark Programs Benchmark 3 Messaging Rate Microbenchmarks A 12 IB0054606 02 A...
Page 202: ...B SRP Configuration OFED SRP Configuration B 26 IB0054606 02 A Notes...
Page 206: ...C Integration with a Batch Queuing System Clean up PSM Shared Memory Files C 4 IB0054606 02 A...
Page 238: ...E ULP Troubleshooting Troubleshooting SRP Issues E 20 IB0054606 02 A...
Page 242: ...F Write Combining Verify Write Combining is Working F 4 IB0054606 02 A Notes...
Page 280: ...G Commands and Files Summary of Configuration Files G 38 IB0054606 02 A...
Page 283: ......
