4
–
Running MPI on QLogic Adapters
Open MPI
4-10
IB0054606-02 A
To explicitly disable context sharing, set this environment variable in one of the
two following ways:
PSM_SHAREDCONTEXTS=0
PSM_SHAREDCONTEXTS=NO
The default value of
PSM_SHAREDCONTEXTS
is
1
(enabled).
Restricting IB Hardware Contexts in a Batch Environment
If required for resource sharing between multiple jobs in batch systems, you can
restrict the number of IB hardware contexts that are made available on each node
of an MPI job by setting that number in the
PSM_SHAREDCONTEXTS_MAX
or
PSM_RANKS_PER_CONTEXT
environment variables.
For example, if you are running two different jobs on nodes using a QDR IB HCA,
set
PSM_SHAREDCONTEXTS_MAX
to 8 instead of the default 16. Each job would
then have at most 8 of the 16 available hardware contexts. Both of the jobs that
want to share a node would have to set
PSM_SHAREDCONTEXTS_MAX=8
.
Setting
PSM_SHAREDCONTEXTS_MAX=8
as a clusterwide default would
unnecessarily penalize nodes that are dedicated to running single jobs. QLogic
recommends that a per-node setting, or some level of coordination with the job
scheduler with setting the environment variable should be used.
The number of contexts can be explicitly configured with the
cfgctxts
module
parameter. This will override the default settings based on the number of CPUs
present on each node. See
“IB Hardware Contexts on the QDR IB Adapters” on
PSM_RANKS_PER_CONTEXT
provides an alternate way of specifying how PSM
should use contexts. The variable is the number of ranks that will share each
hardware context. The supported values are 1, 2, 3 and 4, where 1 is no context
sharing, 2 is 2-way context sharing, 3 is 3-way context sharing and 4 is the
maximum 4-way context sharing. The same value of
PSM_RANKS_PER_CONTEXT
must be used for all ranks on a node, and typically, you would use the same value
for all nodes in that job. Either
PSM_RANKS_PER_CONTEXT
or
PSM_SHAREDCONTEXTS_MAX
would be used in a particular job, but not both. If
both are used and the settings are incompatible, then PSM will report an error and
the job will fail to start up.
NOTE
MPIs use different methods for propagating environment variables to the
nodes used for the job; See
for examples. Open MPI will
automatically propagate PSM environment variables.
Summary of Contents for OFED+ Host
Page 1: ...IB0054606 02 A OFED Host Software Release 1 5 4 User Guide...
Page 14: ...xiv IB0054606 02 A OFED Host Software Release 1 5 4 User Guide...
Page 22: ...1 Introduction Interoperability 1 4 IB0054606 02 A...
Page 96: ...4 Running MPI on QLogic Adapters Debugging MPI Programs 4 24 IB0054606 02 A...
Page 140: ...6 SHMEM Description and Configuration SHMEM Benchmark Programs 6 32 IB0054606 02 A...
Page 148: ...8 Dispersive Routing 8 4 IB0054606 02 A...
Page 164: ...9 gPXE HTTP Boot Setup 9 16 IB0054606 02 A...
Page 176: ...A Benchmark Programs Benchmark 3 Messaging Rate Microbenchmarks A 12 IB0054606 02 A...
Page 202: ...B SRP Configuration OFED SRP Configuration B 26 IB0054606 02 A Notes...
Page 206: ...C Integration with a Batch Queuing System Clean up PSM Shared Memory Files C 4 IB0054606 02 A...
Page 238: ...E ULP Troubleshooting Troubleshooting SRP Issues E 20 IB0054606 02 A...
Page 242: ...F Write Combining Verify Write Combining is Working F 4 IB0054606 02 A Notes...
Page 280: ...G Commands and Files Summary of Configuration Files G 38 IB0054606 02 A...
Page 283: ......
