6
–
SHMEM Description and Configuration
Implementation Behavior
6-16
IB0054606-02 A
For a put operation, these descriptions use the terms "local completion" and
“remote completion”. Once a put is locally complete, the source buffer on the
initiating PE is available for reuse. Until a put is locally complete the source buffer
must not be modified since that buffer is in use for the put operation. A blocking
put is locally complete immediately upon return from the put. A non-blocking put is
not locally complete upon return from the put
—
different mechanisms are used for
detecting local completion using either an explicit handle (use
shmem_test_nb()
or
shmem_wait_nb())
or a NULL handle (use
shmem_quiet()
). Once a put is remotely complete the destination buffer on the
target PE is fully written and available for use. The mechanism provided by
SHMEM for detecting remote completion are described below.
shmem_fence()
- This function ensures that all puts issued by this PE prior
to the fence will become remotely visible before any puts issued by this PE
after the fence. The call does not necessarily imply that any of the prior puts
are actually remotely visible at the point of the fence, only that this ordering
is guaranteed.
shmem_quiet()
- This function waits for remote completion of all puts
issued by this PE prior to the quiet operation. Therefore, once the quiet
operation returns, it is guaranteed that all those puts will be remotely visible
to other PEs. This guarantee of remote completion applies to all
puts
—
blocking puts, non-blocking puts with handles, and non-blocking puts
with NULL handles. Additionally, this function additionally waits for local
completion of non-blocking puts and non-blocking gets that were issued with
a NULL handle.
shmem_test_nb()
and
shmem_wait_nb()
can be used to test and wait
for local completion of a
non_blocking
operation. For a non-blocking put,
this does not indicate whether remote completion has occurred.
Additional properties of the QLogic SHMEM implementation are:
The QLogic SHMEM implementation makes no guarantees as to the
ordering in which the bytes of a put operation are delivered into the remote
memory. It is *not* a safe assumption to poll or read certain bytes of the put
destination buffer (for example, the last 8 bytes) to look for a change in value
and then infer that the entirety of the put has arrived. The correct
mechanism for this is to use the
shmem_quiet()
operation to force remote
completion, or to use the following type of sequence:
Initiator side:
Issue a batch of puts all unordered with respect to each other
shmem_fence()
Summary of Contents for OFED+ Host
Page 1: ...IB0054606 02 A OFED Host Software Release 1 5 4 User Guide...
Page 14: ...xiv IB0054606 02 A OFED Host Software Release 1 5 4 User Guide...
Page 22: ...1 Introduction Interoperability 1 4 IB0054606 02 A...
Page 96: ...4 Running MPI on QLogic Adapters Debugging MPI Programs 4 24 IB0054606 02 A...
Page 140: ...6 SHMEM Description and Configuration SHMEM Benchmark Programs 6 32 IB0054606 02 A...
Page 148: ...8 Dispersive Routing 8 4 IB0054606 02 A...
Page 164: ...9 gPXE HTTP Boot Setup 9 16 IB0054606 02 A...
Page 176: ...A Benchmark Programs Benchmark 3 Messaging Rate Microbenchmarks A 12 IB0054606 02 A...
Page 202: ...B SRP Configuration OFED SRP Configuration B 26 IB0054606 02 A Notes...
Page 206: ...C Integration with a Batch Queuing System Clean up PSM Shared Memory Files C 4 IB0054606 02 A...
Page 238: ...E ULP Troubleshooting Troubleshooting SRP Issues E 20 IB0054606 02 A...
Page 242: ...F Write Combining Verify Write Combining is Working F 4 IB0054606 02 A Notes...
Page 280: ...G Commands and Files Summary of Configuration Files G 38 IB0054606 02 A...
Page 283: ......
