115 lines
4 KiB
Text
115 lines
4 KiB
Text
|
INFINIBAND MIDLAYER LOCKING
|
||
|
|
|
||
|
|
This guide is an attempt to make explicit the locking assumptions
|
||
|
|
made by the InfiniBand midlayer. It describes the requirements on
|
||
|
|
both low-level drivers that sit below the midlayer and upper level
|
||
|
|
protocols that use the midlayer.
|
||
|
|
|
||
|
|
Sleeping and interrupt context
|
||
|
|
|
||
|
|
With the following exceptions, a low-level driver implementation of
|
||
|
|
all of the methods in struct ib_device may sleep. The exceptions
|
||
|
|
are any methods from the list:
|
||
|
|
|
||
|
|
create_ah
|
||
|
|
modify_ah
|
||
|
|
query_ah
|
||
|
|
destroy_ah
|
||
|
|
bind_mw
|
||
|
|
post_send
|
||
|
|
post_recv
|
||
|
|
poll_cq
|
||
|
|
req_notify_cq
|
||
|
|
map_phys_fmr
|
||
|
|
|
||
|
|
which may not sleep and must be callable from any context.
|
||
|
|
|
||
|
|
The corresponding functions exported to upper level protocol
|
||
|
|
consumers:
|
||
|
|
|
||
|
|
ib_create_ah
|
||
|
|
ib_modify_ah
|
||
|
|
ib_query_ah
|
||
|
|
ib_destroy_ah
|
||
|
|
ib_bind_mw
|
||
|
|
ib_post_send
|
||
|
|
ib_post_recv
|
||
|
|
ib_req_notify_cq
|
||
|
|
ib_map_phys_fmr
|
||
|
|
|
||
|
|
are therefore safe to call from any context.
|
||
|
|
|
||
|
|
In addition, the function
|
||
|
|
|
||
|
|
ib_dispatch_event
|
||
|
|
|
||
|
|
used by low-level drivers to dispatch asynchronous events through
|
||
|
|
the midlayer is also safe to call from any context.
|
||
|
|
|
||
|
|
Reentrancy
|
||
|
|
|
||
|
|
All of the methods in struct ib_device exported by a low-level
|
||
|
|
driver must be fully reentrant. The low-level driver is required to
|
||
|
|
perform all synchronization necessary to maintain consistency, even
|
||
|
|
if multiple function calls using the same object are run
|
||
|
|
simultaneously.
|
||
|
|
|
||
|
|
The IB midlayer does not perform any serialization of function calls.
|
||
|
|
|
||
|
|
Because low-level drivers are reentrant, upper level protocol
|
||
|
|
consumers are not required to perform any serialization. However,
|
||
|
|
some serialization may be required to get sensible results. For
|
||
|
|
example, a consumer may safely call ib_poll_cq() on multiple CPUs
|
||
|
|
simultaneously. However, the ordering of the work completion
|
||
|
|
information between different calls of ib_poll_cq() is not defined.
|
||
|
|
|
||
|
|
Callbacks
|
||
|
|
|
||
|
|
A low-level driver must not perform a callback directly from the
|
||
|
|
same callchain as an ib_device method call. For example, it is not
|
||
|
|
allowed for a low-level driver to call a consumer's completion event
|
||
|
|
handler directly from its post_send method. Instead, the low-level
|
||
|
|
driver should defer this callback by, for example, scheduling a
|
||
|
|
tasklet to perform the callback.
|
||
|
|
|
||
|
|
The low-level driver is responsible for ensuring that multiple
|
||
|
|
completion event handlers for the same CQ are not called
|
||
|
|
simultaneously. The driver must guarantee that only one CQ event
|
||
|
|
handler for a given CQ is running at a time. In other words, the
|
||
|
|
following situation is not allowed:
|
||
|
|
|
||
|
|
CPU1 CPU2
|
||
|
|
|
||
|
|
low-level driver ->
|
||
|
|
consumer CQ event callback:
|
||
|
|
/* ... */
|
||
|
|
ib_req_notify_cq(cq, ...);
|
||
|
|
low-level driver ->
|
||
|
|
/* ... */ consumer CQ event callback:
|
||
|
|
/* ... */
|
||
|
|
return from CQ event handler
|
||
|
|
|
||
|
|
The context in which completion event and asynchronous event
|
||
|
|
callbacks run is not defined. Depending on the low-level driver, it
|
||
|
|
may be process context, softirq context, or interrupt context.
|
||
|
|
Upper level protocol consumers may not sleep in a callback.
|
||
|
|
|
||
|
|
Hot-plug
|
||
|
|
|
||
|
|
A low-level driver announces that a device is ready for use by
|
||
|
|
consumers when it calls ib_register_device(), all initialization
|
||
|
|
must be complete before this call. The device must remain usable
|
||
|
|
until the driver's call to ib_unregister_device() has returned.
|
||
|
|
|
||
|
|
A low-level driver must call ib_register_device() and
|
||
|
|
ib_unregister_device() from process context. It must not hold any
|
||
|
|
semaphores that could cause deadlock if a consumer calls back into
|
||
|
|
the driver across these calls.
|
||
|
|
|
||
|
|
An upper level protocol consumer may begin using an IB device as
|
||
|
|
soon as the add method of its struct ib_client is called for that
|
||
|
|
device. A consumer must finish all cleanup and free all resources
|
||
|
|
relating to a device before returning from the remove method.
|
||
|
|
|
||
|
|
A consumer is permitted to sleep in its add and remove methods.
|