Adds per cpu exch pool for these reasons:-
1. Currently an EM instance is shared across all cpus to manage
all exches for all cpus. This required em_lock across all
cpus for an exch alloc, free, lookup and reset each frame
and that made em_lock expensive, so instead having per cpu
exch pool with their own per cpu pool lock will likely reduce
locking contention in fast path for an exch alloc, free and
lookup.
2. Per cpu exch pool will likely improve cache hit ratio since
all frames of an exch will be processed on the same cpu on
which exch originated.
This patch is only prep work to help in keeping complexity of next
patch low, so this patch only sets up per cpu exch pool and related
helper funcs to be used by next patch. The next patch fully makes
use of per cpu exch pool in all code paths ie. tx, rx and reset.
Divides per EM exch id range equally across all cpus to setup per
cpu exch pool. This division is such that lower bits of exch id
carries cpu number info on which exch originated, later a simple
bitwise AND operation on exch id of incoming frame with fc_cpu_mask
retrieves cpu number info to direct all frames to same cpu on which
exch originated. This required a global fc_cpu_mask and fc_cpu_order
initialized to max possible cpus number nr_cpu_ids rounded up to 2's
power, this will be used in mapping exch id and exch ptr array
index in pool during exch allocation, find or reset code paths.
Adds a check in fc_exch_mgr_alloc() to ensure specified min_xid
lower bits are zero since these bits are used to carry cpu info.
Adds and initializes struct fc_exch_pool with all required fields
to manage exches in pool.
Allocates per cpu struct fc_exch_pool with memory for exches array
for range of exches per pool. The exches array memory is followed
by struct fc_exch_pool.
Adds fc_exch_ptr_get/set() helper functions to get/set exch ptr in
pool exches array at specified array index.
Increases default FCOE_MAX_XID to 0x0FFF from 0x07EF, so that more
exches are available per cpu after above described exch id range
division across all cpus to each pool.
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Modifies current code to use EM anchor list in EM allocation, EM free,
EM reset, exch allocation and exch lookup code paths.
1. Modifies fc_exch_mgr_alloc to accept EM match function and then
have allocated EM added to the lport using fc_exch_mgr_add API
while also updating EM kref for newly added EM.
2. Updates fc_exch_mgr_free API to accept only lport pointer instead
EM and then have this API free all EMs of the lport from EM anchor
list.
3. Removes single lport pointer link from the EM, which was used in
associating lport pointer in newly allocated exchange. Instead have
lport pointer passed along new exchange allocation call path and
then store passed lport pointer in newly allocated exchange, this
will allow a single EM instance to be used across more than one
lport and used in EM reset to reset only lport specific exchanges.
4. Modifies fc_exch_mgr_reset to reset all EMs from the EM anchor list
of the lport, adds additional exch lport pointer (ep->lp) check for
shared EM case to reset exchange specific to a lport requested reset.
5. Updates exch allocation API fc_exch_alloc to use EM anchor list and
its anchor match func pointer. The fc_exch_alloc will walk the list
of EMs until it finds a match, a match will be either null match
func pointer or call to match function returning true value.
6. Updates fc_exch_recv to accept incoming frame on local port using
only lport pointer and frame pointer without specifying EM instance
of incoming frame. Instead modified fc_exch_recv to locate EM for the
incoming frame by matching xid of incoming frame against a EM xid range.
This change was required to use EM list in libfc Rx path and after this
change the lport fc_exch_mgr pointer emp is not needed anymore, so
removed emp pointer.
7. Updates fnic for removed lport emp pointer and above modified libfc APIs
fc_exch_recv, fc_exch_mgr_alloc and fc_exch_mgr_free.
8. Removes exch_get and exch_put from libfc_function_template as these
are no longer needed with EM anchor list and its match function use.
Also removes its default function fc_exch_get.
A defect this patch introduced regarding the libfc initialization order in
the fnic driver was fixed by Joe Eykholt <jeykholt@cisco.com>.
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Currently there is a 1:1 relationship between the lport
and exchange manager. This macro takes an EM as an argument
and determines the lport from it. However, later patches
will use an EM list per lport, so we will no longer have
this 1:1 relationship- this macro must change.
The FC_EM_DBG macro is rarely used. There are four callers,
two can use FC_LPORT_DBG instead and two can be removed
since they're not necessary. This patch makes those changes
and removes the macro.
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
Adds EM list using a anchor struct fc_exch_mgr_anchor, anchor is used
to allow same EM instance sharing across more than one lport on a eth
device, this implementation is per discussed design posted at
http://www.open-fcoe.org/pipermail/devel/2009-June/002566.html.
The shared EM is required for multiple lports on eth device when
using multiple VLANs or NPIV.
Adds fc_exch_mgr_add API to add a EM to the lport and fc_exch_mgr_del
API to delete previously added EM.
Also adds function fc_exch_mgr_destroy() to destroy allocated EM.
The kref is added to the EM to keep track of EM usage count, the EM is
destroyed when no longer in use upon kref reaching to zero.
The caller can specify match function to fc_exch_mgr_add, this
will be used in determining exchange allocation from its EM or not.
Moved calling of fcoe_em_config below fcoe_libfc_config calling,
so that list head lp->ema_list is initialized before configuring
EM.
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
State RPORT_ST_NONE was intented to be an invalid state (0), never used.
This was a misguided attempt to be sure it was always initialized.
Having an extra state meaning nothing requires switch statements to
have a case covering that state.
State NONE has been used instead to mean the remote port is being deleted.
Changing the name to RPORT_ST_DELETE.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
The state NONE was meant to be invalid, but has been used as
the initial state. Rename it to be DISABLED, as more descriptive.
Further patches will make it the like the RESET state, except
it won't transition to FLOGI until fc_lport_fabric_login() is called.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
libfc debug messages currently show 'lport: <fc-id>:'
wher <fc-id> is the hex assigned port-id. When the lport
is logged off, that will be zero, so its hard to distinguish
which instance is involved. The FC-ID can change
if the port is re-patched or changes VSANs.
Two lports may even have the same FC-ID if connected to isolated SANs.
Change the debug messages to print the SCSI host number "hostN:",
which will not change for the life of the lport.
Still show the FC_ID on lport messages.
Also, add a macro to FC_RPORT_ID_DBG for rport debugging where there's
no rdata structure involved. It takes the lport and port_id as parameters.
Use this in fc_rport_recv_plogi_req() and fc_rport_recv_logo_req().
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This is unlikely to cause any problems, but the libfc debug macros
introduce extra undesirable semicolons.
Signed-off-by: Joe Eykholt <jeykholt@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Signed-off-by: James Bottomley <James.Bottomley@suse.de>
This patch adds the /sys/module/libfc/parameters/debug_logging
file to sysfs as a module parameter. It accepts an integer
bitmask for logging. Currently it supports:
bit
LSB 0 = general libfc debugging
1 = lport debugging
2 = disc debugging
3 = rport debugging
4 = fcp debugging
5 = EM debugging
6 = exch/seq debugging
7 = scsi logging (mostly error handling)
the other bits are not used at this time.
The patch converts all of the libfc source files to use
these new macros and removes the old FC_DBG macro.
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
This allows fnic to configure number of retries for lport and rport
separately.
Signed-off-by: Abhijeet Joglekar <abjoglek@cisco.com>
Acked-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Rogue ports are currently not tracked on any list. The only reference
to them is through any outstanding exchanges pending on the rogue ports.
If the module is removed while a retry is set on a rogue port
(say a Plogi retry for instance), this retry is not cancelled because there
is no reference to the rogue port in the discovery rports list. Thus the
local port can clean itself up, delete the exchange pool, and then the
rogue port timeout can fire and try to start up another exchange.
This patch tracks the rogue ports in a new list disc->rogue_rports. Creating
a new list instead of using the disc->rports list keeps remote port code
change to a minimum.
1) Whenever a rogue port is created, it is immediately added to the
disc->rogue_rports list.
2) When the rogues port goes to ready, it is removed from the rogue list
and the real remote port is added to the disc->rports list
3) The removal of the rogue from the disc->rogue_rports list is done in
the context of the fc_rport_work() workQ thread in discovery callback.
4) Real rports are removed from the disc->rports list like before. Lookup
is done only in the real rports list. This avoids making large changes
to the remote port code.
5) In fc_disc_stop_rports, the rogues list is traversed in addition to the
real list to stop the rogue ports and issue logoffs on them. This way, rogue
ports get cleaned up when the local port goes away.
6) rogue remote ports are not removed from the list right away, but
removed late in fc_rport_work() context, multiple threads can find the same
remote port in the list and call rport_logoff(). Rport_logoff() only
continues with the logoff if port is not in NONE state, thus preventing
multiple logoffs and multiple list deletions.
7) Since the rport is removed from the disc list at a later stage
(in the disc callback), incoming frames can find the rport even if
rport_logoff() has been called on the rport. When rport_logoff() is called,
the rport state is set to NONE, and we are trying to cancel all exchanges
and retries on that port. While in this state, if an incoming
Plogi/Prli/Logo or other frames match the rport, we should not reply
because the rport is in the NONE state. Just drop the frame, since the
rport will be deleted soon in the disc callback (fc_rport_work)
8) In fc_disc_single(), remove rport lookup and call to fc_disc_del_target.
fc_disc_single() is called from recv_rscn_req() where rport lookup
and rport_logoff is already done.
Signed-off-by: Abhijeet Joglekar <abjoglek@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
The foce_softc mem was reserved by libfc_host_alloc as well as
by fcoe_host_alloc.
Removes one liner fcoe_host_alloc completely, instead directly calls
libfc_host_alloc to alloc scsi_host with libfc for just one fcoe_softc
as fcoe private data.
Moves libfc_host_alloc to libfc.h since it is a libfc API, placed
lport_priv API adjacent to libfc_host_alloc since this is related
to scsi_host priv data.
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Remove the hotplug creation of dev_stats, we allocate for all possible CPUs
now when we allocate the lport.
v2: Durring the 2.6.30 merge window, before these patches were comitted,
'percpu_ptr' was renamed 'per_cpu_ptr'. This latest update updates this
patch for the name change.
Signed-off-by: Yi Zou <yi.zou@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
When LLD supports direct data placement (ddp) for large receive of an scsi
i/o coming into fc_fcp, we call into libfc_function_template's ddp_setup()
to prepare for a ddp of large receive for this read I/O. When I/O is complete,
we call the corresponding ddp_done() to get the length of data ddped as well
as to let LLD do clean up.
fc_fcp_ddp_setup()/fc_fcp_ddp_done() are added to setup and complete a ddped
read I/O described by the given fc_fcp_pkt. They would call into corresponding
ddp_setup/ddp_done implemented by the fcoe layer. Eventually, fcoe layer calls
into LLD's ddp_setup/ddp_done provided through net_device
Signed-off-by: Yi Zou <yi.zou@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
This checks if net_devices supports FCoE offload ops in netdev_ops and it
if it does, then sets up the corresponding flags in the associated fc_lport.
For large send offload, the maximum length supported in one large send is now
described by the added lso_max in fc_lport, which is setup initially from
netdev->gso_max_size.
Signed-off-by: Yi Zou <yi.zou@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
1) Added '()' for function names in kerneldoc comments
2) Changed comment bookends from '**/' to '*/'. The comment on the the
mailing list was that '**/' "is consistently unconventional. Not
wrong, just odd." The Documentation/kernel-doc-nano-HOWTO.txt
states that kerneldoc comment blocks should end with '**/' but most
(if not all) instance I found under drivers/scsi/ were only using
the '*/' so I converted to that style.
3) Removed incorrect linebreaks in kerneldoc comments where found
4) Removed a few unnecessary blank comment lines in kerneldoc comment
blocks
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
Made the comments more like the comments for struct scsi_host_template.
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
The fcoe_xmit could call fc_pause in case the pending skb queue len is larger
than FCOE_MAX_QUEUE_DEPTH, the fc_pause was trying to grab lport->lp_muex to
change lport->link_status and that had these issues :-
1. The fcoe_xmit was getting called with bh disabled, thus causing
"BUG: scheduling while atomic" when grabbing lport->lp_muex with bh disabled.
2. fc_linkup and fc_linkdown function calls lport_enter function with
lport->lp_mutex held and these enter function in turn calls fcoe_xmit to send
lport related FC frame, e.g. fc_linkup => fc_lport_enter_flogi to send flogi
req. In this case grabbing the same lport->lp_mutex again in fc_puase from
fcoe_xmit would cause deadlock.
The lport->lp_mutex was used for setting FC_PAUSE in fcoe_xmit path but
FC_PAUSE bit was not used anywhere beside just setting and clear this
bit in lport->link_status, instead used a separate field qfull in fc_lport
to eliminate need for lport->lp_mutex to track pending queue full condition
and in turn avoid above described two locking issues.
Also added check for lp->qfull in fc_fcp_lport_queue_ready to trigger
SCSI_MLQUEUE_HOST_BUSY when lp->qfull is set to prevent more scsi-ml cmds
while lp->qfull is set.
This patch eliminated FC_LINK_UP and FC_PAUSE and instead used dedicated
fields in fc_lport for this, this simplified all related conditional
code.
Also removed fc_pause and fc_unpause functions and instead used newly added
lport->qfull directly in fcoe.
Signed-off-by: Vasu Dev <vasu.dev@intel.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
fc_exch_mgr structure is private to fc_exch.c. To export exch_mgr_reset to
transport, transport needs access to the exch manager. Change
exch_mgr_reset to use lport param which is the shared structure between
libFC and transport.
Alternatively, fc_exch_mgr definition can be moved to libfc.h so that lport
can be accessed from mp*.
Signed-off-by: Abhijeet Joglekar <abjoglek@cisco.com>
Signed-off-by: Robert Love <robert.w.love@intel.com>
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
libFC is composed of 4 blocks supported by an exchange manager
and a framing library. The upper 4 layers are fc_lport, fc_disc,
fc_rport and fc_fcp. A LLD that uses libfc could choose to
either use libfc's block, or using the transport template
defined in libfc.h, override one or more blocks with its own
implementation.
The EM (Exchange Manager) manages exhcanges/sequences for all
commands- ELS, CT and FCP.
The framing library frames ELS and CT commands.
The fc_lport block manages the library's representation of the
host's FC enabled ports.
The fc_disc block manages discovery of targets as well as
handling changes that occur in the FC fabric (via. RSCN events).
The fc_rport block manages the library's representation of other
entities in the FC fabric. Currently the library uses this block
for targets, its peer when in point-to-point mode and the
directory server, but can be extended for other entities if
needed.
The fc_fcp block interacts with the scsi-ml and handles all
I/O.
Signed-off-by: Robert Love <robert.w.love@intel.com>
[jejb: added include of delay.h to fix ppc64 compile prob spotted by sfr]
Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
