fabric(7) @VERSION@ fabric(7)NAMEfabric - Fabric Interface Library
SYNOPSIS
#include <rdma/fabric.h>
Libfabric is a high-performance fabric software library designed to
provide low-latency interfaces to fabric hardware.
OVERVIEW
Libfabric provides 'process direct I/O' to application software commu‐
nicating across fabric software and hardware. Process direct I/O, his‐
torically referred to as RDMA, allows an application to directly access
network resources without operating system interventions. Data trans‐
fers can occur directly to and from application memory.
There are two components to the libfabric software:
Fabric Providers : Conceptually, a fabric provider may be viewed as a
local hardware NIC driver, though a provider is not limited by this
definition. The first component of libfabric is a general purpose
framework that is capable of handling different types of fabric hard‐
ware. All fabric hardware devices and their software drivers are
required to support this framework. Devices and the drivers that plug
into the libfabric framework are referred to as fabric providers, or
simply providers. Provider details may be found in fi_provider(7).
Fabric Interfaces : The second component is a set of communication
operations. Libfabric defines several sets of communication functions
that providers can support. It is not required that providers imple‐
ment all the interfaces that are defined; however, providers clearly
indicate which interfaces they do support.
FABRIC INTERFACES
The fabric interfaces are designed such that they are cohesive and not
simply a union of disjoint interfaces. The interfaces are logically
divided into two groups: control interfaces and communication opera‐
tions. The control interfaces are a common set of operations that pro‐
vide access to local communication resources, such as address vectors
and event queues. The communication operations expose particular mod‐
els of communication and fabric functionality, such as message queues,
remote memory access, and atomic operations. Communication operations
are associated with fabric endpoints.
Applications will typically use the control interfaces to discover
local capabilities and allocate necessary resources. They will then
allocate and configure a communication endpoint to send and receive
data, or perform other types of data transfers, with remote endpoints.
CONTROL INTERFACES
The control interfaces APIs provide applications access to network
resources. This involves listing all the interfaces available, obtain‐
ing the capabilities of the interfaces and opening a provider.
fi_getinfo - Fabric Information : The fi_getinfo call is the base call
used to discover and request fabric services offered by the system.
Applications can use this call to indicate the type of communication
that they desire. The results from fi_getinfo, fi_info, are used to
reserve and configure fabric resources.
fi_getinfo returns a list of fi_info structures. Each structure refer‐
ences a single fabric provider, indicating the interfaces that the
provider supports, along with a named set of resources. A fabric
provider may include multiple fi_info structures in the returned list.
fi_fabric - Fabric Domain : A fabric domain represents a collection of
hardware and software resources that access a single physical or vir‐
tual network. All network ports on a system that can communicate with
each other through the fabric belong to the same fabric domain. A fab‐
ric domain shares network addresses and can span multiple providers.
libfabric supports systems connected to multiple fabrics.
fi_domain - Access Domains : An access domain represents a single logi‐
cal connection into a fabric. It may map to a single physical or vir‐
tual NIC or a port. An access domain defines the boundary across which
fabric resources may be associated. Each access domain belongs to a
single fabric domain.
fi_endpoint - Fabric Endpoint : A fabric endpoint is a communication
portal. An endpoint may be either active or passive. Passive end‐
points are used to listen for connection requests. Active endpoints
can perform data transfers. Endpoints are configured with specific
communication capabilities and data transfer interfaces.
fi_eq - Event Queue : Event queues, are used to collect and report the
completion of asynchronous operations and events. Event queues report
events that are not directly associated with data transfer operations.
fi_cq - Completion Queue : Completion queues are high-performance event
queues used to report the completion of data transfer operations.
fi_cntr - Event Counters : Event counters are used to report the number
of completed asynchronous operations. Event counters are considered
light-weight, in that a completion simply increments a counter, rather
than placing an entry into an event queue.
fi_mr - Memory Region : Memory regions describe application local mem‐
ory buffers. In order for fabric resources to access application mem‐
ory, the application must first grant permission to the fabric provider
by constructing a memory region. Memory regions are required for spe‐
cific types of data transfer operations, such as RMA transfers (see
below).
fi_av - Address Vector : Address vectors are used to map higher level
addresses, such as IP addresses, which may be more natural for an
application to use, into fabric specific addresses. The use of address
vectors allows providers to reduce the amount of memory required to
maintain large address look-up tables, and eliminate expensive address
resolution and look-up methods during data transfer operations.
DATA TRANSFER INTERFACES
Fabric endpoints are associated with multiple data transfer interfaces.
Each interface set is designed to support a specific style of communi‐
cation, with an endpoint allowing the different interfaces to be used
in conjunction. The following data transfer interfaces are defined by
libfabric.
fi_msg - Message Queue : Message queues expose a simple, message-based
FIFO queue interface to the application. Message data transfers allow
applications to send and receive data with message boundaries being
maintained.
fi_tagged - Tagged Message Queues : Tagged message lists expose
send/receive data transfer operations built on the concept of tagged
messaging. The tagged message queue is conceptually similar to stan‐
dard message queues, but with the addition of 64-bit tags for each mes‐
sage. Sent messages are matched with receive buffers that are tagged
with a similar value.
fi_rma - Remote Memory Access : RMA transfers are one-sided operations
that read or write data directly to a remote memory region. Other than
defining the appropriate memory region, RMA operations do not require
interaction at the target side for the data transfer to complete.
fi_atomic - Atomic : Atomic operations can perform one of several oper‐
ations on a remote memory region. Atomic operations include well-known
functionality, such as atomic-add and compare-and-swap, plus several
other pre-defined calls. Unlike other data transfer interfaces, atomic
operations are aware of the data formatting at the target memory
region.
LOGGING INTERFACE
Logging can be controlled using the FI_LOG_LEVEL, FI_LOG_PROV, and
FI_LOG_SUBSYS environment variables.
FI_LOG_LEVEL : FI_LOG_LEVEL controls the amount of logging data that is
output. The following log levels are defined.
· Warn : Warn is the least verbose setting and is intended for report‐
ing errors or warnings.
· Trace : Trace is more verbose and is meant to include non-detailed
output helpful to tracing program execution.
· Info : Info is high traffic and meant for detailed output.
· Debug : Debug is high traffic and is likely to impact application
performance. Debug output is only available if the library has been
compiled with debugging enabled.
FI_LOG_PROV : The FI_LOG_PROV environment variable enables or disables
logging from specific providers. Providers can be enabled by listing
them in a comma separated fashion. If the list begins with the '^'
symbol, then the list will be negated. By default all providers are
enabled.
Example: To enable logging from the psm and sockets provider:
FI_LOG_PROV="psm,sockets"
Example: To enable logging from providers other than psm:
FI_LOG_PROV="^psm"
FI_LOG_SUBSYS : The FI_LOG_SUBSYS environment variable enables or dis‐
ables logging at the subsystem level. The syntax for enabling or dis‐
abling subsystems is similar to that used for FI_LOG_PROV. The follow‐
ing subsystems are defined.
· core : Provides output related to the core framework and its manage‐
ment of providers.
· fabric : Provides output specific to interactions associated with the
fabric object.
· domain : Provides output specific to interactions associated with the
domain object.
· ep_ctrl : Provides output specific to endpoint non-data transfer
operations, such as CM operations.
· ep_data : Provides output specific to endpoint data transfer opera‐
tions.
· av : Provides output specific to address vector operations.
· cq : Provides output specific to completion queue operations.
· eq : Provides output specific to event queue operations.
· mr : Provides output specific to memory registration.
NOTES
Because libfabric is designed to provide applications direct access to
fabric hardware, there are limits on how libfabric resources may be
used in conjunction with system calls. These limitations are notable
for developers who may be familiar programming to the sockets inter‐
face. Although limits are provider specific, the following restric‐
tions apply to many providers and should be adhered to by applications
desiring portability across providers.
fork : Fabric resources are not guaranteed to be available by child
processes. This includes objects, such as endpoints and completion
queues, as well as application controlled data buffers which have been
assigned to the network. For example, data buffers that have been reg‐
istered with a fabric domain may not be available in a child process
because of copy on write restrictions.
SEE ALSOfi_provider(7), fi_getinfo(3), fi_endpoint(3), fi_domain(3), fi_av(3),
fi_eq(3), fi_cq(3), fi_cntr(3), fi_mr(3)AUTHORS
OpenFabrics.
Libfabric Programmer's Manual 2016-06-29 fabric(7)
