forked from luck/tmp_suning_uos_patched
97162a1ee8
The InfiniBand docs are plain text with no markups. So, all we needed to do were to add the title markups and some markup sequences in order to properly parse tables, lists and literal blocks. At its new index.rst, let's add a :orphan: while this is not linked to the main index.rst file, in order to avoid build warnings. Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org> Signed-off-by: Jason Gunthorpe <jgg@mellanox.com>
116 lines
4.4 KiB
ReStructuredText
116 lines
4.4 KiB
ReStructuredText
==================
|
|
IP over InfiniBand
|
|
==================
|
|
|
|
The ib_ipoib driver is an implementation of the IP over InfiniBand
|
|
protocol as specified by RFC 4391 and 4392, issued by the IETF ipoib
|
|
working group. It is a "native" implementation in the sense of
|
|
setting the interface type to ARPHRD_INFINIBAND and the hardware
|
|
address length to 20 (earlier proprietary implementations
|
|
masqueraded to the kernel as ethernet interfaces).
|
|
|
|
Partitions and P_Keys
|
|
=====================
|
|
|
|
When the IPoIB driver is loaded, it creates one interface for each
|
|
port using the P_Key at index 0. To create an interface with a
|
|
different P_Key, write the desired P_Key into the main interface's
|
|
/sys/class/net/<intf name>/create_child file. For example::
|
|
|
|
echo 0x8001 > /sys/class/net/ib0/create_child
|
|
|
|
This will create an interface named ib0.8001 with P_Key 0x8001. To
|
|
remove a subinterface, use the "delete_child" file::
|
|
|
|
echo 0x8001 > /sys/class/net/ib0/delete_child
|
|
|
|
The P_Key for any interface is given by the "pkey" file, and the
|
|
main interface for a subinterface is in "parent."
|
|
|
|
Child interface create/delete can also be done using IPoIB's
|
|
rtnl_link_ops, where children created using either way behave the same.
|
|
|
|
Datagram vs Connected modes
|
|
===========================
|
|
|
|
The IPoIB driver supports two modes of operation: datagram and
|
|
connected. The mode is set and read through an interface's
|
|
/sys/class/net/<intf name>/mode file.
|
|
|
|
In datagram mode, the IB UD (Unreliable Datagram) transport is used
|
|
and so the interface MTU has is equal to the IB L2 MTU minus the
|
|
IPoIB encapsulation header (4 bytes). For example, in a typical IB
|
|
fabric with a 2K MTU, the IPoIB MTU will be 2048 - 4 = 2044 bytes.
|
|
|
|
In connected mode, the IB RC (Reliable Connected) transport is used.
|
|
Connected mode takes advantage of the connected nature of the IB
|
|
transport and allows an MTU up to the maximal IP packet size of 64K,
|
|
which reduces the number of IP packets needed for handling large UDP
|
|
datagrams, TCP segments, etc and increases the performance for large
|
|
messages.
|
|
|
|
In connected mode, the interface's UD QP is still used for multicast
|
|
and communication with peers that don't support connected mode. In
|
|
this case, RX emulation of ICMP PMTU packets is used to cause the
|
|
networking stack to use the smaller UD MTU for these neighbours.
|
|
|
|
Stateless offloads
|
|
==================
|
|
|
|
If the IB HW supports IPoIB stateless offloads, IPoIB advertises
|
|
TCP/IP checksum and/or Large Send (LSO) offloading capability to the
|
|
network stack.
|
|
|
|
Large Receive (LRO) offloading is also implemented and may be turned
|
|
on/off using ethtool calls. Currently LRO is supported only for
|
|
checksum offload capable devices.
|
|
|
|
Stateless offloads are supported only in datagram mode.
|
|
|
|
Interrupt moderation
|
|
====================
|
|
|
|
If the underlying IB device supports CQ event moderation, one can
|
|
use ethtool to set interrupt mitigation parameters and thus reduce
|
|
the overhead incurred by handling interrupts. The main code path of
|
|
IPoIB doesn't use events for TX completion signaling so only RX
|
|
moderation is supported.
|
|
|
|
Debugging Information
|
|
=====================
|
|
|
|
By compiling the IPoIB driver with CONFIG_INFINIBAND_IPOIB_DEBUG set
|
|
to 'y', tracing messages are compiled into the driver. They are
|
|
turned on by setting the module parameters debug_level and
|
|
mcast_debug_level to 1. These parameters can be controlled at
|
|
runtime through files in /sys/module/ib_ipoib/.
|
|
|
|
CONFIG_INFINIBAND_IPOIB_DEBUG also enables files in the debugfs
|
|
virtual filesystem. By mounting this filesystem, for example with::
|
|
|
|
mount -t debugfs none /sys/kernel/debug
|
|
|
|
it is possible to get statistics about multicast groups from the
|
|
files /sys/kernel/debug/ipoib/ib0_mcg and so on.
|
|
|
|
The performance impact of this option is negligible, so it
|
|
is safe to enable this option with debug_level set to 0 for normal
|
|
operation.
|
|
|
|
CONFIG_INFINIBAND_IPOIB_DEBUG_DATA enables even more debug output in
|
|
the data path when data_debug_level is set to 1. However, even with
|
|
the output disabled, enabling this configuration option will affect
|
|
performance, because it adds tests to the fast path.
|
|
|
|
References
|
|
==========
|
|
|
|
Transmission of IP over InfiniBand (IPoIB) (RFC 4391)
|
|
http://ietf.org/rfc/rfc4391.txt
|
|
|
|
IP over InfiniBand (IPoIB) Architecture (RFC 4392)
|
|
http://ietf.org/rfc/rfc4392.txt
|
|
|
|
IP over InfiniBand: Connected Mode (RFC 4755)
|
|
http://ietf.org/rfc/rfc4755.txt
|