From patchwork Thu Apr 27 00:52:10 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "John Daley (johndale)" X-Patchwork-Id: 23929 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 319D92C01; Thu, 27 Apr 2017 02:52:20 +0200 (CEST) Received: from alln-iport-5.cisco.com (alln-iport-5.cisco.com [173.37.142.92]) by dpdk.org (Postfix) with ESMTP id 9F7FC2C01 for ; Thu, 27 Apr 2017 02:52:18 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=6095; q=dns/txt; s=iport; t=1493254338; x=1494463938; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Qf0xJhBmTjS0sAUqzKGUu67HajiBqrzQcMPcm/UWZRM=; b=K/ABduqmf/hNuv1nPBP/Ic9+wzRFxrRngryBVI2bi4C2P2hOBlo550Tj cyLxu98JEVag0+KcMAWnwkFMYFhSUxehDDjnVIe1PJtZIYSBbF88lfBQr 1DBAvFtpFatUQ/2UO6+V0rEu3PgEbRSxosemo+H4mwCDbwbCdFv3Bb0D6 A=; X-IronPort-AV: E=Sophos;i="5.37,256,1488844800"; d="scan'208";a="416531598" Received: from rcdn-core-4.cisco.com ([173.37.93.155]) by alln-iport-5.cisco.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Apr 2017 00:52:17 +0000 Received: from cisco.com (savbu-usnic-a.cisco.com [10.193.184.48]) by rcdn-core-4.cisco.com (8.14.5/8.14.5) with ESMTP id v3R0qHnO012129; Thu, 27 Apr 2017 00:52:17 GMT Received: by cisco.com (Postfix, from userid 392789) id 4C8273FAAF2F; Wed, 26 Apr 2017 17:52:17 -0700 (PDT) From: John Daley To: ferruh.yigit@intel.com, john.mcnamara@intel.com Cc: dev@dpdk.org, John Daley Date: Wed, 26 Apr 2017 17:52:10 -0700 Message-Id: <20170427005210.22440-1-johndale@cisco.com> X-Mailer: git-send-email 2.12.0 In-Reply-To: <20170426055619.16452-1-johndale@cisco.com> References: <20170426055619.16452-1-johndale@cisco.com> Subject: [dpdk-dev] [PATCH v2] doc: enic SR-IOV configurations X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Document SR-IOV passthrough setup and limitations for enic PMD. Signed-off-by: John Daley Acked-by: John McNamara --- v2 - fix typos, URL targets and formatting in SR-IOV section. doc/guides/nics/enic.rst | 93 +++++++++++++++++++++++++++++++++++++++ doc/guides/nics/features/enic.ini | 1 + 2 files changed, 94 insertions(+) diff --git a/doc/guides/nics/enic.rst b/doc/guides/nics/enic.rst index c535b589c..f744151f5 100644 --- a/doc/guides/nics/enic.rst +++ b/doc/guides/nics/enic.rst @@ -140,6 +140,81 @@ Masking of these feilds for partial match is also supported. Without advanced filter support, the flow director is limited to IPv4 perfect filtering of the 5-tuple with no masking of fields supported. +SR-IOV mode utilization +----------------------- + +UCS blade servers configured with dynamic vNIC connection policies in UCS +manager are capable of supporting assigned devices on virtual machines (VMs) +through a KVM hypervisor. Assigned devices, also known as 'passthrough' +devices, are SR-IOV virtual functions (VFs) on the host which are exposed +to VM instances. + +The Cisco Virtual Machine Fabric Extender (VM-FEX) gives the VM a dedicated +interface on the Fabric Interconnect (FI). Layer 2 switching is done at +the FI. This may eliminate the requirement for software switching on the +host to route intra-host VM traffic. + +Please refer to `Creating a Dynamic vNIC Connection Policy +`_ +for information on configuring SR-IOV Adapter policies using UCS manager. + +Once the policies are in place and the host OS is rebooted, VFs should be +visible on the host, E.g.: + +.. code-block:: console + + # lspci | grep Cisco | grep Ethernet + 0d:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2) + 0d:00.1 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + 0d:00.2 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + 0d:00.3 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + 0d:00.4 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + 0d:00.5 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + 0d:00.6 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + 0d:00.7 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + +Enable Intel IOMMU on the host and install KVM and libvirt. A VM instance should +be created with an assigned device. When using libvirt, this configuration can +be done within the domain (i.e. VM) config file. For example this entry maps +host VF 0d:00:01 into the VM. + +.. code-block:: console + + + + +
+ + +Alternatively, the configuration can be done in a separate file using the +``network`` keyword. These methods are described in the libvirt documentation for +`Network XML format `_. + +When the VM instance is started, the enic KVM driver will bind the host VF to +vfio, complete provisioning on the FI and bring up the link. + +.. note:: + + It is not possible to use a VF directly from the host because it is not + fully provisioned until the hypervisor brings up the VM that it is assigned + to. + +In the VM instance, the VF will now be visible. E.g., here the VF 00:04.0 is +seen on the VM instance and should be available for binding to a DPDK. + +.. code-block:: console + + # lspci | grep Ether + 00:04.0 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + +Follow the normal DPDK install procedure, binding the VF to either ``igb_uio`` +or ``vfio`` in non-IOMMU mode. + +Please see :ref:`Limitations ` for limitations in +the use of SR-IOV. + +.. _enic_limitations: + Limitations ----------- @@ -169,6 +244,23 @@ Limitations - Flow director features are not supported on generation 1 Cisco VIC adapters (M81KR and P81E) +- **SR-IOV** + + - KVM hypervisor support only. VMware has not been tested. + - Requires VM-FEX, and so is only available on UCS managed servers connected + to Fabric Interconnects. It is not on standalone C-Series servers. + - VF devices are not usable directly from the host. They can only be used + as assigned devices on VM instances. + - Currently, unbind of the enic kernel mode driver 'enic' on the VM instance + may hang. As a workaround, enic.ko should blacklisted or removed from the + boot process. + - pci_generic cannot be used as the uio module in the VM. igb_uio or + vfio in non-IOMMU mode can be used. + - The number of RQs in UCSM dynamic vNIC configurations must be at least 2. + - The number of SR-IOV devices is limited to 256. Components on target system + might limit this number to fewer than 256. + + How to build the suite? ----------------------- The build instructions for the DPDK suite should be followed. By default @@ -216,6 +308,7 @@ Supported features - IPV4, IPV6 and TCP RSS hashing - Scattered Rx - MTU update +- SR-IOV on UCS managed servers connected to Fabric Interconnects. Known bugs and Unsupported features in this release --------------------------------------------------- diff --git a/doc/guides/nics/features/enic.ini b/doc/guides/nics/features/enic.ini index 51b330fa8..94e7f3cba 100644 --- a/doc/guides/nics/features/enic.ini +++ b/doc/guides/nics/features/enic.ini @@ -15,6 +15,7 @@ Promiscuous mode = Y Unicast MAC filter = Y Multicast MAC filter = Y RSS hash = Y +SR-IOV = Y VLAN filter = Y CRC offload = Y VLAN offload = Y