From patchwork Wed Apr 26 05:56:19 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "John Daley (johndale)" X-Patchwork-Id: 23889 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 5EC335A6A; Wed, 26 Apr 2017 07:56:43 +0200 (CEST) Received: from alln-iport-5.cisco.com (alln-iport-5.cisco.com [173.37.142.92]) by dpdk.org (Postfix) with ESMTP id 022385A44 for ; Wed, 26 Apr 2017 07:56:40 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=5356; q=dns/txt; s=iport; t=1493186201; x=1494395801; h=from:to:cc:subject:date:message-id; bh=/LDc9x9jsTolI5G/jmPgWjgAtVR8SFMA7/21qgtfzxg=; b=IYEYFKJgqZhm9zNSpehXaL5taOgdZSpdlbsyj0w5+SIVXPVS2BHRL9UD ag6iiFqEOS/EAF2GDbUCWWwXJhA1naI0EGEPvArwbfcmIPfl+ehxYh7C9 S3JaZWlTPXzfrkw0ZiT27uhvrTsXBzyRwXWCNBmTzrtVmTNHHeQcLSxHx 4=; X-IronPort-AV: E=Sophos;i="5.37,253,1488844800"; d="scan'208";a="415987178" Received: from alln-core-1.cisco.com ([173.36.13.131]) by alln-iport-5.cisco.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 26 Apr 2017 05:56:40 +0000 Received: from cisco.com (savbu-usnic-a.cisco.com [10.193.184.48]) by alln-core-1.cisco.com (8.14.5/8.14.5) with ESMTP id v3Q5udfm000388; Wed, 26 Apr 2017 05:56:39 GMT Received: by cisco.com (Postfix, from userid 392789) id B9BB33FAAF2C; Tue, 25 Apr 2017 22:56:39 -0700 (PDT) From: John Daley To: john.mcnamara@intel.com Cc: dev@dpdk.org, John Daley Date: Tue, 25 Apr 2017 22:56:19 -0700 Message-Id: <20170426055619.16452-1-johndale@cisco.com> X-Mailer: git-send-email 2.12.0 Subject: [dpdk-dev] [PATCH] doc: enic SR-IOV configurations X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Document SR-IOV passthrough setup and limitations for enic PMD. Signed-off-by: John Daley --- doc/guides/nics/enic.rst | 94 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) diff --git a/doc/guides/nics/enic.rst b/doc/guides/nics/enic.rst index c535b589c..8f8ae1f9c 100644 --- a/doc/guides/nics/enic.rst +++ b/doc/guides/nics/enic.rst @@ -140,6 +140,83 @@ Masking of these feilds for partial match is also supported. Without advanced filter support, the flow director is limited to IPv4 perfect filtering of the 5-tuple with no masking of fields supported. +SR-IOV mode utilization +----------------------- + +UCS blade servers configured with dynamic vNIC connection policies in UCS +manager are capable of supporting assigned devices on virtual machines (VMs) +through a KVM hypervisor. Assigned devices, also known as 'passthrough' +devices, are SR-IOV virtual functions (VFs) on the host which are exposed +to VM instances. + +The Cisco Virtual Machine Fabric Extender (VM-FEX) gives the VM a dedicated +interface on the Fabric Interconnect (FI). Layer 2 switching is done at +the FI. This may eliminate the requirement for software switching on the +host to route intra-host VM traffic. + +Please refer to `Creating a Dynamic vNIC Connection Policy +`_ +for information on configuring SR-IOV Adapter policies using UCS manager. + +Once the policies are in place and the host OS is reboot, VFs should be visible +on the host, E.g.: + +.. code-block:: c + + # lspci | grep Cisco | grep Ethernet + 0d:00.0 Ethernet controller: Cisco Systems Inc VIC Ethernet NIC (rev a2) + 0d:00.1 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + 0d:00.2 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + 0d:00.3 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + 0d:00.4 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + 0d:00.5 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + 0d:00.6 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + 0d:00.7 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + +Enable Intel IOMMU on the host and install KVM and libvirt. A VM instance should +be created with an assigned device. When using libvirt, this configuration can +be done within the domain (i.e. VM) config file. For example this entry maps +host VF 0d:00:01 into the VM. + +.. code-block:: c + + + + +
+ + +This configuration method is explained in more detail +`here `_. +Alternatively, the configuration can be done in a separate file using the +``network`` keyword. This method is described here: +`https://libvirt.org/formatnetwork.html `_ + +When the VM instance is started, the enic KVM driver will bind the host VF to +vfio, complete provisioning on the FI and bring up the link. + +.. note:: + + It is not possible to use a VF directly from the host because it is not + fully provisioned until the hypervisor brings up the VM that it is assigned + to. + +In the VM instance, the VF will now be visible. E.g., here the VF 00:04.0 is +seen on the VM instance and should be available for binding to a DPDK. + +.. code-block:: c + + # lspci | grep Ether + 00:04.0 Ethernet controller: Cisco Systems Inc VIC SR-IOV VF (rev a2) + +Follow the normal DPDK install proceedure, binding the VF to either ``igb_uio`` +or ``vfio`` in non-IOMMU mode. + +Please see :ref:`Limitations ` for limitations in +the use of SR-IOV. + +.. _enic_limitations: + Limitations ----------- @@ -169,6 +246,23 @@ Limitations - Flow director features are not supported on generation 1 Cisco VIC adapters (M81KR and P81E) +- **SR-IOV** + + - KVM hypervisor support only. VMware has not been tested. + - Requires VM-FEX, and so is only available on UCS managed servers connected + to Fabric Interrconnects. It is not on standalone C-Series servers. + - VF devices are not usable directly from the host. They can only be used + as assigned devices on VM instances. + - Currently, unbind of the enic kernel mode driver 'enic' on the VM instance + may hang. As a workaround, enic.ko should blacklisted or removed from the + boot process. + - pci_generic cannot be used as the uio module in the VM. igb_uio or + vfio in non-IOMMU mode can be used. + - The number of RQs in UCSM dynamic vNIC configurations must be at least 2. + - The number of SR-IOV devices is limited to 256. Components on target system + might limit this number to fewer than 256. + + How to build the suite? ----------------------- The build instructions for the DPDK suite should be followed. By default