From patchwork Thu Mar 9 19:20:46 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Fiona Trahe X-Patchwork-Id: 21665 X-Patchwork-Delegate: pablo.de.lara.guarch@intel.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 5CFD2CFD6; Thu, 9 Mar 2017 20:21:42 +0100 (CET) Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id A5A93CF80 for ; Thu, 9 Mar 2017 20:21:30 +0100 (CET) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 Mar 2017 11:21:08 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos; i="5.36,137,1486454400"; d="scan'208"; a="1120602421" Received: from sivswdev01.ir.intel.com (HELO localhost.localdomain) ([10.237.217.45]) by fmsmga001.fm.intel.com with ESMTP; 09 Mar 2017 11:21:06 -0800 From: Fiona Trahe To: dev@dpdk.org, pablo.de.lara.guarch@intel.com, arkadiuszx.kusztal@intel.com Cc: deepak.k.jain@intel.com, fiona.trahe@intel.com, john.griffin@intel.com Date: Thu, 9 Mar 2017 19:20:46 +0000 Message-Id: <1489087247-456-3-git-send-email-fiona.trahe@intel.com> X-Mailer: git-send-email 1.7.0.7 In-Reply-To: <1487691610-11974-1-git-send-email-fiona.trahe@intel.com> References: <1487691610-11974-1-git-send-email-fiona.trahe@intel.com> Subject: [dpdk-dev] [PATCH v2 2/3] doc: restructure QAT PMD guide X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Restructure QAT PMD instructions and add a device table to minimise duplication for each device and make it easier to add devices. Fix some device name typos. Signed-off-by: Fiona Trahe --- doc/guides/cryptodevs/qat.rst | 370 +++++++++++++++++------------------------- 1 file changed, 148 insertions(+), 222 deletions(-) diff --git a/doc/guides/cryptodevs/qat.rst b/doc/guides/cryptodevs/qat.rst index 9ecd19b..8a31741 100644 --- a/doc/guides/cryptodevs/qat.rst +++ b/doc/guides/cryptodevs/qat.rst @@ -30,9 +30,12 @@ Intel(R) QuickAssist (QAT) Crypto Poll Mode Driver ================================================== -The QAT PMD provides poll mode crypto driver support for **Intel QuickAssist -Technology DH895xxC**, **Intel QuickAssist Technology C62x** and -**Intel QuickAssist Technology C3xxx** hardware accelerator. +The QAT PMD provides poll mode crypto driver support for the following +hardware accelerator devices: + +* ``Intel QuickAssist Technology DH895xCC`` +* ``Intel QuickAssist Technology C62x`` +* ``Intel QuickAssist Technology C3xxx`` Features @@ -84,37 +87,116 @@ Limitations Installation ------------ -To use the DPDK QAT PMD an SRIOV-enabled QAT kernel driver is required. The -VF devices exposed by this driver will be used by QAT PMD. - -To enable QAT in DPDK, follow the instructions mentioned in -http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html +To enable QAT in DPDK, follow the instructions for modifying the compile-time +configuration file as described `here `_. -Quick instructions as follows: +Quick instructions are as follows: .. code-block:: console + cd to the top-level DPDK directory make config T=x86_64-native-linuxapp-gcc sed -i 's,\(CONFIG_RTE_LIBRTE_PMD_QAT\)=n,\1=y,' build/.config make -If you are running on kernel 4.4 or greater, see instructions for -`Installation using kernel.org driver`_ below. If you are on a kernel earlier -than 4.4, see `Installation using 01.org QAT driver`_. +To use the DPDK QAT PMD an SRIOV-enabled QAT kernel driver is required. The +VF devices exposed by this driver will be used by QAT PMD. The devices and +available kernel drivers and device ids are : -For **Intel QuickAssist Technology C62x** and **Intel QuickAssist Technology C3xxx** -device, kernel 4.5 or greater is needed. -See instructions for `Installation using kernel.org driver`_ below. +.. csv-table:: + :header: "device", "driver", "kernel module", "pci driver", "PF DID", "num PFs", "VF DID", "num VFs per PF" + :stub-columns: 1 + DH895xCC,01.org,icp_qa_al, n/a,435,1,443,32 + DH895xCC,4.4+,qat_dh895xcc,dh895xcc,435,1,443,32 + C62x,4.5+,qat_c62x,c6xx,37c8,3,37c9,16 + C3xxx,4.5+,qat_c3xxx,c3xxx,19e2,1,19e3,16 -Installation using 01.org QAT driver + +The ``driver`` column indicates either the linux kernel version in which support +for this device was introduced or a driver available on Intel's 01.org website. +There are both linux and 01.org kernel drivers available for some devices. + + +If you are running on a kernel which includes a driver for your device, see +`Installation using kernel.org driver`_ below. Otherwise see +`Installation using 01.org QAT driver`_. + + +Installation using kernel.org driver ------------------------------------ -NOTE: There is no driver available for **Intel QuickAssist Technology C62x** and -**Intel QuickAssist Technology C3xxx** devices on 01.org. +The examples below are based on the C62x device, if you have a different device +use the corresponding values in the above table. + +In BIOS ensure that SRIOV is enabled and either +a) disable VT-d or +b) enable VT-d and set ``"intel_iommu=on iommu=pt"`` in the grub file. + +Check that the QAT driver is loaded on your system, by executing:: + + lsmod | grep qa + +You should see the kernel module for your device listed, e.g.:: + + qat_c62x 5626 0 + intel_qat 82336 1 qat_c62x + +Next, you need to expose the Virtual Functions (VFs) using the sysfs file system. + +First find the bdfs of the physical functions (PFs) of your device, e.g.:: + + lspci -d : 37c8 + +You should see output similar to:: + + 1a:00.0 Co-processor: Intel Corporation Device 37c8 + 3d:00.0 Co-processor: Intel Corporation Device 37c8 + 3f:00.0 Co-processor: Intel Corporation Device 37c8 + + +Enable the VFs for each PF by echoing the number of VFs per PF to the pci driver:: + + echo 16 > /sys/bus/pci/drivers/c6xx/0000\:1a\:00.0/sriov_numvfs + echo 16 > /sys/bus/pci/drivers/c6xx/0000\:3d\:00.0/sriov_numvfs + echo 16 > /sys/bus/pci/drivers/c6xx/0000\:3f\:00.0/sriov_numvfs + + +Check that the VFs are available for use - e.g. +``lspci -d:37c9`` should list 48 VF devices available for a ``C62x`` device. + +To complete the installation - follow instructions in `Binding the available VFs to the DPDK UIO driver`_. + +**Notes**: + +If the QAT kernel modules are not loaded and you see an error like +``Failed to load MMP firmware qat_895xcc_mmp.bin`` in kernel logs, this may be +as a result of not using a distribution, but just updating the kernel directly. + +Download firmware from the kernel firmware repo at: +http://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git/tree/ + +Copy qat binaries to /lib/firmware: + +* ``cp qat_895xcc.bin /lib/firmware`` +* ``cp qat_895xcc_mmp.bin /lib/firmware`` + +cd to your linux source root directory and start the qat kernel modules: + +* ``insmod ./drivers/crypto/qat/qat_common/intel_qat.ko`` +* ``insmod ./drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko`` + + +If you see the following warning in /var/log/messages it can be ignored: +``IOMMU should be enabled for SR-IOV to work correctly`` + + + +Installation using 01.org QAT driver +------------------------------------ Download the latest QuickAssist Technology Driver from `01.org -`_ +`_. Consult the *Getting Started Guide* at the same URL for further information. The steps below assume you are: @@ -142,11 +224,13 @@ Build and install the SRIOV-enabled QAT driver:: ./installer.sh install QAT1.6 host You can use ``cat /proc/icp_dh895xcc_dev0/version`` to confirm the driver is correctly installed. -You can use ``lspci -d:443`` to confirm the bdf of the 32 VF devices are available per ``DH895xCC`` device. +You can use ``lspci -d:443`` to confirm the bdf of the 32 VF devices available per ``DH895xCC`` device. To complete the installation - follow instructions in `Binding the available VFs to the DPDK UIO driver`_. -**Note**: If using a later kernel and the build fails with an error relating to ``strict_stroul`` not being available apply the following patch: +**Notes**: + +If using a later kernel and the build fails with an error relating to ``strict_stroul`` not being available apply the following patch: .. code-block:: diff @@ -186,230 +270,72 @@ If the build or install fails due to mismatching kernel sources you may need to * ``sudo yum install kernel-devel-`uname -r``` -Installation using kernel.org driver ------------------------------------- - -For **Intel QuickAssist Technology DH895xxC**: - -Assuming you are running on at least a 4.4 kernel, you can use the stock kernel.org QAT -driver to start the QAT hardware. - -The steps below assume you are: - -* Running DPDK on a platform with one ``DH895xCC`` device. -* On a kernel at least version 4.4. - -In BIOS ensure that SRIOV is enabled and either -a) disable VT-d or -b) enable VT-d and set ``"intel_iommu=on iommu=pt"`` in the grub file. - -Ensure the QAT driver is loaded on your system, by executing:: - - lsmod | grep qat - -You should see the following output:: - - qat_dh895xcc 5626 0 - intel_qat 82336 1 qat_dh895xcc - -Next, you need to expose the Virtual Functions (VFs) using the sysfs file system. - -First find the bdf of the physical function (PF) of the DH895xCC device:: - - lspci -d : 435 - -You should see output similar to:: - - 03:00.0 Co-processor: Intel Corporation Coleto Creek PCIe Endpoint - -Using the sysfs, enable the VFs:: - - echo 32 > /sys/bus/pci/drivers/dh895xcc/0000\:03\:00.0/sriov_numvfs - -If you get an error, it's likely you're using a QAT kernel driver earlier than kernel 4.4. - -To verify that the VFs are available for use - use ``lspci -d:443`` to confirm -the bdf of the 32 VF devices are available per ``DH895xCC`` device. - -To complete the installation - follow instructions in `Binding the available VFs to the DPDK UIO driver`_. - -**Note**: If the QAT kernel modules are not loaded and you see an error like - ``Failed to load MMP firmware qat_895xcc_mmp.bin`` this may be as a - result of not using a distribution, but just updating the kernel directly. - -Download firmware from the kernel firmware repo at: -http://git.kernel.org/cgit/linux/kernel/git/firmware/linux-firmware.git/tree/ - -Copy qat binaries to /lib/firmware: -* ``cp qat_895xcc.bin /lib/firmware`` -* ``cp qat_895xcc_mmp.bin /lib/firmware`` - -cd to your linux source root directory and start the qat kernel modules: -* ``insmod ./drivers/crypto/qat/qat_common/intel_qat.ko`` -* ``insmod ./drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko`` - -**Note**:The following warning in /var/log/messages can be ignored: - ``IOMMU should be enabled for SR-IOV to work correctly`` - -For **Intel QuickAssist Technology C62x**: -Assuming you are running on at least a 4.5 kernel, you can use the stock kernel.org QAT -driver to start the QAT hardware. - -The steps below assume you are: - -* Running DPDK on a platform with one ``C62x`` device. -* On a kernel at least version 4.5. - -In BIOS ensure that SRIOV is enabled and either -a) disable VT-d or -b) enable VT-d and set ``"intel_iommu=on iommu=pt"`` in the grub file. - -Ensure the QAT driver is loaded on your system, by executing:: - - lsmod | grep qat - -You should see the following output:: - - qat_c62x 16384 0 - intel_qat 122880 1 qat_c62x - -Next, you need to expose the VFs using the sysfs file system. - -First find the bdf of the C62x device:: - - lspci -d:37c8 - -You should see output similar to:: - - 1a:00.0 Co-processor: Intel Corporation Device 37c8 - 3d:00.0 Co-processor: Intel Corporation Device 37c8 - 3f:00.0 Co-processor: Intel Corporation Device 37c8 - -For each c62x device there are 3 PFs. -Using the sysfs, for each PF, enable the 16 VFs:: - - echo 16 > /sys/bus/pci/drivers/c6xx/0000\:1a\:00.0/sriov_numvfs - -If you get an error, it's likely you're using a QAT kernel driver earlier than kernel 4.5. - -To verify that the VFs are available for use - use ``lspci -d:37c9`` to confirm -the bdf of the 48 VF devices are available per ``C62x`` device. - -To complete the installation - follow instructions in `Binding the available VFs to the DPDK UIO driver`_. - -For **Intel QuickAssist Technology C3xxx**: -Assuming you are running on at least a 4.5 kernel, you can use the stock kernel.org QAT -driver to start the QAT hardware. - -The steps below assume you are: - -* Running DPDK on a platform with one ``C3xxx`` device. -* On a kernel at least version 4.5. - -In BIOS ensure that SRIOV is enabled and either -a) disable VT-d or -b) enable VT-d and set ``"intel_iommu=on iommu=pt"`` in the grub file. - -Ensure the QAT driver is loaded on your system, by executing:: - - lsmod | grep qat - -You should see the following output:: - - qat_c3xxx 16384 0 - intel_qat 122880 1 qat_c3xxx - -Next, you need to expose the Virtual Functions (VFs) using the sysfs file system. - -First find the bdf of the physical function (PF) of the C3xxx device - - lspci -d:19e2 - -You should see output similar to:: - - 01:00.0 Co-processor: Intel Corporation Device 19e2 - -For c3xxx device there is 1 PFs. -Using the sysfs, enable the 16 VFs:: - - echo 16 > /sys/bus/pci/drivers/c3xxx/0000\:01\:00.0/sriov_numvfs - -If you get an error, it's likely you're using a QAT kernel driver earlier than kernel 4.5. - -To verify that the VFs are available for use - use ``lspci -d:19e3`` to confirm -the bdf of the 16 VF devices are available per ``C3xxx`` device. -To complete the installation - follow instructions in `Binding the available VFs to the DPDK UIO driver`_. Binding the available VFs to the DPDK UIO driver ------------------------------------------------ -For **Intel(R) QuickAssist Technology DH895xcc** device: -The unbind command below assumes ``bdfs`` of ``03:01.00-03:04.07``, if yours are different adjust the unbind command below:: - cd $RTE_SDK - modprobe uio - insmod ./build/kmod/igb_uio.ko +Unbind the VFs from the stock driver so they can be bound to the uio driver. - for device in $(seq 1 4); do \ - for fn in $(seq 0 7); do \ - echo -n 0000:03:0${device}.${fn} > \ - /sys/bus/pci/devices/0000\:03\:0${device}.${fn}/driver/unbind; \ - done; \ - done +For **Intel(R) QuickAssist Technology DH895xCC** device: - echo "8086 0443" > /sys/bus/pci/drivers/igb_uio/new_id +The unbind command below assumes ``bdfs`` of ``03:01.00-03:04.07``, if your VFs +are different adjust the unbind command below:: -You can use ``lspci -vvd:443`` to confirm that all devices are now in use by igb_uio kernel driver. + for device in $(seq 1 4); do \ + for fn in $(seq 0 7); do \ + echo -n 0000:03:0${device}.${fn} > \ + /sys/bus/pci/devices/0000\:03\:0${device}.${fn}/driver/unbind; \ + done; \ + done For **Intel(R) QuickAssist Technology C62x** device: -The unbind command below assumes ``bdfs`` of ``1a:01.00-1a:02.07``, ``3d:01.00-3d:02.07`` and ``3f:01.00-3f:02.07``, -if yours are different adjust the unbind command below:: - - cd $RTE_SDK - modprobe uio - insmod ./build/kmod/igb_uio.ko - for device in $(seq 1 2); do \ - for fn in $(seq 0 7); do \ - echo -n 0000:1a:0${device}.${fn} > \ - /sys/bus/pci/devices/0000\:1a\:0${device}.${fn}/driver/unbind; \ +The unbind command below assumes ``bdfs`` of ``1a:01.00-1a:02.07``, +``3d:01.00-3d:02.07`` and ``3f:01.00-3f:02.07``, if your VFs are different adjust +the unbind command below:: - echo -n 0000:3d:0${device}.${fn} > \ - /sys/bus/pci/devices/0000\:3d\:0${device}.${fn}/driver/unbind; \ + for device in $(seq 1 2); do \ + for fn in $(seq 0 7); do \ + echo -n 0000:1a:0${device}.${fn} > \ + /sys/bus/pci/devices/0000\:1a\:0${device}.${fn}/driver/unbind; \ - echo -n 0000:3f:0${device}.${fn} > \ - /sys/bus/pci/devices/0000\:3f\:0${device}.${fn}/driver/unbind; \ - done; \ - done + echo -n 0000:3d:0${device}.${fn} > \ + /sys/bus/pci/devices/0000\:3d\:0${device}.${fn}/driver/unbind; \ - echo "8086 37c9" > /sys/bus/pci/drivers/igb_uio/new_id - -You can use ``lspci -vvd:37c9`` to confirm that all devices are now in use by igb_uio kernel driver. + echo -n 0000:3f:0${device}.${fn} > \ + /sys/bus/pci/devices/0000\:3f\:0${device}.${fn}/driver/unbind; \ + done; \ + done For **Intel(R) QuickAssist Technology C3xxx** device: -The unbind command below assumes ``bdfs`` of ``01:01.00-01:02.07``, -if yours are different adjust the unbind command below:: - cd $RTE_SDK - modprobe uio - insmod ./build/kmod/igb_uio.ko - - for device in $(seq 1 2); do \ - for fn in $(seq 0 7); do \ - echo -n 0000:01:0${device}.${fn} > \ - /sys/bus/pci/devices/0000\:01\:0${device}.${fn}/driver/unbind; \ +The unbind command below assumes ``bdfs`` of ``01:01.00-01:02.07``, +if your VFs are different adjust the unbind command below:: - done; \ - done + for device in $(seq 1 2); do \ + for fn in $(seq 0 7); do \ + echo -n 0000:01:0${device}.${fn} > \ + /sys/bus/pci/devices/0000\:01\:0${device}.${fn}/driver/unbind; \ + done; \ + done - echo "8086 19e3" > /sys/bus/pci/drivers/igb_uio/new_id +**Bind to the DPDK uio driver.** +Install the DPDK igb_uio driver, bind the VF PCI Device id to it and use lspci +to confirm the VF devices are now in use by igb_uio kernel driver, +e.g. for the C62x device:: + -You can use ``lspci -vvd:19e3`` to confirm that all devices are now in use by igb_uio kernel driver. + cd to the top-level DPDK directory + modprobe uio + insmod ./build/kmod/igb_uio.ko + echo "8086 37c9" > /sys/bus/pci/drivers/igb_uio/new_id + lspci -vvd:37c9 -The other way to bind the VFs to the DPDK UIO driver is by using the ``dpdk-devbind.py`` script: +Another way to bind the VFs to the DPDK UIO driver is by using the ``dpdk-devbind.py`` script: .. code-block:: console - cd $RTE_SDK + cd to the top-level DPDK directory ./usertools/dpdk-devbind.py -b igb_uio 0000:03:01.1