From patchwork Sat Nov 4 10:08:02 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stephen Hemminger X-Patchwork-Id: 31172 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id EE5261B5E0; Sat, 4 Nov 2017 11:08:06 +0100 (CET) Received: from mail-pg0-f66.google.com (mail-pg0-f66.google.com [74.125.83.66]) by dpdk.org (Postfix) with ESMTP id B66811B5E0 for ; Sat, 4 Nov 2017 11:08:04 +0100 (CET) Received: by mail-pg0-f66.google.com with SMTP id b192so4495946pga.2 for ; Sat, 04 Nov 2017 03:08:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=WOU9bnqFBcw9X0WLJLJ49ve+DsOtFKpzI15f0DNOVVg=; b=FWGZBVkDYT/O4SfFu9h+j+CqURskZvDIoHcTFj2NYI3rmbX13AiIofmBQzrs+60vx2 93nQ2krEm8z9mggeLeRuj+1jqBhpwEgjRTXcXgngdC7vrvs2PK5vynjWboXFJZkyZVqg VyC1RBDQE45jp580mBX4MbhS8kzXR9Ish7I9M3/qzJYmtjegKhxfbos3foql7P+8Rol7 BUvADz3srgxE6KMOzyWMxIkda56toSAe6jF475KWTnrJda7Nxjx0/gjwZNQJTY8QHY5F mVjoY+TnTii8I63geIbbbb6b/sBBG05M0nq82oE/XWKXYsDLjkMzdOcw7JHKjkKHK0vz c1/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=WOU9bnqFBcw9X0WLJLJ49ve+DsOtFKpzI15f0DNOVVg=; b=cMaA+NKJQJ+v1A49ksmZrRmuXboRUMPtjicPgbBG7mDVrujO8Uc8nABH8p+PDWgEmo 2mVBj0YucKPT23NCyllrMI2PyWU4Dv3wNx9mec2Y51LWNlNX0cfPl4i3KeWSTJc0s0BX K1IPq+50xgS06Fe140euOBDGEAEc5SGCtpfbyzNT3U6bbjTy/5u2YpE1xeW6ep6zvp4i m4yNykmeEewxQxAbnwId7QUsA238ivpkGzfMIoq8WlvRrilFjqrY/Ql45jIcLWU6C8tN j/1PSEKdDG1vNVVpaXHMjUKXd+e+SytMEoppFI0hOEymn8hiROE278RCMW8Drz3VY9yB GOOw== X-Gm-Message-State: AMCzsaUFN5LE9izWTnKgpUXmSF9kKpHx6MORo19K/pWS96nkDtijYHQ5 mNeUzgXj8QXicrWo3Rh3jjY2NBkctiFEP1mWJ0LmJg== X-Google-Smtp-Source: ABhQp+QAo7V3o4RNV3rBKLM+C2AgKhJlgjn5i2Df/J2KeP9j1GPvGNgry6qoq5s/OyAgN8PCADoRmiS06ZysyyZZ7iQ= X-Received: by 10.98.36.135 with SMTP id k7mr10342652pfk.63.1509790083746; Sat, 04 Nov 2017 03:08:03 -0700 (PDT) MIME-Version: 1.0 Received: by 10.100.181.108 with HTTP; Sat, 4 Nov 2017 03:08:02 -0700 (PDT) Received: by 10.100.181.108 with HTTP; Sat, 4 Nov 2017 03:08:02 -0700 (PDT) In-Reply-To: References: <20171103003005.44339-1-ferruh.yigit@intel.com> <20171103005100.44633-1-ferruh.yigit@intel.com> From: Stephen Hemminger Date: Sat, 4 Nov 2017 11:08:02 +0100 Message-ID: To: Ferruh Yigit Cc: "Roberts, Lee A." , "Tan, Jianfeng" , Thomas Monjalon , "dev@dpdk.org" , "stable@dpdk.org" , "Wu, Jingjing" , Shijith Thotton , Gregory Etelson , Harish Patil , George Prekas , "Gonzalez Monroy, Sergio" , Rasesh Mody X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH v2] igb_uio: add config option to control reset X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Nov 4, 2017 01:03, "Ferruh Yigit" wrote: On 11/3/2017 12:42 PM, Roberts, Lee A. wrote: > > >> -----Original Message----- >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Tan, Jianfeng >> Sent: Thursday, November 02, 2017 8:57 PM >> To: Ferruh Yigit ; Thomas Monjalon < thomas@monjalon.net> >> Cc: dev@dpdk.org; stable@dpdk.org; Jingjing Wu ; Shijith Thotton >> ; Gregory Etelson ; Harish Patil >> ; George Prekas ; Sergio Gonzalez Monroy >> ; Rasesh Mody >> Subject: Re: [dpdk-dev] [PATCH v2] igb_uio: add config option to control reset >> >> >> >> On 11/3/2017 8:51 AM, Ferruh Yigit wrote: >>> Adding a compile time configuration option to control device reset done >>> during DPDK application exit. >>> >>> Config option is CONFIG_RTE_EAL_IGB_UIO_RESET and enabled by default, >>> so by default reset will happen. Having this reset is safer to be sure >>> device left in a proper case. >>> >>> But for special cases [1] it is possible to disable the config option >>> to prevent the device reset. >>> >>> [1] >>> http://dpdk.org/ml/archives/dev/2017-November/080927.html >>> >>> Fixes: b58eedfc7dd5 ("igb_uio: issue FLR during open and release of device file") >>> Cc: stable@dpdk.org >>> >>> Signed-off-by: Ferruh Yigit >> >> Realize that we do have a pci_clear_master() in the release() to disable >> the DMA from device until the next open() will enable the DMA again . >> Here is my: >> >> Reviewed-by: Jianfeng Tan >> >> Thanks, >> Jianfeng >> >>> --- >>> Cc: Jianfeng Tan >>> Cc: Jingjing Wu >>> Cc: Shijith Thotton >>> Cc: Gregory Etelson >>> Cc: Harish Patil >>> Cc: George Prekas >>> Cc: Sergio Gonzalez Monroy >>> Cc: Rasesh Mody >>> >>> v2: >>> * fix typo in commit log >>> --- >>> config/common_base | 1 + >>> config/common_linuxapp | 1 + >>> lib/librte_eal/linuxapp/igb_uio/igb_uio.c | 2 ++ >>> 3 files changed, 4 insertions(+) >>> >>> diff --git a/config/common_base b/config/common_base >>> index 82ee75456..2a9947420 100644 >>> --- a/config/common_base >>> +++ b/config/common_base >>> @@ -102,6 +102,7 @@ CONFIG_RTE_LIBEAL_USE_HPET=n >>> CONFIG_RTE_EAL_ALLOW_INV_SOCKET_ID=n >>> CONFIG_RTE_EAL_ALWAYS_PANIC_ON_ERROR=n >>> CONFIG_RTE_EAL_IGB_UIO=n >>> +CONFIG_RTE_EAL_IGB_UIO_RESET=n >>> CONFIG_RTE_EAL_VFIO=n >>> CONFIG_RTE_MALLOC_DEBUG=n >>> CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=n >>> diff --git a/config/common_linuxapp b/config/common_linuxapp >>> index 74c7d64ec..b3a602909 100644 >>> --- a/config/common_linuxapp >>> +++ b/config/common_linuxapp >>> @@ -37,6 +37,7 @@ CONFIG_RTE_EXEC_ENV_LINUXAPP=y >>> >>> CONFIG_RTE_EAL_NUMA_AWARE_HUGEPAGES=y >>> CONFIG_RTE_EAL_IGB_UIO=y >>> +CONFIG_RTE_EAL_IGB_UIO_RESET=y >>> CONFIG_RTE_EAL_VFIO=y >>> CONFIG_RTE_KNI_KMOD=y >>> CONFIG_RTE_LIBRTE_KNI=y >>> diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c >>> index fd320d87d..0325722c0 100644 >>> --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c >>> +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c >>> @@ -360,7 +360,9 @@ igbuio_pci_release(struct uio_info *info, struct inode *inode) >>> /* stop the device from further DMA */ >>> pci_clear_master(dev); >>> >>> +#ifdef RTE_EAL_IGB_UIO_RESET >>> pci_reset_function(dev); >>> +#endif >>> >>> return 0; >>> } > > A compile time configuration option makes life very difficult for application providers. > > Consider the case where an application such as Open vSwitch with DPDK support is being provided > with a Linux distribution. One would want the Open vSwitch binary to support as many vendor NICs > as possible---without the need to recompile. With a change such as this, one would need to have > different versions of the kernel igb_uio module to support different NICs. Agreed, I am against adding more compile time options although I am end up sending a few of them these days. > The Linux kernel is already aware of, and provides work-arounds for, various PCI quirks. > For example, see linux/drivers/pci/quirks.c (http://elixir.free-electrons. com/linux/latest/source/drivers/pci/quirks.c). > > At this point in igb_uio.c, one is aware of the struct pci_dev "dev" for the device in question. > Access to the vendor and device information should be simple: > > struct pci_dev { > struct list_head bus_list;/* node in per-bus list */ > struct pci_bus*bus;/* bus this device is on */ > struct pci_bus*subordinate;/* bus this device bridges to */ > > void*sysdata;/* hook for sys-specific extension */ > struct proc_dir_entry *procent;/* device entry in /proc/bus/pci */ > struct pci_slot*slot;/* Physical slot this device is in */ > > unsigned intdevfn;/* encoded device & function index */ > unsigned shortvendor; > unsigned shortdevice; > unsigned shortsubsystem_vendor; > unsigned shortsubsystem_device; > ... > > One could imagine using logic to implement corresponding PCI quirks that can be evaluated > at runtime. For example (in pseudocode), > > if not (vendor = "Cavium" and device = "bnx2x") > then pci_reset_function(dev); It wouldn't be nice to add device specific checks into generic igb_uio module, but also it is not nice to add compile time option, comparing two I would be OK to device checks. What do you think about following? If there is no objection and Rasesh confirms that patch is working, I can send a proper patch for it. Personal preference is for more concise: static book is_device_excluded(const struct pci_dev *pdev) { return pci_match_id(no_reset_pci_tbl, pdev); } static int igbuio_pci_release(struct uio_info *info, struct inode *inode) { @@ -360,7 +368,8 @@ igbuio_pci_release(struct uio_info *info, struct inode *inode) /* stop the device from further DMA */ pci_clear_master(dev); - pci_reset_function(dev); + if (!is_device_excluded_from_reset(dev)) + pci_reset_function(dev); return 0; } > > There are other possible implementations. If there are enough quirks, one might have action > functions defined---and a table of function pointers associated with each PMD to select the > proper action. > > - Lee Roberts > > > > diff --git a/lib/librte_eal/linuxapp/igb_uio/compat.h b/lib/librte_eal/linuxapp/igb_uio/compat.h index 30508f35c..264206af3 100644 --- a/lib/librte_eal/linuxapp/igb_uio/compat.h +++ b/lib/librte_eal/linuxapp/igb_uio/compat.h @@ -134,3 +134,21 @@ static bool pci_check_and_mask_intx(struct pci_dev *pdev) #endif +#define BROADCOM_PCI_VENDOR_ID 0x14E4 +static const struct pci_device_id no_reset_pci_tbl[] = { + { PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x168a) }, /* 57800 */ + { PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x164f) }, /* 57711 */ + { PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x168e) }, /* 57810 */ + { PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x163d) }, /* 57811 */ + { PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x168d) }, /* 57840_OBS */ + { PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16a1) }, /* 57840_4_10 */ + { PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16a2) }, /* 57840_2_20 */ + { PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16ae) }, /* 57810_MF */ + { PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x163e) }, /* 57811_MF */ + { PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16a4) }, /* 57840_MF */ + { PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16a9) }, /* 57800_VF */ + { PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16af) }, /* 57810_VF */ + { PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x163f) }, /* 57811_VF */ + { PCI_DEVICE(BROADCOM_PCI_VENDOR_ID, 0x16ad) }, /* 57840_VF */ + { 0 }, +}; diff --git a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c index fd320d87d..b0d92b51e 100644 --- a/lib/librte_eal/linuxapp/igb_uio/igb_uio.c +++ b/lib/librte_eal/linuxapp/igb_uio/igb_uio.c @@ -348,6 +348,14 @@ igbuio_pci_open(struct uio_info *info, struct inode *inode) return 0; } +static int is_device_excluded_from_reset(struct pci_dev *pdev) +{ + if (pci_match_id(no_reset_pci_tbl, pdev)) + return 1; + + return 0; +} +