From patchwork Fri Jun 22 11:51:05 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Jia" X-Patchwork-Id: 41389 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 566B91BB32; Fri, 22 Jun 2018 13:48:09 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 9DBB51BB15 for ; Fri, 22 Jun 2018 13:48:05 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Jun 2018 04:48:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,256,1526367600"; d="scan'208";a="49078344" Received: from jeffguo-s2600wt2.sh.intel.com (HELO localhost.localdomain) ([10.67.110.10]) by fmsmga007.fm.intel.com with ESMTP; 22 Jun 2018 04:48:02 -0700 From: Jeff Guo To: stephen@networkplumber.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, konstantin.ananyev@intel.com, gaetan.rivet@6wind.com, jingjing.wu@intel.com, thomas@monjalon.net, motih@mellanox.com, matan@mellanox.com, harry.van.haaren@intel.com, qi.z.zhang@intel.com, shaopeng.he@intel.com Cc: jblunck@infradead.org, shreyansh.jain@nxp.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com Date: Fri, 22 Jun 2018 19:51:05 +0800 Message-Id: <1529668268-7462-2-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1529668268-7462-1-git-send-email-jia.guo@intel.com> References: <1498711073-42917-1-git-send-email-jia.guo@intel.com> <1529668268-7462-1-git-send-email-jia.guo@intel.com> Subject: [dpdk-dev] [PATCH v2 1/4] bus/pci: handle device hot unplug X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When a hardware device is removed physically or the software disables it, the hot unplug occur. App need to call ether dev API to detach the device, to unplug the device at the bus level and make access to the device invalid. But the problem is that, the removal of the device from the software lists is not going to be instantaneous, at this time if the data path still read/write the device, it will cause MMIO error and result of the app crash out. So a hot unplug handle mechanism need to guaranty app will not crash out when hot unplug device. To handle device hot unplug is bus-specific behavior, this patch introduces a bus ops so that each kind of bus can implement its own logic. Further, this patch implements the ops for PCI bus: remap a dummy memory to avoid bus read/write error. Signed-off-by: Jeff Guo --- v2->v1(v21): refind commit log --- drivers/bus/pci/pci_common.c | 65 +++++++++++++++++++++++++++++++++ drivers/bus/pci/pci_common_uio.c | 33 +++++++++++++++++ drivers/bus/pci/private.h | 12 ++++++ lib/librte_eal/common/include/rte_bus.h | 16 ++++++++ 4 files changed, 126 insertions(+) diff --git a/drivers/bus/pci/pci_common.c b/drivers/bus/pci/pci_common.c index 7215aae..74d9aa8 100644 --- a/drivers/bus/pci/pci_common.c +++ b/drivers/bus/pci/pci_common.c @@ -472,6 +472,70 @@ pci_find_device(const struct rte_device *start, rte_dev_cmp_t cmp, return NULL; } +static struct rte_pci_device * +pci_find_device_by_addr(void *failure_addr) +{ + struct rte_pci_device *pdev = NULL; + int i; + + FOREACH_DEVICE_ON_PCIBUS(pdev) { + for (i = 0; i != RTE_DIM(pdev->mem_resource); i++) { + if ((uint64_t)(uintptr_t)failure_addr >= + (uint64_t)(uintptr_t)pdev->mem_resource[i].addr && + (uint64_t)(uintptr_t)failure_addr < + (uint64_t)(uintptr_t)pdev->mem_resource[i].addr + + pdev->mem_resource[i].len) { + RTE_LOG(ERR, EAL, "Failure address " + "%16.16"PRIx64" belongs to " + "device %s!\n", + (uint64_t)(uintptr_t)failure_addr, + pdev->device.name); + return pdev; + } + } + } + return NULL; +} +static int +pci_handle_hot_unplug(struct rte_device *dev, void *failure_addr) +{ + struct rte_pci_device *pdev = NULL; + int ret = 0; + + if (dev != NULL) + pdev = RTE_DEV_TO_PCI(dev); + else + pdev = pci_find_device_by_addr(failure_addr); + + if (!pdev) + return -1; + + /* remap resources for devices */ + switch (pdev->kdrv) { + case RTE_KDRV_VFIO: +#ifdef VFIO_PRESENT + /* TODO */ + ret = -1; +#endif + break; + case RTE_KDRV_IGB_UIO: + case RTE_KDRV_UIO_GENERIC: + case RTE_KDRV_NIC_UIO: + ret = pci_uio_remap_resource(pdev); + break; + default: + RTE_LOG(DEBUG, EAL, + "Not managed by a supported kernel driver, skipped\n"); + ret = -1; + break; + } + + if (ret != 0) + RTE_LOG(ERR, EAL, "Failed to handle hot unplug of device %s", + pdev->name); + return ret; +} + static int pci_plug(struct rte_device *dev) { @@ -502,6 +566,7 @@ struct rte_pci_bus rte_pci_bus = { .unplug = pci_unplug, .parse = pci_parse, .get_iommu_class = rte_pci_get_iommu_class, + .handle_hot_unplug = pci_handle_hot_unplug, }, .device_list = TAILQ_HEAD_INITIALIZER(rte_pci_bus.device_list), .driver_list = TAILQ_HEAD_INITIALIZER(rte_pci_bus.driver_list), diff --git a/drivers/bus/pci/pci_common_uio.c b/drivers/bus/pci/pci_common_uio.c index 54bc20b..7ea73db 100644 --- a/drivers/bus/pci/pci_common_uio.c +++ b/drivers/bus/pci/pci_common_uio.c @@ -146,6 +146,39 @@ pci_uio_unmap(struct mapped_pci_resource *uio_res) } } +/* remap the PCI resource of a PCI device in anonymous virtual memory */ +int +pci_uio_remap_resource(struct rte_pci_device *dev) +{ + int i; + void *map_address; + + if (dev == NULL) + return -1; + + /* Remap all BARs */ + for (i = 0; i != PCI_MAX_RESOURCE; i++) { + /* skip empty BAR */ + if (dev->mem_resource[i].phys_addr == 0) + continue; + map_address = mmap(dev->mem_resource[i].addr, + (size_t)dev->mem_resource[i].len, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0); + if (map_address == MAP_FAILED) { + RTE_LOG(ERR, EAL, + "Cannot remap resource for device %s\n", + dev->name); + return -1; + } + RTE_LOG(INFO, EAL, + "Successful remap resource for device %s\n", + dev->name); + } + + return 0; +} + static struct mapped_pci_resource * pci_uio_find_resource(struct rte_pci_device *dev) { diff --git a/drivers/bus/pci/private.h b/drivers/bus/pci/private.h index 88fa587..5551506 100644 --- a/drivers/bus/pci/private.h +++ b/drivers/bus/pci/private.h @@ -173,6 +173,18 @@ void pci_uio_free_resource(struct rte_pci_device *dev, struct mapped_pci_resource *uio_res); /** + * Remap the PCI resource of a PCI device in anonymous virtual memory. + * + * @param dev + * Point to the struct rte pci device. + * @return + * - On success, zero. + * - On failure, a negative value. + */ +int +pci_uio_remap_resource(struct rte_pci_device *dev); + +/** * Map device memory to uio resource * * This function is private to EAL. diff --git a/lib/librte_eal/common/include/rte_bus.h b/lib/librte_eal/common/include/rte_bus.h index eb9eded..6a5609f 100644 --- a/lib/librte_eal/common/include/rte_bus.h +++ b/lib/librte_eal/common/include/rte_bus.h @@ -168,6 +168,20 @@ typedef int (*rte_bus_unplug_t)(struct rte_device *dev); typedef int (*rte_bus_parse_t)(const char *name, void *addr); /** + * Implementation a specific hot unplug handler, which is responsible + * for handle the failure when hot unplug the device, guaranty the system + * would not hung in the case. + * @param dev + * Pointer of the device structure. + * + * @return + * 0 on success. + * !0 on error. + */ +typedef int (*rte_bus_handle_hot_unplug_t)(struct rte_device *dev, + void *dev_addr); + +/** * Bus scan policies */ enum rte_bus_scan_mode { @@ -209,6 +223,8 @@ struct rte_bus { rte_bus_plug_t plug; /**< Probe single device for drivers */ rte_bus_unplug_t unplug; /**< Remove single device from driver */ rte_bus_parse_t parse; /**< Parse a device name */ + rte_bus_handle_hot_unplug_t handle_hot_unplug; /**< handle hot unplug + device event */ struct rte_bus_conf conf; /**< Bus configuration */ rte_bus_get_iommu_class_t get_iommu_class; /**< Get iommu class */ }; From patchwork Fri Jun 22 11:51:06 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Jia" X-Patchwork-Id: 41390 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0600E1BB4D; Fri, 22 Jun 2018 13:48:12 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 6C60E1BB2C for ; Fri, 22 Jun 2018 13:48:08 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Jun 2018 04:48:07 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,256,1526367600"; d="scan'208";a="49078353" Received: from jeffguo-s2600wt2.sh.intel.com (HELO localhost.localdomain) ([10.67.110.10]) by fmsmga007.fm.intel.com with ESMTP; 22 Jun 2018 04:48:04 -0700 From: Jeff Guo To: stephen@networkplumber.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, konstantin.ananyev@intel.com, gaetan.rivet@6wind.com, jingjing.wu@intel.com, thomas@monjalon.net, motih@mellanox.com, matan@mellanox.com, harry.van.haaren@intel.com, qi.z.zhang@intel.com, shaopeng.he@intel.com Cc: jblunck@infradead.org, shreyansh.jain@nxp.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com Date: Fri, 22 Jun 2018 19:51:06 +0800 Message-Id: <1529668268-7462-3-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1529668268-7462-1-git-send-email-jia.guo@intel.com> References: <1498711073-42917-1-git-send-email-jia.guo@intel.com> <1529668268-7462-1-git-send-email-jia.guo@intel.com> Subject: [dpdk-dev] [PATCH v2 2/4] eal: add failure handle mechanism for hot plug X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This patch introduces a failure handler mechanism to handle device hot unplug event. When device be hot plug out, the device resource become invalid, if this resource is still be unexpected read/write, system will crash. This patch let framework help application to handle this fault. When sigbus error occur, check the failure address and accordingly remap the invalid memory for the corresponding device, that could guaranty the application not to be shut down when hot unplug devices. Signed-off-by: Jeff Guo --- v2->v1(v21): refine commit log --- lib/librte_eal/linuxapp/eal/eal_dev.c | 154 +++++++++++++++++++++++++++++++++- 1 file changed, 153 insertions(+), 1 deletion(-) diff --git a/lib/librte_eal/linuxapp/eal/eal_dev.c b/lib/librte_eal/linuxapp/eal/eal_dev.c index 1cf6aeb..3067f39 100644 --- a/lib/librte_eal/linuxapp/eal/eal_dev.c +++ b/lib/librte_eal/linuxapp/eal/eal_dev.c @@ -4,6 +4,8 @@ #include #include +#include +#include #include #include @@ -14,15 +16,27 @@ #include #include #include +#include +#include +#include #include "eal_private.h" static struct rte_intr_handle intr_handle = {.fd = -1 }; static bool monitor_started; +extern struct rte_bus_list rte_bus_list; + #define EAL_UEV_MSG_LEN 4096 #define EAL_UEV_MSG_ELEM_LEN 128 +/* spinlock for device failure process */ +static rte_spinlock_t dev_failure_lock = RTE_SPINLOCK_INITIALIZER; + +static struct sigaction sigbus_action_old; + +static int sigbus_need_recover; + static void dev_uev_handler(__rte_unused void *param); /* identify the system layer which reports this event. */ @@ -34,6 +48,93 @@ enum eal_dev_event_subsystem { }; static int +dev_uev_failure_process(struct rte_device *dev, void *dev_addr) +{ + struct rte_bus *bus; + int ret = 0; + + if (!dev && !dev_addr) { + return -EINVAL; + } else if (dev) { + bus = rte_bus_find_by_device_name(dev->name); + if (bus->handle_hot_unplug) { + /** + * call bus ops to handle hot unplug. + */ + ret = bus->handle_hot_unplug(dev, dev_addr); + if (ret) { + RTE_LOG(ERR, EAL, + "Cannot handle hot unplug " + "for device %s " + "on the bus %s.\n ", + dev->name, bus->name); + } + } else { + RTE_LOG(ERR, EAL, + "Not support handle hot unplug for bus %s!\n", + bus->name); + ret = -ENOTSUP; + } + } else { + TAILQ_FOREACH(bus, &rte_bus_list, next) { + if (bus->handle_hot_unplug) { + /** + * call bus ops to handle hot unplug. + */ + ret = bus->handle_hot_unplug(dev, dev_addr); + if (ret) + RTE_LOG(ERR, EAL, + "Cannot handle hot unplug " + "for the device " + "on the bus %s!\n", bus->name); + else + break; + } else { + RTE_LOG(ERR, EAL, + "Not support handle hot unplug " + "for bus %s!\n", bus->name); + ret = -ENOTSUP; + } + } + } + return ret; +} + +static void +sigbus_action_recover(void) +{ + if (sigbus_need_recover) { + sigaction(SIGBUS, &sigbus_action_old, NULL); + sigbus_need_recover = 0; + } +} + +static void sigbus_handler(int signum __rte_unused, siginfo_t *info, + void *ctx __rte_unused) +{ + int ret; + + RTE_LOG(DEBUG, EAL, "Thread[%d] catch SIGBUS, fault address:%p\n", + (int)pthread_self(), info->si_addr); + rte_spinlock_lock(&dev_failure_lock); + ret = dev_uev_failure_process(NULL, info->si_addr); + rte_spinlock_unlock(&dev_failure_lock); + if (!ret) + RTE_LOG(DEBUG, EAL, + "Success to handle SIGBUS error for hot unplug!\n"); + else + rte_exit(EXIT_FAILURE, "exit for SIGBUS error!"); +} + +static int cmp_dev_name(const struct rte_device *dev, + const void *_name) +{ + const char *name = _name; + + return strcmp(dev->name, name); +} + +static int dev_uev_socket_fd_create(void) { struct sockaddr_nl addr; @@ -147,6 +248,9 @@ dev_uev_handler(__rte_unused void *param) struct rte_dev_event uevent; int ret; char buf[EAL_UEV_MSG_LEN]; + struct rte_bus *bus; + struct rte_device *dev; + const char *busname; memset(&uevent, 0, sizeof(struct rte_dev_event)); memset(buf, 0, EAL_UEV_MSG_LEN); @@ -171,13 +275,50 @@ dev_uev_handler(__rte_unused void *param) RTE_LOG(DEBUG, EAL, "receive uevent(name:%s, type:%d, subsystem:%d)\n", uevent.devname, uevent.type, uevent.subsystem); - if (uevent.devname) + switch (uevent.subsystem) { + case EAL_DEV_EVENT_SUBSYSTEM_PCI: + case EAL_DEV_EVENT_SUBSYSTEM_UIO: + busname = "pci"; + break; + default: + break; + } + + if (uevent.devname) { + if (uevent.type == RTE_DEV_EVENT_REMOVE) { + bus = rte_bus_find_by_name(busname); + if (bus == NULL) { + RTE_LOG(ERR, EAL, "Cannot find bus (%s)\n", + uevent.devname); + return; + } + dev = bus->find_device(NULL, cmp_dev_name, + uevent.devname); + if (dev == NULL) { + RTE_LOG(ERR, EAL, + "Cannot find unplugged device (%s)\n", + uevent.devname); + return; + } + rte_spinlock_lock(&dev_failure_lock); + ret = dev_uev_failure_process(dev, NULL); + rte_spinlock_unlock(&dev_failure_lock); + if (ret) { + RTE_LOG(ERR, EAL, "Driver cannot remap the " + "device (%s)\n", + dev->name); + return; + } + } dev_callback_process(uevent.devname, uevent.type); + } } int __rte_experimental rte_dev_event_monitor_start(void) { + sigset_t mask; + struct sigaction action; int ret; if (monitor_started) @@ -197,6 +338,14 @@ rte_dev_event_monitor_start(void) return -1; } + /* register sigbus handler */ + sigemptyset(&mask); + sigaddset(&mask, SIGBUS); + action.sa_flags = SA_SIGINFO; + action.sa_mask = mask; + action.sa_sigaction = sigbus_handler; + sigbus_need_recover = !sigaction(SIGBUS, &action, &sigbus_action_old); + monitor_started = true; return 0; @@ -217,8 +366,11 @@ rte_dev_event_monitor_stop(void) return ret; } + sigbus_action_recover(); + close(intr_handle.fd); intr_handle.fd = -1; monitor_started = false; + return 0; } From patchwork Fri Jun 22 11:51:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Guo, Jia" X-Patchwork-Id: 41391 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6F5E71BB69; Fri, 22 Jun 2018 13:48:12 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 9F2CA1BB3A for ; Fri, 22 Jun 2018 13:48:10 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Jun 2018 04:48:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,256,1526367600"; d="scan'208";a="49078357" Received: from jeffguo-s2600wt2.sh.intel.com (HELO localhost.localdomain) ([10.67.110.10]) by fmsmga007.fm.intel.com with ESMTP; 22 Jun 2018 04:48:07 -0700 From: Jeff Guo To: stephen@networkplumber.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, konstantin.ananyev@intel.com, gaetan.rivet@6wind.com, jingjing.wu@intel.com, thomas@monjalon.net, motih@mellanox.com, matan@mellanox.com, harry.van.haaren@intel.com, qi.z.zhang@intel.com, shaopeng.he@intel.com Cc: jblunck@infradead.org, shreyansh.jain@nxp.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com Date: Fri, 22 Jun 2018 19:51:07 +0800 Message-Id: <1529668268-7462-4-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1529668268-7462-1-git-send-email-jia.guo@intel.com> References: <1498711073-42917-1-git-send-email-jia.guo@intel.com> <1529668268-7462-1-git-send-email-jia.guo@intel.com> Subject: [dpdk-dev] [PATCH v2 3/4] igb_uio: fix uio release issue when hot unplug X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When hot unplug device, the kernel will release the device resource in the kernel side, such as the fd sys file will disappear, and the irq will be released. At this time, if igb uio driver still try to release this resource, it will cause kernel crash. On the other hand, something like interrupt disabling do not automatically process in kernel side. If not handler it, this redundancy and dirty thing will affect the interrupt resource be used by other device. So the igb_uio driver have to check the hot plug status, and the corresponding process should be taken in igb uio driver. This patch propose to add structure of rte_udev_state into rte_uio_pci_dev of igb_uio kernel driver, which will record the state of uio device, such as probed/opened/released/removed/unplug. When detect the unexpected removal which cause of hot unplug behavior, it will corresponding disable interrupt resource, while for the part of releasement which kernel have already handle, just skip it to avoid double free or null pointer kernel crash issue. Signed-off-by: Jeff Guo --- v2->v1(v21): add uio device state to check hot plug unexpected removmal. fix igb uio kernel driver issue. --- kernel/linux/igb_uio/igb_uio.c | 50 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 45 insertions(+), 5 deletions(-) diff --git a/kernel/linux/igb_uio/igb_uio.c b/kernel/linux/igb_uio/igb_uio.c index cd9b7e7..fdd692a 100644 --- a/kernel/linux/igb_uio/igb_uio.c +++ b/kernel/linux/igb_uio/igb_uio.c @@ -19,6 +19,15 @@ #include "compat.h" +/* uio pci device state */ +enum rte_udev_state { + RTE_UDEV_PROBED, + RTE_UDEV_OPENNED, + RTE_UDEV_RELEASED, + RTE_UDEV_REMOVED, + RTE_UDEV_UNPLUG +}; + /** * A structure describing the private information for a uio device. */ @@ -28,6 +37,7 @@ struct rte_uio_pci_dev { enum rte_intr_mode mode; struct mutex lock; int refcnt; + enum rte_udev_state state; }; static char *intr_mode; @@ -194,12 +204,20 @@ igbuio_pci_irqhandler(int irq, void *dev_id) { struct rte_uio_pci_dev *udev = (struct rte_uio_pci_dev *)dev_id; struct uio_info *info = &udev->info; + struct pci_dev *pdev = udev->pdev; /* Legacy mode need to mask in hardware */ if (udev->mode == RTE_INTR_MODE_LEGACY && !pci_check_and_mask_intx(udev->pdev)) return IRQ_NONE; + /* check the uevent of the kobj */ + if ((&pdev->dev.kobj)->state_remove_uevent_sent == 1) { + dev_notice(&pdev->dev, "device:%s, sent remove uevent!\n", + (&pdev->dev.kobj)->name); + udev->state = RTE_UDEV_UNPLUG; + } + uio_event_notify(info); /* Message signal mode, no share IRQ and automasked */ @@ -308,7 +326,6 @@ igbuio_pci_disable_interrupts(struct rte_uio_pci_dev *udev) #endif } - /** * This gets called while opening uio device file. */ @@ -330,24 +347,33 @@ igbuio_pci_open(struct uio_info *info, struct inode *inode) /* enable interrupts */ err = igbuio_pci_enable_interrupts(udev); - mutex_unlock(&udev->lock); if (err) { dev_err(&dev->dev, "Enable interrupt fails\n"); + pci_clear_master(dev); return err; } + udev->state = RTE_UDEV_OPENNED; + mutex_unlock(&udev->lock); return 0; } +/** + * This gets called while closing uio device file. + */ static int igbuio_pci_release(struct uio_info *info, struct inode *inode) { + struct rte_uio_pci_dev *udev = info->priv; struct pci_dev *dev = udev->pdev; + if (udev->state == RTE_UDEV_REMOVED) + return 0; + mutex_lock(&udev->lock); if (--udev->refcnt > 0) { mutex_unlock(&udev->lock); - return 0; + return -1; } /* disable interrupts */ @@ -355,7 +381,7 @@ igbuio_pci_release(struct uio_info *info, struct inode *inode) /* stop the device from further DMA */ pci_clear_master(dev); - + udev->state = RTE_UDEV_RELEASED; mutex_unlock(&udev->lock); return 0; } @@ -557,6 +583,7 @@ igbuio_pci_probe(struct pci_dev *dev, const struct pci_device_id *id) (unsigned long long)map_dma_addr, map_addr); } + udev->state = RTE_UDEV_PROBED; return 0; fail_remove_group: @@ -573,11 +600,24 @@ igbuio_pci_probe(struct pci_dev *dev, const struct pci_device_id *id) static void igbuio_pci_remove(struct pci_dev *dev) { + struct rte_uio_pci_dev *udev = pci_get_drvdata(dev); + int ret; + + /* handler hot unplug */ + if (udev->state == RTE_UDEV_OPENNED || + udev->state == RTE_UDEV_UNPLUG) { + dev_notice(&dev->dev, "Unexpected removal!\n"); + ret = igbuio_pci_release(&udev->info, NULL); + if (ret) + return; + udev->state = RTE_UDEV_REMOVED; + return; + } mutex_destroy(&udev->lock); - sysfs_remove_group(&dev->dev.kobj, &dev_attr_grp); uio_unregister_device(&udev->info); + sysfs_remove_group(&dev->dev.kobj, &dev_attr_grp); igbuio_pci_release_iomem(&udev->info); pci_disable_device(dev); pci_set_drvdata(dev, NULL); From patchwork Fri Jun 22 11:51:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Guo, Jia" X-Patchwork-Id: 41392 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 9609F1BB71; Fri, 22 Jun 2018 13:48:14 +0200 (CEST) Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 22E101BB4F for ; Fri, 22 Jun 2018 13:48:12 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Jun 2018 04:48:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,256,1526367600"; d="scan'208";a="49078365" Received: from jeffguo-s2600wt2.sh.intel.com (HELO localhost.localdomain) ([10.67.110.10]) by fmsmga007.fm.intel.com with ESMTP; 22 Jun 2018 04:48:10 -0700 From: Jeff Guo To: stephen@networkplumber.org, bruce.richardson@intel.com, ferruh.yigit@intel.com, konstantin.ananyev@intel.com, gaetan.rivet@6wind.com, jingjing.wu@intel.com, thomas@monjalon.net, motih@mellanox.com, matan@mellanox.com, harry.van.haaren@intel.com, qi.z.zhang@intel.com, shaopeng.he@intel.com Cc: jblunck@infradead.org, shreyansh.jain@nxp.com, dev@dpdk.org, jia.guo@intel.com, helin.zhang@intel.com Date: Fri, 22 Jun 2018 19:51:08 +0800 Message-Id: <1529668268-7462-5-git-send-email-jia.guo@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1529668268-7462-1-git-send-email-jia.guo@intel.com> References: <1498711073-42917-1-git-send-email-jia.guo@intel.com> <1529668268-7462-1-git-send-email-jia.guo@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v2 4/4] app/testpmd: show example to handle hot unplug X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Use testpmd for example, to show how an application smoothly handle failure when device being hot unplug. If app have enabled the device event monitor and register the hot plug event’s callback before running, once app detect the removal event, the callback would be called. It will first stop the packet forwarding, then stop the port, close the port, and finally detach the port to remove the device out from the device lists. Signed-off-by: Jeff Guo Acked-by: Bernard Iremonger --- v2->v1(v21): rebase testpmd code --- app/test-pmd/testpmd.c | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index 24c1998..286f242 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -1951,9 +1951,10 @@ eth_dev_event_callback_unregister(void) void attach_port(char *identifier) { - portid_t pi = 0; unsigned int socket_id; + portid_t pi = rte_eth_dev_count_avail(); + printf("Attaching a new port...\n"); if (identifier == NULL) { @@ -2125,16 +2126,25 @@ check_all_ports_link_status(uint32_t port_mask) static void rmv_event_callback(void *arg) { + struct rte_eth_dev *dev; + int need_to_start = 0; int org_no_link_check = no_link_check; portid_t port_id = (intptr_t)arg; RTE_ETH_VALID_PORTID_OR_RET(port_id); + dev = &rte_eth_devices[port_id]; + + if (dev->state == RTE_ETH_DEV_UNUSED) + return; + + printf("removing device %s\n", dev->device->name); if (!test_done && port_is_forwarding(port_id)) { need_to_start = 1; stop_packet_forwarding(); } + no_link_check = 1; stop_port(port_id); no_link_check = org_no_link_check; @@ -2196,6 +2206,9 @@ static void eth_dev_event_callback(char *device_name, enum rte_dev_event_type type, __rte_unused void *arg) { + uint16_t port_id; + int ret; + if (type >= RTE_DEV_EVENT_MAX) { fprintf(stderr, "%s called upon invalid event %d\n", __func__, type); @@ -2206,9 +2219,12 @@ eth_dev_event_callback(char *device_name, enum rte_dev_event_type type, case RTE_DEV_EVENT_REMOVE: RTE_LOG(ERR, EAL, "The device: %s has been removed!\n", device_name); - /* TODO: After finish failure handle, begin to stop - * packet forward, stop port, close port, detach port. - */ + ret = rte_eth_dev_get_port_by_name(device_name, &port_id); + if (ret) { + printf("can not get port by device %s!\n", device_name); + return; + } + rmv_event_callback((void *)(intptr_t)port_id); break; case RTE_DEV_EVENT_ADD: RTE_LOG(ERR, EAL, "The device: %s has been added!\n", @@ -2736,7 +2752,6 @@ main(int argc, char** argv) return -1; } eth_dev_event_callback_register(); - } if (start_port(RTE_PORT_ALL) != 0)