From patchwork Fri Sep 3 12:41:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Harman Kalra X-Patchwork-Id: 97935 X-Patchwork-Delegate: david.marchand@redhat.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B65ADA0C54; Fri, 3 Sep 2021 14:42:32 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A53B5410DC; Fri, 3 Sep 2021 14:42:32 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 5297F40E3C for ; Fri, 3 Sep 2021 14:42:31 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 1837tMpt012027; Fri, 3 Sep 2021 05:42:30 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : subject : date : message-id : in-reply-to : references : mime-version : content-type; s=pfpt0220; bh=veU8F+KNPOa8eUCuLRnEJSdTCC2EzK5thxPz+nsSEPs=; b=HO+zhQQ9axhcmEMZ7w17PyQOpOmN34tJJQnvLvHEPDdJdzLLP03j7j09Ek5sYRuDzY8X fjKceIYkM10SqZvvDCejD+kZ9UsZrS2f5IZvTSI8zv8BJVWDOAOPTtTkjGzjfVFSCmFz uXWTbCIt8tm4GQpXmMZHaa2SeOuyUVzixR/5sTSTZQZK7kfCrdnfDcGqewlsNIzsQYbJ gMxByt+AXb8q2DjYI+ZWNHoHSHiO9TXOeF5dAytw0IjRMj/NeHWEzUKEe+PwjdafB8pE Py3uw+S1q/HA809DMes+wKOlBI6BqxOkQq5gmhwtKaWDD1fLfefQSFFM+cJfkWX25CpO mQ== Received: from dc5-exch01.marvell.com ([199.233.59.181]) by mx0b-0016f401.pphosted.com with ESMTP id 3aufr890w8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Fri, 03 Sep 2021 05:42:30 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Fri, 3 Sep 2021 05:42:28 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.18 via Frontend Transport; Fri, 3 Sep 2021 05:42:28 -0700 Received: from localhost.localdomain (unknown [10.29.52.211]) by maili.marvell.com (Postfix) with ESMTP id 5FD435B6942; Fri, 3 Sep 2021 05:42:27 -0700 (PDT) From: Harman Kalra To: , Anatoly Burakov , Harman Kalra Date: Fri, 3 Sep 2021 18:11:01 +0530 Message-ID: <20210903124102.47425-7-hkalra@marvell.com> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20210903124102.47425-1-hkalra@marvell.com> References: <20210826145726.102081-1-hkalra@marvell.com> <20210903124102.47425-1-hkalra@marvell.com> MIME-Version: 1.0 X-Proofpoint-ORIG-GUID: DfbgGWheBvX2hHb29tiMi48_OUzz1yF1 X-Proofpoint-GUID: DfbgGWheBvX2hHb29tiMi48_OUzz1yF1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475 definitions=2021-09-03_03,2021-09-03_01,2020-04-07_01 Subject: [dpdk-dev] [PATCH v1 6/7] eal/interrupts: make interrupt handle structure opaque X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Moving interrupt handle structure definition inside the c file to make its fields totally opaque to the outside world. Dynamically allocating the efds and elist array os intr_handle structure, based on size provided by user. Eg size can be MSIX interrupts supported by a PCI device. Signed-off-by: Harman Kalra --- drivers/bus/pci/linux/pci_vfio.c | 7 + lib/eal/common/eal_common_interrupts.c | 172 ++++++++++++++++++++++++- lib/eal/include/meson.build | 1 - lib/eal/include/rte_eal_interrupts.h | 72 ----------- lib/eal/include/rte_interrupts.h | 24 +++- 5 files changed, 196 insertions(+), 80 deletions(-) delete mode 100644 lib/eal/include/rte_eal_interrupts.h diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c index f920163580..6af8279189 100644 --- a/drivers/bus/pci/linux/pci_vfio.c +++ b/drivers/bus/pci/linux/pci_vfio.c @@ -266,6 +266,13 @@ pci_vfio_setup_interrupts(struct rte_pci_device *dev, int vfio_dev_fd) return -1; } + /* Reallocate the efds and elist fields of intr_handle based + * on PCI device MSIX size. + */ + if (rte_intr_handle_event_list_update(dev->intr_handle, + irq.count)) + return -1; + /* if this vector cannot be used with eventfd, fail if we explicitly * specified interrupt type, otherwise continue */ if ((irq.flags & VFIO_IRQ_INFO_EVENTFD) == 0) { diff --git a/lib/eal/common/eal_common_interrupts.c b/lib/eal/common/eal_common_interrupts.c index 2e4fed96f0..caddf9b0ad 100644 --- a/lib/eal/common/eal_common_interrupts.c +++ b/lib/eal/common/eal_common_interrupts.c @@ -11,6 +11,29 @@ #include +struct rte_intr_handle { + RTE_STD_C11 + union { + struct { + /** VFIO/UIO cfg device file descriptor */ + int dev_fd; + int fd; /**< interrupt event file descriptor */ + }; + void *handle; /**< device driver handle (Windows) */ + }; + bool alloc_from_hugepage; + enum rte_intr_handle_type type; /**< handle type */ + uint32_t max_intr; /**< max interrupt requested */ + uint32_t nb_efd; /**< number of available efd(event fd) */ + uint8_t efd_counter_size; /**< size of efd counter, used for vdev */ + uint16_t nb_intr; + /**< Max vector count, default RTE_MAX_RXTX_INTR_VEC_ID */ + int *efds; /**< intr vectors/efds mapping */ + struct rte_epoll_event *elist; /**< intr vector epoll event */ + uint16_t vec_list_size; + int *intr_vec; /**< intr vector number array */ +}; + struct rte_intr_handle *rte_intr_handle_instance_alloc(int size, bool from_hugepage) @@ -31,11 +54,40 @@ struct rte_intr_handle *rte_intr_handle_instance_alloc(int size, } for (i = 0; i < size; i++) { + if (from_hugepage) + intr_handle[i].efds = rte_zmalloc(NULL, + RTE_MAX_RXTX_INTR_VEC_ID * sizeof(uint32_t), 0); + else + intr_handle[i].efds = calloc(1, + RTE_MAX_RXTX_INTR_VEC_ID * sizeof(uint32_t)); + if (!intr_handle[i].efds) { + RTE_LOG(ERR, EAL, "Fail to allocate event fd list\n"); + rte_errno = ENOMEM; + goto fail; + } + + if (from_hugepage) + intr_handle[i].elist = rte_zmalloc(NULL, + RTE_MAX_RXTX_INTR_VEC_ID * + sizeof(struct rte_epoll_event), 0); + else + intr_handle[i].elist = calloc(1, + RTE_MAX_RXTX_INTR_VEC_ID * + sizeof(struct rte_epoll_event)); + if (!intr_handle[i].elist) { + RTE_LOG(ERR, EAL, "fail to allocate event fd list\n"); + rte_errno = ENOMEM; + goto fail; + } intr_handle[i].nb_intr = RTE_MAX_RXTX_INTR_VEC_ID; intr_handle[i].alloc_from_hugepage = from_hugepage; } return intr_handle; +fail: + free(intr_handle->efds); + free(intr_handle); + return NULL; } struct rte_intr_handle *rte_intr_handle_instance_index_get( @@ -73,12 +125,48 @@ int rte_intr_handle_instance_index_set(struct rte_intr_handle *intr_handle, } intr_handle[index].fd = src->fd; - intr_handle[index].vfio_dev_fd = src->vfio_dev_fd; + intr_handle[index].dev_fd = src->dev_fd; + intr_handle[index].type = src->type; intr_handle[index].max_intr = src->max_intr; intr_handle[index].nb_efd = src->nb_efd; intr_handle[index].efd_counter_size = src->efd_counter_size; + if (intr_handle[index].nb_intr != src->nb_intr) { + if (src->alloc_from_hugepage) + intr_handle[index].efds = + rte_realloc(intr_handle[index].efds, + src->nb_intr * + sizeof(uint32_t), 0); + else + intr_handle[index].efds = + realloc(intr_handle[index].efds, + src->nb_intr * sizeof(uint32_t)); + if (intr_handle[index].efds == NULL) { + RTE_LOG(ERR, EAL, "Failed to realloc the efds list"); + rte_errno = ENOMEM; + goto fail; + } + + if (src->alloc_from_hugepage) + intr_handle[index].elist = + rte_realloc(intr_handle[index].elist, + src->nb_intr * + sizeof(struct rte_epoll_event), 0); + else + intr_handle[index].elist = + realloc(intr_handle[index].elist, + src->nb_intr * + sizeof(struct rte_epoll_event)); + if (intr_handle[index].elist == NULL) { + RTE_LOG(ERR, EAL, "Failed to realloc the event list"); + rte_errno = ENOMEM; + goto fail; + } + + intr_handle[index].nb_intr = src->nb_intr; + } + memcpy(intr_handle[index].efds, src->efds, src->nb_intr); memcpy(intr_handle[index].elist, src->elist, src->nb_intr); @@ -87,6 +175,45 @@ int rte_intr_handle_instance_index_set(struct rte_intr_handle *intr_handle, return rte_errno; } +int rte_intr_handle_event_list_update(struct rte_intr_handle *intr_handle, + int size) +{ + if (intr_handle == NULL) { + RTE_LOG(ERR, EAL, "Interrupt instance unallocated\n"); + rte_errno = ENOTSUP; + goto fail; + } + + if (size == 0) { + RTE_LOG(ERR, EAL, "Size can't be zero\n"); + rte_errno = EINVAL; + goto fail; + } + + intr_handle->efds = realloc(intr_handle->efds, + size * sizeof(uint32_t)); + if (intr_handle->efds == NULL) { + RTE_LOG(ERR, EAL, "Failed to realloc the efds list"); + rte_errno = ENOMEM; + goto fail; + } + + intr_handle->elist = realloc(intr_handle->elist, + size * sizeof(struct rte_epoll_event)); + if (intr_handle->elist == NULL) { + RTE_LOG(ERR, EAL, "Failed to realloc the event list"); + rte_errno = ENOMEM; + goto fail; + } + + intr_handle->nb_intr = size; + + return 0; +fail: + return rte_errno; +} + + void rte_intr_handle_instance_free(struct rte_intr_handle *intr_handle) { if (intr_handle == NULL) { @@ -94,10 +221,15 @@ void rte_intr_handle_instance_free(struct rte_intr_handle *intr_handle) rte_errno = ENOTSUP; } - if (intr_handle->alloc_from_hugepage) + if (intr_handle->alloc_from_hugepage) { + rte_free(intr_handle->efds); + rte_free(intr_handle->elist); rte_free(intr_handle); - else + } else { + free(intr_handle->efds); + free(intr_handle->elist); free(intr_handle); + } } int rte_intr_handle_fd_set(struct rte_intr_handle *intr_handle, int fd) @@ -164,7 +296,7 @@ int rte_intr_handle_dev_fd_set(struct rte_intr_handle *intr_handle, int fd) goto fail; } - intr_handle->vfio_dev_fd = fd; + intr_handle->dev_fd = fd; return 0; fail: @@ -179,7 +311,7 @@ int rte_intr_handle_dev_fd_get(const struct rte_intr_handle *intr_handle) goto fail; } - return intr_handle->vfio_dev_fd; + return intr_handle->dev_fd; fail: return rte_errno; } @@ -300,6 +432,12 @@ int *rte_intr_handle_efds_base(struct rte_intr_handle *intr_handle) goto fail; } + if (!intr_handle->efds) { + RTE_LOG(ERR, EAL, "Event fd list not allocated\n"); + rte_errno = ENOTSUP; + goto fail; + } + return intr_handle->efds; fail: return NULL; @@ -314,6 +452,12 @@ int rte_intr_handle_efds_index_get(const struct rte_intr_handle *intr_handle, goto fail; } + if (!intr_handle->efds) { + RTE_LOG(ERR, EAL, "Event fd list not allocated\n"); + rte_errno = EFAULT; + goto fail; + } + if (index >= intr_handle->nb_intr) { RTE_LOG(ERR, EAL, "Invalid size %d, max limit %d\n", index, intr_handle->nb_intr); @@ -335,6 +479,12 @@ int rte_intr_handle_efds_index_set(struct rte_intr_handle *intr_handle, goto fail; } + if (!intr_handle->efds) { + RTE_LOG(ERR, EAL, "Event fd list not allocated\n"); + rte_errno = EFAULT; + goto fail; + } + if (index >= intr_handle->nb_intr) { RTE_LOG(ERR, EAL, "Invalid size %d, max limit %d\n", index, intr_handle->nb_intr); @@ -358,6 +508,12 @@ struct rte_epoll_event *rte_intr_handle_elist_index_get( goto fail; } + if (!intr_handle->elist) { + RTE_LOG(ERR, EAL, "Event list not allocated\n"); + rte_errno = ENOTSUP; + goto fail; + } + if (index >= intr_handle->nb_intr) { RTE_LOG(ERR, EAL, "Invalid size %d, max limit %d\n", index, intr_handle->nb_intr); @@ -379,6 +535,12 @@ int rte_intr_handle_elist_index_set(struct rte_intr_handle *intr_handle, goto fail; } + if (!intr_handle->elist) { + RTE_LOG(ERR, EAL, "Event list not allocated\n"); + rte_errno = ENOTSUP; + goto fail; + } + if (index >= intr_handle->nb_intr) { RTE_LOG(ERR, EAL, "Invalid size %d, max limit %d\n", index, intr_handle->nb_intr); diff --git a/lib/eal/include/meson.build b/lib/eal/include/meson.build index 8e258607b8..86468d1a2b 100644 --- a/lib/eal/include/meson.build +++ b/lib/eal/include/meson.build @@ -49,7 +49,6 @@ headers += files( 'rte_version.h', 'rte_vfio.h', ) -indirect_headers += files('rte_eal_interrupts.h') # special case install the generic headers, since they go in a subdir generic_headers = files( diff --git a/lib/eal/include/rte_eal_interrupts.h b/lib/eal/include/rte_eal_interrupts.h deleted file mode 100644 index 216aece61b..0000000000 --- a/lib/eal/include/rte_eal_interrupts.h +++ /dev/null @@ -1,72 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2010-2014 Intel Corporation - */ - -#ifndef _RTE_INTERRUPTS_H_ -#error "don't include this file directly, please include generic " -#endif - -/** - * @file rte_eal_interrupts.h - * @internal - * - * Contains function prototypes exposed by the EAL for interrupt handling by - * drivers and other DPDK internal consumers. - */ - -#ifndef _RTE_EAL_INTERRUPTS_H_ -#define _RTE_EAL_INTERRUPTS_H_ - -#define RTE_MAX_RXTX_INTR_VEC_ID 512 -#define RTE_INTR_VEC_ZERO_OFFSET 0 -#define RTE_INTR_VEC_RXTX_OFFSET 1 - -/** - * The interrupt source type, e.g. UIO, VFIO, ALARM etc. - */ -enum rte_intr_handle_type { - RTE_INTR_HANDLE_UNKNOWN = 0, /**< generic unknown handle */ - RTE_INTR_HANDLE_UIO, /**< uio device handle */ - RTE_INTR_HANDLE_UIO_INTX, /**< uio generic handle */ - RTE_INTR_HANDLE_VFIO_LEGACY, /**< vfio device handle (legacy) */ - RTE_INTR_HANDLE_VFIO_MSI, /**< vfio device handle (MSI) */ - RTE_INTR_HANDLE_VFIO_MSIX, /**< vfio device handle (MSIX) */ - RTE_INTR_HANDLE_ALARM, /**< alarm handle */ - RTE_INTR_HANDLE_EXT, /**< external handler */ - RTE_INTR_HANDLE_VDEV, /**< virtual device */ - RTE_INTR_HANDLE_DEV_EVENT, /**< device event handle */ - RTE_INTR_HANDLE_VFIO_REQ, /**< VFIO request handle */ - RTE_INTR_HANDLE_MAX /**< count of elements */ -}; - -/** Handle for interrupts. */ -struct rte_intr_handle { - RTE_STD_C11 - union { - struct { - RTE_STD_C11 - union { - /** VFIO device file descriptor */ - int vfio_dev_fd; - /** UIO cfg file desc for uio_pci_generic */ - int uio_cfg_fd; - }; - int fd; /**< interrupt event file descriptor */ - }; - void *handle; /**< device driver handle (Windows) */ - }; - bool alloc_from_hugepage; - enum rte_intr_handle_type type; /**< handle type */ - uint32_t max_intr; /**< max interrupt requested */ - uint32_t nb_efd; /**< number of available efd(event fd) */ - uint8_t efd_counter_size; /**< size of efd counter, used for vdev */ - uint16_t nb_intr; - /**< Max vector count, default RTE_MAX_RXTX_INTR_VEC_ID */ - int efds[RTE_MAX_RXTX_INTR_VEC_ID]; /**< intr vectors/efds mapping */ - struct rte_epoll_event elist[RTE_MAX_RXTX_INTR_VEC_ID]; - /**< intr vector epoll event */ - uint16_t vec_list_size; - int *intr_vec; /**< intr vector number array */ -}; - -#endif /* _RTE_EAL_INTERRUPTS_H_ */ diff --git a/lib/eal/include/rte_interrupts.h b/lib/eal/include/rte_interrupts.h index afc3262967..7dfb849eea 100644 --- a/lib/eal/include/rte_interrupts.h +++ b/lib/eal/include/rte_interrupts.h @@ -25,9 +25,29 @@ extern "C" { /** Interrupt handle */ struct rte_intr_handle; -#define RTE_INTR_HANDLE_DEFAULT_SIZE 1 +#define RTE_MAX_RXTX_INTR_VEC_ID 512 +#define RTE_INTR_VEC_ZERO_OFFSET 0 +#define RTE_INTR_VEC_RXTX_OFFSET 1 + +/** + * The interrupt source type, e.g. UIO, VFIO, ALARM etc. + */ +enum rte_intr_handle_type { + RTE_INTR_HANDLE_UNKNOWN = 0, /**< generic unknown handle */ + RTE_INTR_HANDLE_UIO, /**< uio device handle */ + RTE_INTR_HANDLE_UIO_INTX, /**< uio generic handle */ + RTE_INTR_HANDLE_VFIO_LEGACY, /**< vfio device handle (legacy) */ + RTE_INTR_HANDLE_VFIO_MSI, /**< vfio device handle (MSI) */ + RTE_INTR_HANDLE_VFIO_MSIX, /**< vfio device handle (MSIX) */ + RTE_INTR_HANDLE_ALARM, /**< alarm handle */ + RTE_INTR_HANDLE_EXT, /**< external handler */ + RTE_INTR_HANDLE_VDEV, /**< virtual device */ + RTE_INTR_HANDLE_DEV_EVENT, /**< device event handle */ + RTE_INTR_HANDLE_VFIO_REQ, /**< VFIO request handle */ + RTE_INTR_HANDLE_MAX /**< count of elements */ +}; -#include "rte_eal_interrupts.h" +#define RTE_INTR_HANDLE_DEFAULT_SIZE 1 /** Function to be registered for the specific interrupt */ typedef void (*rte_intr_callback_fn)(void *cb_arg);