From patchwork Wed Sep 29 14:52:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dmitry Kozlyuk X-Patchwork-Id: 100002 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6873AA0547; Wed, 29 Sep 2021 16:53:16 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id E4A81410F0; Wed, 29 Sep 2021 16:53:12 +0200 (CEST) Received: from AZHDRRW-EX02.NVIDIA.COM (azhdrrw-ex02.nvidia.com [20.64.145.131]) by mails.dpdk.org (Postfix) with ESMTP id 0CFD7410EF for ; Wed, 29 Sep 2021 16:53:11 +0200 (CEST) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.171) by mxs.oss.nvidia.com (10.13.234.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.858.15; Wed, 29 Sep 2021 07:53:09 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JyVsoXd5mDM7FTjyNY3OQPu2K1X7xTTrHSJhFYBQuHl6Vs+ybnZzv5wiZkSck6MX5GQkymC6yVXINjKlzgX9d1D9zRpkd1jhHzfCN+aS8eiu9ETNbp2GPxpmVsQ7o3GVn/i12SkNvpXjfeUlO93svcDfwG4/McLJvJeHiDskxvKnk1IyvZ3eUw7II9t7WiNzct6EGZUK/B/2FRMLXnUaMTNJ7QJki7qldFRAmvGQcmH+TwdUkrTEhkAyJUGA00aoIyhIsIJV5eBonTjdPDQ26bqI8+o43mrqDOCJjF5sVDyved6Q+OcNAUDF9+XpHtOBCAE71OBCUGhVvD0dcZC+qw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=Yqdi7iBuhvqCl4Rr50p60IujMrdrZZ2Pg3HtVXrPlsI=; b=YtlKOGY2uaWl4XmnSXIj8YEMMbEBxohpgxyFKKT7/o8imzsPpj07qNQRve+cCXd7XgnLn9aUPpcGe3f3JWfYCTSv0VZsE5l5qwLZIlcSfmDYkUcCzJsLHNHkTM6X7juL42Rb0hDxmxDJo0DQrDY3Pc/c1BzdHeQkDXoRHvKJ4HOvx6YEy05O0DMyVcBfu34LpOdGoUvSvn+IERLi93uYFa7kKEqQUKfglEH6xnZfg+p32cVi9nlrGwpI+p4I0l6gDFmZSMV/DbLDT3rZ2sI6n3kLpEwkID8/+fbVI4vl43ccbK0TEpqfYV9R76ey/Wg8zGytbfDizRVjr+NosX6VPw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=6wind.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Yqdi7iBuhvqCl4Rr50p60IujMrdrZZ2Pg3HtVXrPlsI=; b=k7hzpYy363dyl+NNRfqaIJbzMKW1ecg7qApZ/kFmwMiiDmgxXLX057Gfsw5c1V7dnOrOOvlwgwfbB02uYO1C3MDMzg3d/MLJcoSHDcvRZVtJTB88PXyH960+n75GOhijcKDSueExQbsvLTH+LBFIEDJWF2ueeI/le6lXYlnwiuF6gwRZSLRJV+9qOTfiRH4GRaWowOTJA7Zjh9RkI0UHszC7OseSR2O/yJLysdxTOV50Ezl1txZZEXJr7dSCxT6R+iA4j/PSI2DAKBt6bnat53RF1Bw5GGp+5DM3CCCS/IytYnuxIEF4R7/hoOJO60SVfUQKfPF51eWXNR6X2uqh7g== Received: from DM5PR21CA0011.namprd21.prod.outlook.com (2603:10b6:3:ac::21) by CY4PR1201MB0005.namprd12.prod.outlook.com (2603:10b6:903:d1::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.13; Wed, 29 Sep 2021 14:53:08 +0000 Received: from DM6NAM11FT065.eop-nam11.prod.protection.outlook.com (2603:10b6:3:ac:cafe::7) by DM5PR21CA0011.outlook.office365.com (2603:10b6:3:ac::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.1 via Frontend Transport; Wed, 29 Sep 2021 14:53:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; 6wind.com; dkim=none (message not signed) header.d=none;6wind.com; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by DM6NAM11FT065.mail.protection.outlook.com (10.13.172.109) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4566.14 via Frontend Transport; Wed, 29 Sep 2021 14:53:07 +0000 Received: from DRHQMAIL107.nvidia.com (10.27.9.16) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 29 Sep 2021 07:53:06 -0700 Received: from nvidia.com (172.20.187.5) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 29 Sep 2021 14:53:04 +0000 From: To: CC: Dmitry Kozlyuk , Matan Azrad , Olivier Matz , Andrew Rybchenko , Ray Kinsella , "Anatoly Burakov" Date: Wed, 29 Sep 2021 17:52:46 +0300 Message-ID: <20210929145249.2176811-2-dkozlyuk@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210929145249.2176811-1-dkozlyuk@nvidia.com> References: <20210818090755.2419483-1-dkozlyuk@nvidia.com> <20210929145249.2176811-1-dkozlyuk@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To DRHQMAIL107.nvidia.com (10.27.9.16) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: fc4f14b9-3767-4873-b68c-08d98358d94f X-MS-TrafficTypeDiagnostic: CY4PR1201MB0005: X-Microsoft-Antispam-PRVS: X-MS-Exchange-Transport-Forked: True X-MS-Oob-TLC-OOBClassifiers: OLM:1332; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: M9wEYG+7OhFLI1uF2VpndUDSbwk/ij9btVCmy/Q7aJvTEQzmrZ6r9iYcHQT3ZO+AVnymj7mR8MNEHkQRQhWBYVHWd8VXkG8bgo1GTkjgWo/MTuefbfdWsLM6VzBVvtMQ7Yx+BqnwIb6Z9QCg98iJAKfioJjEoijfyfZYZwOhmCEQZkN+7Usg1Gu+hPgeWEOxXELadQdOZZstPmHcKovapxgF72MND66L+jCGmicP8JfciSHp77NBQ62/GMyuho2IbeZxQdLnViSmr9cNySfhXAOboBN0CchaO9i1UpPktvDWzdR3eSzmSBiYq8aQSA2oDQxu+IZ3y4tsQK5SQDl3U7xwN5AjP9vQNp13YHAUP+cJiBA4HJgtM32yzOssJgELKYPfNWAeLGPZLR3Og/ETn5mOd+suv8hnpavvimJJ9irYpiaHAyyuuILEOqDIZu56epFErO+rsuW3NBAXtZ48+RNAa4+nL4ozoDSvqnwm8m/txfm+uuFOjQf6Ynd8YsV2LHFiyO+9j+5x2p3QVYQ1J7FyTfuGHupvQFuv0DhvaIfkpUCrsem+CsCePGSZXNFGg2g8+zkGBrLWTQ9oHmkirGHDT9ZxE+H0Z0qN8MGYC5v3pMzwZDWzc7mOkw+aZNVpYfHKS6kX+HyVKpOAIQ7CkFr1j8pK3BVHBQaPf1v38hDcELH8yTVZgbt2PeT5mxJLdLOET2NBEBYandEUNqLsCQ== X-Forefront-Antispam-Report: CIP:216.228.112.32; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid01.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(30864003)(6916009)(5660300002)(316002)(4326008)(2906002)(70586007)(55016002)(86362001)(7696005)(2876002)(82310400003)(54906003)(70206006)(426003)(83380400001)(2616005)(1076003)(336012)(47076005)(186003)(8676002)(356005)(8936002)(26005)(36756003)(16526019)(6666004)(107886003)(6286002)(508600001)(36860700001)(7636003); DIR:OUT; SFP:1101; X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Sep 2021 14:53:07.4748 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: fc4f14b9-3767-4873-b68c-08d98358d94f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.32]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT065.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR1201MB0005 Subject: [dpdk-dev] [PATCH v2 1/4] mempool: add event callbacks X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Dmitry Kozlyuk Performance of MLX5 PMD of different classes can benefit if PMD knows which memory it will need to handle in advance, before the first mbuf is sent to the PMD. It is impractical, however, to consider all allocated memory for this purpose. Most often mbuf memory comes from mempools that can come and go. PMD can enumerate existing mempools on device start, but it also needs to track creation and destruction of mempools after the forwarding starts but before an mbuf from the new mempool is sent to the device. Add an internal API to register callback for mempool lify cycle events, currently RTE_MEMPOOL_EVENT_READY (after populating) and RTE_MEMPOOL_EVENT_DESTROY (before freeing): * rte_mempool_event_callback_register() * rte_mempool_event_callback_unregister() Provide a unit test for the new API. Signed-off-by: Dmitry Kozlyuk Acked-by: Matan Azrad --- app/test/test_mempool.c | 75 ++++++++++++++++++++ lib/mempool/rte_mempool.c | 143 +++++++++++++++++++++++++++++++++++++- lib/mempool/rte_mempool.h | 56 +++++++++++++++ lib/mempool/version.map | 8 +++ 4 files changed, 279 insertions(+), 3 deletions(-) diff --git a/app/test/test_mempool.c b/app/test/test_mempool.c index 7675a3e605..0c4ed7c60b 100644 --- a/app/test/test_mempool.c +++ b/app/test/test_mempool.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -471,6 +472,74 @@ test_mp_mem_init(struct rte_mempool *mp, data->ret = 0; } +struct test_mempool_events_data { + struct rte_mempool *mp; + enum rte_mempool_event event; + bool invoked; +}; + +static void +test_mempool_events_cb(enum rte_mempool_event event, + struct rte_mempool *mp, void *arg) +{ + struct test_mempool_events_data *data = arg; + + data->mp = mp; + data->event = event; + data->invoked = true; +} + +static int +test_mempool_events(int (*populate)(struct rte_mempool *mp)) +{ + struct test_mempool_events_data data; + struct rte_mempool *mp; + int ret; + + ret = rte_mempool_event_callback_register(NULL, &data); + RTE_TEST_ASSERT_NOT_EQUAL(ret, 0, "Registered a NULL callback"); + + memset(&data, 0, sizeof(data)); + ret = rte_mempool_event_callback_register(test_mempool_events_cb, + &data); + RTE_TEST_ASSERT_EQUAL(ret, 0, "Failed to register the callback: %s", + rte_strerror(rte_errno)); + + mp = rte_mempool_create_empty("empty", MEMPOOL_SIZE, + MEMPOOL_ELT_SIZE, 0, 0, + SOCKET_ID_ANY, 0); + RTE_TEST_ASSERT_NOT_NULL(mp, "Cannot create an empty mempool: %s", + rte_strerror(rte_errno)); + RTE_TEST_ASSERT_EQUAL(data.invoked, false, + "Callback invoked on an empty mempool creation"); + + rte_mempool_set_ops_byname(mp, rte_mbuf_best_mempool_ops(), NULL); + ret = populate(mp); + RTE_TEST_ASSERT_EQUAL(ret, (int)mp->size, "Failed to populate the mempool: %s", + rte_strerror(rte_errno)); + RTE_TEST_ASSERT_EQUAL(data.invoked, true, + "Callback not invoked on an empty mempool population"); + RTE_TEST_ASSERT_EQUAL(data.event, RTE_MEMPOOL_EVENT_READY, + "Wrong callback invoked, expected READY"); + RTE_TEST_ASSERT_EQUAL(data.mp, mp, + "Callback invoked for a wrong mempool"); + + memset(&data, 0, sizeof(data)); + rte_mempool_free(mp); + RTE_TEST_ASSERT_EQUAL(data.invoked, true, + "Callback not invoked on mempool destruction"); + RTE_TEST_ASSERT_EQUAL(data.event, RTE_MEMPOOL_EVENT_DESTROY, + "Wrong callback invoked, expected DESTROY"); + RTE_TEST_ASSERT_EQUAL(data.mp, mp, + "Callback invoked for a wrong mempool"); + + ret = rte_mempool_event_callback_unregister(test_mempool_events_cb, + &data); + RTE_TEST_ASSERT_EQUAL(ret, 0, "Failed to unregister the callback: %s", + rte_strerror(rte_errno)); + return 0; +} + static int test_mempool(void) { @@ -645,6 +714,12 @@ test_mempool(void) if (test_mempool_basic(default_pool, 1) < 0) GOTO_ERR(ret, err); + /* test mempool event callbacks */ + if (test_mempool_events(rte_mempool_populate_default) < 0) + GOTO_ERR(ret, err); + if (test_mempool_events(rte_mempool_populate_anon) < 0) + GOTO_ERR(ret, err); + rte_mempool_list_dump(stdout); ret = 0; diff --git a/lib/mempool/rte_mempool.c b/lib/mempool/rte_mempool.c index 59a588425b..c6cb99ba48 100644 --- a/lib/mempool/rte_mempool.c +++ b/lib/mempool/rte_mempool.c @@ -42,6 +42,18 @@ static struct rte_tailq_elem rte_mempool_tailq = { }; EAL_REGISTER_TAILQ(rte_mempool_tailq) +TAILQ_HEAD(mempool_callback_list, rte_tailq_entry); + +static struct rte_tailq_elem callback_tailq = { + .name = "RTE_MEMPOOL_CALLBACK", +}; +EAL_REGISTER_TAILQ(callback_tailq) + +/* Invoke all registered mempool event callbacks. */ +static void +mempool_event_callback_invoke(enum rte_mempool_event event, + struct rte_mempool *mp); + #define CACHE_FLUSHTHRESH_MULTIPLIER 1.5 #define CALC_CACHE_FLUSHTHRESH(c) \ ((typeof(c))((c) * CACHE_FLUSHTHRESH_MULTIPLIER)) @@ -360,6 +372,10 @@ rte_mempool_populate_iova(struct rte_mempool *mp, char *vaddr, STAILQ_INSERT_TAIL(&mp->mem_list, memhdr, next); mp->nb_mem_chunks++; + /* Report the mempool as ready only when fully populated. */ + if (mp->populated_size >= mp->size) + mempool_event_callback_invoke(RTE_MEMPOOL_EVENT_READY, mp); + rte_mempool_trace_populate_iova(mp, vaddr, iova, len, free_cb, opaque); return i; @@ -722,6 +738,7 @@ rte_mempool_free(struct rte_mempool *mp) } rte_mcfg_tailq_write_unlock(); + mempool_event_callback_invoke(RTE_MEMPOOL_EVENT_DESTROY, mp); rte_mempool_trace_free(mp); rte_mempool_free_memchunks(mp); rte_mempool_ops_free(mp); @@ -779,9 +796,9 @@ rte_mempool_cache_free(struct rte_mempool_cache *cache) /* create an empty mempool */ struct rte_mempool * -rte_mempool_create_empty(const char *name, unsigned n, unsigned elt_size, - unsigned cache_size, unsigned private_data_size, - int socket_id, unsigned flags) +rte_mempool_create_empty(const char *name, unsigned int n, + unsigned int elt_size, unsigned int cache_size, + unsigned int private_data_size, int socket_id, unsigned int flags) { char mz_name[RTE_MEMZONE_NAMESIZE]; struct rte_mempool_list *mempool_list; @@ -1343,3 +1360,123 @@ void rte_mempool_walk(void (*func)(struct rte_mempool *, void *), rte_mcfg_mempool_read_unlock(); } + +struct mempool_callback { + rte_mempool_event_callback *func; + void *arg; +}; + +static void +mempool_event_callback_invoke(enum rte_mempool_event event, + struct rte_mempool *mp) +{ + struct mempool_callback_list *list; + struct rte_tailq_entry *te; + void *tmp_te; + + rte_mcfg_tailq_read_lock(); + list = RTE_TAILQ_CAST(callback_tailq.head, mempool_callback_list); + TAILQ_FOREACH_SAFE(te, list, next, tmp_te) { + struct mempool_callback *cb = te->data; + rte_mcfg_tailq_read_unlock(); + cb->func(event, mp, cb->arg); + rte_mcfg_tailq_read_lock(); + } + rte_mcfg_tailq_read_unlock(); +} + +int +rte_mempool_event_callback_register(rte_mempool_event_callback *func, + void *arg) +{ + struct mempool_callback_list *list; + struct rte_tailq_entry *te = NULL; + struct mempool_callback *cb; + void *tmp_te; + int ret; + + if (func == NULL) { + rte_errno = EINVAL; + return -rte_errno; + } + + rte_mcfg_mempool_read_lock(); + rte_mcfg_tailq_write_lock(); + + list = RTE_TAILQ_CAST(callback_tailq.head, mempool_callback_list); + TAILQ_FOREACH_SAFE(te, list, next, tmp_te) { + struct mempool_callback *cb = + (struct mempool_callback *)te->data; + if (cb->func == func && cb->arg == arg) { + ret = -EEXIST; + goto exit; + } + } + + te = rte_zmalloc("MEMPOOL_TAILQ_ENTRY", sizeof(*te), 0); + if (te == NULL) { + RTE_LOG(ERR, MEMPOOL, + "Cannot allocate event callback tailq entry!\n"); + ret = -ENOMEM; + goto exit; + } + + cb = rte_malloc("MEMPOOL_EVENT_CALLBACK", sizeof(*cb), 0); + if (cb == NULL) { + RTE_LOG(ERR, MEMPOOL, + "Cannot allocate event callback!\n"); + rte_free(te); + ret = -ENOMEM; + goto exit; + } + + cb->func = func; + cb->arg = arg; + te->data = cb; + TAILQ_INSERT_TAIL(list, te, next); + ret = 0; + +exit: + rte_mcfg_tailq_write_unlock(); + rte_mcfg_mempool_read_unlock(); + rte_errno = -ret; + return ret; +} + +int +rte_mempool_event_callback_unregister(rte_mempool_event_callback *func, + void *arg) +{ + struct mempool_callback_list *list; + struct rte_tailq_entry *te = NULL; + struct mempool_callback *cb; + int ret; + + if (rte_eal_process_type() != RTE_PROC_PRIMARY) { + rte_errno = EPERM; + return -1; + } + + rte_mcfg_mempool_read_lock(); + rte_mcfg_tailq_write_lock(); + ret = -ENOENT; + list = RTE_TAILQ_CAST(callback_tailq.head, mempool_callback_list); + TAILQ_FOREACH(te, list, next) { + cb = (struct mempool_callback *)te->data; + if (cb->func == func && cb->arg == arg) + break; + } + if (te != NULL) { + TAILQ_REMOVE(list, te, next); + ret = 0; + } + rte_mcfg_tailq_write_unlock(); + rte_mcfg_mempool_read_unlock(); + + if (ret == 0) { + rte_free(te); + rte_free(cb); + } + rte_errno = -ret; + return ret; +} diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h index 4235d6f0bf..c81e488851 100644 --- a/lib/mempool/rte_mempool.h +++ b/lib/mempool/rte_mempool.h @@ -1775,6 +1775,62 @@ void rte_mempool_walk(void (*func)(struct rte_mempool *, void *arg), int rte_mempool_get_page_size(struct rte_mempool *mp, size_t *pg_sz); +/** + * Mempool event type. + * @internal + */ +enum rte_mempool_event { + /** Occurs after a mempool is successfully populated. */ + RTE_MEMPOOL_EVENT_READY = 0, + /** Occurs before destruction of a mempool begins. */ + RTE_MEMPOOL_EVENT_DESTROY = 1, +}; + +/** + * @internal + * Mempool event callback. + */ +typedef void (rte_mempool_event_callback)( + enum rte_mempool_event event, + struct rte_mempool *mp, + void *arg); + +/** + * @internal + * Register a callback invoked on mempool life cycle event. + * Callbacks will be invoked in the process that creates the mempool. + * + * @param cb + * Callback function. + * @param cb_arg + * User data. + * + * @return + * 0 on success, negative on failure and rte_errno is set. + */ +__rte_internal +int +rte_mempool_event_callback_register(rte_mempool_event_callback *cb, + void *cb_arg); + +/** + * @internal + * Unregister a callback added with rte_mempool_event_callback_register(). + * @p cb and @p arg must exactly match registration parameters. + * + * @param cb + * Callback function. + * @param cb_arg + * User data. + * + * @return + * 0 on success, negative on failure and rte_errno is set. + */ +__rte_internal +int +rte_mempool_event_callback_unregister(rte_mempool_event_callback *cb, + void *cb_arg); + #ifdef __cplusplus } #endif diff --git a/lib/mempool/version.map b/lib/mempool/version.map index 9f77da6fff..1b7d7c5456 100644 --- a/lib/mempool/version.map +++ b/lib/mempool/version.map @@ -64,3 +64,11 @@ EXPERIMENTAL { __rte_mempool_trace_ops_free; __rte_mempool_trace_set_ops_byname; }; + +INTERNAL { + global: + + # added in 21.11 + rte_mempool_event_callback_register; + rte_mempool_event_callback_unregister; +}; From patchwork Wed Sep 29 14:52:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dmitry Kozlyuk X-Patchwork-Id: 100003 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 26A8FA0547; Wed, 29 Sep 2021 16:53:24 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 5AA37410FF; Wed, 29 Sep 2021 16:53:15 +0200 (CEST) Received: from AZHDRRW-EX02.NVIDIA.COM (azhdrrw-ex02.nvidia.com [20.64.145.131]) by mails.dpdk.org (Postfix) with ESMTP id 45BFC410EF for ; Wed, 29 Sep 2021 16:53:12 +0200 (CEST) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (104.47.66.41) by mxs.oss.nvidia.com (10.13.234.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.858.15; Wed, 29 Sep 2021 07:53:11 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HznNskJ1AX3CJBxesT8Nb/28X+GLKzQpUJJ3J0BvKgI89CDxyIgejA26eH9sM9VWduoQALcdNwVh7EBE5jswGbh//B0n2p5wOW7MEKVKhmMJjtGNsj20SIejgDmfQic/OuCKXr59O8GH2WsUvQ7x5aC4fUhiQQv37UFpYGcGtzmG3IaQxt6G1dSCEISgFfMDqui91hIDHA/pyRVu+b6hv86qusb3zmS0NuEoOedokskSHAse7/OjP7EhNC0dpSbiN8wDZSDKLxdltbesBLmhuctQIsD7iaZVWbvU/e5JdGaE4AW2JTlMpNx5Jbd0rt9/dVQrS9dQH6GmqhXTKzPktQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=gjsAWiWZUcrM6X2JItyUUXXz6ncRYoTc2/jMGU1dsic=; b=d53iYflrX2+hNL7MYB1Ndk+CWCqF1ty+Ccuk6HouiPQ+c7q0nI9X8yzI8Bqmo58bafgUDe5qkC5fhKYMBXqgDCv/W9whCb/CLjObtZw8SzYplevAt+HQ0IeurD9iSRkbrc5zTXMiG8t6p7W3DFHEDaLThKa16d2i+/IQvf7gnxGYZvi1mXfN5osvaWKVmWwDA37C7geYpf9IdZDjJAeMCAx+tY0iqzdkGTDshN4tLVjtwclzz1i6aXOdbsKk0MCH2OO4Zdl+wJwT2ziGfvGMjKNvgtvPt1YYmXuk5ivGFPinYIpT5P1BF63xFv35o6/IpMl8ox/OgKbb09e+/f1imQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gjsAWiWZUcrM6X2JItyUUXXz6ncRYoTc2/jMGU1dsic=; b=Dc0q8h1sb6qi/S4Yruel5AXfkx/qbM2xYuPDnGtalahTAqZMpcYtRXE2ykkRCCel3vnjyQ29qePy/r0VXROU/P4rWOHFmUBTxwOQpttugsh2csVyACZ7iRRWWYgZ8YeD1kh9/BG1EKRGzmGSWl8TvGN0JiOcmBNrmZtXCGPhpQ9BCS3fta36wJRscCj4x4VBy+pTFEsqC92BR7UEHlO7mlIy1oaz2HigtcF6u1yBeed8Tb8Tpum/G1HXi1FWn4TeeZMWsz4dXrvgCEG91N2pCNJTek80cij+tkG8ebKMqX4FalxIq55i1y/m8Leu5pGv4csyLhHu6J0Xnj2Zhj7kfA== Received: from DM6PR12CA0021.namprd12.prod.outlook.com (2603:10b6:5:1c0::34) by CH2PR12MB4183.namprd12.prod.outlook.com (2603:10b6:610:7a::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.14; Wed, 29 Sep 2021 14:53:09 +0000 Received: from DM6NAM11FT064.eop-nam11.prod.protection.outlook.com (2603:10b6:5:1c0:cafe::d9) by DM6PR12CA0021.outlook.office365.com (2603:10b6:5:1c0::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.15 via Frontend Transport; Wed, 29 Sep 2021 14:53:09 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by DM6NAM11FT064.mail.protection.outlook.com (10.13.172.234) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4566.14 via Frontend Transport; Wed, 29 Sep 2021 14:53:08 +0000 Received: from DRHQMAIL107.nvidia.com (10.27.9.16) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 29 Sep 2021 14:53:08 +0000 Received: from nvidia.com (172.20.187.5) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 29 Sep 2021 14:53:06 +0000 From: To: CC: Dmitry Kozlyuk , Matan Azrad , Olivier Matz , Andrew Rybchenko Date: Wed, 29 Sep 2021 17:52:47 +0300 Message-ID: <20210929145249.2176811-3-dkozlyuk@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210929145249.2176811-1-dkozlyuk@nvidia.com> References: <20210818090755.2419483-1-dkozlyuk@nvidia.com> <20210929145249.2176811-1-dkozlyuk@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To DRHQMAIL107.nvidia.com (10.27.9.16) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f62129d2-49b1-4391-50d3-08d98358da31 X-MS-TrafficTypeDiagnostic: CH2PR12MB4183: X-Microsoft-Antispam-PRVS: X-MS-Exchange-Transport-Forked: True X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3vR/12SUseJWZfvUcRChKB3AZOJKd5ZTbBSLJonyWFWIUDsZ1OTHacvO6oWJE/hzSzAU0Iyh4r5mde3aTIZ2ltsZuNe3N6OD7Dy5EeyIgJeZFgyTMZHKNlFYkFu/mx9ZMvJ/rhMdn2MLYU8ud0jeVuStuDATHUnELoW1+bKig4GcbIv+cTRFHXmHJOPJA/v1TSSM0DmPuuUu2bbjrhHcMGfkjLEjwRmMMythWtQmihglAnXofQaqo4dVFMa2ROlfudGKUC7wUe12gKPy/UROva9OH24Xmzwmz4a2yq8Q+cDVFIi9nWPrKEhfOfrQ97OODcnz6nxdJ0dYXlhlP2ftqhRiOMkJXFq3hBDTHAUVDchMwt9xKA8pY+byuwVvaUvytKh/4FoN0SQW9EsXdtqnMCl0xwJdWVyp1FQGq+rYtO7IyseUDsboBN9a+sx01azZjSe9sfxI6DKB7fWtQUBjDrTxElRyddbNhgs8O0VN5JlJ4SfzZ0ZG3+toXbqer/x1WhVdo4zZwSCyv1MR9QaM3WT0g9zkcSbo136IlHrx5FA0gp8MhkZ/RGCaEPDgH0AabGkD8F57zE5YLHqAOTVxNn4XcD2SvdvggaeJOon2l05U+yRP2qcqX1TSotIzct/LTZ7r9AXeHXxiSA8y12GLJMTWTsOP940dvUNI7adTaf0zbccAZUc5U4XCEmmLETPM54/c7Y6pZqzp+Z0dkHehbg== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(6916009)(83380400001)(186003)(2876002)(7696005)(16526019)(70206006)(316002)(356005)(508600001)(7636003)(26005)(36756003)(6286002)(1076003)(107886003)(70586007)(6666004)(82310400003)(4326008)(8936002)(426003)(8676002)(55016002)(336012)(47076005)(5660300002)(2616005)(86362001)(2906002)(36860700001)(54906003); DIR:OUT; SFP:1101; X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Sep 2021 14:53:08.9443 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f62129d2-49b1-4391-50d3-08d98358da31 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT064.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4183 Subject: [dpdk-dev] [PATCH v2 2/4] mempool: add non-IO flag X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Dmitry Kozlyuk Mempool is a generic allocator that is not necessarily used for device IO operations and its memory for DMA. Add MEMPOOL_F_NON_IO flag to mark such mempools. Signed-off-by: Dmitry Kozlyuk Acked-by: Matan Azrad --- doc/guides/rel_notes/release_21_11.rst | 3 +++ lib/mempool/rte_mempool.h | 4 ++++ 2 files changed, 7 insertions(+) diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index f85dc99c8b..873beda633 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -155,6 +155,9 @@ API Changes the crypto/security operation. This field will be used to communicate events such as soft expiry with IPsec in lookaside mode. +* mempool: Added ``MEMPOOL_F_NON_IO`` flag to give a hint to DPDK components + that objects from this pool will not be used for device IO (e.g. DMA). + ABI Changes ----------- diff --git a/lib/mempool/rte_mempool.h b/lib/mempool/rte_mempool.h index c81e488851..4d18957d6d 100644 --- a/lib/mempool/rte_mempool.h +++ b/lib/mempool/rte_mempool.h @@ -263,6 +263,7 @@ struct rte_mempool { #define MEMPOOL_F_SC_GET 0x0008 /**< Default get is "single-consumer".*/ #define MEMPOOL_F_POOL_CREATED 0x0010 /**< Internal: pool is created. */ #define MEMPOOL_F_NO_IOVA_CONTIG 0x0020 /**< Don't need IOVA contiguous objs. */ +#define MEMPOOL_F_NON_IO 0x0040 /**< Not used for device IO (DMA). */ /** * @internal When debug is enabled, store some statistics. @@ -992,6 +993,9 @@ typedef void (rte_mempool_ctor_t)(struct rte_mempool *, void *); * "single-consumer". Otherwise, it is "multi-consumers". * - MEMPOOL_F_NO_IOVA_CONTIG: If set, allocated objects won't * necessarily be contiguous in IO memory. + * - MEMPOOL_F_NO_IO: If set, the mempool is considered to be + * never used for device IO, i.e. DMA operations, + * which may affect some PMD behavior. * @return * The pointer to the new allocated mempool, on success. NULL on error * with rte_errno set appropriately. Possible rte_errno values include: From patchwork Wed Sep 29 14:52:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dmitry Kozlyuk X-Patchwork-Id: 100004 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AA745A0547; Wed, 29 Sep 2021 16:53:30 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 71E0641104; Wed, 29 Sep 2021 16:53:17 +0200 (CEST) Received: from AZHDRRW-EX01.nvidia.com (azhdrrw-ex01.nvidia.com [20.51.104.162]) by mails.dpdk.org (Postfix) with ESMTP id 7044E41101 for ; Wed, 29 Sep 2021 16:53:15 +0200 (CEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.170) by mxs.oss.nvidia.com (10.13.234.36) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.858.15; Wed, 29 Sep 2021 07:53:13 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jLEloz3R3rVcbl2hVgokFDrt5rnh8WARyQD4WbgtW4loQ2Ggpar4S8QamJYPZKkRPK81z7gF9Vyj2qkK9fwON09u4HhvpC42oPvNi/tWUrhqbqmhvHVf7dRoHD8HOKyL6oNVl/XVmYYUaXEMjeJzkh368ObsJJ7FON49PUsFA8CO+3Dtw8emSwTtJdKCjEIbK5wGDlUOTVqmDjc8iLtaWsjBvjJlraaCfCDHd234jcU07jfriVko+PGnHkxHy4DjNtnzZDHN5U7nWJWazl2y6sALLwkk8HuwZ1HmBJhhQzqxrDlgc7VmbNI1M5faDjkZyPa9hnWjxjjPbzyUqo1Jsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=ZyEWEpfM0rOS/QhYKFcxlE3mAtw3Gxjq+XTtG57aZtE=; b=Bzm3VanSdQ9R084U+HsvRY4SSCmBCVTg3gaLM6wSdh1PR9YJEAH0EVUmGG3q7nJEzT9FBlQyvfZESOjASBwJMlSuSpIKOh60O4q6n3vZvGTcJffZLLNmJX79yS/aYoPrdqHh0HII++fs+4CjwVM8wJq2rxfdPq0wzreeCewNUKRFgGUuMZNhHgNg+yc11Wj58YtPVxonQCzAvQbGjHOHoBsp1LzqZ3fXE5nPi9/wG1i0iO327xhbY0ZnKX/DrOV6rEC1ze6DarrtZj552IwHzO9ptr7lKoPjvnHMuB6mCGwwgmtpPDV/mcijE/SPzxt+zqS64SI2xjhn5mEZGhp/mg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.35) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZyEWEpfM0rOS/QhYKFcxlE3mAtw3Gxjq+XTtG57aZtE=; b=Y6NycSqOW5EtXipHcwI04Mpx+NboBVt27unETNLkEfGYeXl50a+wawerxCu8zA+VeVDPYGjlOGXg9VIXaQP6pg5Fi95RT3Dnx+Aj4bGehp9n71ckG2XwpGbq3S7aUfM1OUugaquqXPGOCahkfiE1AW1pYodbNDZNUEg2kDvM8BCjyWV74Wm65zrvSgr/0H6y0Qy5uI7Mq5ZMtF+IEuXg1+ZAvbfnaY4Q+nUSIImu9FapaROZLrrUp9AmnHqDTggMhKf9Lf30cnCFYVXC/LhnOs0bte/zgspLbJwjLn8xt3DIJDZov3NMb9xpS6lNBybCThm74QtxjZjq4E+iAgnSGw== Received: from MWHPR2001CA0004.namprd20.prod.outlook.com (2603:10b6:301:15::14) by BL0PR12MB4609.namprd12.prod.outlook.com (2603:10b6:208:8d::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.14; Wed, 29 Sep 2021 14:53:11 +0000 Received: from CO1NAM11FT026.eop-nam11.prod.protection.outlook.com (2603:10b6:301:15:cafe::67) by MWHPR2001CA0004.outlook.office365.com (2603:10b6:301:15::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.14 via Frontend Transport; Wed, 29 Sep 2021 14:53:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.35) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.35 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.35; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.35) by CO1NAM11FT026.mail.protection.outlook.com (10.13.175.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4566.14 via Frontend Transport; Wed, 29 Sep 2021 14:53:11 +0000 Received: from DRHQMAIL107.nvidia.com (10.27.9.16) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 29 Sep 2021 14:53:10 +0000 Received: from nvidia.com (172.20.187.5) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 29 Sep 2021 14:53:08 +0000 From: To: CC: Dmitry Kozlyuk , Matan Azrad , Viacheslav Ovsiienko , Ray Kinsella , Anatoly Burakov Date: Wed, 29 Sep 2021 17:52:48 +0300 Message-ID: <20210929145249.2176811-4-dkozlyuk@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210929145249.2176811-1-dkozlyuk@nvidia.com> References: <20210818090755.2419483-1-dkozlyuk@nvidia.com> <20210929145249.2176811-1-dkozlyuk@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To DRHQMAIL107.nvidia.com (10.27.9.16) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d1d8ccd1-6f17-45e5-a7c5-08d98358db81 X-MS-TrafficTypeDiagnostic: BL0PR12MB4609: X-Microsoft-Antispam-PRVS: X-MS-Exchange-Transport-Forked: True X-MS-Oob-TLC-OOBClassifiers: OLM:34; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2hvXxKWOKjd/Fob2OT3R3fHZWK9Gvxz1Ibv5GVNbREW4D4mmNnSWq/XRbCANh70tsU3kTG1ot5XpwLKK5VEpw869ZEB/AqrQhEftnXdtsYplJm7E9jMBqoIdJmdtBjOuNRIV9gjOKnRcO7EwUocThe8Mqu/jPfNG2pWNHb7S+4oqEB5PGS7cSqRo8QFf//qJaCXfWvBzywZLhtS4Hwm+Y5snski9g1/dN7JmwLFI7RWAMS1Skkt9MRzKbvHhJHofx++Nlx+7k5oQbTQTqkQmUXS9mbGd5SUR8UINlhle8qjfQpT3wyS3TqsFM5O62YHvyEI/hYFh6seC4eZAcu3RC2VKvaaVXlkRrLa7bwRNGbP2ir/2bx+yxpcyKCqx3bOwXrl4I3O+9kauF3G8vQw4BTifUjyTJaMPLMbJI1XgZclhwxAcyH5X6pB5A1gr7vH7w2n8guks+gJ6nHNV3bCJr3tB8LFqGroC/SxSUeKWeiduWcHha5dpqo0nVXrs9M51HTpiRYL/pRSDmoejf/6b6g+HIDDokgIlM0zQ045kvMHz66+DLPFT9gbogJpp1AmMQ3XQ+Nhg0f/15Pvt8xWA2Lmaoyx55SXyRn3lQzM9bxz8V/sQowfhj0pg84NDkJF0/YxEX6zUSmZIQIvuDZF63vAXKYONxvUIGFAQXyo/OXpey1iM8GYt/WnZSldC7grAjIk/wgKt7r3R5jRyGYS0Nw== X-Forefront-Antispam-Report: CIP:216.228.112.35; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid02.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(6286002)(7696005)(82310400003)(36860700001)(7636003)(16526019)(107886003)(6916009)(83380400001)(5660300002)(426003)(47076005)(4326008)(36756003)(336012)(356005)(2616005)(8936002)(6666004)(54906003)(30864003)(8676002)(36906005)(316002)(2876002)(70586007)(508600001)(70206006)(55016002)(86362001)(2906002)(1076003)(186003)(26005); DIR:OUT; SFP:1101; X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Sep 2021 14:53:11.0165 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d1d8ccd1-6f17-45e5-a7c5-08d98358db81 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.35]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT026.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4609 Subject: [dpdk-dev] [PATCH v2 3/4] common/mlx5: add mempool registration facilities X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Dmitry Kozlyuk Add internal API to register mempools, that is, to create memory regions (MR) for their memory and store them in a separate database. Implementation deals with multi-process, so that class drivers don't need to. Each protection domain has its own database. Memory regions can be shared within a database if they represent a single hugepage covering one or more mempools entirely. Add internal API to lookup an MR key for an address that belongs to a known mempool. It is a responsibility of a class driver to extract the mempool from an mbuf. Signed-off-by: Dmitry Kozlyuk Acked-by: Matan Azrad --- drivers/common/mlx5/mlx5_common_mp.c | 50 +++ drivers/common/mlx5/mlx5_common_mp.h | 14 + drivers/common/mlx5/mlx5_common_mr.c | 580 +++++++++++++++++++++++++++ drivers/common/mlx5/mlx5_common_mr.h | 17 + drivers/common/mlx5/version.map | 5 + 5 files changed, 666 insertions(+) diff --git a/drivers/common/mlx5/mlx5_common_mp.c b/drivers/common/mlx5/mlx5_common_mp.c index 673a7c31de..6dfc5535e0 100644 --- a/drivers/common/mlx5/mlx5_common_mp.c +++ b/drivers/common/mlx5/mlx5_common_mp.c @@ -54,6 +54,56 @@ mlx5_mp_req_mr_create(struct mlx5_mp_id *mp_id, uintptr_t addr) return ret; } +/** + * @param mp_id + * ID of the MP process. + * @param share_cache + * Shared MR cache. + * @param pd + * Protection domain. + * @param mempool + * Mempool to register or unregister. + * @param reg + * True to register the mempool, False to unregister. + */ +int +mlx5_mp_req_mempool_reg(struct mlx5_mp_id *mp_id, + struct mlx5_mr_share_cache *share_cache, void *pd, + struct rte_mempool *mempool, bool reg) +{ + struct rte_mp_msg mp_req; + struct rte_mp_msg *mp_res; + struct rte_mp_reply mp_rep; + struct mlx5_mp_param *req = (struct mlx5_mp_param *)mp_req.param; + struct mlx5_mp_arg_mempool_reg *arg = &req->args.mempool_reg; + struct mlx5_mp_param *res; + struct timespec ts = {.tv_sec = MLX5_MP_REQ_TIMEOUT_SEC, .tv_nsec = 0}; + enum mlx5_mp_req_type type; + int ret; + + MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_SECONDARY); + type = reg ? MLX5_MP_REQ_MEMPOOL_REGISTER : + MLX5_MP_REQ_MEMPOOL_UNREGISTER; + mp_init_msg(mp_id, &mp_req, type); + arg->share_cache = share_cache; + arg->pd = pd; + arg->mempool = mempool; + ret = rte_mp_request_sync(&mp_req, &mp_rep, &ts); + if (ret) { + DRV_LOG(ERR, "port %u request to primary process failed", + mp_id->port_id); + return -rte_errno; + } + MLX5_ASSERT(mp_rep.nb_received == 1); + mp_res = &mp_rep.msgs[0]; + res = (struct mlx5_mp_param *)mp_res->param; + ret = res->result; + if (ret) + rte_errno = -ret; + mlx5_free(mp_rep.msgs); + return ret; +} + /** * Request Verbs queue state modification to the primary process. * diff --git a/drivers/common/mlx5/mlx5_common_mp.h b/drivers/common/mlx5/mlx5_common_mp.h index 6829141fc7..527bf3cad8 100644 --- a/drivers/common/mlx5/mlx5_common_mp.h +++ b/drivers/common/mlx5/mlx5_common_mp.h @@ -14,6 +14,8 @@ enum mlx5_mp_req_type { MLX5_MP_REQ_VERBS_CMD_FD = 1, MLX5_MP_REQ_CREATE_MR, + MLX5_MP_REQ_MEMPOOL_REGISTER, + MLX5_MP_REQ_MEMPOOL_UNREGISTER, MLX5_MP_REQ_START_RXTX, MLX5_MP_REQ_STOP_RXTX, MLX5_MP_REQ_QUEUE_STATE_MODIFY, @@ -33,6 +35,12 @@ struct mlx5_mp_arg_queue_id { uint16_t queue_id; /* DPDK queue ID. */ }; +struct mlx5_mp_arg_mempool_reg { + struct mlx5_mr_share_cache *share_cache; + void *pd; /* NULL for MLX5_MP_REQ_MEMPOOL_UNREGISTER */ + struct rte_mempool *mempool; +}; + /* Pameters for IPC. */ struct mlx5_mp_param { enum mlx5_mp_req_type type; @@ -41,6 +49,8 @@ struct mlx5_mp_param { RTE_STD_C11 union { uintptr_t addr; /* MLX5_MP_REQ_CREATE_MR */ + struct mlx5_mp_arg_mempool_reg mempool_reg; + /* MLX5_MP_REQ_MEMPOOL_(UN)REGISTER */ struct mlx5_mp_arg_queue_state_modify state_modify; /* MLX5_MP_REQ_QUEUE_STATE_MODIFY */ struct mlx5_mp_arg_queue_id queue_id; @@ -91,6 +101,10 @@ void mlx5_mp_uninit_secondary(const char *name); __rte_internal int mlx5_mp_req_mr_create(struct mlx5_mp_id *mp_id, uintptr_t addr); __rte_internal +int mlx5_mp_req_mempool_reg(struct mlx5_mp_id *mp_id, + struct mlx5_mr_share_cache *share_cache, void *pd, + struct rte_mempool *mempool, bool reg); +__rte_internal int mlx5_mp_req_queue_state_modify(struct mlx5_mp_id *mp_id, struct mlx5_mp_arg_queue_state_modify *sm); __rte_internal diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c index 98fe8698e2..2e039a4e70 100644 --- a/drivers/common/mlx5/mlx5_common_mr.c +++ b/drivers/common/mlx5/mlx5_common_mr.c @@ -2,7 +2,10 @@ * Copyright 2016 6WIND S.A. * Copyright 2020 Mellanox Technologies, Ltd */ +#include + #include +#include #include #include #include @@ -21,6 +24,29 @@ struct mr_find_contig_memsegs_data { const struct rte_memseg_list *msl; }; +/* Virtual memory range. */ +struct mlx5_range { + uintptr_t start; + uintptr_t end; +}; + +/** Memory region for a mempool. */ +struct mlx5_mempool_mr { + struct mlx5_pmd_mr pmd_mr; + uint32_t refcnt; /**< Number of mempools sharing this MR. */ +}; + +/* Mempool registration. */ +struct mlx5_mempool_reg { + LIST_ENTRY(mlx5_mempool_reg) next; + /** Registered mempool, used to designate registrations. */ + struct rte_mempool *mp; + /** Memory regions for the address ranges of the mempool. */ + struct mlx5_mempool_mr *mrs; + /** Number of memory regions. */ + unsigned int mrs_n; +}; + /** * Expand B-tree table to a given size. Can't be called with holding * memory_hotplug_lock or share_cache.rwlock due to rte_realloc(). @@ -1191,3 +1217,557 @@ mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused) rte_rwlock_read_unlock(&share_cache->rwlock); #endif } + +static int +mlx5_range_compare_start(const void *lhs, const void *rhs) +{ + const struct mlx5_range *r1 = lhs, *r2 = rhs; + + if (r1->start > r2->start) + return 1; + else if (r1->start < r2->start) + return -1; + return 0; +} + +static void +mlx5_range_from_mempool_chunk(struct rte_mempool *mp, void *opaque, + struct rte_mempool_memhdr *memhdr, + unsigned int idx) +{ + struct mlx5_range *ranges = opaque, *range = &ranges[idx]; + uint64_t page_size = rte_mem_page_size(); + + RTE_SET_USED(mp); + range->start = RTE_ALIGN_FLOOR((uintptr_t)memhdr->addr, page_size); + range->end = RTE_ALIGN_CEIL(range->start + memhdr->len, page_size); +} + +/** + * Get VA-contiguous ranges of the mempool memory. + * Each range start and end is aligned to the system page size. + * + * @param[in] mp + * Analyzed mempool. + * @param[out] out + * Receives the ranges, caller must release it with free(). + * @param[out] ount_n + * Receives the number of @p out elements. + * + * @return + * 0 on success, (-1) on failure. + */ +static int +mlx5_get_mempool_ranges(struct rte_mempool *mp, struct mlx5_range **out, + unsigned int *out_n) +{ + struct mlx5_range *chunks; + unsigned int chunks_n = mp->nb_mem_chunks, contig_n, i; + + /* Collect page-aligned memory ranges of the mempool. */ + chunks = calloc(sizeof(chunks[0]), chunks_n); + if (chunks == NULL) + return -1; + rte_mempool_mem_iter(mp, mlx5_range_from_mempool_chunk, chunks); + /* Merge adjacent chunks and place them at the beginning. */ + qsort(chunks, chunks_n, sizeof(chunks[0]), mlx5_range_compare_start); + contig_n = 1; + for (i = 1; i < chunks_n; i++) + if (chunks[i - 1].end != chunks[i].start) { + chunks[contig_n - 1].end = chunks[i - 1].end; + chunks[contig_n] = chunks[i]; + contig_n++; + } + /* Extend the last contiguous chunk to the end of the mempool. */ + chunks[contig_n - 1].end = chunks[i - 1].end; + *out = chunks; + *out_n = contig_n; + return 0; +} + +/** + * Analyze mempool memory to select memory ranges to register. + * + * @param[in] mp + * Mempool to analyze. + * @param[out] out + * Receives memory ranges to register, aligned to the system page size. + * The caller must release them with free(). + * @param[out] out_n + * Receives the number of @p out items. + * @param[out] share_hugepage + * Receives True if the entire pool resides within a single hugepage. + * + * @return + * 0 on success, (-1) on failure. + */ +static int +mlx5_mempool_reg_analyze(struct rte_mempool *mp, struct mlx5_range **out, + unsigned int *out_n, bool *share_hugepage) +{ + struct mlx5_range *ranges = NULL; + unsigned int i, ranges_n = 0; + struct rte_memseg_list *msl; + + if (mlx5_get_mempool_ranges(mp, &ranges, &ranges_n) < 0) { + DRV_LOG(ERR, "Cannot get address ranges for mempool %s", + mp->name); + return -1; + } + /* Check if the hugepage of the pool can be shared. */ + *share_hugepage = false; + msl = rte_mem_virt2memseg_list((void *)ranges[0].start); + if (msl != NULL) { + uint64_t hugepage_sz = 0; + + /* Check that all ranges are on pages of the same size. */ + for (i = 0; i < ranges_n; i++) { + if (hugepage_sz != 0 && hugepage_sz != msl->page_sz) + break; + hugepage_sz = msl->page_sz; + } + if (i == ranges_n) { + /* + * If the entire pool is within one hugepage, + * combine all ranges into one of the hugepage size. + */ + uintptr_t reg_start = ranges[0].start; + uintptr_t reg_end = ranges[ranges_n - 1].end; + uintptr_t hugepage_start = + RTE_ALIGN_FLOOR(reg_start, hugepage_sz); + uintptr_t hugepage_end = hugepage_start + hugepage_sz; + if (reg_end < hugepage_end) { + ranges[0].start = hugepage_start; + ranges[0].end = hugepage_end; + ranges_n = 1; + *share_hugepage = true; + } + } + } + *out = ranges; + *out_n = ranges_n; + return 0; +} + +/** Create a registration object for the mempool. */ +static struct mlx5_mempool_reg * +mlx5_mempool_reg_create(struct rte_mempool *mp, unsigned int mrs_n) +{ + struct mlx5_mempool_reg *mpr = NULL; + + mpr = mlx5_malloc(MLX5_MEM_RTE | MLX5_MEM_ZERO, + sizeof(*mpr) + mrs_n * sizeof(mpr->mrs[0]), + RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY); + if (mpr == NULL) { + DRV_LOG(ERR, "Cannot allocate mempool %s registration object", + mp->name); + return NULL; + } + mpr->mp = mp; + mpr->mrs = (struct mlx5_mempool_mr *)(mpr + 1); + mpr->mrs_n = mrs_n; + return mpr; +} + +/** + * Destroy a mempool registration object. + * + * @param standalone + * Whether @p mpr owns its MRs excludively, i.e. they are not shared. + */ +static void +mlx5_mempool_reg_destroy(struct mlx5_mr_share_cache *share_cache, + struct mlx5_mempool_reg *mpr, bool standalone) +{ + if (standalone) { + unsigned int i; + + for (i = 0; i < mpr->mrs_n; i++) + share_cache->dereg_mr_cb(&mpr->mrs[i].pmd_mr); + } + mlx5_free(mpr); +} + +/** Find registration object of a mempool. */ +static struct mlx5_mempool_reg * +mlx5_mempool_reg_lookup(struct mlx5_mr_share_cache *share_cache, + struct rte_mempool *mp) +{ + struct mlx5_mempool_reg *mpr; + + LIST_FOREACH(mpr, &share_cache->mempool_reg_list, next) + if (mpr->mp == mp) + break; + return mpr; +} + +/** Increment reference counters of MRs used in the registration. */ +static void +mlx5_mempool_reg_attach(struct mlx5_mempool_reg *mpr) +{ + unsigned int i; + + for (i = 0; i < mpr->mrs_n; i++) + __atomic_add_fetch(&mpr->mrs[i].refcnt, 1, __ATOMIC_RELAXED); +} + +/** + * Decrement reference counters of MRs used in the registration. + * + * @return True if no more references to @p mpr MRs exist, False otherwise. + */ +static bool +mlx5_mempool_reg_detach(struct mlx5_mempool_reg *mpr) +{ + unsigned int i; + bool ret = false; + + for (i = 0; i < mpr->mrs_n; i++) + ret |= __atomic_sub_fetch(&mpr->mrs[i].refcnt, 1, + __ATOMIC_RELAXED) == 0; + return ret; +} + +static int +mlx5_mr_mempool_register_primary(struct mlx5_mr_share_cache *share_cache, + void *pd, struct rte_mempool *mp) +{ + struct mlx5_range *ranges = NULL; + struct mlx5_mempool_reg *mpr, *new_mpr; + unsigned int i, ranges_n; + bool share_hugepage; + int ret = -1; + + /* Early check to avoid unnecessary creation of MRs. */ + rte_rwlock_read_lock(&share_cache->rwlock); + mpr = mlx5_mempool_reg_lookup(share_cache, mp); + rte_rwlock_read_unlock(&share_cache->rwlock); + if (mpr != NULL) { + DRV_LOG(DEBUG, "Mempool %s is already registered for PD %p", + mp->name, pd); + rte_errno = EEXIST; + goto exit; + } + if (mlx5_mempool_reg_analyze(mp, &ranges, &ranges_n, + &share_hugepage) < 0) { + DRV_LOG(ERR, "Cannot get mempool %s memory ranges", mp->name); + rte_errno = ENOMEM; + goto exit; + } + new_mpr = mlx5_mempool_reg_create(mp, ranges_n); + if (new_mpr == NULL) { + DRV_LOG(ERR, + "Cannot create a registration object for mempool %s in PD %p", + mp->name, pd); + rte_errno = ENOMEM; + goto exit; + } + /* + * If the entire mempool fits in a single hugepage, the MR for this + * hugepage can be shared across mempools that also fit in it. + */ + if (share_hugepage) { + rte_rwlock_write_lock(&share_cache->rwlock); + LIST_FOREACH(mpr, &share_cache->mempool_reg_list, next) { + if (mpr->mrs[0].pmd_mr.addr == (void *)ranges[0].start) + break; + } + if (mpr != NULL) { + new_mpr->mrs = mpr->mrs; + mlx5_mempool_reg_attach(new_mpr); + LIST_INSERT_HEAD(&share_cache->mempool_reg_list, + new_mpr, next); + } + rte_rwlock_write_unlock(&share_cache->rwlock); + if (mpr != NULL) { + DRV_LOG(DEBUG, "Shared MR %#x in PD %p for mempool %s with mempool %s", + mpr->mrs[0].pmd_mr.lkey, pd, mp->name, + mpr->mp->name); + ret = 0; + goto exit; + } + } + for (i = 0; i < ranges_n; i++) { + struct mlx5_mempool_mr *mr = &new_mpr->mrs[i]; + const struct mlx5_range *range = &ranges[i]; + size_t len = range->end - range->start; + + if (share_cache->reg_mr_cb(pd, (void *)range->start, len, + &mr->pmd_mr) < 0) { + DRV_LOG(ERR, + "Failed to create an MR in PD %p for address range " + "[0x%" PRIxPTR ", 0x%" PRIxPTR "] (%zu bytes) for mempool %s", + pd, range->start, range->end, len, mp->name); + break; + } + DRV_LOG(DEBUG, + "Created a new MR %#x in PD %p for address range " + "[0x%" PRIxPTR ", 0x%" PRIxPTR "] (%zu bytes) for mempool %s", + mr->pmd_mr.lkey, pd, range->start, range->end, len, + mp->name); + } + if (i != ranges_n) { + mlx5_mempool_reg_destroy(share_cache, new_mpr, true); + rte_errno = EINVAL; + goto exit; + } + /* Concurrent registration is not supposed to happen. */ + rte_rwlock_write_lock(&share_cache->rwlock); + mpr = mlx5_mempool_reg_lookup(share_cache, mp); + if (mpr == NULL) { + mlx5_mempool_reg_attach(new_mpr); + LIST_INSERT_HEAD(&share_cache->mempool_reg_list, + new_mpr, next); + ret = 0; + } + rte_rwlock_write_unlock(&share_cache->rwlock); + if (mpr != NULL) { + DRV_LOG(DEBUG, "Mempool %s is already registered for PD %p", + mp->name, pd); + mlx5_mempool_reg_destroy(share_cache, new_mpr, true); + rte_errno = EEXIST; + goto exit; + } +exit: + free(ranges); + return ret; +} + +static int +mlx5_mr_mempool_register_secondary(struct mlx5_mr_share_cache *share_cache, + void *pd, struct rte_mempool *mp, + struct mlx5_mp_id *mp_id) +{ + if (mp_id == NULL) { + rte_errno = EINVAL; + return -1; + } + return mlx5_mp_req_mempool_reg(mp_id, share_cache, pd, mp, true); +} + +/** + * Register the memory of a mempool in the protection domain. + * + * @param share_cache + * Shared MR cache of the protection domain. + * @param pd + * Protection domain object. + * @param mp + * Mempool to register. + * @param mp_id + * Multi-process identifier, may be NULL for the primary process. + * + * @return + * 0 on success, (-1) on failure and rte_errno is set. + */ +int +mlx5_mr_mempool_register(struct mlx5_mr_share_cache *share_cache, void *pd, + struct rte_mempool *mp, struct mlx5_mp_id *mp_id) +{ + if (mp->flags & MEMPOOL_F_NON_IO) + return 0; + switch (rte_eal_process_type()) { + case RTE_PROC_PRIMARY: + return mlx5_mr_mempool_register_primary(share_cache, pd, mp); + case RTE_PROC_SECONDARY: + return mlx5_mr_mempool_register_secondary(share_cache, pd, mp, + mp_id); + default: + return -1; + } +} + +static int +mlx5_mr_mempool_unregister_primary(struct mlx5_mr_share_cache *share_cache, + struct rte_mempool *mp) +{ + struct mlx5_mempool_reg *mpr; + bool standalone = false; + + rte_rwlock_write_lock(&share_cache->rwlock); + LIST_FOREACH(mpr, &share_cache->mempool_reg_list, next) + if (mpr->mp == mp) { + LIST_REMOVE(mpr, next); + standalone = mlx5_mempool_reg_detach(mpr); + if (standalone) + /* + * The unlock operation below provides a memory + * barrier due to its store-release semantics. + */ + ++share_cache->dev_gen; + break; + } + rte_rwlock_write_unlock(&share_cache->rwlock); + if (mpr == NULL) { + rte_errno = ENOENT; + return -1; + } + mlx5_mempool_reg_destroy(share_cache, mpr, standalone); + return 0; +} + +static int +mlx5_mr_mempool_unregister_secondary(struct mlx5_mr_share_cache *share_cache, + struct rte_mempool *mp, + struct mlx5_mp_id *mp_id) +{ + if (mp_id == NULL) { + rte_errno = EINVAL; + return -1; + } + return mlx5_mp_req_mempool_reg(mp_id, share_cache, NULL, mp, false); +} + +/** + * Unregister the memory of a mempool from the protection domain. + * + * @param share_cache + * Shared MR cache of the protection domain. + * @param mp + * Mempool to unregister. + * @param mp_id + * Multi-process identifier, may be NULL for the primary process. + * + * @return + * 0 on success, (-1) on failure and rte_errno is set. + */ +int +mlx5_mr_mempool_unregister(struct mlx5_mr_share_cache *share_cache, + struct rte_mempool *mp, struct mlx5_mp_id *mp_id) +{ + if (mp->flags & MEMPOOL_F_NON_IO) + return 0; + switch (rte_eal_process_type()) { + case RTE_PROC_PRIMARY: + return mlx5_mr_mempool_unregister_primary(share_cache, mp); + case RTE_PROC_SECONDARY: + return mlx5_mr_mempool_unregister_secondary(share_cache, mp, + mp_id); + default: + return -1; + } +} + +/** + * Lookup a MR key by and address in a registered mempool. + * + * @param mpr + * Mempool registration object. + * @param addr + * Address within the mempool. + * @param entry + * Bottom-half cache entry to fill. + * + * @return + * MR key or UINT32_MAX on failure, which can only happen + * if the address is not from within the mempool. + */ +static uint32_t +mlx5_mempool_reg_addr2mr(struct mlx5_mempool_reg *mpr, uintptr_t addr, + struct mr_cache_entry *entry) +{ + uint32_t lkey = UINT32_MAX; + unsigned int i; + + for (i = 0; i < mpr->mrs_n; i++) { + const struct mlx5_pmd_mr *mr = &mpr->mrs[i].pmd_mr; + uintptr_t mr_addr = (uintptr_t)mr->addr; + + if (mr_addr <= addr) { + lkey = rte_cpu_to_be_32(mr->lkey); + entry->start = mr_addr; + entry->end = mr_addr + mr->len; + entry->lkey = lkey; + break; + } + } + return lkey; +} + +/** + * Update bottom-half cache from the list of mempool registrations. + * + * @param share_cache + * Pointer to a global shared MR cache. + * @param mr_ctrl + * Per-queue MR control handle. + * @param entry + * Pointer to an entry in the bottom-half cache to update + * with the MR lkey looked up. + * @param mp + * Mempool containing the address. + * @param addr + * Address to lookup. + * @return + * MR lkey on success, UINT32_MAX on failure. + */ +static uint32_t +mlx5_lookup_mempool_regs(struct mlx5_mr_share_cache *share_cache, + struct mlx5_mr_ctrl *mr_ctrl, + struct mr_cache_entry *entry, + struct rte_mempool *mp, uintptr_t addr) +{ + struct mlx5_mr_btree *bt = &mr_ctrl->cache_bh; + struct mlx5_mempool_reg *mpr; + uint32_t lkey = UINT32_MAX; + + /* If local cache table is full, try to double it. */ + if (unlikely(bt->len == bt->size)) + mr_btree_expand(bt, bt->size << 1); + /* Look up in mempool registrations. */ + rte_rwlock_read_lock(&share_cache->rwlock); + mpr = mlx5_mempool_reg_lookup(share_cache, mp); + if (mpr != NULL) + lkey = mlx5_mempool_reg_addr2mr(mpr, addr, entry); + rte_rwlock_read_unlock(&share_cache->rwlock); + /* + * Update local cache. Even if it fails, return the found entry + * to update top-half cache. Next time, this entry will be found + * in the global cache. + */ + if (lkey != UINT32_MAX) + mr_btree_insert(bt, entry); + return lkey; +} + +/** + * Bottom-half lookup for the address from the mempool. + * + * @param share_cache + * Pointer to a global shared MR cache. + * @param mr_ctrl + * Per-queue MR control handle. + * @param mp + * Mempool containing the address. + * @param addr + * Address to lookup. + * @return + * MR lkey on success, UINT32_MAX on failure. + */ +uint32_t +mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache, + struct mlx5_mr_ctrl *mr_ctrl, + struct rte_mempool *mp, uintptr_t addr) +{ + struct mr_cache_entry *repl = &mr_ctrl->cache[mr_ctrl->head]; + uint32_t lkey; + uint16_t bh_idx = 0; + + /* Binary-search MR translation table. */ + lkey = mr_btree_lookup(&mr_ctrl->cache_bh, &bh_idx, addr); + /* Update top-half cache. */ + if (likely(lkey != UINT32_MAX)) { + *repl = (*mr_ctrl->cache_bh.table)[bh_idx]; + } else { + lkey = mlx5_lookup_mempool_regs(share_cache, mr_ctrl, repl, + mp, addr); + /* Can only fail if the address is not from the mempool. */ + if (unlikely(lkey == UINT32_MAX)) + return UINT32_MAX; + } + /* Update the most recently used entry. */ + mr_ctrl->mru = mr_ctrl->head; + /* Point to the next victim, the oldest. */ + mr_ctrl->head = (mr_ctrl->head + 1) % MLX5_MR_CACHE_N; + return lkey; +} diff --git a/drivers/common/mlx5/mlx5_common_mr.h b/drivers/common/mlx5/mlx5_common_mr.h index 6e465a05e9..685ac98e08 100644 --- a/drivers/common/mlx5/mlx5_common_mr.h +++ b/drivers/common/mlx5/mlx5_common_mr.h @@ -13,6 +13,7 @@ #include #include +#include #include #include "mlx5_glue.h" @@ -75,6 +76,7 @@ struct mlx5_mr_ctrl { } __rte_packed; LIST_HEAD(mlx5_mr_list, mlx5_mr); +LIST_HEAD(mlx5_mempool_reg_list, mlx5_mempool_reg); /* Global per-device MR cache. */ struct mlx5_mr_share_cache { @@ -83,6 +85,7 @@ struct mlx5_mr_share_cache { struct mlx5_mr_btree cache; /* Global MR cache table. */ struct mlx5_mr_list mr_list; /* Registered MR list. */ struct mlx5_mr_list mr_free_list; /* Freed MR list. */ + struct mlx5_mempool_reg_list mempool_reg_list; /* Mempool database. */ mlx5_reg_mr_t reg_mr_cb; /* Callback to reg_mr func */ mlx5_dereg_mr_t dereg_mr_cb; /* Callback to dereg_mr func */ } __rte_packed; @@ -136,6 +139,10 @@ uint32_t mlx5_mr_addr2mr_bh(void *pd, struct mlx5_mp_id *mp_id, struct mlx5_mr_ctrl *mr_ctrl, uintptr_t addr, unsigned int mr_ext_memseg_en); __rte_internal +uint32_t mlx5_mr_mempool2mr_bh(struct mlx5_mr_share_cache *share_cache, + struct mlx5_mr_ctrl *mr_ctrl, + struct rte_mempool *mp, uintptr_t addr); +__rte_internal void mlx5_mr_release_cache(struct mlx5_mr_share_cache *mr_cache); __rte_internal void mlx5_mr_dump_cache(struct mlx5_mr_share_cache *share_cache __rte_unused); @@ -179,4 +186,14 @@ mlx5_common_verbs_dereg_mr(struct mlx5_pmd_mr *pmd_mr); __rte_internal void mlx5_mr_free(struct mlx5_mr *mr, mlx5_dereg_mr_t dereg_mr_cb); + +__rte_internal +int +mlx5_mr_mempool_register(struct mlx5_mr_share_cache *share_cache, void *pd, + struct rte_mempool *mp, struct mlx5_mp_id *mp_id); +__rte_internal +int +mlx5_mr_mempool_unregister(struct mlx5_mr_share_cache *share_cache, + struct rte_mempool *mp, struct mlx5_mp_id *mp_id); + #endif /* RTE_PMD_MLX5_COMMON_MR_H_ */ diff --git a/drivers/common/mlx5/version.map b/drivers/common/mlx5/version.map index e5cb6b7060..d5e9635a14 100644 --- a/drivers/common/mlx5/version.map +++ b/drivers/common/mlx5/version.map @@ -149,4 +149,9 @@ INTERNAL { mlx5_realloc; mlx5_translate_port_name; # WINDOWS_NO_EXPORT + + mlx5_mr_mempool_register; + mlx5_mr_mempool_unregister; + mlx5_mp_req_mempool_reg; + mlx5_mr_mempool2mr_bh; }; From patchwork Wed Sep 29 14:52:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dmitry Kozlyuk X-Patchwork-Id: 100005 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 02BC2A0547; Wed, 29 Sep 2021 16:53:39 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CDBA641123; Wed, 29 Sep 2021 16:53:20 +0200 (CEST) Received: from AZHDRRW-EX02.NVIDIA.COM (azhdrrw-ex02.nvidia.com [20.64.145.131]) by mails.dpdk.org (Postfix) with ESMTP id 52B32410F8 for ; Wed, 29 Sep 2021 16:53:16 +0200 (CEST) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.171) by mxs.oss.nvidia.com (10.13.234.37) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.858.15; Wed, 29 Sep 2021 07:53:14 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=O4nybuXwgzJ7jnfY/jGHJcyB88KRFM+hDldA+RXMW6gmUWTVbiTpO3Ss7xC9tfz6t15H8NP/TICq+WFs+Pu/Bo60uPiNV6ACiNn01XrxMDvPJ0Oflt4qD9I1Owng/mX5F5Jb5kAn4h+U633zAoZvHmE2lSqVhJGkat+202d8uBKm7edQmGxLoApxFKnDKQVSg/e+iXitme1TjlRcpu205siwEV0WkFEcz6XWbkRHUh4gbHkDJuXksfIjSSetl3MxJQDgFYXzqjy6ajiAnlQTRFGnblA/hYEeppsvxeCIyrihHF/vSWedo+/Oer/Hu0ao1+q49yT+7XWhg8DO52CgdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=45En4/UTddHYiaYH5+wGO4vOR97JPbEpC7XWybNPFIQ=; b=fCRh7GmqjMPJE/SKRtlMpHXW8hx6NceXdYF8oWBlaGtkZHeQDPeV7TUlc/SfJmoTEP7yGyvt/k4+XXrEZU3UOfq3N9zd3KFYcjosXnHFYwSFuhKcCl1BV/faiJOxJw4S59SLV95M4IvLWI4zmT4FDVTrzlmzvp+KLDmD1Aaz1VNEpwOEx4+EEZtD0TIKGD5HfkInuwnOBPOSn8a8p0RBaNY1v8lViCuHfj7pYkezcAM0MXTBnjx34YaiAKLW3RKdiDi7FT4towP4347u9Su+kH8HonocikasGzNxyngIpRIx9+mneywdLW6Whe3zHj6leDYcHYyu0nNd0/F+Kcnxvg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=45En4/UTddHYiaYH5+wGO4vOR97JPbEpC7XWybNPFIQ=; b=ZIR+E138aoSmxQIJJNc+XFf210EQihnnoQUVFPJ7olSNmIfs6/HRoZpL+h07Q8OKnim88NRe4B2fBhn12OQHOfRZML1bWk91E1CfcsFX3FQMQPFAeOQBvNNqnYcF26k80rBWy+kWSHWBinKHPzGOhwnfg/RrxFV5UVxp4/V4tq7Mg9bFe/2fXtbv3uDqlswnDD9LkOLcgcJVjcDyINMa8aseqmE6BBMch0Fe94CXho/GJ5kOLlgf5KYDC9Nu3JEul9E6Y61BXnr++x+MtxXpwdvYnSnXQwqPhbxmNbjNbylRThWmIDXZeMi8c75TQDeLtT+17ej+eAdU/J7pzGLHuw== Received: from DM5PR12CA0011.namprd12.prod.outlook.com (2603:10b6:4:1::21) by BN6PR1201MB0195.namprd12.prod.outlook.com (2603:10b6:405:53::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.15; Wed, 29 Sep 2021 14:53:13 +0000 Received: from DM6NAM11FT060.eop-nam11.prod.protection.outlook.com (2603:10b6:4:1:cafe::8c) by DM5PR12CA0011.outlook.office365.com (2603:10b6:4:1::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4566.13 via Frontend Transport; Wed, 29 Sep 2021 14:53:13 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by DM6NAM11FT060.mail.protection.outlook.com (10.13.173.63) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4566.14 via Frontend Transport; Wed, 29 Sep 2021 14:53:12 +0000 Received: from DRHQMAIL107.nvidia.com (10.27.9.16) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 29 Sep 2021 07:53:12 -0700 Received: from nvidia.com (172.20.187.5) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 29 Sep 2021 14:53:10 +0000 From: To: CC: Dmitry Kozlyuk , Matan Azrad , Viacheslav Ovsiienko Date: Wed, 29 Sep 2021 17:52:49 +0300 Message-ID: <20210929145249.2176811-5-dkozlyuk@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210929145249.2176811-1-dkozlyuk@nvidia.com> References: <20210818090755.2419483-1-dkozlyuk@nvidia.com> <20210929145249.2176811-1-dkozlyuk@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To DRHQMAIL107.nvidia.com (10.27.9.16) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 39eb58c0-9362-48b0-7ec0-08d98358dc9b X-MS-TrafficTypeDiagnostic: BN6PR1201MB0195: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2201; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: v3oACQe6xjEf6AU9DsyTqOZkJd02p50QYPXUsL5bdzMqWjrTb2+OLCGzjCEZ9MrVTwyuESPr+j5R62FAEPXqu1cHLbCc4ndm+x7UFv/D2CHDKx/MK1+d/umF41m3NSwGJOt1dlB0t+yEFA/Gq5Ylt08kKcQn8FU1Q7XJURVXe2JHVkBCs6oexUNuGsKXGGeLnO5ecp4bvFfvpKb1t/mr+veg7I2bysAvqGug/fO2SxkvDZfHAyulWREV0oe2UcQbcZZRpiOcp982J3eIEQf91A2Oi/stdgx+9EH4eIgRp3kJgFBZsC10Ax37E5sRknDc03sBZacrWtAUSqXSeVYkn9uTu10dBf5OFpeZJOInauHUbcCexoO/tO8vVZpgjhcujow47oSDcVNOJze2K/VbuTshEMhFRejrXZeUWkciHOskHarA6ONX1cn2iIgnQy/96kbjN8X/pnpvc36pUPsKamLEYussMjMT/l4uAdQrHE2gxUlZywaRVjt9bU92kjzWwmN3QZXn/ICsj7JMq1LS+DaW8XEsnteFEQZ0ULr6nbVMDl2K0CuWxiWkqTxPg1jiywEF/c4vCgPWzoZa2vQLDGV7shz5TbsTAEhqK8fGKfv7hxWiUiSIOjdvVZEFrEZNP8O/Sfk4N6EIyLV6/FDDmwnpXCcrbSTn3ZxFLbpEEv5EuscG8ueWlK1RHAQdD/JU1Y7uQ/mgANiyjW/1JsmttlBhbbZLaBrASCGcy3M+7yU= X-Forefront-Antispam-Report: CIP:216.228.112.32; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid01.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(30864003)(86362001)(2876002)(107886003)(6286002)(8676002)(70206006)(356005)(70586007)(36756003)(7636003)(7696005)(47076005)(83380400001)(6916009)(6666004)(26005)(16526019)(36860700001)(8936002)(508600001)(55016002)(186003)(426003)(336012)(2906002)(82310400003)(54906003)(2616005)(4326008)(5660300002)(1076003)(316002)(309714004); DIR:OUT; SFP:1101; X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Sep 2021 14:53:12.9595 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 39eb58c0-9362-48b0-7ec0-08d98358dc9b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.32]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT060.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR1201MB0195 Subject: [dpdk-dev] [PATCH v2 4/4] net/mlx5: support mempool registration X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Dmitry Kozlyuk When the first port in a given protection domain (PD) starts, install a mempool event callback for this PD and register all existing memory regions (MR) for it. When the last port in a PD closes, remove the callback and unregister all mempools for this PD. This behavior can be switched off with a new devarg: mr_mempool_reg_en. On TX slow path, i.e. when an MR key for the address of the buffer to send is not in the local cache, first try to retrieve it from the database of registered mempools. Supported are direct and indirect mbufs, as well as externally-attached ones from MLX5 MPRQ feature. Lookup in the database of non-mempool memory is used as the last resort. RX mempools are registered regardless of the devarg value. On RX data path only the local cache and the mempool database is used. If implicit mempool registration is disabled, these mempools are unregistered at port stop, releasing the MRs. Signed-off-by: Dmitry Kozlyuk Acked-by: Matan Azrad --- doc/guides/nics/mlx5.rst | 11 ++ doc/guides/rel_notes/release_21_11.rst | 6 + drivers/net/mlx5/linux/mlx5_mp_os.c | 44 +++++++ drivers/net/mlx5/linux/mlx5_os.c | 4 +- drivers/net/mlx5/mlx5.c | 152 +++++++++++++++++++++++++ drivers/net/mlx5/mlx5.h | 10 ++ drivers/net/mlx5/mlx5_mr.c | 120 +++++-------------- drivers/net/mlx5/mlx5_mr.h | 2 - drivers/net/mlx5/mlx5_rx.h | 21 ++-- drivers/net/mlx5/mlx5_rxq.c | 13 +++ drivers/net/mlx5/mlx5_trigger.c | 77 +++++++++++-- drivers/net/mlx5/windows/mlx5_os.c | 1 + 12 files changed, 345 insertions(+), 116 deletions(-) diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst index bae73f42d8..58d1c5b65c 100644 --- a/doc/guides/nics/mlx5.rst +++ b/doc/guides/nics/mlx5.rst @@ -1001,6 +1001,17 @@ Driver options Enabled by default. +- ``mr_mempool_reg_en`` parameter [int] + + A nonzero value enables implicit registration of DMA memory of all mempools + except those having ``MEMPOOL_F_NON_IO``. The effect is that when a packet + from a mempool is transmitted, its memory is already registered for DMA + in the PMD and no registration will happen on the data path. The tradeoff is + extra work on the creation of each mempool and increased HW resource use + if some mempools are not used with MLX5 devices. + + Enabled by default. + - ``representor`` parameter [list] This parameter can be used to instantiate DPDK Ethernet devices from diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst index 873beda633..1fc09faf96 100644 --- a/doc/guides/rel_notes/release_21_11.rst +++ b/doc/guides/rel_notes/release_21_11.rst @@ -106,6 +106,12 @@ New Features * Added tests to validate packets hard expiry. * Added tests to verify tunnel header verification in IPsec inbound. +* **Updated Mellanox mlx5 driver.** + + Updated the Mellanox mlx5 driver with new features and improvements, including: + + * Added implicit mempool registration to avoid data path hiccups (opt-out). + Removed Items ------------- diff --git a/drivers/net/mlx5/linux/mlx5_mp_os.c b/drivers/net/mlx5/linux/mlx5_mp_os.c index 3a4aa766f8..d2ac375a47 100644 --- a/drivers/net/mlx5/linux/mlx5_mp_os.c +++ b/drivers/net/mlx5/linux/mlx5_mp_os.c @@ -20,6 +20,45 @@ #include "mlx5_tx.h" #include "mlx5_utils.h" +/** + * Handle a port-agnostic message. + * + * @return + * 0 on success, 1 when message is not port-agnostic, (-1) on error. + */ +static int +mlx5_mp_os_handle_port_agnostic(const struct rte_mp_msg *mp_msg, + const void *peer) +{ + struct rte_mp_msg mp_res; + struct mlx5_mp_param *res = (struct mlx5_mp_param *)mp_res.param; + const struct mlx5_mp_param *param = + (const struct mlx5_mp_param *)mp_msg->param; + const struct mlx5_mp_arg_mempool_reg *mpr; + struct mlx5_mp_id mp_id; + + switch (param->type) { + case MLX5_MP_REQ_MEMPOOL_REGISTER: + mlx5_mp_id_init(&mp_id, param->port_id); + mp_init_msg(&mp_id, &mp_res, param->type); + mpr = ¶m->args.mempool_reg; + res->result = mlx5_mr_mempool_register(mpr->share_cache, + mpr->pd, mpr->mempool, + NULL); + return rte_mp_reply(&mp_res, peer); + case MLX5_MP_REQ_MEMPOOL_UNREGISTER: + mlx5_mp_id_init(&mp_id, param->port_id); + mp_init_msg(&mp_id, &mp_res, param->type); + mpr = ¶m->args.mempool_reg; + res->result = mlx5_mr_mempool_unregister(mpr->share_cache, + mpr->mempool, NULL); + return rte_mp_reply(&mp_res, peer); + default: + return 1; + } + return -1; +} + int mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer) { @@ -34,6 +73,11 @@ mlx5_mp_os_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer) int ret; MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY); + /* Port-agnostic messages. */ + ret = mlx5_mp_os_handle_port_agnostic(mp_msg, peer); + if (ret <= 0) + return ret; + /* Port-specific messages. */ if (!rte_eth_dev_is_valid_port(param->port_id)) { rte_errno = ENODEV; DRV_LOG(ERR, "port %u invalid port ID", param->port_id); diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 470b16cb9a..78ed588746 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -1034,8 +1034,7 @@ mlx5_dev_spawn(struct rte_device *dpdk_dev, err = mlx5_proc_priv_init(eth_dev); if (err) return NULL; - mp_id.port_id = eth_dev->data->port_id; - strlcpy(mp_id.name, MLX5_MP_NAME, RTE_MP_MAX_NAME_LEN); + mlx5_mp_id_init(&mp_id, eth_dev->data->port_id); /* Receive command fd from primary process */ err = mlx5_mp_req_verbs_cmd_fd(&mp_id); if (err < 0) @@ -2133,6 +2132,7 @@ mlx5_os_config_default(struct mlx5_dev_config *config) config->txqs_inline = MLX5_ARG_UNSET; config->vf_nl_en = 1; config->mr_ext_memseg_en = 1; + config->mr_mempool_reg_en = 1; config->mprq.max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN; config->mprq.min_rxqs_num = MLX5_MPRQ_MIN_RXQS; config->dv_esw_en = 1; diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index f84e061fe7..a16be05f7c 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -178,6 +178,9 @@ /* Device parameter to configure allow or prevent duplicate rules pattern. */ #define MLX5_ALLOW_DUPLICATE_PATTERN "allow_duplicate_pattern" +/* Device parameter to configure implicit registration of mempool memory. */ +#define MLX5_MR_MEMPOOL_REG_EN "mr_mempool_reg_en" + /* Shared memory between primary and secondary processes. */ struct mlx5_shared_data *mlx5_shared_data; @@ -1085,6 +1088,141 @@ mlx5_alloc_rxtx_uars(struct mlx5_dev_ctx_shared *sh, return err; } +/** + * Unregister the mempool from the protection domain. + * + * @param sh + * Pointer to the device shared context. + * @param mp + * Mempool being unregistered. + */ +static void +mlx5_dev_ctx_shared_mempool_unregister(struct mlx5_dev_ctx_shared *sh, + struct rte_mempool *mp) +{ + struct mlx5_mp_id mp_id; + + mlx5_mp_id_init(&mp_id, 0); + if (mlx5_mr_mempool_unregister(&sh->share_cache, mp, &mp_id) < 0) + DRV_LOG(WARNING, "Failed to unregister mempool %s for PD %p: %s", + mp->name, sh->pd, rte_strerror(rte_errno)); +} + +/** + * rte_mempool_walk() callback to register mempools + * for the protection domain. + * + * @param mp + * The mempool being walked. + * @param arg + * Pointer to the device shared context. + */ +static void +mlx5_dev_ctx_shared_mempool_register_cb(struct rte_mempool *mp, void *arg) +{ + struct mlx5_dev_ctx_shared *sh = arg; + struct mlx5_mp_id mp_id; + int ret; + + mlx5_mp_id_init(&mp_id, 0); + ret = mlx5_mr_mempool_register(&sh->share_cache, sh->pd, mp, &mp_id); + if (ret < 0 && rte_errno != EEXIST) + DRV_LOG(ERR, "Failed to register existing mempool %s for PD %p: %s", + mp->name, sh->pd, rte_strerror(rte_errno)); +} + +/** + * rte_mempool_walk() callback to unregister mempools + * from the protection domain. + * + * @param mp + * The mempool being walked. + * @param arg + * Pointer to the device shared context. + */ +static void +mlx5_dev_ctx_shared_mempool_unregister_cb(struct rte_mempool *mp, void *arg) +{ + mlx5_dev_ctx_shared_mempool_unregister + ((struct mlx5_dev_ctx_shared *)arg, mp); +} + +/** + * Mempool life cycle callback for Ethernet devices. + * + * @param event + * Mempool life cycle event. + * @param mp + * Associated mempool. + * @param arg + * Pointer to a device shared context. + */ +static void +mlx5_dev_ctx_shared_mempool_event_cb(enum rte_mempool_event event, + struct rte_mempool *mp, void *arg) +{ + struct mlx5_dev_ctx_shared *sh = arg; + struct mlx5_mp_id mp_id; + + switch (event) { + case RTE_MEMPOOL_EVENT_READY: + mlx5_mp_id_init(&mp_id, 0); + if (mlx5_mr_mempool_register(&sh->share_cache, sh->pd, mp, + &mp_id) < 0) + DRV_LOG(ERR, "Failed to register new mempool %s for PD %p: %s", + mp->name, sh->pd, rte_strerror(rte_errno)); + break; + case RTE_MEMPOOL_EVENT_DESTROY: + mlx5_dev_ctx_shared_mempool_unregister(sh, mp); + break; + } +} + +/** + * Callback used when implicit mempool registration is disabled + * in order to track Rx mempool destruction. + * + * @param event + * Mempool life cycle event. + * @param mp + * An Rx mempool registered explicitly when the port is started. + * @param arg + * Pointer to a device shared context. + */ +static void +mlx5_dev_ctx_shared_rx_mempool_event_cb(enum rte_mempool_event event, + struct rte_mempool *mp, void *arg) +{ + struct mlx5_dev_ctx_shared *sh = arg; + + if (event == RTE_MEMPOOL_EVENT_DESTROY) + mlx5_dev_ctx_shared_mempool_unregister(sh, mp); +} + +int +mlx5_dev_ctx_shared_mempool_subscribe(struct rte_eth_dev *dev) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_dev_ctx_shared *sh = priv->sh; + int ret; + + /* Check if we only need to track Rx mempool destruction. */ + if (!priv->config.mr_mempool_reg_en) { + ret = rte_mempool_event_callback_register + (mlx5_dev_ctx_shared_rx_mempool_event_cb, sh); + return ret == 0 || rte_errno == EEXIST ? 0 : ret; + } + /* Callback for this shared context may be already registered. */ + ret = rte_mempool_event_callback_register + (mlx5_dev_ctx_shared_mempool_event_cb, sh); + if (ret != 0 && rte_errno != EEXIST) + return ret; + /* Register mempools only once for this shared context. */ + if (ret == 0) + rte_mempool_walk(mlx5_dev_ctx_shared_mempool_register_cb, sh); + return 0; +} + /** * Allocate shared device context. If there is multiport device the * master and representors will share this context, if there is single @@ -1282,6 +1420,8 @@ mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn, void mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh) { + int ret; + pthread_mutex_lock(&mlx5_dev_ctx_list_mutex); #ifdef RTE_LIBRTE_MLX5_DEBUG /* Check the object presence in the list. */ @@ -1302,6 +1442,15 @@ mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh) MLX5_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY); if (--sh->refcnt) goto exit; + /* Stop watching for mempool events and unregister all mempools. */ + ret = rte_mempool_event_callback_unregister + (mlx5_dev_ctx_shared_mempool_event_cb, sh); + if (ret < 0 && rte_errno == ENOENT) + ret = rte_mempool_event_callback_unregister + (mlx5_dev_ctx_shared_rx_mempool_event_cb, sh); + if (ret == 0) + rte_mempool_walk(mlx5_dev_ctx_shared_mempool_unregister_cb, + sh); /* Remove from memory callback device list. */ rte_rwlock_write_lock(&mlx5_shared_data->mem_event_rwlock); LIST_REMOVE(sh, mem_event_cb); @@ -1991,6 +2140,8 @@ mlx5_args_check(const char *key, const char *val, void *opaque) config->decap_en = !!tmp; } else if (strcmp(MLX5_ALLOW_DUPLICATE_PATTERN, key) == 0) { config->allow_duplicate_pattern = !!tmp; + } else if (strcmp(MLX5_MR_MEMPOOL_REG_EN, key) == 0) { + config->mr_mempool_reg_en = !!tmp; } else { DRV_LOG(WARNING, "%s: unknown parameter", key); rte_errno = EINVAL; @@ -2051,6 +2202,7 @@ mlx5_args(struct mlx5_dev_config *config, struct rte_devargs *devargs) MLX5_SYS_MEM_EN, MLX5_DECAP_EN, MLX5_ALLOW_DUPLICATE_PATTERN, + MLX5_MR_MEMPOOL_REG_EN, NULL, }; struct rte_kvargs *kvlist; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index e02714e231..89630d569b 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -155,6 +155,13 @@ struct mlx5_flow_dump_ack { /** Key string for IPC. */ #define MLX5_MP_NAME "net_mlx5_mp" +/** Initialize a multi-process ID. */ +static inline void +mlx5_mp_id_init(struct mlx5_mp_id *mp_id, uint16_t port_id) +{ + mp_id->port_id = port_id; + strlcpy(mp_id->name, MLX5_MP_NAME, RTE_MP_MAX_NAME_LEN); +} LIST_HEAD(mlx5_dev_list, mlx5_dev_ctx_shared); @@ -270,6 +277,8 @@ struct mlx5_dev_config { unsigned int dv_miss_info:1; /* restore packet after partial hw miss */ unsigned int allow_duplicate_pattern:1; /* Allow/Prevent the duplicate rules pattern. */ + unsigned int mr_mempool_reg_en:1; + /* Allow/prevent implicit mempool memory registration. */ struct { unsigned int enabled:1; /* Whether MPRQ is enabled. */ unsigned int stride_num_n; /* Number of strides. */ @@ -1497,6 +1506,7 @@ struct mlx5_dev_ctx_shared * mlx5_alloc_shared_dev_ctx(const struct mlx5_dev_spawn_data *spawn, const struct mlx5_dev_config *config); void mlx5_free_shared_dev_ctx(struct mlx5_dev_ctx_shared *sh); +int mlx5_dev_ctx_shared_mempool_subscribe(struct rte_eth_dev *dev); void mlx5_free_table_hash_list(struct mlx5_priv *priv); int mlx5_alloc_table_hash_list(struct mlx5_priv *priv); void mlx5_set_min_inline(struct mlx5_dev_spawn_data *spawn, diff --git a/drivers/net/mlx5/mlx5_mr.c b/drivers/net/mlx5/mlx5_mr.c index 44afda731f..55d27b50b9 100644 --- a/drivers/net/mlx5/mlx5_mr.c +++ b/drivers/net/mlx5/mlx5_mr.c @@ -65,30 +65,6 @@ mlx5_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr, } } -/** - * Bottom-half of LKey search on Rx. - * - * @param rxq - * Pointer to Rx queue structure. - * @param addr - * Search key. - * - * @return - * Searched LKey on success, UINT32_MAX on no match. - */ -uint32_t -mlx5_rx_addr2mr_bh(struct mlx5_rxq_data *rxq, uintptr_t addr) -{ - struct mlx5_rxq_ctrl *rxq_ctrl = - container_of(rxq, struct mlx5_rxq_ctrl, rxq); - struct mlx5_mr_ctrl *mr_ctrl = &rxq->mr_ctrl; - struct mlx5_priv *priv = rxq_ctrl->priv; - - return mlx5_mr_addr2mr_bh(priv->sh->pd, &priv->mp_id, - &priv->sh->share_cache, mr_ctrl, addr, - priv->config.mr_ext_memseg_en); -} - /** * Bottom-half of LKey search on Tx. * @@ -128,9 +104,36 @@ mlx5_tx_addr2mr_bh(struct mlx5_txq_data *txq, uintptr_t addr) uint32_t mlx5_tx_mb2mr_bh(struct mlx5_txq_data *txq, struct rte_mbuf *mb) { + struct mlx5_txq_ctrl *txq_ctrl = + container_of(txq, struct mlx5_txq_ctrl, txq); + struct mlx5_mr_ctrl *mr_ctrl = &txq->mr_ctrl; + struct mlx5_priv *priv = txq_ctrl->priv; uintptr_t addr = (uintptr_t)mb->buf_addr; uint32_t lkey; + if (priv->config.mr_mempool_reg_en) { + struct rte_mempool *mp = NULL; + struct mlx5_mprq_buf *buf; + + if (!RTE_MBUF_HAS_EXTBUF(mb)) { + mp = mlx5_mb2mp(mb); + } else if (mb->shinfo->free_cb == mlx5_mprq_buf_free_cb) { + /* Recover MPRQ mempool. */ + buf = mb->shinfo->fcb_opaque; + mp = buf->mp; + } + if (mp != NULL) { + lkey = mlx5_mr_mempool2mr_bh(&priv->sh->share_cache, + mr_ctrl, mp, addr); + /* + * Lookup can only fail on invalid input, e.g. "addr" + * is not from "mp" or "mp" has MEMPOOL_F_NON_IO set. + */ + if (lkey != UINT32_MAX) + return lkey; + } + /* Fallback for generic mechanism in corner cases. */ + } lkey = mlx5_tx_addr2mr_bh(txq, addr); if (lkey == UINT32_MAX && rte_errno == ENXIO) { /* Mempool may have externally allocated memory. */ @@ -392,72 +395,3 @@ mlx5_tx_update_ext_mp(struct mlx5_txq_data *txq, uintptr_t addr, mlx5_mr_update_ext_mp(ETH_DEV(priv), mr_ctrl, mp); return mlx5_tx_addr2mr_bh(txq, addr); } - -/* Called during rte_mempool_mem_iter() by mlx5_mr_update_mp(). */ -static void -mlx5_mr_update_mp_cb(struct rte_mempool *mp __rte_unused, void *opaque, - struct rte_mempool_memhdr *memhdr, - unsigned mem_idx __rte_unused) -{ - struct mr_update_mp_data *data = opaque; - struct rte_eth_dev *dev = data->dev; - struct mlx5_priv *priv = dev->data->dev_private; - - uint32_t lkey; - - /* Stop iteration if failed in the previous walk. */ - if (data->ret < 0) - return; - /* Register address of the chunk and update local caches. */ - lkey = mlx5_mr_addr2mr_bh(priv->sh->pd, &priv->mp_id, - &priv->sh->share_cache, data->mr_ctrl, - (uintptr_t)memhdr->addr, - priv->config.mr_ext_memseg_en); - if (lkey == UINT32_MAX) - data->ret = -1; -} - -/** - * Register entire memory chunks in a Mempool. - * - * @param dev - * Pointer to Ethernet device. - * @param mr_ctrl - * Pointer to per-queue MR control structure. - * @param mp - * Pointer to registering Mempool. - * - * @return - * 0 on success, -1 on failure. - */ -int -mlx5_mr_update_mp(struct rte_eth_dev *dev, struct mlx5_mr_ctrl *mr_ctrl, - struct rte_mempool *mp) -{ - struct mr_update_mp_data data = { - .dev = dev, - .mr_ctrl = mr_ctrl, - .ret = 0, - }; - uint32_t flags = rte_pktmbuf_priv_flags(mp); - - if (flags & RTE_PKTMBUF_POOL_F_PINNED_EXT_BUF) { - /* - * The pinned external buffer should be registered for DMA - * operations by application. The mem_list of the pool contains - * the list of chunks with mbuf structures w/o built-in data - * buffers and DMA actually does not happen there, no need - * to create MR for these chunks. - */ - return 0; - } - DRV_LOG(DEBUG, "Port %u Rx queue registering mp %s " - "having %u chunks.", dev->data->port_id, - mp->name, mp->nb_mem_chunks); - rte_mempool_mem_iter(mp, mlx5_mr_update_mp_cb, &data); - if (data.ret < 0 && rte_errno == ENXIO) { - /* Mempool may have externally allocated memory. */ - return mlx5_mr_update_ext_mp(dev, mr_ctrl, mp); - } - return data.ret; -} diff --git a/drivers/net/mlx5/mlx5_mr.h b/drivers/net/mlx5/mlx5_mr.h index 4a7fab6df2..c984e777b5 100644 --- a/drivers/net/mlx5/mlx5_mr.h +++ b/drivers/net/mlx5/mlx5_mr.h @@ -22,7 +22,5 @@ void mlx5_mr_mem_event_cb(enum rte_mem_event event_type, const void *addr, size_t len, void *arg); -int mlx5_mr_update_mp(struct rte_eth_dev *dev, struct mlx5_mr_ctrl *mr_ctrl, - struct rte_mempool *mp); #endif /* RTE_PMD_MLX5_MR_H_ */ diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h index 3f2b99fb65..3952e64d50 100644 --- a/drivers/net/mlx5/mlx5_rx.h +++ b/drivers/net/mlx5/mlx5_rx.h @@ -275,13 +275,11 @@ uint16_t mlx5_rx_burst_vec(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t mlx5_rx_burst_mprq_vec(void *dpdk_rxq, struct rte_mbuf **pkts, uint16_t pkts_n); -/* mlx5_mr.c */ - -uint32_t mlx5_rx_addr2mr_bh(struct mlx5_rxq_data *rxq, uintptr_t addr); +static int mlx5_rxq_mprq_enabled(struct mlx5_rxq_data *rxq); /** - * Query LKey from a packet buffer for Rx. No need to flush local caches for Rx - * as mempool is pre-configured and static. + * Query LKey from a packet buffer for Rx. No need to flush local caches + * as the Rx mempool database entries are valid for the lifetime of the queue. * * @param rxq * Pointer to Rx queue structure. @@ -290,11 +288,14 @@ uint32_t mlx5_rx_addr2mr_bh(struct mlx5_rxq_data *rxq, uintptr_t addr); * * @return * Searched LKey on success, UINT32_MAX on no match. + * This function always succeeds on valid input. */ static __rte_always_inline uint32_t mlx5_rx_addr2mr(struct mlx5_rxq_data *rxq, uintptr_t addr) { struct mlx5_mr_ctrl *mr_ctrl = &rxq->mr_ctrl; + struct mlx5_rxq_ctrl *rxq_ctrl; + struct rte_mempool *mp; uint32_t lkey; /* Linear search on MR cache array. */ @@ -302,8 +303,14 @@ mlx5_rx_addr2mr(struct mlx5_rxq_data *rxq, uintptr_t addr) MLX5_MR_CACHE_N, addr); if (likely(lkey != UINT32_MAX)) return lkey; - /* Take slower bottom-half (Binary Search) on miss. */ - return mlx5_rx_addr2mr_bh(rxq, addr); + /* + * Slower search in the mempool database on miss. + * During queue creation rxq->sh is not yet set, so we use rxq_ctrl. + */ + rxq_ctrl = container_of(rxq, struct mlx5_rxq_ctrl, rxq); + mp = mlx5_rxq_mprq_enabled(rxq) ? rxq->mprq_mp : rxq->mp; + return mlx5_mr_mempool2mr_bh(&rxq_ctrl->priv->sh->share_cache, + mr_ctrl, mp, addr); } #define mlx5_rx_mb2mr(rxq, mb) mlx5_rx_addr2mr(rxq, (uintptr_t)((mb)->buf_addr)) diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index abd8ce7989..4696f0b851 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -1165,6 +1165,7 @@ mlx5_mprq_alloc_mp(struct rte_eth_dev *dev) unsigned int strd_sz_n = 0; unsigned int i; unsigned int n_ibv = 0; + int ret; if (!mlx5_mprq_enabled(dev)) return 0; @@ -1244,6 +1245,16 @@ mlx5_mprq_alloc_mp(struct rte_eth_dev *dev) rte_errno = ENOMEM; return -rte_errno; } + ret = mlx5_mr_mempool_register(&priv->sh->share_cache, priv->sh->pd, + mp, &priv->mp_id); + if (ret < 0 && rte_errno != EEXIST) { + ret = rte_errno; + DRV_LOG(ERR, "port %u failed to register a mempool for Multi-Packet RQ", + dev->data->port_id); + rte_mempool_free(mp); + rte_errno = ret; + return -rte_errno; + } priv->mprq_mp = mp; exit: /* Set mempool for each Rx queue. */ @@ -1446,6 +1457,8 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, /* rte_errno is already set. */ goto error; } + /* Rx queues don't use this pointer, but we want a valid structure. */ + tmpl->rxq.mr_ctrl.dev_gen_ptr = &priv->sh->share_cache.dev_gen; tmpl->socket = socket; if (dev->data->dev_conf.intr_conf.rxq) tmpl->irq = 1; diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index 54173bfacb..3cbf5816a1 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -105,6 +105,59 @@ mlx5_txq_start(struct rte_eth_dev *dev) return -rte_errno; } +/** + * Translate the chunk address to MR key in order to put in into the cache. + */ +static void +mlx5_rxq_mempool_register_cb(struct rte_mempool *mp, void *opaque, + struct rte_mempool_memhdr *memhdr, + unsigned int idx) +{ + struct mlx5_rxq_data *rxq = opaque; + + RTE_SET_USED(mp); + RTE_SET_USED(idx); + mlx5_rx_addr2mr(rxq, (uintptr_t)memhdr->addr); +} + +/** + * Register Rx queue mempools and fill the Rx queue cache. + * This function tolerates repeated mempool registration. + * + * @param[in] rxq_ctrl + * Rx queue control data. + * + * @return + * 0 on success, (-1) on failure and rte_errno is set. + */ +static int +mlx5_rxq_mempool_register(struct mlx5_rxq_ctrl *rxq_ctrl) +{ + struct mlx5_priv *priv = rxq_ctrl->priv; + struct rte_mempool *mp; + uint32_t s; + int ret = 0; + + mlx5_mr_flush_local_cache(&rxq_ctrl->rxq.mr_ctrl); + /* MPRQ mempool is registered on creation, just fill the cache. */ + if (mlx5_rxq_mprq_enabled(&rxq_ctrl->rxq)) { + rte_mempool_mem_iter(rxq_ctrl->rxq.mprq_mp, + mlx5_rxq_mempool_register_cb, + &rxq_ctrl->rxq); + return 0; + } + for (s = 0; s < rxq_ctrl->rxq.rxseg_n; s++) { + mp = rxq_ctrl->rxq.rxseg[s].mp; + ret = mlx5_mr_mempool_register(&priv->sh->share_cache, + priv->sh->pd, mp, &priv->mp_id); + if (ret < 0 && rte_errno != EEXIST) + return ret; + rte_mempool_mem_iter(mp, mlx5_rxq_mempool_register_cb, + &rxq_ctrl->rxq); + } + return 0; +} + /** * Stop traffic on Rx queues. * @@ -152,18 +205,13 @@ mlx5_rxq_start(struct rte_eth_dev *dev) if (!rxq_ctrl) continue; if (rxq_ctrl->type == MLX5_RXQ_TYPE_STANDARD) { - /* Pre-register Rx mempools. */ - if (mlx5_rxq_mprq_enabled(&rxq_ctrl->rxq)) { - mlx5_mr_update_mp(dev, &rxq_ctrl->rxq.mr_ctrl, - rxq_ctrl->rxq.mprq_mp); - } else { - uint32_t s; - - for (s = 0; s < rxq_ctrl->rxq.rxseg_n; s++) - mlx5_mr_update_mp - (dev, &rxq_ctrl->rxq.mr_ctrl, - rxq_ctrl->rxq.rxseg[s].mp); - } + /* + * Pre-register the mempools. Regardless of whether + * the implicit registration is enabled or not, + * Rx mempool destruction is tracked to free MRs. + */ + if (mlx5_rxq_mempool_register(rxq_ctrl) < 0) + goto error; ret = rxq_alloc_elts(rxq_ctrl); if (ret) goto error; @@ -1124,6 +1172,11 @@ mlx5_dev_start(struct rte_eth_dev *dev) dev->data->port_id, strerror(rte_errno)); goto error; } + if (mlx5_dev_ctx_shared_mempool_subscribe(dev) != 0) { + DRV_LOG(ERR, "port %u failed to subscribe for mempool life cycle: %s", + dev->data->port_id, rte_strerror(rte_errno)); + goto error; + } rte_wmb(); dev->tx_pkt_burst = mlx5_select_tx_function(dev); dev->rx_pkt_burst = mlx5_select_rx_function(dev); diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c index 26fa927039..149253d174 100644 --- a/drivers/net/mlx5/windows/mlx5_os.c +++ b/drivers/net/mlx5/windows/mlx5_os.c @@ -1116,6 +1116,7 @@ mlx5_os_net_probe(struct rte_device *dev) dev_config.txqs_inline = MLX5_ARG_UNSET; dev_config.vf_nl_en = 0; dev_config.mr_ext_memseg_en = 1; + dev_config.mr_mempool_reg_en = 1; dev_config.mprq.max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN; dev_config.mprq.min_rxqs_num = MLX5_MPRQ_MIN_RXQS; dev_config.dv_esw_en = 0;