From patchwork Tue Aug 17 13:44:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michael Baum X-Patchwork-Id: 96991 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 24431A0548; Tue, 17 Aug 2021 15:45:19 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 11954407FF; Tue, 17 Aug 2021 15:45:19 +0200 (CEST) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2055.outbound.protection.outlook.com [40.107.237.55]) by mails.dpdk.org (Postfix) with ESMTP id B01F24014E for ; Tue, 17 Aug 2021 15:45:17 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WS19GMdsCG+3/sVuY4JF70GBWTUv5kG8J+GzRqI1YmMKXE7ejpGpJjyxAeiWd9uKqSQPNaiytgGesA/hSyZj3YAAHLVl6VW8JDBIcK552gZjF0Zk0tEkhMtoeF/m0gIy3zZxtzgL9x9PDWez99x+DNksV1HRIa6mYNSsMtOfrUTXGI7teljhYf5H08aPi1wd12M0uoh2VAG2IATkoh5LwJJH5UtlFAZv8Lpc2qcKlqoEWFAe3gsJEkm+JZQq+k8xayQI6+tDL6oZlxVNZ3/4eYHrp0NR80aP0fm9QNX79ez32u6/fqhvjlEsfx8fUEtNMoow7nMyHQ2Svsw2/QgDNQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zh2wMhEcw5bDgxtXMUYaSypauitP9Q9+NTdUJOVbLM8=; b=gf1pkJDl5/c5cueizLoLEXSVGNIFSj3aD9BaTi4+DM1wajTEJgT0i/9zww1QSzg9SmRNS6LlQQdfcjMW5N1L3Sufku8S3Jk555Hvu/J0eznf+PkLr/ydpbz5opI+5h/qpLbeXPJch9mi1eosdv8JNlVZ4fhwR1o3TZZbLuQ9D8YxgQsBNIE+FA/NiGeFXDW6PV3NzHDjbQrad7fRz+J3AkxHgce+0sYrbrXyfa/PPbYSbTfDAQt4MmmsAATNL/PSbPnx6RG/k78mqdpMG6YQurjehr+snsMFQryWI0kp6hnsxAxGDu4Cd3VSqtChz+xyUD+Qj7M9OGeef57xvMEkfQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zh2wMhEcw5bDgxtXMUYaSypauitP9Q9+NTdUJOVbLM8=; b=LorfKw6B89dH/gPmfoJwIOQiZXbF+OjrUe890FJ3PnBVvXdViEBN6rCAKoMlBoWnchAjSMpztoQiCjsNm/cdoGJ0NY0GEaTtnrD5bdAnQDCkTqi8l8/nsitUKb1V4HHP3hkbxFZelySYY65uwl2Kry/lv25sKscePiHbsDwD3owbPjy8+Yec8r4NiQWnzUX93W9rzq3OcoWo2UGlMFStmKAeIkge8TaXce2mscxD0vhEBuvEheVBsM8c0USeVvahN7+eTGJZFZsG2dCsfbPr89OBU9DNV6jcl390+behIMNmkJdDTyBHYJXSUhAZXSI0oq9LnqJf9EJzkkmFkZrRAA== Received: from MWHPR03CA0010.namprd03.prod.outlook.com (2603:10b6:300:117::20) by CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.17; Tue, 17 Aug 2021 13:45:16 +0000 Received: from CO1NAM11FT012.eop-nam11.prod.protection.outlook.com (2603:10b6:300:117:cafe::8a) by MWHPR03CA0010.outlook.office365.com (2603:10b6:300:117::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Tue, 17 Aug 2021 13:45:16 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by CO1NAM11FT012.mail.protection.outlook.com (10.13.175.192) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4415.16 via Frontend Transport; Tue, 17 Aug 2021 13:45:15 +0000 Received: from DRHQMAIL107.nvidia.com (10.27.9.16) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 17 Aug 2021 06:45:15 -0700 Received: from nvidia.com (172.20.187.6) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 17 Aug 2021 13:45:13 +0000 From: Michael Baum To: CC: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Tue, 17 Aug 2021 16:44:20 +0300 Message-ID: <20210817134441.1966618-1-michaelba@nvidia.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To DRHQMAIL107.nvidia.com (10.27.9.16) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 71942ae2-bd14-4088-235d-08d961853eb2 X-MS-TrafficTypeDiagnostic: CH2PR12MB4294: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4714; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1u3ln6k12swTeLyQpGDxlvDf2H/rkH7WaebagqdOTVyFX9mg0F73ZaRAlFonZSGeey8wDVQ18zwFizU2B7lRAjJiiciqIKVNVnAWBF94f/6jHF+oczEQWManYupCSH3TRg7j0g6drUTo6K07PRwTQ03FxyKY1K0xyZ07hExZeG4TE7j4BYWkNU1WrrO7ZE+ze12+C1UEj2kzGc6nVHdf1v3OUWryvTLqVIUUrXhyf6bq394TBD6eqAySezD4ohOSX48M/SKgeuePt7rv3Kzki568U+5tOpuxC3nRCaIMRjM+1wtaoIGdJuhiXChTjBqN3PYxja0376jvlS8yznes2aKG25HgOR8TGd0wB9Xq1q0dAUnfG/m06ocfSrfab1gxl3ZPMPgH+KGE4xKEnNcKhf5zJl0n68/QJIgSYaeFKxloed9TgoZE5yIsrxALdGRVEB+KPNs4Y97FX4w/bJ6fhNew6BvmmtT5GDgeliLdTa1BgMgBt1OTem1rlkMMJkNNoLSIqaiTmZVryJWfPBtZ/do3J/aIeUbbLuQL+fv/Sor4xcncBlgaGXVPBH/nO/uiutYBK2mHWgEJoLTEk3Ru/tuTwELTarg455JcUUo9tDTyNHNZwgid3O4PLWh4cWt4ClkOMszAmZK2/SPD2sxwx/tSi7jfi/0RkR1+Exa+2wMP4MtMizD9AZmN7u8xD1Y83Fzr2nvRyBjTwpZ6MW5WPg== X-Forefront-Antispam-Report: CIP:216.228.112.32; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid01.nvidia.com; CAT:NONE; SFS:(4636009)(136003)(396003)(346002)(39860400002)(376002)(36840700001)(46966006)(1076003)(47076005)(36756003)(36860700001)(54906003)(8676002)(316002)(26005)(107886003)(16526019)(186003)(86362001)(6286002)(7636003)(4326008)(5660300002)(83380400001)(82310400003)(426003)(2906002)(6916009)(478600001)(82740400003)(356005)(70206006)(2616005)(70586007)(7696005)(8936002)(55016002)(6666004)(336012); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Aug 2021 13:45:15.8623 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 71942ae2-bd14-4088-235d-08d961853eb2 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.32]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT012.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4294 Subject: [dpdk-dev] [RFC 00/21] mlx5: sharing global MR cache between drivers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" There are 5 classes of mlx5 drivers (net\eth, RegEx, vDPA, compress and crypto). The various drivers are registered under the common mlx5 driver and are managed by it. The common driver probing calls in a loop to the probe function of each driver registered to it. Each driver creates for itself all the objects required for communication with the hardware and a global MR cache that manages memory mappings. The management of the caches separately by the different drivers is not very efficient. In fact, the same memory is mapped multiple times to the HW when more than 1 class use the device. This feature will move management to the common driver in two phases. Phase 1: sharing HW objects between drivers on the same device The communication with the hardware - for any MR handle - is conducted by the Protection Domain, so we are motivated to share it between the drivers. However, to create it, we need to give the context of the device, so the context must also be shared between the drivers. At this point, we will share the next trio between the drivers (CTX, PD, pdn) to create an infrastructure that will allow sharing of dependent objects, particularly the global MR cache. The common driver itself will create this trio individually for all drivers before calling their probe function. Then, as a parameter to the probe function, it will give them a pointer to the structure containing the trio. Phase 2: sharing global MR cache between drivers on the same device The common driver will add to the structure containing the trio and the structure that manages the global MR cache and keep a list of such structures for memory management. In each driver, each queue will manage its own local MR cache. If the queue does not find its cache, it will search the global MR cache shared by all. Caching access will be through the pointer that the driver received as a parameter in probing. Michael Baum (21): net/mlx5: fix shared device context creation error flow net/mlx5: fix PCI probing error flow common/mlx5: add context device structure compress/mlx5: use context device structure crypto/mlx5: use context device structure regex/mlx5: use context device structure net/mlx5: improve probe function on Windows net/mlx5: improve probe function on Linux net/mlx5: improve spawn function net/mlx5: use context device structure net/mlx5: move NUMA node field to context device common/mlx5: add ROCE disable in context device creation vdpa/mlx5: use context device structure mlx5: update device sent to probing mlx5: share context device structure between drivers common/mlx5: add HCA attributes to context device structure regex/mlx5: use HCA attributes from context device vdpa/mlx5: use HCA attributes from context device compress/mlx5: use HCA attributes from context device crypto/mlx5: use HCA attributes from context device net/mlx5: use HCA attributes from context device drivers/common/mlx5/linux/mlx5_common_os.c | 268 ++++++++- drivers/common/mlx5/mlx5_common.c | 273 +++++++++- drivers/common/mlx5/mlx5_common.h | 35 +- drivers/common/mlx5/mlx5_common_private.h | 6 - drivers/common/mlx5/version.map | 2 + drivers/common/mlx5/windows/mlx5_common_os.c | 207 ++++++- drivers/compress/mlx5/mlx5_compress.c | 112 +--- drivers/crypto/mlx5/mlx5_crypto.c | 111 +--- drivers/crypto/mlx5/mlx5_crypto.h | 4 +- drivers/crypto/mlx5/mlx5_crypto_dek.c | 5 +- drivers/net/mlx5/linux/mlx5_ethdev_os.c | 8 +- drivers/net/mlx5/linux/mlx5_mp_os.c | 9 +- drivers/net/mlx5/linux/mlx5_os.c | 543 +++++++++---------- drivers/net/mlx5/linux/mlx5_verbs.c | 55 +- drivers/net/mlx5/mlx5.c | 85 ++- drivers/net/mlx5/mlx5.h | 17 +- drivers/net/mlx5/mlx5_devx.c | 35 +- drivers/net/mlx5/mlx5_flow.c | 6 +- drivers/net/mlx5/mlx5_flow_aso.c | 24 +- drivers/net/mlx5/mlx5_flow_dv.c | 51 +- drivers/net/mlx5/mlx5_flow_verbs.c | 4 +- drivers/net/mlx5/mlx5_mr.c | 14 +- drivers/net/mlx5/mlx5_txpp.c | 27 +- drivers/net/mlx5/windows/mlx5_ethdev_os.c | 14 +- drivers/net/mlx5/windows/mlx5_os.c | 285 ++-------- drivers/regex/mlx5/mlx5_regex.c | 74 +-- drivers/regex/mlx5/mlx5_regex.h | 23 +- drivers/regex/mlx5/mlx5_regex_control.c | 12 +- drivers/regex/mlx5/mlx5_regex_fastpath.c | 18 +- drivers/regex/mlx5/mlx5_rxp.c | 64 ++- drivers/vdpa/mlx5/mlx5_vdpa.c | 210 +------ drivers/vdpa/mlx5/mlx5_vdpa.h | 4 +- drivers/vdpa/mlx5/mlx5_vdpa_event.c | 19 +- drivers/vdpa/mlx5/mlx5_vdpa_lm.c | 6 +- drivers/vdpa/mlx5/mlx5_vdpa_mem.c | 13 +- drivers/vdpa/mlx5/mlx5_vdpa_steer.c | 10 +- drivers/vdpa/mlx5/mlx5_vdpa_virtq.c | 16 +- 37 files changed, 1414 insertions(+), 1255 deletions(-)