From patchwork Fri Jan 14 10:52:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dmitry Kozlyuk X-Patchwork-Id: 105812 X-Patchwork-Delegate: rasland@nvidia.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 12AA4A0032; Fri, 14 Jan 2022 11:52:44 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 88A5042776; Fri, 14 Jan 2022 11:52:43 +0100 (CET) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2043.outbound.protection.outlook.com [40.107.93.43]) by mails.dpdk.org (Postfix) with ESMTP id 7253F40DDD; Fri, 14 Jan 2022 11:52:42 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NxMglCnSceHJkwJqDtBFJSOaVN55e/200di6tz61Qs2Gi1hxo5AjGfMnf/oy55jMqt7IG5ZXtGn2Ba1WWH+Pea+59oS3AopzwQIHHZ1+YTDVXwBt+yU3iBiwG58Yjw68Yj8gu4dJ4ZtHxPhaRt3Hy1f6RPnPvPpkzaLQGz4c1021Hl9lgfGBwbvRUPHPcKARo8EUaeNxucArnSJfRhy0t2X/RA+9eo4wzcFFv0sG645G4vWewWghpfBjprUVkNlveyBGtcTlK6PrBKLT775RFQXiKRWtuZ/LAf5qQS3CVAtiTcVgtqSsccVAKiwTBbcdJe3CFcEvUF5COGW/oYhzQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iqdnNcN5qWs/PphcQ2lSxVRAVoq8a5LStURZeTkpygg=; b=dyB+luHxhpYdUqwMuIZC1nqBwyvQ0CwiQ3hYd2CNdijt/0PSjjEXYMRXhYM+/xiiuWNMxUs/s8VlKmKQysvpMzVplWPjQggcrSEHd7CK8nRziSR26CguIb+FwR46X7OTBOLx1LRKw+wOtUmX/76pXQhYrgj9/aVEgznIFWivvHAVV8PO057dxxGOPmay1sqLtIq5AKaIJkCmp3EEcq+Ve3JVO4wbkWLG4TUA4+NqozLZ18emN1wW1400nllMTXcqXVCKuUZ3XULx0U1sBdrExhblcEVpkdbENhHUqoc794WCtCIekIdOTmcnMpd0AlPvAk8d4qlMQdpeeG+7p0sMlQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.234) smtp.rcpttodomain=huawei.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iqdnNcN5qWs/PphcQ2lSxVRAVoq8a5LStURZeTkpygg=; b=pCEMj1z+3vW1B1pZY54vlAQb7VKpsEiw2ni1w7U6XJQtFnsLfp7fyzjfRv00T/GgQJEgcGaHTSJpzkeylCMdr4fmwwp8EdI3e4QbB/urNjztMZtSlxnZf9R/56Byp8eQ0g+ommRsiLTFhA3X0l8lofH0PJZ6WFPg/mFfMWTfjEWde4nTImAY5DLz2qosMYrWeTrwta4964x/VLgNVjpmurCuSVCI0UJlVQbpGYmmW16uEN366d4EamjSNmcbIUdWILKjW8pzhHJwe9kJEwYCfkMmHl9o3Z21lYd5/sREawQ60cZ/ebCIJOq/RT3QjpXbf3HEGABF1Oe03G//mvuhZg== Received: from MW4PR04CA0204.namprd04.prod.outlook.com (2603:10b6:303:86::29) by DM6PR12MB4763.namprd12.prod.outlook.com (2603:10b6:5:36::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4888.11; Fri, 14 Jan 2022 10:52:40 +0000 Received: from CO1NAM11FT033.eop-nam11.prod.protection.outlook.com (2603:10b6:303:86:cafe::44) by MW4PR04CA0204.outlook.office365.com (2603:10b6:303:86::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4888.10 via Frontend Transport; Fri, 14 Jan 2022 10:52:39 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.234) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.234 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.234; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.234) by CO1NAM11FT033.mail.protection.outlook.com (10.13.174.247) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4888.9 via Frontend Transport; Fri, 14 Jan 2022 10:52:39 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL101.nvidia.com (10.27.9.10) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Fri, 14 Jan 2022 10:52:38 +0000 Received: from nvidia.com (10.127.8.11) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.2.986.9; Fri, 14 Jan 2022 02:52:36 -0800 From: Dmitry Kozlyuk To: CC: , Wang Yunjian , "Viacheslav Ovsiienko" , Matan Azrad Subject: [PATCH] common/mlx5: fix MR lookup for non-contiguous mempool Date: Fri, 14 Jan 2022 12:52:17 +0200 Message-ID: <20220114105217.343139-1-dkozlyuk@nvidia.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Originating-IP: [10.127.8.11] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: cec5c5dc-ec92-4cff-5813-08d9d74bfbbc X-MS-TrafficTypeDiagnostic: DM6PR12MB4763:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2958; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iP0QDSI2v6WIZL19hT/SHE7x2vMQT5eIEaZ5kb83WTVkxSRe57lp3CfYzKy4dc/cm76Kgf1Fvi5N/up0Ej4zjwxMTR1Ak75XP90iB+mEQ+Rx8+W8xzXgEE4U1IKJYV2Dp8TIcfdfzfTtAdJDbhxGxjVPMhKElndx1E77SRoUOSFBgQi7KgHRb90Y3//X2Jyoqaub1CthWrxKkb3SA/OQ/ay3psTp1Lq4+y/oVp/Y4UhPfB89BZ4EgWS3ovheKyKBgqUVh2YRr8UBBkkNz98FlR04Lto+nSIkUoFfx1u5L1n+Tcv3Mh9J8dC5iUpyBse0/yNkcuT0CIl8Oudj3uU0HGr/RPDdTTrXJ1+9esNhfydXW+u38ZO/7/uO+m4j8kBa8kYudeA+CJzbq/ZBKq+pmtrfZWEG0uWnaCQ3M4QtRQ/1eWMC0SFm1kSEDYValGjJ1Om7cJkQHfzGN0z4nRL5CWJfoUCckWYZs9ekbA9ih1oqqrEdj0rmoI2+Es/gExegyjondlB92j5yWjBwLcybXHXVuxYFbNH86aKBTe94I40BuR4rTvNpuTgNHKD2mc3uPFGP22O725Yhe7oxD35u06C0inlYdvVkHALtPcaujSMj0X7AspCPsOmdRzzv6VBSDy0ZOuAh1vcjdUcWw28RTwWOdLEUzDEXGXiy617CepHETP6lq2C5qrCDjOtdkhfRq94J5ZxIyihoBxs9NBxXLdcTFD+JmcIOGSGxDA/XUQt2aStHIBMF2t0WwRtp8OnrELwfACYVkn+9z4+AstSDkIT1Y5gpWyHuXkxaEGHIyx8= X-Forefront-Antispam-Report: CIP:12.22.5.234; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(40470700002)(54906003)(2616005)(70586007)(70206006)(336012)(107886003)(6666004)(426003)(5660300002)(316002)(356005)(4326008)(81166007)(36860700001)(83380400001)(6286002)(40460700001)(186003)(16526019)(26005)(47076005)(86362001)(2906002)(55016003)(508600001)(1076003)(36756003)(8936002)(82310400004)(7696005)(6916009)(8676002)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Jan 2022 10:52:39.4485 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: cec5c5dc-ec92-4cff-5813-08d9d74bfbbc X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.234]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT033.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4763 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Memory region (MR) lookup by address inside mempool MRs was not accounting for the upper bound of an MR. For mempools covered by multiple MRs this could return a wrong MR LKey, typically resulting in an unrecoverable TxQ failure: mlx5_net: Cannot change Tx QP state to INIT Invalid argument Corresponding message from /var/log/dpdk_mlx5_port_X_txq_Y_index_Z*: Unexpected CQE error syndrome 0x04 CQN = 128 SQN = 4848 wqe_counter = 0 wq_ci = 9 cq_ci = 122 This is likely to happen with --legacy-mem and IOVA-as-PA, because EAL intentionally maps pages at non-adjacent PA to non-adjacent VA in this mode, and MLX5 PMD works with VA. Fixes: 690b2a88c2f7 ("common/mlx5: add mempool registration facilities") Cc: stable@dpdk.org Reported-by: Wang Yunjian Signed-off-by: Dmitry Kozlyuk Reviewed-by: Viacheslav Ovsiienko --- drivers/common/mlx5/mlx5_common_mr.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/drivers/common/mlx5/mlx5_common_mr.c b/drivers/common/mlx5/mlx5_common_mr.c index 1537b5d428..5f7e4f6734 100644 --- a/drivers/common/mlx5/mlx5_common_mr.c +++ b/drivers/common/mlx5/mlx5_common_mr.c @@ -1834,12 +1834,13 @@ mlx5_mempool_reg_addr2mr(struct mlx5_mempool_reg *mpr, uintptr_t addr, for (i = 0; i < mpr->mrs_n; i++) { const struct mlx5_pmd_mr *mr = &mpr->mrs[i].pmd_mr; - uintptr_t mr_addr = (uintptr_t)mr->addr; + uintptr_t mr_start = (uintptr_t)mr->addr; + uintptr_t mr_end = mr_start + mr->len; - if (mr_addr <= addr) { + if (mr_start <= addr && addr < mr_end) { lkey = rte_cpu_to_be_32(mr->lkey); - entry->start = mr_addr; - entry->end = mr_addr + mr->len; + entry->start = mr_start; + entry->end = mr_end; entry->lkey = lkey; break; }