From patchwork Sat Jan 8 00:20:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Agostini X-Patchwork-Id: 105690 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 08A0AA04A6; Fri, 7 Jan 2022 17:10:20 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8B59C40143; Fri, 7 Jan 2022 17:10:19 +0100 (CET) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2078.outbound.protection.outlook.com [40.107.244.78]) by mails.dpdk.org (Postfix) with ESMTP id 9EF6E40042 for ; Fri, 7 Jan 2022 17:10:17 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WHwApGozWsT5C5DsArlmVY9HboXLTV8fxRyaHSEailfWTmm8ipR0LPSro2R3+fCre1wbAwNWRhK9+lq27+lAE2rErF5D7XJ/J3uJg/O6cKTROxYOKhvlLc3L40QSr+/sAwIy9juFmdff2JXT3CHmmoTCb97f6FpmU3pQO5sfq4HsrbomR04dABV5840hclzZcbewV1Fj+oR4da2S8b+CvqcRm4b5fSKYcdyeOftkZBaQWaJSDb0rbP0zY/GOyvA+uyFty13Jz2Vyk/Ce5rW0kZ49hBt01m2gjxA9mon5/VCmGxOGf3FAMkPaYA5gu25F5bTD3oizFS05EsbPpadWKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=7B906MrpXTxj/G3PqMvKWJ7BiBXknFULgIYvhvITmx4=; b=YJ3MULmR3VD9X3XQexfVUFAnMxh2OidMESapeU9oYDuFlCIxvVIN3tBJXayEv3Lb19e2y7mHVRPwOf/hRHJ+Qmqx4eVGjYMOEIVNnbMVOAq3SXY/G1xh0rP28tVyHqSU9/XW1s452f/shQxWU9/ph1LGcWmW5OsBueyLRVdF+l/XDlR9WGGKDEVl9wct/phSBBMaZZiFoXfIt2/t63+U4HKx69lR6p3HZ84NWTTGiUvUUbQVTxK+cpGxugq5mZmbRvA5mU14FEE6Thj3o9g9zZKoZ+9POYBNCHeFlGQitwMCp5/a0Ial0XKtfLC46sajt7Q5IV6CnfLKtpVBYLHwEw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.236) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7B906MrpXTxj/G3PqMvKWJ7BiBXknFULgIYvhvITmx4=; b=NaHAH8EwCVTI0A4bL7cVIp/5zYNEuvi5gVxJ9VEz9+KvjS+kyH5R4d/KSDRGORLWpMnKN7VfFlSsY5bjJuZYXnIU6nIhE05QunakSHW6MPKflJYtin/XMSooVrnhwRvZRGWoQwn4JVSPj+iXRuR641hWRJQSVrjhMxUdsQ/HiGjzTZEY2Wl32hECWny+mZ5XuPajKAmybnbPe1Fj8xAuSrsUu8Tv6GuugVBv4U+h88FPlCWb2uvlTr3MgN/CHEfi4WzoXAf7AW4jBMY/A5tg/s3Q/80cofRxkHAy98q7cO9FC5XQ5AZht/5vFamL8lucBBSlba9ikwhla+VPVkYsxQ== Received: from DM5PR20CA0027.namprd20.prod.outlook.com (2603:10b6:3:13d::13) by BY5PR12MB4258.namprd12.prod.outlook.com (2603:10b6:a03:20d::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4867.9; Fri, 7 Jan 2022 16:10:16 +0000 Received: from DM6NAM11FT005.eop-nam11.prod.protection.outlook.com (2603:10b6:3:13d:cafe::cc) by DM5PR20CA0027.outlook.office365.com (2603:10b6:3:13d::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4867.9 via Frontend Transport; Fri, 7 Jan 2022 16:10:15 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.236) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.236 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.236; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.236) by DM6NAM11FT005.mail.protection.outlook.com (10.13.172.238) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4867.7 via Frontend Transport; Fri, 7 Jan 2022 16:10:15 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL109.nvidia.com (10.27.9.19) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Fri, 7 Jan 2022 16:10:14 +0000 Received: from nvidia.com (172.20.187.5) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.2.986.9; Fri, 7 Jan 2022 08:10:14 -0800 From: To: CC: Elena Agostini Subject: [PATCH v2 1/3] gpudev: mem alloc aligned memory Date: Sat, 8 Jan 2022 00:20:01 +0000 Message-ID: <20220108002003.21153-1-eagostini@nvidia.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220104014721.1799-1-eagostini@nvidia.com> References: <20220104014721.1799-1-eagostini@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0516ab2b-ec21-42be-027e-08d9d1f83144 X-MS-TrafficTypeDiagnostic: BY5PR12MB4258:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 17hAqBAXWh2Tr4FyGwq62JNA//OdAmrVKl6GdUPGzS1Cjs9ZtJoRYD2RjYABoLTPr24gSoTVKwbj9QOaTj0gb/1WpIlA2OX4jnk9ETYhhhVvPUr8krcd2IPBulJ2E95UvntJkBOVnTZ2HOh5zLmX7znNwi77HC1vE7f7SErpSk7b6ddl7GlVrOvCm6jp9hONsaPcFNG1rAJQt60zisa9UxKXgryHUKxFUjwDP4L54cQVE979xiJPB5tKHF2ql3HGXF+4T/1jZ1yhTZzJhnv56ehZu9ZeB6lQc4ChXCpl7XL6ytEFJNuOrJ9bdRUYvp3AAnZS/e86OAx3ZYCwArxj7HOtoUuCGHCh3KvEBr1IBqVzhaUR37c1OoYe4TWs4fP9By1nkibundAO8SraAyhlQvzuooZHtsQp460U+xEc1mqJi+wCnG38rO8zpSu3VgZVYXrKWKSR0dcbjOhiHHNmbzuwmrAf+eEcVOiz4YfQURp4Hyog7tsoK8wPZSoPMzFWqIP6YSlFQ6PtGSMQW4ON/KJzaHsTNiLB1jJRBsiiWCkYUZoOAszE9V3KGPZ8mwOWLJWlgN6I4mIWZG1t/YUUJCiV40AafRn3mXFdTC/QviFCNVprpxGPE8eJ1Zn7uVMGBAC6evq7NIIcOQjIS4q4tu7dHZHdX7nh5/zp1zODelpqf5iYRM7NA6LGBJg2LGV12S9aN02ubfifoTmtpVuMzI7HOimNKBkw04rr1otYU27NAPybmwgmFPD3POmREWqivot4iONL83ihaYlKRxNXxJxUvJj36T7s3cPEM30vV2U= X-Forefront-Antispam-Report: CIP:12.22.5.236; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(4636009)(36840700001)(40470700002)(46966006)(5660300002)(8936002)(81166007)(26005)(2876002)(6916009)(82310400004)(1076003)(55016003)(47076005)(356005)(8676002)(40460700001)(86362001)(4326008)(107886003)(6286002)(36756003)(70586007)(70206006)(426003)(336012)(7696005)(2616005)(316002)(36860700001)(6666004)(2906002)(16526019)(186003)(508600001)(83380400001)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jan 2022 16:10:15.6552 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0516ab2b-ec21-42be-027e-08d9d1f83144 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.236]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT005.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4258 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Elena Agostini Similarly to rte_malloc, rte_gpu_mem_alloc accept as input the memory alignment size. GPU driver should return GPU memory address aligned with the input value. Changelog: - rte_gpu_mem_alloc parameters order Signed-off-by: Elena Agostini --- lib/gpudev/gpudev.c | 10 ++++++++-- lib/gpudev/gpudev_driver.h | 2 +- lib/gpudev/rte_gpudev.h | 10 +++++++--- 3 files changed, 16 insertions(+), 6 deletions(-) diff --git a/lib/gpudev/gpudev.c b/lib/gpudev/gpudev.c index 9ae36dbae9..59e2169292 100644 --- a/lib/gpudev/gpudev.c +++ b/lib/gpudev/gpudev.c @@ -527,7 +527,7 @@ rte_gpu_info_get(int16_t dev_id, struct rte_gpu_info *info) } void * -rte_gpu_mem_alloc(int16_t dev_id, size_t size) +rte_gpu_mem_alloc(int16_t dev_id, size_t size, unsigned int align) { struct rte_gpu *dev; void *ptr; @@ -549,7 +549,13 @@ rte_gpu_mem_alloc(int16_t dev_id, size_t size) if (size == 0) /* dry-run */ return NULL; - ret = dev->ops.mem_alloc(dev, size, &ptr); + if (align && !rte_is_power_of_2(align)) { + GPU_LOG(ERR, "requested alignment is not a power of two %u", align); + rte_errno = EINVAL; + return NULL; + } + + ret = dev->ops.mem_alloc(dev, size, align, &ptr); switch (ret) { case 0: diff --git a/lib/gpudev/gpudev_driver.h b/lib/gpudev/gpudev_driver.h index cb7b101f2f..0ed7478e9b 100644 --- a/lib/gpudev/gpudev_driver.h +++ b/lib/gpudev/gpudev_driver.h @@ -27,7 +27,7 @@ enum rte_gpu_state { struct rte_gpu; typedef int (rte_gpu_close_t)(struct rte_gpu *dev); typedef int (rte_gpu_info_get_t)(struct rte_gpu *dev, struct rte_gpu_info *info); -typedef int (rte_gpu_mem_alloc_t)(struct rte_gpu *dev, size_t size, void **ptr); +typedef int (rte_gpu_mem_alloc_t)(struct rte_gpu *dev, size_t size, unsigned int align, void **ptr); typedef int (rte_gpu_mem_free_t)(struct rte_gpu *dev, void *ptr); typedef int (rte_gpu_mem_register_t)(struct rte_gpu *dev, size_t size, void *ptr); typedef int (rte_gpu_mem_unregister_t)(struct rte_gpu *dev, void *ptr); diff --git a/lib/gpudev/rte_gpudev.h b/lib/gpudev/rte_gpudev.h index fa3f3aad4f..9e2e2c5dce 100644 --- a/lib/gpudev/rte_gpudev.h +++ b/lib/gpudev/rte_gpudev.h @@ -364,18 +364,22 @@ int rte_gpu_info_get(int16_t dev_id, struct rte_gpu_info *info); * @param size * Number of bytes to allocate. * Requesting 0 will do nothing. - * + * @param align + * If 0, the return is a pointer that is suitably aligned for any kind of + * variable (in the same manner as malloc()). + * Otherwise, the return is a pointer that is a multiple of *align*. In + * this case, it must obviously be a power of two. * @return * A pointer to the allocated memory, otherwise NULL and rte_errno is set: * - ENODEV if invalid dev_id - * - EINVAL if reserved flags + * - EINVAL if align is not a power of two * - ENOTSUP if operation not supported by the driver * - E2BIG if size is higher than limit * - ENOMEM if out of space * - EPERM if driver error */ __rte_experimental -void *rte_gpu_mem_alloc(int16_t dev_id, size_t size) +void *rte_gpu_mem_alloc(int16_t dev_id, size_t size, unsigned int align) __rte_alloc_size(2); /** From patchwork Sat Jan 8 00:20:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Agostini X-Patchwork-Id: 105691 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BBC91A04A6; Fri, 7 Jan 2022 17:10:25 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 610E04068E; Fri, 7 Jan 2022 17:10:20 +0100 (CET) Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2066.outbound.protection.outlook.com [40.107.223.66]) by mails.dpdk.org (Postfix) with ESMTP id 3643440042 for ; Fri, 7 Jan 2022 17:10:18 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=V1/mv455Q2cqPuB2ENlPXccVAZiru+1/jovMKPxghmHrQfFcYmd8DNph4qAUYuPJzMBEa8T4K7m3qGwat1RymvSQBc/g3fF2w8zagcFLA0wvD/MoElLn3D/eLsjB9z1srLdkdzBV7OztYrwBw596FtU3iigxybY0Mdu0WuZdYAZFuwREWF6GOqX8HFxlru3DaOA0JvXOsmtE/m3j2ixSp04pRf/ubkDVnjik21dSHAki7qhoBCNRzH/FJPBDoKKR8R2rnnhA7IQ1E5HJprZWHprVRAcpwjVbv+6QRXb0bucEAy560Fho9cag9h7PDTRE9cRx5ejgEPChO9lxtgN9hw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=OkdTox4plqxUBYxrxXqv2iVwrfQvqv/v85nXlPQy2mQ=; b=CPnP5qFlKKipVdoo3Dei8x4O+w5b4cFBKlZ77NCdaHSkH5PuiUN4macB1zox6KpywS/7Qd8b7MvUBpOVUMbtDfLkoMh14pn1BsT9IOokuG9G3PHAYZadsdycCCJ+LuVRFtAs0mrVB/m/QVVLrML7lljQUegHSKKTVj9UbBJX1f6UHOZZA1zee70g2RY9CgxYqtM4ip9dhxGJMLdDvlr2H+s//YV5I9MgVlVg3IcEXaciOzwdVgKpXS8dWsIrar/A6gIxNjB0+C9yWIWf7AnUmV/gCqtOVSJAJ2Huk4xC5gVAT17I3pk79DQmzPHREVB/5IlCAZvWCS9YdFm76BaatA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.236) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OkdTox4plqxUBYxrxXqv2iVwrfQvqv/v85nXlPQy2mQ=; b=EJXwOK3EsF4X6QCw0OJdRmvmxr1SMrnDMHGxGgSJ2JHTpF9ej2urCyrmckJVBg/UMssr7U+ViPo9YI8E5obxg74OfXJFq0zAWawg4m7rj9VBFldnM4I3tsiQ1NDdKPOaXfp72cbFS529/ip2M6g3Zzq/42Q7DEP9B1vZEEqFa/AkXY6720kOTRBxI/dDv+uThdRVKGX6ZHqlpxvmJNwJFpv3CAhx4viTb73IMpFTJbood4uId3BRHKVVxsO41E3aoCwo8TjwD3imr6AOOR2b/t9sLEWETyCiL+FNnDM+86doUSzyFGVbqeE6AIP484ODIIHmD1/0yywbz5kH4Cjxsw== Received: from DM5PR20CA0042.namprd20.prod.outlook.com (2603:10b6:3:13d::28) by MWHPR12MB1471.namprd12.prod.outlook.com (2603:10b6:301:e::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4867.9; Fri, 7 Jan 2022 16:10:16 +0000 Received: from DM6NAM11FT005.eop-nam11.prod.protection.outlook.com (2603:10b6:3:13d:cafe::ff) by DM5PR20CA0042.outlook.office365.com (2603:10b6:3:13d::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4867.7 via Frontend Transport; Fri, 7 Jan 2022 16:10:16 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.236) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.236 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.236; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.236) by DM6NAM11FT005.mail.protection.outlook.com (10.13.172.238) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4867.7 via Frontend Transport; Fri, 7 Jan 2022 16:10:16 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL109.nvidia.com (10.27.9.19) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Fri, 7 Jan 2022 16:10:15 +0000 Received: from nvidia.com (172.20.187.5) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.2.986.9; Fri, 7 Jan 2022 08:10:14 -0800 From: To: CC: Elena Agostini Subject: [PATCH v2 2/3] app/test-gpudev: test aligned memory allocation Date: Sat, 8 Jan 2022 00:20:02 +0000 Message-ID: <20220108002003.21153-2-eagostini@nvidia.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220108002003.21153-1-eagostini@nvidia.com> References: <20220104014721.1799-1-eagostini@nvidia.com> <20220108002003.21153-1-eagostini@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 76a905b2-69de-4f15-c1a0-08d9d1f831a5 X-MS-TrafficTypeDiagnostic: MWHPR12MB1471:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:510; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: HdNuUU0sdBIwlvfJHTMfjw8cgb4c2xvI8EtzBLiOAYx5vJxSb9xbnP7FHPGGYMAuLEFPNbHGnKoTzTNf61UnZ5QkmLiey2EJRaIyGDN41Oam1zAQZXMoBINulGZGTj8Q3QbrpWqFSNw1LTCrQzLcJjHkE74hgLRwWpfOMgC5GCOx5gxd8qquAekaxtK0jZGifquarjJwL4SuTI+baU6dXh++e6An/oDaIKhZvv5lFIE7Gj++fdLqygmOEtjv6A8ciDVc3icCGpwG7frumYuJmIXV+6XIknrkfSrCj7K0ImQ5IoqF3FEItSR+9i3kBTMryLmFAbonGan9bOo4QVHl3lNC9aJvEg4oYJuEKoMqjBlptAkmcktfw7quc550BETpzL7BkrJ9MQM9u8u2iQN7gLuYc2zQJdeqqx0+4kYQHESi84OFcCWtds0fr/tADPv3spGQ230t5ehsdZ3AR32DAzvDlVu97UEaCGXCuqM/hxZksM+Af3jgn3eD9p3aTI4Mno3LRmHTix/N4VGu+bagEVAxzrZr+kOR/NSphp9vgE9rgFnXeQ0Rkp/11CYEhs85VBSB5m0lzhJ2hgrFz9J28Qf6b9tTMYOS1U9D3nhfRmwza9Zl8zmkGMfIKrSSRXlFucH8C4F/vDdmtnjr5Rj+cfDoERU1jL7GctXI5ZEI787ZQ+zIPKpj5AljyTnNed4xEK8wi21qJc9Q8SVNsqJYO/kqZXOTeX4P2NAFUatv6Crkk0JAgAyP31LyWm0oMxhm1I7iRPRQEMyuAl3lB6R3kJnwvc98yTPdYYwhAMGcETo= X-Forefront-Antispam-Report: CIP:12.22.5.236; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(4636009)(40470700002)(46966006)(36840700001)(2876002)(81166007)(2906002)(36756003)(7696005)(6286002)(508600001)(55016003)(70206006)(47076005)(4326008)(70586007)(83380400001)(6916009)(5660300002)(356005)(336012)(426003)(2616005)(6666004)(16526019)(82310400004)(186003)(86362001)(8936002)(8676002)(36860700001)(316002)(40460700001)(1076003)(107886003)(26005)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jan 2022 16:10:16.3270 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 76a905b2-69de-4f15-c1a0-08d9d1f831a5 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.236]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT005.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1471 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Elena Agostini Update gpudev app to test GPU memory aligned allocation. Signed-off-by: Elena Agostini --- app/test-gpudev/main.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/app/test-gpudev/main.c b/app/test-gpudev/main.c index 5c1aa3d52f..f36f46cbca 100644 --- a/app/test-gpudev/main.c +++ b/app/test-gpudev/main.c @@ -69,11 +69,12 @@ alloc_gpu_memory(uint16_t gpu_id) void *ptr_2 = NULL; size_t buf_bytes = 1024; int ret; + unsigned align = 4096; printf("\n=======> TEST: Allocate GPU memory\n\n"); - /* Alloc memory on GPU 0 */ - ptr_1 = rte_gpu_mem_alloc(gpu_id, buf_bytes); + /* Alloc memory on GPU 0 without any specific alignment */ + ptr_1 = rte_gpu_mem_alloc(gpu_id, buf_bytes, 0); if (ptr_1 == NULL) { fprintf(stderr, "rte_gpu_mem_alloc GPU memory returned error\n"); goto error; @@ -81,7 +82,8 @@ alloc_gpu_memory(uint16_t gpu_id) printf("GPU memory allocated at 0x%p size is %zd bytes\n", ptr_1, buf_bytes); - ptr_2 = rte_gpu_mem_alloc(gpu_id, buf_bytes); + /* Alloc memory on GPU 0 with 4kB alignment */ + ptr_2 = rte_gpu_mem_alloc(gpu_id, buf_bytes, align); if (ptr_2 == NULL) { fprintf(stderr, "rte_gpu_mem_alloc GPU memory returned error\n"); goto error; @@ -89,6 +91,11 @@ alloc_gpu_memory(uint16_t gpu_id) printf("GPU memory allocated at 0x%p size is %zd bytes\n", ptr_2, buf_bytes); + if (((uintptr_t)ptr_2) % align) { + fprintf(stderr, "Memory address 0x%p is not aligned to %u\n", ptr_2, align); + goto error; + } + ret = rte_gpu_mem_free(gpu_id, (uint8_t *)(ptr_1)+0x700); if (ret < 0) { printf("GPU memory 0x%p NOT freed: GPU driver didn't find this memory address internally.\n", From patchwork Sat Jan 8 00:20:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Agostini X-Patchwork-Id: 105692 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C0B52A04A6; Fri, 7 Jan 2022 17:10:32 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9FE3641141; Fri, 7 Jan 2022 17:10:21 +0100 (CET) Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2052.outbound.protection.outlook.com [40.107.220.52]) by mails.dpdk.org (Postfix) with ESMTP id 5196F40042 for ; Fri, 7 Jan 2022 17:10:19 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Zx+prNlZicwU9LUPw8D3MyOhbgljYD59F0sCEmMtoMvMZ4xNZh3+fHFKWA3mnDgTJ9WcizQrCPFd9E3bunVXzy3upqXUts4ogTM/D0ettIZ12Tprkvz71J22JBeZBNwJhcLMO52NdtBchDcbBilY9ZYTBf0skN28oT0Z0qOJGHYo+GUqEWQFThwIoX9uVAAGWD6HdgSzGQvcV5gO6/zNywoFaLBdPons8t8NqKCETGBKrJwLdcreyYtmhlr4D9BzxoipVKPKVDLr4lYeULURj8Sryfqq0G6FOiKgyzC29l1x+9hDkwDoKnd/yhd9T04lpqgMNiS/TbDQpexmItx+pw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=a03DfUynRhyTyJpArg5xa6hs1G1OEEhemAVjGSwUFxI=; b=oEHw6V356nP6hItql7u0oiB6NfnDiS8+EtpjIt4G2UyHCYJ7n2X10e7MLC9v2AcNAw7pSId0d1cMrAz+mJUhQFlZxwJRtD1GjwSurOd/K/TDwSAdT8WxR87QHDm9el3pDeJlFSnGxYrt8yktqNQ1AiIQOnkqi9QGKV3bw9GJa7kOIlMr7zR6n/X3RzEEWQD0glRGh9uiJyRYy+YbGzfpJPL+0w1RhM1MgVZQTvfLbMyKZIW6OWX88FAeSKSlOL7Sjuhj1UOniF3aGBD/WwSK1FVCatzwNxvpKIs+BK9kyNCaiAfgZ9hp7wEE3mxohc1v/mISWN6mdOkgwYAsJuuadQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=a03DfUynRhyTyJpArg5xa6hs1G1OEEhemAVjGSwUFxI=; b=F1sTRMhYIuCvilqKyrqxPVBK10mSg+09cnsgaiBbAAJnvnWQSrRG+TdZAlAy8gHaqnpivPSluxcad7J2bir1zzqNxC8UgOVRG2Ho8haAIQ9bizDs0aAzUI5U6FnuK2bCR6hd1ThgEJ5HwmZA3e3sPhce21eqobRhk7LecqzKQr5e3SPGyQJ5cQ/oB/hDpj4FSm7Jmk4BXxHcMkAnQrKAk1FRn2jBcN0kbnjDUPP3kl8VnZtm+wXrUg6OUBScZ7bgMucx4/+NNS0dLybWDYQlJSrEkm0a93YeotQ1Jq5eNtegCcUqIq0rWPqyMWvJfKcTbh1NZB3kavxAehaI5Zsdhw== Received: from CO2PR04CA0153.namprd04.prod.outlook.com (2603:10b6:104::31) by CH2PR12MB3894.namprd12.prod.outlook.com (2603:10b6:610:2b::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4867.7; Fri, 7 Jan 2022 16:10:16 +0000 Received: from CO1NAM11FT042.eop-nam11.prod.protection.outlook.com (2603:10b6:104:0:cafe::7d) by CO2PR04CA0153.outlook.office365.com (2603:10b6:104::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4867.9 via Frontend Transport; Fri, 7 Jan 2022 16:10:16 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.235) by CO1NAM11FT042.mail.protection.outlook.com (10.13.174.250) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4867.7 via Frontend Transport; Fri, 7 Jan 2022 16:10:16 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Fri, 7 Jan 2022 16:10:15 +0000 Received: from nvidia.com (172.20.187.5) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.2.986.9; Fri, 7 Jan 2022 08:10:15 -0800 From: To: CC: Elena Agostini Subject: [PATCH v2 3/3] gpu/cuda: mem alloc aligned memory Date: Sat, 8 Jan 2022 00:20:03 +0000 Message-ID: <20220108002003.21153-3-eagostini@nvidia.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220108002003.21153-1-eagostini@nvidia.com> References: <20220104014721.1799-1-eagostini@nvidia.com> <20220108002003.21153-1-eagostini@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0d1e7347-2582-494d-2716-08d9d1f831a4 X-MS-TrafficTypeDiagnostic: CH2PR12MB3894:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:428; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: jvsbYtzHHnwtcdRQP4Pb5uSnZ63OZ0uyo8gx6zfAaB2VbaUdPGR7zL4HsebVaHw8s+ic6dSiVvbbty/QN/s9JKLduRw1mbXH9FyrMRXpZNFVyHOl8glla6RxyZhF5agvCA8x5FK39MInK3haAIO1agokWn2y0ddY02Jvi17tZJWriOjBnW6RBWnEsnq71NObj2+zz9aLDfK1SETC5QqYWhmu33IsVKlg/v7xyKqYhKYujlfJULkd0FoKsTui7uToMh6T75LuJ6t5JnNQV0PdgRpnXSYtbQRdh1aZBA43PVZQ/JkCGP7b8I9cTdkkB3Jb7X3prf7Wi6UgQJF7I8ohkGtN96Zkh0EK0pKZeQOjgcf1WUL88pM9Cq05bvmI37Gvz9of7yUsQQfxxR1yEy+oLMcmgjAW08WSN5f4VO6jZIVqOAofiBezA0k9jl1YSg+D2AT56J2mI4TTPR9OgRumnzCPsB1PGh3jWp5r/EQQII+ubtrz2nJP1NgHrBXyXAAgPqwweqJSj6cXBao4GXe/u1bnWocFN/PUJWLLAEkQoq5WnUQXI0af7AOh7tDWKSaeKCUUEyWIOWXzc7i98Y4qUvk9gHDYkzmYWkzPpjna7XmEu5eIj94GvY96oYgTaCRSqSqiWVbL4ATJy8TW1sYfseDUQ22XPCCOrdNz83zo7N1XbkByQdiX6xL7zN4iDx4RQSf1xd2Dla2oMAihDShhFhAnWk+7UZw+un41GG/82MkR3Z3KlJyW21BoaBh/1PgR8zQIu6IvhOKPXDqzwLAo5uauoAzOWieOLVS0zQkLuts= X-Forefront-Antispam-Report: CIP:12.22.5.235; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(40470700002)(36756003)(2906002)(1076003)(7696005)(86362001)(508600001)(36860700001)(186003)(26005)(336012)(47076005)(2616005)(426003)(8676002)(8936002)(5660300002)(16526019)(83380400001)(356005)(6916009)(81166007)(55016003)(82310400004)(6666004)(6286002)(40460700001)(2876002)(4326008)(70586007)(70206006)(107886003)(316002)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Jan 2022 16:10:16.3322 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0d1e7347-2582-494d-2716-08d9d1f831a4 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.235]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT042.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB3894 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org From: Elena Agostini Implement aligned GPU memory allocation in GPU CUDA driver. Changelog: - cuda_mem_alloc parameters order Signed-off-by: Elena Agostini --- drivers/gpu/cuda/cuda.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/drivers/gpu/cuda/cuda.c b/drivers/gpu/cuda/cuda.c index 882df08e56..dc8d3d3b5a 100644 --- a/drivers/gpu/cuda/cuda.c +++ b/drivers/gpu/cuda/cuda.c @@ -139,8 +139,10 @@ typedef uintptr_t cuda_ptr_key; /* Single entry of the memory list */ struct mem_entry { CUdeviceptr ptr_d; + CUdeviceptr ptr_orig_d; void *ptr_h; size_t size; + size_t size_orig; struct rte_gpu *dev; CUcontext ctx; cuda_ptr_key pkey; @@ -569,7 +571,7 @@ cuda_dev_info_get(struct rte_gpu *dev, struct rte_gpu_info *info) */ static int -cuda_mem_alloc(struct rte_gpu *dev, size_t size, void **ptr) +cuda_mem_alloc(struct rte_gpu *dev, size_t size, unsigned int align, void **ptr) { CUresult res; const char *err_string; @@ -610,8 +612,10 @@ cuda_mem_alloc(struct rte_gpu *dev, size_t size, void **ptr) /* Allocate memory */ mem_alloc_list_tail->size = size; - res = pfn_cuMemAlloc(&(mem_alloc_list_tail->ptr_d), - mem_alloc_list_tail->size); + mem_alloc_list_tail->size_orig = size + align; + + res = pfn_cuMemAlloc(&(mem_alloc_list_tail->ptr_orig_d), + mem_alloc_list_tail->size_orig); if (res != 0) { pfn_cuGetErrorString(res, &(err_string)); rte_cuda_log(ERR, "cuCtxSetCurrent current failed with %s", @@ -620,6 +624,13 @@ cuda_mem_alloc(struct rte_gpu *dev, size_t size, void **ptr) return -rte_errno; } + + /* Align memory address */ + mem_alloc_list_tail->ptr_d = mem_alloc_list_tail->ptr_orig_d; + if (align && ((uintptr_t)mem_alloc_list_tail->ptr_d) % align) + mem_alloc_list_tail->ptr_d += (align - + (((uintptr_t)mem_alloc_list_tail->ptr_d) % align)); + /* GPUDirect RDMA attribute required */ res = pfn_cuPointerSetAttribute(&flag, CU_POINTER_ATTRIBUTE_SYNC_MEMOPS, @@ -634,7 +645,6 @@ cuda_mem_alloc(struct rte_gpu *dev, size_t size, void **ptr) mem_alloc_list_tail->pkey = get_hash_from_ptr((void *)mem_alloc_list_tail->ptr_d); mem_alloc_list_tail->ptr_h = NULL; - mem_alloc_list_tail->size = size; mem_alloc_list_tail->dev = dev; mem_alloc_list_tail->ctx = (CUcontext)((uintptr_t)dev->mpshared->info.context); mem_alloc_list_tail->mtype = GPU_MEM; @@ -761,6 +771,7 @@ cuda_mem_register(struct rte_gpu *dev, size_t size, void *ptr) mem_alloc_list_tail->dev = dev; mem_alloc_list_tail->ctx = (CUcontext)((uintptr_t)dev->mpshared->info.context); mem_alloc_list_tail->mtype = CPU_REGISTERED; + mem_alloc_list_tail->ptr_orig_d = mem_alloc_list_tail->ptr_d; /* Restore original ctx as current ctx */ res = pfn_cuCtxSetCurrent(current_ctx); @@ -796,7 +807,7 @@ cuda_mem_free(struct rte_gpu *dev, void *ptr) } if (mem_item->mtype == GPU_MEM) { - res = pfn_cuMemFree(mem_item->ptr_d); + res = pfn_cuMemFree(mem_item->ptr_orig_d); if (res != 0) { pfn_cuGetErrorString(res, &(err_string)); rte_cuda_log(ERR, "cuMemFree current failed with %s",