Cover Detail
Show a cover letter.
GET /api/covers/105529/?format=api
{ "id": 105529, "url": "http://patchwork.dpdk.org/api/covers/105529/?format=api", "web_url": "http://patchwork.dpdk.org/project/dpdk/cover/20211230143744.3550098-1-dkozlyuk@nvidia.com/", "project": { "id": 1, "url": "http://patchwork.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<20211230143744.3550098-1-dkozlyuk@nvidia.com>", "list_archive_url": "https://inbox.dpdk.org/dev/20211230143744.3550098-1-dkozlyuk@nvidia.com", "date": "2021-12-30T14:37:38", "name": "[RFC,0/6] Fast restart with many hugepages", "submitter": { "id": 2248, "url": "http://patchwork.dpdk.org/api/people/2248/?format=api", "name": "Dmitry Kozlyuk", "email": "dkozlyuk@nvidia.com" }, "mbox": "http://patchwork.dpdk.org/project/dpdk/cover/20211230143744.3550098-1-dkozlyuk@nvidia.com/mbox/", "series": [ { "id": 21042, "url": "http://patchwork.dpdk.org/api/series/21042/?format=api", "web_url": "http://patchwork.dpdk.org/project/dpdk/list/?series=21042", "date": "2021-12-30T14:37:38", "name": "Fast restart with many hugepages", "version": 1, "mbox": "http://patchwork.dpdk.org/series/21042/mbox/" } ], "comments": "http://patchwork.dpdk.org/api/covers/105529/comments/", "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@inbox.dpdk.org", "Delivered-To": "patchwork@inbox.dpdk.org", "Received": [ "from mails.dpdk.org (mails.dpdk.org [217.70.189.124])\n\tby inbox.dpdk.org (Postfix) with ESMTP id 30A15A04A5;\n\tThu, 30 Dec 2021 15:38:11 +0100 (CET)", "from [217.70.189.124] (localhost [127.0.0.1])\n\tby mails.dpdk.org (Postfix) with ESMTP id DD70D410F1;\n\tThu, 30 Dec 2021 15:38:10 +0100 (CET)", "from NAM11-CO1-obe.outbound.protection.outlook.com\n (mail-co1nam11on2048.outbound.protection.outlook.com [40.107.220.48])\n by mails.dpdk.org (Postfix) with ESMTP id 3B23540F35\n for <dev@dpdk.org>; Thu, 30 Dec 2021 15:38:10 +0100 (CET)", "from CO2PR18CA0046.namprd18.prod.outlook.com (2603:10b6:104:2::14)\n by CH0PR12MB5267.namprd12.prod.outlook.com (2603:10b6:610:d2::18) with\n Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4823.19; Thu, 30 Dec\n 2021 14:38:08 +0000", "from CO1NAM11FT053.eop-nam11.prod.protection.outlook.com\n (2603:10b6:104:2:cafe::86) by CO2PR18CA0046.outlook.office365.com\n (2603:10b6:104:2::14) with Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4844.14 via Frontend\n Transport; Thu, 30 Dec 2021 14:38:08 +0000", "from mail.nvidia.com (12.22.5.236) by\n CO1NAM11FT053.mail.protection.outlook.com (10.13.175.63) with Microsoft SMTP\n Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id\n 15.20.4844.14 via Frontend Transport; Thu, 30 Dec 2021 14:38:08 +0000", "from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL109.nvidia.com\n (10.27.9.19) with Microsoft SMTP Server (TLS) id 15.0.1497.18;\n Thu, 30 Dec 2021 14:38:01 +0000", "from nvidia.com (172.20.187.6) by rnnvmail201.nvidia.com\n (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2,\n cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.2.986.9; Thu, 30 Dec 2021\n 06:37:59 -0800" ], "ARC-Seal": "i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;\n b=EbS7Ee0cD7h3xQBIfYwPV2PQ5XyROJeQcveTPwsEXxlzgMzQ2dB6s4kw/hpQwMCJMMwL3hmfzZ07oEjhuOnRu4KgsY7vRb6S31nEoIukVBikuuElzdqLEiiieVo5nRdBmqrNiwqoo/qtBPErtSHMvgLqBgzbU75v8s7JdnbSYYYWYyZLObMCU5YY5mEtvhbDnKlmAXFyQHBtIJQmcvNn+CS0vmJmfdYq+C2ENemzFlRBobXLCZIHz65Y4b2bUA6lzrTlmRXMyGb8mrx6Pv1OYWTzAid8EfWbnla75J8sdLoEnUT6taUS0NK6xgh84nAqT0I5TV2Ld7nntfzutEw2iQ==", "ARC-Message-Signature": "i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;\n s=arcselector9901;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1;\n bh=JLjampzGceTINDko69741thqY2OH3IaiPNC3I4piBgA=;\n b=RPqG1KuTrxm9rYvp+PbRAzBgyxONzdkVkEU4YD41eXocx0IBm7PkaGZgMTu1MsNT0M9wQKXp5o9fh8ENMXDeFhWUN2JcyeppjnUTBrDjek6/+BtP1YbF85oRLHcVwlUD4oBAj0L0EqyW4wjJBzKCykxabue7pytXBpHeguY7QkdVslhiKTDQZODIdSj1mLPoikUlVbYrcHA9WolW8KBbvzda6p02vXGg57ekpq1TQdcbo4Wwh+Ia0QUk2al9OyywD4wXq6/0hqxXhGtYc0lahhinUcB+QZqcfXKOFxKcxzy0yTwC2OKbrXYEPahErsbipPQQZyFXdZ89yozLarf62g==", "ARC-Authentication-Results": "i=1; mx.microsoft.com 1; spf=pass (sender ip is\n 12.22.5.236) smtp.rcpttodomain=redhat.com smtp.mailfrom=nvidia.com;\n dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com;\n dkim=none (message not signed); arc=none", "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com;\n s=selector2;\n h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;\n bh=JLjampzGceTINDko69741thqY2OH3IaiPNC3I4piBgA=;\n b=ra7e1Z1zrzTkA3ONdiZA6Ls+YFk56SBQFnd3tyx+McNJxDBwASMU470+XQNYtxb82O6SYcPnRG7gi6kt66qLqZMWQxgCprD1YD0zZ8/T+LRDssv7B2NZ3QR3biJwZWOOSj2wrWGI4Ot5J5hZcb6CGKLQgOToqEh8q3ZrVHQemQZoCBIvZJrGZDfIOImswOQ5RNGC48VohcCVjlTjjXJttDipUu4bq19rdAl/Nnkx5vVXlhob9hP+8kAjbKqEBid34fQCpsiwEde73QCwuQrj8Y4P7+m4mN5u3kjTjfB9Dplr5wSmofHvlwzFAaKW3AlfydTbAIV7wSpMk0pbOPjllA==", "X-MS-Exchange-Authentication-Results": "spf=pass (sender IP is 12.22.5.236)\n smtp.mailfrom=nvidia.com; dkim=none (message not signed)\n header.d=none;dmarc=pass action=none header.from=nvidia.com;", "Received-SPF": "Pass (protection.outlook.com: domain of nvidia.com designates\n 12.22.5.236 as permitted sender) receiver=protection.outlook.com;\n client-ip=12.22.5.236; helo=mail.nvidia.com;", "From": "Dmitry Kozlyuk <dkozlyuk@nvidia.com>", "To": "<dev@dpdk.org>", "CC": "Anatoly Burakov <anatoly.burakov@intel.com>, Viacheslav Ovsiienko\n <viacheslavo@nvidia.com>, David Marchand <david.marchand@redhat.com>, \"Thomas\n Monjalon\" <thomas@monjalon.net>, Lior Margalit <lmargalit@nvidia.com>", "Subject": "[RFC PATCH 0/6] Fast restart with many hugepages", "Date": "Thu, 30 Dec 2021 16:37:38 +0200", "Message-ID": "<20211230143744.3550098-1-dkozlyuk@nvidia.com>", "X-Mailer": "git-send-email 2.25.1", "MIME-Version": "1.0", "Content-Transfer-Encoding": "8bit", "Content-Type": "text/plain", "X-Originating-IP": "[172.20.187.6]", "X-ClientProxiedBy": "HQMAIL105.nvidia.com (172.20.187.12) To\n rnnvmail201.nvidia.com (10.129.68.8)", "X-EOPAttributedMessage": "0", "X-MS-PublicTrafficType": "Email", "X-MS-Office365-Filtering-Correlation-Id": "0c30b0ab-e952-4026-df7c-08d9cba1ff43", "X-MS-TrafficTypeDiagnostic": "CH0PR12MB5267:EE_", "X-LD-Processed": "43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr", "X-Microsoft-Antispam-PRVS": "\n <CH0PR12MB526721E1FCC063A296DDC6AFB9459@CH0PR12MB5267.namprd12.prod.outlook.com>", "X-MS-Oob-TLC-OOBClassifiers": "OLM:8882;", "X-MS-Exchange-SenderADCheck": "1", "X-MS-Exchange-AntiSpam-Relay": "0", "X-Microsoft-Antispam": "BCL:0;", "X-Microsoft-Antispam-Message-Info": "\n iSRrYpevtMu4dytlmaTkC8j06w+AgNZN4x8ZtWS6yGILH9L7kVbaffJV0jpab2tMofDBWAPtvtrXmWgb39oNbTFfbFfkK5Y/2yIzrDmPqjc3mZ0sRKQEt+Gpdn7QqerpRZ1QsOCS/P8MoK0C8jYFuDMAV3HiLNvtXTvjLKDQosAKkzMaKpIuezpdxws6Y3qC3+Kp5nRFXOep+RXPmO+hvVIIfdbKrE78OhXXRtpn1jEt3lP7Nku8AEGsSEQSp0ldjohRlbxx+Un3CzO8sLqp39t5DzJpoxMZmTzkkqeG6Q+ekmfWEBx1lakoMM6q9GK4oj5H34YdMElPr+s5ZnPBoGfR72tnssP8TufiEytIxqVZ4tUAW6u0qHEMvWv2Sa9eiS3E4/qGyDol2kV9Qv4CpvmEnRIlncLjDfGssyOyQxNoyzEriotBh+fy2GfMWJ80Hx25n+PigL5BXCEwKUtTJWaAMdqXRkL2Z+5IVH3PJRxGCKYLKAaCp19Whja4ufYns62jo8D1a49yvK42p0uHi7DFwtq3ap1CB1D7GylCIEtYdMaNU31T+AvmsaQoGdqPSw2/HVbTV9hd08pbhwynDc9/i+OskkxgX226taBo2yCiPvsbnms0N01jquyFMcwaa/B4n/yV+fYPTjf8hHQwxXf9UHCzbXvvPoFdGmv38vbCFD1nsE4vEt15+qYWQ2OWj6oesHKYkS44ikClwKG54zlW9nx7GBETQfydXxvUACmL3jdnRCKHMgBlMTAO1ovY1lsUxJb8x4LsmVVnscY+GL0QYpaZBmm6m3OIhAZ1LkL686LRRj0FAdxWAqfHCUxfehAzwRPeseES+zkiSbS/xnK/RXFEmW5TtYjenXLvBXigNlmcLyJoIvLdhnXGXi5i", "X-Forefront-Antispam-Report": "CIP:12.22.5.236; CTRY:US; LANG:en; SCL:1; SRV:;\n IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE;\n SFS:(4636009)(40470700002)(36840700001)(46966006)(336012)(966005)(81166007)(16526019)(186003)(426003)(6286002)(8936002)(36860700001)(7696005)(2906002)(86362001)(6916009)(316002)(40460700001)(70586007)(26005)(107886003)(83380400001)(356005)(508600001)(47076005)(70206006)(55016003)(82310400004)(36756003)(1076003)(8676002)(6666004)(5660300002)(4326008)(2616005)(54906003)(36900700001);\n DIR:OUT; SFP:1101;", "X-OriginatorOrg": "Nvidia.com", "X-MS-Exchange-CrossTenant-OriginalArrivalTime": "30 Dec 2021 14:38:08.1353 (UTC)", "X-MS-Exchange-CrossTenant-Network-Message-Id": "\n 0c30b0ab-e952-4026-df7c-08d9cba1ff43", "X-MS-Exchange-CrossTenant-Id": "43083d15-7273-40c1-b7db-39efd9ccc17a", "X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp": "\n TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.236];\n Helo=[mail.nvidia.com]", "X-MS-Exchange-CrossTenant-AuthSource": "\n CO1NAM11FT053.eop-nam11.prod.protection.outlook.com", "X-MS-Exchange-CrossTenant-AuthAs": "Anonymous", "X-MS-Exchange-CrossTenant-FromEntityHeader": "HybridOnPrem", "X-MS-Exchange-Transport-CrossTenantHeadersStamped": "CH0PR12MB5267", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.29", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n <mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n <mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org" }, "content": "This patchset is a new design and implementation of [1].\n\n# Problem Statement\n\nLarge allocations that involve mapping new hugepages are slow.\nThis is problematic, for example, in the following use case.\nA single-process application allocates ~1TB of mempools at startup.\nSometimes the app needs to restart as quick as possible.\nAllocating the hugepages anew takes as long as 15 seconds,\nwhile the new process could just pick up all the memory\nleft by the old one (reinitializing the contents as needed).\n\nAlmost all of mmap(2) time spent in the kernel\nis clearing the memory, i.e. filling it with zeros.\nThis is done if a file in hugetlbfs is mapped\nfor the first time system-wide, i.e. a hugepage is committed\nto prevent data leaks from the previous users of the same hugepage.\nFor example, mapping 32 GB from a new file may take 2.16 seconds,\nwhile mapping the same pages again takes only 0.3 ms.\nSecurity put aside, e.g. when the environment is controlled,\nthis effort is wasted for the memory intended for DMA,\nbecause its content will be overwritten anyway.\n\nLinux EAL explicitly removes hugetlbfs files at initialization\nand before mapping to force the kernel clear the memory.\nThis allows the memory allocator to clean memory on only on freeing.\n\n# Solution\n\nAdd a new mode allowing EAL to remap existing hugepage files.\nWhile it is intended to make restarts faster in the first place,\nit makes any startup faster except the cold one\n(with no existing files).\n\nIt is the administrator who accepts security risks\nimplied by reusing hugepages.\nThe new mode is an opt-in and a warning is logged.\n\nThe feature is Linux-only as it is related\nto mapping hugepages from files which only Linux does.\nIt is inherently incompatible with --in-memory,\nfor --huge-unlink see below.\n\nThere is formally no breakage of API contract,\nbut there is a behavior change in the new mode:\nrte_malloc*() and rte_memzone_reserve*() may return dirty memory\n(previously they were returning clean memory from free heap elements).\nTheir contract has always explicitly allowed this,\nbut still there may be users relying on the traditional behavior.\nSuch users will need to fix their code to use the new mode.\n\n# Implementation\n\n## User Interface\n\nThere is --huge-unlink switch in the same area to remove hugepage files\nbefore mapping them. It is infeasible to use with the new mode,\nbecause the point is to keep hugepage files for fast future restarts.\nExtend --huge-unlink option to represent only valid combinations:\n\n* --huge-unlink=existing OR no option (for compatibility):\n unlink files at initialization\n and before opening them as a precaution.\n\n* --huge-unlink=always OR just --huge-unlink (for compatibility):\n same as above + unlink created files before mapping.\n\n* --huge-unlink=never:\n the new mode, do not unlink hugepages files, reuse them.\n\nThis option was always Linux-only, but it is kept as common\nin case there are users who expect it to be a no-op on other systems.\n(Adding a separate --huge-reuse option was also considered,\nbut there is no obvious benefit and more combinations to test.)\n\n## EAL\n\nIf a memseg is mapped dirty, it is marked with RTE_MEMSEG_FLAG_DIRTY\nso that the memory allocator may clear the memory if need be.\nSee patch 4/6 description for details.\n\nThe memory manager tracks whether an element is clean or dirty.\nIf rte_zmalloc*() allocates from a dirty element,\nthe memory is cleared before handling it to the user.\nOn freeing, the allocator joins adjacent free elements,\nbut in the new mode it may not be feasible to clear the free memory\nif the joint element is dirty (contains dirty parts).\nIn any case, memory will be cleared only once,\neither on freeing or on allocation. See patch 2/6 for details.\nPatch 6/6 adds a benchmark to see how time is distributed\nbetween allocation and freeing in different modes.\n\nBesides clearing memory, each mmap() call takes some time which adds up.\nEAL does one call per hugepage, 1024 calls for 1 TB may take ~300 ms.\nIt does so in order to be able to unmap the segments one by one.\nHowever, segments from initial allocation (-m) are never unmapped.\nIdeally, initial allocation should take one mmap() call per memory type\n(i.e. per NUMA node per page size) if --single-file-segments is used.\nThis further optimization is not implemented in current version.\n\n[1]: http://inbox.dpdk.org/dev/20211011085644.2716490-3-dkozlyuk@nvidia.com/\n\nDmitry Kozlyuk (6):\n doc: add hugepage mapping details\n mem: add dirty malloc element support\n eal: refactor --huge-unlink storage\n eal/linux: allow hugepage file reuse\n eal: allow hugepage file reuse with --huge-unlink\n app/test: add allocator performance benchmark\n\n app/test/meson.build | 2 +\n app/test/test_malloc_perf.c | 174 ++++++++++++++++++\n doc/guides/linux_gsg/linux_eal_parameters.rst | 21 ++-\n .../prog_guide/env_abstraction_layer.rst | 94 +++++++++-\n doc/guides/rel_notes/release_22_03.rst | 7 +\n lib/eal/common/eal_common_options.c | 46 ++++-\n lib/eal/common/eal_internal_cfg.h | 10 +-\n lib/eal/common/malloc_elem.c | 22 ++-\n lib/eal/common/malloc_elem.h | 11 +-\n lib/eal/common/malloc_heap.c | 18 +-\n lib/eal/common/rte_malloc.c | 21 ++-\n lib/eal/include/rte_memory.h | 8 +-\n lib/eal/linux/eal_hugepage_info.c | 59 ++++--\n lib/eal/linux/eal_memalloc.c | 164 ++++++++++-------\n lib/eal/linux/eal_memory.c | 2 +-\n 15 files changed, 537 insertions(+), 122 deletions(-)\n create mode 100644 app/test/test_malloc_perf.c" }