From patchwork Fri Aug 11 20:00:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adam Hassick X-Patchwork-Id: 47 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2545043038; Fri, 11 Aug 2023 22:03:02 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1EF1140F16; Fri, 11 Aug 2023 22:03:02 +0200 (CEST) Received: from mail-qt1-f196.google.com (mail-qt1-f196.google.com [209.85.160.196]) by mails.dpdk.org (Postfix) with ESMTP id 1396E40144 for ; Fri, 11 Aug 2023 22:03:01 +0200 (CEST) Received: by mail-qt1-f196.google.com with SMTP id d75a77b69052e-4103c8157ccso3561351cf.1 for ; Fri, 11 Aug 2023 13:03:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; t=1691784180; x=1692388980; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=QYgZVmhrpAPXfK91eJ5pZfEVC3ZP9fPp+kEHjbbbrsM=; b=XcBAH0gWi1UM2m+9o3fdiw7v6Kh5jXrGEp447wlDzC0ECtj34Ce8xhFS42Xgyq/SVN 0ENOWXqvaPEqpPy8pqZKy0BzkPXXaEdcQDiX/i2soMTwyCkUSeAqE2LkPUOrZ25bEcaQ RdUpeXO0sPm+Zq0wPmPWOplygG9a9bAi/7T9Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691784180; x=1692388980; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=QYgZVmhrpAPXfK91eJ5pZfEVC3ZP9fPp+kEHjbbbrsM=; b=R/0Y5cgmaksLSgagfSD9D2ecUXtw+bR3gbaN5tuOs4ts1S/TSlop5CrXtCu9dlKX47 vpyIA0gQBmRtm1N3YIbI6KjKaF1Jq7WWJVnprp24E3A8C2C19zQig7U7aEHQENl0d7G1 m7/9ek1Y10SSdnyQgUpzxjderLi/wu+0KQ+M94S8wG4X/DPkV9q6kCzSj/kJ3ptAbKxT SuiSW8JM/Df0y5+t7d+4d0gTDQHZoex1bOAXc6hyZ2e0D+AesEhBfQKsQmuc/CD9AiNf PErCTsbtsQZHg1ZzXDP49ku5LRqRRBgjOzTsoLaxMqVoMCXiRS5VeaY868rsAxK6r/uw errg== X-Gm-Message-State: AOJu0YwMZngO8trgWTD46wF7Y/FKAt+/bduRRwIUD/MdAmjl+fpogm8a sn54m9V0SWe/+89u0kP8SYNUJeuSHNOVmEWp3Txo4nxIowly27oBuAT4QPkLIq47XSKArcg1494 5C/WlL1eGilx3yRzXNKEQfXtos2SUxEH2Jb/ACb3S8UcZLj4mYNi99ZFeKp7jFsPAlhvamw== X-Google-Smtp-Source: AGHT+IFLfyYq3EzK1TP81xvbGJmL56TRB5IDhc2zN13aVwhYDUIyTzJ2HwcKH+9NKiybCumnhA9QDA== X-Received: by 2002:ac8:7d94:0:b0:40f:f3d0:2ab6 with SMTP id c20-20020ac87d94000000b0040ff3d02ab6mr3785827qtd.6.1691784180374; Fri, 11 Aug 2023 13:03:00 -0700 (PDT) Received: from pogmachine2.loudonlune.net ([216.212.51.182]) by smtp.gmail.com with ESMTPSA id f19-20020ac84993000000b004053bcffe49sm1399874qtq.9.2023.08.11.13.02.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Aug 2023 13:03:00 -0700 (PDT) From: Adam Hassick To: ci@dpdk.org Cc: aconole@redhat.com, alialnu@nvidia.com, Adam Hassick Subject: [PATCH v9 0/6] Community Lab Containers and Builder Engine Date: Fri, 11 Aug 2023 16:00:12 -0400 Message-ID: <20230811200018.5650-1-ahassick@iol.unh.edu> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org This patch series contains a new version of the DPDK CI containers. The old version was tied very tightly to the Community Lab Infrastructure, so it was not suitable for general use. This version is designed to make adding new OSes or OS versions as easy as possible. The minimum functionality can easily be built on any system that can compile DPDK. It includes support for building containers for other non-native architectures (ex: arm containers on x86) and for baking ABI references into the images. The inventory file as added in this patch series defines what the community lab currently supports. If you want to build these yourself, don't try to do parallel Makefile builds on non-server hardware. Libabigail is built into the containers, and since it is not avaliable in all distros it is compiled from source for many targets. If embedding the abi is enabled (DPDK_CI_CONTAINERS_BUILD_ABI=Y), with the current settings, DPDK will be compiled twice for every target in the inventory file. This will become three times when DPDK main has ABI stability again. Due to how the containers are constructed, building on non-native architectures is especially expensive because DPDK is compiled with an emulated compiler in a qemu vm that only has one thread. --- v2: * Added poetry.lock * Remove duplicates in inventory file * Fix typo in base container template v3: * Libabigail is now not built by default unless DPDK_CI_CONTAINERS_BUILD_ABI=y * Made minimum podman version 4.0.0 due to dependencies on cache mounts v4: * Fix whitespace errors v5: * Updated container templates and inventory * Add support for OCI manifests * Update README to reflect changes v6: * Fix spelling errors and whitespace errors v7: * Add workaround to Arch template to install Python packages as root * Compare against v4 and polish the patchset v8: * Fix typos * Correct the default for the dpdk-stable clone URL v9: * Add ACVP FIPS container * Run template rendering script through black * Update copyright years * Remove unneeded comments Owen Hilyard (6): containers/docs: Add container builder start containers/inventory: Add inventory for container builder containers/builder: Dockerfile creation script containers/templates: Templates for Dockerfiles containers/container_builder: Container for python scripts containers/Makefile: Makefile to automate builds containers/Makefile | 250 ++++++++++ containers/README.md | 178 +++++++ containers/container_builder.dockerfile | 26 + containers/template_engine/inventory.yaml | 470 ++++++++++++++++++ .../template_engine/inventory_schema.json | 221 ++++++++ containers/template_engine/make_dockerfile.py | 358 +++++++++++++ containers/template_engine/poetry.lock | 220 ++++++++ containers/template_engine/pyproject.toml | 21 + .../templates/containers.makefile.j2 | 73 +++ .../templates/containers/alpine.dockerfile.j2 | 3 + .../templates/containers/arch.dockerfile.j2 | 39 ++ .../templates/containers/base.dockerfile.j2 | 120 +++++ .../containers/centos8.dockerfile.j2 | 21 + .../containers/centos9.dockerfile.j2 | 17 + .../templates/containers/debian.dockerfile.j2 | 7 + .../containers/debian10.dockerfile.j2 | 3 + .../containers/debian11.dockerfile.j2 | 3 + .../debian11_arm_ipsec.dockerfile.j2 | 16 + .../containers/debian_bullseye.dockerfile.j2 | 3 + .../containers/debian_buster.dockerfile.j2 | 3 + .../templates/containers/fedora.dockerfile.j2 | 11 + .../containers/fedora36_clang.dockerfile.j2 | 7 + .../containers/fedora_clang.dockerfile.j2 | 8 + .../containers/fedora_coverity.dockerfile.j2 | 10 + .../containers/opensuse.dockerfile.j2 | 10 + .../containers/redhat_family.dockerfile.j2 | 5 + .../templates/containers/rhel.dockerfile.j2 | 16 + .../templates/containers/rhel7.dockerfile.j2 | 15 + .../templates/containers/rhel8.dockerfile.j2 | 15 + .../templates/containers/rhel9.dockerfile.j2 | 19 + .../templates/containers/rpm.dockerfile.j2 | 3 + .../templates/containers/ubuntu.dockerfile.j2 | 3 + .../containers/ubuntu20.04.dockerfile.j2 | 12 + .../containers/ubuntu22.04.dockerfile.j2 | 3 + .../containers/ubuntu_cross.dockerfile.j2 | 11 + .../containers/ubuntu_sve.dockerfile.j2 | 12 + 36 files changed, 2212 insertions(+) create mode 100644 containers/Makefile create mode 100644 containers/README.md create mode 100644 containers/container_builder.dockerfile create mode 100644 containers/template_engine/inventory.yaml create mode 100644 containers/template_engine/inventory_schema.json create mode 100755 containers/template_engine/make_dockerfile.py create mode 100644 containers/template_engine/poetry.lock create mode 100644 containers/template_engine/pyproject.toml create mode 100644 containers/template_engine/templates/containers.makefile.j2 create mode 100644 containers/template_engine/templates/containers/alpine.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/arch.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/base.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/centos8.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/centos9.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian10.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian11.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian11_arm_ipsec.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/debian_buster.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora_clang.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/fedora_coverity.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/opensuse.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/redhat_family.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel7.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel8.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rhel9.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/rpm.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2 create mode 100644 containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2