[v7,0/6] Community Lab Containers and Builder Engine

  This patch series contains a new version of the DPDK CI containers. The old
version was tied very tightly to the Community Lab Infrastructure, so it was
not suitable for general use. This version is designed to make adding new
OSes or OS versions as easy as possible. The minimum functionality can easily
be built on any system that can compile DPDK. It includes support for
building containers for other non-native architectures (ex: arm containers
on x86) and for baking ABI references into the images.

The inventory file as added in this patch series defines what the community lab
currently supports.

If you want to build these yourself, don't try to do parallel Makefile builds
on non-server hardware. Libabigail is built into the containers, and since it
is not avaliable in all distros it is compiled from source for many targets.
If embedding the abi is enabled (DPDK_CI_CONTAINERS_BUILD_ABI=Y), with the
current settings, DPDK will be compiled twice for every target in the
inventory file. This will become three times when DPDK main has ABI stability
again. Due to how the containers are constructed, building on non-native
architectures is especially expensive because DPDK is compiled with an emulated
compiler in a qemu vm that only has one thread.

---

v2:
* Added poetry.lock
* Remove duplicates in inventory file
* Fix typo in base container template

v3:
* Libabigail is now not built by default unless DPDK_CI_CONTAINERS_BUILD_ABI=y
* Made minimum podman version 4.0.0 due to dependencies on cache mounts

v4:
* Fix whitespace errors

v5:
* Updated container templates and inventory
* Add support for OCI manifests
* Update README to reflect changes

v6:
* Fix spelling errors and whitespace errors

v7:
* Add workaround to Arch template to install Python packages as root
* Compare against v4 and polish the patchset

Owen Hilyard (6):
  * Add README file for containers * Add pyproject file with required
    dependencies for building containers
  Adds a yaml file used to define what containers should be built, what
    packages the containers should have, what platforms to build for and
    how they should be tagged.
  This script will template out all of the Dockerfiles based on the
    definitions provided in the inventory using the jinja2 templating
    library.
  Adds a variety of extensible templates used to create the Dockerfiles
    for each target. All templates inherit from base, and most distros
    and distro families have their own templates that are used to define
    common functionality. Multiple versions of a distro may use a single
    template if they are similar enough (ex: fedora).
  Adds a container that can be used to run the python scripts to create
    the Dockerfiles for the CI containers. This removes a large number
    of python environment dependencies from the host requirements.
  The Makefile that can be used to build all of the container images
    using "make build", and can also be used to push them to a remote
    repository (for use in CI).

 containers/Makefile                           | 252 ++++++++++
 containers/README.md                          | 178 +++++++
 containers/container_builder.dockerfile       |  26 +
 containers/template_engine/inventory.yaml     | 470 ++++++++++++++++++
 .../template_engine/inventory_schema.json     | 221 ++++++++
 containers/template_engine/make_dockerfile.py | 358 +++++++++++++
 containers/template_engine/poetry.lock        | 220 ++++++++
 containers/template_engine/pyproject.toml     |  21 +
 .../templates/containers.makefile.j2          |  73 +++
 .../templates/containers/alpine.dockerfile.j2 |   3 +
 .../templates/containers/arch.dockerfile.j2   |  39 ++
 .../templates/containers/base.dockerfile.j2   | 120 +++++
 .../containers/centos8.dockerfile.j2          |  21 +
 .../containers/centos9.dockerfile.j2          |  17 +
 .../templates/containers/debian.dockerfile.j2 |   7 +
 .../containers/debian10.dockerfile.j2         |   3 +
 .../containers/debian11.dockerfile.j2         |   3 +
 .../debian11_arm_ipsec.dockerfile.j2          |  16 +
 .../containers/debian_bullseye.dockerfile.j2  |   3 +
 .../containers/debian_buster.dockerfile.j2    |   3 +
 .../templates/containers/fedora.dockerfile.j2 |  11 +
 .../containers/fedora36_clang.dockerfile.j2   |   7 +
 .../containers/fedora_clang.dockerfile.j2     |   8 +
 .../containers/fedora_coverity.dockerfile.j2  |  10 +
 .../containers/opensuse.dockerfile.j2         |  10 +
 .../containers/redhat_family.dockerfile.j2    |   5 +
 .../templates/containers/rhel.dockerfile.j2   |  16 +
 .../templates/containers/rhel7.dockerfile.j2  |  15 +
 .../templates/containers/rhel8.dockerfile.j2  |  15 +
 .../templates/containers/rhel9.dockerfile.j2  |  19 +
 .../templates/containers/rpm.dockerfile.j2    |   3 +
 .../templates/containers/ubuntu.dockerfile.j2 |   3 +
 .../containers/ubuntu20.04.dockerfile.j2      |  12 +
 .../containers/ubuntu22.04.dockerfile.j2      |   3 +
 .../containers/ubuntu_cross.dockerfile.j2     |  11 +
 .../containers/ubuntu_sve.dockerfile.j2       |  12 +
 36 files changed, 2214 insertions(+)
 create mode 100644 containers/Makefile
 create mode 100644 containers/README.md
 create mode 100644 containers/container_builder.dockerfile
 create mode 100644 containers/template_engine/inventory.yaml
 create mode 100644 containers/template_engine/inventory_schema.json
 create mode 100755 containers/template_engine/make_dockerfile.py
 create mode 100644 containers/template_engine/poetry.lock
 create mode 100644 containers/template_engine/pyproject.toml
 create mode 100644 containers/template_engine/templates/containers.makefile.j2
 create mode 100644 containers/template_engine/templates/containers/alpine.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/arch.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/base.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/centos8.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/centos9.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/debian.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/debian10.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/debian11.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/debian11_arm_ipsec.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/debian_buster.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/fedora.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/fedora_clang.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/fedora_coverity.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/opensuse.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/redhat_family.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/rhel.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/rhel7.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/rhel8.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/rhel9.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/rpm.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/ubuntu.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2
 create mode 100644 containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2
  

> From: Adam Hassick <ahassick@iol.unh.edu>
> Sent: Tuesday, July 11, 2023 11:21 PM
> To: ci@dpdk.org
> Cc: Ali Alnubani <alialnu@nvidia.com>; aconole@redhat.com; Adam Hassick
> <ahassick@iol.unh.edu>
> Subject: [PATCH v7 0/6] Community Lab Containers and Builder Engine
> 
> This patch series contains a new version of the DPDK CI containers. The old
> version was tied very tightly to the Community Lab Infrastructure, so it was
> not suitable for general use. This version is designed to make adding new
> OSes or OS versions as easy as possible. The minimum functionality can easily
> be built on any system that can compile DPDK. It includes support for
> building containers for other non-native architectures (ex: arm containers
> on x86) and for baking ABI references into the images.
> 
> The inventory file as added in this patch series defines what the community
> lab
> currently supports.
> 
> If you want to build these yourself, don't try to do parallel Makefile builds
> on non-server hardware. Libabigail is built into the containers, and since it
> is not avaliable in all distros it is compiled from source for many targets.
> If embedding the abi is enabled (DPDK_CI_CONTAINERS_BUILD_ABI=Y), with
> the
> current settings, DPDK will be compiled twice for every target in the
> inventory file. This will become three times when DPDK main has ABI stability
> again. Due to how the containers are constructed, building on non-native
> architectures is especially expensive because DPDK is compiled with an
> emulated
> compiler in a qemu vm that only has one thread.
> 
> ---
> 
> v2:
> * Added poetry.lock
> * Remove duplicates in inventory file
> * Fix typo in base container template
> 
> v3:
> * Libabigail is now not built by default unless
> DPDK_CI_CONTAINERS_BUILD_ABI=y
> * Made minimum podman version 4.0.0 due to dependencies on cache
> mounts
> 
> v4:
> * Fix whitespace errors
> 
> v5:
> * Updated container templates and inventory
> * Add support for OCI manifests
> * Update README to reflect changes
> 
> v6:
> * Fix spelling errors and whitespace errors
> 
> v7:
> * Add workaround to Arch template to install Python packages as root
> * Compare against v4 and polish the patchset
> 
> Owen Hilyard (6):
>   * Add README file for containers * Add pyproject file with required
>     dependencies for building containers
>   Adds a yaml file used to define what containers should be built, what
>     packages the containers should have, what platforms to build for and
>     how they should be tagged.
>   This script will template out all of the Dockerfiles based on the
>     definitions provided in the inventory using the jinja2 templating
>     library.
>   Adds a variety of extensible templates used to create the Dockerfiles
>     for each target. All templates inherit from base, and most distros
>     and distro families have their own templates that are used to define
>     common functionality. Multiple versions of a distro may use a single
>     template if they are similar enough (ex: fedora).
>   Adds a container that can be used to run the python scripts to create
>     the Dockerfiles for the CI containers. This removes a large number
>     of python environment dependencies from the host requirements.
>   The Makefile that can be used to build all of the container images
>     using "make build", and can also be used to push them to a remote
>     repository (for use in CI).
> 
>  containers/Makefile                           | 252 ++++++++++
>  containers/README.md                          | 178 +++++++
>  containers/container_builder.dockerfile       |  26 +
>  containers/template_engine/inventory.yaml     | 470 ++++++++++++++++++
>  .../template_engine/inventory_schema.json     | 221 ++++++++
>  containers/template_engine/make_dockerfile.py | 358 +++++++++++++
>  containers/template_engine/poetry.lock        | 220 ++++++++
>  containers/template_engine/pyproject.toml     |  21 +
>  .../templates/containers.makefile.j2          |  73 +++
>  .../templates/containers/alpine.dockerfile.j2 |   3 +
>  .../templates/containers/arch.dockerfile.j2   |  39 ++
>  .../templates/containers/base.dockerfile.j2   | 120 +++++
>  .../containers/centos8.dockerfile.j2          |  21 +
>  .../containers/centos9.dockerfile.j2          |  17 +
>  .../templates/containers/debian.dockerfile.j2 |   7 +
>  .../containers/debian10.dockerfile.j2         |   3 +
>  .../containers/debian11.dockerfile.j2         |   3 +
>  .../debian11_arm_ipsec.dockerfile.j2          |  16 +
>  .../containers/debian_bullseye.dockerfile.j2  |   3 +
>  .../containers/debian_buster.dockerfile.j2    |   3 +
>  .../templates/containers/fedora.dockerfile.j2 |  11 +
>  .../containers/fedora36_clang.dockerfile.j2   |   7 +
>  .../containers/fedora_clang.dockerfile.j2     |   8 +
>  .../containers/fedora_coverity.dockerfile.j2  |  10 +
>  .../containers/opensuse.dockerfile.j2         |  10 +
>  .../containers/redhat_family.dockerfile.j2    |   5 +
>  .../templates/containers/rhel.dockerfile.j2   |  16 +
>  .../templates/containers/rhel7.dockerfile.j2  |  15 +
>  .../templates/containers/rhel8.dockerfile.j2  |  15 +
>  .../templates/containers/rhel9.dockerfile.j2  |  19 +
>  .../templates/containers/rpm.dockerfile.j2    |   3 +
>  .../templates/containers/ubuntu.dockerfile.j2 |   3 +
>  .../containers/ubuntu20.04.dockerfile.j2      |  12 +
>  .../containers/ubuntu22.04.dockerfile.j2      |   3 +
>  .../containers/ubuntu_cross.dockerfile.j2     |  11 +
>  .../containers/ubuntu_sve.dockerfile.j2       |  12 +
>  36 files changed, 2214 insertions(+)
>  create mode 100644 containers/Makefile
>  create mode 100644 containers/README.md
>  create mode 100644 containers/container_builder.dockerfile
>  create mode 100644 containers/template_engine/inventory.yaml
>  create mode 100644 containers/template_engine/inventory_schema.json
>  create mode 100755 containers/template_engine/make_dockerfile.py
>  create mode 100644 containers/template_engine/poetry.lock
>  create mode 100644 containers/template_engine/pyproject.toml
>  create mode 100644
> containers/template_engine/templates/containers.makefile.j2
>  create mode 100644
> containers/template_engine/templates/containers/alpine.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/arch.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/base.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/centos8.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/centos9.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/debian.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/debian10.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/debian11.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/debian11_arm_ipsec.dock
> erfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/debian_bullseye.dockerfile
> .j2
>  create mode 100644
> containers/template_engine/templates/containers/debian_buster.dockerfile.j
> 2
>  create mode 100644
> containers/template_engine/templates/containers/fedora.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/fedora36_clang.dockerfile.
> j2
>  create mode 100644
> containers/template_engine/templates/containers/fedora_clang.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/fedora_coverity.dockerfile.
> j2
>  create mode 100644
> containers/template_engine/templates/containers/opensuse.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/redhat_family.dockerfile.j
> 2
>  create mode 100644
> containers/template_engine/templates/containers/rhel.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/rhel7.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/rhel8.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/rhel9.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/rpm.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/ubuntu.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2
>  create mode 100644
> containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j
> 2
>  create mode 100644
> containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2
> 
> --
> 2.34.1
> 

Tested on X86 with DPDK_CI_CONTAINERS_ONLY_HOST_ARCH=Y.
Noticed a few typos, the rest looks good to me.

Tested-by: Ali Alnubani <alialnu@nvidia.com>
Acked-by: Ali Alnubani <alialnu@nvidia.com>

Thanks,
Ali
  

Adam Hassick <ahassick@iol.unh.edu> writes:

> This patch series contains a new version of the DPDK CI containers. The old
> version was tied very tightly to the Community Lab Infrastructure, so it was
> not suitable for general use. This version is designed to make adding new
> OSes or OS versions as easy as possible. The minimum functionality can easily
> be built on any system that can compile DPDK. It includes support for
> building containers for other non-native architectures (ex: arm containers
> on x86) and for baking ABI references into the images.
>
> The inventory file as added in this patch series defines what the community lab
> currently supports.
>
> If you want to build these yourself, don't try to do parallel Makefile builds
> on non-server hardware. Libabigail is built into the containers, and since it
> is not avaliable in all distros it is compiled from source for many targets.
> If embedding the abi is enabled (DPDK_CI_CONTAINERS_BUILD_ABI=Y), with the
> current settings, DPDK will be compiled twice for every target in the
> inventory file. This will become three times when DPDK main has ABI stability
> again. Due to how the containers are constructed, building on non-native
> architectures is especially expensive because DPDK is compiled with an emulated
> compiler in a qemu vm that only has one thread.
>
> ---
>
> v2:
> * Added poetry.lock
> * Remove duplicates in inventory file
> * Fix typo in base container template
>
> v3:
> * Libabigail is now not built by default unless DPDK_CI_CONTAINERS_BUILD_ABI=y
> * Made minimum podman version 4.0.0 due to dependencies on cache mounts
>
> v4:
> * Fix whitespace errors
>
> v5:
> * Updated container templates and inventory
> * Add support for OCI manifests
> * Update README to reflect changes
>
> v6:
> * Fix spelling errors and whitespace errors
>
> v7:
> * Add workaround to Arch template to install Python packages as root
> * Compare against v4 and polish the patchset

Please can you repost with the spelling fixes, and also fixing Owen's
email address? (it is currently listed like a mailing list archive).

> Owen Hilyard (6):
>   * Add README file for containers * Add pyproject file with required
>     dependencies for building containers
>   Adds a yaml file used to define what containers should be built, what
>     packages the containers should have, what platforms to build for and
>     how they should be tagged.
>   This script will template out all of the Dockerfiles based on the
>     definitions provided in the inventory using the jinja2 templating
>     library.
>   Adds a variety of extensible templates used to create the Dockerfiles
>     for each target. All templates inherit from base, and most distros
>     and distro families have their own templates that are used to define
>     common functionality. Multiple versions of a distro may use a single
>     template if they are similar enough (ex: fedora).
>   Adds a container that can be used to run the python scripts to create
>     the Dockerfiles for the CI containers. This removes a large number
>     of python environment dependencies from the host requirements.
>   The Makefile that can be used to build all of the container images
>     using "make build", and can also be used to push them to a remote
>     repository (for use in CI).
>
>  containers/Makefile                           | 252 ++++++++++
>  containers/README.md                          | 178 +++++++
>  containers/container_builder.dockerfile       |  26 +
>  containers/template_engine/inventory.yaml     | 470 ++++++++++++++++++
>  .../template_engine/inventory_schema.json     | 221 ++++++++
>  containers/template_engine/make_dockerfile.py | 358 +++++++++++++
>  containers/template_engine/poetry.lock        | 220 ++++++++
>  containers/template_engine/pyproject.toml     |  21 +
>  .../templates/containers.makefile.j2          |  73 +++
>  .../templates/containers/alpine.dockerfile.j2 |   3 +
>  .../templates/containers/arch.dockerfile.j2   |  39 ++
>  .../templates/containers/base.dockerfile.j2   | 120 +++++
>  .../containers/centos8.dockerfile.j2          |  21 +
>  .../containers/centos9.dockerfile.j2          |  17 +
>  .../templates/containers/debian.dockerfile.j2 |   7 +
>  .../containers/debian10.dockerfile.j2         |   3 +
>  .../containers/debian11.dockerfile.j2         |   3 +
>  .../debian11_arm_ipsec.dockerfile.j2          |  16 +
>  .../containers/debian_bullseye.dockerfile.j2  |   3 +
>  .../containers/debian_buster.dockerfile.j2    |   3 +
>  .../templates/containers/fedora.dockerfile.j2 |  11 +
>  .../containers/fedora36_clang.dockerfile.j2   |   7 +
>  .../containers/fedora_clang.dockerfile.j2     |   8 +
>  .../containers/fedora_coverity.dockerfile.j2  |  10 +
>  .../containers/opensuse.dockerfile.j2         |  10 +
>  .../containers/redhat_family.dockerfile.j2    |   5 +
>  .../templates/containers/rhel.dockerfile.j2   |  16 +
>  .../templates/containers/rhel7.dockerfile.j2  |  15 +
>  .../templates/containers/rhel8.dockerfile.j2  |  15 +
>  .../templates/containers/rhel9.dockerfile.j2  |  19 +
>  .../templates/containers/rpm.dockerfile.j2    |   3 +
>  .../templates/containers/ubuntu.dockerfile.j2 |   3 +
>  .../containers/ubuntu20.04.dockerfile.j2      |  12 +
>  .../containers/ubuntu22.04.dockerfile.j2      |   3 +
>  .../containers/ubuntu_cross.dockerfile.j2     |  11 +
>  .../containers/ubuntu_sve.dockerfile.j2       |  12 +
>  36 files changed, 2214 insertions(+)
>  create mode 100644 containers/Makefile
>  create mode 100644 containers/README.md
>  create mode 100644 containers/container_builder.dockerfile
>  create mode 100644 containers/template_engine/inventory.yaml
>  create mode 100644 containers/template_engine/inventory_schema.json
>  create mode 100755 containers/template_engine/make_dockerfile.py
>  create mode 100644 containers/template_engine/poetry.lock
>  create mode 100644 containers/template_engine/pyproject.toml
>  create mode 100644 containers/template_engine/templates/containers.makefile.j2
>  create mode 100644 containers/template_engine/templates/containers/alpine.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/arch.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/base.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/centos8.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/centos9.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/debian.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/debian10.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/debian11.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/debian11_arm_ipsec.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/debian_buster.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/fedora.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/fedora_clang.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/fedora_coverity.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/opensuse.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/redhat_family.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/rhel.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/rhel7.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/rhel8.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/rhel9.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/rpm.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/ubuntu.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2
>  create mode 100644 containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2
  

These issues are fixed in v8.

On Mon, Jul 17, 2023 at 3:07 PM Aaron Conole <aconole@redhat.com> wrote:

> Adam Hassick <ahassick@iol.unh.edu> writes:
>
> > This patch series contains a new version of the DPDK CI containers. The
> old
> > version was tied very tightly to the Community Lab Infrastructure, so it
> was
> > not suitable for general use. This version is designed to make adding new
> > OSes or OS versions as easy as possible. The minimum functionality can
> easily
> > be built on any system that can compile DPDK. It includes support for
> > building containers for other non-native architectures (ex: arm
> containers
> > on x86) and for baking ABI references into the images.
> >
> > The inventory file as added in this patch series defines what the
> community lab
> > currently supports.
> >
> > If you want to build these yourself, don't try to do parallel Makefile
> builds
> > on non-server hardware. Libabigail is built into the containers, and
> since it
> > is not avaliable in all distros it is compiled from source for many
> targets.
> > If embedding the abi is enabled (DPDK_CI_CONTAINERS_BUILD_ABI=Y), with
> the
> > current settings, DPDK will be compiled twice for every target in the
> > inventory file. This will become three times when DPDK main has ABI
> stability
> > again. Due to how the containers are constructed, building on non-native
> > architectures is especially expensive because DPDK is compiled with an
> emulated
> > compiler in a qemu vm that only has one thread.
> >
> > ---
> >
> > v2:
> > * Added poetry.lock
> > * Remove duplicates in inventory file
> > * Fix typo in base container template
> >
> > v3:
> > * Libabigail is now not built by default unless
> DPDK_CI_CONTAINERS_BUILD_ABI=y
> > * Made minimum podman version 4.0.0 due to dependencies on cache mounts
> >
> > v4:
> > * Fix whitespace errors
> >
> > v5:
> > * Updated container templates and inventory
> > * Add support for OCI manifests
> > * Update README to reflect changes
> >
> > v6:
> > * Fix spelling errors and whitespace errors
> >
> > v7:
> > * Add workaround to Arch template to install Python packages as root
> > * Compare against v4 and polish the patchset
>
> Please can you repost with the spelling fixes, and also fixing Owen's
> email address? (it is currently listed like a mailing list archive).
>
> > Owen Hilyard (6):
> >   * Add README file for containers * Add pyproject file with required
> >     dependencies for building containers
> >   Adds a yaml file used to define what containers should be built, what
> >     packages the containers should have, what platforms to build for and
> >     how they should be tagged.
> >   This script will template out all of the Dockerfiles based on the
> >     definitions provided in the inventory using the jinja2 templating
> >     library.
> >   Adds a variety of extensible templates used to create the Dockerfiles
> >     for each target. All templates inherit from base, and most distros
> >     and distro families have their own templates that are used to define
> >     common functionality. Multiple versions of a distro may use a single
> >     template if they are similar enough (ex: fedora).
> >   Adds a container that can be used to run the python scripts to create
> >     the Dockerfiles for the CI containers. This removes a large number
> >     of python environment dependencies from the host requirements.
> >   The Makefile that can be used to build all of the container images
> >     using "make build", and can also be used to push them to a remote
> >     repository (for use in CI).
> >
> >  containers/Makefile                           | 252 ++++++++++
> >  containers/README.md                          | 178 +++++++
> >  containers/container_builder.dockerfile       |  26 +
> >  containers/template_engine/inventory.yaml     | 470 ++++++++++++++++++
> >  .../template_engine/inventory_schema.json     | 221 ++++++++
> >  containers/template_engine/make_dockerfile.py | 358 +++++++++++++
> >  containers/template_engine/poetry.lock        | 220 ++++++++
> >  containers/template_engine/pyproject.toml     |  21 +
> >  .../templates/containers.makefile.j2          |  73 +++
> >  .../templates/containers/alpine.dockerfile.j2 |   3 +
> >  .../templates/containers/arch.dockerfile.j2   |  39 ++
> >  .../templates/containers/base.dockerfile.j2   | 120 +++++
> >  .../containers/centos8.dockerfile.j2          |  21 +
> >  .../containers/centos9.dockerfile.j2          |  17 +
> >  .../templates/containers/debian.dockerfile.j2 |   7 +
> >  .../containers/debian10.dockerfile.j2         |   3 +
> >  .../containers/debian11.dockerfile.j2         |   3 +
> >  .../debian11_arm_ipsec.dockerfile.j2          |  16 +
> >  .../containers/debian_bullseye.dockerfile.j2  |   3 +
> >  .../containers/debian_buster.dockerfile.j2    |   3 +
> >  .../templates/containers/fedora.dockerfile.j2 |  11 +
> >  .../containers/fedora36_clang.dockerfile.j2   |   7 +
> >  .../containers/fedora_clang.dockerfile.j2     |   8 +
> >  .../containers/fedora_coverity.dockerfile.j2  |  10 +
> >  .../containers/opensuse.dockerfile.j2         |  10 +
> >  .../containers/redhat_family.dockerfile.j2    |   5 +
> >  .../templates/containers/rhel.dockerfile.j2   |  16 +
> >  .../templates/containers/rhel7.dockerfile.j2  |  15 +
> >  .../templates/containers/rhel8.dockerfile.j2  |  15 +
> >  .../templates/containers/rhel9.dockerfile.j2  |  19 +
> >  .../templates/containers/rpm.dockerfile.j2    |   3 +
> >  .../templates/containers/ubuntu.dockerfile.j2 |   3 +
> >  .../containers/ubuntu20.04.dockerfile.j2      |  12 +
> >  .../containers/ubuntu22.04.dockerfile.j2      |   3 +
> >  .../containers/ubuntu_cross.dockerfile.j2     |  11 +
> >  .../containers/ubuntu_sve.dockerfile.j2       |  12 +
> >  36 files changed, 2214 insertions(+)
> >  create mode 100644 containers/Makefile
> >  create mode 100644 containers/README.md
> >  create mode 100644 containers/container_builder.dockerfile
> >  create mode 100644 containers/template_engine/inventory.yaml
> >  create mode 100644 containers/template_engine/inventory_schema.json
> >  create mode 100755 containers/template_engine/make_dockerfile.py
> >  create mode 100644 containers/template_engine/poetry.lock
> >  create mode 100644 containers/template_engine/pyproject.toml
> >  create mode 100644
> containers/template_engine/templates/containers.makefile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/alpine.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/arch.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/base.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/centos8.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/centos9.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/debian.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/debian10.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/debian11.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/debian11_arm_ipsec.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/debian_buster.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/fedora.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/fedora_clang.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/fedora_coverity.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/opensuse.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/redhat_family.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/rhel.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/rhel7.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/rhel8.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/rhel9.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/rpm.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/ubuntu.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2
> >  create mode 100644
> containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2
>
>
  

Adam Hassick <ahassick@iol.unh.edu> writes:

> These issues are fixed in v8.

Thanks

> On Mon, Jul 17, 2023 at 3:07 PM Aaron Conole <aconole@redhat.com> wrote:
>
>  Adam Hassick <ahassick@iol.unh.edu> writes:
>
>  > This patch series contains a new version of the DPDK CI containers. The old
>  > version was tied very tightly to the Community Lab Infrastructure, so it was
>  > not suitable for general use. This version is designed to make adding new
>  > OSes or OS versions as easy as possible. The minimum functionality can easily
>  > be built on any system that can compile DPDK. It includes support for
>  > building containers for other non-native architectures (ex: arm containers
>  > on x86) and for baking ABI references into the images.
>  >
>  > The inventory file as added in this patch series defines what the community lab
>  > currently supports.
>  >
>  > If you want to build these yourself, don't try to do parallel Makefile builds
>  > on non-server hardware. Libabigail is built into the containers, and since it
>  > is not avaliable in all distros it is compiled from source for many targets.
>  > If embedding the abi is enabled (DPDK_CI_CONTAINERS_BUILD_ABI=Y), with the
>  > current settings, DPDK will be compiled twice for every target in the
>  > inventory file. This will become three times when DPDK main has ABI stability
>  > again. Due to how the containers are constructed, building on non-native
>  > architectures is especially expensive because DPDK is compiled with an emulated
>  > compiler in a qemu vm that only has one thread.
>  >
>  > ---
>  >
>  > v2:
>  > * Added poetry.lock
>  > * Remove duplicates in inventory file
>  > * Fix typo in base container template
>  >
>  > v3:
>  > * Libabigail is now not built by default unless DPDK_CI_CONTAINERS_BUILD_ABI=y
>  > * Made minimum podman version 4.0.0 due to dependencies on cache mounts
>  >
>  > v4:
>  > * Fix whitespace errors
>  >
>  > v5:
>  > * Updated container templates and inventory
>  > * Add support for OCI manifests
>  > * Update README to reflect changes
>  >
>  > v6:
>  > * Fix spelling errors and whitespace errors
>  >
>  > v7:
>  > * Add workaround to Arch template to install Python packages as root
>  > * Compare against v4 and polish the patchset
>
>  Please can you repost with the spelling fixes, and also fixing Owen's
>  email address? (it is currently listed like a mailing list archive).
>
>  > Owen Hilyard (6):
>  >   * Add README file for containers * Add pyproject file with required
>  >     dependencies for building containers
>  >   Adds a yaml file used to define what containers should be built, what
>  >     packages the containers should have, what platforms to build for and
>  >     how they should be tagged.
>  >   This script will template out all of the Dockerfiles based on the
>  >     definitions provided in the inventory using the jinja2 templating
>  >     library.
>  >   Adds a variety of extensible templates used to create the Dockerfiles
>  >     for each target. All templates inherit from base, and most distros
>  >     and distro families have their own templates that are used to define
>  >     common functionality. Multiple versions of a distro may use a single
>  >     template if they are similar enough (ex: fedora).
>  >   Adds a container that can be used to run the python scripts to create
>  >     the Dockerfiles for the CI containers. This removes a large number
>  >     of python environment dependencies from the host requirements.
>  >   The Makefile that can be used to build all of the container images
>  >     using "make build", and can also be used to push them to a remote
>  >     repository (for use in CI).
>  >
>  >  containers/Makefile                           | 252 ++++++++++
>  >  containers/README.md                          | 178 +++++++
>  >  containers/container_builder.dockerfile       |  26 +
>  >  containers/template_engine/inventory.yaml     | 470 ++++++++++++++++++
>  >  .../template_engine/inventory_schema.json     | 221 ++++++++
>  >  containers/template_engine/make_dockerfile.py | 358 +++++++++++++
>  >  containers/template_engine/poetry.lock        | 220 ++++++++
>  >  containers/template_engine/pyproject.toml     |  21 +
>  >  .../templates/containers.makefile.j2          |  73 +++
>  >  .../templates/containers/alpine.dockerfile.j2 |   3 +
>  >  .../templates/containers/arch.dockerfile.j2   |  39 ++
>  >  .../templates/containers/base.dockerfile.j2   | 120 +++++
>  >  .../containers/centos8.dockerfile.j2          |  21 +
>  >  .../containers/centos9.dockerfile.j2          |  17 +
>  >  .../templates/containers/debian.dockerfile.j2 |   7 +
>  >  .../containers/debian10.dockerfile.j2         |   3 +
>  >  .../containers/debian11.dockerfile.j2         |   3 +
>  >  .../debian11_arm_ipsec.dockerfile.j2          |  16 +
>  >  .../containers/debian_bullseye.dockerfile.j2  |   3 +
>  >  .../containers/debian_buster.dockerfile.j2    |   3 +
>  >  .../templates/containers/fedora.dockerfile.j2 |  11 +
>  >  .../containers/fedora36_clang.dockerfile.j2   |   7 +
>  >  .../containers/fedora_clang.dockerfile.j2     |   8 +
>  >  .../containers/fedora_coverity.dockerfile.j2  |  10 +
>  >  .../containers/opensuse.dockerfile.j2         |  10 +
>  >  .../containers/redhat_family.dockerfile.j2    |   5 +
>  >  .../templates/containers/rhel.dockerfile.j2   |  16 +
>  >  .../templates/containers/rhel7.dockerfile.j2  |  15 +
>  >  .../templates/containers/rhel8.dockerfile.j2  |  15 +
>  >  .../templates/containers/rhel9.dockerfile.j2  |  19 +
>  >  .../templates/containers/rpm.dockerfile.j2    |   3 +
>  >  .../templates/containers/ubuntu.dockerfile.j2 |   3 +
>  >  .../containers/ubuntu20.04.dockerfile.j2      |  12 +
>  >  .../containers/ubuntu22.04.dockerfile.j2      |   3 +
>  >  .../containers/ubuntu_cross.dockerfile.j2     |  11 +
>  >  .../containers/ubuntu_sve.dockerfile.j2       |  12 +
>  >  36 files changed, 2214 insertions(+)
>  >  create mode 100644 containers/Makefile
>  >  create mode 100644 containers/README.md
>  >  create mode 100644 containers/container_builder.dockerfile
>  >  create mode 100644 containers/template_engine/inventory.yaml
>  >  create mode 100644 containers/template_engine/inventory_schema.json
>  >  create mode 100755 containers/template_engine/make_dockerfile.py
>  >  create mode 100644 containers/template_engine/poetry.lock
>  >  create mode 100644 containers/template_engine/pyproject.toml
>  >  create mode 100644 containers/template_engine/templates/containers.makefile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/alpine.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/arch.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/base.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/centos8.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/centos9.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/debian.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/debian10.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/debian11.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/debian11_arm_ipsec.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/debian_bullseye.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/debian_buster.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/fedora.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/fedora36_clang.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/fedora_clang.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/fedora_coverity.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/opensuse.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/redhat_family.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/rhel.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/rhel7.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/rhel8.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/rhel9.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/rpm.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/ubuntu.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/ubuntu20.04.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/ubuntu22.04.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/ubuntu_cross.dockerfile.j2
>  >  create mode 100644 containers/template_engine/templates/containers/ubuntu_sve.dockerfile.j2