[v6,1/5] ci: replace IPsec-mb package install

  From: Ciara Power <ciara.power@intel.com>

The IPsec-mb version that is available through current package
managers is 1.2.
This release moves the minimum required IPsec-mb version for IPsec-mb
based SW PMDs to 1.4.
To compile these PMDs, a manual step is added to install IPsec-mb v1.4
using dpkg.

Signed-off-by: Ciara Power <ciara.power@intel.com>
---
 .github/workflows/build.yml | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)
  

Hello,

On Tue, Mar 12, 2024 at 2:50 PM Brian Dooley <brian.dooley@intel.com> wrote:
>
> From: Ciara Power <ciara.power@intel.com>
>
> The IPsec-mb version that is available through current package
> managers is 1.2.
> This release moves the minimum required IPsec-mb version for IPsec-mb
> based SW PMDs to 1.4.
> To compile these PMDs, a manual step is added to install IPsec-mb v1.4
> using dpkg.
>
> Signed-off-by: Ciara Power <ciara.power@intel.com>
> ---
>  .github/workflows/build.yml | 25 ++++++++++++++++++++++---
>  1 file changed, 22 insertions(+), 3 deletions(-)
>
> diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
> index 776fbf6f30..ed44b1f730 100644
> --- a/.github/workflows/build.yml
> +++ b/.github/workflows/build.yml
> @@ -106,9 +106,15 @@ jobs:
>        run: sudo apt update || true
>      - name: Install packages
>        run: sudo apt install -y ccache libarchive-dev libbsd-dev libbpf-dev
> -        libfdt-dev libibverbs-dev libipsec-mb-dev libisal-dev libjansson-dev
> +        libfdt-dev libibverbs-dev libisal-dev libjansson-dev
>          libnuma-dev libpcap-dev libssl-dev ninja-build pkg-config python3-pip
>          python3-pyelftools python3-setuptools python3-wheel zlib1g-dev
> +    - name: Install ipsec-mb library
> +      run: |
> +        wget "https://launchpad.net/ubuntu/+archive/primary/+files/libipsec-mb-dev_1.4-3_amd64.deb"
> +        wget "https://launchpad.net/ubuntu/+archive/primary/+files/libipsec-mb1_1.4-3_amd64.deb"
> +        sudo dpkg -i libipsec-mb1_1.4-3_amd64.deb
> +        sudo dpkg -i libipsec-mb-dev_1.4-3_amd64.deb

I am not enthousiastic at advertising a kind of out of tree approach.
That's a bit like if NVIDIA asked us to stop testing distribution
rdma-core packages and instead rely on MOFED.

Why are we removing support for versions that are packaged by the main
distributions?
  

Hi David,

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, March 12, 2024 1:54 PM
> To: Dooley, Brian <brian.dooley@intel.com>
> Cc: Aaron Conole <aconole@redhat.com>; Michael Santana
> <maicolgabriel@hotmail.com>; dev@dpdk.org; gakhil@marvell.com; De Lara
> Guarch, Pablo <pablo.de.lara.guarch@intel.com>; probb@iol.unh.edu;
> wathsala.vithanage@arm.com; Power, Ciara <ciara.power@intel.com>
> Subject: Re: [PATCH v6 1/5] ci: replace IPsec-mb package install
> 
> Hello,
> 
> On Tue, Mar 12, 2024 at 2:50 PM Brian Dooley <brian.dooley@intel.com>
> wrote:
> >
> > From: Ciara Power <ciara.power@intel.com>
> >
> > The IPsec-mb version that is available through current package
> > managers is 1.2.
> > This release moves the minimum required IPsec-mb version for IPsec-mb
> > based SW PMDs to 1.4.
> > To compile these PMDs, a manual step is added to install IPsec-mb v1.4
> > using dpkg.
> >
> > Signed-off-by: Ciara Power <ciara.power@intel.com>
> > ---
> >  .github/workflows/build.yml | 25 ++++++++++++++++++++++---
> >  1 file changed, 22 insertions(+), 3 deletions(-)
> >
> > diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
> > index 776fbf6f30..ed44b1f730 100644
> > --- a/.github/workflows/build.yml
> > +++ b/.github/workflows/build.yml
> > @@ -106,9 +106,15 @@ jobs:
> >        run: sudo apt update || true
> >      - name: Install packages
> >        run: sudo apt install -y ccache libarchive-dev libbsd-dev libbpf-dev
> > -        libfdt-dev libibverbs-dev libipsec-mb-dev libisal-dev libjansson-dev
> > +        libfdt-dev libibverbs-dev libisal-dev libjansson-dev
> >          libnuma-dev libpcap-dev libssl-dev ninja-build pkg-config python3-pip
> >          python3-pyelftools python3-setuptools python3-wheel
> > zlib1g-dev
> > +    - name: Install ipsec-mb library
> > +      run: |
> > +        wget "https://launchpad.net/ubuntu/+archive/primary/+files/libipsec-
> mb-dev_1.4-3_amd64.deb"
> > +        wget "https://launchpad.net/ubuntu/+archive/primary/+files/libipsec-
> mb1_1.4-3_amd64.deb"
> > +        sudo dpkg -i libipsec-mb1_1.4-3_amd64.deb
> > +        sudo dpkg -i libipsec-mb-dev_1.4-3_amd64.deb
> 
> I am not enthousiastic at advertising a kind of out of tree approach.
> That's a bit like if NVIDIA asked us to stop testing distribution rdma-core
> packages and instead rely on MOFED.
> 
> Why are we removing support for versions that are packaged by the main
> distributions?

With Ubuntu 22.04, ipsec-mb v1.2 is the version available through the package manager.
We were aiming to make v1.4 the minimum version for ipsec-mb PMDs from this release onwards,
removing the many ifdef codepaths in the PMDs for older versions. (patch included in this patchset)

Some of the other CI environments were updated to install v1.4 already to support this change,
but we found the github CI robot was limited for ipsec-mb versions when using the package manager.
It had some failures comparing ABI with v1.2 installed (SW PMDs compiled in reference build, but not compiled after patch).

To support the new minimum SW PMD ipsec-mb version for this CI, we thought installing v1.4 like this would suffice.

Thanks,
Ciara
  

On Tue, Mar 12, 2024 at 2:50 PM Brian Dooley <brian.dooley@intel.com> wrote:
> @@ -187,11 +193,18 @@ jobs:
>        run: docker exec -i dpdk dnf update -y
>      - name: Install packages
>        if: steps.image_cache.outputs.cache-hit != 'true'
> -      run: docker exec -i dpdk dnf install -y ccache intel-ipsec-mb-devel
> +      run: docker exec -i dpdk dnf install -y ccache

Removing ipsec-mb means we lose build coverage for those crypto
drivers on Fedora.


>          isa-l-devel jansson-devel libarchive-devel libatomic libbsd-devel
>          libbpf-devel libfdt-devel libpcap-devel libxdp-devel ninja-build
>          numactl-devel openssl-devel python3-pip python3-pyelftools
>          python3-setuptools python3-wheel rdma-core-devel zlib-devel
> +    - name: Install ipsec-mb library
> +      if: steps.image_cache.outputs.cache-hit != 'true'
> +      run: |
> +        wget "https://launchpad.net/ubuntu/+archive/primary/+files/libipsec-mb-dev_1.4-3_amd64.deb"
> +        wget "https://launchpad.net/ubuntu/+archive/primary/+files/libipsec-mb1_1.4-3_amd64.deb"
> +        sudo dpkg -i libipsec-mb1_1.4-3_amd64.deb
> +        sudo dpkg -i libipsec-mb-dev_1.4-3_amd64.deb

And this hunk is useless.
This installs the deb in the Ubuntu "host", not in the Fedora container.
  

On Tue, Mar 12, 2024 at 4:26 PM Power, Ciara <ciara.power@intel.com> wrote:
> > From: David Marchand <david.marchand@redhat.com>
> > On Tue, Mar 12, 2024 at 2:50 PM Brian Dooley <brian.dooley@intel.com>
> > wrote:
> > >
> > > From: Ciara Power <ciara.power@intel.com>
> > >
> > > The IPsec-mb version that is available through current package
> > > managers is 1.2.
> > > This release moves the minimum required IPsec-mb version for IPsec-mb
> > > based SW PMDs to 1.4.
> > > To compile these PMDs, a manual step is added to install IPsec-mb v1.4
> > > using dpkg.
> > >
> > > Signed-off-by: Ciara Power <ciara.power@intel.com>
> > > ---
> > >  .github/workflows/build.yml | 25 ++++++++++++++++++++++---
> > >  1 file changed, 22 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
> > > index 776fbf6f30..ed44b1f730 100644
> > > --- a/.github/workflows/build.yml
> > > +++ b/.github/workflows/build.yml
> > > @@ -106,9 +106,15 @@ jobs:
> > >        run: sudo apt update || true
> > >      - name: Install packages
> > >        run: sudo apt install -y ccache libarchive-dev libbsd-dev libbpf-dev
> > > -        libfdt-dev libibverbs-dev libipsec-mb-dev libisal-dev libjansson-dev
> > > +        libfdt-dev libibverbs-dev libisal-dev libjansson-dev
> > >          libnuma-dev libpcap-dev libssl-dev ninja-build pkg-config python3-pip
> > >          python3-pyelftools python3-setuptools python3-wheel
> > > zlib1g-dev
> > > +    - name: Install ipsec-mb library
> > > +      run: |
> > > +        wget "https://launchpad.net/ubuntu/+archive/primary/+files/libipsec-
> > mb-dev_1.4-3_amd64.deb"
> > > +        wget "https://launchpad.net/ubuntu/+archive/primary/+files/libipsec-
> > mb1_1.4-3_amd64.deb"
> > > +        sudo dpkg -i libipsec-mb1_1.4-3_amd64.deb
> > > +        sudo dpkg -i libipsec-mb-dev_1.4-3_amd64.deb
> >
> > I am not enthousiastic at advertising a kind of out of tree approach.
> > That's a bit like if NVIDIA asked us to stop testing distribution rdma-core
> > packages and instead rely on MOFED.
> >
> > Why are we removing support for versions that are packaged by the main
> > distributions?
>
> With Ubuntu 22.04, ipsec-mb v1.2 is the version available through the package manager.
> We were aiming to make v1.4 the minimum version for ipsec-mb PMDs from this release onwards,
> removing the many ifdef codepaths in the PMDs for older versions. (patch included in this patchset)
>
> Some of the other CI environments were updated to install v1.4 already to support this change,
> but we found the github CI robot was limited for ipsec-mb versions when using the package manager.
> It had some failures comparing ABI with v1.2 installed (SW PMDs compiled in reference build, but not compiled after patch).

Such a change means that users of the Ubuntu/Fedora dpdk package lose
access to those drivers hypothetically.
"Hypothetically", because in reality, Ubuntu and others distributions
won't update to non LTS versions.

On the other hand, if a user was building DPDK (and not the one
provided by the distribution), now the user has to stop using the
ipsec mb provided by the distribution: building/packaging/maintaining
the ipsec mb library is now forced on the user plate.

I am unclear if this qualifies as a ABI breakage, but I am not
confortable with this change.
  

Hi,

On 08/03/2024 16:05, Power, Ciara wrote:
> I think, for basic compile to work for both repos, any functions in the
> ipsec_mb.h file need to be defined in a C file to avoid issues.
> But, I understand only SNOW3G + ZUC are the focus for the Arm ipsec-mb
> repo, so a HMAC related function like the one throwing the error isn't
> used.
>
> Perhaps having empty stubs for all other functions such as the
> imb_hmac_ipad_opad would be sufficient to allow compiling, because PMD
> expects it to exist somewhere in the library.

We've stubbed the method hmac_ipad_opad in the Arm ipsec-mb repo. This 
is now tagged as SECLIB-IPSEC-2024.03.12. See here: 
https://gitlab.arm.com/arm-reference-solutions/ipsec-mb/-/tree/SECLIB-IPSEC-2024.03.12

Thanks,
Jack
  

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, March 12, 2024 4:14 PM
> To: Power, Ciara <ciara.power@intel.com>
> Cc: Dooley, Brian <brian.dooley@intel.com>; Aaron Conole
> <aconole@redhat.com>; Michael Santana <maicolgabriel@hotmail.com>;
> dev@dpdk.org; gakhil@marvell.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; probb@iol.unh.edu;
> wathsala.vithanage@arm.com; Thomas Monjalon <thomas@monjalon.net>;
> Richardson, Bruce <bruce.richardson@intel.com>
> Subject: Re: [PATCH v6 1/5] ci: replace IPsec-mb package install
> 
> On Tue, Mar 12, 2024 at 4:26 PM Power, Ciara <ciara.power@intel.com> wrote:
> > > From: David Marchand <david.marchand@redhat.com> On Tue, Mar 12,
> > > 2024 at 2:50 PM Brian Dooley <brian.dooley@intel.com>
> > > wrote:
> > > >
> > > > From: Ciara Power <ciara.power@intel.com>
> > > >
> > > > The IPsec-mb version that is available through current package
> > > > managers is 1.2.
> > > > This release moves the minimum required IPsec-mb version for
> > > > IPsec-mb based SW PMDs to 1.4.
> > > > To compile these PMDs, a manual step is added to install IPsec-mb
> > > > v1.4 using dpkg.
> > > >
> > > > Signed-off-by: Ciara Power <ciara.power@intel.com>
> > > > ---
> > > >  .github/workflows/build.yml | 25 ++++++++++++++++++++++---
> > > >  1 file changed, 22 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/.github/workflows/build.yml
> > > > b/.github/workflows/build.yml index 776fbf6f30..ed44b1f730 100644
> > > > --- a/.github/workflows/build.yml
> > > > +++ b/.github/workflows/build.yml
> > > > @@ -106,9 +106,15 @@ jobs:
> > > >        run: sudo apt update || true
> > > >      - name: Install packages
> > > >        run: sudo apt install -y ccache libarchive-dev libbsd-dev libbpf-dev
> > > > -        libfdt-dev libibverbs-dev libipsec-mb-dev libisal-dev libjansson-dev
> > > > +        libfdt-dev libibverbs-dev libisal-dev libjansson-dev
> > > >          libnuma-dev libpcap-dev libssl-dev ninja-build pkg-config python3-
> pip
> > > >          python3-pyelftools python3-setuptools python3-wheel
> > > > zlib1g-dev
> > > > +    - name: Install ipsec-mb library
> > > > +      run: |
> > > > +        wget
> > > > + "https://launchpad.net/ubuntu/+archive/primary/+files/libipsec-
> > > mb-dev_1.4-3_amd64.deb"
> > > > +        wget
> > > > + "https://launchpad.net/ubuntu/+archive/primary/+files/libipsec-
> > > mb1_1.4-3_amd64.deb"
> > > > +        sudo dpkg -i libipsec-mb1_1.4-3_amd64.deb
> > > > +        sudo dpkg -i libipsec-mb-dev_1.4-3_amd64.deb
> > >
> > > I am not enthousiastic at advertising a kind of out of tree approach.
> > > That's a bit like if NVIDIA asked us to stop testing distribution
> > > rdma-core packages and instead rely on MOFED.
> > >
> > > Why are we removing support for versions that are packaged by the
> > > main distributions?
> >
> > With Ubuntu 22.04, ipsec-mb v1.2 is the version available through the
> package manager.
> > We were aiming to make v1.4 the minimum version for ipsec-mb PMDs from
> > this release onwards, removing the many ifdef codepaths in the PMDs
> > for older versions. (patch included in this patchset)
> >
> > Some of the other CI environments were updated to install v1.4 already
> > to support this change, but we found the github CI robot was limited for ipsec-
> mb versions when using the package manager.
> > It had some failures comparing ABI with v1.2 installed (SW PMDs compiled in
> reference build, but not compiled after patch).
> 
> Such a change means that users of the Ubuntu/Fedora dpdk package lose access
> to those drivers hypothetically.
> "Hypothetically", because in reality, Ubuntu and others distributions won't
> update to non LTS versions.
> 
> On the other hand, if a user was building DPDK (and not the one provided by
> the distribution), now the user has to stop using the ipsec mb provided by the
> distribution: building/packaging/maintaining the ipsec mb library is now forced
> on the user plate.
> 
> I am unclear if this qualifies as a ABI breakage, but I am not confortable with this
> change.

Hi David,

Ah, okay - thanks for the explanation.
Those are points I had missed, but it makes sense.

We will drop the version bump to v1.4 for this release, and revisit in a later release when suitable.

Thanks,
Ciara
  

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Tuesday, March 12, 2024 4:05 PM
> To: Dooley, Brian <brian.dooley@intel.com>
> Cc: Aaron Conole <aconole@redhat.com>; Michael Santana
> <maicolgabriel@hotmail.com>; dev@dpdk.org; gakhil@marvell.com; De Lara
> Guarch, Pablo <pablo.de.lara.guarch@intel.com>; probb@iol.unh.edu;
> wathsala.vithanage@arm.com; Power, Ciara <ciara.power@intel.com>
> Subject: Re: [PATCH v6 1/5] ci: replace IPsec-mb package install
> 
> On Tue, Mar 12, 2024 at 2:50 PM Brian Dooley <brian.dooley@intel.com>
> wrote:
> > @@ -187,11 +193,18 @@ jobs:
> >        run: docker exec -i dpdk dnf update -y
> >      - name: Install packages
> >        if: steps.image_cache.outputs.cache-hit != 'true'
> > -      run: docker exec -i dpdk dnf install -y ccache intel-ipsec-mb-devel
> > +      run: docker exec -i dpdk dnf install -y ccache
> 
> Removing ipsec-mb means we lose build coverage for those crypto drivers on
> Fedora.
> 
> 
> >          isa-l-devel jansson-devel libarchive-devel libatomic libbsd-devel
> >          libbpf-devel libfdt-devel libpcap-devel libxdp-devel ninja-build
> >          numactl-devel openssl-devel python3-pip python3-pyelftools
> >          python3-setuptools python3-wheel rdma-core-devel zlib-devel
> > +    - name: Install ipsec-mb library
> > +      if: steps.image_cache.outputs.cache-hit != 'true'
> > +      run: |
> > +        wget "https://launchpad.net/ubuntu/+archive/primary/+files/libipsec-
> mb-dev_1.4-3_amd64.deb"
> > +        wget "https://launchpad.net/ubuntu/+archive/primary/+files/libipsec-
> mb1_1.4-3_amd64.deb"
> > +        sudo dpkg -i libipsec-mb1_1.4-3_amd64.deb
> > +        sudo dpkg -i libipsec-mb-dev_1.4-3_amd64.deb
> 
> And this hunk is useless.
> This installs the deb in the Ubuntu "host", not in the Fedora container.

Thanks, yes, good catch.

This patch isn't needed anymore as per the discussion in the other patch, no version bump for 24.03.

Thanks,
Ciara
  

> -----Original Message-----
> From: Dooley, Brian <brian.dooley@intel.com>
> Sent: Tuesday, March 12, 2024 1:50 PM
> To: Aaron Conole <aconole@redhat.com>; Michael Santana
> <maicolgabriel@hotmail.com>
> Cc: dev@dpdk.org; gakhil@marvell.com; De Lara Guarch, Pablo
> <pablo.de.lara.guarch@intel.com>; probb@iol.unh.edu;
> wathsala.vithanage@arm.com; Power, Ciara <ciara.power@intel.com>
> Subject: [PATCH v6 1/5] ci: replace IPsec-mb package install
> 
> From: Ciara Power <ciara.power@intel.com>
> 
> The IPsec-mb version that is available through current package managers is 1.2.
> This release moves the minimum required IPsec-mb version for IPsec-mb based
> SW PMDs to 1.4.
> To compile these PMDs, a manual step is added to install IPsec-mb v1.4 using
> dpkg.
> 
> Signed-off-by: Ciara Power <ciara.power@intel.com>
> ---

Hi folks,

Due to the time of release, and churn of this patchset - we will hold off the changes until a later release.

This includes nearly all of the patches in this set (unifying SW PMDs, HMAC_ipad_opad API, version bump)
The doc update patch is still applicable and a small change so we can send that along for 24.03 rc3/rc4.

Thanks,
Ciara
  

Recheck-request: iol-unit-arm64-testing

Even though ipsec update is postponed to a later release, I'm putting
in rechecks for all series that have fails for the arm crypto tests
now that we are building from SECLIB-IPSEC-2024.03.12.