Message ID | 1604649795-27476-1-git-send-email-juraj.linkes@pantheon.tech (mailing list archive) |
---|---|
Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id D024DA0524; Fri, 6 Nov 2020 09:03:23 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 383BC2BEA; Fri, 6 Nov 2020 09:03:22 +0100 (CET) Received: from lb.pantheon.sk (lb.pantheon.sk [46.229.239.20]) by dpdk.org (Postfix) with ESMTP id 195062BE2 for <dev@dpdk.org>; Fri, 6 Nov 2020 09:03:21 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by lb.pantheon.sk (Postfix) with ESMTP id 8E143B6B66; Fri, 6 Nov 2020 09:03:17 +0100 (CET) X-Virus-Scanned: amavisd-new at siecit.sk Received: from lb.pantheon.sk ([127.0.0.1]) by localhost (lb.pantheon.sk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yMtZq1t90PVY; Fri, 6 Nov 2020 09:03:16 +0100 (CET) Received: from service-node1.lab.pantheon.local (unknown [46.229.239.141]) by lb.pantheon.sk (Postfix) with ESMTP id 79A7AB6B61; Fri, 6 Nov 2020 09:03:15 +0100 (CET) From: =?utf-8?q?Juraj_Linke=C5=A1?= <juraj.linkes@pantheon.tech> To: bruce.richardson@intel.com, Ruifeng.Wang@arm.com, Honnappa.Nagarahalli@arm.com, Phil.Yang@arm.com, vcchunga@amazon.com, Dharmik.Thakkar@arm.com, jerinjacobk@gmail.com, hemant.agrawal@nxp.com, ajit.khaparde@broadcom.com, ferruh.yigit@intel.com, aconole@redhat.com Cc: dev@dpdk.org, =?utf-8?q?Juraj_Linke=C5=A1?= <juraj.linkes@pantheon.tech> Date: Fri, 6 Nov 2020 09:03:01 +0100 Message-Id: <1604649795-27476-1-git-send-email-juraj.linkes@pantheon.tech> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1604584762-25321-1-git-send-email-juraj.linkes@pantheon.tech> References: <1604584762-25321-1-git-send-email-juraj.linkes@pantheon.tech> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v8 00/14] Arm build options rework X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org Sender: "dev" <dev-bounces@dpdk.org> |
Series |
Arm build options rework
|
|
Message
Juraj Linkeš
Nov. 6, 2020, 8:03 a.m. UTC
The current way of specifying Arm configuration options is insufficient since we can't identify the SoC we're building for from the MIDR information. For example, we can't distinguish between N1SDP, Graviton2 or Ampere Altra. Add a way to specify the cpu count and numa node count for cross builds. We also want to be able to disable which drivers (and possibly libraries) are built without user input. This is useful when building: 1. on an SoC that is slow and we want to build only what is necessary without the user having to check which libraries they have installed 2. a cross build on a fast aarch64 machine but with target SoC which differs in capabilities or libraries. This is achieved by specifying the drivers in cross files. Among libraries, only libnuma can be now disabled. Also add an optional way to discover cpu count a numa node count. Fix -Dmax_lcores and -Dmax_numa_nodes for arm builds. The current implementation adds/supports the following: * x86 -> aarch64 cross build with added config options/disabled drivers/libs * aarch64 -> aarch64 builds for a different SoCs must be done using cross-files * max numa nodes and max lcore may be specified on the command line to overwrite the values for any (native or cross) build v2: Major rework of the whole series. v3: Added numa and core count defaults for x86 default build. Removed numa and core count defaults. Now requiring defaults to be specified in a cross file or on the cmdline. Added FreeBDS support for numa count discovery. v4: Make automatic numa and cpu counts discovery optional. v5: Split the refactor patch into smaller patches. Simplify buildtools/get_numa_count.py. Add more explanation to cover letter. v6: Apply cross file options arch agnostically, not just in Arm cross builds. Streamline Arm build setup and machine args: always use native args in native builds, require implementer ID and part number for cross builds. v7: Arm config options are now organized in one dictionary. Removed unsupported implementers and removed fallback to generic implementer/part number for unknown implementer/part number. Added meson config option arm_soc which can be used to specify configuration to be used, useful for aarch64 -> aarch64 builds. v8: Rebase. Juraj Linkeš (14): build: alias default build as generic build: rename Arm build variables build: remove unused or superfluous variables build: reformat and move Arm config and comments build: simplify how Arm flags are processed build: organize Arm config into dict build: isolate configuration for generic build build: use native machine args in Arm native build ci: switch to generic Arm builds build: optional NUMA and cpu counts detection build: add core and NUMA counts to cross files build: disable Arm drivers build: disable libnuma in cross builds build: add Arm SoC meson option .ci/linux-build.sh | 4 + buildtools/get_cpu_count.py | 7 + buildtools/get_numa_count.py | 22 + buildtools/meson.build | 2 + config/arm/arm64_armada_linux_gcc | 2 +- config/arm/arm64_armv8_linux_gcc | 26 +- config/arm/arm64_bluefield_linux_gcc | 3 +- config/arm/arm64_dpaa_linux_gcc | 2 +- config/arm/arm64_emag_linux_gcc | 2 +- config/arm/arm64_graviton2_linux_gcc | 3 +- config/arm/arm64_n1sdp_linux_gcc | 3 +- config/arm/arm64_octeontx2_linux_gcc | 3 +- config/arm/arm64_stingray_linux_gcc | 3 +- config/arm/arm64_thunderx2_linux_gcc | 3 +- ..._linux_gcc => arm64_thunderxt88_linux_gcc} | 2 +- config/arm/meson.build | 494 ++++++++++++------ config/meson.build | 90 +++- config/x86/meson.build | 2 + doc/guides/prog_guide/build-sdk-meson.rst | 4 +- drivers/meson.build | 6 +- meson.build | 1 + meson_options.txt | 12 +- 22 files changed, 483 insertions(+), 213 deletions(-) create mode 100644 buildtools/get_cpu_count.py create mode 100644 buildtools/get_numa_count.py rename config/arm/{arm64_thunderx_linux_gcc => arm64_thunderxt88_linux_gcc} (92%)
Comments
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Juraj Linkeš > Sent: Friday, November 6, 2020 9:03 AM > > The current way of specifying Arm configuration options is insufficient > since we can't identify the SoC we're building for from the MIDR > information. For example, we can't distinguish between N1SDP, Graviton2 > or Ampere Altra. > > Add a way to specify the cpu count and numa node count for cross > builds. > > We also want to be able to disable which drivers (and possibly > libraries) are built without user input. This is useful when building: > 1. on an SoC that is slow and we want to build only what is necessary > without the user having to check which libraries they have installed > 2. a cross build on a fast aarch64 machine but with target SoC which > differs in capabilities or libraries. > This is achieved by specifying the drivers in cross files. > > Among libraries, only libnuma can be now disabled. > > Also add an optional way to discover cpu count a numa node count. Fix > -Dmax_lcores and -Dmax_numa_nodes for arm builds. > > The current implementation adds/supports the following: > * x86 -> aarch64 cross build with added config options/disabled > drivers/libs > * aarch64 -> aarch64 builds for a different SoCs must be done using > cross-files > * max numa nodes and max lcore may be specified on the command line to > overwrite the values for any (native or cross) build > We are cross building our application firmware x86 -> x86. Our firmware repository includes the cross compiler source code, target libc source code, and so on. This ensures that the entire firmware is exactly the same, independently of which host it was built on. Back in the days, before we started using DPDK, we validated our cross building environment by building the complete firmware for our appliance (including bootloader, Linux kernel, libraries, applications, root file system, etc.) PPC -> x86. It worked. The DPDK cross building system should work in any X -> Y environment, including X -> X cross building. > v2: > Major rework of the whole series. > > v3: > Added numa and core count defaults for x86 default build. > Removed numa and core count defaults. Now requiring defaults to be > specified in a cross file or on the cmdline. > Added FreeBDS support for numa count discovery. > > v4: > Make automatic numa and cpu counts discovery optional. > > v5: > Split the refactor patch into smaller patches. > Simplify buildtools/get_numa_count.py. > Add more explanation to cover letter. > > v6: > Apply cross file options arch agnostically, not just in Arm cross > builds. > Streamline Arm build setup and machine args: always use native args in > native builds, require implementer ID and part number for cross builds. > > v7: > Arm config options are now organized in one dictionary. > Removed unsupported implementers and removed fallback to generic > implementer/part number for unknown implementer/part number. > Added meson config option arm_soc which can be used to specify > configuration to be used, useful for aarch64 -> aarch64 builds. > > v8: > Rebase. >
> -----Original Message----- > From: Morten Brørup <mb@smartsharesystems.com> > Sent: Friday, November 6, 2020 9:23 AM > To: Juraj Linkeš <juraj.linkes@pantheon.tech>; bruce.richardson@intel.com; > Ruifeng.Wang@arm.com; Honnappa.Nagarahalli@arm.com; > Phil.Yang@arm.com; vcchunga@amazon.com; Dharmik.Thakkar@arm.com; > jerinjacobk@gmail.com; hemant.agrawal@nxp.com; > ajit.khaparde@broadcom.com; ferruh.yigit@intel.com; aconole@redhat.com > Cc: dev@dpdk.org > Subject: RE: [dpdk-dev] [PATCH v8 00/14] Arm build options rework > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Juraj Linkeš > > Sent: Friday, November 6, 2020 9:03 AM > > > > The current way of specifying Arm configuration options is > > insufficient since we can't identify the SoC we're building for from > > the MIDR information. For example, we can't distinguish between N1SDP, > > Graviton2 or Ampere Altra. > > > > Add a way to specify the cpu count and numa node count for cross > > builds. > > > > We also want to be able to disable which drivers (and possibly > > libraries) are built without user input. This is useful when building: > > 1. on an SoC that is slow and we want to build only what is necessary > > without the user having to check which libraries they have installed > > 2. a cross build on a fast aarch64 machine but with target SoC which > > differs in capabilities or libraries. > > This is achieved by specifying the drivers in cross files. > > > > Among libraries, only libnuma can be now disabled. > > > > Also add an optional way to discover cpu count a numa node count. Fix > > -Dmax_lcores and -Dmax_numa_nodes for arm builds. > > > > The current implementation adds/supports the following: > > * x86 -> aarch64 cross build with added config options/disabled > > drivers/libs > > * aarch64 -> aarch64 builds for a different SoCs must be done using > > cross-files > > * max numa nodes and max lcore may be specified on the command line to > > overwrite the values for any (native or cross) build > > > > We are cross building our application firmware x86 -> x86. Our firmware > repository includes the cross compiler source code, target libc source code, and > so on. This ensures that the entire firmware is exactly the same, independently > of which host it was built on. > > Back in the days, before we started using DPDK, we validated our cross building > environment by building the complete firmware for our appliance (including > bootloader, Linux kernel, libraries, applications, root file system, etc.) PPC -> > x86. It worked. > > The DPDK cross building system should work in any X -> Y environment, including > X -> X cross building. > The series only modifies aarch64 aspect of the build system. So anything else that worked before should be working as it was. The wording should've been more clearer - The series adds/supports those bullet points. > > v2: > > Major rework of the whole series. > > > > v3: > > Added numa and core count defaults for x86 default build. > > Removed numa and core count defaults. Now requiring defaults to be > > specified in a cross file or on the cmdline. > > Added FreeBDS support for numa count discovery. > > > > v4: > > Make automatic numa and cpu counts discovery optional. > > > > v5: > > Split the refactor patch into smaller patches. > > Simplify buildtools/get_numa_count.py. > > Add more explanation to cover letter. > > > > v6: > > Apply cross file options arch agnostically, not just in Arm cross > > builds. > > Streamline Arm build setup and machine args: always use native args in > > native builds, require implementer ID and part number for cross builds. > > > > v7: > > Arm config options are now organized in one dictionary. > > Removed unsupported implementers and removed fallback to generic > > implementer/part number for unknown implementer/part number. > > Added meson config option arm_soc which can be used to specify > > configuration to be used, useful for aarch64 -> aarch64 builds. > > > > v8: > > Rebase. > >
> From: Juraj Linkeš [mailto:juraj.linkes@pantheon.tech] > Sent: Friday, November 6, 2020 9:40 AM > > > From: Morten Brørup <mb@smartsharesystems.com> > > Sent: Friday, November 6, 2020 9:23 AM > > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Juraj Linkeš > > > Sent: Friday, November 6, 2020 9:03 AM > > > > > > The current way of specifying Arm configuration options is > > > insufficient since we can't identify the SoC we're building for > from > > > the MIDR information. For example, we can't distinguish between > N1SDP, > > > Graviton2 or Ampere Altra. > > > > > > Add a way to specify the cpu count and numa node count for cross > > > builds. > > > > > > We also want to be able to disable which drivers (and possibly > > > libraries) are built without user input. This is useful when > building: > > > 1. on an SoC that is slow and we want to build only what is > necessary > > > without the user having to check which libraries they have > installed > > > 2. a cross build on a fast aarch64 machine but with target SoC > which > > > differs in capabilities or libraries. > > > This is achieved by specifying the drivers in cross files. > > > > > > Among libraries, only libnuma can be now disabled. > > > > > > Also add an optional way to discover cpu count a numa node count. > Fix > > > -Dmax_lcores and -Dmax_numa_nodes for arm builds. > > > > > > The current implementation adds/supports the following: > > > * x86 -> aarch64 cross build with added config options/disabled > > > drivers/libs > > > * aarch64 -> aarch64 builds for a different SoCs must be done using > > > cross-files > > > * max numa nodes and max lcore may be specified on the command line > to > > > overwrite the values for any (native or cross) build > > > > > > > We are cross building our application firmware x86 -> x86. Our > firmware > > repository includes the cross compiler source code, target libc > source code, and > > so on. This ensures that the entire firmware is exactly the same, > independently > > of which host it was built on. > > > > Back in the days, before we started using DPDK, we validated our > cross building > > environment by building the complete firmware for our appliance > (including > > bootloader, Linux kernel, libraries, applications, root file system, > etc.) PPC -> > > x86. It worked. > > > > The DPDK cross building system should work in any X -> Y environment, > including > > X -> X cross building. > > > > The series only modifies aarch64 aspect of the build system. So > anything else that worked before should be working as it was. The > wording should've been more clearer - The series adds/supports those > bullet points. > Bruce, Are all these nice details - such as being able to disable drivers and libraries - also available as configuration options for x86 -> x86 cross building?
On Fri, Nov 06, 2020 at 09:23:01AM +0100, Morten Brørup wrote: > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Juraj Linkeš Sent: > > Friday, November 6, 2020 9:03 AM > > > > The current way of specifying Arm configuration options is insufficient > > since we can't identify the SoC we're building for from the MIDR > > information. For example, we can't distinguish between N1SDP, Graviton2 > > or Ampere Altra. > > > > Add a way to specify the cpu count and numa node count for cross > > builds. > > > > We also want to be able to disable which drivers (and possibly > > libraries) are built without user input. This is useful when building: > > 1. on an SoC that is slow and we want to build only what is necessary > > without the user having to check which libraries they have installed 2. > > a cross build on a fast aarch64 machine but with target SoC which > > differs in capabilities or libraries. This is achieved by specifying > > the drivers in cross files. > > > > Among libraries, only libnuma can be now disabled. > > > > Also add an optional way to discover cpu count a numa node count. Fix > > -Dmax_lcores and -Dmax_numa_nodes for arm builds. > > > > The current implementation adds/supports the following: * x86 -> > > aarch64 cross build with added config options/disabled drivers/libs * > > aarch64 -> aarch64 builds for a different SoCs must be done using > > cross-files * max numa nodes and max lcore may be specified on the > > command line to overwrite the values for any (native or cross) build > > > > We are cross building our application firmware x86 -> x86. Our firmware > repository includes the cross compiler source code, target libc source > code, and so on. This ensures that the entire firmware is exactly the > same, independently of which host it was built on. > > Back in the days, before we started using DPDK, we validated our cross > building environment by building the complete firmware for our appliance > (including bootloader, Linux kernel, libraries, applications, root file > system, etc.) PPC -> x86. It worked. > > The DPDK cross building system should work in any X -> Y environment, > including X -> X cross building. > It should indeed work for x86 -> x86, but I'm not sure anyone has ever tested it yet. There are no cross-files for x86 included in the DPDK config folder, but it should work fine if you create one for your own setup. Some references, since this should all be handled by meson itself transparently: https://mesonbuild.com/Cross-compilation.html https://mesonbuild.com/Machine-files.html Only problem that I think would arise for x86-x86 would be if we had made some assumptions on the "config/x86/meson.build" file about it being a native build, but hopefully there are few or none of those. If you do get to try out this cross-compile, please let us know how it goes for you. Thanks, /Bruce
On Fri, Nov 06, 2020 at 09:56:42AM +0100, Morten Brørup wrote: > > From: Juraj Linkeš [mailto:juraj.linkes@pantheon.tech] > > Sent: Friday, November 6, 2020 9:40 AM > > > > > From: Morten Brørup <mb@smartsharesystems.com> > > > Sent: Friday, November 6, 2020 9:23 AM > > > > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Juraj Linkeš > > > > Sent: Friday, November 6, 2020 9:03 AM > > > > > > > > The current way of specifying Arm configuration options is > > > > insufficient since we can't identify the SoC we're building for > > from > > > > the MIDR information. For example, we can't distinguish between > > N1SDP, > > > > Graviton2 or Ampere Altra. > > > > > > > > Add a way to specify the cpu count and numa node count for cross > > > > builds. > > > > > > > > We also want to be able to disable which drivers (and possibly > > > > libraries) are built without user input. This is useful when > > building: > > > > 1. on an SoC that is slow and we want to build only what is > > necessary > > > > without the user having to check which libraries they have > > installed > > > > 2. a cross build on a fast aarch64 machine but with target SoC > > which > > > > differs in capabilities or libraries. > > > > This is achieved by specifying the drivers in cross files. > > > > > > > > Among libraries, only libnuma can be now disabled. > > > > > > > > Also add an optional way to discover cpu count a numa node count. > > Fix > > > > -Dmax_lcores and -Dmax_numa_nodes for arm builds. > > > > > > > > The current implementation adds/supports the following: > > > > * x86 -> aarch64 cross build with added config options/disabled > > > > drivers/libs > > > > * aarch64 -> aarch64 builds for a different SoCs must be done using > > > > cross-files > > > > * max numa nodes and max lcore may be specified on the command line > > to > > > > overwrite the values for any (native or cross) build > > > > > > > > > > We are cross building our application firmware x86 -> x86. Our > > firmware > > > repository includes the cross compiler source code, target libc > > source code, and > > > so on. This ensures that the entire firmware is exactly the same, > > independently > > > of which host it was built on. > > > > > > Back in the days, before we started using DPDK, we validated our > > cross building > > > environment by building the complete firmware for our appliance > > (including > > > bootloader, Linux kernel, libraries, applications, root file system, > > etc.) PPC -> > > > x86. It worked. > > > > > > The DPDK cross building system should work in any X -> Y environment, > > including > > > X -> X cross building. > > > > > > > The series only modifies aarch64 aspect of the build system. So > > anything else that worked before should be working as it was. The > > wording should've been more clearer - The series adds/supports those > > bullet points. > > > > Bruce, > > Are all these nice details - such as being able to disable drivers and libraries - also available as configuration options for x86 -> x86 cross building? Yes and no. Disabling libraries is not currently supported for anything, but there were proposals made to add a build option to support it. In terms of disabling driver that is supported via the existing build option, and the support added in this set will also work for specifying drivers to disable on x86 too. /Bruce