[v2,1/7] build: output a dependency log in build directory

Message ID 20240802124411.485430-2-bruce.richardson@intel.com (mailing list archive)
State Superseded
Delegated to: Thomas Monjalon
Headers
Series record and rework component dependencies |

Checks

Context Check Description
ci/checkpatch success coding style OK

Commit Message

Bruce Richardson Aug. 2, 2024, 12:44 p.m. UTC
As meson processes our DPDK source tree it manages dependencies
specified by each individual driver. To enable future analysis of the
dependency links between components, log the dependencies of each DPDK
component as it gets processed. This could potentially allow other tools
to automatically enable or disable components based on the desired end
components to be built, e.g. if the user requests net/ice, ensure that
common/iavf is also enabled in the drivers.

The output file produced is in "dot" or "graphviz" format, which allows
producing a graphical representation of the DPDK dependency tree if so
desired. For example: "dot -Tpng -O build/deps.dot" to produce the
image file "build/deps.dot.png".

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>

---
 app/meson.build        |  1 +
 buildtools/log-deps.py | 43 ++++++++++++++++++++++++++++++++++++++++++
 buildtools/meson.build |  2 ++
 drivers/meson.build    |  1 +
 lib/meson.build        |  1 +
 5 files changed, 48 insertions(+)
 create mode 100644 buildtools/log-deps.py

--
2.43.0
  

Comments

Burakov, Anatoly Sept. 2, 2024, 2:34 p.m. UTC | #1
On 8/2/2024 2:44 PM, Bruce Richardson wrote:
> As meson processes our DPDK source tree it manages dependencies
> specified by each individual driver. To enable future analysis of the
> dependency links between components, log the dependencies of each DPDK
> component as it gets processed. This could potentially allow other tools
> to automatically enable or disable components based on the desired end
> components to be built, e.g. if the user requests net/ice, ensure that
> common/iavf is also enabled in the drivers.
> 
> The output file produced is in "dot" or "graphviz" format, which allows
> producing a graphical representation of the DPDK dependency tree if so
> desired. For example: "dot -Tpng -O build/deps.dot" to produce the
> image file "build/deps.dot.png".
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>
> 
> ---

<snip>

> +try:
> +    contents = file_to_list(depsfile)
> +except FileNotFoundError:
> +    contents = ['digraph {\n', '}\n']
> +
> +component = sys.argv[1]
> +if len(sys.argv) > 2:
> +    contents[-1] = f'"{component}" -> {{ "{"\", \"".join(sys.argv[2:])}" }}\n'

This is invalid syntax prior to Python 3.12. You're doing escaped quotes 
("\", \"") inside a f-string block ({}). My machine has Python 3.9 and 
this breaks the script.
  
Bruce Richardson Sept. 3, 2024, 8:31 a.m. UTC | #2
On Mon, Sep 02, 2024 at 04:34:34PM +0200, Burakov, Anatoly wrote:
> On 8/2/2024 2:44 PM, Bruce Richardson wrote:
> > As meson processes our DPDK source tree it manages dependencies
> > specified by each individual driver. To enable future analysis of the
> > dependency links between components, log the dependencies of each DPDK
> > component as it gets processed. This could potentially allow other tools
> > to automatically enable or disable components based on the desired end
> > components to be built, e.g. if the user requests net/ice, ensure that
> > common/iavf is also enabled in the drivers.
> > 
> > The output file produced is in "dot" or "graphviz" format, which allows
> > producing a graphical representation of the DPDK dependency tree if so
> > desired. For example: "dot -Tpng -O build/deps.dot" to produce the
> > image file "build/deps.dot.png".
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > Acked-by: Konstantin Ananyev <konstantin.ananyev@huawei.com>
> > 
> > ---
> 
> <snip>
> 
> > +try:
> > +    contents = file_to_list(depsfile)
> > +except FileNotFoundError:
> > +    contents = ['digraph {\n', '}\n']
> > +
> > +component = sys.argv[1]
> > +if len(sys.argv) > 2:
> > +    contents[-1] = f'"{component}" -> {{ "{"\", \"".join(sys.argv[2:])}" }}\n'
> 
> This is invalid syntax prior to Python 3.12. You're doing escaped quotes
> ("\", \"") inside a f-string block ({}). My machine has Python 3.9 and this
> breaks the script.
> 
Thanks for the heads-up. I'll do a respin when I can.
  

Patch

diff --git a/app/meson.build b/app/meson.build
index 5b2c80c7a1..837a57ad0a 100644
--- a/app/meson.build
+++ b/app/meson.build
@@ -76,6 +76,7 @@  foreach app:apps

     if build
         subdir(name)
+        run_command([log_deps_cmd, 'dpdk-' + name, deps], check: false)
         if not build and require_apps
             error('Cannot build explicitly requested app "@0@".\n'.format(name)
                   + '\tReason: ' + reason)
diff --git a/buildtools/log-deps.py b/buildtools/log-deps.py
new file mode 100644
index 0000000000..a4331fa15b
--- /dev/null
+++ b/buildtools/log-deps.py
@@ -0,0 +1,43 @@ 
+#! /usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 Intel Corporation
+
+"""Utility script to build up a list of dependencies from meson."""
+
+import os
+import sys
+
+
+def file_to_list(filename):
+    """Read file into a list of strings."""
+    with open(filename) as f:
+        return f.readlines()
+
+
+def list_to_file(filename, lines):
+    """Write a list of strings out to a file."""
+    with open(filename, 'w') as f:
+        f.writelines(lines)
+
+
+depsfile = f'{os.environ["MESON_BUILD_ROOT"]}/deps.dot'
+
+# to reset the deps file on each build, the script is called without any params
+if len(sys.argv) == 1:
+    os.remove(depsfile)
+    sys.exit(0)
+
+try:
+    contents = file_to_list(depsfile)
+except FileNotFoundError:
+    contents = ['digraph {\n', '}\n']
+
+component = sys.argv[1]
+if len(sys.argv) > 2:
+    contents[-1] = f'"{component}" -> {{ "{"\", \"".join(sys.argv[2:])}" }}\n'
+else:
+    contents[-1] = f'"{component}"\n'
+
+contents.append('}\n')
+
+list_to_file(depsfile, contents)
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 3adf34e1a8..e2ba9d0ad4 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -24,6 +24,8 @@  get_numa_count_cmd = py3 + files('get-numa-count.py')
 get_test_suites_cmd = py3 + files('get-test-suites.py')
 has_hugepages_cmd = py3 + files('has-hugepages.py')
 cmdline_gen_cmd = py3 + files('dpdk-cmdline-gen.py')
+log_deps_cmd = py3 + files('log-deps.py')
+run_command(log_deps_cmd, check: false)  # call with no parameters to reset the file

 # install any build tools that end-users might want also
 install_data([
diff --git a/drivers/meson.build b/drivers/meson.build
index 66931d4241..d56308f1c1 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -154,6 +154,7 @@  foreach subpath:subdirs
         if build
             # pull in driver directory which should update all the local variables
             subdir(drv_path)
+            run_command([log_deps_cmd, class + '_' + name, deps], check: false)

             if dpdk_conf.get('RTE_IOVA_IN_MBUF') == 0 and require_iova_in_mbuf
                 build = false
diff --git a/lib/meson.build b/lib/meson.build
index 162287753f..106c2a947c 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -160,6 +160,7 @@  foreach l:libraries

     if build
         subdir(l)
+        run_command([log_deps_cmd, l, deps], check: false)
         if not build and require_libs
             error('Cannot build explicitly requested lib "@0@".\n'.format(name)
                     +'\tReason: ' + reason)