[3/3] doc: rewrite shell scripts in Python

Message ID 20220331212830.708912-4-dmitry.kozliuk@gmail.com (mailing list archive)
State Superseded, archived
Delegated to: Thomas Monjalon
Headers
Series doc: build on Windows |

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/Intel-compilation success Compilation OK
ci/iol-mellanox-Performance success Performance Testing PASS
ci/iol-intel-Performance success Performance Testing PASS
ci/intel-Testing success Testing PASS
ci/github-robot: build success github build: passed
ci/iol-intel-Functional success Functional Testing PASS
ci/iol-abi-testing success Testing PASS
ci/iol-aarch64-unit-testing success Testing PASS
ci/iol-x86_64-compile-testing success Testing PASS
ci/iol-aarch64-compile-testing success Testing PASS
ci/iol-x86_64-unit-testing success Testing PASS

Commit Message

Dmitry Kozlyuk March 31, 2022, 9:28 p.m. UTC
  Shell used in documentation generation could not run on Windows.
Rewrite scripts in Python.
New scripts use proper path separators and handle paths with spaces.

Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
 doc/api/generate_doxygen.py  | 19 +++++++++++++++++++
 doc/api/generate_doxygen.sh  | 12 ------------
 doc/api/generate_examples.py | 31 +++++++++++++++++++++++++++++++
 doc/api/generate_examples.sh | 20 --------------------
 doc/api/meson.build          |  6 +++---
 5 files changed, 53 insertions(+), 35 deletions(-)
 create mode 100644 doc/api/generate_doxygen.py
 delete mode 100755 doc/api/generate_doxygen.sh
 create mode 100644 doc/api/generate_examples.py
 delete mode 100755 doc/api/generate_examples.sh
  

Comments

Bruce Richardson April 1, 2022, 4:31 p.m. UTC | #1
On Fri, Apr 01, 2022 at 12:28:30AM +0300, Dmitry Kozlyuk wrote:
> Shell used in documentation generation could not run on Windows.
> Rewrite scripts in Python.
> New scripts use proper path separators and handle paths with spaces.
> 
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---

Generally looks ok to me, couple of minor comments inline below.

Reviewed-by: Bruce Richardson <bruce.richardson@intel.com>

>  doc/api/generate_doxygen.py  | 19 +++++++++++++++++++
>  doc/api/generate_doxygen.sh  | 12 ------------
>  doc/api/generate_examples.py | 31 +++++++++++++++++++++++++++++++
>  doc/api/generate_examples.sh | 20 --------------------
>  doc/api/meson.build          |  6 +++---
>  5 files changed, 53 insertions(+), 35 deletions(-)
>  create mode 100644 doc/api/generate_doxygen.py
>  delete mode 100755 doc/api/generate_doxygen.sh
>  create mode 100644 doc/api/generate_examples.py
>  delete mode 100755 doc/api/generate_examples.sh
> 
<snip>

> diff --git a/doc/api/generate_examples.py b/doc/api/generate_examples.py
> new file mode 100644
> index 0000000000..66933f9472
> --- /dev/null
> +++ b/doc/api/generate_examples.py
> @@ -0,0 +1,31 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# (c) 2018 Luca Boccassi <bluca@debian.org>
> +# (c) 2022 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> +
> +import os, sys
> +
> +examples_dir, api_examples = sys.argv[1:]
> +
> +sources = []
> +with open(f'{api_examples}.d', 'w') as dep:
> +    print(f'{api_examples}:', end=' ', file=dep)
> +    for root, _, files in os.walk(examples_dir):
> +        for name in files:
> +            is_source = name.endswith('.c')
> +            if is_source or name == 'meson.meson.build':

duplicate "meson."

> +                path = os.path.join(root, name)
> +                if is_source:
> +                    sources.append(path)
> +                print(path , end=' ', file=dep)
> +
> +sys.stdout = open(api_examples, 'w')

While this is a literal translation of what is done in the .sh file, I
wonder if, for consistency, in the python versions we should always either
use stdout redirection or always use "file=" option to printing. Right now
we have a mix of both.

> +print('/**')
> +print('@page examples DPDK Example Programs')
> +print()

Is this additional print done deliberately for clarity vs just putting '\n'
on previous line?

> +for path in sources:
> +    # Produce consistent output with forward slashes on all systems.
> +    # Every \ in paths within examples directory is a separator, not escape.
> +    relpath = os.path.relpath(path, examples_dir).replace('\\', '/')
> +    print(f'@example examples/{relpath}')
> +print('*/')
  
Dmitry Kozlyuk April 1, 2022, 9:47 p.m. UTC | #2
2022-04-01 17:31 (UTC+0100), Bruce Richardson:
> On Fri, Apr 01, 2022 at 12:28:30AM +0300, Dmitry Kozlyuk wrote:
[...]
> > +            if is_source or name == 'meson.meson.build':  
> 
> duplicate "meson."

Thanks!

> > +                path = os.path.join(root, name)
> > +                if is_source:
> > +                    sources.append(path)
> > +                print(path , end=' ', file=dep)
> > +
> > +sys.stdout = open(api_examples, 'w')  
> 
> While this is a literal translation of what is done in the .sh file, I
> wonder if, for consistency, in the python versions we should always either
> use stdout redirection or always use "file=" option to printing. Right now
> we have a mix of both.

I was wondering about this too.
Will change to "file=" (see also below) because explicit is better.

> > +print('/**')
> > +print('@page examples DPDK Example Programs')
> > +print()  
> 
> Is this additional print done deliberately for clarity vs just putting '\n'
> on previous line?

Yes. I think '''-string will be cleaner, especially with "file=".
  

Patch

diff --git a/doc/api/generate_doxygen.py b/doc/api/generate_doxygen.py
new file mode 100644
index 0000000000..2ccedf844e
--- /dev/null
+++ b/doc/api/generate_doxygen.py
@@ -0,0 +1,19 @@ 
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# (c) 2018 Luca Boccassi <bluca@debian.org>
+# (c) 2022 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import os, re, subprocess, sys
+
+pattern = re.compile('^Preprocessing (.*)...$')
+out_dir, *doxygen_command = sys.argv[1:]
+out_file = os.path.join(os.path.dirname(out_dir), 'doxygen.out')
+dep_file = f'{out_dir}.d'
+with open(out_file, 'w') as out:
+    subprocess.run(doxygen_command, check=True, stdout=out)
+with open(out_file) as out, open(dep_file, 'w') as dep:
+    print(f'{out_dir}:', end=' ', file=dep)
+    for line in out:
+        match = re.match(pattern, line)
+        if match:
+            print(match.group(1), end=' ', file=dep)
diff --git a/doc/api/generate_doxygen.sh b/doc/api/generate_doxygen.sh
deleted file mode 100755
index 1940ea46e2..0000000000
--- a/doc/api/generate_doxygen.sh
+++ /dev/null
@@ -1,12 +0,0 @@ 
-#! /bin/sh -e
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright 2018 Luca Boccassi <bluca@debian.org>
-
-DOXYCONF=$1
-OUTDIR=$2
-
-OUT_FILE=$(dirname $OUTDIR)/doxygen.out
-
-# run doxygen, capturing all the header files it processed
-doxygen "${DOXYCONF}" > $OUT_FILE
-echo "$OUTDIR: $(awk '/Preprocessing/ {printf("%s ", substr($2, 1, length($2) - 3))}' $OUT_FILE)" > $OUTDIR.d
diff --git a/doc/api/generate_examples.py b/doc/api/generate_examples.py
new file mode 100644
index 0000000000..66933f9472
--- /dev/null
+++ b/doc/api/generate_examples.py
@@ -0,0 +1,31 @@ 
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# (c) 2018 Luca Boccassi <bluca@debian.org>
+# (c) 2022 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import os, sys
+
+examples_dir, api_examples = sys.argv[1:]
+
+sources = []
+with open(f'{api_examples}.d', 'w') as dep:
+    print(f'{api_examples}:', end=' ', file=dep)
+    for root, _, files in os.walk(examples_dir):
+        for name in files:
+            is_source = name.endswith('.c')
+            if is_source or name == 'meson.meson.build':
+                path = os.path.join(root, name)
+                if is_source:
+                    sources.append(path)
+                print(path , end=' ', file=dep)
+
+sys.stdout = open(api_examples, 'w')
+print('/**')
+print('@page examples DPDK Example Programs')
+print()
+for path in sources:
+    # Produce consistent output with forward slashes on all systems.
+    # Every \ in paths within examples directory is a separator, not escape.
+    relpath = os.path.relpath(path, examples_dir).replace('\\', '/')
+    print(f'@example examples/{relpath}')
+print('*/')
diff --git a/doc/api/generate_examples.sh b/doc/api/generate_examples.sh
deleted file mode 100755
index 48574563ca..0000000000
--- a/doc/api/generate_examples.sh
+++ /dev/null
@@ -1,20 +0,0 @@ 
-#! /bin/sh -e
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright 2018 Luca Boccassi <bluca@debian.org>
-
-EXAMPLES_DIR=$1
-API_EXAMPLES=$2
-
-FIND=find
-
-# generate a .d file including both C files and also build files, so we can
-# detect both file changes and file additions/deletions
-echo "$API_EXAMPLES: $($FIND ${EXAMPLES_DIR} -type f \( -name '*.c' -o -name 'meson.build' \) | tr '\n' ' ' )" > ${API_EXAMPLES}.d
-
-exec > "${API_EXAMPLES}"
-printf '/**\n'
-printf '@page examples DPDK Example Programs\n\n'
-$FIND "${EXAMPLES_DIR}" -type f -name '*.c' |
-	sed "s|${EXAMPLES_DIR}|@example examples|" |
-	LC_ALL=C sort
-printf '*/\n'
diff --git a/doc/api/meson.build b/doc/api/meson.build
index 53338a02d6..2876a78a7e 100644
--- a/doc/api/meson.build
+++ b/doc/api/meson.build
@@ -11,8 +11,8 @@  endif
 # is in a subdirectory that is created at build time and thus it cannot
 # be an individual custom_target, we need to wrap the doxygen call in a
 # script to run the CSS modification afterwards
-generate_doxygen = find_program('generate_doxygen.sh')
-generate_examples = find_program('generate_examples.sh')
+generate_doxygen = py3 + files('generate_doxygen.py')
+generate_examples = py3 + files('generate_examples.py')
 
 htmldir = join_paths(get_option('datadir'), 'doc', 'dpdk')
 
@@ -51,7 +51,7 @@  doxy_build = custom_target('doxygen',
         input: doxy_conf,
         output: 'html',
         depfile: 'html.d',
-        command: [generate_doxygen, '@INPUT@', '@OUTPUT@'],
+        command: [generate_doxygen, '@OUTPUT@', doxygen, '@INPUT@'],
         install: get_option('enable_docs'),
         install_dir: htmldir,
         build_by_default: get_option('enable_docs'))