[3/3] doc: rewrite shell scripts in Python
Checks
Commit Message
Shell used in documentation generation could not run on Windows.
Rewrite scripts in Python.
New scripts use proper path separators and handle paths with spaces.
Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
---
doc/api/generate_doxygen.py | 19 +++++++++++++++++++
doc/api/generate_doxygen.sh | 12 ------------
doc/api/generate_examples.py | 31 +++++++++++++++++++++++++++++++
doc/api/generate_examples.sh | 20 --------------------
doc/api/meson.build | 6 +++---
5 files changed, 53 insertions(+), 35 deletions(-)
create mode 100644 doc/api/generate_doxygen.py
delete mode 100755 doc/api/generate_doxygen.sh
create mode 100644 doc/api/generate_examples.py
delete mode 100755 doc/api/generate_examples.sh
Comments
On Fri, Apr 01, 2022 at 12:28:30AM +0300, Dmitry Kozlyuk wrote:
> Shell used in documentation generation could not run on Windows.
> Rewrite scripts in Python.
> New scripts use proper path separators and handle paths with spaces.
>
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> ---
Generally looks ok to me, couple of minor comments inline below.
Reviewed-by: Bruce Richardson <bruce.richardson@intel.com>
> doc/api/generate_doxygen.py | 19 +++++++++++++++++++
> doc/api/generate_doxygen.sh | 12 ------------
> doc/api/generate_examples.py | 31 +++++++++++++++++++++++++++++++
> doc/api/generate_examples.sh | 20 --------------------
> doc/api/meson.build | 6 +++---
> 5 files changed, 53 insertions(+), 35 deletions(-)
> create mode 100644 doc/api/generate_doxygen.py
> delete mode 100755 doc/api/generate_doxygen.sh
> create mode 100644 doc/api/generate_examples.py
> delete mode 100755 doc/api/generate_examples.sh
>
<snip>
> diff --git a/doc/api/generate_examples.py b/doc/api/generate_examples.py
> new file mode 100644
> index 0000000000..66933f9472
> --- /dev/null
> +++ b/doc/api/generate_examples.py
> @@ -0,0 +1,31 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# (c) 2018 Luca Boccassi <bluca@debian.org>
> +# (c) 2022 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> +
> +import os, sys
> +
> +examples_dir, api_examples = sys.argv[1:]
> +
> +sources = []
> +with open(f'{api_examples}.d', 'w') as dep:
> + print(f'{api_examples}:', end=' ', file=dep)
> + for root, _, files in os.walk(examples_dir):
> + for name in files:
> + is_source = name.endswith('.c')
> + if is_source or name == 'meson.meson.build':
duplicate "meson."
> + path = os.path.join(root, name)
> + if is_source:
> + sources.append(path)
> + print(path , end=' ', file=dep)
> +
> +sys.stdout = open(api_examples, 'w')
While this is a literal translation of what is done in the .sh file, I
wonder if, for consistency, in the python versions we should always either
use stdout redirection or always use "file=" option to printing. Right now
we have a mix of both.
> +print('/**')
> +print('@page examples DPDK Example Programs')
> +print()
Is this additional print done deliberately for clarity vs just putting '\n'
on previous line?
> +for path in sources:
> + # Produce consistent output with forward slashes on all systems.
> + # Every \ in paths within examples directory is a separator, not escape.
> + relpath = os.path.relpath(path, examples_dir).replace('\\', '/')
> + print(f'@example examples/{relpath}')
> +print('*/')
2022-04-01 17:31 (UTC+0100), Bruce Richardson:
> On Fri, Apr 01, 2022 at 12:28:30AM +0300, Dmitry Kozlyuk wrote:
[...]
> > + if is_source or name == 'meson.meson.build':
>
> duplicate "meson."
Thanks!
> > + path = os.path.join(root, name)
> > + if is_source:
> > + sources.append(path)
> > + print(path , end=' ', file=dep)
> > +
> > +sys.stdout = open(api_examples, 'w')
>
> While this is a literal translation of what is done in the .sh file, I
> wonder if, for consistency, in the python versions we should always either
> use stdout redirection or always use "file=" option to printing. Right now
> we have a mix of both.
I was wondering about this too.
Will change to "file=" (see also below) because explicit is better.
> > +print('/**')
> > +print('@page examples DPDK Example Programs')
> > +print()
>
> Is this additional print done deliberately for clarity vs just putting '\n'
> on previous line?
Yes. I think '''-string will be cleaner, especially with "file=".
new file mode 100644
@@ -0,0 +1,19 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# (c) 2018 Luca Boccassi <bluca@debian.org>
+# (c) 2022 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import os, re, subprocess, sys
+
+pattern = re.compile('^Preprocessing (.*)...$')
+out_dir, *doxygen_command = sys.argv[1:]
+out_file = os.path.join(os.path.dirname(out_dir), 'doxygen.out')
+dep_file = f'{out_dir}.d'
+with open(out_file, 'w') as out:
+ subprocess.run(doxygen_command, check=True, stdout=out)
+with open(out_file) as out, open(dep_file, 'w') as dep:
+ print(f'{out_dir}:', end=' ', file=dep)
+ for line in out:
+ match = re.match(pattern, line)
+ if match:
+ print(match.group(1), end=' ', file=dep)
deleted file mode 100755
@@ -1,12 +0,0 @@
-#! /bin/sh -e
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright 2018 Luca Boccassi <bluca@debian.org>
-
-DOXYCONF=$1
-OUTDIR=$2
-
-OUT_FILE=$(dirname $OUTDIR)/doxygen.out
-
-# run doxygen, capturing all the header files it processed
-doxygen "${DOXYCONF}" > $OUT_FILE
-echo "$OUTDIR: $(awk '/Preprocessing/ {printf("%s ", substr($2, 1, length($2) - 3))}' $OUT_FILE)" > $OUTDIR.d
new file mode 100644
@@ -0,0 +1,31 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# (c) 2018 Luca Boccassi <bluca@debian.org>
+# (c) 2022 Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
+
+import os, sys
+
+examples_dir, api_examples = sys.argv[1:]
+
+sources = []
+with open(f'{api_examples}.d', 'w') as dep:
+ print(f'{api_examples}:', end=' ', file=dep)
+ for root, _, files in os.walk(examples_dir):
+ for name in files:
+ is_source = name.endswith('.c')
+ if is_source or name == 'meson.meson.build':
+ path = os.path.join(root, name)
+ if is_source:
+ sources.append(path)
+ print(path , end=' ', file=dep)
+
+sys.stdout = open(api_examples, 'w')
+print('/**')
+print('@page examples DPDK Example Programs')
+print()
+for path in sources:
+ # Produce consistent output with forward slashes on all systems.
+ # Every \ in paths within examples directory is a separator, not escape.
+ relpath = os.path.relpath(path, examples_dir).replace('\\', '/')
+ print(f'@example examples/{relpath}')
+print('*/')
deleted file mode 100755
@@ -1,20 +0,0 @@
-#! /bin/sh -e
-# SPDX-License-Identifier: BSD-3-Clause
-# Copyright 2018 Luca Boccassi <bluca@debian.org>
-
-EXAMPLES_DIR=$1
-API_EXAMPLES=$2
-
-FIND=find
-
-# generate a .d file including both C files and also build files, so we can
-# detect both file changes and file additions/deletions
-echo "$API_EXAMPLES: $($FIND ${EXAMPLES_DIR} -type f \( -name '*.c' -o -name 'meson.build' \) | tr '\n' ' ' )" > ${API_EXAMPLES}.d
-
-exec > "${API_EXAMPLES}"
-printf '/**\n'
-printf '@page examples DPDK Example Programs\n\n'
-$FIND "${EXAMPLES_DIR}" -type f -name '*.c' |
- sed "s|${EXAMPLES_DIR}|@example examples|" |
- LC_ALL=C sort
-printf '*/\n'
@@ -11,8 +11,8 @@ endif
# is in a subdirectory that is created at build time and thus it cannot
# be an individual custom_target, we need to wrap the doxygen call in a
# script to run the CSS modification afterwards
-generate_doxygen = find_program('generate_doxygen.sh')
-generate_examples = find_program('generate_examples.sh')
+generate_doxygen = py3 + files('generate_doxygen.py')
+generate_examples = py3 + files('generate_examples.py')
htmldir = join_paths(get_option('datadir'), 'doc', 'dpdk')
@@ -51,7 +51,7 @@ doxy_build = custom_target('doxygen',
input: doxy_conf,
output: 'html',
depfile: 'html.d',
- command: [generate_doxygen, '@INPUT@', '@OUTPUT@'],
+ command: [generate_doxygen, '@OUTPUT@', doxygen, '@INPUT@'],
install: get_option('enable_docs'),
install_dir: htmldir,
build_by_default: get_option('enable_docs'))