[v19,5/5] dts: add API doc generation
Checks
Commit Message
The tool used to generate DTS API docs is Sphinx, which is already in
use in DPDK. The same configuration is used to preserve style with one
DTS-specific configuration (so that the DPDK docs are unchanged) that
modifies how the sidebar displays the content. There's other Sphinx
configuration related to Python docstrings which doesn't affect DPDK doc
build. All new configuration is in a conditional block, applied only
when DTS API docs are built to not interfere with DPDK doc build.
Sphinx generates the documentation from Python docstrings. The docstring
format is the Google format [0] which requires the sphinx.ext.napoleon
extension. The other extension, sphinx.ext.intersphinx, enables linking
to objects in external documentations, such as the Python documentation.
There is one requirement for building DTS docs - the same Python version
as DTS or higher, because Sphinx's autodoc extension imports the code.
The dependencies needed to import the code don't have to be satisfied,
as the autodoc extension allows us to mock the imports. The missing
packages are taken from the DTS pyproject.toml file.
And finally, the DTS API docs can be accessed from the DPDK API doxygen
page.
[0] https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings
Cc: Bruce Richardson <bruce.richardson@intel.com>
Signed-off-by: Juraj Linkeš <juraj.linkes@pantheon.tech>
Reviewed-by: Jeremy Spewock <jspewock@iol.unh.edu>
Reviewed-by: Dean Marx <dmarx@iol.unh.edu>
---
buildtools/call-sphinx-build.py | 2 +
buildtools/get-dts-runtime-deps.py | 95 +++++++++++++++++++++++
buildtools/meson.build | 1 +
doc/api/doxy-api-index.md | 3 +
doc/api/doxy-api.conf.in | 2 +
doc/api/dts/custom.css | 1 +
doc/api/dts/meson.build | 31 ++++++++
doc/api/meson.build | 6 +-
doc/guides/conf.py | 44 ++++++++++-
doc/guides/contributing/documentation.rst | 2 +
doc/guides/contributing/patches.rst | 4 +
doc/guides/tools/dts.rst | 39 +++++++++-
doc/meson.build | 1 +
13 files changed, 228 insertions(+), 3 deletions(-)
create mode 100755 buildtools/get-dts-runtime-deps.py
create mode 120000 doc/api/dts/custom.css
create mode 100644 doc/api/dts/meson.build
Comments
On Wed, Aug 21, 2024 at 11:03 AM Juraj Linkeš <juraj.linkes@pantheon.tech>
wrote:
> The tool used to generate DTS API docs is Sphinx, which is already in
> use in DPDK. The same configuration is used to preserve style with one
> DTS-specific configuration (so that the DPDK docs are unchanged) that
> modifies how the sidebar displays the content. There's other Sphinx
> configuration related to Python docstrings which doesn't affect DPDK doc
> build. All new configuration is in a conditional block, applied only
> when DTS API docs are built to not interfere with DPDK doc build.
>
> Sphinx generates the documentation from Python docstrings. The docstring
> format is the Google format [0] which requires the sphinx.ext.napoleon
> extension. The other extension, sphinx.ext.intersphinx, enables linking
> to objects in external documentations, such as the Python documentation.
>
> There is one requirement for building DTS docs - the same Python version
> as DTS or higher, because Sphinx's autodoc extension imports the code.
>
> The dependencies needed to import the code don't have to be satisfied,
> as the autodoc extension allows us to mock the imports. The missing
> packages are taken from the DTS pyproject.toml file.
>
> And finally, the DTS API docs can be accessed from the DPDK API doxygen
> page.
>
Tested-by: Dean Marx <dmarx@iol.unh.edu>
Reviewed-by: Luca Vizzarro <luca.vizzarro@arm.com>
21/08/2024 17:02, Juraj Linkeš:
> +if 'dts' in src:
> + os.environ['DTS_BUILD'] = "y"
That's more precisely "DTS doc build".
I think the variable name DTS_BUILD may be confusing.
[...]
> --- /dev/null
> +++ b/buildtools/get-dts-runtime-deps.py
> @@ -0,0 +1,95 @@
> +#!/usr/bin/env python3
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2024 PANTHEON.tech s.r.o.
> +#
extra blank line above can be removed
> +
> +"""Utilities for DTS dependencies.
> +
> +The module can be used as an executable script,
> +which verifies that the running Python version meets the version requirement of DTS.
> +The script exits with the standard exit codes in this mode (0 is success, 1 is failure).
Given it is just doing a check by default,
the script name could be "check-dts-requirements".
> +
> +The module also contains a function, get_missing_imports,
> +which looks for runtime dependencies in the DTS pyproject.toml file
> +and returns a list of module names used in an import statement (import packages) that are missing.
> +This function is not used when the module is run as a script and is available to be imported.
> +"""
[...]
> + req_deps = _get_dependencies(_DTS_DEP_FILE_PATH)
> + req_deps.pop('python')
> +
> + for req_dep, dep_data in (req_deps | _EXTRA_DEPS).items():
Please could you explain somewhere why _EXTRA_DEPS is needed?
[...]
> +++ b/doc/api/dts/meson.build
> @@ -0,0 +1,31 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2023 PANTHEON.tech s.r.o.
> +
> +sphinx = find_program('sphinx-build', required: get_option('enable_docs'))
> +if not sphinx.found()
> + subdir_done()
> +endif
> +
> +python_ver_satisfied = run_command(get_dts_runtime_deps, check: false).returncode()
> +if python_ver_satisfied != 0
> + subdir_done()
> +endif
Looks simple.
So if I have the right Python but some dependencies are missing,
it will still work the same, right?
I feel the need for dependencies should be explained in the script.
[...]
> --- a/doc/api/meson.build
> +++ b/doc/api/meson.build
> @@ -1,6 +1,11 @@
> # SPDX-License-Identifier: BSD-3-Clause
> # Copyright(c) 2018 Luca Boccassi <bluca@debian.org>
>
> +# initialize common Doxygen configuration
> +cdata = configuration_data()
> +
> +subdir('dts')
Why inserting DTS first before generating DPDK API doc?
[...]
> # set up common Doxygen configuration
> -cdata = configuration_data()
> cdata.set('VERSION', meson.project_version())
> cdata.set('API_EXAMPLES', join_paths(dpdk_build_root, 'doc', 'api', 'examples.dox'))
> cdata.set('OUTPUT', join_paths(dpdk_build_root, 'doc', 'api'))
> diff --git a/doc/guides/conf.py b/doc/guides/conf.py
> index 0f7ff5282d..d7f3030838 100644
> --- a/doc/guides/conf.py
> +++ b/doc/guides/conf.py
> @@ -10,7 +10,7 @@
> from os.path import basename
> from os.path import dirname
> from os.path import join as path_join
> -from sys import argv, stderr
> +from sys import argv, stderr, path
>
> import configparser
>
> @@ -58,6 +58,48 @@
> ("tools/devbind", "dpdk-devbind",
> "check device status and bind/unbind them from drivers", "", 8)]
>
> +# DTS API docs additional configuration
> +if environ.get('DTS_BUILD'):
> + extensions = ['sphinx.ext.napoleon', 'sphinx.ext.autodoc', 'sphinx.ext.intersphinx']
> + # Napoleon enables the Google format of Python doscstrings.
> + napoleon_numpy_docstring = False
> + napoleon_attr_annotations = True
> + napoleon_preprocess_types = True
> +
> + # Autodoc pulls documentation from code.
> + autodoc_default_options = {
> + 'members': True,
> + 'member-order': 'bysource',
> + 'show-inheritance': True,
> + }
> + autodoc_class_signature = 'separated'
> + autodoc_typehints = 'both'
> + autodoc_typehints_format = 'short'
> + autodoc_typehints_description_target = 'documented'
> +
> + # Intersphinx allows linking to external projects, such as Python docs.
> + intersphinx_mapping = {'python': ('https://docs.python.org/3', None)}
I'm not sure about the need for this intersphinx.
> +
> + # DTS docstring options.
> + add_module_names = False
> + toc_object_entries = True
> + toc_object_entries_show_parents = 'hide'
> + # DTS Sidebar config.
> + html_theme_options = {
> + 'collapse_navigation': False,
> + 'navigation_depth': -1, # unlimited depth
> + }
> +
> + # Add path to DTS sources so that Sphinx can find them.
> + dpdk_root = dirname(dirname(dirname(__file__)))
> + path.append(path_join(dpdk_root, 'dts'))
> +
> + # Get missing DTS dependencies. Add path to buildtools to find the get_missing_imports function.
> + path.append(path_join(dpdk_root, 'buildtools'))
> + import importlib
> + # Ignore missing imports from DTS dependencies.
> + autodoc_mock_imports = importlib.import_module('get-dts-runtime-deps').get_missing_imports()
[...]
> +the corresponding changes must be made to DTS api doc sources in ``doc/api/dts``.
api -> API
Except minor corrections and explanations, it looks good.
You can add my ack to the final version.
Acked-by: Thomas Monjalon <thomas@monjalon.net>
On 12. 9. 2024 22:09, Thomas Monjalon wrote:
> 21/08/2024 17:02, Juraj Linkeš:
>> +if 'dts' in src:
>> + os.environ['DTS_BUILD'] = "y"
>
> That's more precisely "DTS doc build".
> I think the variable name DTS_BUILD may be confusing.
Ack, I'll rename the variable.
>
> [...]
>> --- /dev/null
>> +++ b/buildtools/get-dts-runtime-deps.py
>> @@ -0,0 +1,95 @@
>> +#!/usr/bin/env python3
>> +# SPDX-License-Identifier: BSD-3-Clause
>> +# Copyright(c) 2024 PANTHEON.tech s.r.o.
>> +#
>
> extra blank line above can be removed
>
Ack.
>> +
>> +"""Utilities for DTS dependencies.
>> +
>> +The module can be used as an executable script,
>> +which verifies that the running Python version meets the version requirement of DTS.
>> +The script exits with the standard exit codes in this mode (0 is success, 1 is failure).
>
> Given it is just doing a check by default,
> the script name could be "check-dts-requirements".
>
Ack.
>> +
>> +The module also contains a function, get_missing_imports,
>> +which looks for runtime dependencies in the DTS pyproject.toml file
>> +and returns a list of module names used in an import statement (import packages) that are missing.
>> +This function is not used when the module is run as a script and is available to be imported.
>> +"""
> [...]
>> + req_deps = _get_dependencies(_DTS_DEP_FILE_PATH)
>> + req_deps.pop('python')
>> +
>> + for req_dep, dep_data in (req_deps | _EXTRA_DEPS).items():
>
> Please could you explain somewhere why _EXTRA_DEPS is needed?
I'll add this comment above the variable:
# The names of packages used in import statements may be different from
distribution package names.
# We get distribution package names from pyproject.toml.
# _EXTRA_DEPS adds those import names which don't match their
distribution package name.
>
> [...]
>> +++ b/doc/api/dts/meson.build
>> @@ -0,0 +1,31 @@
>> +# SPDX-License-Identifier: BSD-3-Clause
>> +# Copyright(c) 2023 PANTHEON.tech s.r.o.
>> +
>> +sphinx = find_program('sphinx-build', required: get_option('enable_docs'))
>> +if not sphinx.found()
>> + subdir_done()
>> +endif
>> +
>> +python_ver_satisfied = run_command(get_dts_runtime_deps, check: false).returncode()
>> +if python_ver_satisfied != 0
>> + subdir_done()
>> +endif
>
> Looks simple.
> So if I have the right Python but some dependencies are missing,
> it will still work the same, right?
Yes.
> I feel the need for dependencies should be explained in the script.
From my point of view, the script gets the dependencies and it's up to
the caller how they use the list of dependencies.
The caller is conf.py and there's a bit of an explanation:
# Get missing DTS dependencies. Add path to buildtools to find the
get_missing_imports function.
And then:
# Ignore missing imports from DTS dependencies.
So basically get the dependencies so we know what to ignore.
But I could add something to the script if this is not enough.
>
> [...]
>> --- a/doc/api/meson.build
>> +++ b/doc/api/meson.build
>> @@ -1,6 +1,11 @@
>> # SPDX-License-Identifier: BSD-3-Clause
>> # Copyright(c) 2018 Luca Boccassi <bluca@debian.org>
>>
>> +# initialize common Doxygen configuration
>> +cdata = configuration_data()
>> +
>> +subdir('dts')
>
> Why inserting DTS first before generating DPDK API doc?
>
I wanted to put it before subdir_done(). Maybe we could put
subdir('dts') in the else branch and also at the end of the meson.build
file. That could be better.
> [...]
>> # set up common Doxygen configuration
>> -cdata = configuration_data()
>> cdata.set('VERSION', meson.project_version())
>> cdata.set('API_EXAMPLES', join_paths(dpdk_build_root, 'doc', 'api', 'examples.dox'))
>> cdata.set('OUTPUT', join_paths(dpdk_build_root, 'doc', 'api'))
>> diff --git a/doc/guides/conf.py b/doc/guides/conf.py
>> index 0f7ff5282d..d7f3030838 100644
>> --- a/doc/guides/conf.py
>> +++ b/doc/guides/conf.py
>> @@ -10,7 +10,7 @@
>> from os.path import basename
>> from os.path import dirname
>> from os.path import join as path_join
>> -from sys import argv, stderr
>> +from sys import argv, stderr, path
>>
>> import configparser
>>
>> @@ -58,6 +58,48 @@
>> ("tools/devbind", "dpdk-devbind",
>> "check device status and bind/unbind them from drivers", "", 8)]
>>
>> +# DTS API docs additional configuration
>> +if environ.get('DTS_BUILD'):
>> + extensions = ['sphinx.ext.napoleon', 'sphinx.ext.autodoc', 'sphinx.ext.intersphinx']
>> + # Napoleon enables the Google format of Python doscstrings.
>> + napoleon_numpy_docstring = False
>> + napoleon_attr_annotations = True
>> + napoleon_preprocess_types = True
>> +
>> + # Autodoc pulls documentation from code.
>> + autodoc_default_options = {
>> + 'members': True,
>> + 'member-order': 'bysource',
>> + 'show-inheritance': True,
>> + }
>> + autodoc_class_signature = 'separated'
>> + autodoc_typehints = 'both'
>> + autodoc_typehints_format = 'short'
>> + autodoc_typehints_description_target = 'documented'
>> +
>> + # Intersphinx allows linking to external projects, such as Python docs.
>> + intersphinx_mapping = {'python': ('https://docs.python.org/3', None)}
>
> I'm not sure about the need for this intersphinx.
It's not stricly needed, but it produces better documentation, with
links to Python docs for classes and other things found there.
For example:
:class:`~argparse.Action` in a docstring will link to
https://docs.python.org/3/library/argparse.html#argparse.Action
>
>> +
>> + # DTS docstring options.
>> + add_module_names = False
>> + toc_object_entries = True
>> + toc_object_entries_show_parents = 'hide'
>> + # DTS Sidebar config.
>> + html_theme_options = {
>> + 'collapse_navigation': False,
>> + 'navigation_depth': -1, # unlimited depth
>> + }
>> +
>> + # Add path to DTS sources so that Sphinx can find them.
>> + dpdk_root = dirname(dirname(dirname(__file__)))
>> + path.append(path_join(dpdk_root, 'dts'))
>> +
>> + # Get missing DTS dependencies. Add path to buildtools to find the get_missing_imports function.
>> + path.append(path_join(dpdk_root, 'buildtools'))
>> + import importlib
>> + # Ignore missing imports from DTS dependencies.
>> + autodoc_mock_imports = importlib.import_module('get-dts-runtime-deps').get_missing_imports()
> [...]
>> +the corresponding changes must be made to DTS api doc sources in ``doc/api/dts``.
>
> api -> API
>
Ack.
>
> Except minor corrections and explanations, it looks good.
> You can add my ack to the final version.
>
> Acked-by: Thomas Monjalon <thomas@monjalon.net>
>
>
16/09/2024 10:51, Juraj Linkeš:
> On 12. 9. 2024 22:09, Thomas Monjalon wrote:
> > 21/08/2024 17:02, Juraj Linkeš:
> >> + req_deps = _get_dependencies(_DTS_DEP_FILE_PATH)
> >> + req_deps.pop('python')
> >> +
> >> + for req_dep, dep_data in (req_deps | _EXTRA_DEPS).items():
> >
> > Please could you explain somewhere why _EXTRA_DEPS is needed?
>
> I'll add this comment above the variable:
> # The names of packages used in import statements may be different from
> distribution package names.
> # We get distribution package names from pyproject.toml.
> # _EXTRA_DEPS adds those import names which don't match their
> distribution package name.
Good
> > I feel the need for dependencies should be explained in the script.
>
> From my point of view, the script gets the dependencies and it's up to
> the caller how they use the list of dependencies.
>
> The caller is conf.py and there's a bit of an explanation:
> # Get missing DTS dependencies. Add path to buildtools to find the
> get_missing_imports function.
>
> And then:
> # Ignore missing imports from DTS dependencies.
>
> So basically get the dependencies so we know what to ignore.
>
> But I could add something to the script if this is not enough.
The unclear part is how it works without these dependencies.
> >> +# initialize common Doxygen configuration
> >> +cdata = configuration_data()
> >> +
> >> +subdir('dts')
> >
> > Why inserting DTS first before generating DPDK API doc?
>
> I wanted to put it before subdir_done(). Maybe we could put
> subdir('dts') in the else branch and also at the end of the meson.build
> file. That could be better.
Yes
> >> + # Intersphinx allows linking to external projects, such as Python docs.
> >> + intersphinx_mapping = {'python': ('https://docs.python.org/3', None)}
> >
> > I'm not sure about the need for this intersphinx.
>
> It's not stricly needed, but it produces better documentation, with
> links to Python docs for classes and other things found there.
>
> For example:
> :class:`~argparse.Action` in a docstring will link to
> https://docs.python.org/3/library/argparse.html#argparse.Action
If you think it helps, I'm fine.
On 16. 9. 2024 14:48, Thomas Monjalon wrote:
> 16/09/2024 10:51, Juraj Linkeš:
>> On 12. 9. 2024 22:09, Thomas Monjalon wrote:
>>> 21/08/2024 17:02, Juraj Linkeš:
>>>> + req_deps = _get_dependencies(_DTS_DEP_FILE_PATH)
>>>> + req_deps.pop('python')
>>>> +
>>>> + for req_dep, dep_data in (req_deps | _EXTRA_DEPS).items():
>>>
>>> Please could you explain somewhere why _EXTRA_DEPS is needed?
>>
>> I'll add this comment above the variable:
>> # The names of packages used in import statements may be different from
>> distribution package names.
>> # We get distribution package names from pyproject.toml.
>> # _EXTRA_DEPS adds those import names which don't match their
>> distribution package name.
>
> Good
>
Ack.
>
>>> I feel the need for dependencies should be explained in the script.
>>
>> From my point of view, the script gets the dependencies and it's up to
>> the caller how they use the list of dependencies.
>>
>> The caller is conf.py and there's a bit of an explanation:
>> # Get missing DTS dependencies. Add path to buildtools to find the
>> get_missing_imports function.
>>
>> And then:
>> # Ignore missing imports from DTS dependencies.
>>
>> So basically get the dependencies so we know what to ignore.
>>
>> But I could add something to the script if this is not enough.
>
> The unclear part is how it works without these dependencies.
>
It's a bit unclear to me as well. The docs [0] don't say much and the
only thing I found is that object paths from third party libraries are a
bit different:
without dependencies: fabric.Connection
with dependencies: fabric.connection.Connection
Everything else seems to be the same. Do we want to document this and if
so, where would be the best place? In the script or conf.py?
[0]
https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_mock_imports
>
>>>> +# initialize common Doxygen configuration
>>>> +cdata = configuration_data()
>>>> +
>>>> +subdir('dts')
>>>
>>> Why inserting DTS first before generating DPDK API doc?
>>
>> I wanted to put it before subdir_done(). Maybe we could put
>> subdir('dts') in the else branch and also at the end of the meson.build
>> file. That could be better.
>
> Yes
>
Ok, I'll make the change.
>
>>>> + # Intersphinx allows linking to external projects, such as Python docs.
>>>> + intersphinx_mapping = {'python': ('https://docs.python.org/3', None)}
>>>
>>> I'm not sure about the need for this intersphinx.
>>
>> It's not stricly needed, but it produces better documentation, with
>> links to Python docs for classes and other things found there.
>>
>> For example:
>> :class:`~argparse.Action` in a docstring will link to
>> https://docs.python.org/3/library/argparse.html#argparse.Action
>
> If you think it helps, I'm fine.
>
Absolutely, it'll make the docs easier to use.
>
@@ -15,6 +15,8 @@
# set the version in environment for sphinx to pick up
os.environ['DPDK_VERSION'] = version
+if 'dts' in src:
+ os.environ['DTS_BUILD'] = "y"
sphinx_cmd = [sphinx] + extra_args
new file mode 100755
@@ -0,0 +1,95 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2024 PANTHEON.tech s.r.o.
+#
+
+"""Utilities for DTS dependencies.
+
+The module can be used as an executable script,
+which verifies that the running Python version meets the version requirement of DTS.
+The script exits with the standard exit codes in this mode (0 is success, 1 is failure).
+
+The module also contains a function, get_missing_imports,
+which looks for runtime dependencies in the DTS pyproject.toml file
+and returns a list of module names used in an import statement (import packages) that are missing.
+This function is not used when the module is run as a script and is available to be imported.
+"""
+
+import configparser
+import importlib.metadata
+import importlib.util
+import os.path
+import platform
+
+from packaging.version import Version
+
+_VERSION_COMPARISON_CHARS = '^<>='
+_EXTRA_DEPS = {
+ 'invoke': {'version': '>=1.3'},
+ 'paramiko': {'version': '>=2.4'},
+ 'PyYAML': {'version': '^6.0', 'import_package': 'yaml'}
+}
+_DPDK_ROOT = os.path.dirname(os.path.dirname(__file__))
+_DTS_DEP_FILE_PATH = os.path.join(_DPDK_ROOT, 'dts', 'pyproject.toml')
+
+
+def _get_dependencies(cfg_file_path):
+ cfg = configparser.ConfigParser()
+ with open(cfg_file_path) as f:
+ dts_deps_file_str = f.read()
+ dts_deps_file_str = dts_deps_file_str.replace("\n]", "]")
+ cfg.read_string(dts_deps_file_str)
+
+ deps_section = cfg['tool.poetry.dependencies']
+ return {dep: {'version': deps_section[dep].strip('"\'')} for dep in deps_section}
+
+
+def get_missing_imports():
+ """Get missing DTS import packages from third party libraries.
+
+ Scan the DTS pyproject.toml file for dependencies and find those that are not installed.
+ The dependencies in pyproject.toml are listed by their distribution package names,
+ but the function finds the associated import packages - those used in import statements.
+
+ The function is not used when the module is run as a script. It should be imported.
+
+ Returns:
+ A list of missing import packages.
+ """
+ missing_imports = []
+ req_deps = _get_dependencies(_DTS_DEP_FILE_PATH)
+ req_deps.pop('python')
+
+ for req_dep, dep_data in (req_deps | _EXTRA_DEPS).items():
+ req_ver = dep_data['version']
+ try:
+ import_package = dep_data['import_package']
+ except KeyError:
+ import_package = req_dep
+ import_package = import_package.lower().replace('-', '_')
+
+ try:
+ req_ver = Version(req_ver.strip(_VERSION_COMPARISON_CHARS))
+ found_dep_ver = Version(importlib.metadata.version(req_dep))
+ if found_dep_ver < req_ver:
+ print(
+ f'The version "{found_dep_ver}" of package "{req_dep}" '
+ f'is lower than required "{req_ver}".'
+ )
+ except importlib.metadata.PackageNotFoundError:
+ print(f'Package "{req_dep}" not found.')
+ missing_imports.append(import_package)
+
+ return missing_imports
+
+
+if __name__ == '__main__':
+ python_version = _get_dependencies(_DTS_DEP_FILE_PATH).pop('python')
+ if python_version:
+ sys_ver = Version(platform.python_version())
+ req_ver = Version(python_version['version'].strip(_VERSION_COMPARISON_CHARS))
+ if sys_ver < req_ver:
+ print(
+ f'The available Python version "{sys_ver}" is lower than required "{req_ver}".'
+ )
+ exit(1)
@@ -24,6 +24,7 @@ get_numa_count_cmd = py3 + files('get-numa-count.py')
get_test_suites_cmd = py3 + files('get-test-suites.py')
has_hugepages_cmd = py3 + files('has-hugepages.py')
cmdline_gen_cmd = py3 + files('dpdk-cmdline-gen.py')
+get_dts_runtime_deps = py3 + files('get-dts-runtime-deps.py')
# install any build tools that end-users might want also
install_data([
@@ -245,3 +245,6 @@ The public API headers are grouped by topics:
[experimental APIs](@ref rte_compat.h),
[ABI versioning](@ref rte_function_versioning.h),
[version](@ref rte_version.h)
+
+- **tests**:
+ [**DTS**](@dts_api_main_page)
@@ -124,6 +124,8 @@ SEARCHENGINE = YES
SORT_MEMBER_DOCS = NO
SOURCE_BROWSER = YES
+ALIASES = "dts_api_main_page=@DTS_API_MAIN_PAGE@"
+
EXAMPLE_PATH = @TOPDIR@/examples
EXAMPLE_PATTERNS = *.c
EXAMPLE_RECURSIVE = YES
new file mode 120000
@@ -0,0 +1 @@
+../../guides/custom.css
\ No newline at end of file
new file mode 100644
@@ -0,0 +1,31 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2023 PANTHEON.tech s.r.o.
+
+sphinx = find_program('sphinx-build', required: get_option('enable_docs'))
+if not sphinx.found()
+ subdir_done()
+endif
+
+python_ver_satisfied = run_command(get_dts_runtime_deps, check: false).returncode()
+if python_ver_satisfied != 0
+ subdir_done()
+endif
+
+cdata.set('DTS_API_MAIN_PAGE', join_paths('..', 'dts', 'html', 'index.html'))
+
+extra_sphinx_args = ['-E', '-c', join_paths(doc_source_dir, 'guides')]
+if get_option('werror')
+ extra_sphinx_args += '-W'
+endif
+
+htmldir = join_paths(get_option('datadir'), 'doc', 'dpdk', 'dts')
+dts_api_html = custom_target('dts_api_html',
+ output: 'html',
+ command: [sphinx_wrapper, sphinx, meson.project_version(),
+ meson.current_source_dir(), meson.current_build_dir(), extra_sphinx_args],
+ build_by_default: get_option('enable_docs'),
+ install: get_option('enable_docs'),
+ install_dir: htmldir)
+
+doc_targets += dts_api_html
+doc_target_names += 'DTS_API_HTML'
@@ -1,6 +1,11 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(c) 2018 Luca Boccassi <bluca@debian.org>
+# initialize common Doxygen configuration
+cdata = configuration_data()
+
+subdir('dts')
+
doxygen = find_program('doxygen', required: get_option('enable_docs'))
if not doxygen.found()
@@ -30,7 +35,6 @@ example = custom_target('examples.dox',
build_by_default: get_option('enable_docs'))
# set up common Doxygen configuration
-cdata = configuration_data()
cdata.set('VERSION', meson.project_version())
cdata.set('API_EXAMPLES', join_paths(dpdk_build_root, 'doc', 'api', 'examples.dox'))
cdata.set('OUTPUT', join_paths(dpdk_build_root, 'doc', 'api'))
@@ -10,7 +10,7 @@
from os.path import basename
from os.path import dirname
from os.path import join as path_join
-from sys import argv, stderr
+from sys import argv, stderr, path
import configparser
@@ -58,6 +58,48 @@
("tools/devbind", "dpdk-devbind",
"check device status and bind/unbind them from drivers", "", 8)]
+# DTS API docs additional configuration
+if environ.get('DTS_BUILD'):
+ extensions = ['sphinx.ext.napoleon', 'sphinx.ext.autodoc', 'sphinx.ext.intersphinx']
+ # Napoleon enables the Google format of Python doscstrings.
+ napoleon_numpy_docstring = False
+ napoleon_attr_annotations = True
+ napoleon_preprocess_types = True
+
+ # Autodoc pulls documentation from code.
+ autodoc_default_options = {
+ 'members': True,
+ 'member-order': 'bysource',
+ 'show-inheritance': True,
+ }
+ autodoc_class_signature = 'separated'
+ autodoc_typehints = 'both'
+ autodoc_typehints_format = 'short'
+ autodoc_typehints_description_target = 'documented'
+
+ # Intersphinx allows linking to external projects, such as Python docs.
+ intersphinx_mapping = {'python': ('https://docs.python.org/3', None)}
+
+ # DTS docstring options.
+ add_module_names = False
+ toc_object_entries = True
+ toc_object_entries_show_parents = 'hide'
+ # DTS Sidebar config.
+ html_theme_options = {
+ 'collapse_navigation': False,
+ 'navigation_depth': -1, # unlimited depth
+ }
+
+ # Add path to DTS sources so that Sphinx can find them.
+ dpdk_root = dirname(dirname(dirname(__file__)))
+ path.append(path_join(dpdk_root, 'dts'))
+
+ # Get missing DTS dependencies. Add path to buildtools to find the get_missing_imports function.
+ path.append(path_join(dpdk_root, 'buildtools'))
+ import importlib
+ # Ignore missing imports from DTS dependencies.
+ autodoc_mock_imports = importlib.import_module('get-dts-runtime-deps').get_missing_imports()
+
# ####### :numref: fallback ########
# The following hook functions add some simple handling for the :numref:
@@ -133,6 +133,8 @@ added to by the developer.
Building the Documentation
--------------------------
+.. _doc_dependencies:
+
Dependencies
~~~~~~~~~~~~
@@ -499,6 +499,10 @@ The script usage is::
For both of the above scripts, the -n option is used to specify a number of commits from HEAD,
and the -r option allows the user specify a ``git log`` range.
+Additionally, when contributing to the DTS tool, patches should also be checked using
+the ``dts-check-format.sh`` script in the ``devtools`` directory of the DPDK repo.
+To run the script, extra :ref:`Python dependencies <dts_deps>` are needed.
+
.. _contrib_check_compilation:
Checking Compilation
@@ -54,6 +54,7 @@ DTS uses Poetry as its Python dependency management.
Python build/development and runtime environments are the same and DTS development environment,
DTS runtime environment or just plain DTS environment are used interchangeably.
+.. _dts_deps:
Setting up DTS environment
~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -291,8 +292,15 @@ When adding code to the DTS framework, pay attention to the rest of the code
and try not to divert much from it.
The :ref:`DTS developer tools <dts_dev_tools>` will issue warnings
when some of the basics are not met.
+You should also build the :ref:`API documentation <building_api_docs>`
+to address any issues found during the build.
-The code must be properly documented with docstrings.
+The API documentation, which is a helpful reference when developing, may be accessed
+in the code directly or generated with the :ref:`API docs build steps <building_api_docs>`.
+When adding new files or modifying the directory structure,
+the corresponding changes must be made to DTS api doc sources in ``doc/api/dts``.
+
+Speaking of which, the code must be properly documented with docstrings.
The style must conform to the `Google style
<https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings>`_.
See an example of the style `here
@@ -427,6 +435,35 @@ the DTS code check and format script.
Refer to the script for usage: ``devtools/dts-check-format.sh -h``.
+.. _building_api_docs:
+
+Building DTS API docs
+---------------------
+
+The documentation is built using the standard DPDK build system.
+See :doc:`../linux_gsg/build_dpdk` for more details on compiling DPDK with meson.
+
+The :ref:`doc build dependencies <doc_dependencies>` may be installed with Poetry:
+
+.. code-block:: console
+
+ poetry install --no-root --only docs
+ poetry install --no-root --with docs # an alternative that will also install DTS dependencies
+ poetry shell
+
+After executing the meson command, build the documentation with:
+
+.. code-block:: console
+
+ ninja -C build doc
+
+The output is generated in ``build/doc/api/dts/html``.
+
+.. note::
+
+ Make sure to fix any Sphinx warnings when adding or updating docstrings.
+
+
Configuration Schema
--------------------
@@ -1,6 +1,7 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(c) 2018 Luca Boccassi <bluca@debian.org>
+doc_source_dir = meson.current_source_dir()
doc_targets = []
doc_target_names = []
subdir('api')