[v3] doc/guides: add details for new test structure

Message ID 20211018145030.11433-1-ciara.power@intel.com (mailing list archive)
State Accepted, archived
Delegated to: Thomas Monjalon
Headers
Series [v3] doc/guides: add details for new test structure |

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/iol-testing warning apply patch failure
ci/Intel-compilation success Compilation OK
ci/intel-Testing success Testing PASS

Commit Message

Power, Ciara Oct. 18, 2021, 2:50 p.m. UTC
  The testing guide is now updated to include details about
using sub-testsuites. Some example code is given to demonstrate how
they can be used.

A note is also added to highlight the need for using vdev EAL args when
running cryptodev tests.

Signed-off-by: Ciara Power <ciara.power@intel.com>

---
Depends-on: patch-101811 ("guides: add a guide for developing unit tests")

v3:
  - Rebased on v5 of testing doc patch.
  - Moved "Depends-on" tag to notes section of commit.
v2:
  - Updated to depend on v4 of testing doc patch.
  - Added examples for running with cryptodev vdev args.
  - Small formatting fixes.
---
 doc/guides/contributing/unit_test.rst | 93 ++++++++++++++++++++++++++-
 1 file changed, 91 insertions(+), 2 deletions(-)
  

Comments

Power, Ciara Oct. 18, 2021, 2:54 p.m. UTC | #1
>-----Original Message-----
>From: Power, Ciara <ciara.power@intel.com>
>Sent: Monday 18 October 2021 15:51
>To: dev@dpdk.org
>Cc: Zhang, Roy Fan <roy.fan.zhang@intel.com>; aconole@redhat.com; Power,
>Ciara <ciara.power@intel.com>
>Subject: [PATCH v3] doc/guides: add details for new test structure
>
>The testing guide is now updated to include details about using sub-testsuites.
>Some example code is given to demonstrate how they can be used.
>
>A note is also added to highlight the need for using vdev EAL args when
>running cryptodev tests.
>
>Signed-off-by: Ciara Power <ciara.power@intel.com>
>
>---
>Depends-on: patch-101811 ("guides: add a guide for developing unit tests")
>

Adding missed acks from v2.

Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
Acked-by: John McNamara <john.mcnamara@intel.com>
  
Thomas Monjalon Nov. 26, 2021, 4:54 p.m. UTC | #2
> >The testing guide is now updated to include details about using sub-testsuites.
> >Some example code is given to demonstrate how they can be used.
> >
> >A note is also added to highlight the need for using vdev EAL args when
> >running cryptodev tests.
> >
> >Signed-off-by: Ciara Power <ciara.power@intel.com>
> >
> >---
> >Depends-on: patch-101811 ("guides: add a guide for developing unit tests")
> >
> 
> Adding missed acks from v2.
> 
> Acked-by: Fan Zhang <roy.fan.zhang@intel.com>
> Acked-by: John McNamara <john.mcnamara@intel.com>

Applied with few formatting changes, thanks.
  

Patch

diff --git a/doc/guides/contributing/unit_test.rst b/doc/guides/contributing/unit_test.rst
index b207b5aade..4be4acceab 100644
--- a/doc/guides/contributing/unit_test.rst
+++ b/doc/guides/contributing/unit_test.rst
@@ -185,13 +185,21 @@  for interacting with the test harness:
 
   2. unit_test_suite_runner(struct unit_test_suite \*)
      Returns a runner for a full test suite object, which contains
-     a test suite name, setup, tear down, and vector of unit test
-     cases.
+     a test suite name, setup, tear down, a pointer to a list of
+     sub-testsuites, and vector of unit test cases.
 
 Each test suite has a setup and tear down function that runs at the
 beginning and end of the test suite execution.  Each unit test has
 a similar function for test case setup and tear down.
 
+Each testsuite may use a nested list of sub-testsuites,
+which are iterated by the unit_test_suite_runner.
+This support allows for better granularity when designing test suites.
+The sub-testsuites list can also be used in parallel with the vector
+of testcases, in this case the testcases will be run,
+and then each sub-testsuite is executed. To see an example of a
+testsuite using sub-testsuites, see *app/test/test_cryptodev.c*.
+
 Test cases are added to the `.unit_test_cases` element of the appropriate
 unit test suite structure.  An example of both a test suite and a case:
 
@@ -236,6 +244,70 @@  unit test suite structure.  An example of both a test suite and a case:
 The above code block is a small example that can be used to create a
 complete test suite with test case.
 
+Sub-testsuites can be added to the `.unit_test_suites` element of
+the unit test suite structure, for example:
+
+.. code-block:: c
+   :linenos:
+
+   static int testsuite_setup(void) { return TEST_SUCCESS; }
+   static void testsuite_teardown(void) { }
+
+   static int ut_setup(void) { return TEST_SUCCESS; }
+   static void ut_teardown(void) { }
+
+   static int test_case_first(void) { return TEST_SUCCESS; }
+
+   static struct unit_test_suite example_parent_testsuite = {
+          .suite_name = "EXAMPLE PARENT TEST SUITE",
+          .setup = testsuite_setup,
+          .teardown = testsuite_teardown,
+          .unit_test_cases = {TEST_CASES_END()}
+   };
+
+   static int sub_testsuite_setup(void) { return TEST_SUCCESS; }
+   static void sub_testsuite_teardown(void) { }
+
+   static struct unit_test_suite example_sub_testsuite = {
+          .suite_name = "EXAMPLE SUB TEST SUITE",
+          .setup = sub_testsuite_setup,
+          .teardown = sub_testsuite_teardown,
+          .unit_test_cases = {
+               TEST_CASE_ST(ut_setup, ut_teardown, test_case_first),
+
+               TEST_CASES_END(), /**< NULL terminate unit test array */
+          },
+   };
+
+   static struct unit_test_suite end_testsuite = {
+          .suite_name = NULL,
+          .setup = NULL,
+          .teardown = NULL,
+          .unit_test_suites = NULL
+   };
+
+   static int example_tests()
+   {
+       uint8_t ret, i = 0;
+       struct unit_test_suite *sub_suites[] = {
+              &example_sub_testsuite,
+              &end_testsuite /**< NULL test suite to indicate end of list */
+        };
+
+       example_parent_testsuite.unit_test_suites =
+               malloc(sizeof(struct unit_test_suite *) * RTE_DIM(sub_suites));
+
+       for (i = 0; i < RTE_DIM(sub_suites); i++)
+           example_parent_testsuite.unit_test_suites[i] = sub_suites[i];
+
+       ret = unit_test_suite_runner(&example_parent_testsuite);
+       free(example_parent_testsuite.unit_test_suites);
+
+       return ret;
+   }
+
+   REGISTER_TEST_COMMAND(example_autotest, example_tests);
+
 
 Designing a test
 ----------------
@@ -325,3 +397,20 @@  In general, when a test is added to the `dpdk-test` application, it
 probably should be added to a meson test suite, but the choice is
 left to maintainers and individual developers.  Preference is to add
 tests to the meson test suites.
+
+
+Running Cryptodev Tests
+-----------------------
+
+When running cryptodev tests, the user must create any required virtual
+device via EAL args, as this is not automatically done by the test::
+
+  $ ./build/app/test/dpdk-test --vdev crypto_aesni_mb
+  $ meson test -C build --suite driver-tests \
+      --test-args="--vdev crypto_aesni_mb"
+
+.. note::
+
+   The cryptodev_scheduler_autotest is the only exception to this.
+   This vdev will be created automatically by the test app,
+   as it requires a more complex setup than other vdevs.