mbox series

[RFC,v1,0/5] test case blocking and logging

Message ID 20231220103331.60888-1-juraj.linkes@pantheon.tech (mailing list archive)
Headers
Series test case blocking and logging |

Message

Juraj Linkeš Dec. 20, 2023, 10:33 a.m. UTC 
  We currently don't store test cases that couldn't be executed because of
a previous failure, such as when a test suite setup failed, resulting in
no executed test cases.

In order to record the test cases that couldn't be executed, we must
know the lists of test suites and test cases ahead of the actual test
suite execution, as an error could occur before we even start executing
test suites.

In addition, the patch series contains two refactors.

The first refactor is closely related. The dts.py was renamed to
runner.py and given a clear purpose - running the test suites and all
other orchestration needed to run test suites. The logic for this was
not all in the original dts.py module and it was brought there. The
runner is also responsible for recording results, which is the blocked
test cases are recorded.

The other refactor, logging, is related to the first refactor. The
logging module was simplified while extending capabilities. Each test
suite logs into its own log file in addition to the main log file which
the runner must handle (as it knows when we start executing particular
test suites). The runner also handles the switching between execution
stages for the purposes of logging.

Juraj Linkeš (5):
  dts: convert dts.py methods to class
  dts: move test suite execution logic to DTSRunner
  dts: process test suites at the beginning of run
  dts: block all testcases when earlier setup fails
  dts: refactor logging configuration

 dts/framework/config/__init__.py              |   8 +-
 dts/framework/dts.py                          | 228 --------
 dts/framework/logger.py                       | 162 +++---
 dts/framework/remote_session/__init__.py      |   4 +-
 dts/framework/remote_session/os_session.py    |   6 +-
 .../remote_session/remote/__init__.py         |   7 +-
 .../remote/interactive_remote_session.py      |   7 +-
 .../remote/interactive_shell.py               |   7 +-
 .../remote_session/remote/remote_session.py   |   8 +-
 .../remote_session/remote/ssh_session.py      |   5 +-
 dts/framework/runner.py                       | 499 ++++++++++++++++++
 dts/framework/test_result.py                  | 365 +++++++------
 dts/framework/test_suite.py                   | 193 +------
 dts/framework/testbed_model/node.py           |  10 +-
 dts/framework/testbed_model/scapy.py          |   6 +-
 .../testbed_model/traffic_generator.py        |   5 +-
 dts/main.py                                   |  12 +-
 dts/tests/TestSuite_smoke_tests.py            |   2 +-
 18 files changed, 830 insertions(+), 704 deletions(-)
 delete mode 100644 dts/framework/dts.py
 create mode 100644 dts/framework/runner.py

Comments

Jeremy Spewock Jan. 8, 2024, 6:47 p.m. UTC | #1 
Definitely worth-while changes. I looked them over and it all looks good so
far.

+1