User Guide

Requirements

  • Python 3.7+

  • pip

Install

pip install pykiso

Poetry is more appropriate for developers as it automatically creates virtual environments.

git clone https://github.com/eclipse/kiso-testing.git
cd kiso-testing
poetry install
poetry shell

Usage

Once installed the application is bound to pykiso, it can be called with the following arguments:

$ pykiso --help
Usage: pykiso [OPTIONS]

  Embedded Integration Test Framework - CLI Entry Point.

  TAG Filters: any additional option to be passed to the test as tag through
  the pykiso call. Multiple values must be separated with a comma.

  For example: pykiso -c your_config.yaml --branch-level dev,master --variant
  delta

Options:
  -c, --test-configuration-file FILE
                                  path to the test configuration file (in YAML
                                  format)  [required]
  -l, --log-path PATH             path to log-file or folder. If not set will
                                  log to STDOUT
  --log-level [DEBUG|INFO|WARNING|ERROR]
                                  set the verbosity of the logging
  --junit                         enables the generation of a junit report
  --text                          default, test results are only displayed in
                                  the console
  --step-report PATH              generate the step report at the specified
                                  path
  --failfast                      stop the test run on the first error or
                                  failure
  -v, --verbose                   activate the internal framework logs
  -p, --pattern TEXT              test filter pattern, e.g. 'test_suite_1.py'
                                  or 'test_*.py'. Or even more granularly
                                  'test_suite_1.py::TestClass::test_name'
  --version                       Show the version and exit.
  -h, --help                      Show this message and exit.
/home/docs/checkouts/readthedocs.org/user_builds/kiso-testing/envs/0.19.2/lib/python3.7/site-packages/pykiso/interfaces/thread_auxiliary.py:41: FutureWarning: AuxiliaryInterface will be deprecated in a few releases!
  "AuxiliaryInterface will be deprecated in a few releases!", category=FutureWarning

Suitable config files are available in the examples folder.

Demo using example config

pykiso -c ./examples/dummy.yaml --log-level=DEBUG -l killme.log

Basic configuration

The test configuration files are written in YAML.

Let’s use an example to understand the structure.

 1auxiliaries:
 2  aux1:
 3    connectors:
 4        com: chan1
 5    config: null
 6    type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
 7  aux2:
 8    connectors:
 9        com:   chan2
10    type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
11  aux3:
12    connectors:
13        com:   chan4
14        flash: chan3
15    type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
16connectors:
17  chan1:
18    config: null
19    type: ext_lib/cc_example.py:CCExample
20  chan2:
21    type: ext_lib/cc_example.py:CCExample
22  chan4:
23    type: ext_lib/cc_example.py:CCExample
24  chan3:
25    config: null
26    type: pykiso.lib.connectors.cc_flasher_example:FlasherExample
27test_suite_list:
28- suite_dir: test_suite_1
29  test_filter_pattern: '*.py'
30  test_suite_id: 1
31- suite_dir: test_suite_2
32  test_filter_pattern: '*.py'
33  test_suite_id: 2
34
35requirements:
36  - pykiso : '>=0.10.1'
37  - robotframework : 3.2.2
38  - pyyaml: any

Connectors

The connector definition is a named list (dictionary in python) of key-value pairs, namely config and type.

    connectors:
        com:   chan4
        flash: chan3
    type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
connectors:
  chan1:
    config: null
    type: ext_lib/cc_example.py:CCExample
  chan2:
    type: ext_lib/cc_example.py:CCExample

The channel alias will identify this configuration for the auxiliaries.

The config can be omitted, null, or any number of key-value pairs.

The type consists of a module location and a class name that is expected to be found in the module. The location can be a path to a python file (Win/Linux, relative/absolute) or a python module on the python path (e.h. pykiso.lib.connectors.cc_uart).

<chan>:                      # channel alias
    config:                  # channel config, optional
        <key>: <value>       # collection of key-value pairs, e.g. "port: 80"
    type: <module:Class>     # location of the python class that represents this channel

Auxiliaries

The auxiliary definition is a named list (dictionary in python) of key-value pairs, namely config, connectors and type.

auxiliaries:
  aux1:
    connectors:
        com: chan1
    config: null
    type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
  aux2:
    connectors:
        com:   chan2
    type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
  aux3:

The auxiliary alias will identify this configuration for the testcases. When running the tests the testcases can import an auxiliary instance defined here using

from pykiso.auxiliaries import <alias>

The connectors can be omitted, null, or any number of role-connector pairs. The roles are defined in the auxiliary implementation, usual examples are com and flash. The channel aliases are the ones you defined in the connectors section above.

The config can be omitted, null, or any number of key-value pairs.

The type consists of a module location and a class name that is expected to be found in the module. The location can be a path to a python file (Win/Linux, relative/absolute) or a python module on the python path (e.h. pykiso.lib.auxiliaries.communication_auxiliary).

<aux>:                           # aux alias
    connectors:                  # list of connectors this auxiliary needs
        <role>: <channel-alias>  # <role> has to be the name defined in the Auxiliary class,
                                 #  <channel-alias> is the alias defined above
    config:                      # channel config, optional
        <key>: <value>           # collection of key-value pairs, e.g. "port: 80"
    type: <module:Class>         # location of the python class that represents this auxiliary

Test Suites

The test suite definition is a list of key-value pairs.

  chan4:
    type: ext_lib/cc_example.py:CCExample
  chan3:
    config: null
    type: pykiso.lib.connectors.cc_flasher_example:FlasherExample
test_suite_list:
- suite_dir: test_suite_1
  test_filter_pattern: '*.py'
  test_suite_id: 1
- suite_dir: test_suite_2
  test_filter_pattern: '*.py'
  test_suite_id: 2

Each test suite consists of a test_suite_id, a suite_dir and a test_filter_pattern.

For fast test development, the test_filter_pattern can be overwritten from the command line in order to e.g. execute a single test file inside the suite_dir using the CLI argument -p or –pattern:

pykiso -c dummy.yaml

To learn more, please take a look at How to make the most of the config file.

Basic test writing

Flow

  1. Create a root-folder that will contain the tests. Let us call it test-folder.

  2. Create, based on your test-specs, one folder per test-suite.

  3. In each test-suite folder, implement the tests. (See how below)

  4. write a configuration file (see How to make the most of the config file)

  5. If your test-setup is ready, run pykiso -c <ROOT_TEST_DIR>

  6. If the tests fail, you will see it in the the output. For more details, you can take a look at the log file (logs to STDOUT as default).

Note

User can run several test using several times flag -c. If a folder path is specified, a log for each yaml file will be stored. If otherwise a filename is provided, all log information will be in one logfile.

Define the test information

For each test fixture (setup, teardown or test_run), users have to define the test information using the decorator define_test_parameters. This decorator gives access to the following parameters:

  • suite_id: current test suite identification number

  • case_id: current test case identification number (optional for test suite setup and teardown)

  • aux_list: list of used auxiliaries

In order to utilise the SetUp/TearDown test-suite feature, users have to define a class inheriting from BasicTestSuiteSetup or BasicTestSuiteTeardown. For each of these classes, the following methods test_suite_setUp or test_suite_tearDown must be overridden with the behaviour you want to have.

Note

Because the python unittest module is used in the background, all methods
starting with “def test_” are executed automatically

Note

If a test in SuiteSetup raises an exception, all tests which belong to the same suite_id will be skipped.

Find below a full example for a test suite/case declaration :

"""
Add test suite setup fixture, run once at test suite's beginning.
Test Suite Setup Information:
-> suite_id : set to 1
-> case_id : Parameter case_id is not mandatory for setup.
-> aux_list : used aux1 and aux2 is used
"""
  @pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
  class SuiteSetup(pykiso.BasicTestSuiteSetup):
      def test_suite_setUp():
          logging.info("I HAVE RUN THE TEST SUITE SETUP!")
          if aux1.not_properly_configured():
              aux1.configure()
          aux2.configure()
          callback_registering()

"""
Add test suite teardown fixture, run once at test suite's end.
Test Suite Teardown Information:
-> suite_id : set to 1
-> case_id : Parameter case_id is not mandatory for setup.
-> aux_list : used aux1 and aux2 is used
"""
  @pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
  class SuiteTearDown(pykiso.BasicTestSuiteTeardown):
      def test_suite_tearDown():
          logging.info("I HAVE RUN THE TEST SUITE TEARDOWN!")
          callback_unregistering()

"""
Add a test case 1 from test suite 1 using auxiliary 1.
  Test Suite Teardown Information:
-> suite_id : set to 1
-> case_id : set to 1
-> aux_list : used aux1 and aux2 is used
"""
  @pykiso.define_test_parameters(
          suite_id=1,
          case_id=1,
          aux_list=[aux1, aux2]
  )
  class MyTest(pykiso.BasicTest):
      pass

Implementation of Basic Tests

Structure: test-folder/test-suite-1/test_suite_1.py

test_suite_1.py:

 """
 I want to run the following tests documented in the following test-specs <TEST_CASE_SPECS>.
 """
  import pykiso
  from pykiso.auxiliaries import aux1, aux2

"""
Add test suite setup fixture, run once at test suite's beginning.
Parameter case_id is not mandatory for setup.
"""
  @pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
  class SuiteSetup(pykiso.BasicTestSuiteSetup):
      pass

"""
Add test suite teardown fixture, run once at test suite's end.
Parameter case_id is not mandatory for teardown.
"""
  @pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
  class SuiteTearDown(pykiso.BasicTestSuiteTeardown):
      pass

"""
Add a test case 1 from test suite 1 using auxiliary 1.
"""
  @pykiso.define_test_parameters(suite_id=1, case_id=1, aux_list=[aux1])
  class MyTest(pykiso.BasicTest):
      pass

"""
Add a test case 2 from test suite 1 using auxiliary 2.
"""
  @pykiso.define_test_parameters(suite_id=1, case_id=2, aux_list=[aux2])
  class MyTest2(pykiso.BasicTest):
      pass

How are the tests called

Let us imagine we have 2 test-cases which are part of a test-suite.

import pykiso
from pykiso.auxiliaries import aux1, aux2

@pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
class SuiteSetup(pykiso.BasicTestSuiteSetup):
    pass

@pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
class SuiteTearDown(pykiso.BasicTestSuiteTeardown):
    pass

@pykiso.define_test_parameters(suite_id=1, case_id=1, aux_list=[aux1])
class TestCase1(pykiso.BasicTest):
    def setUp(self):
        pass
    def test_run_1(self):
        pass
    def test_run_2(self):
        pass
    def tearDown(self):
        pass

@pykiso.define_test_parameters(suite_id=1, case_id=1, aux_list=[aux1])
class TestCase2(pykiso.BasicTest):
    def setUp(self):
        pass
    def test_run_1(self):
        pass
    def test_run_2(self):
        pass
    def tearDown(self):
        pass

The pykiso will call the elements in the following order:

TestSuiteSetup().test_suite_setUp
TestCase1.setUpClass
    TestCase1().setUp
    TestCase1().test_run
    TestCase1().tearDown
    TestCase1().setUp
    TestCase1().test_run_2
    TestCase1().tearDown
TestCase1.tearDownClass
TestCase2.setUpClass
    TestCase2().setUp
    TestCase2().test_run
    TestCase2().tearDown
    TestCase2().setUp
    TestCase2().test_run_2
    TestCase2().tearDown
TestCase2.tearDownClass
TestSuiteTeardown().test_suite_tearDown

To learn more, please take a look at How to make the most of the tests.