User Guide

Requirements

Python 3.7+
pip

Install

pip install pykiso # Core framework
pip install pykiso[plugins] # To install all plugins
pip install pykiso[can] # To enable you to use CAN related plugins
pip install pykiso[debugger] # To enable you to use JLINK and else related plugins
pip install pykiso[instrument] # To enable you to use instrument control plugins

pip install pykiso[all] # To enable you to install everything we have to offer

Poetry is more appropriate for developers as it automatically creates virtual environments.

git clone https://github.com/eclipse/kiso-testing.git
cd kiso-testing
poetry install --all-extras
poetry shell

Usage

Once installed the application is bound to pykiso, it can be called with the following arguments:

$ pykiso --help
Usage: pykiso [OPTIONS]

  Embedded Integration Test Framework - CLI Entry Point.

  TAG Filters: any additional option to be passed to the test as tag through
  the pykiso call. Multiple values must be separated with a comma.

  For example: pykiso -c your_config.yaml --branch-level dev,master --variant
  delta

Options:
  -c, --test-configuration-file FILE
                                  path to the test configuration file (in YAML
                                  format)  [required]
  -l, --log-path PATH             path to log-file or folder. If not set will
                                  log to STDOUT
  --log-level [DEBUG|INFO|WARNING|ERROR]
                                  set the verbosity of the logging
  --junit                         Enable junit reports, if you want to save
                                  report in specific dir relative to current
                                  or .xml file, use --junit=(name or dir). The
                                  default dir is './reports' and default name
                                  is '%Y-%m-%d_%H-%M-%S-{config_name}.xml'
  --step-report PATH              generate the step report at the specified
                                  path
  --failfast                      stop the test run on the first error or
                                  failure
  -v, --verbose                   activate the internal framework logs
  -p, --pattern TEXT              test filter pattern, e.g. 'test_suite_1.py'
                                  or 'test_*.py'. Or even more granularly
                                  'test_suite_1.py::TestClass::test_name'
  --logger TEXT                   use the specified logger class in pykiso
  --version                       Show the version and exit.
  -h, --help                      Show this message and exit.

Suitable config files are available in the examples folder.

Demo using example config

pykiso -c ./examples/dummy.yaml --log-level=DEBUG -l killme.log

Basic configuration

The test configuration files are written in YAML.

Let’s use an example to understand the structure.

auxiliaries:
  aux1:
    connectors:
        com: chan1
    config: null
    type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
  aux2:
    connectors:
        com:   chan2
    type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
  aux3:
    connectors:
        com:   chan4
        flash: chan3
    type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
connectors:
  chan1:
    config: null
    type: ext_lib/cc_example.py:CCExample
  chan2:
    type: ext_lib/cc_example.py:CCExample
  chan4:
    type: ext_lib/cc_example.py:CCExample
  chan3:
    config: null
    type: pykiso.lib.connectors.cc_flasher_example:FlasherExample
test_suite_list:
- suite_dir: test_suite_1
  test_filter_pattern: '*.py'
  test_suite_id: 1
- suite_dir: test_suite_2
  test_filter_pattern: '*.py'
  test_suite_id: 2

requirements:
  - pykiso : '>=0.10.1'
  - robotframework : 3.2.2
  - pyyaml: any

Connectors

The connector definition is a named list (dictionary in python) of key-value pairs, namely config and type.

  aux3:
    connectors:
        com:   chan4
        flash: chan3
    type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
connectors:
  chan1:
    config: null
    type: ext_lib/cc_example.py:CCExample
  chan2:

The channel alias will identify this configuration for the auxiliaries.

The config can be omitted, null, or any number of key-value pairs.

The type consists of a module location and a class name that is expected to be found in the module. The location can be a path to a python file (Win/Linux, relative/absolute) or a python module on the python path (e.h. pykiso.lib.connectors.cc_uart).

<chan>:                      # channel alias
    config:                  # channel config, optional
        <key>: <value>       # collection of key-value pairs, e.g. "port: 80"
    type: <module:Class>     # location of the python class that represents this channel

Auxiliaries

The auxiliary definition is a named list (dictionary in python) of key-value pairs, namely config, connectors and type.

auxiliaries:
  aux1:
    connectors:
        com: chan1
    config: null
    type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
  aux2:
    connectors:
        com:   chan2
    type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary

The auxiliary alias will identify this configuration for the testcases. When running the tests the testcases can import an auxiliary instance defined here using

from pykiso.auxiliaries import <alias>

The connectors can be omitted, null, or any number of role-connector pairs. The roles are defined in the auxiliary implementation, usual examples are com and flash. The channel aliases are the ones you defined in the connectors section above.

The config can be omitted, null, or any number of key-value pairs.

The type consists of a module location and a class name that is expected to be found in the module. The location can be a path to a python file (Win/Linux, relative/absolute) or a python module on the python path (e.h. pykiso.lib.auxiliaries.communication_auxiliary).

<aux>:                           # aux alias
    connectors:                  # list of connectors this auxiliary needs
        <role>: <channel-alias>  # <role> has to be the name defined in the Auxiliary class,
                                 #  <channel-alias> is the alias defined above
    config:                      # channel config, optional
        <key>: <value>           # collection of key-value pairs, e.g. "port: 80"
    type: <module:Class>         # location of the python class that represents this auxiliary

Test Suites

The test suite definition is a list of key-value pairs.

    type: ext_lib/cc_example.py:CCExample
  chan4:
    type: ext_lib/cc_example.py:CCExample
  chan3:
    config: null
    type: pykiso.lib.connectors.cc_flasher_example:FlasherExample
test_suite_list:
- suite_dir: test_suite_1
  test_filter_pattern: '*.py'
  test_suite_id: 1
- suite_dir: test_suite_2
  test_filter_pattern: '*.py'
  test_suite_id: 2

Each test suite consists of a test_suite_id, a suite_dir and a test_filter_pattern.

For fast test development, the test_filter_pattern can be overwritten from the command line in order to e.g. execute a single test file inside the suite_dir using the CLI argument -p or –pattern:

pykiso -c dummy.yaml

To learn more, please take a look at How to make the most of the config file.

Basic test writing

Flow

Create a root-folder that will contain the tests. Let us call it test-folder.
Create, based on your test-specs, one folder per test-suite.
In each test-suite folder, implement the tests. (See how below)
write a configuration file (see How to make the most of the config file)
If your test-setup is ready, run pykiso -c <ROOT_TEST_DIR>
If the tests fail, you will see it in the the output. For more details, you can take a look at the log file (logs to STDOUT as default).

Note

User can run several test using several times flag -c. If a folder path is specified, a log for each yaml file will be stored. If otherwise a filename is provided, all log information will be in one logfile.

Define the test information

For each test fixture (setup, teardown or test_run), users have to define the test information using the decorator define_test_parameters. This decorator gives access to the following parameters:

suite_id: current test suite identification number
case_id: current test case identification number (optional for test suite setup and teardown)
aux_list: list of used auxiliaries

suite_id and case_id are used to coordinate and define a clear test execution order. It is now optional for pykiso.BasicTest but still mandatory in case of pykiso.RemoteTest as the TestApp protocol relies on these identifiers.

If none of these IDs are defined, the default value of 0 will be used and the alphabetical order will be applied on each .py module and each contained test class.

In other words, If we have the following test suite folder organisation:

test_suite_folder
├── a.py module with classes B/C/A (declare in this order)
├── b.py module with classes Z/G (declare in this order)
├── c.py module with classes S/T + Suite Setup/Teardown

The framework will execute the tests in the following order:

test_suite_setUp (c.SuiteSetup-0-0)
test_run (a.A-0-0)
test_run (a.B-0-0)
test_run (a.C-0-0)
test_run (b.G-0-0)
test_run (b.Z-0-0)
test_run (c.S-0-0)
test_run (c.T-0-0)
test_suite_tearDown (c.SuiteTearDown-0-0)

In order to make use of the SetUp/TearDown test-suite feature, users have to define a class inheriting from BasicTestSuiteSetup or BasicTestSuiteTeardown. For each of these classes, the following methods test_suite_setUp or test_suite_tearDown must be overridden with the behaviour you want to have.

Note

Because the python unittest module is used in the background, all methods

starting with “def test_” are executed automatically

Note

If a test in SuiteSetup raises an exception, all tests which belong to the same suite_id will be skipped.

Find below a full example for a test suite/case declaration :

"""
Add test suite setup fixture, run once at test suite's beginning.
Test Suite Setup Information:
-> suite_id : set to 1
-> case_id : Parameter case_id is not mandatory for setup.
-> aux_list : used aux1 and aux2 is used
"""
@pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
class SuiteSetup(pykiso.BasicTestSuiteSetup):
    def test_suite_setUp():
        logging.info("I HAVE RUN THE TEST SUITE SETUP!")
        if aux1.not_properly_configured():
            aux1.configure()
        aux2.configure()
        callback_registering()

"""
Add test suite teardown fixture, run once at test suite's end.
Test Suite Teardown Information:
-> suite_id : set to 1
-> case_id : Parameter case_id is not mandatory for setup.
-> aux_list : used aux1 and aux2 is used
"""
@pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
class SuiteTearDown(pykiso.BasicTestSuiteTeardown):
    def test_suite_tearDown():
        logging.info("I HAVE RUN THE TEST SUITE TEARDOWN!")
        callback_unregistering()

"""
Add a test case 1 from test suite 1 using auxiliary 1.
    Test Suite Teardown Information:
-> suite_id : set to 1
-> case_id : set to 1
-> aux_list : used aux1 and aux2 is used
"""
@pykiso.define_test_parameters(
        suite_id=1,
        case_id=1,
        aux_list=[aux1, aux2]
)
class MyTest(pykiso.BasicTest):
    pass

Implementation of Basic Tests

Structure: test-folder/test-suite-1/test_suite_1.py

test_suite_1.py:

"""
I want to run the following tests documented in the following test-specs <TEST_CASE_SPECS>.
"""
import pykiso
from pykiso.auxiliaries import aux1, aux2

"""
Add test suite setup fixture, run once at test suite's beginning.
Parameter case_id is not mandatory for setup.
"""
@pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
class SuiteSetup(pykiso.BasicTestSuiteSetup):
    pass

"""
Add test suite teardown fixture, run once at test suite's end.
Parameter case_id is not mandatory for teardown.
"""
@pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
class SuiteTearDown(pykiso.BasicTestSuiteTeardown):
    pass

"""
Add a test case 1 from test suite 1 using auxiliary 1.
"""
@pykiso.define_test_parameters(suite_id=1, case_id=1, aux_list=[aux1])
class MyTest(pykiso.BasicTest):
    pass

"""
Add a test case 2 from test suite 1 using auxiliary 2.
"""
@pykiso.define_test_parameters(suite_id=1, case_id=2, aux_list=[aux2])
class MyTest2(pykiso.BasicTest):
    pass

How are the tests called

Let us imagine we have 2 test-cases which are part of a test-suite.

import pykiso
from pykiso.auxiliaries import aux1, aux2

@pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
class SuiteSetup(pykiso.BasicTestSuiteSetup):
    pass

@pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
class SuiteTearDown(pykiso.BasicTestSuiteTeardown):
    pass

@pykiso.define_test_parameters(suite_id=1, case_id=1, aux_list=[aux1])
class TestCase1(pykiso.BasicTest):
    def setUp(self):
        pass
    def test_run_1(self):
        pass
    def test_run_2(self):
        pass
    def tearDown(self):
        pass

@pykiso.define_test_parameters(suite_id=1, case_id=1, aux_list=[aux1])
class TestCase2(pykiso.BasicTest):
    def setUp(self):
        pass
    def test_run_1(self):
        pass
    def test_run_2(self):
        pass
    def tearDown(self):
        pass

The pykiso will call the elements in the following order:

TestSuiteSetup().test_suite_setUp

    TestCase1.setUpClass

        TestCase1().setUp
        TestCase1().test_run
        TestCase1().tearDown

        TestCase1().setUp
        TestCase1().test_run_2
        TestCase1().tearDown

    TestCase1.tearDownClass

    TestCase2.setUpClass

        TestCase2().setUp
        TestCase2().test_run
        TestCase2().tearDown

        TestCase2().setUp
        TestCase2().test_run_2
        TestCase2().tearDown

    TestCase2.tearDownClass

TestSuiteTeardown().test_suite_tearDown

To learn more, please take a look at How to make the most of the tests.

NEW Pykiso as a simulator

Introduction

Pykiso consists of two main components:

The testing framework
The test environment creation and management

Together, these components enable embedded software engineers to efficiently test their software in a familiar manner. Pykiso’s testing framework adopts an approach similar to popular embedded testing frameworks, such as Google Test, making it intuitive for engineers with experience in embedded systems.

Over the years, we have identified two main groups of Pykiso users:

Those who embrace our opinionated approach to testing embedded software.
Those who appreciate the core principles and resources but prefer a different approach to testing.

By decoupling the testing framework from the test environment creation, Pykiso now caters to both groups, offering flexibility while retaining the benefits of its robust structure.

Workflow Overview

Create a test environment Begin by defining your test environment in a configuration file. (Refer to Basic configuration for more details.)

Strip the test suites section If needed, you can remove the Test Suites section from the configuration file to simplify the setup.

Write your (test) script Create a Python script, import the Pykiso library, import the test environment and use the auxiliaries to interact with the system under test.

This workflow allows users to leverage Pykiso’s testing capabilities while maintaining flexibility in how they define and manage their test environments.

Example

Definition of the test environment:

auxiliaries:
  com_aux_sender:
    connectors:
        com: loopback
    type: pykiso.lib.auxiliaries.communication_auxiliary:CommunicationAuxiliary
  com_aux_receiver:
    connectors:
        com: loopback
    type: pykiso.lib.auxiliaries.communication_auxiliary:CommunicationAuxiliary
connectors:
  loopback:
    type: pykiso.lib.connectors.cc_raw_loopback:CCLoopback

Creation of the test script:

from pathlib import Path

# Import pykiso
import pykiso

# Load the test environment configuration
pykiso.load_config(Path(__file__).parent.resolve() / "serial.yaml")

# From the pykiso library, import the type of auxiliary you defined in the configuration
# Here, it would be the CommunicationAuxiliary class
from pykiso.lib.auxiliaries.communication_auxiliary import CommunicationAuxiliary


def first_test():
    """Ping-pong test between sender and receiver (with no context manager)
    """
    # Get the instance of the sender and receiver defined in the configuration
    sender = CommunicationAuxiliary.get_instance('com_aux_sender')
    receiver = CommunicationAuxiliary.get_instance('com_aux_receiver')
    # Start the sender and receiver
    sender.start()
    receiver.start()
    # Use the auxiliaries for my test
    sender.send_message("Hello, World!")
    assert receiver.receive_message(timeout_in_s = 2) == "Hello, World!"
    print("Test passed!")
    # Stop the sender and receiver
    sender.stop()
    receiver.stop()

def second_test():
    """Ping-pong test between sender and receiver (with context manager)
    """
    # Get the instance of the sender and receiver defined in the configuration
    sender = CommunicationAuxiliary.get_instance('com_aux_sender')
    receiver = CommunicationAuxiliary.get_instance('com_aux_receiver')
    # Start the auxiliaries with a context manager
    with sender as sender, receiver as receiver:
        # Use the auxiliaries for my test
        sender.send_message("Hello, World!")
        assert receiver.receive_message(timeout_in_s = 2) == "Hello, World!"
    print("Second test passed!")

if __name__ == "__main__":
    first_test()
    second_test()

User Guide

Requirements

Install

Usage

Demo using example config

Basic configuration

Connectors

Auxiliaries

Test Suites

Basic test writing

Flow

Define the test information

Implementation of Basic Tests

How are the tests called

** NEW ** Pykiso as a simulator

Introduction

Workflow Overview

Example

NEW Pykiso as a simulator