Pykiso
Introduction
Integration Test Framework (Pykiso) is a framework that can be used for both white-box and black-box testing as well as in the integration and system testing.
Quality Goals
The framework tries to achieve the following quality goals:
Quality Goal (with prio) |
Scenarios |
---|---|
Portability |
The framework shall run on linux, windows, macOS |
The framework shall run on a raspberryPI or a regular laptop |
|
Modularity |
The framework shall allow me to implement complex logic and to run it over any communication port |
The framework shall allow me to add any communication port |
|
The framework shall allow me to use private modules within my tests if it respects its APIs |
|
The framework shall allow me to define my own test approach |
|
Correctness |
The framework shall verify that its inputs (test-setup) are correct before performing any test |
The framework shall execute the provided tests always in the same order |
|
Usability |
The framework shall feel familiar for embedded developers |
The framework shall feel familiar for system tester |
|
The framework shall generate test reports that are human and machine readable |
|
Performance (new) |
The framework shall use only the right/reasonable amount of resources to run (real-time timings) |
Design Overview
The pykiso Testing Framework is built in a modular and configurable way with abstractions both for entities (e.g. a handler for the device under test) and communication (e.g. UART or TCP/IP).
The tests leverage the python unittest-Framework which has a similar flavor as many available major unit testing frameworks and thus comes with an ecosystem of tools and utilities.
Test Coordinator
The test-coordinator is the central module setting up and running the tests. Based on a configuration file (in YAML), it does the following:
instantiate the selected connectors
instantiate the selected auxiliaries
provide the auxiliaries with the matching connectors
generate the list of tests to perform
provide the testcases with the auxiliaries they need
verify if the tests can be performed
for remote tests (see ../remote_test/remote_test) flash and run and synchronize the tests on the auxiliaries
gather the reports and publish the results
Auxiliary
The auxiliary provides to the test-coordinator an interface to interact with the physical or digital auxiliary target. It is composed by 2 blocks:
instance creation / deletion
connectors to facilitate interaction and communication with the device (e.g. messaging with UART)
For example auxiliaries like the one interacting with cloud services, we may have:
A communication channel (cchannel) like REST
Create an Auxiliary
Detailed information can be found here How to create an auxiliary.
Connector
Communication Channel
The Communication Channel - also known as cchannel - is the medium to communicate with auxiliary target. Example include UART, UDP, USB, REST,… The communication protocol itself can be auxiliary specific.
Create a Connector
Detailed information can be found here How to create a connector.
Dynamic Import Linking
The pykiso framework was developed with modularity and reusability in mind. To avoid close coupling between testcases and auxiliaries as well as between auxiliaries and connectors, the linking between those components is defined in a config file (see Test Configuration File) and performed by the TestCoordinator.
Different instances of connectors and auxiliaries are given aliases which identify them within the test session.
Let’s say we have this (abridged) config file:
connectors:
my_chan: # Alias of the connector
type: ...
auxiliaries:
my_aux: # Alias of the auxiliary
connectors:
com: my_chan # Reference to the connector
type: ...
The auxiliary my_aux will automatically be initialised with my_chan as its com channel.
When writing your testcases, the auxiliary will then be available under its defined alias.
from pykiso.auxiliaries import my_aux
The pykiso.auxiliaries
is a magic package that only exists in the pykiso
package after the TestCoordinator
has processed the config file.
It will include all instances of the defined auxiliares, available at their defined alias.
Usage
Flow
Create a root-folder that will contain the tests. Let us call it test-folder.
Create, based on your test-specs, one folder per test-suite.
In each test-suite folder, implement the tests. (See how below)
write a configuration file (see Test Configuration File)
If your test-setup is ready, run
pykiso -c <ROOT_TEST_DIR>
If the tests fail, you will see it in the the output. For more details, you can take a look at the log file (logs to STDOUT as default).
Note
User can run several test using several times flag -c. If a folder path is specified, a log for each yaml file will be stored. If otherwise a filename is provided, all log information will be in one logfile.
Define the test information
For each test fixture (setup, teardown or test_run), users have to define the test information using the decorator define_test_parameters. This decorator gives access to the following parameters:
suite_id
: current test suite identification numbercase_id
: current test case identification number (optional for test suite setup and teardown)aux_list
: list of used auxiliaries
In order to link the architecture requirement to the test, an additional reference can be added into the test_run decorator:
test_ids
: optional requirements linked to the test that need to be defined as follow:
{"Component1": ["Req1", "Req2"], "Component2": ["Req3"]}
In order to run only a subset of tests, an additional reference can be added to the test_run decorator:
tag : [optional] the variant can be defined like:
{"variant": ["variant2", "variant1"], "branch_level": ["daily", "nightly"]}
Both parameters (variant/branch_level), will play the role of filter to fine tune the test collection and at the end ensure the execution of very specific tests subset.
Note
cli tags must be given in pairs. If one key has multiple values seperate them with a comma
pykiso -c configuration_file --variant var1,var2 --branch-level daily,nightly
test case tags |
cli tags |
executed |
---|---|---|
“branch_level”: [“daily”,”nightly”] |
branch_level nightly |
🗸 |
“branch_level”: [“daily”,”nightly”] |
branch_level nightly,daily |
🗸 |
“branch_level”: [“daily”,”nightly”] |
branch_level master |
✗ |
“branch_level”: [“daily”,”nightly”],”variant”:[“var1”] |
branch_level nightly |
🗸 |
“branch_level”: [“daily”,”nightly”],”variant”:[“var1”] |
variant var1 |
🗸 |
“branch_level”: [“daily”,”nightly”],”variant”:[“var1”] |
variant var2 |
✗ |
“branch_level”: [“daily”,”nightly”],”variant”:[“var1”] |
branch_level daily variant var1 |
✗ |
In order to utilise the SetUp/TearDown test-suite feature, users have to define a class inheriting from
BasicTestSuiteSetup
or
BasicTestSuiteTeardown
.
For each of these classes, the following methods test_suite_setUp
or test_suite_tearDown
must be
overridden with the behaviour you want to have.
Note
Note
If a test in SuiteSetup raises an exception, all tests which belong to the same suite_id will be skipped.
Find below a full example for a test suite/case declaration :
"""
Add test suite setup fixture, run once at test suite's beginning.
Test Suite Setup Information:
-> suite_id : set to 1
-> case_id : Parameter case_id is not mandatory for setup.
-> aux_list : used aux1 and aux2 is used
"""
@pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
class SuiteSetup(pykiso.BasicTestSuiteSetup):
def test_suite_setUp():
logging.info("I HAVE RUN THE TEST SUITE SETUP!")
if aux1.not_properly_configured():
aux1.configure()
aux2.configure()
callback_registering()
"""
Add test suite teardown fixture, run once at test suite's end.
Test Suite Teardown Information:
-> suite_id : set to 1
-> case_id : Parameter case_id is not mandatory for setup.
-> aux_list : used aux1 and aux2 is used
"""
@pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
class SuiteTearDown(pykiso.BasicTestSuiteTeardown):
def test_suite_tearDown():
logging.info("I HAVE RUN THE TEST SUITE TEARDOWN!")
callback_unregistering()
"""
Add a test case 1 from test suite 1 using auxiliary 1.
Test Suite Teardown Information:
-> suite_id : set to 1
-> case_id : set to 1
-> aux_list : used aux1 and aux2 is used
-> test_ids: [optional] store the requirements into the report
-> tag: [optional] dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed
"""
@pykiso.define_test_parameters(
suite_id=1,
case_id=1,
aux_list=[aux1, aux2],
test_ids={"Component1": ["Req1", "Req2"]},
tag={"variant": ["variant2", "variant1"], "branch_level": ["daily", "nightly"]},
)
class MyTest(pykiso.BasicTest):
pass
Implementation of Basic Tests
Structure: test-folder/test-suite-1/test_suite_1.py
test_suite_1.py:
"""
I want to run the following tests documented in the following test-specs <TEST_CASE_SPECS>.
"""
import pykiso
from pykiso.auxiliaries import aux1, aux2
"""
Add test suite setup fixture, run once at test suite's beginning.
Parameter case_id is not mandatory for setup.
"""
@pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
class SuiteSetup(pykiso.BasicTestSuiteSetup):
pass
"""
Add test suite teardown fixture, run once at test suite's end.
Parameter case_id is not mandatory for teardown.
"""
@pykiso.define_test_parameters(suite_id=1, aux_list=[aux1, aux2])
class SuiteTearDown(pykiso.BasicTestSuiteTeardown):
pass
"""
Add a test case 1 from test suite 1 using auxiliary 1.
"""
@pykiso.define_test_parameters(suite_id=1, case_id=1, aux_list=[aux1])
class MyTest(pykiso.BasicTest):
pass
"""
Add a test case 2 from test suite 1 using auxiliary 2.
"""
@pykiso.define_test_parameters(suite_id=1, case_id=2, aux_list=[aux2])
class MyTest2(pykiso.BasicTest):
pass
Implementation of Advanced Tests - Auxiliary Interaction
Using the dynamic importing capabilities of the framework we can interact with the auxiliaries directly.
For this test we will assume that we have configured a
CommunicationAuxiliary
and a connector that supports raw messaging.
"""
send a message, receive a response, compare to expected response
"""
import pykiso
from pykiso.auxiliaries import com_aux
@pykiso.define_test_parameters(suite_id=2, case_id=1, aux_list=[com_aux])
class ComTest(pykiso.BasicTest):
STIMULUS = b"stimulus message"
RESPONSE = b"expected reply"
def test_run(self):
com_aux.send_message(STIMULUS)
resp = com_aux.receive_message()
self.assertEqual(resp, RESPONSE)
We can use the configured and instantiated auxiliary com_aux
(imported by it’s alias) in the test directly.
Implementation of Advanced Tests - Custom Setup
If you need to have more complex tests, you can do the following:
BasicTest
is a specific implementation ofunittest.TestCase
therefore it contains 3 steps/methods setUp(), tearDown() and test_run() that can be overwritten.BasicTest
will contain the list of auxiliaries you can use. It will be hold in the attributetest_auxiliary_list
.BasicTest
also contains the following informationtest_section_id
,test_suite_id
,test_case_id
.Import logging or/and message (if needed) to communicate with the **auxiliary**(in that case use RemoteTest instead of BasicTest)
test_suite_2.py:
"""
I want to run the following tests documented in the following test-specs <TEST_CASE_SPECS>.
"""
import pykiso
from pykiso import message
from pykiso.auxiliaries import aux1
@pykiso.define_test_parameters(suite_id=2, case_id=1, aux_list=[aux1])
class MyTest(pykiso.BasicTest):
def setUp(self):
# I loop through all the auxiliaries
for aux in self.test_auxiliary_list:
if aux.name == "aux1": # If I find the auxiliary to which I need to send a special message, I compose the message and send it.
# Compose the message to send with some additional information
tlv = { TEST_REPORT:"Give me something" }
testcase_setup_special_message = message.Message(msg_type=message.MessageType.COMMAND, sub_type=message.MessageCommandType.TEST_CASE_SETUP,
test_section=self.test_section_id, test_suite=self.test_suite_id, test_case=self.test_case_id, tlv_dict=tlv)
# Send the message
aux.run_command(testcase_setup_special_message, blocking=True, timeout_in_s=10)
else: # Do not forget to send a setup message to the other auxiliaries!
# Compose the normal message
testcase_setup_basic_message = message.Message(msg_type=message.MessageType.COMMAND, sub_type=message.MessageCommandType.TEST_CASE_SETUP,
test_section=self.test_section_id, test_suite=self.test_suite_id, test_case=self.test_case_id)
# Send the message
aux.run_command(testcase_setup_basic_message, blocking=True, timeout_in_s=10)
Implementation of Advanced Tests - Test Templates
Because we are python based, you can until some extend, design and implement parts of the framework to fulfil your needs. For example:
test_suite_3.py:
import pykiso
from pykiso import message
from pykiso.auxiliaries import aux1
class MyTestTemplate(pykiso.BasicTest):
def test_run(self):
# Prepare message to send
testcase_run_message = message.Message(msg_type=message.MessageType.COMMAND, sub_type=message.MessageCommandType.TEST_CASE_RUN,
test_section=self.test_section_id, test_suite=self.test_suite_id, test_case=self.test_case_id)
# Send test start through all auxiliaries
for aux in self.test_auxiliary_list:
if aux.run_command(testcase_run_message, blocking=True, timeout_in_s=10) is not True:
self.cleanup_and_skip("{} could not be run!".format(aux))
# Device will reboot, wait for the reboot report
for aux in self.test_auxiliary_list:
if aux.name == "DeviceUnderTest":
report = aux.wait_and_get_report(blocking=True, timeout_in_s=10) # Wait for a report from the DeviceUnderTest
break
# Check if the report for the reboot was received.
report is not None and report.get_message_type() == message.MessageType.REPORT and report.get_message_sub_type() == message.MessageReportType.TEST_PASS:
pass # We can continue
else:
self.cleanup_and_skip("Device failed rebooting")
# Loop until all reports are received
list_of_aux_with_received_reports = [False]*len(self.test_auxiliary_list)
while False in list_of_aux_with_received_reports:
# Loop through all auxiliaries
for i, aux in enumerate(self.test_auxiliary_list):
if list_of_aux_with_received_reports[i] == False:
# Wait for a report
reported_message = aux.wait_and_get_report()
# Check the received message
list_of_aux_with_received_reports[i] = self.evaluate_message(aux, reported_message)
@pykiso.define_test_parameters(suite_id=3, case_id=1, aux_list=[aux1])
class MyTest(MyTestTemplate):
pass
@pykiso.define_test_parameters(suite_id=3, case_id=2, aux_list=[aux1])
class MyTest2(MyTestTemplate):
pass
Implementation of Advanced Tests - Repeat testCases
Decorator: retry mechanism for testCase.
The aim is to cover the 2 following cases:
Unstable test : get the test pass within the {max_try} attempt
Stability test : run {max_try} time the test expecting no error
The retry_test_case comes with the possibility to re-run the setUp and tearDown methods automatically.
- type max_try
int
- param max_try
maximum number of try to get the test pass.
- type rerun_setup
bool
- param rerun_setup
call the “setUp” method of the test.
- type rerun_teardown
bool
- param rerun_teardown
call the “tearDown” method of the test.
- type stability_test
bool
- param stability_test
run {max_try} time the test and raise an exception if an error occurs.
- return
None, a testCase is not supposed to return anything.
- raise Exception
if stability_test, the exception that occurred during the execution; if not stability_test, the exception that occurred at the last try.
test_suite_1.py:
# define an external iterator that can be used for retry_test_case demo
side_effect = cycle([False, False, True])
@pykiso.define_test_parameters()
class MyTest1(pykiso.BasicTest):
"""This test case definition will override the setUp, test_run and tearDown method."""
@pykiso.retry_test_case(max_try=3)
def setUp(self):
"""Hook method from unittest in order to execute code before test case run.
In this case the default setUp method is overridden, allowing us to apply the
retry_test_case's decorator. The syntax super() access to the BasicTest and
we will run the default setUp()
"""
super().setUp()
@pykiso.retry_test_case(max_try=5, rerun_setup=True, rerun_teardown=False)
def test_run(self):
"""In this case the default test_run method is overridden and
instead of calling test_run from BasicTest class the following
code is called.
Here, the test pass at the 3rd attempt out of 5. The setup and
tearDown methods are called for each attempt.
"""
logging.info(
f"--------------- RUN: {self.test_suite_id}, {self.test_case_id} ---------------"
)
self.assertTrue(next(side_effect))
logging.info(f"I HAVE RUN 0.1.1 for variant {self.variant}!")
@pykiso.retry_test_case(max_try=3, stability_test=True)
def tearDown(self):
"""Hook method from unittest in order to execute code after the test case ran.
In this case the default tearDown method is overridden, allowing us to apply the
retry_test_case's decorator. The syntax super() access to the BasicTest and
we will run the default tearDown().
The retry_test_case has stability test activated, so the tearDown method will
be run 3 times.
"""
super().tearDown()
Add Config File
For details see ../getting_started/config_file.
Example:
1auxiliaries:
2 aux1:
3 connectors:
4 com: chan1
5 config: null
6 type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
7 aux2:
8 connectors:
9 com: chan2
10 type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
11 aux3:
12 connectors:
13 com: chan4
14 flash: chan3
15 type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
16connectors:
17 chan1:
18 config: null
19 type: ext_lib/cc_example.py:CCExample
20 chan2:
21 type: ext_lib/cc_example.py:CCExample
22 chan4:
23 type: ext_lib/cc_example.py:CCExample
24 chan3:
25 config: null
26 type: pykiso.lib.connectors.cc_flasher_example:FlasherExample
27test_suite_list:
28- suite_dir: test_suite_1
29 test_filter_pattern: '*.py'
30 test_suite_id: 1
31- suite_dir: test_suite_2
32 test_filter_pattern: '*.py'
33 test_suite_id: 2
34
35requirements:
36 - pykiso : '>=0.10.1'
37 - robotframework : 3.2.2
38 - pyyaml: any
Run the tests
pykiso -c <config_file>
To let the user decide which information they want to see in their logs, new log levels have been defined. When launched normally, only the logs from the tests and the framework errors will be active. The option -v (–verbose) should be used to display the internal logs of the framework:
pykiso -c <config_file> -v
or
pykiso -c <config_file> --verbose
Three internal log levels are available: INTERNAL_INFO, INTERNAL_DEBUG, INTERNAL_WARNING. They will then be activated depending on the value of the–log-level option. Error logs level will always be logged, internal or not.
The summary of the activated logs depending of the value of the –log-level and –verbose options can be found in the following table:
verbose == True |
verbose == False |
|
---|---|---|
log-level == DEBUG |
DEBUG, INTERNAL_DEBUG, INFO, INTERNAL_INFO, WARNING, INTERNAL_WARNING, ERROR |
DEBUG, INFO, WARNING, ERROR |
log-level == INFO |
INFO, INTERNAL_INFO, WARNING, INTERNAL_WARNING, ERROR |
INFO, WARNING, ERROR |
log-level == WARNING |
WARNING, INTERNAL_WARNING, ERROR |
WARNING, ERROR |
log-level == ERROR |
ERROR |
ERROR |