Pykiso Design

Introduction

Integration Test Framework (Pykiso) is a framework that can be used for both white-box and black-box testing as well as in the integration and system testing.

Quality Goals

The framework tries to achieve the following quality goals:

Quality Goal (with prio)	Scenarios
Portability	The framework shall run on linux, windows, macOS
	The framework shall run on a raspberryPI or a regular laptop
Modularity	The framework shall allow me to implement complex logic and to run it over any communication port
	The framework shall allow me to add any communication port
	The framework shall allow me to use private modules within my tests if it respects its APIs
	The framework shall allow me to define my own test approach
Correctness	The framework shall verify that its inputs (test-setup) are correct before performing any test
	The framework shall execute the provided tests always in the same order
Usability	The framework shall feel familiar for embedded developers
	The framework shall feel familiar for system tester
	The framework shall generate test reports that are human and machine readable
Performance (new)	The framework shall use only the right/reasonable amount of resources to run (real-time timings)

Design Overview

Figure 1: Integration Test Framework Context

The pykiso Testing Framework is built in a modular and configurable way with abstractions both for entities (e.g. a handler for the device under test) and communication (e.g. UART or TCP/IP).

The tests leverage the python unittest-Framework which has a similar flavor as many available major unit testing frameworks and thus comes with an ecosystem of tools and utilities.

Test Coordinator

The test-coordinator is the central module setting up and running the tests. Based on a configuration file (in YAML), it does the following:

instantiate the selected connectors
instantiate the selected auxiliaries
provide the auxiliaries with the matching connectors
generate the list of tests to perform
provide the testcases with the auxiliaries they need
verify if the tests can be performed
for remote tests (see Remote Test) flash and run and synchronize the tests on the auxiliaries
gather the reports and publish the results

Auxiliary

The auxiliary provides to the test-coordinator an interface to interact with the physical or digital auxiliary target. It is composed by 2 blocks:

instance creation / deletion
connectors to facilitate interaction and communication with the device (e.g. messaging with UART)

For example auxiliaries like the one interacting with cloud services, we may have:

A communication channel (cchannel) like REST

Create an Auxiliary

Detailed information can be found here How to create an auxiliary.

Connector

Communication Channel

The Communication Channel - also known as cchannel - is the medium to communicate with auxiliary target. Example include UART, UDP, USB, REST,… The communication protocol itself can be auxiliary specific.

Create a Connector

Detailed information can be found here How to create a connector.

Dynamic Import Linking

The pykiso framework was developed with modularity and reusability in mind. To avoid close coupling between testcases and auxiliaries as well as between auxiliaries and connectors, the linking between those components is defined in a config file (see How to make the most of the config file) and performed by the TestCoordinator.

Different instances of connectors and auxiliaries are given aliases which identify them within the test session.

Let’s say we have this (abridged) config file:

connectors:
  my_chan:           # Alias of the connector
    type: ...
auxiliaries:
  my_aux:            # Alias of the auxiliary
    connectors:
        com: my_chan # Reference to the connector
    type: ...

The auxiliary my_aux will automatically be initialised with my_chan as its com channel.

When writing your testcases, the auxiliary will then be available under its defined alias.

from pykiso.auxiliaries import my_aux

The pykiso.auxiliaries is a magic package that only exists in the pykiso package after the TestCoordinator has processed the config file. It will include all instances of the defined auxiliares, available at their defined alias.

Usage

Please see How to make the most of the config file to have a deep-dive on how the pykiso configuration work.

Please see How to make the most of the tests to have a deep-dive on how pykiso tests work.