User Guide

Requirements

  • Python 3.6+

  • pip/pipenv (used to get the rest of the requirements)

Install

git clone https://dev-bosch.com/bitbucket/scm/pea/integration-test-framework.git
cd integration-test-framework
pip install .

Pipenv is more appropriate for developers as it automatically creates virtual environments.

git clone https://dev-bosch.com/bitbucket/scm/pea/integration-test-framework.git
cd integration-test-framework
pipenv install --dev
pipenv shell

Usage

Once installed the application is bound to pykiso, it can be called with the following arguments:

$ pykiso --help
Usage: pykiso [OPTIONS] [PATTERN]

  Embedded Integration Test Framework - CLI Entry Point.

  PATTERN: overwrite the test filter pattern from the YAML file (optional)

Options:
  -c, --test-configuration-file FILE
                                  path to the test configuration file (in YAML
                                  format)  [required]
  -l, --log-path PATH             path to log-file or folder. If not set will
                                  log to STDOUT
  --log-level [DEBUG|INFO|WARNING|ERROR]
                                  set the verbosity of the logging
  --junit                         enables the generation of a junit report
  --text                          default, test results are only displayed in
                                  the console
  --variant TEXT                  allow the user to execute a subset of tests
                                  based on variants
  --branch-level TEXT             allow the user to execute a subset of tests
                                  based on branch levels
  --version                       Show the version and exit.
  -h, --help                      Show this message and exit.

Suitable config files are available in the test-examples folder.

Demo using example config

invoke run

Running the Tests

invoke test

or

pytest

Test Configuration File

The test configuration files are written in YAML.

Let’s use an example to understand the structure.

 1auxiliaries:
 2  aux1:
 3    connectors:
 4        com: chan1
 5    config: null
 6    type: ext_lib/example_test_auxiliary.py:ExampleAuxiliary
 7  aux2:
 8    connectors:
 9        com:   chan2
10        flash: chan3
11    type: pykiso.lib.auxiliaries.example_test_auxiliary:ExampleAuxiliary
12  aux3:
13    connectors:
14        com:   chan4
15    type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
16connectors:
17  chan1:
18    config: null
19    type: ext_lib/cc_example.py:CCExample
20  chan2:
21    type: ext_lib/cc_example.py:CCExample
22  chan4:
23    type: ext_lib/cc_example.py:CCExample
24  chan3:
25    config:
26        configKey: "config value"
27    type: ext_lib/cc_example.py:CCExample
28test_suite_list:
29- suite_dir: test_suite_1
30  test_filter_pattern: '*.py'
31  test_suite_id: 1
32- suite_dir: test_suite_2
33  test_filter_pattern: '*.py'
34  test_suite_id: 2
35
36requirements:
37  - pykiso : '>=0.10.1'
38  - robotframework : 3.2.2
39  - pyyaml: any

Connectors

The connector definition is a named list (dictionary in python) of key-value pairs, namely config and type.

  aux3:
    connectors:
        com:   chan4
    type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
connectors:
  chan1:
    config: null
    type: ext_lib/cc_example.py:CCExample
  chan2:
    type: ext_lib/cc_example.py:CCExample

The channel alias will identify this configuration for the auxiliaries.

The config can be omitted, null, or any number of key-value pairs.

The type consists of a module location and a class name that is expected to be found in the module. The location can be a path to a python file (Win/Linux, relative/absolute) or a python module on the python path (e.h. pykiso.lib.connectors.cc_uart).

<chan>:                      # channel alias
    config:                  # channel config, optional
        <key>: <value>       # collection of key-value pairs, e.g. "port: 80"
    type: <module:Class>     # location of the python class that represents this channel

Auxiliaries

The auxiliary definition is a named list (dictionary in python) of key-value pairs, namely config, connectors and type.

auxiliaries:
  aux1:
    connectors:
        com: chan1
    config: null
    type: ext_lib/example_test_auxiliary.py:ExampleAuxiliary
  aux2:
    connectors:
        com:   chan2
        flash: chan3
    type: pykiso.lib.auxiliaries.example_test_auxiliary:ExampleAuxiliary

The auxiliary alias will identify this configuration for the testcases. When running the tests the testcases can import an auxiliary instance defined here using

from pykiso.auxiliaries import <alias>

The connectors can be omitted, null, or any number of role-connector pairs. The roles are defined in the auxiliary implementation, usual examples are com and flash. The channel aliases are the ones you defined in the connectors section above.

The config can be omitted, null, or any number of key-value pairs.

The type consists of a module location and a class name that is expected to be found in the module. The location can be a path to a python file (Win/Linux, relative/absolute) or a python module on the python path (e.h. pykiso.lib.auxiliaries.communication_auxiliary).

<aux>:                           # aux alias
    connectors:                  # list of connectors this auxiliary needs
        <role>: <channel-alias>  # <role> has to be the name defined in the Auxiliary class,
                                 #  <channel-alias> is the alias defined above
    config:                      # channel config, optional
        <key>: <value>           # collection of key-value pairs, e.g. "port: 80"
    type: <module:Class>         # location of the python class that represents this auxiliary

Test Suites

The test suite definition is a list of key-value pairs.

  chan4:
    type: ext_lib/cc_example.py:CCExample
  chan3:
    config:
        configKey: "config value"
    type: ext_lib/cc_example.py:CCExample
test_suite_list:
- suite_dir: test_suite_1
  test_filter_pattern: '*.py'
  test_suite_id: 1
- suite_dir: test_suite_2
  test_filter_pattern: '*.py'
  test_suite_id: 2

Each test suite consists of a test_suite_id, a suite_dir and a test_filter_pattern.

For fast test development, the test_filter_pattern can be overwritten from the command line in order to e.g. execute a single test file inside the suite_dir using the CLI argument PATTERN:

..code:: bash

pykiso -c dummy.yaml test_suite_1.py

Requirements specification

[optional] - Any package specified will be checked.

Use cases:

  • A new feature is introduced and used in the test

  • Breaking change introduced with a new release

  • Specific package used in a test that does not belong to pykiso

#__________________________ Requirements section ________________________
# FEATURE: Check the environment before running the tests
#
# The version can be:
#   - specified alone (minimum version accepted)
#   - conditioned using <, <=, >, >=, == or !=
#   - no specified using 'any' (accept any version)
#
# /!\: If the check fail, the tests will not start and the mismatch
#      displayed
#_______________________________________________________________________
requirements:
  - pykiso : '>=0.10.1'
  - robotframework : 3.2.2
  - pyyaml: any

Real-World Configuration File

 1auxiliaries:
 2  DUT:
 3    connectors:
 4        com: uart
 5    config: null
 6    type: pykiso.lib.auxiliaries.example_test_auxiliary:ExampleAuxiliary
 7connectors:
 8  uart:
 9    config:
10      serialPort: COM3
11    type: pykiso.lib.connectors.cc_uart:CCUart
12test_suite_list:
13- suite_dir: test_suite_1
14  test_filter_pattern: '*.py'
15  test_suite_id: 1

Activation of specific loggers

By default, every logger that does not belong to the pykiso package or that is not an auxiliary logger will see its level set to WARNING even if you have in the command line pykiso –log-level DEBUG. This aims to reduce redundant logs from additional modules during the test execution. For keeping specific loggers to the set log-level, it is possible to set the activate_log parameter in the auxiliary config. The following example activates the jlink logger from the pylink package, imported in cc_rtt_segger.py:

auxiliaries:
  aux1:
    connectors:
      com: rtt_channel
    config:
      activate_log:
      # only specifying pylink will include child loggers
      - pylink.jlink
      - my_pkg
    type: pykiso.lib.auxiliaries.dut_auxiliary:DUTAuxiliary
connectors:
  rtt_channel:
    config: null
    type: pykiso.lib.connectors.cc_rtt_segger:CCRttSegger

Based on this example, by specifying my_pkg, all child loggers will also be set to the set log-level.

Note

If e.g. only the logger my_pkg.module_1 should be set to the level, it should be entered as such.

Ability to use environment variables

It is possible to replace any value by an environment variable in the YAML files. When using environment variables, the following format should be respected: ENV{my-env-var}. In the following example, an environment variable called TEST_SUITE_1 contains the path to the test suite 1 directory.

test_suite_list:
- suite_dir: ENV{TEST_SUITE_1}
  test_filter_pattern: '*.py'
  test_suite_id: 1

Specify files and folders

To specify files and folders you can use absolute or relative paths. Relative paths are always given relative to the location of the yaml file.

Relative path or file locations must always start with “./”

example_config:
    rel_script_path: './script_folder/my_awesome_script.py'
    abs_script_path_win: 'C:/script_folder/my_awesome_script.py'
    abs_script_path_unix: '/home/usr/script_folder/my_awesome_script.py'

Make a proxy auxiliary trace

Proxy auxiliary is capable of creating a trace file, where all received messages at connector level are written. This feature is useful when proxy auxiliary is associated with a connector who doesn’t have any trace capability (in contrast to cc_pcan_can or cc_rtt_segger for example).

Everything is handled at configuration level and especially at yaml file :

proxy_aux:
  connectors:
    # communication channel alias
    com: <channel-alias>
  config:
    # Auxiliaries alias list bound to proxy auxiliary
    aux_list : [<aux alias 1>, <aux alias 2>, <aux alias 3>]
    # activate trace at proxy level, sniff everything received at
    # connector level and write it in .log file.
    activate_trace : True
    # by default the trace is placed where pykiso is launched
    # otherwise user should specify his own path
    # (absolute and relative)
    trace_dir: ./suite_proxy
    # by default the trace file's name is :
    # YY-MM-DD_hh-mm-ss_proxy_logging.log
    # otherwise user should specify his own name
    trace_name: can_trace
  type: pykiso.lib.auxiliaries.proxy_auxiliary:ProxyAuxiliary

Delay an auxiliary start-up

All threaded auxiliaries are capable to delay their start-up (not starting at import level). This means, from user point of view, it’s possible to start it on demand and especially where it’s really needed.

Warning

in a proxy set-up be sure to always start the proxy auxiliary last otherwise an error will occurred due to proxy auxiliary specific import rules

In order to achieved that, a parameter was added at the auxiliary configuration level.

auxiliaries:
  proxy_aux:
    connectors:
        com: can_channel
    config:
      aux_list : [aux1, aux2]
      activate_trace : True
      trace_dir: ./suite_proxy
      trace_name: can_trace
      # if False create the auxiliary instance but don't start it, an
      # additional call of start method has to be performed.
      # By default, auto_start flag is set to True and "normal" ITF aux
      # creation mechanism is used.
      auto_start: False
    type: pykiso.lib.auxiliaries.proxy_auxiliary:ProxyAuxiliary
  aux1:
    connectors:
        com: proxy_com1
    config:
      auto_start: False
    type: pykiso.lib.auxiliaries.communication_auxiliary:CommunicationAuxiliary
  aux2:
    connectors:
        com: proxy_com2
    config:
      auto_start: False
    type: pykiso.lib.auxiliaries.communication_auxiliary:CommunicationAuxiliary

In user’s script simply call the related auxilairy start method:


    def setUp(self):
        """If a fixture is not use just override it like below."""
        # start auxiliary one and two because I need it
        aux1.start()
        aux2.start()
        # start the proxy auxiliary in order to open the connector