API Documentation

Test Cases

Generic Test

module

test_case

synopsis

Basic extensible implementation of a TestCase.

Note

TODO later on will inherit from a metaclass to get the id parameters

class pykiso.test_coordinator.test_case.BasicTest(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)[source]

Base for test-cases.

Initialize generic test-case.

Parameters

  • test_suite_id (int) – test suite identification number

  • test_case_id (int) – test case identification number

  • aux_list (Optional[List[AuxiliaryInterface]]) – list of used auxiliaries

  • setup_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during setup execution

  • run_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during test_run execution

  • teardown_timeout (Optional[int]) – the maximum time (in seconds) used to wait for a report during teardown execution

  • test_ids (Optional[dict]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}

  • tag (Optional[Dict[str, List[str]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed

cleanup_and_skip(aux, info_to_print)[source]

Cleanup auxiliary and log reasons.

Parameters

  • aux (AuxiliaryInterface) – corresponding auxiliary to abort

  • info_to_print (str) – A message you want to print while cleaning up the test

Return type

None

setUp()[source]

Hook method for constructing the test fixture.

Return type

None

classmethod setUpClass()[source]

A class method called before tests in an individual class are run.

This implementation is only mandatory to enable logging in junit report. The logging configuration has to be call inside test runner run, otherwise stdout is never caught.

Return type

None

tearDown()[source]

Hook method for deconstructing the test fixture after testing it.

Return type

None

test_run()[source]

Hook method from unittest in order to execute test case.

Return type

None

pykiso.test_coordinator.test_case.define_test_parameters(suite_id=0, case_id=0, aux_list=None, setup_timeout=None, run_timeout=None, teardown_timeout=None, test_ids=None, tag=None)[source]

Decorator to fill out test parameters of the BasicTest automatically.

pykiso.test_coordinator.test_case.retry_test_case(max_try=2, rerun_setup=False, rerun_teardown=False, stability_test=False)[source]

Decorator: retry mechanism for testCase.

The aim is to cover the 2 following cases:

  • Unstable test : get the test pass within the {max_try} attempt

  • Stability test : run {max_try} time the test expecting no error

The retry_test_case comes with the possibility to re-run the setUp and tearDown methods automatically.

Parameters

  • max_try (int) – maximum number of try to get the test pass.

  • rerun_setup (bool) – call the “setUp” method of the test.

  • rerun_teardown (bool) – call the “tearDown” method of the test.

  • stability_test (bool) – run {max_try} time the test and raise an exception if an error occurs.

Returns

None, a testCase is not supposed to return anything.

Raises

Exception – if stability_test, the exception that occurred during the execution; if not stability_test, the exception that occurred at the last try.

Connectors

pykiso comes with some ready to use implementations of different connectors.

Auxiliaries

Message Protocol

pykiso Control Message Protocol

module

message

synopsis

Message that will be send though the different agents

class pykiso.message.Message(msg_type=0, sub_type=0, error_code=0, test_suite=0, test_case=0, tlv_dict=None)[source]

A message who fit testApp protocol.

The created message is a tlv style message with the following format: TYPE: msg_type | message_token | sub_type | errorCode |

Create a generic message.

Parameters

  • msg_type (MessageType) – Message type

  • sub_type (Message<MessageType>Type) – Message sub-type

  • error_code (integer) – Error value

  • test_suite (integer) – Suite value

  • test_case (integer) – Test value

  • tlv_dict (dict) – Dictionary containing tlvs elements in the form {‘type’:’value’, …}

check_if_ack_message_is_matching(ack_message)[source]

Check if the ack message was for this sent message.

Parameters

ack_message (Message) – received acknowledge message

Return type

bool

Returns

True if message type and token are valid otherwise False

generate_ack_message(ack_type)[source]

Generate acknowledgement to send out.

Parameters

ack_type (int) – ack or nack

Return type

Optional[Message]

Returns

filled acknowledge message otherwise None

classmethod get_crc(serialized_msg, crc_byte_size=2)[source]

Get the CRC checksum for a bytes message.

Parameters

  • serialized_msg (bytes) – message used for the crc calculation

  • crc_byte_size (int) – number of bytes dedicated for the crc

Return type

int

Returns

CRC checksum

get_message_sub_type()[source]

Return actual message subtype.

Return type

int

get_message_tlv_dict()[source]

Return actual message type/length/value dictionary.

Return type

dict

get_message_token()[source]

Return actual message token.

Return type

int

get_message_type()[source]

Return actual message type.

Return type

Union[int, MessageType]

classmethod parse_packet(raw_packet)[source]

Factory function to create a Message object from raw data.

Parameters

raw_packet (bytes) – array of a received message

Return type

Message

Returns

itself

serialize()[source]

Serialize message into raw packet.

Format: | msg_type (1b) | msg_token (1b) | sub_type (1b) | error_code (1b) |
test_section (1b) | test_suite (1b) | test_case (1b) | payload_length (1b) |
tlv_type (1b) | tlv_size (1b) | … | crc_checksum (2b)
Return type

bytes

Returns

bytes representing the Message object

class pykiso.message.MessageAckType(value)[source]

List of possible received messages.

class pykiso.message.MessageCommandType(value)[source]

List of commands allowed.

class pykiso.message.MessageLogType(value)[source]

List of possible received log messages.

class pykiso.message.MessageReportType(value)[source]

List of possible received messages.

class pykiso.message.MessageType(value)[source]

List of messages allowed.

class pykiso.message.TlvKnownTags(value)[source]

List of known / supported tags.

Import Magic

Auxiliary Interface Definition

module

dynamic_loader

synopsis

Import magic that enables aliased auxiliary loading in TestCases

class pykiso.test_setup.dynamic_loader.DynamicImportLinker[source]

Public Interface of Import Magic.

initialises the Loaders, Finders and Caches, implements interfaces to install the magic and register the auxiliaries and connectors.

Initialize attributes.

install()[source]

Install the import hooks with the system.

provide_auxiliary(name, module, aux_cons=None, **config_params)[source]

Provide a auxiliary.

Parameters

  • name (str) – the auxiliary alias

  • module (str) – either ‘python-file-path:Class’ or ‘module:Class’ of the class we want to provide

  • aux_cons – list of connectors this auxiliary has

provide_connector(name, module, **config_params)[source]

Provide a connector.

Parameters

  • name (str) – the connector alias

  • module (str) – either ‘python-file-path:Class’ or ‘module:Class’ of the class we want to provide

uninstall()[source]

Deregister the import hooks, close all running threads, delete all instances.

Config Registry

module

config_registry

synopsis

register auxiliaries and connectors to provide them for import.

class pykiso.test_setup.config_registry.ConfigRegistry[source]

Register auxiliaries with connectors to provide systemwide import statements.

classmethod delete_aux_con()[source]

deregister the import hooks, close all running threads, delete all instances.

Return type

None

classmethod get_all_auxes()[source]

Return all auxiliaires instances and alias

Return type

dict

Returns

dictionary with alias as keys and instances as values

classmethod get_aux_config(name)[source]

Return the registered auxiliary configuration based on his name.

Parameters

name (str) – auxiliary alias

Return type

dict

Returns

auxiliary’s configuration (yaml content)

classmethod get_auxes_alias()[source]

return all created auxiliaries alias.

Return type

list

Returns

list containing all auxiliaries alias

classmethod get_auxes_by_type(aux_type)[source]

Return all auxiliaries who match a specific type.

Parameters

aux_type (Any) – auxiliary class type (DUTAuxiliary, CommunicationAuxiliary…)

Return type

dict

Returns

dictionary with alias as keys and instances as values

classmethod register_aux_con(config)[source]

Create import hooks. Register auxiliaries and connectors.

Parameters

config (dict) – dictionary containing yaml configuration content

Return type

None

Test Suites

Test Suite

module

test_suite

synopsis

Create a generic test-suite based on the connected modules

class pykiso.test_coordinator.test_suite.BaseTestSuite(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)[source]

Initialize generic test-case.

Parameters

  • test_suite_id (int) – test suite identification number

  • test_case_id (int) – test case identification number

  • aux_list (Optional[List[AuxiliaryInterface]]) – list of used auxiliaries

  • setup_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during setup execution

  • run_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during test_run execution

  • teardown_timeout (Optional[int]) – the maximum time (in seconds) used to wait for a report during teardown execution

  • test_ids (Optional[dict]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2 [“Req3”]}

  • tag (Optional[Dict[str, List[str]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed

cleanup_and_skip(aux, info_to_print)[source]

Cleanup auxiliary and log reasons.

Parameters

  • aux (AuxiliaryInterface) – corresponding auxiliary to abort

  • info_to_print (str) – A message you want to print while cleaning up the test

class pykiso.test_coordinator.test_suite.BasicTestSuite(modules_to_add_dir, test_filter_pattern, test_suite_id, args, kwargs)[source]

Inherit from the unittest framework test-suite but build it for our integration tests.

Initialize our custom unittest-test-suite.

Note

  1. Will Load from the given path the integration test modules under test

  2. Sort the given test case list by test suite/case id

  3. Place Test suite setup and teardown respectively at top and bottom of test case list

  4. Add sorted test case list to test suite

class pykiso.test_coordinator.test_suite.BasicTestSuiteSetup(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)[source]

Inherit from unittest testCase and represent setup fixture.

Initialize generic test-case.

Parameters

  • test_suite_id (int) – test suite identification number

  • test_case_id (int) – test case identification number

  • aux_list (Optional[List[AuxiliaryInterface]]) – list of used auxiliaries

  • setup_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during setup execution

  • run_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during test_run execution

  • teardown_timeout (Optional[int]) – the maximum time (in seconds) used to wait for a report during teardown execution

  • test_ids (Optional[dict]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}

  • tag (Optional[Dict[str, List[str]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed

test_suite_setUp()[source]

Test method for constructing the actual test suite.

class pykiso.test_coordinator.test_suite.BasicTestSuiteTeardown(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)[source]

Inherit from unittest testCase and represent teardown fixture.

Initialize generic test-case.

Parameters

  • test_suite_id (int) – test suite identification number

  • test_case_id (int) – test case identification number

  • aux_list (Optional[List[AuxiliaryInterface]]) – list of used auxiliaries

  • setup_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during setup execution

  • run_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during test_run execution

  • teardown_timeout (Optional[int]) – the maximum time (in seconds) used to wait for a report during teardown execution

  • test_ids (Optional[dict]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}

  • tag (Optional[Dict[str, List[str]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed

test_suite_tearDown()[source]

Test method for deconstructing the actual test suite after testing it.

Test Execution

Test Execution

module

test_execution

synopsis

Execute a test environment based on the supplied configuration.

Note

  1. Glob a list of test-suite folders

  2. Generate a list of test-suites with a list of test-cases

  3. Loop per suite

  4. Gather result

class pykiso.test_coordinator.test_execution.ExitCode(value)[source]

List of possible exit codes

pykiso.test_coordinator.test_execution.apply_variant_filter(all_tests_to_run, variants, branch_levels)[source]

Filter the test cases based on the variant string.

Parameters

  • all_tests_to_run (dict) – a dict containing all testsuites and testcases

  • variants (tuple) – encapsulate user’s variant choices

  • branch_levels (tuple) – encapsulate user’s branch level choices

Return type

None

pykiso.test_coordinator.test_execution.create_test_suite(test_suite_dict)[source]

create a test suite based on the config dict

Parameters

test_suite_dict (Dict) – dict created from config with keys ‘suite_dir’, ‘test_filter_pattern’, ‘test_suite_id’

Return type

BasicTestSuite

pykiso.test_coordinator.test_execution.execute(config, report_type='text', variants=None, branch_levels=None, pattern_inject=None, failfast=False)[source]

create test environment base on config

Parameters

  • config (Dict) – dict from converted YAML config file

  • report_type (str) – str to set the type of report wanted, i.e. test or junit

  • variants (Optional[tuple]) – encapsulate user’s variant choices

  • branch_levels (Optional[tuple]) – encapsulate user’s branch level choices

  • pattern_inject (Optional[str]) – optional pattern that will override test_filter_pattern for all suites. Used in test development to run specific tests.

  • failfast (bool) – Stop the test run on the first error or failure

Return type

int

Returns

exit code corresponding to the actual tets exeuciton run state(tests failed, unexpected exception…)

pykiso.test_coordinator.test_execution.failure_and_error_handling(result)[source]

provide necessary information to Jenkins if an error occur during tests execution

Parameters

result (TestResult) – encapsulate all test results from the current run

Return type

int

Returns

an ExitCode object

Test-Message Handling

Handle common communication with device under test

By default, the integration test framework handles internal messaging and control flow using a message format defined in pykiso.Message. pykiso.test_message_handler defines the default messaging protocol from a behavioral point of view.

The general procedure is described in handle_basic_interaction context manager, but specific _MsgHandler_ classes are provided with TestCaseMsgHandler and TestSuiteMsgHandler to provide shorthands for the specialised communication from pykiso.test_case.BasicTest and pykiso.test_suite.BasicTestSuite.

module

test_message_handler

synopsis

default communication between TestManagement and DUT.

pykiso.test_coordinator.test_message_handler.handle_basic_interaction(test_entity, cmd_sub_type, timeout_cmd, timeout_resp)[source]

Handle default communication mechanism between test manager and device under test as follow:

TM |   COMMAND ---------->        | DUT
TM |           <---------- ACK    | DUT
TM |           <---------- LOG    | DUT
TM |   ACK     ---------->        | DUT
...
TM |           <---------- LOG    | DUT
TM |   ACK     ---------->        | DUT
TM |           <---------- REPORT | DUT
TM |   ACK     ---------->        | DUT

This behaviour is implemented here.

Logs can be sent to TM while waiting for report.

Parameters

  • test_entity (Callable) – test instance in use (BaseTestSuite, BasicTest,…)

  • cmd_sub_type (MessageCommandType) – message command sub-type (Test case run, setup,….)

  • timeout_cmd (int) – timeout in seconds for auxiliary run_command

  • timeout_resp (int) – timeout in seconds for auxiliary wait_and_get_report

Return type

List[report_analysis]

Returns

tuple containing current auxiliary, reported message, logging method to use, and pre-defined log message.

class pykiso.test_coordinator.test_message_handler.report_analysis(current_auxiliary, report_message, logging_method, log_message)

Create new instance of report_analysis(current_auxiliary, report_message, logging_method, log_message)

property current_auxiliary

Alias for field number 0

property log_message

Alias for field number 3

property logging_method

Alias for field number 2

property report_message

Alias for field number 1

pykiso.test_coordinator.test_message_handler.test_app_interaction(message_type, timeout_cmd=5)[source]

Handle test app basic interaction depending on the decorated method.

Parameters

  • message_type (MessageCommandType) – message command sub-type (test case/suite run, setup, teardown….)

  • timeout_cmd (int) – timeout in seconds for auxiliary run_command

Return type

Callable

Returns

inner decorator function

test xml result

test_xml_result

module

test_xml_result

synopsis

overwrite xmlrunner.result to be able to add additional data into the xml report.

class pykiso.test_coordinator.test_xml_result.TestInfo(test_result, test_method, outcome=0, err=None, subTest=None, filename=None, lineno=None, doc=None)[source]

This class keeps useful information about the execution of a test method. Used by XmlTestResult

Initialize the TestInfo class and append additional tag

that have to be stored for each test

Parameters

  • test_result (_XMLTestResult) – test result class

  • test_method – test method (dynamically created eg: test_case.MyTest2-1-2)

  • outcome (int) – result of the test (SUCCESS, FAILURE, ERROR, SKIP)

  • err – error cached during test

  • subTest – optional, refer the test id and the test description

  • filename (Optional[str]) – name of the file

  • lineno (Optional[bool]) – store the test line number

  • doc (Optional[str]) – additional documentation to store

class pykiso.test_coordinator.test_xml_result.XmlTestResult(stream=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>, descriptions=True, verbosity=1, elapsed_times=True, properties=None, infoclass=<class 'pykiso.test_coordinator.test_xml_result.TestInfo'>)[source]

Test result class that can express test results in a XML report. Used by XMLTestRunner

Initialize the _XMLTestResult class.

Parameters

  • stream (TextIOWrapper) – buffered text interface to a buffered raw stream

  • descriptions (bool) – include description of the test

  • verbosity (int) – print output into the console

  • elapsed_times (bool) – include the time spend on the test

  • properties – junit testsuite properties

  • infoclass (_TestInfo) – class containing the test information

report_testcase(xml_testsuite, xml_document)

Appends a testcase section to the XML document.