API Documentation

Test Cases

Generic Test

module

test_case

synopsis

Basic extensible implementation of a TestCase, and of a Remote

TestCase for Message Protocol / TestApp usage.

Note

TODO later on will inherit from a metaclass to get the id parameters

class pykiso.test_coordinator.test_case.BasicTest(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)

Base for test-cases.

Initialize generic test-case.

Parameters

• test_suite_id (int) – test suite identification number

• test_case_id (int) – test case identification number

• aux_list (Optional[List[AuxiliaryInterface]]) – list of used auxiliaries

• setup_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during setup execution

• run_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during test_run execution

• teardown_timeout (Optional[int]) – the maximum time (in seconds) used to wait for a report during teardown execution

• test_ids (Optional[dict]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}

• tag (Optional[Dict[str, List[str]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed

cleanup_and_skip(aux, info_to_print)

Cleanup auxiliary and log reasons.

Parameters

• aux (AuxiliaryInterface) – corresponding auxiliary to abort

• info_to_print (str) – A message you want to print while cleaning up the test

Return type

None

setUp()

Startup hook method to execute code before each test method.

Return type

None

classmethod setUpClass()

A class method called before tests in an individual class are run.

This implementation is only mandatory to enable logging in junit report. The logging configuration has to be call inside test runner run, otherwise stdout is never caught.

Return type

None

tearDown()

Closure hook method to execute code after each test method.

Return type

None

class pykiso.test_coordinator.test_case.RemoteTest(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)

Base test-cases for Message Protocol / TestApp usage.

Initialize TestApp test-case.

Parameters

• test_suite_id (int) – test suite identification number

• test_case_id (int) – test case identification number

• aux_list (Optional[List[AuxiliaryInterface]]) – list of used auxiliaries

• setup_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during setup execution

• run_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during test_run execution

• teardown_timeout (Optional[int]) – the maximum time (in seconds) used to wait for a report during teardown execution

• test_ids (Optional[dict]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}

• tag (Optional[Dict[str, List[str]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed

setUp()

Startup hook method to execute code before each test method.

Return type

None

tearDown()

Closure hook method to execute code after each test method.

Return type

None

test_run()

Hook method from unittest in order to execute test case.

Return type

None

pykiso.test_coordinator.test_case.define_test_parameters(suite_id=0, case_id=0, aux_list=None, setup_timeout=None, run_timeout=None, teardown_timeout=None, test_ids=None, tag=None)

Decorator to fill out test parameters of the BasicTest and RemoteTest automatically.

pykiso.test_coordinator.test_case.retry_test_case(max_try=2, rerun_setup=False, rerun_teardown=False, stability_test=False)

Decorator: retry mechanism for testCase.

The aim is to cover the 2 following cases:

• Unstable test : get the test pass within the {max_try} attempt

• Stability test : run {max_try} time the test expecting no error

The retry_test_case comes with the possibility to re-run the setUp and tearDown methods automatically.

Parameters

• max_try (int) – maximum number of try to get the test pass.

• rerun_setup (bool) – call the “setUp” method of the test.

• rerun_teardown (bool) – call the “tearDown” method of the test.

• stability_test (bool) – run {max_try} time the test and raise an exception if an error occurs.

Returns

None, a testCase is not supposed to return anything.

Raises

Exception – if stability_test, the exception that occurred during the execution; if not stability_test, the exception that occurred at the last try.

Connectors

pykiso comes with some ready to use implementations of different connectors.

• Included Connectors
  • 1. Connector Interface
  • 2. cc_example
  • 3. cc_fdx_lauterbach
  • 4. cc_mp_proxy
  • 5. cc_pcan_can
  • 6. cc_proxy
  • 7. cc_raw_loopback
  • 8. cc_rtt_segger
  • 9. cc_socket_can
  • 10. cc_tcp_ip
  • 11. cc_uart
  • 12. cc_udp
  • 13. cc_udp_server
  • 14. cc_usb
  • 15. cc_vector_can
  • 16. cc_visa
• Flashers
  • 1. flash_jlink
  • 2. flash_lauterbach
  • 3. cc_flasher_example

Auxiliaries

• Auxiliary interfaces
  • auxiliary
  • mp_auxiliary
  • simple_auxiliary
  • thread_auxiliary
  • double threaded auxiliary
• Included Auxiliaries
  • 1. acroname_auxiliary
  • 2. communication_auxiliary
  • 3. dut_auxiliary
  • 4. instrument_control_auxiliary
  • 5. mp_proxy_auxiliary
  • 6. proxy_auxiliary
  • 7. record_auxiliary
  • 8. simulated_auxiliary
  • 9. UDS Auxiliary
  • 10. UDS Server Auxiliary

Message Protocol

pykiso Control Message Protocol

module

message

synopsis

Message that will be send though the different agents

class pykiso.message.Message(msg_type=0, sub_type=0, error_code=0, test_suite=0, test_case=0, tlv_dict=None)

A message who fit testApp protocol.

The created message is a tlv style message with the following format: TYPE: msg_type | message_token | sub_type | errorCode |

Create a generic message.

Parameters

• msg_type (MessageType) – Message type

• sub_type (Message<MessageType>Type) – Message sub-type

• error_code (integer) – Error value

• test_suite (integer) – Suite value

• test_case (integer) – Test value

• tlv_dict (dict) – Dictionary containing tlvs elements in the form {‘type’:’value’, …}

check_if_ack_message_is_matching(ack_message)

Check if the ack message was for this sent message.

Parameters

ack_message (Message) – received acknowledge message

Return type

bool

Returns

True if message type and token are valid otherwise False

generate_ack_message(ack_type)

Generate acknowledgement to send out.

Parameters

ack_type (int) – ack or nack

Return type

Optional[Message]

Returns

filled acknowledge message otherwise None

classmethod get_crc(serialized_msg, crc_byte_size=2)

Get the CRC checksum for a bytes message.

Parameters

• serialized_msg (bytes) – message used for the crc calculation

• crc_byte_size (int) – number of bytes dedicated for the crc

Return type

int

Returns

CRC checksum

get_message_sub_type()

Return actual message subtype.

Return type

int

get_message_tlv_dict()

Return actual message type/length/value dictionary.

Return type

dict

get_message_token()

Return actual message token.

Return type

int

get_message_type()

Return actual message type.

Return type

Union[int, MessageType]

classmethod parse_packet(raw_packet)

Factory function to create a Message object from raw data.

Parameters

raw_packet (bytes) – array of a received message

Return type

Message

Returns

itself

serialize()

Serialize message into raw packet.

Format: | msg_type (1b) | msg_token (1b) | sub_type (1b) | error_code (1b) |

test_section (1b) | test_suite (1b) | test_case (1b) | payload_length (1b) |

tlv_type (1b) | tlv_size (1b) | … | crc_checksum (2b)

Return type

bytes

Returns

bytes representing the Message object

class pykiso.message.MessageAckType(value)

List of possible received messages.

class pykiso.message.MessageCommandType(value)

List of commands allowed.

class pykiso.message.MessageLogType(value)

List of possible received log messages.

class pykiso.message.MessageReportType(value)

List of possible received messages.

class pykiso.message.MessageType(value)

List of messages allowed.

class pykiso.message.TlvKnownTags(value)

List of known / supported tags.

Import Magic

Auxiliary Interface Definition

module

dynamic_loader

synopsis

Import magic that enables aliased auxiliary loading in TestCases

class pykiso.test_setup.dynamic_loader.DynamicImportLinker

Public Interface of Import Magic.

initialises the Loaders, Finders and Caches, implements interfaces to install the magic and register the auxiliaries and connectors.

Initialize attributes.

install()

Install the import hooks with the system.

provide_auxiliary(name, module, aux_cons=None, **config_params)

Provide a auxiliary.

Parameters

• name (str) – the auxiliary alias

• module (str) – either ‘python-file-path:Class’ or ‘module:Class’ of the class we want to provide

• aux_cons – list of connectors this auxiliary has

provide_connector(name, module, **config_params)

Provide a connector.

Parameters

• name (str) – the connector alias

• module (str) – either ‘python-file-path:Class’ or ‘module:Class’ of the class we want to provide

uninstall()

Deregister the import hooks, close all running threads, delete all instances.

Config Registry

module

config_registry

synopsis

register auxiliaries and connectors to provide them for import.

class pykiso.test_setup.config_registry.ConfigRegistry

Register auxiliaries with connectors to provide systemwide import statements.

classmethod delete_aux_con()

deregister the import hooks, close all running threads, delete all instances.

Return type

None

classmethod get_all_auxes()

Return all auxiliaires instances and alias

Return type

dict

Returns

dictionary with alias as keys and instances as values

classmethod get_aux_by_alias(alias)

Return the associated auxiliary instance to the given alias.

Parameters

alias (str) – auxiliary’s alias

Return type

Any

Returns

auxiliary instance created by the dymanic loader

classmethod get_aux_config(name)

Return the registered auxiliary configuration based on his name.

Parameters

name (str) – auxiliary alias

Return type

dict

Returns

auxiliary’s configuration (yaml content)

classmethod get_auxes_alias()

return all created auxiliaries alias.

Return type

list

Returns

list containing all auxiliaries alias

classmethod get_auxes_by_type(aux_type)

Return all auxiliaries who match a specific type.

Parameters

aux_type (Any) – auxiliary class type (DUTAuxiliary, CommunicationAuxiliary…)

Return type

dict

Returns

dictionary with alias as keys and instances as values

classmethod register_aux_con(config)

Create import hooks. Register auxiliaries and connectors.

Parameters

config (dict) – dictionary containing yaml configuration content

Return type

None

Test Suites

Test Suite

module

test_suite

synopsis

Create a generic test-suite based on the connected modules, and

gray test-suite for Message Protocol / TestApp usage.

class pykiso.test_coordinator.test_suite.BaseTestSuite(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)

Initialize generic test-case.

Parameters

• test_suite_id (int) – test suite identification number

• test_case_id (int) – test case identification number

• aux_list (Optional[List[AuxiliaryInterface]]) – list of used auxiliaries

• setup_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during setup execution

• run_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during test_run execution

• teardown_timeout (Optional[int]) – the maximum time (in seconds) used to wait for a report during teardown execution

• test_ids (Optional[dict]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}

• tag (Optional[Dict[str, List[str]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed

cleanup_and_skip(aux, info_to_print)

Cleanup auxiliary and log reasons.

Parameters

• aux (AuxiliaryInterface) – corresponding auxiliary to abort

• info_to_print (str) – A message you want to print while cleaning up the test

class pykiso.test_coordinator.test_suite.BasicTestSuite(modules_to_add_dir, test_filter_pattern, test_suite_id, args, kwargs)

Inherit from the unittest framework test-suite but build it for our integration tests.

Initialize our custom unittest-test-suite.

Note

1. Will Load from the given path the integration test modules under test

2. Sort the given test case list by test suite/case id

3. Place Test suite setup and teardown respectively at top and bottom of test case list

4. Add sorted test case list to test suite

check_suite_setup_failed(test, result)

Check if the suite setup has failed and store failed suite id. Search in the global unittest result object, which save all the results of the tests performed up to that point, for a BasicTestSuiteSetup tests which has failed. If the suite setup has failed store the suite id.

Parameters

• test (BasicTest) – test to check

• result (TestResult) – unittest result object

Return type

None

run(result, debug=False)

Override run method from unittest.suite.TestSuite. Added functionality: Skip suite tests if the parent test suite setup has failed.

Parameters

• result (TestResult) – unittest result storage

• debug (bool) – True to enter debug mode, defaults to False

Return type

TestResult

Returns

test suite result

class pykiso.test_coordinator.test_suite.BasicTestSuiteSetup(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)

Inherit from unittest testCase and represent setup fixture.

Initialize Message Protocol / TestApp test-case.

Parameters

• test_suite_id (int) – test suite identification number

• test_case_id (int) – test case identification number

• aux_list (Optional[List[AuxiliaryInterface]]) – list of used auxiliaries

• setup_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during setup execution

• run_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during test_run execution

• teardown_timeout (Optional[int]) – the maximum time (in seconds) used to wait for a report during teardown execution

• test_ids (Optional[dict]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}

• tag (Optional[Dict[str, List[str]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed

test_suite_setUp()

Test method for constructing the actual test suite.

class pykiso.test_coordinator.test_suite.BasicTestSuiteTeardown(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)

Inherit from unittest testCase and represent teardown fixture.

Initialize Message Protocol / TestApp test-case.

Parameters

• test_suite_id (int) – test suite identification number

• test_case_id (int) – test case identification number

• aux_list (Optional[List[AuxiliaryInterface]]) – list of used auxiliaries

• setup_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during setup execution

• run_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during test_run execution

• teardown_timeout (Optional[int]) – the maximum time (in seconds) used to wait for a report during teardown execution

• test_ids (Optional[dict]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}

• tag (Optional[Dict[str, List[str]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed

test_suite_tearDown()

Test method for deconstructing the actual test suite after testing it.

class pykiso.test_coordinator.test_suite.RemoteTestSuiteSetup(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)

Inherit from unittest testCase and represent setup fixture when Message Protocol / TestApp is used.

Initialize Message Protocol / TestApp test-case.

Parameters

• test_suite_id (int) – test suite identification number

• test_case_id (int) – test case identification number

• aux_list (Optional[List[AuxiliaryInterface]]) – list of used auxiliaries

• setup_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during setup execution

• run_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during test_run execution

• teardown_timeout (Optional[int]) – the maximum time (in seconds) used to wait for a report during teardown execution

• test_ids (Optional[dict]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}

• tag (Optional[Dict[str, List[str]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed

test_suite_setUp()

Test method for constructing the actual test suite.

class pykiso.test_coordinator.test_suite.RemoteTestSuiteTeardown(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)

Inherit from unittest testCase and represent teardown fixture when Message Protocol / TestApp is used.

Initialize Message Protocol / TestApp test-case.

Parameters

• test_suite_id (int) – test suite identification number

• test_case_id (int) – test case identification number

• aux_list (Optional[List[AuxiliaryInterface]]) – list of used auxiliaries

• setup_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during setup execution

• run_timeout (Optional[int]) – maximum time (in seconds) used to wait for a report during test_run execution

• teardown_timeout (Optional[int]) – the maximum time (in seconds) used to wait for a report during teardown execution

• test_ids (Optional[dict]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}

• tag (Optional[Dict[str, List[str]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed

test_suite_tearDown()

Test method for deconstructing the actual test suite after testing it.

Test Execution

Test Execution

module

test_execution

synopsis

Execute a test environment based on the supplied configuration.

Note

1. Glob a list of test-suite folders

2. Generate a list of test-suites with a list of test-cases

3. Loop per suite

4. Gather result

class pykiso.test_coordinator.test_execution.ExitCode(value)

List of possible exit codes

pykiso.test_coordinator.test_execution.apply_tag_filter(all_tests_to_run, usr_tags)

Filter the test cases based on user tags.

Parameters

• all_tests_to_run (TestSuite) – a dict containing all testsuites and testcases

• usr_tags (Dict[str, List[str]]) – encapsulate user’s variant choices

Return type

None

pykiso.test_coordinator.test_execution.collect_test_suites(config_test_suite_list, test_filter_pattern=None)

Collect and load all test suites defined in the test configuration.

Parameters

• config_test_suite_list (List[Dict[str, Union[str, int]]]) – list of dictionaries from the configuration file corresponding each to one test suite.

• test_filter_pattern (Optional[str]) – optional filter pattern to overwrite the one defined in the test suite configuration.

Raises

pykiso.TestCollectionError – if any test case inside one of the configured test suites failed to be loaded.

Return type

List[Optional[BasicTestSuite]]

Returns

a list of all loaded test suites.

pykiso.test_coordinator.test_execution.create_test_suite(test_suite_dict)

create a test suite based on the config dict

Parameters

test_suite_dict (Dict[str, Union[str, int]]) – dict created from config with keys ‘suite_dir’, ‘test_filter_pattern’, ‘test_suite_id’

Return type

BasicTestSuite

pykiso.test_coordinator.test_execution.enable_step_report(all_tests_to_run)

Decorate all assert method from Test-Case

This will allow to save the assert inputs in order to generate the step-report

Parameters

all_tests_to_run (TestSuite) – a dict containing all testsuites and testcases

Return type

None

pykiso.test_coordinator.test_execution.execute(config, report_type='text', user_tags=None, step_report=None, pattern_inject=None, failfast=False)

Create test environment based on test configuration.

Parameters

• config (Dict[str, Any]) – dict from converted YAML config file

• report_type (str) – str to set the type of report wanted, i.e. test or junit

• user_tags (Optional[Dict[str, List[str]]]) – test case tags to execute

• step_report (Optional[Path]) – file path for the step report or None

• pattern_inject (Optional[str]) – optional pattern that will override test_filter_pattern for all suites. Used in test development to run specific tests.

• failfast (bool) – stop the test run on the first error or failure.

Return type

int

Returns

exit code corresponding to the result of the test execution (tests failed, unexpected exception, …)

pykiso.test_coordinator.test_execution.failure_and_error_handling(result)

provide necessary information to Jenkins if an error occur during tests execution

Parameters

result (TestResult) – encapsulate all test results from the current run

Return type

int

Returns

an ExitCode object

Test-Message Handling

Handle common communication with device under test

When using a Remote TestCase/TestSuite, the integration test framework handles internal messaging and control flow using a message format defined in pykiso.Message.

pykiso.test_message_handler defines the messaging protocol from a behavioral point of view.

module

test_message_handler

synopsis

default communication between TestManagement and DUT.

pykiso.test_coordinator.test_message_handler.test_app_interaction(message_type, timeout_cmd=5)

Handle test app basic interaction depending on the decorated method.

Parameters

• message_type (MessageCommandType) – message command sub-type (test case/suite run, setup, teardown….)

• timeout_cmd (int) – timeout in seconds for auxiliary run_command

Return type

Callable

Returns

inner decorator function

test xml result

test_xml_result

module

test_xml_result

synopsis

overwrite xmlrunner.result to be able to add additional data into the xml report.

class pykiso.test_coordinator.test_xml_result.TestInfo(test_result, test_method, outcome=0, err=None, subTest=None, filename=None, lineno=None, doc=None)

This class keeps useful information about the execution of a test method. Used by XmlTestResult

Initialize the TestInfo class and append additional tag

that have to be stored for each test

Parameters

• test_result (_XMLTestResult) – test result class

• test_method – test method (dynamically created eg: test_case.MyTest2-1-2)

• outcome (int) – result of the test (SUCCESS, FAILURE, ERROR, SKIP)

• err – error cached during test

• subTest – optional, refer the test id and the test description

• filename (Optional[str]) – name of the file

• lineno (Optional[bool]) – store the test line number

• doc (Optional[str]) – additional documentation to store

class pykiso.test_coordinator.test_xml_result.XmlTestResult(stream=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>, descriptions=True, verbosity=1, elapsed_times=True, properties=None, infoclass=<class 'pykiso.test_coordinator.test_xml_result.TestInfo'>)

Test result class that can express test results in a XML report. Used by XMLTestRunner

Initialize the _XMLTestResult class.

Parameters

• stream (TextIOWrapper) – buffered text interface to a buffered raw stream

• descriptions (bool) – include description of the test

• verbosity (int) – print output into the console

• elapsed_times (bool) – include the time spend on the test

• properties – junit testsuite properties

• infoclass (_TestInfo) – class containing the test information

report_testcase(xml_testsuite, xml_document)

Appends a testcase section to the XML document.