API Documentation
Test Cases
Generic Test
- module
test_case
- synopsis
Basic extensible implementation of a TestCase, and of a Remote
TestCase for Message Protocol / TestApp usage.
Note
TODO later on will inherit from a metaclass to get the id parameters
- class pykiso.test_coordinator.test_case.BasicTest(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, *args, **kwargs)[source]
Base for test-cases.
Initialize generic test-case.
- Parameters
test_suite_id (
int
) – test suite identification numbertest_case_id (
int
) – test case identification numberaux_list (
Optional
[List
[AuxiliaryInterface
]]) – list of used auxiliariessetup_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during setup executionrun_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during test_run executionteardown_timeout (
Optional
[int
]) – the maximum time (in seconds) used to wait for a report during teardown executiontest_ids (
Optional
[dict
]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}tag (
Optional
[Dict
[str
,List
[str
]]]) – dictionary allowing users to filter the tests based on the keys and their value.
- cleanup_and_skip(aux, info_to_print)[source]
Cleanup auxiliary and log reasons.
- Parameters
aux (
AuxiliaryInterface
) – corresponding auxiliary to abortinfo_to_print (
str
) – A message you want to print while cleaning up the test
- Return type
None
- class pykiso.test_coordinator.test_case.RemoteTest(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, *args, **kwargs)[source]
Base test-cases for Message Protocol / TestApp usage.
Initialize TestApp test-case.
- Parameters
test_suite_id (
int
) – test suite identification numbertest_case_id (
int
) – test case identification numberaux_list (
Optional
[List
[AuxiliaryInterface
]]) – list of used auxiliariessetup_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during setup executionrun_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during test_run executionteardown_timeout (
Optional
[int
]) – the maximum time (in seconds) used to wait for a report during teardown executiontest_ids (
Optional
[dict
]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}tag (
Optional
[Dict
[str
,List
[str
]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed
- pykiso.test_coordinator.test_case.define_test_parameters(suite_id=0, case_id=0, aux_list=None, setup_timeout=None, run_timeout=None, teardown_timeout=None, test_ids=None, tag=None)[source]
Decorator to fill out test parameters of the BasicTest and RemoteTest automatically.
- pykiso.test_coordinator.test_case.retry_test_case(max_try=2, rerun_setup=False, rerun_teardown=False, stability_test=False)[source]
Decorator: retry mechanism for testCase.
The aim is to cover the 2 following cases:
Unstable test : get the test pass within the {max_try} attempt
Stability test : run {max_try} time the test expecting no error
The retry_test_case comes with the possibility to re-run the setUp and tearDown methods automatically.
- Parameters
max_try (
int
) – maximum number of try to get the test pass.rerun_setup (
bool
) – call the “setUp” method of the test.rerun_teardown (
bool
) – call the “tearDown” method of the test.stability_test (
bool
) – run {max_try} time the test and raise an exception if an error occurs.
- Returns
None, a testCase is not supposed to return anything.
- Raises
Exception – if stability_test, the exception that occurred during the execution; if not stability_test, the exception that occurred at the last try.
Connectors
pykiso comes with some ready to use implementations of different connectors.
Auxiliaries
Message Protocol
pykiso Control Message Protocol
- module
message
- synopsis
Message that will be send though the different agents
- class pykiso.message.Message(msg_type=0, sub_type=0, error_code=0, test_suite=0, test_case=0, tlv_dict=None)[source]
A message who fit testApp protocol.
The created message is a tlv style message with the following format: TYPE: msg_type | message_token | sub_type | errorCode |
Create a generic message.
- Parameters
msg_type (MessageType) – Message type
sub_type (Message<MessageType>Type) – Message sub-type
error_code (integer) – Error value
test_suite (integer) – Suite value
test_case (integer) – Test value
tlv_dict (dict) – Dictionary containing tlvs elements in the form {‘type’:’value’, …}
- check_if_ack_message_is_matching(ack_message)[source]
Check if the ack message was for this sent message.
- Parameters
ack_message (
Message
) – received acknowledge message- Return type
bool
- Returns
True if message type and token are valid otherwise False
- generate_ack_message(ack_type)[source]
Generate acknowledgement to send out.
- Parameters
ack_type (
int
) – ack or nack- Return type
Optional
[Message
]- Returns
filled acknowledge message otherwise None
- classmethod get_crc(serialized_msg, crc_byte_size=2)[source]
Get the CRC checksum for a bytes message.
- Parameters
serialized_msg (
bytes
) – message used for the crc calculationcrc_byte_size (
int
) – number of bytes dedicated for the crc
- Return type
int
- Returns
CRC checksum
- get_message_tlv_dict()[source]
Return actual message type/length/value dictionary.
- Return type
dict
- get_message_type()[source]
Return actual message type.
- Return type
Union
[int
,MessageType
]
- classmethod parse_packet(raw_packet)[source]
Factory function to create a Message object from raw data.
- Parameters
raw_packet (
bytes
) – array of a received message- Return type
- Returns
itself
- serialize()[source]
Serialize message into raw packet.
- Format: | msg_type (1b) | msg_token (1b) | sub_type (1b) | error_code (1b) |
- test_section (1b) | test_suite (1b) | test_case (1b) | payload_length (1b) |tlv_type (1b) | tlv_size (1b) | … | crc_checksum (2b)
- Return type
bytes
- Returns
bytes representing the Message object
Import Magic
Auxiliary Interface Definition
- module
dynamic_loader
- synopsis
Import magic that enables aliased auxiliary loading in TestCases
- class pykiso.test_setup.dynamic_loader.DynamicImportLinker[source]
Public Interface of Import Magic.
initialises the Loaders, Finders and Caches, implements interfaces to install the magic and register the auxiliaries and connectors.
Initialize attributes.
- provide_auxiliary(name, module, aux_cons=None, **config_params)[source]
Provide a auxiliary.
- Parameters
name (
str
) – the auxiliary aliasmodule (
str
) – either ‘python-file-path:Class’ or ‘module:Class’ of the class we want to provide.aux_cons – list of connectors this auxiliary has
Config Registry
- module
config_registry
- synopsis
register auxiliaries and connectors to provide them for import.
- class pykiso.test_setup.config_registry.ConfigRegistry[source]
Register auxiliaries with connectors to provide systemwide import statements.
Internally patch the user-configuration if multiple auxiliaries share a same communication channel.
- classmethod delete_aux_con()[source]
Deregister the import hooks, close all running threads, delete all instances.
- Return type
None
- classmethod get_all_auxes()[source]
Return all auxiliaires instances and alias
- Return type
Dict[AuxiliaryAlias, AuxiliaryCommon]
- Returns
dictionary with alias as keys and instances as values
- classmethod get_aux_by_alias(alias)[source]
Return the associated auxiliary instance to the given alias.
- Parameters
alias (AuxiliaryAlias) – auxiliary’s alias
- Return type
- Returns
auxiliary instance created by the dymanic loader
- classmethod get_aux_config(name)[source]
Return the registered auxiliary configuration based on his name.
- Parameters
name (AuxiliaryAlias) – auxiliary’s alias
- Return type
AuxiliaryConfig
- Returns
auxiliary’s configuration (yaml content)
- classmethod get_auxes_alias()[source]
return all created auxiliaries alias.
- Return type
List[AuxiliaryAlias]
- Returns
list containing all auxiliaries alias
- classmethod get_auxes_by_type(aux_type)[source]
Return all auxiliaries who match a specific type.
- Parameters
aux_type (Type[AuxiliaryCommon]) – auxiliary class type (DUTAuxiliary, CommunicationAuxiliary…)
- Return type
Dict[str, AuxiliaryCommon]
- Returns
dictionary with alias as keys and instances as values
Test Suites
Test Suite
- module
test_suite
- synopsis
Create a generic test-suite based on the connected modules, and
gray test-suite for Message Protocol / TestApp usage.
- class pykiso.test_coordinator.test_suite.BaseTestSuite(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, *args, **kwargs)[source]
Initialize generic test-case.
- Parameters
test_suite_id (
int
) – test suite identification numbertest_case_id (
int
) – test case identification numberaux_list (
Optional
[List
[AuxiliaryInterface
]]) – list of used auxiliariessetup_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during setup executionrun_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during test_run executionteardown_timeout (
Optional
[int
]) – the maximum time (in seconds) used to wait for a report during teardown executiontest_ids (
Optional
[dict
]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}tag (
Optional
[Dict
[str
,List
[str
]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed
- cleanup_and_skip(aux, info_to_print)[source]
Cleanup auxiliary and log reasons.
- Parameters
aux (
AuxiliaryInterface
) – corresponding auxiliary to abortinfo_to_print (
str
) – A message you want to print while cleaning up the test
- class pykiso.test_coordinator.test_suite.BasicTestSuite(modules_to_add_dir, test_filter_pattern, test_suite_id, *args, **kwargs)[source]
Inherit from the unittest framework test-suite but build it for our integration tests.
Initialize our custom unittest-test-suite.
Note
Will Load from the given path the integration test modules under test
Sort the given test case list by test suite/case id
Place Test suite setup and teardown respectively at top and bottom of test case list
Add sorted test case list to test suite
- check_suite_setup_failed(test, result)[source]
Check if the suite setup has failed and store failed suite id. Search in the global unittest result object, which save all the results of the tests performed up to that point, if the suite setup that ran has failed then store the suite id.
- Parameters
test (BasicTest) – test to check
result (BannerTestResult) – unittest result object
- Return type
None
- run(result, debug=False)[source]
Override run method from unittest.suite.TestSuite. Added functionality: Skip suite tests if the parent test suite setup has failed.
- Parameters
result (BannerTestResult) – unittest result storage
debug (bool) – True to enter debug mode, defaults to False
- Return type
- Returns
test suite result
- class pykiso.test_coordinator.test_suite.BasicTestSuiteSetup(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, *args, **kwargs)[source]
Inherit from unittest testCase and represent setup fixture.
Initialize Message Protocol / TestApp test-case.
- Parameters
test_suite_id (
int
) – test suite identification numbertest_case_id (
int
) – test case identification numberaux_list (
Optional
[List
[AuxiliaryInterface
]]) – list of used auxiliariessetup_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during setup executionrun_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during test_run executionteardown_timeout (
Optional
[int
]) – the maximum time (in seconds) used to wait for a report during teardown executiontest_ids (
Optional
[dict
]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}tag (
Optional
[Dict
[str
,List
[str
]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed
- class pykiso.test_coordinator.test_suite.BasicTestSuiteTeardown(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, *args, **kwargs)[source]
Inherit from unittest testCase and represent teardown fixture.
Initialize Message Protocol / TestApp test-case.
- Parameters
test_suite_id (
int
) – test suite identification numbertest_case_id (
int
) – test case identification numberaux_list (
Optional
[List
[AuxiliaryInterface
]]) – list of used auxiliariessetup_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during setup executionrun_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during test_run executionteardown_timeout (
Optional
[int
]) – the maximum time (in seconds) used to wait for a report during teardown executiontest_ids (
Optional
[dict
]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}tag (
Optional
[Dict
[str
,List
[str
]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed
- class pykiso.test_coordinator.test_suite.RemoteTestSuiteSetup(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, *args, **kwargs)[source]
Inherit from unittest testCase and represent setup fixture when Message Protocol / TestApp is used.
Initialize Message Protocol / TestApp test-case.
- Parameters
test_suite_id (
int
) – test suite identification numbertest_case_id (
int
) – test case identification numberaux_list (
Optional
[List
[AuxiliaryInterface
]]) – list of used auxiliariessetup_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during setup executionrun_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during test_run executionteardown_timeout (
Optional
[int
]) – the maximum time (in seconds) used to wait for a report during teardown executiontest_ids (
Optional
[dict
]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}tag (
Optional
[Dict
[str
,List
[str
]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed
- class pykiso.test_coordinator.test_suite.RemoteTestSuiteTeardown(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, *args, **kwargs)[source]
Inherit from unittest testCase and represent teardown fixture when Message Protocol / TestApp is used.
Initialize Message Protocol / TestApp test-case.
- Parameters
test_suite_id (
int
) – test suite identification numbertest_case_id (
int
) – test case identification numberaux_list (
Optional
[List
[AuxiliaryInterface
]]) – list of used auxiliariessetup_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during setup executionrun_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during test_run executionteardown_timeout (
Optional
[int
]) – the maximum time (in seconds) used to wait for a report during teardown executiontest_ids (
Optional
[dict
]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}tag (
Optional
[Dict
[str
,List
[str
]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed
Test Execution
Test Execution
- module
test_execution
- synopsis
Execute a test environment based on the supplied configuration.
Note
Glob a list of test-suite folders
Generate a list of test-suites with a list of test-cases
Loop per suite
Gather result
- class pykiso.test_coordinator.test_execution.TestFilterPattern(test_file, test_class, test_case)
Create new instance of TestFilterPattern(test_file, test_class, test_case)
- property test_case
Alias for field number 2
- property test_class
Alias for field number 1
- property test_file
Alias for field number 0
- pykiso.test_coordinator.test_execution.apply_tag_filter(all_tests_to_run, usr_tags)[source]
Filter the test cases based on user tags provided via CLI.
- Parameters
all_tests_to_run (
TestSuite
) – a dict containing all testsuites and testcasesusr_tags (
Dict
[str
,List
[str
]]) – encapsulate user’s variant choices
- Return type
None
- pykiso.test_coordinator.test_execution.apply_test_case_filter(all_tests_to_run, test_class_pattern, test_case_pattern)[source]
Apply a filter to run only test cases which matches given expression
- Parameters
all_tests_to_run (
TestSuite
) – a dict containing all testsuites and testcasestest_class_pattern (
str
) – pattern to select test class as unix filename patterntest_case_pattern (
str
) – pattern to select test case as unix filename pattern
- Return type
TestSuite
- Returns
new test suite with filtered test cases
- pykiso.test_coordinator.test_execution.collect_test_suites(config_test_suite_list, test_filter_pattern=None)[source]
Collect and load all test suites defined in the test configuration.
- Parameters
config_test_suite_list (
List
[Dict
[str
,Union
[str
,int
]]]) – list of dictionaries from the configuration file corresponding each to one test suite.test_filter_pattern (
Optional
[str
]) – optional filter pattern to overwrite the one defined in the test suite configuration.
- Raises
pykiso.TestCollectionError – if any test case inside one of the configured test suites failed to be loaded.
- Return type
List
[Optional
[BasicTestSuite
]]- Returns
a list of all loaded test suites.
- pykiso.test_coordinator.test_execution.create_test_suite(test_suite_dict)[source]
create a test suite based on the config dict
- Parameters
test_suite_dict (
Dict
[str
,Union
[str
,int
]]) – dict created from config with keys ‘suite_dir’, ‘test_filter_pattern’, ‘test_suite_id’- Return type
- pykiso.test_coordinator.test_execution.enable_step_report(all_tests_to_run, step_report)[source]
Decorate all assert method from Test-Case.
This will allow to save the assert inputs in order to generate the HTML step report.
- Parameters
all_tests_to_run (
TestSuite
) – a dict containing all testsuites and testcases- Return type
None
- pykiso.test_coordinator.test_execution.execute(config, report_type='text', user_tags=None, step_report=None, pattern_inject=None, failfast=False)[source]
Create test environment based on test configuration.
- Parameters
config (
Dict
[str
,Any
]) – dict from converted YAML config filereport_type (
str
) – str to set the type of report wanted, i.e. test or junituser_tags (
Optional
[Dict
[str
,List
[str
]]]) – test case tags to executestep_report (
Optional
[Path
]) – file path for the step report or Nonepattern_inject (
Optional
[str
]) – optional pattern that will override test_filter_pattern for all suites. Used in test development to run specific tests.failfast (
bool
) – stop the test run on the first error or failure.
- Return type
int
- Returns
exit code corresponding to the result of the test execution (tests failed, unexpected exception, …)
- pykiso.test_coordinator.test_execution.failure_and_error_handling(result)[source]
provide necessary information to Jenkins if an error occur during tests execution
- Parameters
result (
TestResult
) – encapsulate all test results from the current run- Return type
int
- Returns
an ExitCode object
Test-Message Handling
Handle common communication with device under test
When using a Remote TestCase/TestSuite, the integration test framework
handles internal messaging and control flow using a message format
defined in pykiso.Message
.
pykiso.test_message_handler
defines the messaging protocol
from a behavioral point of view.
- module
test_message_handler
- synopsis
default communication between TestManagement and DUT.
- pykiso.test_coordinator.test_message_handler.test_app_interaction(message_type, timeout_cmd=5)[source]
Handle test app basic interaction depending on the decorated method.
- Parameters
message_type (
MessageCommandType
) – message command sub-type (test case/suite run, setup, teardown….)timeout_cmd (
int
) – timeout in seconds for auxiliary run_command
- Return type
Callable
- Returns
inner decorator function
Test Results
XML test result for JUnit support
- module
xml_result
- synopsis
overwrite xmlrunner.result to add the test IDs into the xml report.
- class pykiso.test_result.xml_result.TestInfo(test_result, test_case, outcome=0, err=None, subTest=None, filename=None, lineno=None, doc=None)[source]
This class keeps useful information about the execution of a test method. Used by XmlTestResult
- Initialize the TestInfo class and append additional tag
that have to be stored for each test
- Parameters
test_result (xmlrunner.runner._XMLTestResult) – test result class
test_method – test method (dynamically created eg: test_case.MyTest2-1-2)
outcome (int) – result of the test (SUCCESS, FAILURE, ERROR, SKIP)
err (Optional[Tuple[Type[BaseException], BaseException, TracebackType]]) – exception information of an error that was raised during the test
subTest (TestCase) – optional subTest that was run
filename (Optional[str]) – name of the file
lineno (Optional[bool]) – store the test line number
doc (Optional[str]) – additional documentation to store
- class pykiso.test_result.xml_result.XmlTestResult(stream=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>, descriptions=True, verbosity=0, elapsed_times=True, properties=None, infoclass=<class 'pykiso.test_result.xml_result.TestInfo'>)[source]
Test result class that can express test results in a XML report. Used by XMLTestRunner.
Initialize both base classes with the appropriate parameters.
- Parameters
stream (
TextIOWrapper
) – buffered text interface to a buffered raw streamdescriptions (
bool
) – include description of the testverbosity (
int
) – print output into the consoleelapsed_times (
bool
) – include the time spend on the testproperties (
Optional
[Dict
[str
,Any
]]) – junit testsuite propertiesinfoclass (
_TestInfo
) – class containing the test information
- addSuccess(test)[source]
Calls only _XMLTestResult’s addSuccess as BannerTestResult’s appends TestCase instances to the successes list, but _XMLTestResult wraps them in TestInfo instances with additional attributes.
- Parameters
test (TestCase) – current succeeded TestCase.
- Return type
None
- report_testcase(xml_testsuite, xml_document)
Appends a testcase section to the XML document.
Create a Step report
- module
assert_step_report
- synopsis
Provide a detailed test view containing: - test name - test description - date of execution + elapsed time - information gathered during test - assertion detail: data_in, variable
name of the data_in, expected, message
- class pykiso.test_result.assert_step_report.StepReportData(header=<factory>, message='', success=True, last_error_message='', current_table=None)[source]
- pykiso.test_result.assert_step_report.assert_decorator(assert_method)[source]
Decorator to gather assertion information
- MyTestClass
header: additional data
description: test purpose
file_path: test location
time_result: start/end/elapsed time
- test_list: store the steps result
setUp : list of assert
test_run: list of assert
Note
header is based on the variable
step_report_header
and description is based on the docstring of the test_run methodMyTest(pykiso.BasicTest): def test_run(self): '''Here is my test description''' self.step_report.header["Voltage"] = 5
- Parameters
func – function to decorate (expected assert method)
return: The func output if it exists. Otherwise, None
- pykiso.test_result.assert_step_report.generate_step_report(test_result, output_file)[source]
Generate the HTML step report based on Jinja2 template
- Parameters
test_result (
Union
[BannerTestResult
,XmlTestResult
]) – Result of tests to generate the report fromoutput_file (
str
) – Report output file path
- Return type
None