API Documentation
Test Cases
Generic Test
- module
test_case
- synopsis
Basic extensible implementation of a TestCase, and of a Remote
TestCase for Message Protocol / TestApp usage.
Note
TODO later on will inherit from a metaclass to get the id parameters
- class pykiso.test_coordinator.test_case.BasicTest(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)[source]
Base for test-cases.
Initialize generic test-case.
- Parameters
test_suite_id (
int
) – test suite identification numbertest_case_id (
int
) – test case identification numberaux_list (
Optional
[List
[AuxiliaryInterface
]]) – list of used auxiliariessetup_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during setup executionrun_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during test_run executionteardown_timeout (
Optional
[int
]) – the maximum time (in seconds) used to wait for a report during teardown executiontest_ids (
Optional
[dict
]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}tag (
Optional
[Dict
[str
,List
[str
]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed
- cleanup_and_skip(aux, info_to_print)[source]
Cleanup auxiliary and log reasons.
- Parameters
aux (
AuxiliaryInterface
) – corresponding auxiliary to abortinfo_to_print (
str
) – A message you want to print while cleaning up the test
- Return type
None
- class pykiso.test_coordinator.test_case.RemoteTest(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)[source]
Base test-cases for Message Protocol / TestApp usage.
Initialize TestApp test-case.
- Parameters
test_suite_id (
int
) – test suite identification numbertest_case_id (
int
) – test case identification numberaux_list (
Optional
[List
[AuxiliaryInterface
]]) – list of used auxiliariessetup_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during setup executionrun_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during test_run executionteardown_timeout (
Optional
[int
]) – the maximum time (in seconds) used to wait for a report during teardown executiontest_ids (
Optional
[dict
]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}tag (
Optional
[Dict
[str
,List
[str
]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed
- pykiso.test_coordinator.test_case.define_test_parameters(suite_id=0, case_id=0, aux_list=None, setup_timeout=None, run_timeout=None, teardown_timeout=None, test_ids=None, tag=None)[source]
Decorator to fill out test parameters of the BasicTest and RemoteTest automatically.
- pykiso.test_coordinator.test_case.retry_test_case(max_try=2, rerun_setup=False, rerun_teardown=False, stability_test=False)[source]
Decorator: retry mechanism for testCase.
The aim is to cover the 2 following cases:
Unstable test : get the test pass within the {max_try} attempt
Stability test : run {max_try} time the test expecting no error
The retry_test_case comes with the possibility to re-run the setUp and tearDown methods automatically.
- Parameters
max_try (
int
) – maximum number of try to get the test pass.rerun_setup (
bool
) – call the “setUp” method of the test.rerun_teardown (
bool
) – call the “tearDown” method of the test.stability_test (
bool
) – run {max_try} time the test and raise an exception if an error occurs.
- Returns
None, a testCase is not supposed to return anything.
- Raises
Exception – if stability_test, the exception that occurred during the execution; if not stability_test, the exception that occurred at the last try.
Connectors
pykiso comes with some ready to use implementations of different connectors.
Auxiliaries
Message Protocol
pykiso Control Message Protocol
- module
message
- synopsis
Message that will be send though the different agents
- class pykiso.message.Message(msg_type=0, sub_type=0, error_code=0, test_suite=0, test_case=0, tlv_dict=None)[source]
A message who fit testApp protocol.
The created message is a tlv style message with the following format: TYPE: msg_type | message_token | sub_type | errorCode |
Create a generic message.
- Parameters
msg_type (MessageType) – Message type
sub_type (Message<MessageType>Type) – Message sub-type
error_code (integer) – Error value
test_suite (integer) – Suite value
test_case (integer) – Test value
tlv_dict (dict) – Dictionary containing tlvs elements in the form {‘type’:’value’, …}
- check_if_ack_message_is_matching(ack_message)[source]
Check if the ack message was for this sent message.
- Parameters
ack_message (
Message
) – received acknowledge message- Return type
bool
- Returns
True if message type and token are valid otherwise False
- generate_ack_message(ack_type)[source]
Generate acknowledgement to send out.
- Parameters
ack_type (
int
) – ack or nack- Return type
Optional
[Message
]- Returns
filled acknowledge message otherwise None
- classmethod get_crc(serialized_msg, crc_byte_size=2)[source]
Get the CRC checksum for a bytes message.
- Parameters
serialized_msg (
bytes
) – message used for the crc calculationcrc_byte_size (
int
) – number of bytes dedicated for the crc
- Return type
int
- Returns
CRC checksum
- get_message_tlv_dict()[source]
Return actual message type/length/value dictionary.
- Return type
dict
- get_message_type()[source]
Return actual message type.
- Return type
Union
[int
,MessageType
]
- classmethod parse_packet(raw_packet)[source]
Factory function to create a Message object from raw data.
- Parameters
raw_packet (
bytes
) – array of a received message- Return type
- Returns
itself
- serialize()[source]
Serialize message into raw packet.
- Format: | msg_type (1b) | msg_token (1b) | sub_type (1b) | error_code (1b) |
- test_section (1b) | test_suite (1b) | test_case (1b) | payload_length (1b) |tlv_type (1b) | tlv_size (1b) | … | crc_checksum (2b)
- Return type
bytes
- Returns
bytes representing the Message object
Import Magic
Auxiliary Interface Definition
- module
dynamic_loader
- synopsis
Import magic that enables aliased auxiliary loading in TestCases
- class pykiso.test_setup.dynamic_loader.DynamicImportLinker[source]
Public Interface of Import Magic.
initialises the Loaders, Finders and Caches, implements interfaces to install the magic and register the auxiliaries and connectors.
Initialize attributes.
- provide_auxiliary(name, module, aux_cons=None, **config_params)[source]
Provide a auxiliary.
- Parameters
name (
str
) – the auxiliary aliasmodule (
str
) – either ‘python-file-path:Class’ or ‘module:Class’ of the class we want to provideaux_cons – list of connectors this auxiliary has
Config Registry
- module
config_registry
- synopsis
register auxiliaries and connectors to provide them for import.
- class pykiso.test_setup.config_registry.ConfigRegistry[source]
Register auxiliaries with connectors to provide systemwide import statements.
- classmethod delete_aux_con()[source]
deregister the import hooks, close all running threads, delete all instances.
- Return type
None
- classmethod get_all_auxes()[source]
Return all auxiliaires instances and alias
- Return type
dict
- Returns
dictionary with alias as keys and instances as values
- classmethod get_aux_config(name)[source]
Return the registered auxiliary configuration based on his name.
- Parameters
name (
str
) – auxiliary alias- Return type
dict
- Returns
auxiliary’s configuration (yaml content)
- classmethod get_auxes_alias()[source]
return all created auxiliaries alias.
- Return type
list
- Returns
list containing all auxiliaries alias
Test Suites
Test Suite
- module
test_suite
- synopsis
Create a generic test-suite based on the connected modules, and
gray test-suite for Message Protocol / TestApp usage.
- class pykiso.test_coordinator.test_suite.BaseTestSuite(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)[source]
Initialize generic test-case.
- Parameters
test_suite_id (
int
) – test suite identification numbertest_case_id (
int
) – test case identification numberaux_list (
Optional
[List
[AuxiliaryInterface
]]) – list of used auxiliariessetup_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during setup executionrun_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during test_run executionteardown_timeout (
Optional
[int
]) – the maximum time (in seconds) used to wait for a report during teardown executiontest_ids (
Optional
[dict
]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}tag (
Optional
[Dict
[str
,List
[str
]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed
- cleanup_and_skip(aux, info_to_print)[source]
Cleanup auxiliary and log reasons.
- Parameters
aux (
AuxiliaryInterface
) – corresponding auxiliary to abortinfo_to_print (
str
) – A message you want to print while cleaning up the test
- class pykiso.test_coordinator.test_suite.BasicTestSuite(modules_to_add_dir, test_filter_pattern, test_suite_id, args, kwargs)[source]
Inherit from the unittest framework test-suite but build it for our integration tests.
Initialize our custom unittest-test-suite.
Note
Will Load from the given path the integration test modules under test
Sort the given test case list by test suite/case id
Place Test suite setup and teardown respectively at top and bottom of test case list
Add sorted test case list to test suite
- class pykiso.test_coordinator.test_suite.BasicTestSuiteSetup(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)[source]
Inherit from unittest testCase and represent setup fixture.
Initialize Message Protocol / TestApp test-case.
- Parameters
test_suite_id (
int
) – test suite identification numbertest_case_id (
int
) – test case identification numberaux_list (
Optional
[List
[AuxiliaryInterface
]]) – list of used auxiliariessetup_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during setup executionrun_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during test_run executionteardown_timeout (
Optional
[int
]) – the maximum time (in seconds) used to wait for a report during teardown executiontest_ids (
Optional
[dict
]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}tag (
Optional
[Dict
[str
,List
[str
]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed
- class pykiso.test_coordinator.test_suite.BasicTestSuiteTeardown(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)[source]
Inherit from unittest testCase and represent teardown fixture.
Initialize Message Protocol / TestApp test-case.
- Parameters
test_suite_id (
int
) – test suite identification numbertest_case_id (
int
) – test case identification numberaux_list (
Optional
[List
[AuxiliaryInterface
]]) – list of used auxiliariessetup_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during setup executionrun_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during test_run executionteardown_timeout (
Optional
[int
]) – the maximum time (in seconds) used to wait for a report during teardown executiontest_ids (
Optional
[dict
]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}tag (
Optional
[Dict
[str
,List
[str
]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed
- class pykiso.test_coordinator.test_suite.RemoteTestSuiteSetup(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)[source]
Inherit from unittest testCase and represent setup fixture when Message Protocol / TestApp is used.
Initialize Message Protocol / TestApp test-case.
- Parameters
test_suite_id (
int
) – test suite identification numbertest_case_id (
int
) – test case identification numberaux_list (
Optional
[List
[AuxiliaryInterface
]]) – list of used auxiliariessetup_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during setup executionrun_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during test_run executionteardown_timeout (
Optional
[int
]) – the maximum time (in seconds) used to wait for a report during teardown executiontest_ids (
Optional
[dict
]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}tag (
Optional
[Dict
[str
,List
[str
]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed
- class pykiso.test_coordinator.test_suite.RemoteTestSuiteTeardown(test_suite_id, test_case_id, aux_list, setup_timeout, run_timeout, teardown_timeout, test_ids, tag, args, kwargs)[source]
Inherit from unittest testCase and represent teardown fixture when Message Protocol / TestApp is used.
Initialize Message Protocol / TestApp test-case.
- Parameters
test_suite_id (
int
) – test suite identification numbertest_case_id (
int
) – test case identification numberaux_list (
Optional
[List
[AuxiliaryInterface
]]) – list of used auxiliariessetup_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during setup executionrun_timeout (
Optional
[int
]) – maximum time (in seconds) used to wait for a report during test_run executionteardown_timeout (
Optional
[int
]) – the maximum time (in seconds) used to wait for a report during teardown executiontest_ids (
Optional
[dict
]) – jama references to get the coverage eg: {“Component1”: [“Req1”, “Req2”], “Component2”: [“Req3”]}tag (
Optional
[Dict
[str
,List
[str
]]]) – dictionary containing lists of variants and/or test levels when only a subset of tests needs to be executed
Test Execution
Test Execution
- module
test_execution
- synopsis
Execute a test environment based on the supplied configuration.
Note
Glob a list of test-suite folders
Generate a list of test-suites with a list of test-cases
Loop per suite
Gather result
- pykiso.test_coordinator.test_execution.apply_variant_filter(all_tests_to_run, variants, branch_levels)[source]
Filter the test cases based on the variant string.
- Parameters
all_tests_to_run (
dict
) – a dict containing all testsuites and testcasesvariants (
tuple
) – encapsulate user’s variant choicesbranch_levels (
tuple
) – encapsulate user’s branch level choices
- Return type
None
- pykiso.test_coordinator.test_execution.create_test_suite(test_suite_dict)[source]
create a test suite based on the config dict
- Parameters
test_suite_dict (
Dict
) – dict created from config with keys ‘suite_dir’, ‘test_filter_pattern’, ‘test_suite_id’- Return type
- pykiso.test_coordinator.test_execution.execute(config, report_type='text', variants=None, branch_levels=None, pattern_inject=None, failfast=False)[source]
create test environment base on config
- Parameters
config (
Dict
) – dict from converted YAML config filereport_type (
str
) – str to set the type of report wanted, i.e. test or junitvariants (
Optional
[tuple
]) – encapsulate user’s variant choicesbranch_levels (
Optional
[tuple
]) – encapsulate user’s branch level choicespattern_inject (
Optional
[str
]) – optional pattern that will override test_filter_pattern for all suites. Used in test development to run specific tests.failfast (
bool
) – Stop the test run on the first error or failure
- Return type
int
- Returns
exit code corresponding to the actual tets exeuciton run state(tests failed, unexpected exception…)
Test-Message Handling
Handle common communication with device under test
When using a Remote TestCase/TestSuite, the integration test framework handles
internal messaging and control flow using a message format defined in
pykiso.Message
. pykiso.test_message_handler
defines the messaging protocol from a behavioral point of view.
The general procedure is described in handle_basic_interaction context
manager, but specific _MsgHandler_ classes are provided with
TestCaseMsgHandler
and TestSuiteMsgHandler
to
provide shorthands for the specialised communication from
pykiso.test_case.RemoteTest
and
pykiso.test_suite.RemoteTestSuite
.
- module
test_message_handler
- synopsis
default communication between TestManagement and DUT.
- pykiso.test_coordinator.test_message_handler.handle_basic_interaction(test_entity, cmd_sub_type, timeout_cmd, timeout_resp)[source]
Handle default communication mechanism between test manager and device under test as follow:
TM | COMMAND ----------> | DUT TM | <---------- ACK | DUT TM | <---------- LOG | DUT TM | ACK ----------> | DUT ... TM | <---------- LOG | DUT TM | ACK ----------> | DUT TM | <---------- REPORT | DUT TM | ACK ----------> | DUT
This behaviour is implemented here.
Logs can be sent to TM while waiting for report.
- Parameters
test_entity (
Callable
) – test instance in use (BaseTestSuite, BasicTest,…)cmd_sub_type (
MessageCommandType
) – message command sub-type (Test case run, setup,….)timeout_cmd (
int
) – timeout in seconds for auxiliary run_commandtimeout_resp (
int
) – timeout in seconds for auxiliary wait_and_get_report
- Return type
List
[report_analysis
]- Returns
tuple containing current auxiliary, reported message, logging method to use, and pre-defined log message.
- class pykiso.test_coordinator.test_message_handler.report_analysis(current_auxiliary, report_message, logging_method, log_message)
Create new instance of report_analysis(current_auxiliary, report_message, logging_method, log_message)
- property current_auxiliary
Alias for field number 0
- property log_message
Alias for field number 3
- property logging_method
Alias for field number 2
- property report_message
Alias for field number 1
- pykiso.test_coordinator.test_message_handler.test_app_interaction(message_type, timeout_cmd=5)[source]
Handle test app basic interaction depending on the decorated method.
- Parameters
message_type (
MessageCommandType
) – message command sub-type (test case/suite run, setup, teardown….)timeout_cmd (
int
) – timeout in seconds for auxiliary run_command
- Return type
Callable
- Returns
inner decorator function
test xml result
test_xml_result
- module
test_xml_result
- synopsis
overwrite xmlrunner.result to be able to add additional data into the xml report.
- class pykiso.test_coordinator.test_xml_result.TestInfo(test_result, test_method, outcome=0, err=None, subTest=None, filename=None, lineno=None, doc=None)[source]
This class keeps useful information about the execution of a test method. Used by XmlTestResult
- Initialize the TestInfo class and append additional tag
that have to be stored for each test
- Parameters
test_result (
_XMLTestResult
) – test result classtest_method – test method (dynamically created eg: test_case.MyTest2-1-2)
outcome (
int
) – result of the test (SUCCESS, FAILURE, ERROR, SKIP)err – error cached during test
subTest – optional, refer the test id and the test description
filename (
Optional
[str
]) – name of the filelineno (
Optional
[bool
]) – store the test line numberdoc (
Optional
[str
]) – additional documentation to store
- class pykiso.test_coordinator.test_xml_result.XmlTestResult(stream=<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>, descriptions=True, verbosity=1, elapsed_times=True, properties=None, infoclass=<class 'pykiso.test_coordinator.test_xml_result.TestInfo'>)[source]
Test result class that can express test results in a XML report. Used by XMLTestRunner
Initialize the _XMLTestResult class.
- Parameters
stream (
TextIOWrapper
) – buffered text interface to a buffered raw streamdescriptions (
bool
) – include description of the testverbosity (
int
) – print output into the consoleelapsed_times (
bool
) – include the time spend on the testproperties – junit testsuite properties
infoclass (
_TestInfo
) – class containing the test information
- report_testcase(xml_testsuite, xml_document)
Appends a testcase section to the XML document.