[jira] [Commented] (QPIDIT-93) Optionally produce xUnit XML report with test results

["Kim van der Riet (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/QPIDIT-93?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464023#comment-16464023 ] 

Kim van der Riet commented on QPIDIT-93:
----------------------------------------

A more comprehensive xUnit XML schema is available here

[https://github.com/windyroad/JUnit-Schema]

and which includes a <{{properties>}} element containing 0..n <{{property>}} elements, each of which contains a name/value pair of attributes. This can be used to add additional information about the test in a compliant way as follows:
{noformat}
<testsuite ...>
  <properties>
    <property name="..." value="..."/>
    <property name="..." value="..."/>
    <property name="..." value="..."/>
    ...
  </properties>
  <testcase ... />
  <testcase ... />
  <testcase ... />
  ...
</testsuite>{noformat}
The question is, what information should be included?

A minimum list should be:
 * test machine identity
 * test executable path/name
 * parameters to test executable
 * broker topology (single-node, Sender->B1->B2->B3->Receiver, etc)
 * broker ip address(es)
 * broker configuration(s) - how to achieve this?

In addition, nice-to-have might include:
 * Date/timestamp/duration for test
 * Additional test description as passed on command-line
 * Broker connection properties (if necessary, for both sender and receiver if different)

Each {{<testcase>}} element supports the following attributes:
 * name
 * classname
 * time

but I would like to add the following out-of-band attributes:
 * receiver-client
 * sender-client
 * type
 * duration of individual testcase
 * any other specialized parameter used in individual tests:
 * subtype (amqp_complex_types_test)
 * size (amqp_large_content_test)
 * headers (jms_hdrs_props_test)
 * properties (jms_hdrs_props_test)

Ideas?

> Optionally produce xUnit XML report with test results
> -----------------------------------------------------
>
>                 Key: QPIDIT-93
>                 URL: https://issues.apache.org/jira/browse/QPIDIT-93
>             Project: Apache QPID Interoperability Test Suite
>          Issue Type: Improvement
>    Affects Versions: 0.1.0
>            Reporter: Jiri Daněk
>            Assignee: Kim van der Riet
>            Priority: Minor
>             Fix For: 0.2.0
>
>         Attachments: amqp_types_test.2018-01-26T09-24-24.xml, jms_hdrs_props_test.2018-01-26T09-45-29.xml, xunit.xsd
>
>
> Something like the following should do the job. Instead of https://pypi.python.org/pypi/unittest-xml-reporting, it might be possible to maybe switch from unittest package to py.test, which has xUnit reports built in.
> {noformat}
> diff --git a/src/python/qpid_interop_test/jms_messages_test.py b/src/python/qpid_interop_test/jms_messages_test.py
> index 3b54510..3b94f3a 100755
> --- a/src/python/qpid_interop_test/jms_messages_test.py
> +++ b/src/python/qpid_interop_test/jms_messages_test.py
> @@ -24,19 +24,17 @@ Module to test JMS message types across different APIs
>  from json import dumps
>  from os import getenv, path
> +import xmlrunner
>  from proton import symbol
>  from qpid_interop_test.test_type_map import TestTypeMap
> @@ -302,7 +300,7 @@ class TestOptions(object):
>      Class controlling command-line arguments used to control the test.
>      """
>      def __init__(self, shim_map):
> -        parser = argparse.ArgumentParser(description='Qpid-interop AMQP client interoparability test suite '
> +        parser = argparse.ArgumentParser(description='Qpid-interop AMQP client interoperability test suite '
>                                           'for JMS message types')
>          parser.add_argument('--sender', action='store', default='localhost:5672', metavar='IP-ADDR:PORT',
>                              help='Node to which test suite will send messages.')
> @@ -313,6 +311,8 @@ class TestOptions(object):
>          parser.add_argument('--broker-type', action='store', metavar='BROKER_NAME',
>                              help='Disable test of broker type (using connection properties) by specifying the broker' +
>                              ' name, or "None".')
> +        parser.add_argument('--xunit-report-dir', action='store', metavar='REPORTS_DIRECTORY',
> +                            help='Generate xUnit report into REPORTS_DIRECTORY/TEST-<test_name>-<timestamp>.xml')
>          type_group = parser.add_mutually_exclusive_group()
>          type_group.add_argument('--include-type', action='append', metavar='JMS_MESSAGE-TYPE',
>                                  help='Name of JMS message type to include. Supported types:\n%s' %
> @@ -421,6 +421,7 @@ if __name__ == '__main__':
>              TEST_SUITE.addTest(unittest.makeSuite(test_case_class))
>  
>      # Finally, run all the dynamically created tests
> -    RES = unittest.TextTestRunner(verbosity=2).run(TEST_SUITE)
> +#    RES = unittest.TextTestRunner(verbosity=2).run(TEST_SUITE)  # type: unittest.TextTestResult
> +    RES = xmlrunner.XMLTestRunner(verbosity=2).run(TEST_SUITE)  # type: unittest.TextTestResult
>      if not RES.wasSuccessful():
>          sys.exit(1)
> {noformat}
> The main "problem" seems that each test has completely separate main method and builds up options from scratch. Adding this to every test would mean some unnecessary duplication. So maybe refactor this somewhat at first? On the other hand, there are good reasons to keep the tests simple...
> What do you think?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org