[jira] [Commented] (CASSANDRA-17277) Automate updating tickets with CI results after merge

["Josh McKenzie (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-17277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17493385#comment-17493385 ] 

Josh McKenzie commented on CASSANDRA-17277:
-------------------------------------------

Ok, got something working locally with the following usage ergonomics:
{code:java}
usage: jenkins-jira-integration.py [-h] [--jenurl JENURL] --jenuser JENUSER --jenpass JENPASS --jiraurl JIRAURL --jirauser JIRAUSER --jirapass JIRAPASS --branch b
                                   [--buildnum n] [--verbose]

Parse Jenkins build output and update JIRA tickets with results for a single branch

required arguments:
  --jenurl JENURL      Jenkins server url
  --jenuser JENUSER    JIRA server url
  --jenpass JENPASS    JIRA username
  --jiraurl JIRAURL    JIRA server url
  --jirauser JIRAUSER  JIRA username
  --jirapass JIRAPASS  JIRA password
  --branch b           Branch Versions to pull Jenkins results for
  --buildnum n         Build number to process for CI status

optional arguments:
  --verbose            Verbose logging
{code}
The usage of the above can be triggered upon completion of a CI run in Jenkins that passes in the branch and number of the build; the rest of it is taken care of by the script.

The current data and format of results it'd post into a JIRA comment (I ran this against build 940 on trunk; arbitrary pick):

-----------------------------------------------------------------------

[CI Results]
Branch: trunk, build number: 940
jenkins url: [https://ci-cassandra.apache.org/job/Cassandra-trunk/940/]
JIRA: 17352
commit url: [https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=5c9ba06dd31157cd224af2cec75521fefe2c9883]
affected paths:
 * src/java/org/apache/cassandra/config/DatabaseDescriptor.java
 * src/java/org/apache/cassandra/cql3/functions/UDFunction.java
 * src/java/org/apache/cassandra/config/Config.java
 * src/java/org/apache/cassandra/cql3/functions/ThreadAwareSecurityManager.java

Build Result: UNSTABLE
Passing Tests: 42908
Failing Tests: 11
||Test|Failures|JIRA|
|dtest.cqlsh_tests.test_cqlsh_copy.TestCqlshCopy.test_explicit_column_order_reading|1 of 17|[No JIRA found|https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=496&quickFilter=2252]|
|org.apache.cassandra.distributed.upgrade.CompactStorageUpgradeTest.compactStorageUpgradeTest|4 of 17|[No JIRA found|https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=496&quickFilter=2252]|
|dtest.replica_side_filtering_test.TestSecondaryIndexes.test_complementary_deletion_with_limit_on_partition_key_column_with_empty_partitions|1 of 17|[No JIRA found|https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=496&quickFilter=2252]|
|dtest.snapshot_test.TestArchiveCommitlog.test_archive_commitlog_point_in_time_with_active_commitlog|1 of 17|[No JIRA found|https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=496&quickFilter=2252]|
|dtest.repair_tests.incremental_repair_test.TestIncRepair.test_repaired_tracking_with_partition_deletes|1 of 17|[No JIRA found|https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=496&quickFilter=2252]|
|dtest.paging_test.TestPagingDatasetChanges.test_data_change_impacting_later_page|1 of 17|[No JIRA found|https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=496&quickFilter=2252]|
|dtest.user_functions_test.TestUserFunctions.test_udf_with_udt|1 of 17|[No JIRA found|https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=496&quickFilter=2252]|
|org.apache.cassandra.db.commitlog.CommitLogSegmentManagerCDCTest.testNonblockingShouldMaintainSteadyDiskUsage|13 of 17|CASSANDRA-17233|

For an engineer that worked on a ticket (in this case CASSANDRA-17352), the implication is that special attention should be paid to any test failure with only 1 failure in the last N builds as this is the first time we've seen it so it could have been caused by this change. Notably, [the butler run for 940|https://butler.cassandra.apache.org/#/ci/upstream/compare/Cassandra-trunk/trunk] doesn't show the {{TestArchiveCommitlog}} failure, however in Jenkins if you go to all test failure results for the build [you see it there|https://ci-cassandra.apache.org/job/Cassandra-trunk/940/testReport/]

The above output is created by taking the passed in build number and walking back (and locally caching) the results of all valid previous builds. This local data is kept in a simple JSON file per branch which we then use for subsequent calculation of the # of runs we've seen a given test in and the # of failures; this is significantly faster (and less offensive) than repeatedly downloading 50+M of text data per build.

The inference of the Jira column is based on a query with summary ~ the last 2 tokens of the test name . delimited. If it finds more than 1 it should link to the same open test failures kanban board as linked when no Jira ticket is found. This functionality is of course limited to our consistency in having at least the last 2 tokens of a test failure name in the ticket summary, but that's standard operating procedure for the build lead role so shouldn't prove too brittle. In the event we have 2 failures it'll just redirect to the kanban board; some developer work required but minimal.

I'd _like_ to be able to have the test name link to the jenkins server for the test history, but that requires parsing out which test suite a given test comes from by its name alone; relatively trivial but something I could take as a follow-up to tinker with.

And last but not least, if this Jira inference proves consistent we could augment this script to automatically create Jira tickets upon finding test failures for which we don't yet have tickets.

> Automate updating tickets with CI results after merge
> -----------------------------------------------------
>
>                 Key: CASSANDRA-17277
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-17277
>             Project: Cassandra
>          Issue Type: Task
>            Reporter: Josh McKenzie
>            Assignee: Josh McKenzie
>            Priority: Normal
>
> From [the wiki article|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=199530280]
> {quote}ci-cassandra run bot updating ticket in JIRA w/state of test run for the SHA (JIRA Pending; to be linked)
> {quote}
> We already run CI for every commit (see [example|https://ci-cassandra.apache.org/job/Cassandra-trunk/]). The goal is to have automation that'll tie the results of a commit to the original JIRA ticket and update it automatically w/a comment indicating:
>  * The results of the CI run (duration, pass, fail, # failures)
>  * Any potential regressions in CI from the merge w/links to job history ([example|https://ci-cassandra.apache.org/job/Cassandra-trunk/912/testReport/junit/dtest.cqlsh_tests.test_cqlsh_copy/TestCqlshCopy/test_bulk_round_trip_with_timeouts/history/])
> From a workflow perspective we want to optimize for having as minimal friction as possible to see the impact of one's commit on ci-cassandra and rapidly verify if their change appears to have destabilized any tests on that infrastructure.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org