Re: Testing and jira tickets

[mck <mc...@apache.org>]

On 10 March 2017 at 05:51, Jason Brown <ja...@gmail.com> wrote:
> A nice convention we've stumbled into wrt to patches submitted via Jira is
> to post the results of unit test and dtest runs to the ticket (to show the
> patch doesn't break things). 
> [snip]
> As an example, should contributors/committers run dtests and unit tests on
> *some* machine (publicly available or otherwise), and then post those
> results to the ticket?


Yes please.
 I'm a supporter that nothing should get committed without it passing
 both unit and dtests.
 That any SHA in trunk or any release branch that fails unit or dtests
 is automatically uncommitted (reverted).

I was under the impression that the point of tick-tock was to move the
code towards a stable master approach. And that the lesson learn that
restricting any release to only have one patch version is: regardless of
how good the developers and CI system is; a pretty poor way of trying to
build a stable product. 

So from tick-tock to 4.0, I was really hoping it meant keeping all the
stable master and CI improvements obtained throughout the tick-tock
cycle while re-adding the discipline of ongoing patch versions to
supported releases. (While being realistic to resources available.)


Unfortunately without access to DS' cassci the best that i could do is
this:
  https://issues.apache.org/jira/browse/CASSANDRA-13307?focusedCommentId=15962001&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15962001


And running dtests on ASF's Jenkins was a 30hr turn around.   
:panda_face:
Is there any hope here for us that don't have access to cassci?

~mck

Re: Testing and jira tickets

[Michael Shuler <mi...@pbandjelly.org>]

On 04/12/2017 12:10 AM, mck wrote:
> 
> On 10 March 2017 at 05:51, Jason Brown <ja...@gmail.com> wrote:
>> A nice convention we've stumbled into wrt to patches submitted via Jira is
>> to post the results of unit test and dtest runs to the ticket (to show the
>> patch doesn't break things). 
>> [snip]
>> As an example, should contributors/committers run dtests and unit tests on
>> *some* machine (publicly available or otherwise), and then post those
>> results to the ticket?
> 
> 
> Yes please.
>  I'm a supporter that nothing should get committed without it passing
>  both unit and dtests.
>  That any SHA in trunk or any release branch that fails unit or dtests
>  is automatically uncommitted (reverted).
> 
> I was under the impression that the point of tick-tock was to move the
> code towards a stable master approach. And that the lesson learn that
> restricting any release to only have one patch version is: regardless of
> how good the developers and CI system is; a pretty poor way of trying to
> build a stable product. 
> 
> So from tick-tock to 4.0, I was really hoping it meant keeping all the
> stable master and CI improvements obtained throughout the tick-tock
> cycle while re-adding the discipline of ongoing patch versions to
> supported releases. (While being realistic to resources available.)
> 
> 
> Unfortunately without access to DS' cassci the best that i could do is
> this:
>   https://issues.apache.org/jira/browse/CASSANDRA-13307?focusedCommentId=15962001&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15962001
> 
> 
> And running dtests on ASF's Jenkins was a 30hr turn around.   
> :panda_face:
> Is there any hope here for us that don't have access to cassci?

Everything running the ASF Jenkins jobs is here:
https://git1-us-west.apache.org/repos/asf?p=cassandra-builds.git

Most of that was copied from what runs all the tests on cassci, with the
exception of all the cloud server things we ran on EC2 and OpenStack.

It should be possible with very minimal edits to the Job DSL groovy file
that sets up all the jobs:
https://git1-us-west.apache.org/repos/asf?p=cassandra-builds.git;a=blob;f=jenkins-dsl/cassandra_job_dsl_seed.groovy

..to set up a duplicate Jenkins instance, along with the same parameter
jobs to feed branches to. That Jenkins could be public or private,
depending on the organization. The only edits I know might be needed
would be slaveLabel, largeSlaveLabel, and possibly jdkLabel, but one
could also just mirror those in the Jenkins configuration.

I'd be happy to help anyone that needs advice on setting this up. The
ASF Jenkins is a quite minimal install with very few plugins, but one
would at least need the Job DSL plugin to get started:
https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin

A little trial and error would find any other plugins needed. Post build
task is used, too:
https://wiki.jenkins-ci.org/display/JENKINS/Post+build+task

If someone is interested in splitting up dtest and running it on
multiple machines, I can look into possibly providing some code
examples, but that work was internal and had some machine specific bits,
but is not crazy complex.

The dtest buckets are created with `nosetests --collect-only` and
comparing each test to the last test runs via the Jenkins api on cassci
to split the buckets up based on test run time. Each bucket was scp'ed
to the scratch servers to run. The nosetest.xml results from each
scratch server are fetched afterwards and we use xunitmerge to combine
them, so Jenkins could parse the results in a sane fashion.

That's the basics on how we split up and run dtests in parallel. Since
there's no way to do this on basic Jenkins slaves, they need to run
serially in the ASF Jenkins setup.

-- 
Warm regards,
Michael