Proposal for alteration of criteria for automatic patch verification

[Ian Maxon <im...@uci.edu>]

Hi all,
I'd like to field the idea of changing the criteria we use to allow Jenkins
to give +1 verified on patch sets. Right now, it runs 'mvn verify', which
runs darn near about every test we have. While this was somewhat OK in the
past, I think now, and going forward, it takes a long time, and it will
only take longer as we make the integration tests more thorough.

Additionally, I'd like to expand and re-enable the automated vagrant-based
cluster integration tests, and unfortunately there isn't a simple way to
make these work inside a docker container. I can get it to work by hand,
but to make it automated I would have to fork or submit a patch to the
Jenkins Docker cloud plugin, and it might not be a small or particularly
clean/easy patch either.

I'd like to change the goal that's run in the automatic tests on every
commit to 'mvn test' rather than 'mvn verify'. This will still run all of
the ExecutionTest(s), and the optimizer tests, and so on. It will stop the
recovery tests and asterix installer tests from being run every time,
however.

However these tests are still certainly valuable, so I'd run them on a
timer, perhaps daily, so that we would still catch bugs that these tests
reveal rather promptly. Due to the fact we only commit a single digit
number of times to master each day generally, the granularity of which
commit will have introduced a bug will hopefully be rather low.

Thoughts/comments?

Thanks,
-Ian

Re: Proposal for alteration of criteria for automatic patch verification

[Ian Maxon <im...@uci.edu>]

We're using 'mvn test' now so all the patches should complete in about 10
minutes rather than closer to an hour. The integration test seems to be
working fine, I'll check to make sure it's triggered on the next push to
master.

On Thu, Jan 14, 2016 at 3:07 PM, Ian Maxon <im...@uci.edu> wrote:

> We can definitely have a job that will take a commit as a parameter that
> is run simply on demand rather than via any sort of trigger or timer. I
> have had a job for this actually at various points but I don't know if the
> ones I have right now are pointed at the correct repositories.
>
>
> For the time being, to start impelementing this since there seems to be no
> objection, I have set up a job to run on each commit. If it looks like it's
> working good after the next commit I will change AsterixDB to use 'mvn
> test' instead of 'mvn verify' for automatic verification.
>
> -Ian
>
> On Wed, Jan 13, 2016 at 4:27 PM, Till Westmann <ti...@apache.org> wrote:
>
>> I think that to find out quickly which change introduces the issue, it
>> would be nice if there was an easy way to have Jenkins run the “long suite”
>> for a specific commit. That way we could identify the problematic commit in
>> the standardized environment and without relying on developer setup.
>>
>>
>> On 13 Jan 2016, at 16:20, Mike Carey wrote:
>>
>> Agreed.  I'd vote for making sure that the broken daily test report be
>>> accompanied by a list of the changes that occurred since the last
>>> successful run - and that it be the responsibility of all folks with
>>> changes on that list to gang up and ensure that the problem is identified
>>> and resolved in a timely fashion.  We should probably spell out what that
>>> looks like, but it seems eminently reasonable....
>>>
>>> On 1/13/16 4:03 PM, Chris Hillery wrote:
>>>
>>>> +1 for Ian's initial proposal. This kind of pre-commit validation is
>>>> always
>>>> a balancing act between being comprehensive and being timely, and I
>>>> think
>>>> sticking with the distinction between "mvn test" and "mvn verify" is a
>>>> clean and simple way to divide the tests.
>>>>
>>>> Given that pushing the Gerrit head to ASF is manual anyway, it wouldn't
>>>> be
>>>> TOO much headache to also enact Ildar's suggestion - basically, whoever
>>>> does the ASF push would need to wait to ensure each commit passes the
>>>> corresponding daily tests. However, IMHO at least this is probably not
>>>> necessary. If a bug is uncovered in the daily tests, we'll need to push
>>>> an
>>>> additional commit to fix it, and that additional commit (plus the
>>>> original
>>>> buggy commit) will still end up on ASF anyway. And I worry a bit about
>>>> anything that would cause the ASF Github mirror to stay out of sync with
>>>> Gerrit for very long.
>>>>
>>>> Ceej
>>>> aka Chris Hillery
>>>>
>>>> On Wed, Jan 13, 2016 at 3:42 PM, Ildar Absalyamov <
>>>> ildar.absalyamov@gmail.com> wrote:
>>>>
>>>> It would be really nice to break Jenkins test build into two stage
>>>>> process: ‘mvn test’ on each patch set submission and ‘mvn verify’
>>>>> after the
>>>>> patch received +2 and got merged into Gerrit, BUT before it got merged
>>>>> to
>>>>> ASF repo. Although I am not sure what should be done in a case when the
>>>>> latter build does not pass, since it’s already merged into Gerrit and
>>>>> we
>>>>> cannot revert that.
>>>>>
>>>>> On Jan 13, 2016, at 15:34, Ian Maxon <im...@uci.edu> wrote:
>>>>>>
>>>>>> Hi all,
>>>>>> I'd like to field the idea of changing the criteria we use to allow
>>>>>>
>>>>> Jenkins
>>>>>
>>>>>> to give +1 verified on patch sets. Right now, it runs 'mvn verify',
>>>>>> which
>>>>>> runs darn near about every test we have. While this was somewhat OK in
>>>>>>
>>>>> the
>>>>>
>>>>>> past, I think now, and going forward, it takes a long time, and it
>>>>>> will
>>>>>> only take longer as we make the integration tests more thorough.
>>>>>>
>>>>>> Additionally, I'd like to expand and re-enable the automated
>>>>>>
>>>>> vagrant-based
>>>>>
>>>>>> cluster integration tests, and unfortunately there isn't a simple way
>>>>>> to
>>>>>> make these work inside a docker container. I can get it to work by
>>>>>> hand,
>>>>>> but to make it automated I would have to fork or submit a patch to the
>>>>>> Jenkins Docker cloud plugin, and it might not be a small or
>>>>>> particularly
>>>>>> clean/easy patch either.
>>>>>>
>>>>>> I'd like to change the goal that's run in the automatic tests on every
>>>>>> commit to 'mvn test' rather than 'mvn verify'. This will still run
>>>>>> all of
>>>>>> the ExecutionTest(s), and the optimizer tests, and so on. It will stop
>>>>>>
>>>>> the
>>>>>
>>>>>> recovery tests and asterix installer tests from being run every time,
>>>>>> however.
>>>>>>
>>>>>> However these tests are still certainly valuable, so I'd run them on a
>>>>>> timer, perhaps daily, so that we would still catch bugs that these
>>>>>> tests
>>>>>> reveal rather promptly. Due to the fact we only commit a single digit
>>>>>> number of times to master each day generally, the granularity of which
>>>>>> commit will have introduced a bug will hopefully be rather low.
>>>>>>
>>>>>> Thoughts/comments?
>>>>>>
>>>>>> Thanks,
>>>>>> -Ian
>>>>>>
>>>>> Best regards,
>>>>> Ildar
>>>>>
>>>>>
>>>>>
>

Re: Proposal for alteration of criteria for automatic patch verification

[Ian Maxon <im...@uci.edu>]

We can definitely have a job that will take a commit as a parameter that is
run simply on demand rather than via any sort of trigger or timer. I have
had a job for this actually at various points but I don't know if the ones
I have right now are pointed at the correct repositories.


For the time being, to start impelementing this since there seems to be no
objection, I have set up a job to run on each commit. If it looks like it's
working good after the next commit I will change AsterixDB to use 'mvn
test' instead of 'mvn verify' for automatic verification.

-Ian

On Wed, Jan 13, 2016 at 4:27 PM, Till Westmann <ti...@apache.org> wrote:

> I think that to find out quickly which change introduces the issue, it
> would be nice if there was an easy way to have Jenkins run the “long suite”
> for a specific commit. That way we could identify the problematic commit in
> the standardized environment and without relying on developer setup.
>
>
> On 13 Jan 2016, at 16:20, Mike Carey wrote:
>
> Agreed.  I'd vote for making sure that the broken daily test report be
>> accompanied by a list of the changes that occurred since the last
>> successful run - and that it be the responsibility of all folks with
>> changes on that list to gang up and ensure that the problem is identified
>> and resolved in a timely fashion.  We should probably spell out what that
>> looks like, but it seems eminently reasonable....
>>
>> On 1/13/16 4:03 PM, Chris Hillery wrote:
>>
>>> +1 for Ian's initial proposal. This kind of pre-commit validation is
>>> always
>>> a balancing act between being comprehensive and being timely, and I think
>>> sticking with the distinction between "mvn test" and "mvn verify" is a
>>> clean and simple way to divide the tests.
>>>
>>> Given that pushing the Gerrit head to ASF is manual anyway, it wouldn't
>>> be
>>> TOO much headache to also enact Ildar's suggestion - basically, whoever
>>> does the ASF push would need to wait to ensure each commit passes the
>>> corresponding daily tests. However, IMHO at least this is probably not
>>> necessary. If a bug is uncovered in the daily tests, we'll need to push
>>> an
>>> additional commit to fix it, and that additional commit (plus the
>>> original
>>> buggy commit) will still end up on ASF anyway. And I worry a bit about
>>> anything that would cause the ASF Github mirror to stay out of sync with
>>> Gerrit for very long.
>>>
>>> Ceej
>>> aka Chris Hillery
>>>
>>> On Wed, Jan 13, 2016 at 3:42 PM, Ildar Absalyamov <
>>> ildar.absalyamov@gmail.com> wrote:
>>>
>>> It would be really nice to break Jenkins test build into two stage
>>>> process: ‘mvn test’ on each patch set submission and ‘mvn verify’ after
>>>> the
>>>> patch received +2 and got merged into Gerrit, BUT before it got merged
>>>> to
>>>> ASF repo. Although I am not sure what should be done in a case when the
>>>> latter build does not pass, since it’s already merged into Gerrit and we
>>>> cannot revert that.
>>>>
>>>> On Jan 13, 2016, at 15:34, Ian Maxon <im...@uci.edu> wrote:
>>>>>
>>>>> Hi all,
>>>>> I'd like to field the idea of changing the criteria we use to allow
>>>>>
>>>> Jenkins
>>>>
>>>>> to give +1 verified on patch sets. Right now, it runs 'mvn verify',
>>>>> which
>>>>> runs darn near about every test we have. While this was somewhat OK in
>>>>>
>>>> the
>>>>
>>>>> past, I think now, and going forward, it takes a long time, and it will
>>>>> only take longer as we make the integration tests more thorough.
>>>>>
>>>>> Additionally, I'd like to expand and re-enable the automated
>>>>>
>>>> vagrant-based
>>>>
>>>>> cluster integration tests, and unfortunately there isn't a simple way
>>>>> to
>>>>> make these work inside a docker container. I can get it to work by
>>>>> hand,
>>>>> but to make it automated I would have to fork or submit a patch to the
>>>>> Jenkins Docker cloud plugin, and it might not be a small or
>>>>> particularly
>>>>> clean/easy patch either.
>>>>>
>>>>> I'd like to change the goal that's run in the automatic tests on every
>>>>> commit to 'mvn test' rather than 'mvn verify'. This will still run all
>>>>> of
>>>>> the ExecutionTest(s), and the optimizer tests, and so on. It will stop
>>>>>
>>>> the
>>>>
>>>>> recovery tests and asterix installer tests from being run every time,
>>>>> however.
>>>>>
>>>>> However these tests are still certainly valuable, so I'd run them on a
>>>>> timer, perhaps daily, so that we would still catch bugs that these
>>>>> tests
>>>>> reveal rather promptly. Due to the fact we only commit a single digit
>>>>> number of times to master each day generally, the granularity of which
>>>>> commit will have introduced a bug will hopefully be rather low.
>>>>>
>>>>> Thoughts/comments?
>>>>>
>>>>> Thanks,
>>>>> -Ian
>>>>>
>>>> Best regards,
>>>> Ildar
>>>>
>>>>
>>>>

Re: Proposal for alteration of criteria for automatic patch verification

[Till Westmann <ti...@apache.org>]

I think that to find out quickly which change introduces the issue, it 
would be nice if there was an easy way to have Jenkins run the “long 
suite” for a specific commit. That way we could identify the 
problematic commit in the standardized environment and without relying 
on developer setup.

On 13 Jan 2016, at 16:20, Mike Carey wrote:

> Agreed.  I'd vote for making sure that the broken daily test report be 
> accompanied by a list of the changes that occurred since the last 
> successful run - and that it be the responsibility of all folks with 
> changes on that list to gang up and ensure that the problem is 
> identified and resolved in a timely fashion.  We should probably spell 
> out what that looks like, but it seems eminently reasonable....
>
> On 1/13/16 4:03 PM, Chris Hillery wrote:
>> +1 for Ian's initial proposal. This kind of pre-commit validation is 
>> always
>> a balancing act between being comprehensive and being timely, and I 
>> think
>> sticking with the distinction between "mvn test" and "mvn verify" is 
>> a
>> clean and simple way to divide the tests.
>>
>> Given that pushing the Gerrit head to ASF is manual anyway, it 
>> wouldn't be
>> TOO much headache to also enact Ildar's suggestion - basically, 
>> whoever
>> does the ASF push would need to wait to ensure each commit passes the
>> corresponding daily tests. However, IMHO at least this is probably 
>> not
>> necessary. If a bug is uncovered in the daily tests, we'll need to 
>> push an
>> additional commit to fix it, and that additional commit (plus the 
>> original
>> buggy commit) will still end up on ASF anyway. And I worry a bit 
>> about
>> anything that would cause the ASF Github mirror to stay out of sync 
>> with
>> Gerrit for very long.
>>
>> Ceej
>> aka Chris Hillery
>>
>> On Wed, Jan 13, 2016 at 3:42 PM, Ildar Absalyamov <
>> ildar.absalyamov@gmail.com> wrote:
>>
>>> It would be really nice to break Jenkins test build into two stage
>>> process: ‘mvn test’ on each patch set submission and ‘mvn 
>>> verify’ after the
>>> patch received +2 and got merged into Gerrit, BUT before it got 
>>> merged to
>>> ASF repo. Although I am not sure what should be done in a case when 
>>> the
>>> latter build does not pass, since it’s already merged into Gerrit 
>>> and we
>>> cannot revert that.
>>>
>>>> On Jan 13, 2016, at 15:34, Ian Maxon <im...@uci.edu> wrote:
>>>>
>>>> Hi all,
>>>> I'd like to field the idea of changing the criteria we use to allow
>>> Jenkins
>>>> to give +1 verified on patch sets. Right now, it runs 'mvn verify', 
>>>> which
>>>> runs darn near about every test we have. While this was somewhat OK 
>>>> in
>>> the
>>>> past, I think now, and going forward, it takes a long time, and it 
>>>> will
>>>> only take longer as we make the integration tests more thorough.
>>>>
>>>> Additionally, I'd like to expand and re-enable the automated
>>> vagrant-based
>>>> cluster integration tests, and unfortunately there isn't a simple 
>>>> way to
>>>> make these work inside a docker container. I can get it to work by 
>>>> hand,
>>>> but to make it automated I would have to fork or submit a patch to 
>>>> the
>>>> Jenkins Docker cloud plugin, and it might not be a small or 
>>>> particularly
>>>> clean/easy patch either.
>>>>
>>>> I'd like to change the goal that's run in the automatic tests on 
>>>> every
>>>> commit to 'mvn test' rather than 'mvn verify'. This will still run 
>>>> all of
>>>> the ExecutionTest(s), and the optimizer tests, and so on. It will 
>>>> stop
>>> the
>>>> recovery tests and asterix installer tests from being run every 
>>>> time,
>>>> however.
>>>>
>>>> However these tests are still certainly valuable, so I'd run them 
>>>> on a
>>>> timer, perhaps daily, so that we would still catch bugs that these 
>>>> tests
>>>> reveal rather promptly. Due to the fact we only commit a single 
>>>> digit
>>>> number of times to master each day generally, the granularity of 
>>>> which
>>>> commit will have introduced a bug will hopefully be rather low.
>>>>
>>>> Thoughts/comments?
>>>>
>>>> Thanks,
>>>> -Ian
>>> Best regards,
>>> Ildar
>>>
>>>

Re: Proposal for alteration of criteria for automatic patch verification

[Mike Carey <dt...@gmail.com>]

Agreed.  I'd vote for making sure that the broken daily test report be 
accompanied by a list of the changes that occurred since the last 
successful run - and that it be the responsibility of all folks with 
changes on that list to gang up and ensure that the problem is 
identified and resolved in a timely fashion.  We should probably spell 
out what that looks like, but it seems eminently reasonable....

On 1/13/16 4:03 PM, Chris Hillery wrote:
> +1 for Ian's initial proposal. This kind of pre-commit validation is always
> a balancing act between being comprehensive and being timely, and I think
> sticking with the distinction between "mvn test" and "mvn verify" is a
> clean and simple way to divide the tests.
>
> Given that pushing the Gerrit head to ASF is manual anyway, it wouldn't be
> TOO much headache to also enact Ildar's suggestion - basically, whoever
> does the ASF push would need to wait to ensure each commit passes the
> corresponding daily tests. However, IMHO at least this is probably not
> necessary. If a bug is uncovered in the daily tests, we'll need to push an
> additional commit to fix it, and that additional commit (plus the original
> buggy commit) will still end up on ASF anyway. And I worry a bit about
> anything that would cause the ASF Github mirror to stay out of sync with
> Gerrit for very long.
>
> Ceej
> aka Chris Hillery
>
> On Wed, Jan 13, 2016 at 3:42 PM, Ildar Absalyamov <
> ildar.absalyamov@gmail.com> wrote:
>
>> It would be really nice to break Jenkins test build into two stage
>> process: ‘mvn test’ on each patch set submission and ‘mvn verify’ after the
>> patch received +2 and got merged into Gerrit, BUT before it got merged to
>> ASF repo. Although I am not sure what should be done in a case when the
>> latter build does not pass, since it’s already merged into Gerrit and we
>> cannot revert that.
>>
>>> On Jan 13, 2016, at 15:34, Ian Maxon <im...@uci.edu> wrote:
>>>
>>> Hi all,
>>> I'd like to field the idea of changing the criteria we use to allow
>> Jenkins
>>> to give +1 verified on patch sets. Right now, it runs 'mvn verify', which
>>> runs darn near about every test we have. While this was somewhat OK in
>> the
>>> past, I think now, and going forward, it takes a long time, and it will
>>> only take longer as we make the integration tests more thorough.
>>>
>>> Additionally, I'd like to expand and re-enable the automated
>> vagrant-based
>>> cluster integration tests, and unfortunately there isn't a simple way to
>>> make these work inside a docker container. I can get it to work by hand,
>>> but to make it automated I would have to fork or submit a patch to the
>>> Jenkins Docker cloud plugin, and it might not be a small or particularly
>>> clean/easy patch either.
>>>
>>> I'd like to change the goal that's run in the automatic tests on every
>>> commit to 'mvn test' rather than 'mvn verify'. This will still run all of
>>> the ExecutionTest(s), and the optimizer tests, and so on. It will stop
>> the
>>> recovery tests and asterix installer tests from being run every time,
>>> however.
>>>
>>> However these tests are still certainly valuable, so I'd run them on a
>>> timer, perhaps daily, so that we would still catch bugs that these tests
>>> reveal rather promptly. Due to the fact we only commit a single digit
>>> number of times to master each day generally, the granularity of which
>>> commit will have introduced a bug will hopefully be rather low.
>>>
>>> Thoughts/comments?
>>>
>>> Thanks,
>>> -Ian
>> Best regards,
>> Ildar
>>
>>

Re: Proposal for alteration of criteria for automatic patch verification

[Chris Hillery <ch...@hillery.land>]

+1 for Ian's initial proposal. This kind of pre-commit validation is always
a balancing act between being comprehensive and being timely, and I think
sticking with the distinction between "mvn test" and "mvn verify" is a
clean and simple way to divide the tests.

Given that pushing the Gerrit head to ASF is manual anyway, it wouldn't be
TOO much headache to also enact Ildar's suggestion - basically, whoever
does the ASF push would need to wait to ensure each commit passes the
corresponding daily tests. However, IMHO at least this is probably not
necessary. If a bug is uncovered in the daily tests, we'll need to push an
additional commit to fix it, and that additional commit (plus the original
buggy commit) will still end up on ASF anyway. And I worry a bit about
anything that would cause the ASF Github mirror to stay out of sync with
Gerrit for very long.

Ceej
aka Chris Hillery

On Wed, Jan 13, 2016 at 3:42 PM, Ildar Absalyamov <
ildar.absalyamov@gmail.com> wrote:

> It would be really nice to break Jenkins test build into two stage
> process: ‘mvn test’ on each patch set submission and ‘mvn verify’ after the
> patch received +2 and got merged into Gerrit, BUT before it got merged to
> ASF repo. Although I am not sure what should be done in a case when the
> latter build does not pass, since it’s already merged into Gerrit and we
> cannot revert that.
>
> > On Jan 13, 2016, at 15:34, Ian Maxon <im...@uci.edu> wrote:
> >
> > Hi all,
> > I'd like to field the idea of changing the criteria we use to allow
> Jenkins
> > to give +1 verified on patch sets. Right now, it runs 'mvn verify', which
> > runs darn near about every test we have. While this was somewhat OK in
> the
> > past, I think now, and going forward, it takes a long time, and it will
> > only take longer as we make the integration tests more thorough.
> >
> > Additionally, I'd like to expand and re-enable the automated
> vagrant-based
> > cluster integration tests, and unfortunately there isn't a simple way to
> > make these work inside a docker container. I can get it to work by hand,
> > but to make it automated I would have to fork or submit a patch to the
> > Jenkins Docker cloud plugin, and it might not be a small or particularly
> > clean/easy patch either.
> >
> > I'd like to change the goal that's run in the automatic tests on every
> > commit to 'mvn test' rather than 'mvn verify'. This will still run all of
> > the ExecutionTest(s), and the optimizer tests, and so on. It will stop
> the
> > recovery tests and asterix installer tests from being run every time,
> > however.
> >
> > However these tests are still certainly valuable, so I'd run them on a
> > timer, perhaps daily, so that we would still catch bugs that these tests
> > reveal rather promptly. Due to the fact we only commit a single digit
> > number of times to master each day generally, the granularity of which
> > commit will have introduced a bug will hopefully be rather low.
> >
> > Thoughts/comments?
> >
> > Thanks,
> > -Ian
>
> Best regards,
> Ildar
>
>

Re: Proposal for alteration of criteria for automatic patch verification

[Ian Maxon <im...@uci.edu>]

True, doing it on every merge to master is also an option. I don't think we
should try to discriminate based on which repo it was pushed to, though. If
something were to be wrong I think the correct approach would not be to
rewrite history. That just seems un-necessary, a revert commit should
totally suffice.

On Wed, Jan 13, 2016 at 3:42 PM, Ildar Absalyamov <
ildar.absalyamov@gmail.com> wrote:

> It would be really nice to break Jenkins test build into two stage
> process: ‘mvn test’ on each patch set submission and ‘mvn verify’ after the
> patch received +2 and got merged into Gerrit, BUT before it got merged to
> ASF repo. Although I am not sure what should be done in a case when the
> latter build does not pass, since it’s already merged into Gerrit and we
> cannot revert that.
>
> > On Jan 13, 2016, at 15:34, Ian Maxon <im...@uci.edu> wrote:
> >
> > Hi all,
> > I'd like to field the idea of changing the criteria we use to allow
> Jenkins
> > to give +1 verified on patch sets. Right now, it runs 'mvn verify', which
> > runs darn near about every test we have. While this was somewhat OK in
> the
> > past, I think now, and going forward, it takes a long time, and it will
> > only take longer as we make the integration tests more thorough.
> >
> > Additionally, I'd like to expand and re-enable the automated
> vagrant-based
> > cluster integration tests, and unfortunately there isn't a simple way to
> > make these work inside a docker container. I can get it to work by hand,
> > but to make it automated I would have to fork or submit a patch to the
> > Jenkins Docker cloud plugin, and it might not be a small or particularly
> > clean/easy patch either.
> >
> > I'd like to change the goal that's run in the automatic tests on every
> > commit to 'mvn test' rather than 'mvn verify'. This will still run all of
> > the ExecutionTest(s), and the optimizer tests, and so on. It will stop
> the
> > recovery tests and asterix installer tests from being run every time,
> > however.
> >
> > However these tests are still certainly valuable, so I'd run them on a
> > timer, perhaps daily, so that we would still catch bugs that these tests
> > reveal rather promptly. Due to the fact we only commit a single digit
> > number of times to master each day generally, the granularity of which
> > commit will have introduced a bug will hopefully be rather low.
> >
> > Thoughts/comments?
> >
> > Thanks,
> > -Ian
>
> Best regards,
> Ildar
>
>

Re: Proposal for alteration of criteria for automatic patch verification

[Ildar Absalyamov <il...@gmail.com>]

It would be really nice to break Jenkins test build into two stage process: ‘mvn test’ on each patch set submission and ‘mvn verify’ after the patch received +2 and got merged into Gerrit, BUT before it got merged to ASF repo. Although I am not sure what should be done in a case when the latter build does not pass, since it’s already merged into Gerrit and we cannot revert that.

> On Jan 13, 2016, at 15:34, Ian Maxon <im...@uci.edu> wrote:
> 
> Hi all,
> I'd like to field the idea of changing the criteria we use to allow Jenkins
> to give +1 verified on patch sets. Right now, it runs 'mvn verify', which
> runs darn near about every test we have. While this was somewhat OK in the
> past, I think now, and going forward, it takes a long time, and it will
> only take longer as we make the integration tests more thorough.
> 
> Additionally, I'd like to expand and re-enable the automated vagrant-based
> cluster integration tests, and unfortunately there isn't a simple way to
> make these work inside a docker container. I can get it to work by hand,
> but to make it automated I would have to fork or submit a patch to the
> Jenkins Docker cloud plugin, and it might not be a small or particularly
> clean/easy patch either.
> 
> I'd like to change the goal that's run in the automatic tests on every
> commit to 'mvn test' rather than 'mvn verify'. This will still run all of
> the ExecutionTest(s), and the optimizer tests, and so on. It will stop the
> recovery tests and asterix installer tests from being run every time,
> however.
> 
> However these tests are still certainly valuable, so I'd run them on a
> timer, perhaps daily, so that we would still catch bugs that these tests
> reveal rather promptly. Due to the fact we only commit a single digit
> number of times to master each day generally, the granularity of which
> commit will have introduced a bug will hopefully be rather low.
> 
> Thoughts/comments?
> 
> Thanks,
> -Ian

Best regards,
Ildar

Re: Proposal for alteration of criteria for automatic patch verification

[Mike Carey <dt...@gmail.com>]

+100

It's too bad that Docker wouldn't give in - but - this is a long-overdue 
change to a more leveled testing approach, so I think now's a great time 
to make this change (i.e., this is a fine camel-back-breaking straw).

:-)

On 1/13/16 3:34 PM, Ian Maxon wrote:
> Hi all,
> I'd like to field the idea of changing the criteria we use to allow Jenkins
> to give +1 verified on patch sets. Right now, it runs 'mvn verify', which
> runs darn near about every test we have. While this was somewhat OK in the
> past, I think now, and going forward, it takes a long time, and it will
> only take longer as we make the integration tests more thorough.
>
> Additionally, I'd like to expand and re-enable the automated vagrant-based
> cluster integration tests, and unfortunately there isn't a simple way to
> make these work inside a docker container. I can get it to work by hand,
> but to make it automated I would have to fork or submit a patch to the
> Jenkins Docker cloud plugin, and it might not be a small or particularly
> clean/easy patch either.
>
> I'd like to change the goal that's run in the automatic tests on every
> commit to 'mvn test' rather than 'mvn verify'. This will still run all of
> the ExecutionTest(s), and the optimizer tests, and so on. It will stop the
> recovery tests and asterix installer tests from being run every time,
> however.
>
> However these tests are still certainly valuable, so I'd run them on a
> timer, perhaps daily, so that we would still catch bugs that these tests
> reveal rather promptly. Due to the fact we only commit a single digit
> number of times to master each day generally, the granularity of which
> commit will have introduced a bug will hopefully be rather low.
>
> Thoughts/comments?
>
> Thanks,
> -Ian
>