You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Brahma Reddy Battula <br...@huawei.com> on 2017/09/19 13:48:57 UTC
RE: Trunk fails
Can we run "mvn install" and "compile" for all the modules after applying the patch(we can skip shadeclients)..?
AS today also we seen "mvn install" issue after HADOOP-14771.
--Brahma Reddy Battula
-----Original Message-----
From: Allen Wittenauer [mailto:aw@effectivemachines.com]
Sent: 14 August 2017 22:09
To: Brahma Reddy Battula
Cc: common-dev@hadoop.apache.org
Subject: Re: Trunk fails
> On Aug 14, 2017, at 5:36 AM, Brahma Reddy Battula <br...@huawei.com> wrote:
>
> How about let this comment on Jira if there is any failure(compile/Test)..? so that corresponding Jira reporter/committer can look into it(can reduce/avoid pre-commit failures..?).
>
hadoop-commit-trunk does. We can see that in this particular case very clearly:
After the commit: https://s.apache.org/ufhf
After the revert: https://s.apache.org/3GbA
Old timers will remember when this job wasn’t reliable. They might still believe it isn’t, but that hasn’t been true for a years. It’s definitely in the 99%+ for accuracy (at least for non-native on Linux).
I don’t have the QBT report publishing to JIRA because it never passes. Hadoop’s unit tests are too unreliable for that.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Trunk fails
Posted by Chris Douglas <ch...@gmail.com>.
On Tue, Sep 19, 2017 at 8:55 AM, Allen Wittenauer
<aw...@effectivemachines.com> wrote:
> We need to get over this idea that precommit is going to find all problems every time. Committers actually do need to spend some time with a patch. Besides, in this particular case, it was shading that actually broke… which really makes me want to remove -DskipShade from the pom.xmls. It’s clearly getting abused.
The downstream testing will help, particularly running against shaded
artifacts [1]. Precommit should replace as many of the rote tasks as
possible, particularly checks on invariants like shading. Computers
are good at checking this. Humans are not only bad at it, it's tedious
as hell. Particularly repeating it in every branch.
Again, let's continue disabling, deleting, and reverting changes that
make the CI infra unstable. Precommit is doing its job, but
garbage-in, garbage-out. -C
[1] https://issues.apache.org/jira/browse/HADOOP-13916
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Trunk fails
Posted by Sean Busbey <bu...@cloudera.com>.
On Wed, Sep 20, 2017 at 5:12 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> >
>
> What we could do is have a patch submission process which says "if you are
> playing with packaging, you must declare at the time of patch submission
> that you have run a full mvn clean install". And a commit process which
> says "if you commit a patch which changes the packaging, you need to do a
> build before a test"
>
> This is a variant of what we expect for the hadoop-aws and hadoop-azure
> clients where the submitter has to state the endpoint they ran the
> integration test suite against. Committer is expected to rerun the test
> suite locally before the commit too, for safety
>
> And we should all be trying 'mvn package -Pdist,native" regularly too, and
> playing with the new scripts. We need to find the issues before anyone else
>
> For all this to work, of course, we need reproducible builds. I see my
> mornings build is asking for "json-smart-2.3-SNAPSHOT.pom" as well as the
> doxia stuff. Why is so much -SNAPSHOT stuff getting in? I don't even see a
> ref for json-smart in our POMs
For stability of packaging changes, we could also ask commiters to include
in relevant commits a piece of commit metadata in the message that we issue
from a jenkins job, i.e. give the job a patch and run full nightly QBT
against tree with the patch in place.
--
busbey
Re: Trunk fails
Posted by Steve Loughran <st...@hortonworks.com>.
> On 19 Sep 2017, at 16:55, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 19, 2017, at 6:48 AM, Brahma Reddy Battula <br...@huawei.com> wrote:
>>
>> Can we run "mvn install" and "compile" for all the modules after applying the patch(we can skip shadeclients)
>
> We need to get over this idea that precommit is going to find all problems every time. Committers actually do need to spend some time with a patch. Besides, in this particular case, it was shading that actually broke…
> which really makes me want to remove -DskipShade from the pom.xmls. It’s clearly getting abused.
please dont
What we could do is have a patch submission process which says "if you are playing with packaging, you must declare at the time of patch submission that you have run a full mvn clean install". And a commit process which says "if you commit a patch which changes the packaging, you need to do a build before a test"
This is a variant of what we expect for the hadoop-aws and hadoop-azure clients where the submitter has to state the endpoint they ran the integration test suite against. Committer is expected to rerun the test suite locally before the commit too, for safety
And we should all be trying 'mvn package -Pdist,native" regularly too, and playing with the new scripts. We need to find the issues before anyone else
For all this to work, of course, we need reproducible builds. I see my mornings build is asking for "json-smart-2.3-SNAPSHOT.pom" as well as the doxia stuff. Why is so much -SNAPSHOT stuff getting in? I don't even see a ref for json-smart in our POMs
Re: Trunk fails
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 19, 2017, at 6:48 AM, Brahma Reddy Battula <br...@huawei.com> wrote:
>
> Can we run "mvn install" and "compile" for all the modules after applying the patch(we can skip shadeclients)
We need to get over this idea that precommit is going to find all problems every time. Committers actually do need to spend some time with a patch. Besides, in this particular case, it was shading that actually broke… which really makes me want to remove -DskipShade from the pom.xmls. It’s clearly getting abused.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org