You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Bokor Andras <bo...@freemail.hu> on 2017/07/22 22:28:44 UTC
Trunk fails
Hi all, Trunk currently fails without -DskipShade.To see the full error message check one of the pre-commit buils, eg:https://builds.apache.org/job/PreCommit-HADOOP-Build/12839/artifact/patchprocess/branch-mvninstall-root.txtand look for "Duplicate classes found" string.It's related to shading. I do not know too much about shading so I need some help to invesitage this. Thanks,Andras
Re: Trunk fails
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Jul 22, 2017, at 10:12 PM, Brahma Reddy Battula <br...@hotmail.com> wrote:
>
> FYI..After Jian he comment on YARN-6804,I reverted. Now trunk build will pass.
>
Yup:
https://s.apache.org/S6yd
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Trunk fails
Posted by Chris Douglas <ch...@gmail.com>.
On Tue, Sep 19, 2017 at 8:55 AM, Allen Wittenauer
<aw...@effectivemachines.com> wrote:
> We need to get over this idea that precommit is going to find all problems every time. Committers actually do need to spend some time with a patch. Besides, in this particular case, it was shading that actually broke… which really makes me want to remove -DskipShade from the pom.xmls. It’s clearly getting abused.
The downstream testing will help, particularly running against shaded
artifacts [1]. Precommit should replace as many of the rote tasks as
possible, particularly checks on invariants like shading. Computers
are good at checking this. Humans are not only bad at it, it's tedious
as hell. Particularly repeating it in every branch.
Again, let's continue disabling, deleting, and reverting changes that
make the CI infra unstable. Precommit is doing its job, but
garbage-in, garbage-out. -C
[1] https://issues.apache.org/jira/browse/HADOOP-13916
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Trunk fails
Posted by Sean Busbey <bu...@cloudera.com>.
On Wed, Sep 20, 2017 at 5:12 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> >
>
> What we could do is have a patch submission process which says "if you are
> playing with packaging, you must declare at the time of patch submission
> that you have run a full mvn clean install". And a commit process which
> says "if you commit a patch which changes the packaging, you need to do a
> build before a test"
>
> This is a variant of what we expect for the hadoop-aws and hadoop-azure
> clients where the submitter has to state the endpoint they ran the
> integration test suite against. Committer is expected to rerun the test
> suite locally before the commit too, for safety
>
> And we should all be trying 'mvn package -Pdist,native" regularly too, and
> playing with the new scripts. We need to find the issues before anyone else
>
> For all this to work, of course, we need reproducible builds. I see my
> mornings build is asking for "json-smart-2.3-SNAPSHOT.pom" as well as the
> doxia stuff. Why is so much -SNAPSHOT stuff getting in? I don't even see a
> ref for json-smart in our POMs
For stability of packaging changes, we could also ask commiters to include
in relevant commits a piece of commit metadata in the message that we issue
from a jenkins job, i.e. give the job a patch and run full nightly QBT
against tree with the patch in place.
--
busbey
Re: Trunk fails
Posted by Steve Loughran <st...@hortonworks.com>.
> On 19 Sep 2017, at 16:55, Allen Wittenauer <aw...@effectivemachines.com> wrote:
>
>
>> On Sep 19, 2017, at 6:48 AM, Brahma Reddy Battula <br...@huawei.com> wrote:
>>
>> Can we run "mvn install" and "compile" for all the modules after applying the patch(we can skip shadeclients)
>
> We need to get over this idea that precommit is going to find all problems every time. Committers actually do need to spend some time with a patch. Besides, in this particular case, it was shading that actually broke…
> which really makes me want to remove -DskipShade from the pom.xmls. It’s clearly getting abused.
please dont
What we could do is have a patch submission process which says "if you are playing with packaging, you must declare at the time of patch submission that you have run a full mvn clean install". And a commit process which says "if you commit a patch which changes the packaging, you need to do a build before a test"
This is a variant of what we expect for the hadoop-aws and hadoop-azure clients where the submitter has to state the endpoint they ran the integration test suite against. Committer is expected to rerun the test suite locally before the commit too, for safety
And we should all be trying 'mvn package -Pdist,native" regularly too, and playing with the new scripts. We need to find the issues before anyone else
For all this to work, of course, we need reproducible builds. I see my mornings build is asking for "json-smart-2.3-SNAPSHOT.pom" as well as the doxia stuff. Why is so much -SNAPSHOT stuff getting in? I don't even see a ref for json-smart in our POMs
Re: Trunk fails
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 19, 2017, at 6:48 AM, Brahma Reddy Battula <br...@huawei.com> wrote:
>
> Can we run "mvn install" and "compile" for all the modules after applying the patch(we can skip shadeclients)
We need to get over this idea that precommit is going to find all problems every time. Committers actually do need to spend some time with a patch. Besides, in this particular case, it was shading that actually broke… which really makes me want to remove -DskipShade from the pom.xmls. It’s clearly getting abused.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
RE: Trunk fails
Posted by Brahma Reddy Battula <br...@huawei.com>.
Can we run "mvn install" and "compile" for all the modules after applying the patch(we can skip shadeclients)..?
AS today also we seen "mvn install" issue after HADOOP-14771.
--Brahma Reddy Battula
-----Original Message-----
From: Allen Wittenauer [mailto:aw@effectivemachines.com]
Sent: 14 August 2017 22:09
To: Brahma Reddy Battula
Cc: common-dev@hadoop.apache.org
Subject: Re: Trunk fails
> On Aug 14, 2017, at 5:36 AM, Brahma Reddy Battula <br...@huawei.com> wrote:
>
> How about let this comment on Jira if there is any failure(compile/Test)..? so that corresponding Jira reporter/committer can look into it(can reduce/avoid pre-commit failures..?).
>
hadoop-commit-trunk does. We can see that in this particular case very clearly:
After the commit: https://s.apache.org/ufhf
After the revert: https://s.apache.org/3GbA
Old timers will remember when this job wasn’t reliable. They might still believe it isn’t, but that hasn’t been true for a years. It’s definitely in the 99%+ for accuracy (at least for non-native on Linux).
I don’t have the QBT report publishing to JIRA because it never passes. Hadoop’s unit tests are too unreliable for that.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Trunk fails
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Aug 14, 2017, at 5:36 AM, Brahma Reddy Battula <br...@huawei.com> wrote:
>
> How about let this comment on Jira if there is any failure(compile/Test)..? so that corresponding Jira reporter/committer can look into it(can reduce/avoid pre-commit failures..?).
>
hadoop-commit-trunk does. We can see that in this particular case very clearly:
After the commit: https://s.apache.org/ufhf
After the revert: https://s.apache.org/3GbA
Old timers will remember when this job wasn’t reliable. They might still believe it isn’t, but that hasn’t been true for a years. It’s definitely in the 99%+ for accuracy (at least for non-native on Linux).
I don’t have the QBT report publishing to JIRA because it never passes. Hadoop’s unit tests are too unreliable for that.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
RE: Trunk fails
Posted by Brahma Reddy Battula <br...@huawei.com>.
How about let this comment on Jira if there is any failure(compile/Test)..? so that corresponding Jira reporter/committer can look into it(can reduce/avoid pre-commit failures..?).
--Brahma Reddy Battula
-----Original Message-----
From: Allen Wittenauer [mailto:aw@effectivemachines.com]
Sent: 23 July 2017 22:34
To: common-dev@hadoop.apache.org
Cc: Bokor Andras; Brahma Reddy Battula
Subject: Re: Trunk fails
> On Jul 22, 2017, at 11:31 PM, Brahma Reddy Battula <br...@hotmail.com> wrote:
>
> AFAIK, Patch install(i mean,mvn install) will not run all the projects hence you didn't seen this failure.
Correct. By design, precommit only runs the test suite against the maven modules that the patch modifies to save (significant!) processing time. The number of patches that would benefit from touching the entire source base is relatively small. But they do exist which is why the two follow up jobs are important:
* the hadoop-trunk-commit, which writes the maven repo status at the end of every JIRA. It builds (almost) everything but skips unit tests. It runs every 20 minutes and only if there has been a commit.
* the daily qbt report which builds and runs EVERYTHING: all the modules, all the native code supported by the docker-ized env, all the unit tests. (For those curious, it currently clocks in at around 9.5 hours. That's up by about 1.5 hours since I added it over a year ago.)
In this case, both of these failed and should have been an indication that something was very, very wrong. Committers should be making it a priority to watch for both of those reports to confirm that their commit didn't break anything.
(I'm not sure if we have or ever had any documentation on what happens after a commit. While these jobs have been rewritten, the process of patch verification process from a committer view hasn't changed in probably five years or more. The tribal knowledge is failing to get passed down from one generation to another.)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Trunk fails
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Jul 22, 2017, at 11:31 PM, Brahma Reddy Battula <br...@hotmail.com> wrote:
>
> AFAIK, Patch install(i mean,mvn install) will not run all the projects hence you didn't seen this failure.
Correct. By design, precommit only runs the test suite against the maven modules that the patch modifies to save (significant!) processing time. The number of patches that would benefit from touching the entire source base is relatively small. But they do exist which is why the two follow up jobs are important:
* the hadoop-trunk-commit, which writes the maven repo status at the end of every JIRA. It builds (almost) everything but skips unit tests. It runs every 20 minutes and only if there has been a commit.
* the daily qbt report which builds and runs EVERYTHING: all the modules, all the native code supported by the docker-ized env, all the unit tests. (For those curious, it currently clocks in at around 9.5 hours. That's up by about 1.5 hours since I added it over a year ago.)
In this case, both of these failed and should have been an indication that something was very, very wrong. Committers should be making it a priority to watch for both of those reports to confirm that their commit didn't break anything.
(I'm not sure if we have or ever had any documentation on what happens after a commit. While these jobs have been rewritten, the process of patch verification process from a committer view hasn't changed in probably five years or more. The tribal knowledge is failing to get passed down from one generation to another.)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Trunk fails
Posted by Brahma Reddy Battula <br...@hotmail.com>.
AFAIK, Patch install(i mean,mvn install) will not run all the projects hence you didn't seen this failure. Please see the following for same..
https://builds.apache.org/job/PreCommit-YARN-Build/16510/console
============================================================================
============================================================================
maven install: patch
============================================================================
============================================================================
cd /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry
/opt/maven/bin/mvn -Dmaven.repo.local=/home/jenkins/yetus-m2/hadoop-trunk-patch-1 -Ptest-patch -DskipTests -fae clean install -DskipTests=true -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true > /testptch/hadoop/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-registry.txt 2>&1
Elapsed: 0m 16s
cd /testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
/opt/maven/bin/mvn -Dmaven.repo.local=/home/jenkins/yetus-m2/hadoop-trunk-patch-1 -Ptest-patch -DskipTests -fae clean install -DskipTests=true -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true > /testptch/hadoop/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt 2>&1
Elapsed: 0m 26s
________________________________
From: Jian He <jh...@hortonworks.com>
Sent: Sunday, July 23, 2017 1:26 PM
To: Brahma Reddy Battula
Cc: Bokor Andras; common-dev@hadoop.apache.org; jianhe
Subject: Re: Trunk fails
Thanks Brahma.
The jenkins report before the commit actually did NOT catch the mvninstall issue. Strange.
https://issues.apache.org/jira/browse/YARN-6804?focusedCommentId=16095688&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16095688
On Jul 22, 2017, at 10:12 PM, Brahma Reddy Battula <br...@hotmail.com>> wrote:
FYI..After Jian he comment on YARN-6804,I reverted. Now trunk build will pass.
________________________________
From: Brahma Reddy Battula <br...@hotmail.com>>
Sent: Sunday, July 23, 2017 12:38 PM
To: Bokor Andras; common-dev@hadoop.apache.org<ma...@hadoop.apache.org>; jianhe
Subject: Re: Trunk fails
Yes, it's failing after following commit.
https://issues.apache.org/jira/browse/YARN-6804
<ht...@apache.org> can you please check once,all the pre-commit also have this error.
________________________________
From: Bokor Andras <bo...@freemail.hu>
Sent: Sunday, July 23, 2017 6:33 AM
To: common-dev@hadoop.apache.org
Subject: Re: Trunk fails
Something wrong with my formatting and new lines were skipped. So the correct link for build log is https://builds.apache.org/job/PreCommit-HADOOP-Build/12839/artifact/patchprocess/branch-mvninstall-root.txt
Bokor Andras <bo...@freemail.hu> írta:
Hi all, Trunk currently fails without -DskipShade.To see the full error message check one of the pre-commit buils, eg:https://builds.apache.org/job/PreCommit-HADOOP-Build/12839/artifact/patchprocess/branch-mvninstall-root.txtand look for "Duplicate classes found" string.It's related to shading. I do not know too much about shading so I need some help to invesitage this. Thanks,Andras
Re: Trunk fails
Posted by Jian He <jh...@hortonworks.com>.
Thanks Brahma.
The jenkins report before the commit actually did NOT catch the mvninstall issue. Strange.
https://issues.apache.org/jira/browse/YARN-6804?focusedCommentId=16095688&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16095688
On Jul 22, 2017, at 10:12 PM, Brahma Reddy Battula <br...@hotmail.com>> wrote:
FYI..After Jian he comment on YARN-6804,I reverted. Now trunk build will pass.
________________________________
From: Brahma Reddy Battula <br...@hotmail.com>>
Sent: Sunday, July 23, 2017 12:38 PM
To: Bokor Andras; common-dev@hadoop.apache.org<ma...@hadoop.apache.org>; jianhe
Subject: Re: Trunk fails
Yes, it's failing after following commit.
https://issues.apache.org/jira/browse/YARN-6804
<ht...@apache.org> can you please check once,all the pre-commit also have this error.
________________________________
From: Bokor Andras <bo...@freemail.hu>
Sent: Sunday, July 23, 2017 6:33 AM
To: common-dev@hadoop.apache.org
Subject: Re: Trunk fails
Something wrong with my formatting and new lines were skipped. So the correct link for build log is https://builds.apache.org/job/PreCommit-HADOOP-Build/12839/artifact/patchprocess/branch-mvninstall-root.txt
Bokor Andras <bo...@freemail.hu> írta:
Hi all, Trunk currently fails without -DskipShade.To see the full error message check one of the pre-commit buils, eg:https://builds.apache.org/job/PreCommit-HADOOP-Build/12839/artifact/patchprocess/branch-mvninstall-root.txtand look for "Duplicate classes found" string.It's related to shading. I do not know too much about shading so I need some help to invesitage this. Thanks,Andras
Re: Trunk fails
Posted by Brahma Reddy Battula <br...@hotmail.com>.
FYI..After Jian he comment on YARN-6804,I reverted. Now trunk build will pass.
________________________________
From: Brahma Reddy Battula <br...@hotmail.com>
Sent: Sunday, July 23, 2017 12:38 PM
To: Bokor Andras; common-dev@hadoop.apache.org; jianhe
Subject: Re: Trunk fails
Yes, it's failing after following commit.
https://issues.apache.org/jira/browse/YARN-6804
<ht...@apache.org> can you please check once,all the pre-commit also have this error.
________________________________
From: Bokor Andras <bo...@freemail.hu>
Sent: Sunday, July 23, 2017 6:33 AM
To: common-dev@hadoop.apache.org
Subject: Re: Trunk fails
Something wrong with my formatting and new lines were skipped. So the correct link for build log is https://builds.apache.org/job/PreCommit-HADOOP-Build/12839/artifact/patchprocess/branch-mvninstall-root.txt
Bokor Andras <bo...@freemail.hu> írta:
>Hi all, Trunk currently fails without -DskipShade.To see the full error message check one of the pre-commit buils, eg:https://builds.apache.org/job/PreCommit-HADOOP-Build/12839/artifact/patchprocess/branch-mvninstall-root.txtand look for "Duplicate classes found" string.It's related to shading. I do not know too much about shading so I need some help to invesitage this. Thanks,Andras
Re: Trunk fails
Posted by Brahma Reddy Battula <br...@hotmail.com>.
Yes, it's failing after following commit.
https://issues.apache.org/jira/browse/YARN-6804
<ht...@apache.org> can you please check once,all the pre-commit also have this error.
________________________________
From: Bokor Andras <bo...@freemail.hu>
Sent: Sunday, July 23, 2017 6:33 AM
To: common-dev@hadoop.apache.org
Subject: Re: Trunk fails
Something wrong with my formatting and new lines were skipped. So the correct link for build log is https://builds.apache.org/job/PreCommit-HADOOP-Build/12839/artifact/patchprocess/branch-mvninstall-root.txt
Bokor Andras <bo...@freemail.hu> írta:
>Hi all, Trunk currently fails without -DskipShade.To see the full error message check one of the pre-commit buils, eg:https://builds.apache.org/job/PreCommit-HADOOP-Build/12839/artifact/patchprocess/branch-mvninstall-root.txtand look for "Duplicate classes found" string.It's related to shading. I do not know too much about shading so I need some help to invesitage this. Thanks,Andras
Re: Trunk fails
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
The "update the maven snapshot repo after a commit then update the JIRA issue on the build status" job caught it (https://s.apache.org/bXUp) but looks like it's feedback and the -1 on the mvninstall after it was ignored. *shrugs*
> On Jul 22, 2017, at 3:33 PM, Bokor Andras <bo...@freemail.hu> wrote:
>
> Something wrong with my formatting and new lines were skipped. So the correct link for build log is https://builds.apache.org/job/PreCommit-HADOOP-Build/12839/artifact/patchprocess/branch-mvninstall-root.txt
> Bokor Andras <bo...@freemail.hu> írta:
>> Hi all, Trunk currently fails without -DskipShade.To see the full error message check one of the pre-commit buils, eg:https://builds.apache.org/job/PreCommit-HADOOP-Build/12839/artifact/patchprocess/branch-mvninstall-root.txtand look for "Duplicate classes found" string.It's related to shading. I do not know too much about shading so I need some help to invesitage this. Thanks,Andras
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: Trunk fails
Posted by Bokor Andras <bo...@freemail.hu>.
Something wrong with my formatting and new lines were skipped. So the correct link for build log is https://builds.apache.org/job/PreCommit-HADOOP-Build/12839/artifact/patchprocess/branch-mvninstall-root.txt
Bokor Andras <bo...@freemail.hu> írta:
>Hi all, Trunk currently fails without -DskipShade.To see the full error message check one of the pre-commit buils, eg:https://builds.apache.org/job/PreCommit-HADOOP-Build/12839/artifact/patchprocess/branch-mvninstall-root.txtand look for "Duplicate classes found" string.It's related to shading. I do not know too much about shading so I need some help to invesitage this. Thanks,Andras