You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Andrew Wang <an...@cloudera.com> on 2015/11/26 02:41:20 UTC
Disable some of the Hudson integration comments on JIRA
Hi all,
Right now we get something like 7 comments from Hudson whenever a change is
committed. Would anyone object if I turned off 6 of them? We have
variations like:
Hadoop-trunk-Commit
Hadoop-Hdfs-trunk-Java8
Hadoop-Yarn-trunk
...etc
I propose leaving notifications on for just Hadoop-trunk-Commit.
Side note is that we could probably stand to delete the disabled jobs on
our view, I'll do that if I see a job has been disabled for a while with no
recent builds:
https://builds.apache.org/view/H-L/view/Hadoop/
Best,
Andrew
Re: Disable some of the Hudson integration comments on JIRA
Posted by Ted Yu <yu...@gmail.com>.
Looking at a few Hadoop-trunk-Commit builds, I saw 'Some Enforcer rules
have failed.'
Below was from build #8895 :
[WARNING]
Dependency convergence error for
org.apache.hadoop:hadoop-auth:3.0.0-SNAPSHOT paths to dependency are:
+-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT
+-org.apache.hadoop:hadoop-auth:3.0.0-SNAPSHOT
and
+-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT
+-org.apache.hadoop:hadoop-auth:3.0.0-20151126.050629-7794
[WARNING] Rule 0:
org.apache.maven.plugins.enforcer.DependencyConvergence failed with
message:
Failed while enforcing releasability the error(s) are [
Dependency convergence error for
org.apache.hadoop:hadoop-auth:3.0.0-SNAPSHOT paths to dependency are:
+-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT
+-org.apache.hadoop:hadoop-auth:3.0.0-SNAPSHOT
and
+-org.apache.hadoop:hadoop-common:3.0.0-SNAPSHOT
+-org.apache.hadoop:hadoop-auth:3.0.0-20151126.050629-7794
FYI
On Thu, Nov 26, 2015 at 2:46 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 26 Nov 2015, at 01:41, Andrew Wang <an...@cloudera.com> wrote:
> >
> > Hi all,
> >
> > Right now we get something like 7 comments from Hudson whenever a change
> is
> > committed. Would anyone object if I turned off 6 of them? We have
> > variations like:
> >
> > Hadoop-trunk-Commit
> > Hadoop-Hdfs-trunk-Java8
> > Hadoop-Yarn-trunk
> > ...etc
> >
> > I propose leaving notifications on for just Hadoop-trunk-Commit.
> +1
>
> I'd also like to understand why those changes are always tagged as FAILED
>
Re: Disable some of the Hudson integration comments on JIRA
Posted by Steve Loughran <st...@hortonworks.com>.
> On 26 Nov 2015, at 01:41, Andrew Wang <an...@cloudera.com> wrote:
>
> Hi all,
>
> Right now we get something like 7 comments from Hudson whenever a change is
> committed. Would anyone object if I turned off 6 of them? We have
> variations like:
>
> Hadoop-trunk-Commit
> Hadoop-Hdfs-trunk-Java8
> Hadoop-Yarn-trunk
> ...etc
>
> I propose leaving notifications on for just Hadoop-trunk-Commit.
+1
I'd also like to understand why those changes are always tagged as FAILED
Re: Disable some of the Hudson integration comments on JIRA
Posted by Steve Loughran <st...@hortonworks.com>.
> On 1 Dec 2015, at 00:03, Allen Wittenauer <aw...@altiscale.com> wrote:
>
>
>> On Nov 30, 2015, at 3:46 PM, Chris Nauroth <cn...@hortonworks.com> wrote:
>>
>> To be exhaustive, we'd also add the various -Drequire options for full
>> inclusion of all optional native components. Unfortunately, that gets
>> tricky because of the various build pre-requisites which would need to be
>> present on the Jenkins hosts. This is where the Dockerfile is helpful,
>> because it encapsulates all of that. Maybe this job would benefit from
>> using the Dockerfile.
>>
>
>
> I’ve been noodling over the past few weeks about a tool called ‘qbt’ that would basically manage builds like Yetus manages tests. It’d be able to use the Yetus plug-ins to qualify the build, do reporting, etc.
>
>
>
I've been doing some work again on some databricks-originating code to go through jenkins histories and ID flaky tests, including automated result publishing as a google doc.
https://github.com/steveloughran/spark-test-failures/tree/stevel-version
The original release was hard code to the spark jenkins and builds; I'm fixing my branch to work with ASF jenkins.
Although the original design's publishing to google docs is cute, we could have some other reporters. Generating an HTML page would be ideal for jenkins itself. So jenkins could be used to run analytics workloads against other job histories
Re: Disable some of the Hudson integration comments on JIRA
Posted by Allen Wittenauer <aw...@altiscale.com>.
> On Nov 30, 2015, at 3:46 PM, Chris Nauroth <cn...@hortonworks.com> wrote:
>
> To be exhaustive, we'd also add the various -Drequire options for full
> inclusion of all optional native components. Unfortunately, that gets
> tricky because of the various build pre-requisites which would need to be
> present on the Jenkins hosts. This is where the Dockerfile is helpful,
> because it encapsulates all of that. Maybe this job would benefit from
> using the Dockerfile.
>
I’ve been noodling over the past few weeks about a tool called ‘qbt’ that would basically manage builds like Yetus manages tests. It’d be able to use the Yetus plug-ins to qualify the build, do reporting, etc.
Re: Disable some of the Hudson integration comments on JIRA
Posted by Chris Nauroth <cn...@hortonworks.com>.
To be exhaustive, we'd also add the various -Drequire options for full
inclusion of all optional native components. Unfortunately, that gets
tricky because of the various build pre-requisites which would need to be
present on the Jenkins hosts. This is where the Dockerfile is helpful,
because it encapsulates all of that. Maybe this job would benefit from
using the Dockerfile.
--Chris Nauroth
On 11/30/15, 3:26 PM, "Andrew Wang" <an...@cloudera.com> wrote:
>I went ahead and turned off JIRA notification for all the "other" jobs I
>could find, if you see more than one Hudson email from now on feel free to
>ping me or turn it off yourself. The jobs are also still running, just not
>JIRA commenting anymore.
>
>I did also add the native nvn args to Hadoop-trunk-Commit, feel free to
>append more stuff as desired.
>
>On Mon, Nov 30, 2015 at 3:11 PM, Andrew Wang <an...@cloudera.com>
>wrote:
>
>>
>> On Mon, Nov 30, 2015 at 3:04 PM, Allen Wittenauer <aw...@altiscale.com>
>> wrote:
>>
>>>
>>> > On Nov 30, 2015, at 11:33 AM, Andrew Wang <an...@cloudera.com>
>>> wrote:
>>> >
>>> > Good point Allen. So I guess the broader question is, do we find the
>>> > per-commit tracking build and test useful? With our current flakiness
>>> > levels, there isn't much signal from a FAILED on one of these
>>> integration
>>> > jobs. I think Hadoop-trunk-Commit is still nice as a compilation
>>>check.
>>>
>>> Š except it doesn¹t compile everything ...
>>>
>>> So I see this in the shell step at the top level:
>>
>> $MAVEN_HOME/bin/mvn clean install -DskipTests
>>
>> It's not compiling native code if that was your concern, and I can add
>> -Pnative. Anything else to tack on?
>>
Re: Disable some of the Hudson integration comments on JIRA
Posted by Andrew Wang <an...@cloudera.com>.
I went ahead and turned off JIRA notification for all the "other" jobs I
could find, if you see more than one Hudson email from now on feel free to
ping me or turn it off yourself. The jobs are also still running, just not
JIRA commenting anymore.
I did also add the native nvn args to Hadoop-trunk-Commit, feel free to
append more stuff as desired.
On Mon, Nov 30, 2015 at 3:11 PM, Andrew Wang <an...@cloudera.com>
wrote:
>
> On Mon, Nov 30, 2015 at 3:04 PM, Allen Wittenauer <aw...@altiscale.com>
> wrote:
>
>>
>> > On Nov 30, 2015, at 11:33 AM, Andrew Wang <an...@cloudera.com>
>> wrote:
>> >
>> > Good point Allen. So I guess the broader question is, do we find the
>> > per-commit tracking build and test useful? With our current flakiness
>> > levels, there isn't much signal from a FAILED on one of these
>> integration
>> > jobs. I think Hadoop-trunk-Commit is still nice as a compilation check.
>>
>> … except it doesn’t compile everything ...
>>
>> So I see this in the shell step at the top level:
>
> $MAVEN_HOME/bin/mvn clean install -DskipTests
>
> It's not compiling native code if that was your concern, and I can add
> -Pnative. Anything else to tack on?
>
Re: Disable some of the Hudson integration comments on JIRA
Posted by Andrew Wang <an...@cloudera.com>.
On Mon, Nov 30, 2015 at 3:04 PM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> > On Nov 30, 2015, at 11:33 AM, Andrew Wang <an...@cloudera.com>
> wrote:
> >
> > Good point Allen. So I guess the broader question is, do we find the
> > per-commit tracking build and test useful? With our current flakiness
> > levels, there isn't much signal from a FAILED on one of these integration
> > jobs. I think Hadoop-trunk-Commit is still nice as a compilation check.
>
> … except it doesn’t compile everything ...
>
> So I see this in the shell step at the top level:
$MAVEN_HOME/bin/mvn clean install -DskipTests
It's not compiling native code if that was your concern, and I can add
-Pnative. Anything else to tack on?
Re: Disable some of the Hudson integration comments on JIRA
Posted by Allen Wittenauer <aw...@altiscale.com>.
> On Nov 30, 2015, at 11:33 AM, Andrew Wang <an...@cloudera.com> wrote:
>
> Good point Allen. So I guess the broader question is, do we find the
> per-commit tracking build and test useful? With our current flakiness
> levels, there isn't much signal from a FAILED on one of these integration
> jobs. I think Hadoop-trunk-Commit is still nice as a compilation check.
… except it doesn’t compile everything ...
Re: Disable some of the Hudson integration comments on JIRA
Posted by "Colin P. McCabe" <cm...@apache.org>.
+1.
Colin
On Mon, Nov 30, 2015 at 11:33 AM, Andrew Wang <an...@cloudera.com> wrote:
> Good point Allen. So I guess the broader question is, do we find the
> per-commit tracking build and test useful? With our current flakiness
> levels, there isn't much signal from a FAILED on one of these integration
> jobs. I think Hadoop-trunk-Commit is still nice as a compilation check.
>
> So overall I'm still in favor of disabling the other jobs. There's always
> the backstop of the nightly test runs to help track down new failures, and
> doing a one-day bisect shouldn't be too bad.
>
> If this sounds good, I'll go ahead and do the above.
>
> Best,
> Andrew
>
> On Thu, Nov 26, 2015 at 9:22 AM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
>>
>> > On Nov 25, 2015, at 5:41 PM, Andrew Wang <an...@cloudera.com>
>> wrote:
>> >
>> > Hi all,
>> >
>> > Right now we get something like 7 comments from Hudson whenever a change
>> is
>> > committed. Would anyone object if I turned off 6 of them? We have
>> > variations like:
>> >
>> > Hadoop-trunk-Commit
>> > Hadoop-Hdfs-trunk-Java8
>> > Hadoop-Yarn-trunk
>> > ...etc
>> >
>> > I propose leaving notifications on for just Hadoop-trunk-Commit.
>> >
>>
>> This will give false positives since that job does a -DskipTests.
>>
Re: Disable some of the Hudson integration comments on JIRA
Posted by Andrew Wang <an...@cloudera.com>.
Good point Allen. So I guess the broader question is, do we find the
per-commit tracking build and test useful? With our current flakiness
levels, there isn't much signal from a FAILED on one of these integration
jobs. I think Hadoop-trunk-Commit is still nice as a compilation check.
So overall I'm still in favor of disabling the other jobs. There's always
the backstop of the nightly test runs to help track down new failures, and
doing a one-day bisect shouldn't be too bad.
If this sounds good, I'll go ahead and do the above.
Best,
Andrew
On Thu, Nov 26, 2015 at 9:22 AM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> > On Nov 25, 2015, at 5:41 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> >
> > Hi all,
> >
> > Right now we get something like 7 comments from Hudson whenever a change
> is
> > committed. Would anyone object if I turned off 6 of them? We have
> > variations like:
> >
> > Hadoop-trunk-Commit
> > Hadoop-Hdfs-trunk-Java8
> > Hadoop-Yarn-trunk
> > ...etc
> >
> > I propose leaving notifications on for just Hadoop-trunk-Commit.
> >
>
> This will give false positives since that job does a -DskipTests.
>
Re: Disable some of the Hudson integration comments on JIRA
Posted by Allen Wittenauer <aw...@altiscale.com>.
> On Nov 25, 2015, at 5:41 PM, Andrew Wang <an...@cloudera.com> wrote:
>
> Hi all,
>
> Right now we get something like 7 comments from Hudson whenever a change is
> committed. Would anyone object if I turned off 6 of them? We have
> variations like:
>
> Hadoop-trunk-Commit
> Hadoop-Hdfs-trunk-Java8
> Hadoop-Yarn-trunk
> ...etc
>
> I propose leaving notifications on for just Hadoop-trunk-Commit.
>
This will give false positives since that job does a -DskipTests.
Re: Disable some of the Hudson integration comments on JIRA
Posted by Vinayakumar B <vi...@apache.org>.
Yes, thats good idea.
-Vinay
On Thu, Nov 26, 2015 at 10:29 AM, Zhe Zhang <zh...@cloudera.com> wrote:
> I think that's a good idea. Thanks for the proposal Andrew.
>
> ---
> Zhe Zhang
>
> On Wed, Nov 25, 2015 at 5:41 PM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Hi all,
> >
> > Right now we get something like 7 comments from Hudson whenever a change
> is
> > committed. Would anyone object if I turned off 6 of them? We have
> > variations like:
> >
> > Hadoop-trunk-Commit
> > Hadoop-Hdfs-trunk-Java8
> > Hadoop-Yarn-trunk
> > ...etc
> >
> > I propose leaving notifications on for just Hadoop-trunk-Commit.
> >
> > Side note is that we could probably stand to delete the disabled jobs on
> > our view, I'll do that if I see a job has been disabled for a while with
> no
> > recent builds:
> >
> > https://builds.apache.org/view/H-L/view/Hadoop/
> >
> > Best,
> > Andrew
> >
>
Re: Disable some of the Hudson integration comments on JIRA
Posted by Zhe Zhang <zh...@cloudera.com>.
I think that's a good idea. Thanks for the proposal Andrew.
---
Zhe Zhang
On Wed, Nov 25, 2015 at 5:41 PM, Andrew Wang <an...@cloudera.com>
wrote:
> Hi all,
>
> Right now we get something like 7 comments from Hudson whenever a change is
> committed. Would anyone object if I turned off 6 of them? We have
> variations like:
>
> Hadoop-trunk-Commit
> Hadoop-Hdfs-trunk-Java8
> Hadoop-Yarn-trunk
> ...etc
>
> I propose leaving notifications on for just Hadoop-trunk-Commit.
>
> Side note is that we could probably stand to delete the disabled jobs on
> our view, I'll do that if I see a job has been disabled for a while with no
> recent builds:
>
> https://builds.apache.org/view/H-L/view/Hadoop/
>
> Best,
> Andrew
>