You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Andrew Wang <an...@cloudera.com> on 2016/06/01 03:53:54 UTC
Re: [DISCUSS] Replacing the automatic Jenkins jobs with Apache Yetus qbt
I'm in favor of something that helps unify the current mess of Jenkins
jobs. We do something similar for our internal Hadoop repo: each branch has
a "build.sh" and "test.sh" script that builds and then runs the tests. This
predates Yetus, else we'd probably have used that. So +1 from me.
One thing I also find really helpful is riding over flaky tests, perhaps
via:
http://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html
This would be great for precommit too, since right now someone has to run
the failed tests locally to confirm they're flakes.
Even better is then feeding the success/failure into a DB so you can track
flaky rates over time, which helps you prioritize which flaky to fix first,
and helps you find the change which caused the flakiness.
On Tue, May 31, 2016 at 11:12 AM, Allen Wittenauer <aw...@apache.org> wrote:
>
> Hey gang.
>
> I’ve been thinking a lot over the past few months about the
> various Jenkins jobs that we have that run after a commit. Their job is to
> tell us when a commit break a unit test. They eat up an incredible amount
> of time on the build infrastructure and I’m not really sure if they provide
> much value in their current form. I suspect most of us pretty much ignore
> those emails, however, for a variety of reasons:
>
> a) There are simply too many of them.
> b) The content of the email is hard to quickly parse/read to get useful
> information.
> c) The unit tests are almost always broken.
>
> Worse yet: these jobs don’t actually test everything (all of
> hadoop-tools is missing!) and skip out on some vital information.
>
> With Apache Yetus 0.3.0, we added a mode called ‘qbt’ aka Quality
> Build Tool. It effectively run Apache Yetus over the entire source tree
> and generates the columnar output similar to what you would see during
> precommit. I’ve been running this mode against Apache Hadoop on Jenkins
> for a few months now, working out the kinks, etc, and it was working well
> except for one thing… the generated report was still not great over email:
>
> 1) Couldn’t use HTML since the ASF mailing lists reject HTML-formatted mail
> 2) Mail clients that used proportional fonts were still a big mess
>
> For 0.4.0, I’m working on a feature that adds a ‘brief’ mode [
> YETUS-390 ] with the (long term?) intention of replacing all of these
> emails that get generated by Jenkins. I’d like to hear some feedback on if
> a) I’m just spinning my wheels and no one is actually interested in fixing
> this problem and b) the replacement email generated either needs more info
> or less info.
>
> Here’s the currently generated email. Note that some subsystems
> have been told not to fail the build due to warnings (e.g., cc, javac,
> javadoc, shellcheck, whitespace, etc) but still generate a log of problems
> so that they may be investigated and used for cleanup.
>
> Thanks!
>
> ==== cut here =====
> For more details, see https://builds.apache.org/job/hadoop-qbt/39/
>
> [Jun 1, 2016 12:41:15 AM] (kai.zheng) Revert "HADOOP-12579. Deprecate and
> remove WriteableRPCEngine.
>
>
>
>
> -1 overall
>
>
> The following subsystems reported failures:
> findbugs unit
>
>
> The following subsystems are considered long running:
> unit
>
>
> FindBugs :
>
> module:hadoop-common-project/hadoop-minikdc
> org.apache.hadoop.minikdc.MiniKdc.stop() calls Thread.sleep() with a
> lock held At MiniKdc.java:lock held At MiniKdc.java:[line 345]
>
> Failed junit tests :
>
> hadoop.net.TestClusterTopology
> hadoop.yarn.server.resourcemanager.TestAMAuthorization
> hadoop.yarn.server.resourcemanager.TestClientRMTokens
> hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
> hadoop.yarn.server.TestContainerManagerSecurity
> hadoop.yarn.client.TestGetGroups
> hadoop.yarn.client.api.impl.TestDistributedScheduling
> hadoop.yarn.client.api.impl.TestAMRMProxy
> hadoop.mapreduce.tools.TestCLI
> hadoop.mapred.TestMiniMRChildTask
>
> Timed out junit tests :
>
> org.apache.hadoop.yarn.client.cli.TestYarnCLI
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient
> org.apache.hadoop.yarn.client.api.impl.TestNMClient
>
> Console output:
> https://builds.apache.org/job/hadoop-qbt/39/console
>
> Logs:
>
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-compile-cc-root.txt
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-compile-javac-root.txt
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-checkstyle-root.txt
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-patch-pylint.txt
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-patch-shellcheck.txt
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-patch-shelldocs.txt
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out/whitespace-eol.txt
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out/whitespace-tabs.txt
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out/branch-findbugs-hadoop-common-project_hadoop-minikdc-warnings.html
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out/diff-javadoc-javadoc-root.txt
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out//testptch/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
>
> https://builds.apache.org/job/hadoop-qbt/39/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
>
> Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org
>
> ==== cut here ====
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
Re: [DISCUSS] Replacing the automatic Jenkins jobs with Apache Yetus qbt
Posted by Allen Wittenauer <al...@yahoo.com.INVALID>.
OK, first report was sent out in lieu of the map reduce and common reports. I'll turn off HDFS and YARN next week some time. For the comparable OS X report, I need:
* YARN-5121 rebased and committed
* a fix for YETUS-414 (which I'm testing)
* and potentially need to get INFRA involved to fix the Java installation to support JNI properly (which is why 2/3 ctest failures are happening in libhdfs. The other one appears to be an actual bug.)
After the OS X one is brought live, we'll have a pattern to support other operating systems. (e.g., the PowerPC bits)
> On Jun 3, 2016, at 12:50 PM, Chris Nauroth <cn...@hortonworks.com> wrote:
>
> I think the new formatting makes it a lot easier to pick out a specific
> log file link. Thank you for making that change.
>
> --Chris Nauroth
>
>
>
>
> On 6/3/16, 7:53 AM, "Allen Wittenauer" <al...@yahoo.com.INVALID>
> wrote:
>
>>
>> Thanks all. As soon as YETUS-390 gets committed, I'll start publishing
>> the reports to *-dev.
>>
>> FWIW, here's last night's. It makes it easy to find out now when
>> committers are opening ignoring findbugs warnings. (The latest from
>> YARN-5180. It started showing up yesterday.)
>>
>>
>>
>> For more details, see https://builds.apache.org/job/hadoop-qbt/46/
>>
>> [Jun 2, 2016 8:14:45 PM] (arp) HDFS-10341. Add a metric to expose the
>> timeout number of pending
>>
>>
>>
>>
>> -1 overall
>>
>>
>> The following subsystems voted -1:
>> findbugs unit
>>
>>
>> The following subsystems voted -1 but
>> were configured to be filtered/ignored:
>> cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace
>>
>>
>> The following subsystems are considered long running:
>> (runtime bigger than 1h 0m 0s)
>> unit
>>
>>
>> Specific tests:
>>
>> FindBugs :
>>
>> module:hadoop-common-project/hadoop-minikdc
>> org.apache.hadoop.minikdc.MiniKdc.stop() calls Thread.sleep() with
>> a lock held At MiniKdc.java:lock held At MiniKdc.java:[line 345]
>>
>> FindBugs :
>>
>> module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
>> Redundant nullcheck of execTypeRequest, which is known to be
>> non-null in
>> org.apache.hadoop.yarn.api.records.ResourceRequest.equals(Object)
>> Redundant null check at ResourceRequest.java:is known to be non-null in
>> org.apache.hadoop.yarn.api.records.ResourceRequest.equals(Object)
>> Redundant null check at ResourceRequest.java:[line 361]
>>
>> Failed junit tests :
>>
>> hadoop.hdfs.TestAsyncHDFSWithHA
>> hadoop.hdfs.server.namenode.TestEditLog
>> hadoop.yarn.server.resourcemanager.TestClientRMTokens
>> hadoop.yarn.server.resourcemanager.TestAMAuthorization
>> hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
>> hadoop.yarn.server.TestContainerManagerSecurity
>> hadoop.yarn.client.api.impl.TestDistributedScheduling
>> hadoop.yarn.client.TestGetGroups
>> hadoop.yarn.client.cli.TestLogsCLI
>> hadoop.yarn.client.api.impl.TestAMRMProxy
>> hadoop.mapred.TestMiniMRChildTask
>>
>> Timed out junit tests :
>>
>> org.apache.hadoop.http.TestHttpServerLifecycle
>> org.apache.hadoop.yarn.client.cli.TestYarnCLI
>> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
>> org.apache.hadoop.yarn.client.api.impl.TestYarnClient
>> org.apache.hadoop.yarn.client.api.impl.TestNMClient
>>
>>
>> cc:
>>
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-compile-cc-r
>> oot.txt [4.0K]
>>
>> javac:
>>
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-compile-java
>> c-root.txt [168K]
>>
>> checkstyle:
>>
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-checkstyle-r
>> oot.txt [16M]
>>
>> pylint:
>>
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-patch-pylint
>> .txt [16K]
>>
>> shellcheck:
>>
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-patch-shellc
>> heck.txt [20K]
>>
>> shelldocs:
>>
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-patch-shelld
>> ocs.txt [16K]
>>
>> whitespace:
>>
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/whitespace-eol.tx
>> t [12M]
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/whitespace-tabs.t
>> xt [1.3M]
>>
>> findbugs:
>>
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/branch-findbugs-h
>> adoop-common-project_hadoop-minikdc-warnings.html [8.0K]
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/branch-findbugs-h
>> adoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html [8.0K]
>>
>> javadoc:
>>
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-javadoc-java
>> doc-root.txt [2.3M]
>>
>> unit:
>>
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop
>> -common-project_hadoop-common.txt [116K]
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop
>> -hdfs-project_hadoop-hdfs.txt [144K]
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop
>> -yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcema
>> nager.txt [60K]
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop
>> -yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
>> [268K]
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop
>> -yarn-project_hadoop-yarn_hadoop-yarn-client.txt [908K]
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop
>> -mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclie
>> nt.txt [92K]
>>
>> https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop
>> -mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativet
>> ask.txt [124K]
>>
>> Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSS] Replacing the automatic Jenkins jobs with Apache
Yetus qbt
Posted by Chris Nauroth <cn...@hortonworks.com>.
I think the new formatting makes it a lot easier to pick out a specific
log file link. Thank you for making that change.
--Chris Nauroth
On 6/3/16, 7:53 AM, "Allen Wittenauer" <al...@yahoo.com.INVALID>
wrote:
>
>Thanks all. As soon as YETUS-390 gets committed, I'll start publishing
>the reports to *-dev.
>
>FWIW, here's last night's. It makes it easy to find out now when
>committers are opening ignoring findbugs warnings. (The latest from
>YARN-5180. It started showing up yesterday.)
>
>
>
>For more details, see https://builds.apache.org/job/hadoop-qbt/46/
>
>[Jun 2, 2016 8:14:45 PM] (arp) HDFS-10341. Add a metric to expose the
>timeout number of pending
>
>
>
>
>-1 overall
>
>
>The following subsystems voted -1:
> findbugs unit
>
>
>The following subsystems voted -1 but
>were configured to be filtered/ignored:
> cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace
>
>
>The following subsystems are considered long running:
>(runtime bigger than 1h 0m 0s)
> unit
>
>
>Specific tests:
>
> FindBugs :
>
> module:hadoop-common-project/hadoop-minikdc
> org.apache.hadoop.minikdc.MiniKdc.stop() calls Thread.sleep() with
>a lock held At MiniKdc.java:lock held At MiniKdc.java:[line 345]
>
> FindBugs :
>
> module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
> Redundant nullcheck of execTypeRequest, which is known to be
>non-null in
>org.apache.hadoop.yarn.api.records.ResourceRequest.equals(Object)
>Redundant null check at ResourceRequest.java:is known to be non-null in
>org.apache.hadoop.yarn.api.records.ResourceRequest.equals(Object)
>Redundant null check at ResourceRequest.java:[line 361]
>
> Failed junit tests :
>
> hadoop.hdfs.TestAsyncHDFSWithHA
> hadoop.hdfs.server.namenode.TestEditLog
> hadoop.yarn.server.resourcemanager.TestClientRMTokens
> hadoop.yarn.server.resourcemanager.TestAMAuthorization
> hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
> hadoop.yarn.server.TestContainerManagerSecurity
> hadoop.yarn.client.api.impl.TestDistributedScheduling
> hadoop.yarn.client.TestGetGroups
> hadoop.yarn.client.cli.TestLogsCLI
> hadoop.yarn.client.api.impl.TestAMRMProxy
> hadoop.mapred.TestMiniMRChildTask
>
> Timed out junit tests :
>
> org.apache.hadoop.http.TestHttpServerLifecycle
> org.apache.hadoop.yarn.client.cli.TestYarnCLI
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient
> org.apache.hadoop.yarn.client.api.impl.TestNMClient
>
>
> cc:
>
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-compile-cc-r
>oot.txt [4.0K]
>
> javac:
>
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-compile-java
>c-root.txt [168K]
>
> checkstyle:
>
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-checkstyle-r
>oot.txt [16M]
>
> pylint:
>
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-patch-pylint
>.txt [16K]
>
> shellcheck:
>
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-patch-shellc
>heck.txt [20K]
>
> shelldocs:
>
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-patch-shelld
>ocs.txt [16K]
>
> whitespace:
>
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/whitespace-eol.tx
>t [12M]
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/whitespace-tabs.t
>xt [1.3M]
>
> findbugs:
>
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/branch-findbugs-h
>adoop-common-project_hadoop-minikdc-warnings.html [8.0K]
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/branch-findbugs-h
>adoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html [8.0K]
>
> javadoc:
>
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-javadoc-java
>doc-root.txt [2.3M]
>
> unit:
>
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop
>-common-project_hadoop-common.txt [116K]
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop
>-hdfs-project_hadoop-hdfs.txt [144K]
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop
>-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcema
>nager.txt [60K]
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop
>-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
> [268K]
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop
>-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [908K]
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop
>-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclie
>nt.txt [92K]
>
>https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop
>-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativet
>ask.txt [124K]
>
>Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org
>---------------------------------------------------------------------
>To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSS] Replacing the automatic Jenkins jobs with Apache Yetus qbt
Posted by Allen Wittenauer <al...@yahoo.com.INVALID>.
Thanks all. As soon as YETUS-390 gets committed, I'll start publishing the reports to *-dev.
FWIW, here's last night's. It makes it easy to find out now when committers are opening ignoring findbugs warnings. (The latest from YARN-5180. It started showing up yesterday.)
For more details, see https://builds.apache.org/job/hadoop-qbt/46/
[Jun 2, 2016 8:14:45 PM] (arp) HDFS-10341. Add a metric to expose the timeout number of pending
-1 overall
The following subsystems voted -1:
findbugs unit
The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace
The following subsystems are considered long running:
(runtime bigger than 1h 0m 0s)
unit
Specific tests:
FindBugs :
module:hadoop-common-project/hadoop-minikdc
org.apache.hadoop.minikdc.MiniKdc.stop() calls Thread.sleep() with a lock held At MiniKdc.java:lock held At MiniKdc.java:[line 345]
FindBugs :
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
Redundant nullcheck of execTypeRequest, which is known to be non-null in org.apache.hadoop.yarn.api.records.ResourceRequest.equals(Object) Redundant null check at ResourceRequest.java:is known to be non-null in org.apache.hadoop.yarn.api.records.ResourceRequest.equals(Object) Redundant null check at ResourceRequest.java:[line 361]
Failed junit tests :
hadoop.hdfs.TestAsyncHDFSWithHA
hadoop.hdfs.server.namenode.TestEditLog
hadoop.yarn.server.resourcemanager.TestClientRMTokens
hadoop.yarn.server.resourcemanager.TestAMAuthorization
hadoop.yarn.server.TestMiniYarnClusterNodeUtilization
hadoop.yarn.server.TestContainerManagerSecurity
hadoop.yarn.client.api.impl.TestDistributedScheduling
hadoop.yarn.client.TestGetGroups
hadoop.yarn.client.cli.TestLogsCLI
hadoop.yarn.client.api.impl.TestAMRMProxy
hadoop.mapred.TestMiniMRChildTask
Timed out junit tests :
org.apache.hadoop.http.TestHttpServerLifecycle
org.apache.hadoop.yarn.client.cli.TestYarnCLI
org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
org.apache.hadoop.yarn.client.api.impl.TestYarnClient
org.apache.hadoop.yarn.client.api.impl.TestNMClient
cc:
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-compile-cc-root.txt [4.0K]
javac:
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-compile-javac-root.txt [168K]
checkstyle:
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-checkstyle-root.txt [16M]
pylint:
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-patch-pylint.txt [16K]
shellcheck:
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-patch-shellcheck.txt [20K]
shelldocs:
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-patch-shelldocs.txt [16K]
whitespace:
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/whitespace-eol.txt [12M]
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/whitespace-tabs.txt [1.3M]
findbugs:
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/branch-findbugs-hadoop-common-project_hadoop-minikdc-warnings.html [8.0K]
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api-warnings.html [8.0K]
javadoc:
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/diff-javadoc-javadoc-root.txt [2.3M]
unit:
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [116K]
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [144K]
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [60K]
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [268K]
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt [908K]
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [92K]
https://builds.apache.org/job/hadoop-qbt/46/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [124K]
Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSS] Replacing the automatic Jenkins jobs with Apache Yetus
qbt
Posted by Akira AJISAKA <aj...@oss.nttdata.co.jp>.
+1 from me too.
-Akira
On 6/3/16 01:41, Sangjin Lee wrote:
> Sounds good. +1.
> On Thu, Jun 2, 2016 at 3:28 AM Steve Loughran <st...@hortonworks.com>
> wrote:
>
>>
>>> On 1 Jun 2016, at 04:53, Andrew Wang <an...@cloudera.com> wrote:
>>>
>>> I'm in favor of something that helps unify the current mess of Jenkins
>>> jobs. We do something similar for our internal Hadoop repo: each branch
>> has
>>> a "build.sh" and "test.sh" script that builds and then runs the tests.
>> This
>>> predates Yetus, else we'd probably have used that. So +1 from me.
>>
>> +1 from me too
>>
>>
>> why not start with Hadoop common and if it's happy, go with the rest
>>
>>
>>>
>>> One thing I also find really helpful is riding over flaky tests, perhaps
>>> via:
>>>
>>>
>> http://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html
>>>
>>> This would be great for precommit too, since right now someone has to run
>>> the failed tests locally to confirm they're flakes.
>>>
>>> Even better is then feeding the success/failure into a DB so you can
>> track
>>> flaky rates over time, which helps you prioritize which flaky to fix
>> first,
>>> and helps you find the change which caused the flakiness.
>>>
>>
>>
>> There's a scala project to talk to Jenkins for that; puts them into google
>> docs
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSS] Replacing the automatic Jenkins jobs with Apache Yetus qbt
Posted by Sangjin Lee <sj...@gmail.com>.
Sounds good. +1.
On Thu, Jun 2, 2016 at 3:28 AM Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 1 Jun 2016, at 04:53, Andrew Wang <an...@cloudera.com> wrote:
> >
> > I'm in favor of something that helps unify the current mess of Jenkins
> > jobs. We do something similar for our internal Hadoop repo: each branch
> has
> > a "build.sh" and "test.sh" script that builds and then runs the tests.
> This
> > predates Yetus, else we'd probably have used that. So +1 from me.
>
> +1 from me too
>
>
> why not start with Hadoop common and if it's happy, go with the rest
>
>
> >
> > One thing I also find really helpful is riding over flaky tests, perhaps
> > via:
> >
> >
> http://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html
> >
> > This would be great for precommit too, since right now someone has to run
> > the failed tests locally to confirm they're flakes.
> >
> > Even better is then feeding the success/failure into a DB so you can
> track
> > flaky rates over time, which helps you prioritize which flaky to fix
> first,
> > and helps you find the change which caused the flakiness.
> >
>
>
> There's a scala project to talk to Jenkins for that; puts them into google
> docs
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
Re: [DISCUSS] Replacing the automatic Jenkins jobs with Apache
Yetus qbt
Posted by Steve Loughran <st...@hortonworks.com>.
> On 1 Jun 2016, at 04:53, Andrew Wang <an...@cloudera.com> wrote:
>
> I'm in favor of something that helps unify the current mess of Jenkins
> jobs. We do something similar for our internal Hadoop repo: each branch has
> a "build.sh" and "test.sh" script that builds and then runs the tests. This
> predates Yetus, else we'd probably have used that. So +1 from me.
+1 from me too
why not start with Hadoop common and if it's happy, go with the rest
>
> One thing I also find really helpful is riding over flaky tests, perhaps
> via:
>
> http://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html
>
> This would be great for precommit too, since right now someone has to run
> the failed tests locally to confirm they're flakes.
>
> Even better is then feeding the success/failure into a DB so you can track
> flaky rates over time, which helps you prioritize which flaky to fix first,
> and helps you find the change which caused the flakiness.
>
There's a scala project to talk to Jenkins for that; puts them into google docs
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSS] Replacing the automatic Jenkins jobs with Apache Yetus qbt
Posted by Allen Wittenauer <aw...@apache.org>.
> On May 31, 2016, at 8:53 PM, Andrew Wang <an...@cloudera.com> wrote:
>
> I'm in favor of something that helps unify the current mess of Jenkins
> jobs. We do something similar for our internal Hadoop repo: each branch has
> a "build.sh" and "test.sh" script that builds and then runs the tests. This
> predates Yetus, else we'd probably have used that. So +1 from me.
>
> One thing I also find really helpful is riding over flaky tests, perhaps
> via:
>
> http://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html
>
> This would be great for precommit too, since right now someone has to run
> the failed tests locally to confirm they're flakes.
>
> Even better is then feeding the success/failure into a DB so you can track
> flaky rates over time, which helps you prioritize which flaky to fix first,
> and helps you find the change which caused the flakiness.
Appy over in HBase-land has added flaky detection and the ability to track and ignore them to their Yetus personality. At some point, I’m sure we’ll see that code float upstream in a more generalized manner as they get more experience with it.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org