You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Andrew Palumbo <ap...@outlook.com> on 2015/04/17 21:02:57 UTC
Fwd: Re: h2o failures on jenkins
-------- Forwarded Message --------
Subject: Re: h2o failures on jenkins
Date: Fri, 17 Apr 2015 18:35:42 +0000
From: Anand Avati <av...@gluster.org>
To: Andrew Palumbo <ap...@outlook.com>
I'm testing this patch:
commit fafa597e0637a59249d754fb5a99d9746b1032f3
Author: Anand Avati <avati@redhat.com <ma...@redhat.com>>
Date: Fri Apr 17 11:34:45 2015 -0700
h2o-test: use unique session id strings
Signed-off-by: Anand Avati <avati@redhat.com <ma...@redhat.com>>
diff --git a/h2o/src/test/scala/org/apache/mahout/h2obindings/test/DistributedH2OSuite.scala b/h2o/src/test/scala/org/apache/mahout/h2obindings/test/Distribute
index 45c77a5..abb4289 100644
--- a/h2o/src/test/scala/org/apache/mahout/h2obindings/test/DistributedH2OSuite.scala
+++ b/h2o/src/test/scala/org/apache/mahout/h2obindings/test/DistributedH2OSuite.scala
@@ -30,7 +30,7 @@ trait DistributedH2OSuite extends DistributedMahoutSuite with LoggerConfiguratio
override protected def beforeEach() {
super.beforeEach()
- mahoutCtx = mahoutH2OContext("mah2out")
+ mahoutCtx = mahoutH2OContext("mah2out" + System.currentTimeMillis())
}
override protected def afterEach() {
On Fri, 17 Apr 2015 at 11:31 Anand Avati <avati@gluster.org
<ma...@gluster.org>> wrote:
That might be a quick fix. Just FYI - for a multi-node test
(whenever we start testing that), all nodes of a session should
share the exact string.
On Fri, 17 Apr 2015 at 11:27 Andrew Palumbo <ap.dev@outlook.com
<ma...@outlook.com>> wrote:
I suppose than an easy fix would be to name the session
something like "mah2out"+getcurrenttimemilliseconds () or
something like that.
Sent from my Verizon Wireless 4G LTE smartphone
-------- Original message --------
From: Anand Avati
Date:04/17/2015 2:23 PM (GMT-05:00)
To: Andrew Palumbo
Subject: Re: h2o failures on jenkins
It is quite possible that the two tests (based on hadoop2 and
hadoop1) are accidentally discovering each other when run in
parallel. I think giving each test a separate session id might
fix this problem. Currently the session id string "mah2out" is
hardcoded in the test/ code. If that can be somehow set to a
unique string at runtime (through env variable?), that might do
the trick. Thoughts on what's the best way we could try this?
On Fri, 17 Apr 2015 at 10:57 Andrew Palumbo <ap.dev@outlook.com
<ma...@outlook.com>> wrote:
Hey Anand,
The h2o Module has been failing regularly in the Jenkins
build. It
seems to fail about every other build test. (i.e. it does
still pass).
Jenkins was recently moved to a new server around the same
time that we
started getting this. We also are running tests for 2
profiles now,
hadoop 1 and hadoop 2, which made me think that its possible
that they
are discovering each other and running as a 2 node cloud
(just a theory)?
I was thinking maybe we should have a different test context
for h2o
which does not call
H2O.joinOthers();
Not sure if this is the answer though. Any thoughts?
[32mDistributedDecompositionsSuite: [0m
04-17 17:31:31.521 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: -----
H2O started -----
04-17 17:31:31.577 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
git branch: master
04-17 17:31:31.577 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
git hash: f6be02b33d23e5be437bbd1b4055c0f1955e1083
04-17 17:31:31.577 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
git describe: RELEASE-0.1.24-47-gf6be02b
04-17 17:31:31.577 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
project version: 0.1.25.99999
04-17 17:31:31.577 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Built
by: 'michal'
04-17 17:31:31.577 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Built
on: '2015-02-05 10:44:16'
04-17 17:31:31.578 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
availableProcessors: 16
04-17 17:31:31.581 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
heap totalMemory: 77.4 MB
04-17 17:31:31.581 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
heap maxMemory: 910.3 MB
04-17 17:31:31.581 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
version: Java 1.7.0_25 (from Oracle Corporation)
04-17 17:31:31.581 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: OS
version: Linux 3.13.0-36-lowlatency (i386)
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Possible IP Address: docker0 (docker0),
fe80:0:0:0:5484:7aff:fefe:9799%4
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Possible IP Address: docker0 (docker0), 172.17.42.1
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Possible IP Address: em1 (em1), fe80:0:0:0:7ae3:b5ff:fe17:8984%2
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Possible IP Address: em1 (em1), 67.195.81.190
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Possible IP Address: lo (lo), 0:0:0:0:0:0:0:1%1
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Possible IP Address: lo (lo), 127.0.0.1
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Multiple local IPs detected:
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
/172.17.42.1 <http://172.17.42.1> /67.195.81.190
<http://67.195.81.190>
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Attempting to determine correct address...
04-17 17:31:31.582 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Using
/67.195.81.190 <http://67.195.81.190>
04-17 17:31:31.583 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Internal communication uses port: 54322
04-17 17:31:31.583 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO:
Listening for HTTP and REST traffic
onhttp://67.195.81.190:54321/
04-17 <http://67.195.81.190:54321/04-17> 17:31:31.583
67.195.81.190:54321 <http://67.195.81.190:54321> 17293
#onsSuite INFO: H2O cloud name: 'mah2out' on
/67.195.81.190:54321 <http://67.195.81.190:54321>, discovery
address /229.95.69.48:58719 <http://229.95.69.48:58719>
04-17 17:31:31.583 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: If you
have trouble connecting, try SSH tunneling from your local
machine (e.g., via port 55555):
04-17 17:31:31.583 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: 1.
Open a terminal and run 'ssh -L
55555:localhost:54321jenkins@67.195.81.190
<ma...@67.195.81.190>'
04-17 17:31:31.583 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: 2.
Point your browser tohttp://localhost:55555
04-17 17:31:31.682 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Log
dir: '/tmp/h2o-jenkins/h2ologs'
04-17 17:31:31.682 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Cur
dir:<https://builds.apache.org/job/Mahout-Quality/ws/h2o>
04-17 17:31:31.813 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #onsSuite INFO: Cloud
of size 1 formed [/67.195.81.190:54321
<http://67.195.81.190:54321>]
04-17 17:31:31.993 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -21
04-17 17:31:36.009 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -17
04-17 17:31:37.016 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -16
04-17 17:31:38.023 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -15
04-17 17:31:41.038 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -12
04-17 17:31:42.043 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -11
04-17 17:31:43.048 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -10
04-17 17:31:46.061 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -7
04-17 17:31:47.067 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -6
04-17 17:31:48.071 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -5
04-17 17:31:51.081 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -2
04-17 17:31:52.105 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
Receiver error on port
54322java.lang.ArrayIndexOutOfBoundsException: -1
04-17 17:31:56.118 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: Out of
Memory and no swap space left from /67.195.81.188:54321
<http://67.195.81.188:54321>
04-17 17:31:56.118 67.195.81.190:54321
<http://67.195.81.190:54321> 17293 #UDP-Recv FATAL: Exiting.
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] Skipping Apache Mahout
[INFO] This project has been banned from the build due to
previous failures.
[INFO]
------------------------------------------------------------------------
[INFO]
[INFO]
------------------------------------------------------------------------
[INFO] Skipping Mahout Build Tools
[INFO] This project has been banned from the build due to
previous failures.
[INFO]
------------------------------------------------------------------------
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Mahout Build Tools ................................
SUCCESS [1.989s]
[INFO] Apache Mahout .....................................
SUCCESS [0.206s]
[INFO] Mahout Math .......................................
SUCCESS [1:14.339s]
[INFO] Mahout HDFS .......................................
SUCCESS [3.378s]
[INFO] Mahout Map-Reduce .................................
SUCCESS [12:50.243s]
[INFO] Mahout Integration ................................
SUCCESS [54.793s]
[INFO] Mahout Examples ...................................
SUCCESS [22.309s]
[INFO] Mahout Math Scala bindings ........................
SUCCESS [1:36.708s]
[INFO] Mahout H2O backend ................................
FAILURE [43.843s]
[INFO] Mahout Spark bindings .............................
SKIPPED
[INFO] Mahout Spark bindings shell .......................
SKIPPED
[INFO] Mahout Release Package ............................
SKIPPED
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 17:50.229s
[INFO] Finished at: Fri Apr 17 17:31:56 UTC 2015
[INFO] Final Memory: 57M/438M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.scalatest:scalatest-maven-plugin:1.0:test (test) on
project mahout-h2o_2.10: There are test failures -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run
Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full
debug logging.
[ERROR]
[ERROR] For more information about the errors and possible
solutions, please read the following articles:
[ERROR] [Help
1]http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the
build with the command
[ERROR] mvn <goals> -rf :mahout-h2o_2.10
Build step 'Invoke top-level Maven targets' marked build as
failure
[PMD] Skipping publisher since build result is FAILURE
[TASKS] Skipping publisher since build result is FAILURE
Re: h2o failures on jenkins
Posted by Andrew Palumbo <ap...@outlook.com>.
Thanks Anand. I pushed this as a quick fix which will hopefully keep
Jenkins happy. But we should probably look into a different test
context for the future.
On 04/17/2015 03:06 PM, Anand Avati wrote:
> Fix is posted here: https://github.com/apache/mahout/pull/119
>
> On Fri, 17 Apr 2015 at 12:04 Andrew Palumbo <ap...@outlook.com> wrote:
>
>>
>>
>> -------- Forwarded Message --------
>> Subject: Re: h2o failures on jenkins
>> Date: Fri, 17 Apr 2015 18:35:42 +0000
>> From: Anand Avati <av...@gluster.org>
>> To: Andrew Palumbo <ap...@outlook.com>
>>
>>
>>
>> I'm testing this patch:
>>
>> commit fafa597e0637a59249d754fb5a99d9746b1032f3
>> Author: Anand Avati <avati@redhat.com <ma...@redhat.com>>
>> Date: Fri Apr 17 11:34:45 2015 -0700
>>
>> h2o-test: use unique session id strings
>>
>> Signed-off-by: Anand Avati <avati@redhat.com <mailto:
>> avati@redhat.com>>
>>
>> diff --git
>> a/h2o/src/test/scala/org/apache/mahout/h2obindings/test/DistributedH2OSuite.scala
>> b/h2o/src/test/scala/org/apache/mahout/h2obindings/test/Distribute
>> index 45c77a5..abb4289 100644
>> ---
>> a/h2o/src/test/scala/org/apache/mahout/h2obindings/test/DistributedH2OSuite.scala
>> +++
>> b/h2o/src/test/scala/org/apache/mahout/h2obindings/test/DistributedH2OSuite.scala
>> @@ -30,7 +30,7 @@ trait DistributedH2OSuite extends DistributedMahoutSuite
>> with LoggerConfiguratio
>> override protected def beforeEach() {
>> super.beforeEach()
>>
>> - mahoutCtx = mahoutH2OContext("mah2out")
>> + mahoutCtx = mahoutH2OContext("mah2out" + System.currentTimeMillis())
>> }
>>
>> override protected def afterEach() {
>>
>>
>> On Fri, 17 Apr 2015 at 11:31 Anand Avati <avati@gluster.org
>> <ma...@gluster.org>> wrote:
>>
>> That might be a quick fix. Just FYI - for a multi-node test
>> (whenever we start testing that), all nodes of a session should
>> share the exact string.
>>
>> On Fri, 17 Apr 2015 at 11:27 Andrew Palumbo <ap.dev@outlook.com
>> <ma...@outlook.com>> wrote:
>>
>> I suppose than an easy fix would be to name the session
>> something like "mah2out"+getcurrenttimemilliseconds () or
>> something like that.
>>
>>
>> Sent from my Verizon Wireless 4G LTE smartphone
>>
>>
>> -------- Original message --------
>> From: Anand Avati
>> Date:04/17/2015 2:23 PM (GMT-05:00)
>> To: Andrew Palumbo
>> Subject: Re: h2o failures on jenkins
>>
>> It is quite possible that the two tests (based on hadoop2 and
>> hadoop1) are accidentally discovering each other when run in
>> parallel. I think giving each test a separate session id might
>> fix this problem. Currently the session id string "mah2out" is
>> hardcoded in the test/ code. If that can be somehow set to a
>> unique string at runtime (through env variable?), that might do
>> the trick. Thoughts on what's the best way we could try this?
>>
>>
>> On Fri, 17 Apr 2015 at 10:57 Andrew Palumbo <ap.dev@outlook.com
>> <ma...@outlook.com>> wrote:
>>
>> Hey Anand,
>> The h2o Module has been failing regularly in the Jenkins
>> build. It
>> seems to fail about every other build test. (i.e. it does
>> still pass).
>> Jenkins was recently moved to a new server around the same
>> time that we
>> started getting this. We also are running tests for 2
>> profiles now,
>> hadoop 1 and hadoop 2, which made me think that its possible
>> that they
>> are discovering each other and running as a 2 node cloud
>> (just a theory)?
>>
>> I was thinking maybe we should have a different test context
>> for h2o
>> which does not call
>>
>> H2O.joinOthers();
>>
>> Not sure if this is the answer though. Any thoughts?
>>
>> [32mDistributedDecompositionsSuite: [0m
>> 04-17 17:31:31.521 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: -----
>> H2O started -----
>> 04-17 17:31:31.577 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
>> git branch: master
>> 04-17 17:31:31.577 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
>> git hash: f6be02b33d23e5be437bbd1b4055c0f1955e1083
>> 04-17 17:31:31.577 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
>> git describe: RELEASE-0.1.24-47-gf6be02b
>> 04-17 17:31:31.577 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
>> project version: 0.1.25.99999
>> 04-17 17:31:31.577 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Built
>> by: 'michal'
>> 04-17 17:31:31.577 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Built
>> on: '2015-02-05 10:44:16'
>> 04-17 17:31:31.578 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
>> availableProcessors: 16
>> 04-17 17:31:31.581 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
>> heap totalMemory: 77.4 MB
>> 04-17 17:31:31.581 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
>> heap maxMemory: 910.3 MB
>> 04-17 17:31:31.581 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
>> version: Java 1.7.0_25 (from Oracle Corporation)
>> 04-17 17:31:31.581 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: OS
>> version: Linux 3.13.0-36-lowlatency (i386)
>> 04-17 17:31:31.582 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
>> Possible IP Address: docker0 (docker0),
>> fe80:0:0:0:5484:7aff:fefe:9799%4
>> 04-17 17:31:31.582 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
>> Possible IP Address: docker0 (docker0), 172.17.42.1
>> 04-17 17:31:31.582 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
>> Possible IP Address: em1 (em1),
>> fe80:0:0:0:7ae3:b5ff:fe17:8984%2
>> 04-17 17:31:31.582 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
>> Possible IP Address: em1 (em1), 67.195.81.190
>> 04-17 17:31:31.582 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
>> Possible IP Address: lo (lo), 0:0:0:0:0:0:0:1%1
>> 04-17 17:31:31.582 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
>> Possible IP Address: lo (lo), 127.0.0.1
>> 04-17 17:31:31.582 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
>> Multiple local IPs detected:
>> 04-17 17:31:31.582 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
>> /172.17.42.1 <http://172.17.42.1> /67.195.81.190
>> <http://67.195.81.190>
>> 04-17 17:31:31.582 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
>> Attempting to determine correct address...
>> 04-17 17:31:31.582 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Using
>> /67.195.81.190 <http://67.195.81.190>
>> 04-17 17:31:31.583 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
>> Internal communication uses port: 54322
>> 04-17 17:31:31.583 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
>> Listening for HTTP and REST traffic
>> onhttp://67.195.81.190:54321/
>> 04-17 <http://67.195.81.190:54321/04-17> 17:31:31.583
>> 67.195.81.190:54321 <http://67.195.81.190:54321> 17293
>> #onsSuite INFO: H2O cloud name: 'mah2out' on
>> /67.195.81.190:54321 <http://67.195.81.190:54321>, discovery
>> address /229.95.69.48:58719 <http://229.95.69.48:58719>
>> 04-17 17:31:31.583 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: If you
>> have trouble connecting, try SSH tunneling from your local
>> machine (e.g., via port 55555):
>> 04-17 17:31:31.583 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: 1.
>> Open a terminal and run 'ssh -L
>> 55555:localhost:54321jenkins@67.195.81.190
>> <ma...@67.195.81.190>'
>> 04-17 17:31:31.583 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: 2.
>> Point your browser tohttp://localhost:55555
>> 04-17 17:31:31.682 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Log
>> dir: '/tmp/h2o-jenkins/h2ologs'
>> 04-17 17:31:31.682 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Cur
>> dir:<https://builds.apache.org/job/Mahout-Quality/ws/h2o>
>> 04-17 17:31:31.813 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Cloud
>> of size 1 formed [/67.195.81.190:54321
>> <http://67.195.81.190:54321>]
>> 04-17 17:31:31.993 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
>> Receiver error on port
>> 54322java.lang.ArrayIndexOutOfBoundsException: -21
>> 04-17 17:31:36.009 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
>> Receiver error on port
>> 54322java.lang.ArrayIndexOutOfBoundsException: -17
>> 04-17 17:31:37.016 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
>> Receiver error on port
>> 54322java.lang.ArrayIndexOutOfBoundsException: -16
>> 04-17 17:31:38.023 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
>> Receiver error on port
>> 54322java.lang.ArrayIndexOutOfBoundsException: -15
>> 04-17 17:31:41.038 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
>> Receiver error on port
>> 54322java.lang.ArrayIndexOutOfBoundsException: -12
>> 04-17 17:31:42.043 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
>> Receiver error on port
>> 54322java.lang.ArrayIndexOutOfBoundsException: -11
>> 04-17 17:31:43.048 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
>> Receiver error on port
>> 54322java.lang.ArrayIndexOutOfBoundsException: -10
>> 04-17 17:31:46.061 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
>> Receiver error on port
>> 54322java.lang.ArrayIndexOutOfBoundsException: -7
>> 04-17 17:31:47.067 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
>> Receiver error on port
>> 54322java.lang.ArrayIndexOutOfBoundsException: -6
>> 04-17 17:31:48.071 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
>> Receiver error on port
>> 54322java.lang.ArrayIndexOutOfBoundsException: -5
>> 04-17 17:31:51.081 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
>> Receiver error on port
>> 54322java.lang.ArrayIndexOutOfBoundsException: -2
>> 04-17 17:31:52.105 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
>> Receiver error on port
>> 54322java.lang.ArrayIndexOutOfBoundsException: -1
>> 04-17 17:31:56.118 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: Out of
>> Memory and no swap space left from /67.195.81.188:54321
>> <http://67.195.81.188:54321>
>> 04-17 17:31:56.118 67.195.81.190:54321
>> <http://67.195.81.190:54321> 17293 #UDP-Recv FATAL: Exiting.
>> [INFO]
>> [INFO]
>>
>> ------------------------------------------------------------------------
>> [INFO] Skipping Apache Mahout
>> [INFO] This project has been banned from the build due to
>> previous failures.
>> [INFO]
>>
>> ------------------------------------------------------------------------
>> [INFO]
>> [INFO]
>>
>> ------------------------------------------------------------------------
>> [INFO] Skipping Mahout Build Tools
>> [INFO] This project has been banned from the build due to
>> previous failures.
>> [INFO]
>>
>> ------------------------------------------------------------------------
>> [INFO]
>>
>> ------------------------------------------------------------------------
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] Mahout Build Tools ................................
>> SUCCESS [1.989s]
>> [INFO] Apache Mahout .....................................
>> SUCCESS [0.206s]
>> [INFO] Mahout Math .......................................
>> SUCCESS [1:14.339s]
>> [INFO] Mahout HDFS .......................................
>> SUCCESS [3.378s]
>> [INFO] Mahout Map-Reduce .................................
>> SUCCESS [12:50.243s]
>> [INFO] Mahout Integration ................................
>> SUCCESS [54.793s]
>> [INFO] Mahout Examples ...................................
>> SUCCESS [22.309s]
>> [INFO] Mahout Math Scala bindings ........................
>> SUCCESS [1:36.708s]
>> [INFO] Mahout H2O backend ................................
>> FAILURE [43.843s]
>> [INFO] Mahout Spark bindings .............................
>> SKIPPED
>> [INFO] Mahout Spark bindings shell .......................
>> SKIPPED
>> [INFO] Mahout Release Package ............................
>> SKIPPED
>> [INFO]
>>
>> ------------------------------------------------------------------------
>> [INFO] BUILD FAILURE
>> [INFO]
>>
>> ------------------------------------------------------------------------
>> [INFO] Total time: 17:50.229s
>> [INFO] Finished at: Fri Apr 17 17:31:56 UTC 2015
>> [INFO] Final Memory: 57M/438M
>> [INFO]
>>
>> ------------------------------------------------------------------------
>> [ERROR] Failed to execute goal
>> org.scalatest:scalatest-maven-plugin:1.0:test (test) on
>> project mahout-h2o_2.10: There are test failures -> [Help 1]
>> [ERROR]
>> [ERROR] To see the full stack trace of the errors, re-run
>> Maven with the -e switch.
>> [ERROR] Re-run Maven using the -X switch to enable full
>> debug logging.
>> [ERROR]
>> [ERROR] For more information about the errors and possible
>> solutions, please read the following articles:
>> [ERROR] [Help
>> 1]
>> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
>> [ERROR]
>> [ERROR] After correcting the problems, you can resume the
>> build with the command
>> [ERROR] mvn <goals> -rf :mahout-h2o_2.10
>> Build step 'Invoke top-level Maven targets' marked build as
>> failure
>> [PMD] Skipping publisher since build result is FAILURE
>> [TASKS] Skipping publisher since build result is FAILURE
>>
>>
>>
>>
>>
Re: Re: h2o failures on jenkins
Posted by Anand Avati <av...@gluster.org>.
Fix is posted here: https://github.com/apache/mahout/pull/119
On Fri, 17 Apr 2015 at 12:04 Andrew Palumbo <ap...@outlook.com> wrote:
>
>
>
> -------- Forwarded Message --------
> Subject: Re: h2o failures on jenkins
> Date: Fri, 17 Apr 2015 18:35:42 +0000
> From: Anand Avati <av...@gluster.org>
> To: Andrew Palumbo <ap...@outlook.com>
>
>
>
> I'm testing this patch:
>
> commit fafa597e0637a59249d754fb5a99d9746b1032f3
> Author: Anand Avati <avati@redhat.com <ma...@redhat.com>>
> Date: Fri Apr 17 11:34:45 2015 -0700
>
> h2o-test: use unique session id strings
>
> Signed-off-by: Anand Avati <avati@redhat.com <mailto:
> avati@redhat.com>>
>
> diff --git
> a/h2o/src/test/scala/org/apache/mahout/h2obindings/test/DistributedH2OSuite.scala
> b/h2o/src/test/scala/org/apache/mahout/h2obindings/test/Distribute
> index 45c77a5..abb4289 100644
> ---
> a/h2o/src/test/scala/org/apache/mahout/h2obindings/test/DistributedH2OSuite.scala
> +++
> b/h2o/src/test/scala/org/apache/mahout/h2obindings/test/DistributedH2OSuite.scala
> @@ -30,7 +30,7 @@ trait DistributedH2OSuite extends DistributedMahoutSuite
> with LoggerConfiguratio
> override protected def beforeEach() {
> super.beforeEach()
>
> - mahoutCtx = mahoutH2OContext("mah2out")
> + mahoutCtx = mahoutH2OContext("mah2out" + System.currentTimeMillis())
> }
>
> override protected def afterEach() {
>
>
> On Fri, 17 Apr 2015 at 11:31 Anand Avati <avati@gluster.org
> <ma...@gluster.org>> wrote:
>
> That might be a quick fix. Just FYI - for a multi-node test
> (whenever we start testing that), all nodes of a session should
> share the exact string.
>
> On Fri, 17 Apr 2015 at 11:27 Andrew Palumbo <ap.dev@outlook.com
> <ma...@outlook.com>> wrote:
>
> I suppose than an easy fix would be to name the session
> something like "mah2out"+getcurrenttimemilliseconds () or
> something like that.
>
>
> Sent from my Verizon Wireless 4G LTE smartphone
>
>
> -------- Original message --------
> From: Anand Avati
> Date:04/17/2015 2:23 PM (GMT-05:00)
> To: Andrew Palumbo
> Subject: Re: h2o failures on jenkins
>
> It is quite possible that the two tests (based on hadoop2 and
> hadoop1) are accidentally discovering each other when run in
> parallel. I think giving each test a separate session id might
> fix this problem. Currently the session id string "mah2out" is
> hardcoded in the test/ code. If that can be somehow set to a
> unique string at runtime (through env variable?), that might do
> the trick. Thoughts on what's the best way we could try this?
>
>
> On Fri, 17 Apr 2015 at 10:57 Andrew Palumbo <ap.dev@outlook.com
> <ma...@outlook.com>> wrote:
>
> Hey Anand,
> The h2o Module has been failing regularly in the Jenkins
> build. It
> seems to fail about every other build test. (i.e. it does
> still pass).
> Jenkins was recently moved to a new server around the same
> time that we
> started getting this. We also are running tests for 2
> profiles now,
> hadoop 1 and hadoop 2, which made me think that its possible
> that they
> are discovering each other and running as a 2 node cloud
> (just a theory)?
>
> I was thinking maybe we should have a different test context
> for h2o
> which does not call
>
> H2O.joinOthers();
>
> Not sure if this is the answer though. Any thoughts?
>
> [32mDistributedDecompositionsSuite: [0m
> 04-17 17:31:31.521 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: -----
> H2O started -----
> 04-17 17:31:31.577 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
> git branch: master
> 04-17 17:31:31.577 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
> git hash: f6be02b33d23e5be437bbd1b4055c0f1955e1083
> 04-17 17:31:31.577 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
> git describe: RELEASE-0.1.24-47-gf6be02b
> 04-17 17:31:31.577 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Build
> project version: 0.1.25.99999
> 04-17 17:31:31.577 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Built
> by: 'michal'
> 04-17 17:31:31.577 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Built
> on: '2015-02-05 10:44:16'
> 04-17 17:31:31.578 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
> availableProcessors: 16
> 04-17 17:31:31.581 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
> heap totalMemory: 77.4 MB
> 04-17 17:31:31.581 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
> heap maxMemory: 910.3 MB
> 04-17 17:31:31.581 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Java
> version: Java 1.7.0_25 (from Oracle Corporation)
> 04-17 17:31:31.581 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: OS
> version: Linux 3.13.0-36-lowlatency (i386)
> 04-17 17:31:31.582 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
> Possible IP Address: docker0 (docker0),
> fe80:0:0:0:5484:7aff:fefe:9799%4
> 04-17 17:31:31.582 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
> Possible IP Address: docker0 (docker0), 172.17.42.1
> 04-17 17:31:31.582 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
> Possible IP Address: em1 (em1),
> fe80:0:0:0:7ae3:b5ff:fe17:8984%2
> 04-17 17:31:31.582 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
> Possible IP Address: em1 (em1), 67.195.81.190
> 04-17 17:31:31.582 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
> Possible IP Address: lo (lo), 0:0:0:0:0:0:0:1%1
> 04-17 17:31:31.582 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
> Possible IP Address: lo (lo), 127.0.0.1
> 04-17 17:31:31.582 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
> Multiple local IPs detected:
> 04-17 17:31:31.582 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
> /172.17.42.1 <http://172.17.42.1> /67.195.81.190
> <http://67.195.81.190>
> 04-17 17:31:31.582 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
> Attempting to determine correct address...
> 04-17 17:31:31.582 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Using
> /67.195.81.190 <http://67.195.81.190>
> 04-17 17:31:31.583 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
> Internal communication uses port: 54322
> 04-17 17:31:31.583 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO:
> Listening for HTTP and REST traffic
> onhttp://67.195.81.190:54321/
> 04-17 <http://67.195.81.190:54321/04-17> 17:31:31.583
> 67.195.81.190:54321 <http://67.195.81.190:54321> 17293
> #onsSuite INFO: H2O cloud name: 'mah2out' on
> /67.195.81.190:54321 <http://67.195.81.190:54321>, discovery
> address /229.95.69.48:58719 <http://229.95.69.48:58719>
> 04-17 17:31:31.583 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: If you
> have trouble connecting, try SSH tunneling from your local
> machine (e.g., via port 55555):
> 04-17 17:31:31.583 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: 1.
> Open a terminal and run 'ssh -L
> 55555:localhost:54321jenkins@67.195.81.190
> <ma...@67.195.81.190>'
> 04-17 17:31:31.583 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: 2.
> Point your browser tohttp://localhost:55555
> 04-17 17:31:31.682 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Log
> dir: '/tmp/h2o-jenkins/h2ologs'
> 04-17 17:31:31.682 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Cur
> dir:<https://builds.apache.org/job/Mahout-Quality/ws/h2o>
> 04-17 17:31:31.813 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #onsSuite INFO: Cloud
> of size 1 formed [/67.195.81.190:54321
> <http://67.195.81.190:54321>]
> 04-17 17:31:31.993 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
> Receiver error on port
> 54322java.lang.ArrayIndexOutOfBoundsException: -21
> 04-17 17:31:36.009 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
> Receiver error on port
> 54322java.lang.ArrayIndexOutOfBoundsException: -17
> 04-17 17:31:37.016 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
> Receiver error on port
> 54322java.lang.ArrayIndexOutOfBoundsException: -16
> 04-17 17:31:38.023 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
> Receiver error on port
> 54322java.lang.ArrayIndexOutOfBoundsException: -15
> 04-17 17:31:41.038 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
> Receiver error on port
> 54322java.lang.ArrayIndexOutOfBoundsException: -12
> 04-17 17:31:42.043 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
> Receiver error on port
> 54322java.lang.ArrayIndexOutOfBoundsException: -11
> 04-17 17:31:43.048 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
> Receiver error on port
> 54322java.lang.ArrayIndexOutOfBoundsException: -10
> 04-17 17:31:46.061 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
> Receiver error on port
> 54322java.lang.ArrayIndexOutOfBoundsException: -7
> 04-17 17:31:47.067 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
> Receiver error on port
> 54322java.lang.ArrayIndexOutOfBoundsException: -6
> 04-17 17:31:48.071 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
> Receiver error on port
> 54322java.lang.ArrayIndexOutOfBoundsException: -5
> 04-17 17:31:51.081 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
> Receiver error on port
> 54322java.lang.ArrayIndexOutOfBoundsException: -2
> 04-17 17:31:52.105 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: UDP
> Receiver error on port
> 54322java.lang.ArrayIndexOutOfBoundsException: -1
> 04-17 17:31:56.118 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #UDP-Recv ERRR: Out of
> Memory and no swap space left from /67.195.81.188:54321
> <http://67.195.81.188:54321>
> 04-17 17:31:56.118 67.195.81.190:54321
> <http://67.195.81.190:54321> 17293 #UDP-Recv FATAL: Exiting.
> [INFO]
> [INFO]
>
> ------------------------------------------------------------------------
> [INFO] Skipping Apache Mahout
> [INFO] This project has been banned from the build due to
> previous failures.
> [INFO]
>
> ------------------------------------------------------------------------
> [INFO]
> [INFO]
>
> ------------------------------------------------------------------------
> [INFO] Skipping Mahout Build Tools
> [INFO] This project has been banned from the build due to
> previous failures.
> [INFO]
>
> ------------------------------------------------------------------------
> [INFO]
>
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Mahout Build Tools ................................
> SUCCESS [1.989s]
> [INFO] Apache Mahout .....................................
> SUCCESS [0.206s]
> [INFO] Mahout Math .......................................
> SUCCESS [1:14.339s]
> [INFO] Mahout HDFS .......................................
> SUCCESS [3.378s]
> [INFO] Mahout Map-Reduce .................................
> SUCCESS [12:50.243s]
> [INFO] Mahout Integration ................................
> SUCCESS [54.793s]
> [INFO] Mahout Examples ...................................
> SUCCESS [22.309s]
> [INFO] Mahout Math Scala bindings ........................
> SUCCESS [1:36.708s]
> [INFO] Mahout H2O backend ................................
> FAILURE [43.843s]
> [INFO] Mahout Spark bindings .............................
> SKIPPED
> [INFO] Mahout Spark bindings shell .......................
> SKIPPED
> [INFO] Mahout Release Package ............................
> SKIPPED
> [INFO]
>
> ------------------------------------------------------------------------
> [INFO] BUILD FAILURE
> [INFO]
>
> ------------------------------------------------------------------------
> [INFO] Total time: 17:50.229s
> [INFO] Finished at: Fri Apr 17 17:31:56 UTC 2015
> [INFO] Final Memory: 57M/438M
> [INFO]
>
> ------------------------------------------------------------------------
> [ERROR] Failed to execute goal
> org.scalatest:scalatest-maven-plugin:1.0:test (test) on
> project mahout-h2o_2.10: There are test failures -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run
> Maven with the -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full
> debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible
> solutions, please read the following articles:
> [ERROR] [Help
> 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the
> build with the command
> [ERROR] mvn <goals> -rf :mahout-h2o_2.10
> Build step 'Invoke top-level Maven targets' marked build as
> failure
> [PMD] Skipping publisher since build result is FAILURE
> [TASKS] Skipping publisher since build result is FAILURE
>
>
>
>
>