You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Ferdy Galema <fe...@kalooga.com> on 2011/11/12 19:12:44 UTC
test failures Nutchgora
Hi,
I cannot get a grasp on this. I cleanly checked out Nutchgora, but the
tests just fail. It's TestAPI and TestGoraStorage. I noticed this
problem before a while ago but then it seemed cleaning caches and build
folders resolved it. Now it is rearing it's ugly head again.
Logs indicate that TestAPI failes because of assert failures (RUNNING
job status vs something else) and also connection failures to SQL.
TestGoraStorage failes with the multi process test throwing "to many
open files" (but I can migitate the latter problem by setting lowering
loop times).
Then when I enable building with HBase (decomment lines in ivy) I get
compile errors in TestGoraStorage (at setDeamon call). This has
something to do with multiple version of the HSQL libs?
I'm completely clueless about what this might be. I tried everything,
also using different versions of JDK. Monday I might have more info, as
I will be back on my pc at work where I should have a working copy. Any
help in the meantime would be great.
Ferdy.
Re: test failures Nutchgora
Posted by Ferdy Galema <fe...@kalooga.com>.
Thanks for clearing up some of these things. I've updated NUTCH-1081
with this thread. We should deal with this test soon. Also I'm
definitely up for remaining a single build (and just remove this extra
ant build)
On 11/15/2011 09:35 PM, Lewis John Mcgibbney wrote:
> Hi Ferdy,
>
> OK so the issue is TestAPI... here we go again ;0)
>
> Typical output for the test can be seen here [1]. Just for the record,
> Markus mentioned a while ago that we should have some place to view
> these test output files, so at least we've now found them in the build
> directory inside the Jenkins Nutch workspace.
>
> I remember when we were fixing the tests before, the issue I opened
> [2] actually highlighted that TestAPI was failing when I initially
> opened the issue, however this miraculously resolved itself and after
> lots and lots of testing I could not reproduce. I think we need to
> look really closely at TestAPI, at what it does, and try to identify
> where the bug is! It sounds like there is definitely some fishy
> business going on.
>
> Lets leave this issue open and have a look at it eh?
>
> [1]
> https://builds.apache.org/view/M-R/view/Nutch/job/Nutch-nutchgora/ws/nutchgora/build/test/TEST-org.apache.nutch.api.TestAPI.txt/*view*/
> [2] https://issues.apache.org/jira/browse/NUTCH-1081
>
>
> On Mon, Nov 14, 2011 at 11:28 AM, Ferdy Galema
> <ferdy.galema@kalooga.com <ma...@kalooga.com>> wrote:
>
> I've pinpointed the problem to a single test, namely TestAPI
> (TestGoraStorage seems to work fine, I guess the failure at my
> home workstation was a memory issue). This is what happends after
> I do a clean checkout Nutchgora from my regular workstation.
>
> "ant test"
> TestAPI fails, see extracted lines from TEST-TestAPI.txt:
>
> Caused by: java.io.IOException:
> java.sql.SQLTransientConnectionException:
> java.net.ConnectException: Connection refused
> ...
> java.lang.IllegalStateException: Job in state DEFINE instead of
> RUNNING
> ...
> junit.framework.AssertionFailedError: expected:<[FAILED]> but
> was:<[RUNNING]>
> at org.apache.nutch.api.TestAPI.testJobsAPI(TestAPI.java:118)
>
> The insane part is the fact that I had a working copy of Nutchgora
> just a few moments ago, from which I reverted the local workspace
> changes (that could have absolutely nothing to do with these
> issues) whereafter the test failed again. I re-applied the
> changes, but TestAPI just does not work anymore. This working copy
> is 3 revisions behind head (that is head minus 1 commit of Markus
> and 2 commits of Lewis). Then, "rm build/ runtime/ -r" followed by
> "ant runtime" and "ant test" and suddenly TestAPI worked! But,
> this is not reproducible because rerunning the tests just failed
> it again. So I guess the test just randomly succeeds. I'm sorry if
> this is becoming a really vague description, but it certainly is a
> real vague problem. It's driving me nuts.
>
> So the underlying question is, what is TestAPI
> (NutchServer/NutchApp) and what purpose does it serve? I'm just
> not familiar enough with these components.
>
> By the way, could somebody explain what the difference is between
> the following two build jobs?
> https://builds.apache.org/job/Nutch-nutchgora-ant/
> https://builds.apache.org/job/Nutch-nutchgora/
>
> Because in Nutch-nutchgora-ant, TestAPI also fails:
> [junit] Tests run: 4, Failures: 1, Errors: 0, Time elapsed: 11.081 sec
> [junit] Test org.apache.nutch.api.TestAPI FAILED
>
> In Nutch-nutchgora, it works perfectly fine.
>
> Ferdy.
>
>
> On 11/12/2011 08:57 PM, Andrzej Bialecki wrote:
>
> On 12/11/2011 19:12, Ferdy Galema wrote:
>
> Hi,
>
> I cannot get a grasp on this. I cleanly checked out
> Nutchgora, but the
> tests just fail. It's TestAPI and TestGoraStorage. I
> noticed this
> problem before a while ago but then it seemed cleaning
> caches and build
> folders resolved it. Now it is rearing it's ugly head again.
>
> Logs indicate that TestAPI failes because of assert
> failures (RUNNING
> job status vs something else) and also connection failures
> to SQL.
> TestGoraStorage failes with the multi process test
> throwing "to many
> open files" (but I can migitate the latter problem by
> setting lowering
> loop times).
>
> Then when I enable building with HBase (decomment lines in
> ivy) I get
> compile errors in TestGoraStorage (at setDeamon call).
> This has
> something to do with multiple version of the HSQL libs?
>
> I'm completely clueless about what this might be. I tried
> everything,
> also using different versions of JDK. Monday I might have
> more info, as
> I will be back on my pc at work where I should have a
> working copy. Any
> help in the meantime would be great.
>
>
> Hmmm, I vaguely recall that either Hadoop or HBase comes with
> an incompatible version of HSQL lib. This is a transitive
> dependency of Gora, and when it's pulled in it conflicts with
> the version we use in Nutch.
>
>
>
>
>
> --
> /Lewis/
>
Re: test failures Nutchgora
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Ferdy,
OK so the issue is TestAPI... here we go again ;0)
Typical output for the test can be seen here [1]. Just for the record,
Markus mentioned a while ago that we should have some place to view these
test output files, so at least we've now found them in the build directory
inside the Jenkins Nutch workspace.
I remember when we were fixing the tests before, the issue I opened [2]
actually highlighted that TestAPI was failing when I initially opened the
issue, however this miraculously resolved itself and after lots and lots of
testing I could not reproduce. I think we need to look really closely at
TestAPI, at what it does, and try to identify where the bug is! It sounds
like there is definitely some fishy business going on.
Lets leave this issue open and have a look at it eh?
[1]
https://builds.apache.org/view/M-R/view/Nutch/job/Nutch-nutchgora/ws/nutchgora/build/test/TEST-org.apache.nutch.api.TestAPI.txt/*view*/
[2] https://issues.apache.org/jira/browse/NUTCH-1081
On Mon, Nov 14, 2011 at 11:28 AM, Ferdy Galema <fe...@kalooga.com>wrote:
> I've pinpointed the problem to a single test, namely TestAPI
> (TestGoraStorage seems to work fine, I guess the failure at my home
> workstation was a memory issue). This is what happends after I do a clean
> checkout Nutchgora from my regular workstation.
>
> "ant test"
> TestAPI fails, see extracted lines from TEST-TestAPI.txt:
>
> Caused by: java.io.IOException: java.sql.**SQLTransientConnectionExceptio*
> *n: java.net.ConnectException: Connection refused
> ...
> java.lang.**IllegalStateException: Job in state DEFINE instead of RUNNING
> ...
> junit.framework.**AssertionFailedError: expected:<[FAILED]> but
> was:<[RUNNING]>
> at org.apache.nutch.api.TestAPI.**testJobsAPI(TestAPI.java:118)
>
> The insane part is the fact that I had a working copy of Nutchgora just a
> few moments ago, from which I reverted the local workspace changes (that
> could have absolutely nothing to do with these issues) whereafter the test
> failed again. I re-applied the changes, but TestAPI just does not work
> anymore. This working copy is 3 revisions behind head (that is head minus 1
> commit of Markus and 2 commits of Lewis). Then, "rm build/ runtime/ -r"
> followed by "ant runtime" and "ant test" and suddenly TestAPI worked! But,
> this is not reproducible because rerunning the tests just failed it again.
> So I guess the test just randomly succeeds. I'm sorry if this is becoming a
> really vague description, but it certainly is a real vague problem. It's
> driving me nuts.
>
> So the underlying question is, what is TestAPI (NutchServer/NutchApp) and
> what purpose does it serve? I'm just not familiar enough with these
> components.
>
> By the way, could somebody explain what the difference is between the
> following two build jobs?
> https://builds.apache.org/job/**Nutch-nutchgora-ant/<https://builds.apache.org/job/Nutch-nutchgora-ant/>
> https://builds.apache.org/job/**Nutch-nutchgora/<https://builds.apache.org/job/Nutch-nutchgora/>
>
> Because in Nutch-nutchgora-ant, TestAPI also fails:
> [junit] Tests run: 4, Failures: 1, Errors: 0, Time elapsed: 11.081 sec
> [junit] Test org.apache.nutch.api.TestAPI FAILED
>
> In Nutch-nutchgora, it works perfectly fine.
>
> Ferdy.
>
>
> On 11/12/2011 08:57 PM, Andrzej Bialecki wrote:
>
>> On 12/11/2011 19:12, Ferdy Galema wrote:
>>
>>> Hi,
>>>
>>> I cannot get a grasp on this. I cleanly checked out Nutchgora, but the
>>> tests just fail. It's TestAPI and TestGoraStorage. I noticed this
>>> problem before a while ago but then it seemed cleaning caches and build
>>> folders resolved it. Now it is rearing it's ugly head again.
>>>
>>> Logs indicate that TestAPI failes because of assert failures (RUNNING
>>> job status vs something else) and also connection failures to SQL.
>>> TestGoraStorage failes with the multi process test throwing "to many
>>> open files" (but I can migitate the latter problem by setting lowering
>>> loop times).
>>>
>>> Then when I enable building with HBase (decomment lines in ivy) I get
>>> compile errors in TestGoraStorage (at setDeamon call). This has
>>> something to do with multiple version of the HSQL libs?
>>>
>>> I'm completely clueless about what this might be. I tried everything,
>>> also using different versions of JDK. Monday I might have more info, as
>>> I will be back on my pc at work where I should have a working copy. Any
>>> help in the meantime would be great.
>>>
>>
>> Hmmm, I vaguely recall that either Hadoop or HBase comes with an
>> incompatible version of HSQL lib. This is a transitive dependency of Gora,
>> and when it's pulled in it conflicts with the version we use in Nutch.
>>
>>
>>
--
*Lewis*
Re: test failures Nutchgora
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi,
Firstly let me separate some things here for clarification before we
discuss the TestAPI in more detail.
On Mon, Nov 14, 2011 at 11:28 AM, Ferdy Galema <fe...@kalooga.com>wrote:
> Then, "rm build/ runtime/ -r" followed by "ant runtime" and "ant test"
> and suddenly TestAPI worked!
>
My own advice is, if your wanting to clean out your working copy of
Nutchgora then use "ant clean" check out some of the other tasks, I think
they are nearer the bottom of the build.xml
>
> By the way, could somebody explain what the difference is between the
> following two build jobs?
> https://builds.apache.org/job/**Nutch-nutchgora-ant/<https://builds.apache.org/job/Nutch-nutchgora-ant/>
> https://builds.apache.org/job/**Nutch-nutchgora/<https://builds.apache.org/job/Nutch-nutchgora/>
>
>
> In Nutch-nutchgora, it works perfectly fine.
OK so Nutch-nutchgora-ant was a temporary build that Oliver Lamy set up,
this was when our builds were really in the sh*t and was an on the fly
solution to getting a test of sorts up and running. The reason it has
remained is that it has simply slipped under the radar until you brought it
up.
Hopefully this has clarified some of the smaller points you make here
Ferdy. In all honesty, we can progress with killing the Nutch-nutchgora-ant
and Nutch-trunk-ant builds. Actually I will disable them both just now.
There is no harm in having disabled builds stored in the Nutch Jenkins area.
Please read other thread for my other input on the TestAPI fiasco.
Thanks
Re: test failures Nutchgora
Posted by Ferdy Galema <fe...@kalooga.com>.
I've pinpointed the problem to a single test, namely TestAPI
(TestGoraStorage seems to work fine, I guess the failure at my home
workstation was a memory issue). This is what happends after I do a
clean checkout Nutchgora from my regular workstation.
"ant test"
TestAPI fails, see extracted lines from TEST-TestAPI.txt:
Caused by: java.io.IOException:
java.sql.SQLTransientConnectionException: java.net.ConnectException:
Connection refused
...
java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING
...
junit.framework.AssertionFailedError: expected:<[FAILED]> but
was:<[RUNNING]>
at org.apache.nutch.api.TestAPI.testJobsAPI(TestAPI.java:118)
The insane part is the fact that I had a working copy of Nutchgora just
a few moments ago, from which I reverted the local workspace changes
(that could have absolutely nothing to do with these issues) whereafter
the test failed again. I re-applied the changes, but TestAPI just does
not work anymore. This working copy is 3 revisions behind head (that is
head minus 1 commit of Markus and 2 commits of Lewis). Then, "rm build/
runtime/ -r" followed by "ant runtime" and "ant test" and suddenly
TestAPI worked! But, this is not reproducible because rerunning the
tests just failed it again. So I guess the test just randomly succeeds.
I'm sorry if this is becoming a really vague description, but it
certainly is a real vague problem. It's driving me nuts.
So the underlying question is, what is TestAPI (NutchServer/NutchApp)
and what purpose does it serve? I'm just not familiar enough with these
components.
By the way, could somebody explain what the difference is between the
following two build jobs?
https://builds.apache.org/job/Nutch-nutchgora-ant/
https://builds.apache.org/job/Nutch-nutchgora/
Because in Nutch-nutchgora-ant, TestAPI also fails:
[junit] Tests run: 4, Failures: 1, Errors: 0, Time elapsed: 11.081 sec
[junit] Test org.apache.nutch.api.TestAPI FAILED
In Nutch-nutchgora, it works perfectly fine.
Ferdy.
On 11/12/2011 08:57 PM, Andrzej Bialecki wrote:
> On 12/11/2011 19:12, Ferdy Galema wrote:
>> Hi,
>>
>> I cannot get a grasp on this. I cleanly checked out Nutchgora, but the
>> tests just fail. It's TestAPI and TestGoraStorage. I noticed this
>> problem before a while ago but then it seemed cleaning caches and build
>> folders resolved it. Now it is rearing it's ugly head again.
>>
>> Logs indicate that TestAPI failes because of assert failures (RUNNING
>> job status vs something else) and also connection failures to SQL.
>> TestGoraStorage failes with the multi process test throwing "to many
>> open files" (but I can migitate the latter problem by setting lowering
>> loop times).
>>
>> Then when I enable building with HBase (decomment lines in ivy) I get
>> compile errors in TestGoraStorage (at setDeamon call). This has
>> something to do with multiple version of the HSQL libs?
>>
>> I'm completely clueless about what this might be. I tried everything,
>> also using different versions of JDK. Monday I might have more info, as
>> I will be back on my pc at work where I should have a working copy. Any
>> help in the meantime would be great.
>
> Hmmm, I vaguely recall that either Hadoop or HBase comes with an
> incompatible version of HSQL lib. This is a transitive dependency of
> Gora, and when it's pulled in it conflicts with the version we use in
> Nutch.
>
>
Re: test failures Nutchgora
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 12/11/2011 19:12, Ferdy Galema wrote:
> Hi,
>
> I cannot get a grasp on this. I cleanly checked out Nutchgora, but the
> tests just fail. It's TestAPI and TestGoraStorage. I noticed this
> problem before a while ago but then it seemed cleaning caches and build
> folders resolved it. Now it is rearing it's ugly head again.
>
> Logs indicate that TestAPI failes because of assert failures (RUNNING
> job status vs something else) and also connection failures to SQL.
> TestGoraStorage failes with the multi process test throwing "to many
> open files" (but I can migitate the latter problem by setting lowering
> loop times).
>
> Then when I enable building with HBase (decomment lines in ivy) I get
> compile errors in TestGoraStorage (at setDeamon call). This has
> something to do with multiple version of the HSQL libs?
>
> I'm completely clueless about what this might be. I tried everything,
> also using different versions of JDK. Monday I might have more info, as
> I will be back on my pc at work where I should have a working copy. Any
> help in the meantime would be great.
Hmmm, I vaguely recall that either Hadoop or HBase comes with an
incompatible version of HSQL lib. This is a transitive dependency of
Gora, and when it's pulled in it conflicts with the version we use in Nutch.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com