You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Chris Mattmann <ch...@jpl.nasa.gov> on 2005/09/27 06:59:58 UTC
failing of org.apache.nutch.tools.TestSegmentMergeTool?
Hi there,
I just noticed after checking out the latest SVN of Nutch that I am
currently failing the TestSegmentMergeTool Junit test when I type "ant test"
for Nutch. Is anyone experiencing the same problem? Here is the relevant
information which I captured out of the
$NUTCH_HOME/build/test/TEST-org.apache.nutch.tools.TestSegmentMergeTool.txt
file:
Testsuite: org.apache.nutch.tools.TestSegmentMergeTool
Tests run: 3, Failures: 1, Errors: 0, Time elapsed: 46.256 sec
------------- Standard Error -----------------
050926 215316 parsing
file:/C:/Program%20Files/eclipse/workspace/nutch/conf/nutch-default.xml
050926 215316 parsing
file:/C:/Program%20Files/eclipse/workspace/nutch/build/test/classes/nutch-si
te.xml
050926 215316 Using URL normalizer: org.apache.nutch.net.BasicUrlNormalizer
050926 215321 No FS indicated, using default:local
050926 215321 * Opening 10 segments:
050926 215321 - segment seg0: 500 records.
050926 215321 - segment seg1: 500 records.
050926 215321 - segment seg2: 500 records.
050926 215321 - segment seg3: 500 records.
050926 215321 - segment seg4: 500 records.
050926 215321 - segment seg5: 500 records.
050926 215321 - segment seg6: 500 records.
050926 215321 - segment seg7: 500 records.
050926 215321 - segment seg8: 500 records.
050926 215321 - segment seg9: 500 records.
050926 215321 * TOTAL 5000 input records in 10 segments.
050926 215321 * Creating master index...
050926 215328 * Creating index took 6356 ms
050926 215328 * Optimizing index took 0 ms
050926 215328 * Removing duplicate entries...
050926 215328 * Deduplicating took 652 ms
050926 215328 * Merging all segments into output
050926 215333 * Merging took 4381 ms
050926 215333 * Deleting old segments...
050926 215333 Finished SegmentMergeTool: INPUT: 5000 -> OUTPUT: 5000 entries
in 12.15 s (416.66666 entries/sec).
050926 215339 No FS indicated, using default:local
050926 215339 * Opening 10 segments:
050926 215339 - segment seg0: 500 records.
050926 215339 - segment seg1: 500 records.
050926 215339 - segment seg2: 500 records.
050926 215339 - segment seg3: 500 records.
050926 215339 - segment seg4: 500 records.
050926 215339 - segment seg5: 500 records.
050926 215339 - segment seg6: 500 records.
050926 215339 - segment seg7: 500 records.
050926 215339 - segment seg8: 500 records.
050926 215339 - segment seg9: 500 records.
050926 215339 * TOTAL 5000 input records in 10 segments.
050926 215339 * Creating master index...
050926 215344 * Creating index took 5083 ms
050926 215344 * Optimizing index took 0 ms
050926 215344 * Removing duplicate entries...
050926 215344 * Deduplicating took 150 ms
050926 215344 * Merging all segments into output
050926 215345 * Merging took 662 ms
050926 215345 * Deleting old segments...
050926 215345 Finished SegmentMergeTool: INPUT: 5000 -> OUTPUT: 500 entries
in 6.316 s (833.3333 entries/sec).
java.lang.Exception: Missing or invalid 'fetcher' or 'fetcher_output'
directory in
c:\DOCUME~1\mattmann\LOCALS~1\Temp\.smttest63088\output\.fastmerge_index
at
org.apache.nutch.segment.SegmentReader.isParsedSegment(SegmentReader.java:16
8)
at
org.apache.nutch.segment.SegmentReader.<init>(SegmentReader.java:143)
at
org.apache.nutch.segment.SegmentReader.<init>(SegmentReader.java:82)
at
org.apache.nutch.tools.TestSegmentMergeTool.testSameMerge(TestSegmentMergeTo
ol.java:185)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at junit.framework.TestCase.runTest(TestCase.java:154)
at junit.framework.TestCase.runBare(TestCase.java:127)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRu
nner.java:289)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestR
unner.java:523)
junit.framework.AssertionFailedError: Missing or invalid 'fetcher' or
'fetcher_output' directory in
c:\DOCUME~1\mattmann\LOCALS~1\Temp\.smttest63088\output\.fastmerge_index
at junit.framework.Assert.fail(Assert.java:47)
at
org.apache.nutch.tools.TestSegmentMergeTool.testSameMerge(TestSegmentMergeTo
ol.java:190)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at junit.framework.TestCase.runTest(TestCase.java:154)
at junit.framework.TestCase.runBare(TestCase.java:127)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRu
nner.java:289)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestR
unner.java:523)
050926 215351 No FS indicated, using default:local
050926 215351 * Opening 10 segments:
050926 215351 - segment seg0: 0 records.
050926 215351 - segment seg1: 500 records.
050926 215351 Unable to fix segment seg2: Missing data file in
c:\DOCUME~1\mattmann\LOCALS~1\Temp\.smttest63089\segments\seg2\content,
impossible to fix this.
050926 215351 - fixed fetcher
050926 215351 - fixed content
050926 215351 - fixed parse_data
050926 215351 - fixed parse_text
050926 215351 Finished fixing seg3
050926 215351 - segment seg3: 500 records.
050926 215351 - segment seg4: 500 records.
050926 215351 - segment seg5: 500 records.
050926 215351 - segment seg6: 500 records.
050926 215351 - segment seg7: 500 records.
050926 215351 - segment seg8: 500 records.
050926 215351 - segment seg9: 500 records.
050926 215351 * TOTAL 4000 input records in 9 segments.
050926 215351 * Creating master index...
050926 215352 - segment seg1 truncated to 376 records
050926 215355 * Creating index took 4020 ms
050926 215355 * Optimizing index took 0 ms
050926 215355 * Removing duplicate entries...
050926 215356 * Deduplicating took 311 ms
050926 215356 * Merging all segments into output
050926 215402 * Merging took 6356 ms
050926 215402 * Deleting old segments...
050926 215402 Finished SegmentMergeTool: INPUT: 4000 -> OUTPUT: 3875 entries
in 11.218 s (363.63635 entries/sec).
------------- ---------------- ---------------
Testcase: testUniqueMerge took 16.982 sec
Testcase: testSameMerge took 12.181 sec
FAILED
Missing or invalid 'fetcher' or 'fetcher_output' directory in
c:\DOCUME~1\mattmann\LOCALS~1\Temp\.smttest63088\output\.fastmerge_index
junit.framework.AssertionFailedError: Missing or invalid 'fetcher' or
'fetcher_output' directory in
c:\DOCUME~1\mattmann\LOCALS~1\Temp\.smttest63088\output\.fastmerge_index
at
org.apache.nutch.tools.TestSegmentMergeTool.testSameMerge(TestSegmentMergeTo
ol.java:194)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
Testcase: testCorruptSegmentMerge took 17.093 sec
Any ideas why this is happening?
Thanks,
Chris
______________________________________________
Chris A. Mattmann
Chris.Mattmann@jpl.nasa.gov
Staff Member
Modeling and Data Management Systems Section (387)
Data Management Systems and Technologies Group
_________________________________________________
Jet Propulsion Laboratory Pasadena, CA
Office: 171-266B Mailstop: 171-246
_______________________________________________________
Disclaimer: The opinions presented within are my own and do not reflect
those of either NASA, JPL, or the California Institute of Technology.
Re: failing of org.apache.nutch.tools.TestSegmentMergeTool?
Posted by Chris Mattmann <ch...@jpl.nasa.gov>.
You know what the crazy thing is:
Seemingly, all tests pass now. And I didn't change a thing. Honest. I swear.
Very strange, indeed, but I'm happy because at least the tests are passing!
:-)
Cheers,
Chris
On 9/27/05 12:29 PM, "Paul Baclace" <pe...@baclace.net> wrote:
> Chris Mattmann wrote:
>> I just noticed after checking out the latest SVN of Nutch that I am
>> currently failing the TestSegmentMergeTool Junit test when I type "ant test"
>> for Nutch.
>
> I'm on the mapred branch, not the trunk, and all tests pass.
>
> One thing I have noticed is that it is best to start with 'ant clean'
> and if you made any mods to the conf files, rewind them back by copying
> the x.template files to x.
>
> Paul
______________________________________________
Chris A. Mattmann
Chris.Mattmann@jpl.nasa.gov
Staff Member
Modeling and Data Management Systems Section (387)
Data Management Systems and Technologies Group
_________________________________________________
Jet Propulsion Laboratory Pasadena, CA
Office: 171-266B Mailstop: 171-246
_______________________________________________________
Disclaimer: The opinions presented within are my own and do not reflect
those of either NASA, JPL, or the California Institute of Technology.
Re: failing of org.apache.nutch.tools.TestSegmentMergeTool?
Posted by Paul Baclace <pe...@baclace.net>.
Chris Mattmann wrote:
> I just noticed after checking out the latest SVN of Nutch that I am
> currently failing the TestSegmentMergeTool Junit test when I type "ant test"
> for Nutch.
I'm on the mapred branch, not the trunk, and all tests pass.
One thing I have noticed is that it is best to start with 'ant clean'
and if you made any mods to the conf files, rewind them back by copying
the x.template files to x.
Paul