You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by spamsucks <sp...@rhoderunner.com> on 2006/12/22 17:32:21 UTC

PhasedFileSystem Exception in trunk build

I pulled from trunk yesterday, and doing a Intranet crawl, it fails.

Running on cygwin/XP

Any guidance?  Thanks.

2006-12-22 11:10:00,613 INFO  indexer.IndexingFilters - Adding 
org.apache.nutch.indexer.basic.BasicIndexingFilter
2006-12-22 11:10:01,363 WARN  mapred.LocalJobRunner - job_g7zf6p
java.lang.UnsupportedOperationException: Operation not supported
 at 
org.apache.hadoop.mapred.PhasedFileSystem.deleteRaw(PhasedFileSystem.java:291)
 at org.apache.hadoop.fs.FileSystem.delete(FileSystem.java:402)
 at 
org.apache.nutch.indexer.Indexer$OutputFormat.getRecordWriter(Indexer.java:97)
 at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:287)
 at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:137)

Re: PhasedFileSystem Exception in trunk build

Posted by Andrzej Bialecki <ab...@getopt.org>.
spamsucks wrote:
> I added the property to hadoop-site.xml, and it now fails with a 
> different error.  How about some more magical properties?

Nope ;) I think this indicates that one of your indexes is corrupt or 
absent.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

Re: PhasedFileSystem Exception in trunk build

Posted by spamsucks <sp...@rhoderunner.com>.
I added the property to hadoop-site.xml, and it now fails with a different 
error.  How about some more magical properties?
Thanks.
Phillip

<property>
  <name>mapred.speculative.execution</name>
  <value>false</value>
  <description>from the mailing list
  </description>
</property>


Stack trace:

2006-12-22 13:40:46,040 WARN  mapred.LocalJobRunner 
(LocalJobRunner.java:run(147)) - job_ambth9
java.lang.ArrayIndexOutOfBoundsException: -1
        at 
org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:109)
        at 
org.apache.nutch.indexer.DeleteDuplicates$InputFormat$DDRecordReader.next(DeleteDuplicates.java:177)
        at org.apache.hadoop.mapred.MapTask$3.next(MapTask.java:203)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:46)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:215)
        at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:109)
Exception in thread "main" java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:399)
        at 
org.apache.nutch.indexer.DeleteDuplicates.dedup(DeleteDuplicates.java:433)
        at org.apache.nutch.crawl.Crawl.main(Crawl.java:131)



----- Original Message ----- 
From: "Andrzej Bialecki" <ab...@getopt.org>
To: <nu...@lucene.apache.org>
Sent: Friday, December 22, 2006 12:50 PM
Subject: Re: PhasedFileSystem Exception in trunk build


> spamsucks wrote:
>> I pulled from trunk yesterday, and doing a Intranet crawl, it fails.
>>
>> Running on cygwin/XP
>>
>> Any guidance?  Thanks.
>>
>> 2006-12-22 11:10:00,613 INFO  indexer.IndexingFilters - Adding 
>> org.apache.nutch.indexer.basic.BasicIndexingFilter
>> 2006-12-22 11:10:01,363 WARN  mapred.LocalJobRunner - job_g7zf6p
>> java.lang.UnsupportedOperationException: Operation not supported
>> at 
>> org.apache.hadoop.mapred.PhasedFileSystem.deleteRaw(PhasedFileSystem.java:291)
>> at org.apache.hadoop.fs.FileSystem.delete(FileSystem.java:402)
>> at 
>> org.apache.nutch.indexer.Indexer$OutputFormat.getRecordWriter(Indexer.java:97)
>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:287)
>> at 
>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:137)
>
> Please set "mapred.speculative.execution" to false, in hadoop-site.xml. 
> This is addressed in the upcoming update of Hadoop v. 0.9.2.
>
> -- 
> Best regards,
> Andrzej Bialecki     <><
> ___. ___ ___ ___ _ _   __________________________________
> [__ || __|__/|__||\/|  Information Retrieval, Semantic Web
> ___|||__||  \|  ||  |  Embedded Unix, System Integration
> http://www.sigram.com  Contact: info at sigram dot com
>
>

Re: PhasedFileSystem Exception in trunk build

Posted by Andrzej Bialecki <ab...@getopt.org>.
spamsucks wrote:
> I pulled from trunk yesterday, and doing a Intranet crawl, it fails.
>
> Running on cygwin/XP
>
> Any guidance?  Thanks.
>
> 2006-12-22 11:10:00,613 INFO  indexer.IndexingFilters - Adding 
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2006-12-22 11:10:01,363 WARN  mapred.LocalJobRunner - job_g7zf6p
> java.lang.UnsupportedOperationException: Operation not supported
> at 
> org.apache.hadoop.mapred.PhasedFileSystem.deleteRaw(PhasedFileSystem.java:291) 
>
> at org.apache.hadoop.fs.FileSystem.delete(FileSystem.java:402)
> at 
> org.apache.nutch.indexer.Indexer$OutputFormat.getRecordWriter(Indexer.java:97) 
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:287)
> at 
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:137)

Please set "mapred.speculative.execution" to false, in hadoop-site.xml. 
This is addressed in the upcoming update of Hadoop v. 0.9.2.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com