You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Hetal Shah <he...@investorsprovident.com> on 2007/01/31 12:27:59 UTC
Dedup index error
Hey guys
Been breaking my head over this error for a while now, but don't seem to be
getting anywhere! I have tried creating / recreating the index several
times, and also made sure that all settings were as "per the book". I read
somewhere on one of the other posts that this error could be due to a
corrupted index, but somehow, I don't think that's the case. I only have a
few urls in the index with depth 1, so it's not even a large crawl!
There are two directories in my crawled/indexes directory, viz. part-00000
and part-00001.
PS. This is a fresh install of nutch with a fresh index.
Please help before I go insane!!!!
Error log
Dedup: starting
Dedup: adding indexes in: crawled/indexes
DeleteDuplicates: java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:402)
at
org.apache.nutch.indexer.DeleteDuplicates.dedup(DeleteDuplicates.java:439)
at
org.apache.nutch.indexer.DeleteDuplicates.run(DeleteDuplicates.java:506)
at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:189)
at
org.apache.nutch.indexer.DeleteDuplicates.main(DeleteDuplicates.java:490)
Additional error log
Task TASKID="tip_0009_m_000001" TASK_TYPE="MAP" TASK_STATUS="FAILED"
FINISH_TIME="1170237489795" ERROR="java.lang.ArrayIndexOutOfBoundsException:
-1
at org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:109)
at
org.apache.nutch.indexer.DeleteDuplicates$InputFormat$DDRecordReader.next(De
leteDuplicates.java:176)
at org.apache.hadoop.mapred.MapTask$2.next(MapTask.java:166)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:46)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:183)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1367)
Thanks
Hetal
RE: Dedup index error
Posted by Hetal Shah <he...@investorsprovident.com>.
Thanks Andrzej. I don't think my scenario would be applicable in real-life
situations. However, it would be great to know where the root of the problem
lies.
I have managed to dedup a larger index, and is working perfect. So your
theory is correct. I guess it's a matter of digging a little deeper to
eliminate this once and for all.
Thanks.
-----Original Message-----
From: Andrzej Bialecki [mailto:ab@getopt.org]
Sent: 01 February 2007 17:59
To: nutch-user@lucene.apache.org
Subject: Re: Dedup index error
Hetal Shah wrote:
> Another quick update:
>
> I ran Luke on the index, and part-00000 works fine, whereas part-00001
> comes up as corrupt or missing. Now seeing from the list of files in
> both these directories, we know that there is nothing in part-00001 -
> so why does it get generated? And if it does, why does dedup not handle it
gracefully?
>
> I also ran a merge on the two indexes, and it worked fine.
>
> So that rests the case that both the indexes are corrupted. This
> brings me to understand that since I only had two pages indexed and
> the index was small, part-00001 came up with nothing, and dedup does not
handle it????
>
> Any thoughts?
>
There seems to be an issue with the document partitioning - it seems that
for larger numbers of document the partitioning schema generates at least
one document per partition, but in your case there were too few documents to
fill the second partition ... I need to check where the problem originates -
however, this should not happen if you index more documents than 2 * the
number of reduce tasks.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \|
|| | Embedded Unix, System Integration http://www.sigram.com Contact:
info at sigram dot com
Re: Dedup index error
Posted by Andrzej Bialecki <ab...@getopt.org>.
Hetal Shah wrote:
> Another quick update:
>
> I ran Luke on the index, and part-00000 works fine, whereas part-00001 comes
> up as corrupt or missing. Now seeing from the list of files in both these
> directories, we know that there is nothing in part-00001 - so why does it
> get generated? And if it does, why does dedup not handle it gracefully?
>
> I also ran a merge on the two indexes, and it worked fine.
>
> So that rests the case that both the indexes are corrupted. This brings me
> to understand that since I only had two pages indexed and the index was
> small, part-00001 came up with nothing, and dedup does not handle it????
>
> Any thoughts?
>
There seems to be an issue with the document partitioning - it seems
that for larger numbers of document the partitioning schema generates at
least one document per partition, but in your case there were too few
documents to fill the second partition ... I need to check where the
problem originates - however, this should not happen if you index more
documents than 2 * the number of reduce tasks.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
RE: Dedup index error
Posted by Hetal Shah <he...@investorsprovident.com>.
Another quick update:
I ran Luke on the index, and part-00000 works fine, whereas part-00001 comes
up as corrupt or missing. Now seeing from the list of files in both these
directories, we know that there is nothing in part-00001 - so why does it
get generated? And if it does, why does dedup not handle it gracefully?
I also ran a merge on the two indexes, and it worked fine.
So that rests the case that both the indexes are corrupted. This brings me
to understand that since I only had two pages indexed and the index was
small, part-00001 came up with nothing, and dedup does not handle it????
Any thoughts?
-- Hetal Shah wrote: --
That's what I had read on another post as well, but somehow, I can't
understand how it can be corrupted! It's not even a massive index. Just a
couple of urls. Every step that I followed was per the tutorials on the wiki
page.
Here's the list under /indexes:
drwxr-xr-x 2 root root 4096 Jan 31 16:21 part-00000
drwxr-xr-x 2 root root 4096 Jan 31 16:21 part-00001
This is what's under part-00000
-rw-r--r-- 1 root root 2 Jan 31 16:21 _2.f0
-rw-r--r-- 1 root root 2 Jan 31 16:21 _2.f1
-rw-r--r-- 1 root root 2 Jan 31 16:21 _2.f2
-rw-r--r-- 1 root root 2 Jan 31 16:21 _2.f3
-rw-r--r-- 1 root root 2 Jan 31 16:21 _2.f4
-rw-r--r-- 1 root root 2 Jan 31 16:21 _2.f5
-rw-r--r-- 1 root root 399 Jan 31 16:21 _2.fdt
-rw-r--r-- 1 root root 16 Jan 31 16:21 _2.fdx
-rw-r--r-- 1 root root 74 Jan 31 16:21 _2.fnm
-rw-r--r-- 1 root root 945 Jan 31 16:21 _2.frq
-rw-r--r-- 1 root root 1790 Jan 31 16:21 _2.prx
-rw-r--r-- 1 root root 105 Jan 31 16:21 _2.tii
-rw-r--r-- 1 root root 6850 Jan 31 16:21 _2.tis
-rw-r--r-- 1 root root 4 Jan 31 16:21 deletable
-rw-r--r-- 1 root root 0 Jan 31 16:21 index.done
-rw-r--r-- 1 root root 27 Jan 31 16:21 segments
This is what's under part-00001
-rw-r--r-- 1 root root 0 Jan 31 16:21 index.done
-rw-r--r-- 1 root root 20 Jan 31 16:21 segments
By the way, also to mention here that I am running dedup on the DFS system.
I haven't tried running it on the local system yet, but does that matter?
Thanks for your help.
RE: Dedup index error
Posted by Hetal Shah <he...@investorsprovident.com>.
That's what I had read on another post as well, but somehow, I can't
understand how it can be corrupted! It's not even a massive index. Just a
couple of urls. Every step that I followed was per the tutorials on the wiki
page.
Here's the list under /indexes:
drwxr-xr-x 2 root root 4096 Jan 31 16:21 part-00000
drwxr-xr-x 2 root root 4096 Jan 31 16:21 part-00001
This is what's under part-00000
-rw-r--r-- 1 root root 2 Jan 31 16:21 _2.f0
-rw-r--r-- 1 root root 2 Jan 31 16:21 _2.f1
-rw-r--r-- 1 root root 2 Jan 31 16:21 _2.f2
-rw-r--r-- 1 root root 2 Jan 31 16:21 _2.f3
-rw-r--r-- 1 root root 2 Jan 31 16:21 _2.f4
-rw-r--r-- 1 root root 2 Jan 31 16:21 _2.f5
-rw-r--r-- 1 root root 399 Jan 31 16:21 _2.fdt
-rw-r--r-- 1 root root 16 Jan 31 16:21 _2.fdx
-rw-r--r-- 1 root root 74 Jan 31 16:21 _2.fnm
-rw-r--r-- 1 root root 945 Jan 31 16:21 _2.frq
-rw-r--r-- 1 root root 1790 Jan 31 16:21 _2.prx
-rw-r--r-- 1 root root 105 Jan 31 16:21 _2.tii
-rw-r--r-- 1 root root 6850 Jan 31 16:21 _2.tis
-rw-r--r-- 1 root root 4 Jan 31 16:21 deletable
-rw-r--r-- 1 root root 0 Jan 31 16:21 index.done
-rw-r--r-- 1 root root 27 Jan 31 16:21 segments
This is what's under part-00001
-rw-r--r-- 1 root root 0 Jan 31 16:21 index.done
-rw-r--r-- 1 root root 20 Jan 31 16:21 segments
By the way, also to mention here that I am running dedup on the DFS system.
I haven't tried running it on the local system yet, but does that matter?
Thanks for your help.
Hetal Shah wrote:
> Hey guys
>
> Been breaking my head over this error for a while now, but don't seem
> to be getting anywhere! I have tried creating / recreating the index
> several times, and also made sure that all settings were as "per the
> book". I read somewhere on one of the other posts that this error
> could be due to a corrupted index, but somehow, I don't think that's
> the case. I only have a few urls in the index with depth 1, so it's not
even a large crawl!
>
> There are two directories in my crawled/indexes directory, viz.
> part-00000 and part-00001.
>
Could you do an 'ls -l' to show the content and sizes of these parts?
>
> Task TASKID="tip_0009_m_000001" TASK_TYPE="MAP" TASK_STATUS="FAILED"
> FINISH_TIME="1170237489795"
ERROR="java.lang.ArrayIndexOutOfBoundsException:
> -1
> at
> org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:109)
>
This usually indicates that one or more indexes under crawled/indexes is
invalid - nonexistent, incomplete or corrupt.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \|
|| | Embedded Unix, System Integration http://www.sigram.com Contact:
info at sigram dot com
Re: Dedup index error
Posted by Andrzej Bialecki <ab...@getopt.org>.
Hetal Shah wrote:
> Hey guys
>
> Been breaking my head over this error for a while now, but don't seem to be
> getting anywhere! I have tried creating / recreating the index several
> times, and also made sure that all settings were as "per the book". I read
> somewhere on one of the other posts that this error could be due to a
> corrupted index, but somehow, I don't think that's the case. I only have a
> few urls in the index with depth 1, so it's not even a large crawl!
>
> There are two directories in my crawled/indexes directory, viz. part-00000
> and part-00001.
>
Could you do an 'ls -l' to show the content and sizes of these parts?
>
> Task TASKID="tip_0009_m_000001" TASK_TYPE="MAP" TASK_STATUS="FAILED"
> FINISH_TIME="1170237489795" ERROR="java.lang.ArrayIndexOutOfBoundsException:
> -1
> at org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:109)
>
This usually indicates that one or more indexes under crawled/indexes is
invalid - nonexistent, incomplete or corrupt.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com