You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Mathijs Homminga <ma...@knowlogy.nl> on 2007/02/25 22:21:43 UTC

Recovering aborted fetch

Hi,

While fetching a segment with 4M documents, we ran out of diskspace.
We guess that the fetcher has fetched (and parsed) about 80 percent of 
the documents, so it would be great if we could continue our crawl somehow.

The segment directory does not contain a crawl_fetch subdirectory yet. 
But we have a /tmp/hadoop/mapred/ (Local FS) directory.

Is there some way we can use the data in the temporary mapred directory 
to create the crawl_fetch data in order to continue our crawl?

Thanks!
Mathijs


Re: Recovering aborted fetch

Posted by Charlie Williams <cw...@gmail.com>.
Our fetch completed, but during the reduce phase we had an IO error, there
was no information provided beyond this in the logs.


On 2/26/07, Mathijs Homminga <ma...@knowlogy.nl> wrote:
>
> Hi Andrzej,
>
> The job stopped because there was no space left on the disk:
>
> FATAL fetcher.Fetcher - org.apache.hadoop.fs.FSError:
> java.io.IOException: No space left on device
> FATAL fetcher.Fetcher - at
> org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(
> LocalFileSystem.java:150)
> FATAL fetcher.Fetcher - at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(
> FSDataOutputStream.java:112)
>
> We use a local FS. Temporary data is stored in /tmp/hadoop/mapred/
>
> Mathijs
>
> Andrzej Bialecki wrote:
> > Charlie Williams wrote:
> >> I had this same problem, we had gathered about 90% of a 1.5M page
> >> fetch only
> >> to have the system crash at the reduce phase. We now do cycles of
> >> about 50k
> >> pages at a time to minimize loss.
> >
> >
> > I may have a possible workaround for this issue. It would help to know
> > the failure modes in your cases - i.e. what were the symptoms, and
> > what lead to the ultimate job failure - was it a real crash of the
> > machine, the tasktracker, or the task was submitted too many times, or
> > it appeared to be stuck for a long time, or you just killed the job -
> > and whether you used the regex URLFilter.
> >
> > BTW. creating a "fetcher.done" file doesn't work in 0.8+ - the output
> > itself was simply discarded when the job ended without closing its
> > file descriptors. If you were running on a local FS you may be able to
> > recover some of the data from temporary files - but if you ran with
> > DFS then the files physically don't exist anywhere, and all data has
> > been discarded ...
> >
>
>

Re: Recovering aborted fetch

Posted by Shailendra Mudgal <mu...@gmail.com>.
Hi Andrzej,

Thanks for the tool. It allowed me to recover a failed fetch due to Disk
space.

Regards,
Vipin

On Mon, Mar 12, 2007 at 7:35 PM, Mathijs Homminga <
mathijs.homminga@knowlogy.nl> wrote:

> Hi all,
>
> We managed to recover most of our data using Andrzej's LocalFetchRecover
> tool.
>
> For more info see:
> http://issues.apache.org/jira/browse/NUTCH-451
>
> Mathijs
>
>
>
> Andrzej Bialecki wrote:
>
>> Mathijs Homminga wrote:
>>
>>> Hi Andrzej,
>>>
>>> Thanks for the tool!
>>>
>>> I found one 'map_xxxxxx' directory which matches the date my segment was
>>> created.
>>> It contains a 'part-0.out' file with a timestamp that matches the time of
>>> the last entries in my log file (just before the process stopped).
>>>
>>> I followed the preparation steps and ran the tool. However, I got the
>>> following error:
>>>
>>
>> The SequenceFile has the following structure (approximately): first, three
>> letter magic 'SEQ', and then either the fully qualified class names for
>> key/value, or abbreviated classnames obtained from the mapping in
>> WritableName. Please check what is this class name in part-0.out file -
>> apparently Hadoop can't find the right mapping (you can send me the first
>> bytes of this file off the list - use 'dd if=part-0.out of=data.out bs=512
>> count=1').
>>
>>  By looking at the Hadoop sources I noticed that the FetcherOutput class
>>> mentioned in this error message is determined by the SequenceFile class and
>>> obtained from the sequence file itself.
>>>
>>
>> The class itself is not obtained from the file - it's loaded from the
>> classpath. The thing that is missing here is the right name of the class as
>> determined by the symbolic name inside the SequenceFile.
>>
>>
>

Re: Recovering aborted fetch

Posted by Mathijs Homminga <ma...@knowlogy.nl>.
Hi all,

We managed to recover most of our data using Andrzej's LocalFetchRecover 
tool.

For more info see:
http://issues.apache.org/jira/browse/NUTCH-451

Mathijs


Andrzej Bialecki wrote:
> Mathijs Homminga wrote:
>> Hi Andrzej,
>>
>> Thanks for the tool!
>>
>> I found one 'map_xxxxxx' directory which matches the date my segment 
>> was created.
>> It contains a 'part-0.out' file with a timestamp that matches the 
>> time of the last entries in my log file (just before the process 
>> stopped).
>>
>> I followed the preparation steps and ran the tool. However, I got the 
>> following error:
>
> The SequenceFile has the following structure (approximately): first, 
> three letter magic 'SEQ', and then either the fully qualified class 
> names for key/value, or abbreviated classnames obtained from the 
> mapping in WritableName. Please check what is this class name in 
> part-0.out file - apparently Hadoop can't find the right mapping (you 
> can send me the first bytes of this file off the list - use 'dd 
> if=part-0.out of=data.out bs=512 count=1').
>
>> By looking at the Hadoop sources I noticed that the FetcherOutput 
>> class mentioned in this error message is determined by the 
>> SequenceFile class and obtained from the sequence file itself.
>
> The class itself is not obtained from the file - it's loaded from the 
> classpath. The thing that is missing here is the right name of the 
> class as determined by the symbolic name inside the SequenceFile.
>


Re: Recovering aborted fetch

Posted by Andrzej Bialecki <ab...@getopt.org>.
Mathijs Homminga wrote:
> Hi Andrzej,
>
> Thanks for the tool!
>
> I found one 'map_xxxxxx' directory which matches the date my segment 
> was created.
> It contains a 'part-0.out' file with a timestamp that matches the time 
> of the last entries in my log file (just before the process stopped).
>
> I followed the preparation steps and ran the tool. However, I got the 
> following error:

The SequenceFile has the following structure (approximately): first, 
three letter magic 'SEQ', and then either the fully qualified class 
names for key/value, or abbreviated classnames obtained from the mapping 
in WritableName. Please check what is this class name in part-0.out file 
- apparently Hadoop can't find the right mapping (you can send me the 
first bytes of this file off the list - use 'dd if=part-0.out 
of=data.out bs=512 count=1').

> By looking at the Hadoop sources I noticed that the FetcherOutput 
> class mentioned in this error message is determined by the 
> SequenceFile class and obtained from the sequence file itself.

The class itself is not obtained from the file - it's loaded from the 
classpath. The thing that is missing here is the right name of the class 
as determined by the symbolic name inside the SequenceFile.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com



Re: Recovering aborted fetch

Posted by Mathijs Homminga <ma...@knowlogy.nl>.
Hi Andrzej,

Thanks for the tool!

I found one 'map_xxxxxx' directory which matches the date my segment was 
created.
It contains a 'part-0.out' file with a timestamp that matches the time 
of the last entries in my log file (just before the process stopped).

I followed the preparation steps and ran the tool. However, I got the 
following error:

2007-02-27 11:27:00,416 WARN  mapred.LocalJobRunner 
(LocalJobRunner.java:run(120)) - job_sygdrx
java.io.IOException: wrong value class:  is not class 
org.apache.nutch.fetcher.FetcherOutput
        at 
org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:346)
        at 
org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:58)
        at org.apache.hadoop.mapred.MapTask$3.next(MapTask.java:119)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:46)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:129)
        at 
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:91)
2007-02-27 11:27:01,321 WARN  util.ToolBase (ToolBase.java:doMain(185)) 
- Job failed!

By looking at the Hadoop sources I noticed that the FetcherOutput class 
mentioned in this error message is determined by the SequenceFile class 
and obtained from the sequence file itself. Which, I think, indicates 
that the part-00000 file I use as input for the tool does indeed contain 
FetcherOutput object(s).
I get the same error when I remove the following line from 
LocalFetcherRecover.java:
job.setOutputValueClass(FetcherOutput.class);

Any clues?
Btw, we use Nutch 0.8.1.

Thanks,
Mathijs





Andrzej Bialecki wrote:
> Mathijs Homminga wrote:
>> Hi Andrzej,
>>
>> The job stopped because there was no space left on the disk:
>>
>> FATAL fetcher.Fetcher - org.apache.hadoop.fs.FSError: 
>> java.io.IOException: No space left on device
>> FATAL fetcher.Fetcher - at 
>> org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(LocalFileSystem.java:150) 
>>
>> FATAL fetcher.Fetcher - at 
>> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:112) 
>>
>>
>> We use a local FS. Temporary data is stored in /tmp/hadoop/mapred/
>
>
> Ok, in your case this partial data may be recoverable, but with some 
> manual work involved ...
>
> At this stage, I'm assuming that even if you started the reduce phase 
> its output won't be usable at all. So, we need to start from the data 
> contained in partial map outputs. Map outputs are a set of 
> SequenceFile's containing pairs of <Text, FetcherOutput> data. Umm, 
> forgot to ask you - are you running trunk/ or Nutch 0.8 ? If trunk, 
> then use the Text class, if 0.8 - replace all occurrences of Text with 
> UTF8.
>
> This is such a common problem that I created a special tool to address 
> this - please see http://issues.apache.org/jira/browse/NUTCH-451 .
>
> Let me repeat what the javadoc says, so that there's no 
> misunderstanding: if you use DFS and your fetch job is aborted, there 
> is no way in the world to recover the data - it's permanently lost. If 
> you run with a local FS, you can try this tool and hope for the best.
>


Re: Recovering aborted fetch

Posted by Andrzej Bialecki <ab...@getopt.org>.
Mathijs Homminga wrote:
> Hi Andrzej,
>
> The job stopped because there was no space left on the disk:
>
> FATAL fetcher.Fetcher - org.apache.hadoop.fs.FSError: 
> java.io.IOException: No space left on device
> FATAL fetcher.Fetcher - at 
> org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(LocalFileSystem.java:150) 
>
> FATAL fetcher.Fetcher - at 
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:112) 
>
>
> We use a local FS. Temporary data is stored in /tmp/hadoop/mapred/


Ok, in your case this partial data may be recoverable, but with some 
manual work involved ...

At this stage, I'm assuming that even if you started the reduce phase 
its output won't be usable at all. So, we need to start from the data 
contained in partial map outputs. Map outputs are a set of 
SequenceFile's containing pairs of <Text, FetcherOutput> data. Umm, 
forgot to ask you - are you running trunk/ or Nutch 0.8 ? If trunk, then 
use the Text class, if 0.8 - replace all occurrences of Text with UTF8.

This is such a common problem that I created a special tool to address 
this - please see http://issues.apache.org/jira/browse/NUTCH-451 .

Let me repeat what the javadoc says, so that there's no 
misunderstanding: if you use DFS and your fetch job is aborted, there is 
no way in the world to recover the data - it's permanently lost. If you 
run with a local FS, you can try this tool and hope for the best.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com



Re: Recovering aborted fetch

Posted by Mathijs Homminga <ma...@knowlogy.nl>.
Hi Andrzej,

The job stopped because there was no space left on the disk:

FATAL fetcher.Fetcher - org.apache.hadoop.fs.FSError: 
java.io.IOException: No space left on device
FATAL fetcher.Fetcher - at 
org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(LocalFileSystem.java:150)
FATAL fetcher.Fetcher - at 
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:112)

We use a local FS. Temporary data is stored in /tmp/hadoop/mapred/
 
Mathijs

Andrzej Bialecki wrote:
> Charlie Williams wrote:
>> I had this same problem, we had gathered about 90% of a 1.5M page 
>> fetch only
>> to have the system crash at the reduce phase. We now do cycles of 
>> about 50k
>> pages at a time to minimize loss.
>
>
> I may have a possible workaround for this issue. It would help to know 
> the failure modes in your cases - i.e. what were the symptoms, and 
> what lead to the ultimate job failure - was it a real crash of the 
> machine, the tasktracker, or the task was submitted too many times, or 
> it appeared to be stuck for a long time, or you just killed the job - 
> and whether you used the regex URLFilter.
>
> BTW. creating a "fetcher.done" file doesn't work in 0.8+ - the output 
> itself was simply discarded when the job ended without closing its 
> file descriptors. If you were running on a local FS you may be able to 
> recover some of the data from temporary files - but if you ran with 
> DFS then the files physically don't exist anywhere, and all data has 
> been discarded ...
>


Re: Recovering aborted fetch

Posted by Andrzej Bialecki <ab...@getopt.org>.
Charlie Williams wrote:
> I had this same problem, we had gathered about 90% of a 1.5M page 
> fetch only
> to have the system crash at the reduce phase. We now do cycles of 
> about 50k
> pages at a time to minimize loss.


I may have a possible workaround for this issue. It would help to know 
the failure modes in your cases - i.e. what were the symptoms, and what 
lead to the ultimate job failure - was it a real crash of the machine, 
the tasktracker, or the task was submitted too many times, or it 
appeared to be stuck for a long time, or you just killed the job - and 
whether you used the regex URLFilter.

BTW. creating a "fetcher.done" file doesn't work in 0.8+ - the output 
itself was simply discarded when the job ended without closing its file 
descriptors. If you were running on a local FS you may be able to 
recover some of the data from temporary files - but if you ran with DFS 
then the files physically don't exist anywhere, and all data has been 
discarded ...

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com



Re: Recovering aborted fetch

Posted by Charlie Williams <cw...@gmail.com>.
I had this same problem, we had gathered about 90% of a 1.5M page fetch only
to have the system crash at the reduce phase. We now do cycles of about 50k
pages at a time to minimize loss.

-Charlie

On 2/26/07, Mathijs Homminga <ma...@knowlogy.nl> wrote:
>
> :(
>
> I read something about creating a 'fetcher.done' file which can do some
> magic.
> Could that help us out?
>
> Mathijs
>
> rubdabadub wrote:
> > Hi:
> >
> > I am Sorry to say that you need to fetch again i.e your last segment.
> > I know the feeling :-( AFAIK there is no way in 0.8 restart a failed
> > crawl. I have found having small segment i.e generating small fetch
> > list and merging all the segment later is the only way to avoid such
> > situation.
> >
> > Regards
> >
> > On 2/25/07, Mathijs Homminga <ma...@knowlogy.nl> wrote:
> >> Hi,
> >>
> >> While fetching a segment with 4M documents, we ran out of diskspace.
> >> We guess that the fetcher has fetched (and parsed) about 80 percent of
> >> the documents, so it would be great if we could continue our crawl
> >> somehow.
> >>
> >> The segment directory does not contain a crawl_fetch subdirectory yet.
> >> But we have a /tmp/hadoop/mapred/ (Local FS) directory.
> >>
> >> Is there some way we can use the data in the temporary mapred directory
> >> to create the crawl_fetch data in order to continue our crawl?
> >>
> >> Thanks!
> >> Mathijs
> >>
> >>
>
>

Re: Recovering aborted fetch

Posted by Mathijs Homminga <ma...@knowlogy.nl>.
:(

I read something about creating a 'fetcher.done' file which can do some 
magic.
Could that help us out?

Mathijs

rubdabadub wrote:
> Hi:
>
> I am Sorry to say that you need to fetch again i.e your last segment.
> I know the feeling :-( AFAIK there is no way in 0.8 restart a failed
> crawl. I have found having small segment i.e generating small fetch
> list and merging all the segment later is the only way to avoid such
> situation.
>
> Regards
>
> On 2/25/07, Mathijs Homminga <ma...@knowlogy.nl> wrote:
>> Hi,
>>
>> While fetching a segment with 4M documents, we ran out of diskspace.
>> We guess that the fetcher has fetched (and parsed) about 80 percent of
>> the documents, so it would be great if we could continue our crawl 
>> somehow.
>>
>> The segment directory does not contain a crawl_fetch subdirectory yet.
>> But we have a /tmp/hadoop/mapred/ (Local FS) directory.
>>
>> Is there some way we can use the data in the temporary mapred directory
>> to create the crawl_fetch data in order to continue our crawl?
>>
>> Thanks!
>> Mathijs
>>
>>


Re: Recovering aborted fetch

Posted by rubdabadub <ru...@gmail.com>.
Hi:

I am Sorry to say that you need to fetch again i.e your last segment.
I know the feeling :-( AFAIK there is no way in 0.8 restart a failed
crawl. I have found having small segment i.e generating small fetch
list and merging all the segment later is the only way to avoid such
situation.

Regards

On 2/25/07, Mathijs Homminga <ma...@knowlogy.nl> wrote:
> Hi,
>
> While fetching a segment with 4M documents, we ran out of diskspace.
> We guess that the fetcher has fetched (and parsed) about 80 percent of
> the documents, so it would be great if we could continue our crawl somehow.
>
> The segment directory does not contain a crawl_fetch subdirectory yet.
> But we have a /tmp/hadoop/mapred/ (Local FS) directory.
>
> Is there some way we can use the data in the temporary mapred directory
> to create the crawl_fetch data in order to continue our crawl?
>
> Thanks!
> Mathijs
>
>