You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Manoharam Reddy <ma...@gmail.com> on 2007/05/29 06:38:29 UTC
mergesegs is not functioning properly
My segment merger is not functioning properly. I am unable to figure
out the problem.
These are the commands I am using.
bin/nutch inject crawl/crawldb seedurls
In a loop iterating 10 times:-
bin/nutch generate crawl/crawldb crawl/segments -topN 1000 -adddays 5
segment=`ls -d crawl/segments/* | tail -1`
bin/nutch fetch $segment -threads 50
bin/nutch updatedb crawl/crawldb $segment
After loop:-
bin/nutch mergesegs crawl/merged_segments crawl/segments/*
rm -rf crawl/segments/*
mv --verbose crawl/merged_segments/* crawl/segments
rm -rf crawl/merged_segments
Merging 10 segments to crawl/MERGEDsegments/20050529095045
SegmentMerger: adding crawl/segments/20050528144604
SegmentMerger: adding crawl/segments/20050528144619
SegmentMerger: adding crawl/segments/20050528145426
SegmentMerger: adding crawl/segments/20050528151323
SegmentMerger: adding crawl/segments/20050528164032
SegmentMerger: adding crawl/segments/20050528170544
SegmentMerger: adding crawl/segments/20050528192341
SegmentMerger: adding crawl/segments/20050528203512
SegmentMerger: adding crawl/segments/20050528210029
SegmentMerger: adding crawl/segments/20050529055733
SegmentMerger: using segment data from: crawl_generate
`crawl/MERGEDsegments/20050529095045' -> `crawl/segments/20050529095045'
As can be seen here, only crawl_generate was used to merge. Other
folders like parse_data, crawl_fetch were not used. Why?
So while inverting the links with this command:-
bin/nutch invertlinks crawl/linkdb crawl/segments/*
I get this error:-
LinkDb: starting
LinkDb: linkdb: crawl/linkdb
LinkDb: URL normalize: true
LinkDb: URL filter: true
LinkDb: adding segment: crawl/segments/20050529095045
LinkDb: org.apache.hadoop.mapred.InvalidInputException: Input path
doesnt exist : /opt/nutch-0.9infy00/crawl/segments/20050529095045/parse_data
at org.apache.hadoop.mapred.InputFormatBase.validateInput(InputFormatBase.java:138)
Please help.
Re: mergesegs is not functioning properly
Posted by Manoharam Reddy <ma...@gmail.com>.
I went back to my logs and found the cause of the error:-
-----
fetch of http://shoppingcenter/home.asp failed with:
java.lang.OutOfMemoryError: Java heap space
heap space
at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:45)
at java.lang.StringBuilder.<init>(StringBuilder.java:68)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:557)
at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:470)
at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:505)
at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:189)
at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:477)
java.lang.NullPointerException
at org.apache.hadoop.fs.FSDataInputStream$Buffer.getPos(FSDataInputStream.java:87)
at org.apache.hadoop.fs.FSDataInputStream.getPos(FSDataInputStream.java:125)
at org.apache.hadoop.io.SequenceFile$Reader.getPosition(SequenceFile.java:1736)
at org.apache.hadoop.mapred.SequenceFileRecordReader.getProgress(SequenceFileRecordReader.java:108)
at org.apache.hadoop.mapred.MapTask$1.getProgress(MapTask.java:165)
at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:155)
at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:115)
fetcher caught:java.lang.NullPointerException
fetcher caught:java.lang.NullPointerException
CrawlDb update: starting
CrawlDb update: db: crawl/crawldb
CrawlDb update: segments: [crawl/segments/20070529170525]
CrawlDb update: additions allowed: true
CrawlDb update: URL normalizing: false
CrawlDb update: URL filtering: false
- skipping invalid segment crawl/segments/20070529170525
CrawlDb update: Merging segment data into db.
CrawlDb update: done
-----
The NullPointerException was mentioned multiple times in the stdout
log. But I have removed them above to maintain clarity.
As a result this segment had only crawl_generate and nothing else. Can
anyone please explain me what caused this error? How can I prevent
this error from happening?
On 5/29/07, Andrzej Bialecki <ab...@getopt.org> wrote:
> Manoharam Reddy wrote:
> > My segment merger is not functioning properly. I am unable to figure
> > out the problem.
> >
> > These are the commands I am using.
> >
> > bin/nutch inject crawl/crawldb seedurls
> >
> > In a loop iterating 10 times:-
> >
> > bin/nutch generate crawl/crawldb crawl/segments -topN 1000 -adddays 5
> > segment=`ls -d crawl/segments/* | tail -1`
> > bin/nutch fetch $segment -threads 50
> > bin/nutch updatedb crawl/crawldb $segment
> >
> > After loop:-
> >
> > bin/nutch mergesegs crawl/merged_segments crawl/segments/*
> > rm -rf crawl/segments/*
> > mv --verbose crawl/merged_segments/* crawl/segments
> > rm -rf crawl/merged_segments
> >
> > Merging 10 segments to crawl/MERGEDsegments/20050529095045
> > SegmentMerger: adding crawl/segments/20050528144604
> > SegmentMerger: adding crawl/segments/20050528144619
> > SegmentMerger: adding crawl/segments/20050528145426
> > SegmentMerger: adding crawl/segments/20050528151323
> > SegmentMerger: adding crawl/segments/20050528164032
> > SegmentMerger: adding crawl/segments/20050528170544
> > SegmentMerger: adding crawl/segments/20050528192341
> > SegmentMerger: adding crawl/segments/20050528203512
> > SegmentMerger: adding crawl/segments/20050528210029
> > SegmentMerger: adding crawl/segments/20050529055733
> > SegmentMerger: using segment data from: crawl_generate
> > `crawl/MERGEDsegments/20050529095045' -> `crawl/segments/20050529095045'
> >
> > As can be seen here, only crawl_generate was used to merge. Other
> > folders like parse_data, crawl_fetch were not used. Why?
>
> This behavior is described in the javadoc of SegmentMerger. In general
> case, users may wish to merge segments at different stages of processing
> - only generated, fetched but not parsed, and parsed. It's easy to do
> this if segments are homogenous, i.e. they all contain the same parts.
>
> However, if segments are heterogenous, i.e some of them are processed
> further than others, we cannot merge all their parts, because we will
> get an incomplete segment as a result (e.g for some urls we will have
> parse_data, for other urls it will be missing).
>
> In such cases SegmentMerger processes only the lowest common
> denominator, i.e. only those segment parts that are present in all input
> segments - and disregards any other existing parts.
>
> That's a long answer to your problem, which is that one or more of your
> input segments isn't fetched yet - don't include that segment on the
> list of input segments, and all should be fine.
>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
Re: mergesegs is not functioning properly
Posted by Andrzej Bialecki <ab...@getopt.org>.
Manoharam Reddy wrote:
> My segment merger is not functioning properly. I am unable to figure
> out the problem.
>
> These are the commands I am using.
>
> bin/nutch inject crawl/crawldb seedurls
>
> In a loop iterating 10 times:-
>
> bin/nutch generate crawl/crawldb crawl/segments -topN 1000 -adddays 5
> segment=`ls -d crawl/segments/* | tail -1`
> bin/nutch fetch $segment -threads 50
> bin/nutch updatedb crawl/crawldb $segment
>
> After loop:-
>
> bin/nutch mergesegs crawl/merged_segments crawl/segments/*
> rm -rf crawl/segments/*
> mv --verbose crawl/merged_segments/* crawl/segments
> rm -rf crawl/merged_segments
>
> Merging 10 segments to crawl/MERGEDsegments/20050529095045
> SegmentMerger: adding crawl/segments/20050528144604
> SegmentMerger: adding crawl/segments/20050528144619
> SegmentMerger: adding crawl/segments/20050528145426
> SegmentMerger: adding crawl/segments/20050528151323
> SegmentMerger: adding crawl/segments/20050528164032
> SegmentMerger: adding crawl/segments/20050528170544
> SegmentMerger: adding crawl/segments/20050528192341
> SegmentMerger: adding crawl/segments/20050528203512
> SegmentMerger: adding crawl/segments/20050528210029
> SegmentMerger: adding crawl/segments/20050529055733
> SegmentMerger: using segment data from: crawl_generate
> `crawl/MERGEDsegments/20050529095045' -> `crawl/segments/20050529095045'
>
> As can be seen here, only crawl_generate was used to merge. Other
> folders like parse_data, crawl_fetch were not used. Why?
This behavior is described in the javadoc of SegmentMerger. In general
case, users may wish to merge segments at different stages of processing
- only generated, fetched but not parsed, and parsed. It's easy to do
this if segments are homogenous, i.e. they all contain the same parts.
However, if segments are heterogenous, i.e some of them are processed
further than others, we cannot merge all their parts, because we will
get an incomplete segment as a result (e.g for some urls we will have
parse_data, for other urls it will be missing).
In such cases SegmentMerger processes only the lowest common
denominator, i.e. only those segment parts that are present in all input
segments - and disregards any other existing parts.
That's a long answer to your problem, which is that one or more of your
input segments isn't fetched yet - don't include that segment on the
list of input segments, and all should be fine.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com