You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Ar...@csiro.au on 2010/07/01 01:48:42 UTC
Generator problems in Nutch 1.1
Hi,
I am trying to use Nutch 1.1 to build a complete index of our corporate web sites. I am using a script based on this one:
http://wiki.apache.org/nutch/Crawl
The text of my script is included below. I set the crawling depth to 100 to make sure that everything is indexed, expecting that the process will stop after about 20 iterations. The problem is that the Generator keeps re-scheduling fetching of failed URLs. The process stopped because the max number of iterations (the depth) was reached. After a few iterations, only failing URLs were being repeatedly fetched. I checked the log for one of them. It failed with code 403 and was re-scheduled for fetching 94 times. This does not seem right.
Another problem that I've noticed is that sometimes the Generators schedules fetching of just one URL per iteration. This happened once and I did not try to repeat this effect, but this does not seem right either.
Here is my script text:
-----------------------------------------------
#!/bin/bash
depth=100
threads=10
adddays=5
topN=50000
NUTCH_HOME=/data/HORUS_1/nutch1.1
NUTCH_HEAPSIZE=8000
nutch=$NUTCH_HOME
JAVA_HOME=/usr/lib/jvm/java-6-sun
export NUTCH_HOME
export JAVA_HOME
export NUTCH_HEAPSIZE
steps=6
echo "----- Inject (Step 1 of $steps) -----"
$nutch/bin/nutch inject $nutch/crawl/crawldb $nutch/crawl/seed
echo "----- Generate, Fetch, Parse, Update (Step 2 of $steps) -----"
for((i=0; i < $depth; i++))
do
echo "--- Beginning crawl at depth `expr $i + 1` of $depth ---"
$nutch/bin/nutch generate $nutch/crawl/crawldb $nutch/crawl/segments $topN -adddays $adddays
if [ $? -ne 0 ]
then
echo "Stopping at depth $depth. No more URLs to fetch."
break
fi
segment=`ls -d $nutch/crawl/segments/* | tail -1`
$nutch/bin/nutch fetch $segment -threads $threads
if [ $? -ne 0 ]
then
echo "fetch $segment at depth $depth failed. Deleting it."
rm -rf $segment
continue
fi
# echo "--- Parsing Segment $segment ---"
# $nutch/bin/nutch parse $segment
$nutch/bin/nutch updatedb $nutch/crawl/crawldb $segment
done
echo "----- Invert Links (Step 3 of $steps) -----"
$nutch/bin/nutch invertlinks $nutch/crawl/linkdb $nutch/crawl/segments/*
echo "----- Index (Step 4 of $steps) -----"
$nutch/bin/nutch index $nutch/crawl/preIndex $nutch/crawl/crawldb $nutch/crawl/linkdb $nutch/crawl/segments/*
echo "----- Dedup (Step 5 of $steps) -----"
$nutch/bin/nutch dedup $nutch/crawl/preIndex
echo "----- Merge Indexes (Step 6 of $steps) -----"
$nutch/bin/nutch merge $nutch/crawl/index $nutch/crawl/preIndex
# in nutch-site, hadoop.tmp.dir points to crawl/tmp
rm -rf $nutch/crawl/tmp/*
-----------------------------------------------
Is anyone experiencing same problems? Is there anything wrong in what I am doing?
Regards,
Arkadi
RE: Generator problems in Nutch 1.1
Posted by Ar...@csiro.au.
>>you have set adddays to 5 days.
>>in this case it will be refetched and refetched again.
>>try adddays with value 0.
This helped. However, the other problem is still there. The Generator schedules for fetching 1 URL per iteration. Please see an extract from the log. These are fetching attempts - a partial output of "grep fetching hadoop.log".
2010-07-01 20:14:34,397 INFO fetcher.Fetcher - fetching http://www.atnf.csiro.au/vlbi/wiki/index.php?n=Pmcal.Pmcal?month=5&day=1&year=2002
2010-07-01 20:15:08,188 INFO fetcher.Fetcher - fetching http://www.atnf.csiro.au/vlbi/wiki/index.php?n=Pmcal.Pmcal?month=4&day=1&year=2002
2010-07-01 20:15:43,830 INFO fetcher.Fetcher - fetching http://www.atnf.csiro.au/vlbi/wiki/index.php?n=Pmcal.Pmcal?month=3&day=1&year=2002
2010-07-01 20:16:17,529 INFO fetcher.Fetcher - fetching http://www.atnf.csiro.au/vlbi/wiki/index.php?n=Pmcal.Pmcal?month=2&day=1&year=2002
2010-07-01 20:16:50,117 INFO fetcher.Fetcher - fetching http://www.atnf.csiro.au/vlbi/wiki/index.php?n=Pmcal.Pmcal?month=1&day=1&year=2002
2010-07-01 20:17:26,211 INFO fetcher.Fetcher - fetching http://www.atnf.csiro.au/vlbi/wiki/index.php?n=Pmcal.Pmcal?month=12&day=1&year=2001
The time gaps are explained by different iterations.
This would not trouble me, but the crawling was finished because the depth was reached. There are still 1000+ of URLs with status db_unfetched and score 30 left in the crwaldb. They should have been scheduled for fetching. Most fetched URLs have this score.
>
>I will try 0, thanks. The interesting thing is, I don't recall this
>happening with Nutch 1.0. I used it with the same script. The other
>problem that I mentioned, scheduling 1 new URL per iteration, is also
>new. I definitely did not have it with Nutch 1.0.
>
>
>Regards,
>
>Arkadi
>
>>
>>Arkadi.Kosmynin@csiro.au schrieb:
>>> Hi,
>>>
>>> I am trying to use Nutch 1.1 to build a complete index of our
>>corporate web sites. I am using a script based on this one:
>>>
>>> http://wiki.apache.org/nutch/Crawl
>>>
>>> The text of my script is included below. I set the crawling depth to
>>100 to make sure that everything is indexed, expecting that the process
>>will stop after about 20 iterations. The problem is that the Generator
>>keeps re-scheduling fetching of failed URLs. The process stopped
>because
>>the max number of iterations (the depth) was reached. After a few
>>iterations, only failing URLs were being repeatedly fetched. I checked
>>the log for one of them. It failed with code 403 and was re-scheduled
>>for fetching 94 times. This does not seem right.
>>>
>>> Another problem that I've noticed is that sometimes the Generators
>>schedules fetching of just one URL per iteration. This happened once
>and
>>I did not try to repeat this effect, but this does not seem right
>>either.
>>>
>>> Here is my script text:
>>>
>>> -----------------------------------------------
>>> #!/bin/bash
>>>
>>> depth=100
>>> threads=10
>>> adddays=5
>>> topN=50000
>>>
>>> NUTCH_HOME=/data/HORUS_1/nutch1.1
>>> NUTCH_HEAPSIZE=8000
>>> nutch=$NUTCH_HOME
>>> JAVA_HOME=/usr/lib/jvm/java-6-sun
>>> export NUTCH_HOME
>>> export JAVA_HOME
>>> export NUTCH_HEAPSIZE
>>>
>>> steps=6
>>> echo "----- Inject (Step 1 of $steps) -----"
>>> $nutch/bin/nutch inject $nutch/crawl/crawldb $nutch/crawl/seed
>>>
>>> echo "----- Generate, Fetch, Parse, Update (Step 2 of $steps) -----"
>>> for((i=0; i < $depth; i++))
>>> do
>>> echo "--- Beginning crawl at depth `expr $i + 1` of $depth ---"
>>> $nutch/bin/nutch generate $nutch/crawl/crawldb
>$nutch/crawl/segments
>>$topN -adddays $adddays
>>> if [ $? -ne 0 ]
>>> then
>>> echo "Stopping at depth $depth. No more URLs to fetch."
>>> break
>>> fi
>>> segment=`ls -d $nutch/crawl/segments/* | tail -1`
>>>
>>> $nutch/bin/nutch fetch $segment -threads $threads
>>> if [ $? -ne 0 ]
>>> then
>>> echo "fetch $segment at depth $depth failed. Deleting it."
>>> rm -rf $segment
>>> continue
>>> fi
>>>
>>> # echo "--- Parsing Segment $segment ---"
>>> # $nutch/bin/nutch parse $segment
>>>
>>> $nutch/bin/nutch updatedb $nutch/crawl/crawldb $segment
>>> done
>>>
>>> echo "----- Invert Links (Step 3 of $steps) -----"
>>> $nutch/bin/nutch invertlinks $nutch/crawl/linkdb
>>$nutch/crawl/segments/*
>>>
>>> echo "----- Index (Step 4 of $steps) -----"
>>> $nutch/bin/nutch index $nutch/crawl/preIndex $nutch/crawl/crawldb
>>$nutch/crawl/linkdb $nutch/crawl/segments/*
>>>
>>> echo "----- Dedup (Step 5 of $steps) -----"
>>> $nutch/bin/nutch dedup $nutch/crawl/preIndex
>>>
>>> echo "----- Merge Indexes (Step 6 of $steps) -----"
>>> $nutch/bin/nutch merge $nutch/crawl/index $nutch/crawl/preIndex
>>>
>>> # in nutch-site, hadoop.tmp.dir points to crawl/tmp
>>> rm -rf $nutch/crawl/tmp/*
>>> -----------------------------------------------
>>>
>>> Is anyone experiencing same problems? Is there anything wrong in what
>>I am doing?
>>>
>>> Regards,
>>>
>>> Arkadi
>>>
>>>
>>>
>>>
>>>
>>>
>>>
RE: Generator problems in Nutch 1.1
Posted by Ar...@csiro.au.
Hi Reinhard,
>-----Original Message-----
>From: reinhard schwab [mailto:reinhard.schwab@aon.at]
>Sent: Thursday, July 01, 2010 2:10 PM
>To: user@nutch.apache.org
>Subject: Re: Generator problems in Nutch 1.1
>
>could you dump the entry of this url in crawl db with
>
>bin/nutch readdb crawl/crawldb -url <url>
Here it is:
URL: http://www.atnf.csiro.au/people/bkoribal/obs/C848.html
Version: 7
Status: 3 (db_gone)
Fetch time: Thu Jul 01 20:51:33 GMT+10:00 2010
Modified time: Thu Jan 01 10:00:00 GMT+10:00 1970
Retries since fetch: 94
Retry interval: 2592000 seconds (30 days)
Score: 1.0
Signature: null
Metadata: _pst_: exception(16), lastModified=0: Http code=403, url=http://www.atnf.csiro.au/people/bkoribal/obs/C848.html
Please note that this is not the only URL which was repeatedly re-fetched. There were a number of them, about 150.
>it also depends on your configuration.
>what is the value for
>db.fetch.interval.default and for
>db.fetch.schedule.class?
I did not change these parameters in nutch-default.xml:
<property>
<name>db.default.fetch.interval</name>
<value>30</value>
<description>(DEPRECATED) The default number of days between re-fetches of a page.
</description>
</property>
<property>
<name>db.fetch.interval.default</name>
<value>2592000</value>
<description>The default number of seconds between re-fetches of a page (30 days).
</description>
</property>
<property>
<name>db.fetch.interval.max</name>
<value>7776000</value>
<description>The maximum number of seconds between re-fetches of a page
(90 days). After this period every page in the db will be re-tried, no
matter what is its status.
</description>
</property>
<property>
<name>db.fetch.schedule.class</name>
<value>org.apache.nutch.crawl.DefaultFetchSchedule</value>
<description>The implementation of fetch schedule. DefaultFetchSchedule simply
adds the original fetchInterval to the last fetch time, regardless of
page changes.</description>
</property>
>
>a guess. i have not done crawling with nutch for a while, so im not
>sure.
>i guess it has CrawlDatum.STATUS_FETCH_RETRY when crawling and it uses
>
> schedule.setPageRetrySchedule((Text)key, result, prevFetchTime,
> prevModifiedTime, fetch.getFetchTime());
>
>in AbstractFetchSchedule you can read
>
> /**
> * This method adjusts the fetch schedule if fetching needs to be
> * re-tried due to transient errors. The default implementation
> * sets the next fetch time 1 day in the future and increases
> * the retry counter.
> * @param url URL of the page
> * @param datum page information
> * @param prevFetchTime previous fetch time
> * @param prevModifiedTime previous modified time
> * @param fetchTime current fetch time
> * @return adjusted page information, including all original
>information.
> * NOTE: this may be a different instance than {@param datum}, but
> * implementations should make sure that it contains at least all
> * information from {@param datum}.
> */
> public CrawlDatum setPageRetrySchedule(Text url, CrawlDatum datum,
> long prevFetchTime, long prevModifiedTime, long fetchTime) {
> datum.setFetchTime(fetchTime + (long)SECONDS_PER_DAY*1000);
> datum.setRetriesSinceFetch(datum.getRetriesSinceFetch() + 1);
> return datum;
> }
>
>you have set adddays to 5 days.
>in this case it will be refetched and refetched again.
>try adddays with value 0.
>or change the code in AbstractFetchSchedule
>
>datum.setFetchTime(fetchTime + (long)SECONDS_PER_DAY*1000);
I will try 0, thanks. The interesting thing is, I don't recall this happening with Nutch 1.0. I used it with the same script. The other problem that I mentioned, scheduling 1 new URL per iteration, is also new. I definitely did not have it with Nutch 1.0.
Regards,
Arkadi
>
>Arkadi.Kosmynin@csiro.au schrieb:
>> Hi,
>>
>> I am trying to use Nutch 1.1 to build a complete index of our
>corporate web sites. I am using a script based on this one:
>>
>> http://wiki.apache.org/nutch/Crawl
>>
>> The text of my script is included below. I set the crawling depth to
>100 to make sure that everything is indexed, expecting that the process
>will stop after about 20 iterations. The problem is that the Generator
>keeps re-scheduling fetching of failed URLs. The process stopped because
>the max number of iterations (the depth) was reached. After a few
>iterations, only failing URLs were being repeatedly fetched. I checked
>the log for one of them. It failed with code 403 and was re-scheduled
>for fetching 94 times. This does not seem right.
>>
>> Another problem that I've noticed is that sometimes the Generators
>schedules fetching of just one URL per iteration. This happened once and
>I did not try to repeat this effect, but this does not seem right
>either.
>>
>> Here is my script text:
>>
>> -----------------------------------------------
>> #!/bin/bash
>>
>> depth=100
>> threads=10
>> adddays=5
>> topN=50000
>>
>> NUTCH_HOME=/data/HORUS_1/nutch1.1
>> NUTCH_HEAPSIZE=8000
>> nutch=$NUTCH_HOME
>> JAVA_HOME=/usr/lib/jvm/java-6-sun
>> export NUTCH_HOME
>> export JAVA_HOME
>> export NUTCH_HEAPSIZE
>>
>> steps=6
>> echo "----- Inject (Step 1 of $steps) -----"
>> $nutch/bin/nutch inject $nutch/crawl/crawldb $nutch/crawl/seed
>>
>> echo "----- Generate, Fetch, Parse, Update (Step 2 of $steps) -----"
>> for((i=0; i < $depth; i++))
>> do
>> echo "--- Beginning crawl at depth `expr $i + 1` of $depth ---"
>> $nutch/bin/nutch generate $nutch/crawl/crawldb $nutch/crawl/segments
>$topN -adddays $adddays
>> if [ $? -ne 0 ]
>> then
>> echo "Stopping at depth $depth. No more URLs to fetch."
>> break
>> fi
>> segment=`ls -d $nutch/crawl/segments/* | tail -1`
>>
>> $nutch/bin/nutch fetch $segment -threads $threads
>> if [ $? -ne 0 ]
>> then
>> echo "fetch $segment at depth $depth failed. Deleting it."
>> rm -rf $segment
>> continue
>> fi
>>
>> # echo "--- Parsing Segment $segment ---"
>> # $nutch/bin/nutch parse $segment
>>
>> $nutch/bin/nutch updatedb $nutch/crawl/crawldb $segment
>> done
>>
>> echo "----- Invert Links (Step 3 of $steps) -----"
>> $nutch/bin/nutch invertlinks $nutch/crawl/linkdb
>$nutch/crawl/segments/*
>>
>> echo "----- Index (Step 4 of $steps) -----"
>> $nutch/bin/nutch index $nutch/crawl/preIndex $nutch/crawl/crawldb
>$nutch/crawl/linkdb $nutch/crawl/segments/*
>>
>> echo "----- Dedup (Step 5 of $steps) -----"
>> $nutch/bin/nutch dedup $nutch/crawl/preIndex
>>
>> echo "----- Merge Indexes (Step 6 of $steps) -----"
>> $nutch/bin/nutch merge $nutch/crawl/index $nutch/crawl/preIndex
>>
>> # in nutch-site, hadoop.tmp.dir points to crawl/tmp
>> rm -rf $nutch/crawl/tmp/*
>> -----------------------------------------------
>>
>> Is anyone experiencing same problems? Is there anything wrong in what
>I am doing?
>>
>> Regards,
>>
>> Arkadi
>>
>>
>>
>>
>>
>>
>>
Re: Generator problems in Nutch 1.1
Posted by reinhard schwab <re...@aon.at>.
could you dump the entry of this url in crawl db with
bin/nutch readdb crawl/crawldb -url <url>
it also depends on your configuration.
what is the value for
db.fetch.interval.default and for
db.fetch.schedule.class?
a guess. i have not done crawling with nutch for a while, so im not sure.
i guess it has CrawlDatum.STATUS_FETCH_RETRY when crawling and it uses
schedule.setPageRetrySchedule((Text)key, result, prevFetchTime,
prevModifiedTime, fetch.getFetchTime());
in AbstractFetchSchedule you can read
/**
* This method adjusts the fetch schedule if fetching needs to be
* re-tried due to transient errors. The default implementation
* sets the next fetch time 1 day in the future and increases
* the retry counter.
* @param url URL of the page
* @param datum page information
* @param prevFetchTime previous fetch time
* @param prevModifiedTime previous modified time
* @param fetchTime current fetch time
* @return adjusted page information, including all original information.
* NOTE: this may be a different instance than {@param datum}, but
* implementations should make sure that it contains at least all
* information from {@param datum}.
*/
public CrawlDatum setPageRetrySchedule(Text url, CrawlDatum datum,
long prevFetchTime, long prevModifiedTime, long fetchTime) {
datum.setFetchTime(fetchTime + (long)SECONDS_PER_DAY*1000);
datum.setRetriesSinceFetch(datum.getRetriesSinceFetch() + 1);
return datum;
}
you have set adddays to 5 days.
in this case it will be refetched and refetched again.
try adddays with value 0.
or change the code in AbstractFetchSchedule
datum.setFetchTime(fetchTime + (long)SECONDS_PER_DAY*1000);
Arkadi.Kosmynin@csiro.au schrieb:
> Hi,
>
> I am trying to use Nutch 1.1 to build a complete index of our corporate web sites. I am using a script based on this one:
>
> http://wiki.apache.org/nutch/Crawl
>
> The text of my script is included below. I set the crawling depth to 100 to make sure that everything is indexed, expecting that the process will stop after about 20 iterations. The problem is that the Generator keeps re-scheduling fetching of failed URLs. The process stopped because the max number of iterations (the depth) was reached. After a few iterations, only failing URLs were being repeatedly fetched. I checked the log for one of them. It failed with code 403 and was re-scheduled for fetching 94 times. This does not seem right.
>
> Another problem that I've noticed is that sometimes the Generators schedules fetching of just one URL per iteration. This happened once and I did not try to repeat this effect, but this does not seem right either.
>
> Here is my script text:
>
> -----------------------------------------------
> #!/bin/bash
>
> depth=100
> threads=10
> adddays=5
> topN=50000
>
> NUTCH_HOME=/data/HORUS_1/nutch1.1
> NUTCH_HEAPSIZE=8000
> nutch=$NUTCH_HOME
> JAVA_HOME=/usr/lib/jvm/java-6-sun
> export NUTCH_HOME
> export JAVA_HOME
> export NUTCH_HEAPSIZE
>
> steps=6
> echo "----- Inject (Step 1 of $steps) -----"
> $nutch/bin/nutch inject $nutch/crawl/crawldb $nutch/crawl/seed
>
> echo "----- Generate, Fetch, Parse, Update (Step 2 of $steps) -----"
> for((i=0; i < $depth; i++))
> do
> echo "--- Beginning crawl at depth `expr $i + 1` of $depth ---"
> $nutch/bin/nutch generate $nutch/crawl/crawldb $nutch/crawl/segments $topN -adddays $adddays
> if [ $? -ne 0 ]
> then
> echo "Stopping at depth $depth. No more URLs to fetch."
> break
> fi
> segment=`ls -d $nutch/crawl/segments/* | tail -1`
>
> $nutch/bin/nutch fetch $segment -threads $threads
> if [ $? -ne 0 ]
> then
> echo "fetch $segment at depth $depth failed. Deleting it."
> rm -rf $segment
> continue
> fi
>
> # echo "--- Parsing Segment $segment ---"
> # $nutch/bin/nutch parse $segment
>
> $nutch/bin/nutch updatedb $nutch/crawl/crawldb $segment
> done
>
> echo "----- Invert Links (Step 3 of $steps) -----"
> $nutch/bin/nutch invertlinks $nutch/crawl/linkdb $nutch/crawl/segments/*
>
> echo "----- Index (Step 4 of $steps) -----"
> $nutch/bin/nutch index $nutch/crawl/preIndex $nutch/crawl/crawldb $nutch/crawl/linkdb $nutch/crawl/segments/*
>
> echo "----- Dedup (Step 5 of $steps) -----"
> $nutch/bin/nutch dedup $nutch/crawl/preIndex
>
> echo "----- Merge Indexes (Step 6 of $steps) -----"
> $nutch/bin/nutch merge $nutch/crawl/index $nutch/crawl/preIndex
>
> # in nutch-site, hadoop.tmp.dir points to crawl/tmp
> rm -rf $nutch/crawl/tmp/*
> -----------------------------------------------
>
> Is anyone experiencing same problems? Is there anything wrong in what I am doing?
>
> Regards,
>
> Arkadi
>
>
>
>
>
>
>