You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Andy Xue <an...@gmail.com> on 2012/06/11 10:59:27 UTC
Generator: 0 records selected for fetching, exiting ...
Hi all:
This is regarding an error I encountered when doing a distributed crawl :
"Generator: 0 records selected for fetching, exiting ..."
I understand that this is a well answered issue which normally is caused by
either seed list or url filters. (or when topN is smaller than the number
of reducers which is not in my case)
However, mine is a little bit different. It runs well in local mode, but
fails after the first level when running in distributed mode with several
computers.
After successfully crawled the first level (i.e., fetched all urls in the
seed list), the generator of the second level fails with the following log:
========================================================================
---- generating ----
(started at Mon Jun 11 18:33:57 2012)
12/06/11 18:34:00 INFO crawl.Generator: Generator: starting at 2012-06-11
18:34:00
12/06/11 18:34:00 INFO crawl.Generator: Generator: Selecting best-scoring
urls due for fetch.
12/06/11 18:34:00 INFO crawl.Generator: Generator: filtering: true
12/06/11 18:34:00 INFO crawl.Generator: Generator: normalizing: true
12/06/11 18:34:00 INFO crawl.Generator: Generator: topN: 5000000
12/06/11 18:34:01 INFO mapred.FileInputFormat: Total input paths to process
: 6
12/06/11 18:34:04 INFO mapred.JobClient: Running job: job_201206111824_0007
12/06/11 18:34:05 INFO mapred.JobClient: map 0% reduce 0%
12/06/11 18:34:20 INFO mapred.JobClient: map 18% reduce 0%
12/06/11 18:34:26 INFO mapred.JobClient: map 31% reduce 0%
12/06/11 18:34:29 INFO mapred.JobClient: map 37% reduce 2%
12/06/11 18:34:32 INFO mapred.JobClient: map 50% reduce 7%
12/06/11 18:34:35 INFO mapred.JobClient: map 56% reduce 9%
12/06/11 18:34:38 INFO mapred.JobClient: map 68% reduce 11%
12/06/11 18:34:41 INFO mapred.JobClient: map 75% reduce 14%
12/06/11 18:34:44 INFO mapred.JobClient: map 87% reduce 19%
12/06/11 18:34:47 INFO mapred.JobClient: map 93% reduce 23%
12/06/11 18:34:50 INFO mapred.JobClient: map 100% reduce 25%
12/06/11 18:34:53 INFO mapred.JobClient: map 100% reduce 27%
12/06/11 18:34:56 INFO mapred.JobClient: map 100% reduce 65%
12/06/11 18:34:59 INFO mapred.JobClient: map 100% reduce 100%
12/06/11 18:35:04 INFO mapred.JobClient: Job complete: job_201206111824_0007
12/06/11 18:35:04 INFO mapred.JobClient: Counters: 30
12/06/11 18:35:04 INFO mapred.JobClient: Job Counters
12/06/11 18:35:04 INFO mapred.JobClient: Launched reduce tasks=6
12/06/11 18:35:04 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=154922
12/06/11 18:35:04 INFO mapred.JobClient: Total time spent by all
reduces waiting after reserving slots (ms)=0
12/06/11 18:35:04 INFO mapred.JobClient: Total time spent by all maps
waiting after reserving slots (ms)=0
12/06/11 18:35:04 INFO mapred.JobClient: Rack-local map tasks=2
12/06/11 18:35:04 INFO mapred.JobClient: Launched map tasks=32
12/06/11 18:35:04 INFO mapred.JobClient: Data-local map tasks=30
12/06/11 18:35:04 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=204710
12/06/11 18:35:04 INFO mapred.JobClient: File Input Format Counters
12/06/11 18:35:04 INFO mapred.JobClient: Bytes Read=643181
12/06/11 18:35:04 INFO mapred.JobClient: File Output Format Counters
12/06/11 18:35:04 INFO mapred.JobClient: Bytes Written=0
12/06/11 18:35:04 INFO mapred.JobClient: FileSystemCounters
12/06/11 18:35:04 INFO mapred.JobClient: FILE_BYTES_READ=36
12/06/11 18:35:04 INFO mapred.JobClient: HDFS_BYTES_READ=651494
12/06/11 18:35:04 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1311646
12/06/11 18:35:04 INFO mapred.JobClient: Map-Reduce Framework
12/06/11 18:35:04 INFO mapred.JobClient: Map output materialized
bytes=1152
12/06/11 18:35:04 INFO mapred.JobClient: Map input records=4259
12/06/11 18:35:04 INFO mapred.JobClient: Reduce shuffle bytes=1128
12/06/11 18:35:04 INFO mapred.JobClient: Spilled Records=0
12/06/11 18:35:04 INFO mapred.JobClient: Map output bytes=0
12/06/11 18:35:04 INFO mapred.JobClient: Total committed heap usage
(bytes)=5583339520
12/06/11 18:35:04 INFO mapred.JobClient: CPU time spent (ms)=55840
12/06/11 18:35:04 INFO mapred.JobClient: Map input bytes=618447
12/06/11 18:35:04 INFO mapred.JobClient: SPLIT_RAW_BYTES=3712
12/06/11 18:35:04 INFO mapred.JobClient: Combine input records=0
12/06/11 18:35:04 INFO mapred.JobClient: Reduce input records=0
12/06/11 18:35:04 INFO mapred.JobClient: Reduce input groups=0
12/06/11 18:35:04 INFO mapred.JobClient: Combine output records=0
12/06/11 18:35:04 INFO mapred.JobClient: Physical memory (bytes)
snapshot=7267901440
12/06/11 18:35:04 INFO mapred.JobClient: Reduce output records=0
12/06/11 18:35:04 INFO mapred.JobClient: Virtual memory (bytes)
snapshot=109405061120
12/06/11 18:35:04 INFO mapred.JobClient: Map output records=0
12/06/11 18:35:04 WARN crawl.Generator: Generator: 0 records selected for
fetching, exiting ...
========================================================================
Some diagnosis that I have done:
* Did a local crawl and it successfully finished two levels;
* Experimented distributed crawling with different (and different number
of) urls in the seed list;
* Used "nutch org.apache.nutch.net.URLFilterChecker -allCombined" to test
some urls in the seed list and the un-fetched web pages in the crawldb (I
did check the content of crawldb and it does contain plenty of positively
scored "db_unfetched" web pages which are linked from the urls in the seed
list.)
One entry in the crawldb looks like this. "generate.min.score" is set to 0.
======================================
http://www.ci.redmond.wa.us/ Version: 7
Status: 1 (db_unfetched)
Fetch time: Tue Jun 12 02:59:27 EST 2012
Modified time: Thu Jan 01 10:00:00 EST 1970
Retries since fetch: 0
Retry interval: 2592000 seconds (30 days)
Score: 0.05
Signature: null
Metadata:
======================================
These computers used in the distributed crawl were functioning fine
previously when I finished plenty of crawls in the past month. Maybe it is
because I modified some of the code (my customised filters) and some
properties in the configuration files.
Any suggestions? It is driving me crazy now...
Thanks for your help. Even a wild thought or guess would be helpful for me
to test out.
Appreciate your time.
Andy
Re: Generator: 0 records selected for fetching, exiting ...
Posted by Andy Xue <an...@gmail.com>.
Hi:
It is caused by different operating system clocks on these slaves
computers. Never thought that this could affect the crawling process. This
is enlightening -__-..
Appreciate it.
Regards
Andy
On 11 June 2012 19:22, Markus Jelsma <ma...@openindex.io> wrote:
> Hi
>
> This CrawlDatum's FetchTime is tomorrow in EST
> Fetch time: Tue Jun 12 02:59:27 EST 2012
>
>
> -----Original message-----
> > From:Andy Xue <an...@gmail.com>
> > Sent: Mon 11-Jun-2012 11:00
> > To: user@nutch.apache.org
> > Subject: Generator: 0 records selected for fetching, exiting ...
> >
> > Hi all:
> >
> > This is regarding an error I encountered when doing a distributed crawl :
> > "Generator: 0 records selected for fetching, exiting ..."
> > I understand that this is a well answered issue which normally is caused
> by
> > either seed list or url filters. (or when topN is smaller than the number
> > of reducers which is not in my case)
> > However, mine is a little bit different. It runs well in local mode, but
> > fails after the first level when running in distributed mode with several
> > computers.
> >
> > After successfully crawled the first level (i.e., fetched all urls in the
> > seed list), the generator of the second level fails with the following
> log:
> > ========================================================================
> > ---- generating ----
> > (started at Mon Jun 11 18:33:57 2012)
> > 12/06/11 18:34:00 INFO crawl.Generator: Generator: starting at 2012-06-11
> > 18:34:00
> > 12/06/11 18:34:00 INFO crawl.Generator: Generator: Selecting best-scoring
> > urls due for fetch.
> > 12/06/11 18:34:00 INFO crawl.Generator: Generator: filtering: true
> > 12/06/11 18:34:00 INFO crawl.Generator: Generator: normalizing: true
> > 12/06/11 18:34:00 INFO crawl.Generator: Generator: topN: 5000000
> > 12/06/11 18:34:01 INFO mapred.FileInputFormat: Total input paths to
> process
> > : 6
> > 12/06/11 18:34:04 INFO mapred.JobClient: Running job:
> job_201206111824_0007
> > 12/06/11 18:34:05 INFO mapred.JobClient: map 0% reduce 0%
> > 12/06/11 18:34:20 INFO mapred.JobClient: map 18% reduce 0%
> > 12/06/11 18:34:26 INFO mapred.JobClient: map 31% reduce 0%
> > 12/06/11 18:34:29 INFO mapred.JobClient: map 37% reduce 2%
> > 12/06/11 18:34:32 INFO mapred.JobClient: map 50% reduce 7%
> > 12/06/11 18:34:35 INFO mapred.JobClient: map 56% reduce 9%
> > 12/06/11 18:34:38 INFO mapred.JobClient: map 68% reduce 11%
> > 12/06/11 18:34:41 INFO mapred.JobClient: map 75% reduce 14%
> > 12/06/11 18:34:44 INFO mapred.JobClient: map 87% reduce 19%
> > 12/06/11 18:34:47 INFO mapred.JobClient: map 93% reduce 23%
> > 12/06/11 18:34:50 INFO mapred.JobClient: map 100% reduce 25%
> > 12/06/11 18:34:53 INFO mapred.JobClient: map 100% reduce 27%
> > 12/06/11 18:34:56 INFO mapred.JobClient: map 100% reduce 65%
> > 12/06/11 18:34:59 INFO mapred.JobClient: map 100% reduce 100%
> > 12/06/11 18:35:04 INFO mapred.JobClient: Job complete:
> job_201206111824_0007
> > 12/06/11 18:35:04 INFO mapred.JobClient: Counters: 30
> > 12/06/11 18:35:04 INFO mapred.JobClient: Job Counters
> > 12/06/11 18:35:04 INFO mapred.JobClient: Launched reduce tasks=6
> > 12/06/11 18:35:04 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=154922
> > 12/06/11 18:35:04 INFO mapred.JobClient: Total time spent by all
> > reduces waiting after reserving slots (ms)=0
> > 12/06/11 18:35:04 INFO mapred.JobClient: Total time spent by all maps
> > waiting after reserving slots (ms)=0
> > 12/06/11 18:35:04 INFO mapred.JobClient: Rack-local map tasks=2
> > 12/06/11 18:35:04 INFO mapred.JobClient: Launched map tasks=32
> > 12/06/11 18:35:04 INFO mapred.JobClient: Data-local map tasks=30
> > 12/06/11 18:35:04 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=204710
> > 12/06/11 18:35:04 INFO mapred.JobClient: File Input Format Counters
> > 12/06/11 18:35:04 INFO mapred.JobClient: Bytes Read=643181
> > 12/06/11 18:35:04 INFO mapred.JobClient: File Output Format Counters
> > 12/06/11 18:35:04 INFO mapred.JobClient: Bytes Written=0
> > 12/06/11 18:35:04 INFO mapred.JobClient: FileSystemCounters
> > 12/06/11 18:35:04 INFO mapred.JobClient: FILE_BYTES_READ=36
> > 12/06/11 18:35:04 INFO mapred.JobClient: HDFS_BYTES_READ=651494
> > 12/06/11 18:35:04 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1311646
> > 12/06/11 18:35:04 INFO mapred.JobClient: Map-Reduce Framework
> > 12/06/11 18:35:04 INFO mapred.JobClient: Map output materialized
> > bytes=1152
> > 12/06/11 18:35:04 INFO mapred.JobClient: Map input records=4259
> > 12/06/11 18:35:04 INFO mapred.JobClient: Reduce shuffle bytes=1128
> > 12/06/11 18:35:04 INFO mapred.JobClient: Spilled Records=0
> > 12/06/11 18:35:04 INFO mapred.JobClient: Map output bytes=0
> > 12/06/11 18:35:04 INFO mapred.JobClient: Total committed heap usage
> > (bytes)=5583339520
> > 12/06/11 18:35:04 INFO mapred.JobClient: CPU time spent (ms)=55840
> > 12/06/11 18:35:04 INFO mapred.JobClient: Map input bytes=618447
> > 12/06/11 18:35:04 INFO mapred.JobClient: SPLIT_RAW_BYTES=3712
> > 12/06/11 18:35:04 INFO mapred.JobClient: Combine input records=0
> > 12/06/11 18:35:04 INFO mapred.JobClient: Reduce input records=0
> > 12/06/11 18:35:04 INFO mapred.JobClient: Reduce input groups=0
> > 12/06/11 18:35:04 INFO mapred.JobClient: Combine output records=0
> > 12/06/11 18:35:04 INFO mapred.JobClient: Physical memory (bytes)
> > snapshot=7267901440
> > 12/06/11 18:35:04 INFO mapred.JobClient: Reduce output records=0
> > 12/06/11 18:35:04 INFO mapred.JobClient: Virtual memory (bytes)
> > snapshot=109405061120
> > 12/06/11 18:35:04 INFO mapred.JobClient: Map output records=0
> > 12/06/11 18:35:04 WARN crawl.Generator: Generator: 0 records selected for
> > fetching, exiting ...
> > ========================================================================
> >
> > Some diagnosis that I have done:
> > * Did a local crawl and it successfully finished two levels;
> > * Experimented distributed crawling with different (and different number
> > of) urls in the seed list;
> > * Used "nutch org.apache.nutch.net.URLFilterChecker -allCombined" to test
> > some urls in the seed list and the un-fetched web pages in the crawldb (I
> > did check the content of crawldb and it does contain plenty of positively
> > scored "db_unfetched" web pages which are linked from the urls in the
> seed
> > list.)
> >
> > One entry in the crawldb looks like this. "generate.min.score" is set to
> 0.
> > ======================================
> > http://www.ci.redmond.wa.us/ Version: 7
> > Status: 1 (db_unfetched)
> > Fetch time: Tue Jun 12 02:59:27 EST 2012
> > Modified time: Thu Jan 01 10:00:00 EST 1970
> > Retries since fetch: 0
> > Retry interval: 2592000 seconds (30 days)
> > Score: 0.05
> > Signature: null
> > Metadata:
> > ======================================
> >
> >
> > These computers used in the distributed crawl were functioning fine
> > previously when I finished plenty of crawls in the past month. Maybe it
> is
> > because I modified some of the code (my customised filters) and some
> > properties in the configuration files.
> >
> > Any suggestions? It is driving me crazy now...
> > Thanks for your help. Even a wild thought or guess would be helpful for
> me
> > to test out.
> >
> > Appreciate your time.
> >
> > Andy
> >
>
RE: Generator: 0 records selected for fetching, exiting ...
Posted by Markus Jelsma <ma...@openindex.io>.
Hi
This CrawlDatum's FetchTime is tomorrow in EST
Fetch time: Tue Jun 12 02:59:27 EST 2012
-----Original message-----
> From:Andy Xue <an...@gmail.com>
> Sent: Mon 11-Jun-2012 11:00
> To: user@nutch.apache.org
> Subject: Generator: 0 records selected for fetching, exiting ...
>
> Hi all:
>
> This is regarding an error I encountered when doing a distributed crawl :
> "Generator: 0 records selected for fetching, exiting ..."
> I understand that this is a well answered issue which normally is caused by
> either seed list or url filters. (or when topN is smaller than the number
> of reducers which is not in my case)
> However, mine is a little bit different. It runs well in local mode, but
> fails after the first level when running in distributed mode with several
> computers.
>
> After successfully crawled the first level (i.e., fetched all urls in the
> seed list), the generator of the second level fails with the following log:
> ========================================================================
> ---- generating ----
> (started at Mon Jun 11 18:33:57 2012)
> 12/06/11 18:34:00 INFO crawl.Generator: Generator: starting at 2012-06-11
> 18:34:00
> 12/06/11 18:34:00 INFO crawl.Generator: Generator: Selecting best-scoring
> urls due for fetch.
> 12/06/11 18:34:00 INFO crawl.Generator: Generator: filtering: true
> 12/06/11 18:34:00 INFO crawl.Generator: Generator: normalizing: true
> 12/06/11 18:34:00 INFO crawl.Generator: Generator: topN: 5000000
> 12/06/11 18:34:01 INFO mapred.FileInputFormat: Total input paths to process
> : 6
> 12/06/11 18:34:04 INFO mapred.JobClient: Running job: job_201206111824_0007
> 12/06/11 18:34:05 INFO mapred.JobClient: map 0% reduce 0%
> 12/06/11 18:34:20 INFO mapred.JobClient: map 18% reduce 0%
> 12/06/11 18:34:26 INFO mapred.JobClient: map 31% reduce 0%
> 12/06/11 18:34:29 INFO mapred.JobClient: map 37% reduce 2%
> 12/06/11 18:34:32 INFO mapred.JobClient: map 50% reduce 7%
> 12/06/11 18:34:35 INFO mapred.JobClient: map 56% reduce 9%
> 12/06/11 18:34:38 INFO mapred.JobClient: map 68% reduce 11%
> 12/06/11 18:34:41 INFO mapred.JobClient: map 75% reduce 14%
> 12/06/11 18:34:44 INFO mapred.JobClient: map 87% reduce 19%
> 12/06/11 18:34:47 INFO mapred.JobClient: map 93% reduce 23%
> 12/06/11 18:34:50 INFO mapred.JobClient: map 100% reduce 25%
> 12/06/11 18:34:53 INFO mapred.JobClient: map 100% reduce 27%
> 12/06/11 18:34:56 INFO mapred.JobClient: map 100% reduce 65%
> 12/06/11 18:34:59 INFO mapred.JobClient: map 100% reduce 100%
> 12/06/11 18:35:04 INFO mapred.JobClient: Job complete: job_201206111824_0007
> 12/06/11 18:35:04 INFO mapred.JobClient: Counters: 30
> 12/06/11 18:35:04 INFO mapred.JobClient: Job Counters
> 12/06/11 18:35:04 INFO mapred.JobClient: Launched reduce tasks=6
> 12/06/11 18:35:04 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=154922
> 12/06/11 18:35:04 INFO mapred.JobClient: Total time spent by all
> reduces waiting after reserving slots (ms)=0
> 12/06/11 18:35:04 INFO mapred.JobClient: Total time spent by all maps
> waiting after reserving slots (ms)=0
> 12/06/11 18:35:04 INFO mapred.JobClient: Rack-local map tasks=2
> 12/06/11 18:35:04 INFO mapred.JobClient: Launched map tasks=32
> 12/06/11 18:35:04 INFO mapred.JobClient: Data-local map tasks=30
> 12/06/11 18:35:04 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=204710
> 12/06/11 18:35:04 INFO mapred.JobClient: File Input Format Counters
> 12/06/11 18:35:04 INFO mapred.JobClient: Bytes Read=643181
> 12/06/11 18:35:04 INFO mapred.JobClient: File Output Format Counters
> 12/06/11 18:35:04 INFO mapred.JobClient: Bytes Written=0
> 12/06/11 18:35:04 INFO mapred.JobClient: FileSystemCounters
> 12/06/11 18:35:04 INFO mapred.JobClient: FILE_BYTES_READ=36
> 12/06/11 18:35:04 INFO mapred.JobClient: HDFS_BYTES_READ=651494
> 12/06/11 18:35:04 INFO mapred.JobClient: FILE_BYTES_WRITTEN=1311646
> 12/06/11 18:35:04 INFO mapred.JobClient: Map-Reduce Framework
> 12/06/11 18:35:04 INFO mapred.JobClient: Map output materialized
> bytes=1152
> 12/06/11 18:35:04 INFO mapred.JobClient: Map input records=4259
> 12/06/11 18:35:04 INFO mapred.JobClient: Reduce shuffle bytes=1128
> 12/06/11 18:35:04 INFO mapred.JobClient: Spilled Records=0
> 12/06/11 18:35:04 INFO mapred.JobClient: Map output bytes=0
> 12/06/11 18:35:04 INFO mapred.JobClient: Total committed heap usage
> (bytes)=5583339520
> 12/06/11 18:35:04 INFO mapred.JobClient: CPU time spent (ms)=55840
> 12/06/11 18:35:04 INFO mapred.JobClient: Map input bytes=618447
> 12/06/11 18:35:04 INFO mapred.JobClient: SPLIT_RAW_BYTES=3712
> 12/06/11 18:35:04 INFO mapred.JobClient: Combine input records=0
> 12/06/11 18:35:04 INFO mapred.JobClient: Reduce input records=0
> 12/06/11 18:35:04 INFO mapred.JobClient: Reduce input groups=0
> 12/06/11 18:35:04 INFO mapred.JobClient: Combine output records=0
> 12/06/11 18:35:04 INFO mapred.JobClient: Physical memory (bytes)
> snapshot=7267901440
> 12/06/11 18:35:04 INFO mapred.JobClient: Reduce output records=0
> 12/06/11 18:35:04 INFO mapred.JobClient: Virtual memory (bytes)
> snapshot=109405061120
> 12/06/11 18:35:04 INFO mapred.JobClient: Map output records=0
> 12/06/11 18:35:04 WARN crawl.Generator: Generator: 0 records selected for
> fetching, exiting ...
> ========================================================================
>
> Some diagnosis that I have done:
> * Did a local crawl and it successfully finished two levels;
> * Experimented distributed crawling with different (and different number
> of) urls in the seed list;
> * Used "nutch org.apache.nutch.net.URLFilterChecker -allCombined" to test
> some urls in the seed list and the un-fetched web pages in the crawldb (I
> did check the content of crawldb and it does contain plenty of positively
> scored "db_unfetched" web pages which are linked from the urls in the seed
> list.)
>
> One entry in the crawldb looks like this. "generate.min.score" is set to 0.
> ======================================
> http://www.ci.redmond.wa.us/ Version: 7
> Status: 1 (db_unfetched)
> Fetch time: Tue Jun 12 02:59:27 EST 2012
> Modified time: Thu Jan 01 10:00:00 EST 1970
> Retries since fetch: 0
> Retry interval: 2592000 seconds (30 days)
> Score: 0.05
> Signature: null
> Metadata:
> ======================================
>
>
> These computers used in the distributed crawl were functioning fine
> previously when I finished plenty of crawls in the past month. Maybe it is
> because I modified some of the code (my customised filters) and some
> properties in the configuration files.
>
> Any suggestions? It is driving me crazy now...
> Thanks for your help. Even a wild thought or guess would be helpful for me
> to test out.
>
> Appreciate your time.
>
> Andy
>