jobtracker is local

[Nick Duan <nd...@mcdonaldbradley.com>]

Hi!  I am trying to follow the tutorial (version 0.8) and run the nutch
generate command to create a fetch list.  The result as follows:

bin/nutch generate ../devel/crawl/crawldb ../devel/crawl/segments
Generator: Selecting best-scoring urls due for fetch.
Generator: starting
Generator: segment: ../devel/crawl/segments/20080220163725
Generator: filtering: true
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls by host, for politeness.
Generator: done.

What does "jobtracker is local" mean here?  Shouldn't be multiple
partitions generated?

Thanks!

ND

Re: jobtracker is local

[Andrzej Bialecki <ab...@getopt.org>]

Nick Duan wrote:

> What does "jobtracker is local" mean here?  Shouldn't be multiple
> partitions generated?

This means that you are running Nutch on top of Hadoop in a so called 
"local" mode (i.e. in a single JVM), so there is no point in 
distributing the fetchlist to many partitions, because in any case there 
will be just a single JVM working with all urls.


-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com