You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Kelvin Tan <ke...@relevanz.com> on 2005/09/01 17:53:51 UTC

To mapred or not

Seeing mapred is about to be folded into trunk, 3 questions:

1. Any benchmarks/estimates on when the scalability of map-reduce surpasses its overhead/complexity? e.g. with > 10 reduce workers..
2. Will there be an option of a plain vanilla single-box Nutch crawler vs a map-reduce version?
3. What are the options for users who don't want to jump onboard map-red? Will pre-mapred be actively maintained?

thanks..
k


Re: To mapred or not

Posted by Stefan Groschupf <sg...@media-style.com>.
> In some cases, though, focused crawling requirements may require  
> extra data to be stored, which is not useful for whole-web, for  
> example, storing a url's parent and seed url and its depth 
> (essential for crawl scopes).

Sounds like meta data for a page. :)
Some time ago I submit a patch to the issue tracking, we use this  
meta data here in a project to decide if the page should be crawled  
or not.. and to give meta data from a 'mother pager' to a child.

I still believe flexible page meta data would be a big help in many  
cases and I believe that to map reduce and 'merge-in' meta data as  
Doug suggest it, isn't that powerful, since a identically key for the  
page and the meta datum are required.
Just my 2 cents..
Greetings,
Stefan



Re: To mapred or not

Posted by Kelvin Tan <ke...@relevanz.com>.

On Thu, 01 Sep 2005 09:36:19 -0700, Doug Cutting wrote:
> It would be worth considering which features of your constrained
> crawler   could be cast as improvements to Nutch's existing tools
> (e.g., more seed url formats, more output formats, http 1.1, custom
> scopes, etc.) and which require a different control flow (online
> fetchlist building?).   In some cases (e.g., fetch prioritization)
> perhaps a new Plugin should be added to Nutch.

In most cases, it is merely a generalization of what Nutch already has, introducing interfaces where appropriate to make it easier to modify behavior. I've come to see the importance of making scoring pluggable (essential for focused crawling), and also both host-based (current nutch-84) and score-based (current nutch) fetch prioritization. 

There are some departures which need to be reconciled, in particular the role of fetchlists and the way they are built. However, I do not see any major incompatibilities between whole-web and focused crawling requirements. 

In some cases, though, focused crawling requirements may require extra data to be stored, which is not useful for whole-web, for example, storing a url's parent and seed url and its depth(essential for crawl scopes).

k


Re: To mapred or not

Posted by Doug Cutting <cu...@nutch.org>.
Kelvin Tan wrote:
> Seeing mapred is about to be folded into trunk, 3 questions:
> 
> 1. Any benchmarks/estimates on when the scalability of map-reduce surpasses its overhead/complexity? e.g. with > 10 reduce workers..

I think that with as few as two boxes it will outperform uniprocessor 
Nutch.  This will not be true for very small collections, since the 
overhead of starting JVMs can dominate those.  (MapReduce runs each task 
in a separate JVM, for robustness.)

> 2. Will there be an option of a plain vanilla single-box Nutch crawler vs a map-reduce version?

Yes, there already is.  By default MapReduce runs on a single box in a 
single JVM.  To run on multiple boxes in multiple JVMs one must alter 
the default configuration (to name the jobtracker server) and start the 
jobtracker daemon and one or more tasktracker daemons.  There are shell 
scripts to assist with the management of daemons.

> 3. What are the options for users who don't want to jump onboard map-red? Will pre-mapred be actively maintained?

The MapReduce versons of Nutch tools (inject, generate, fetch, etc.) are 
not a large amount of code.  One could easily build compatible 
non-MapReduce versions of these tools.  But then we'd be maintaining two 
versions, so we should avoid this as much as possible.  However, if some 
applications need a very different control flow, then that might warrant 
this.  For example, one might write a crawler that combines a number of 
these tools in a single process, e.g., using RDBMS to keep track of urls 
and links while crawling, updating the database in real-time as URLs are 
fetched.  Such an architecture would not be as scalable, but might excel 
in other ways.

It would be worth considering which features of your constrained crawler 
  could be cast as improvements to Nutch's existing tools (e.g., more 
seed url formats, more output formats, http 1.1, custom scopes, etc.) 
and which require a different control flow (online fetchlist building?). 
  In some cases (e.g., fetch prioritization) perhaps a new Plugin should 
be added to Nutch.

In the mapred branch the webdb has been decomposed into a crawldb and a 
linkdb.  The crawldb is much smaller and simpler than the former webdb, 
containing only an entry for each known URL.  This makes updates much 
faster while crawling.  The linkdb contains only the link graph, and 
needs only be updated prior to indexing, not with each step while 
crawling as before.

Doug