You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Kelvin Tan <ke...@relevanz.com> on 2005/09/01 17:53:51 UTC
To mapred or not
Seeing mapred is about to be folded into trunk, 3 questions:
1. Any benchmarks/estimates on when the scalability of map-reduce surpasses its overhead/complexity? e.g. with > 10 reduce workers..
2. Will there be an option of a plain vanilla single-box Nutch crawler vs a map-reduce version?
3. What are the options for users who don't want to jump onboard map-red? Will pre-mapred be actively maintained?
thanks..
k
Re: To mapred or not
Posted by Stefan Groschupf <sg...@media-style.com>.
> In some cases, though, focused crawling requirements may require
> extra data to be stored, which is not useful for whole-web, for
> example, storing a url's parent and seed url and its depth
> (essential for crawl scopes).
Sounds like meta data for a page. :)
Some time ago I submit a patch to the issue tracking, we use this
meta data here in a project to decide if the page should be crawled
or not.. and to give meta data from a 'mother pager' to a child.
I still believe flexible page meta data would be a big help in many
cases and I believe that to map reduce and 'merge-in' meta data as
Doug suggest it, isn't that powerful, since a identically key for the
page and the meta datum are required.
Just my 2 cents..
Greetings,
Stefan
Re: To mapred or not
Posted by Kelvin Tan <ke...@relevanz.com>.
On Thu, 01 Sep 2005 09:36:19 -0700, Doug Cutting wrote:
> It would be worth considering which features of your constrained
> crawler could be cast as improvements to Nutch's existing tools
> (e.g., more seed url formats, more output formats, http 1.1, custom
> scopes, etc.) and which require a different control flow (online
> fetchlist building?). In some cases (e.g., fetch prioritization)
> perhaps a new Plugin should be added to Nutch.
In most cases, it is merely a generalization of what Nutch already has, introducing interfaces where appropriate to make it easier to modify behavior. I've come to see the importance of making scoring pluggable (essential for focused crawling), and also both host-based (current nutch-84) and score-based (current nutch) fetch prioritization.
There are some departures which need to be reconciled, in particular the role of fetchlists and the way they are built. However, I do not see any major incompatibilities between whole-web and focused crawling requirements.
In some cases, though, focused crawling requirements may require extra data to be stored, which is not useful for whole-web, for example, storing a url's parent and seed url and its depth(essential for crawl scopes).
k
Re: To mapred or not
Posted by Doug Cutting <cu...@nutch.org>.
Kelvin Tan wrote:
> Seeing mapred is about to be folded into trunk, 3 questions:
>
> 1. Any benchmarks/estimates on when the scalability of map-reduce surpasses its overhead/complexity? e.g. with > 10 reduce workers..
I think that with as few as two boxes it will outperform uniprocessor
Nutch. This will not be true for very small collections, since the
overhead of starting JVMs can dominate those. (MapReduce runs each task
in a separate JVM, for robustness.)
> 2. Will there be an option of a plain vanilla single-box Nutch crawler vs a map-reduce version?
Yes, there already is. By default MapReduce runs on a single box in a
single JVM. To run on multiple boxes in multiple JVMs one must alter
the default configuration (to name the jobtracker server) and start the
jobtracker daemon and one or more tasktracker daemons. There are shell
scripts to assist with the management of daemons.
> 3. What are the options for users who don't want to jump onboard map-red? Will pre-mapred be actively maintained?
The MapReduce versons of Nutch tools (inject, generate, fetch, etc.) are
not a large amount of code. One could easily build compatible
non-MapReduce versions of these tools. But then we'd be maintaining two
versions, so we should avoid this as much as possible. However, if some
applications need a very different control flow, then that might warrant
this. For example, one might write a crawler that combines a number of
these tools in a single process, e.g., using RDBMS to keep track of urls
and links while crawling, updating the database in real-time as URLs are
fetched. Such an architecture would not be as scalable, but might excel
in other ways.
It would be worth considering which features of your constrained crawler
could be cast as improvements to Nutch's existing tools (e.g., more
seed url formats, more output formats, http 1.1, custom scopes, etc.)
and which require a different control flow (online fetchlist building?).
In some cases (e.g., fetch prioritization) perhaps a new Plugin should
be added to Nutch.
In the mapred branch the webdb has been decomposed into a crawldb and a
linkdb. The crawldb is much smaller and simpler than the former webdb,
containing only an entry for each known URL. This makes updates much
faster while crawling. The linkdb contains only the link graph, and
needs only be updated prior to indexing, not with each step while
crawling as before.
Doug