You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Ken van Mulder <ke...@wavefire.com> on 2005/12/09 00:59:31 UTC
nutch questions
Hey folks,
We're looking at launching a search engine in the beginning of the new
year that will eventually grow to being a multi-billion page index.
Three questions:
First, and most important for now, does anyone have any useful numbers
for what the hardware requirements are to run such an engine? I have
numbers for how fast I can get the crawler's working. But not for how
many pages can be served off of each search node and how much processing
power is required for the indexing, etc.
Second, what all needs to be done to Nutch yet in order for it to be
able to handle billions of pages? Is there a general list of requirements?
Third, if nutch isn't capable of doing what we need, what is the
expected upper limit for it? Using the map/reduce version.
Thanks,
--
Ken van Mulder
Wavefire Technologies Corporation
http://www.wavefire.com
250.717.0200 (ext 113)
Re: nutch questions
Posted by Stefan Groschupf <sg...@media-style.com>.
Ken,
> Thanks Stefan. I'll resend this to the user list as well. Just
> thought the dev list might be better since we're using the map/
> reduce version.
it is just that may other user would be interested to get such
information as well and there are a lot of the developers also
subscribed to the user list.
Cheers,
Stefan
Re: nutch questions
Posted by Ken van Mulder <ke...@wavefire.com>.
Thanks Stefan. I'll resend this to the user list as well. Just thought
the dev list might be better since we're using the map/reduce version.
Thanks!
Stefan Groschupf wrote:
> Ken,
> may the user mailing list would be a better place for such questions.
> The size of your index depends on you configuration(what kind of index
> filter plugins you use)
>
> You can say a document in the index needs 10KB plus the meta data like
> date, content type or category of the page.
> Storing the pages content took around 64KB for each page.
> You also need to store a linkgraph and a list of known urls - web db.
> I would say each 100 Mio document require 1 TB of storage.
>
> Information about query speed can be found in the index, as a role of
> thumb 4 GB of RAM can handle 20 queries per second by 2 Million
> documents per box.
> So in general you need many boxes, but the more expansive part of such
> a project is bandwidth.
>
> Nutch 0.8 works well, however you have to write some custom jobs to get
> some standart jobs done, also storing index on the distributed
> filesystem and search it from there is very very slow. Beside that
> nutch has serious problems with spam detection in very large indexes.
>
> HTH
> Stefan
>
>
>
>
> Am 09.12.2005 um 00:59 schrieb Ken van Mulder:
>
>> Hey folks,
>>
>> We're looking at launching a search engine in the beginning of the
>> new year that will eventually grow to being a multi-billion page
>> index. Three questions:
>>
>> First, and most important for now, does anyone have any useful
>> numbers for what the hardware requirements are to run such an engine?
>> I have numbers for how fast I can get the crawler's working. But not
>> for how many pages can be served off of each search node and how much
>> processing power is required for the indexing, etc.
>>
>> Second, what all needs to be done to Nutch yet in order for it to be
>> able to handle billions of pages? Is there a general list of
>> requirements?
>>
>> Third, if nutch isn't capable of doing what we need, what is the
>> expected upper limit for it? Using the map/reduce version.
>>
>> Thanks,
>>
>> --
>> Ken van Mulder
>> Wavefire Technologies Corporation
>>
>> http://www.wavefire.com
>> 250.717.0200 (ext 113)
>>
>
>
>
--
Ken van Mulder
Wavefire Technologies Corporation
http://www.wavefire.com
250.717.0200 (ext 113)
Re: nutch questions
Posted by Ken van Mulder <ke...@wavefire.com>.
What about how much hardware is required to index data in a reasonable
amount of time? Say you had 3 crawlers running full time (avg 30
pages/s) on a billion page index. How much is required to keep up with
the indexing of that?
Does doubling the ram on a search node have a linear progression?
We'll be doing our own tests shortly, but knowing what to expect would
be helpful. Actually getting a billion pages into an index will take a
bit of time too.
Thanks,
-Ken
Stefan Groschupf wrote:
> Ken,
> may the user mailing list would be a better place for such questions.
> The size of your index depends on you configuration(what kind of index
> filter plugins you use)
>
> You can say a document in the index needs 10KB plus the meta data like
> date, content type or category of the page.
> Storing the pages content took around 64KB for each page.
> You also need to store a linkgraph and a list of known urls - web db.
> I would say each 100 Mio document require 1 TB of storage.
>
> Information about query speed can be found in the index, as a role of
> thumb 4 GB of RAM can handle 20 queries per second by 2 Million
> documents per box.
> So in general you need many boxes, but the more expansive part of such
> a project is bandwidth.
>
> Nutch 0.8 works well, however you have to write some custom jobs to get
> some standart jobs done, also storing index on the distributed
> filesystem and search it from there is very very slow. Beside that
> nutch has serious problems with spam detection in very large indexes.
>
> HTH
> Stefan
>
>
>
>
> Am 09.12.2005 um 00:59 schrieb Ken van Mulder:
>
>> Hey folks,
>>
>> We're looking at launching a search engine in the beginning of the
>> new year that will eventually grow to being a multi-billion page
>> index. Three questions:
>>
>> First, and most important for now, does anyone have any useful
>> numbers for what the hardware requirements are to run such an engine?
>> I have numbers for how fast I can get the crawler's working. But not
>> for how many pages can be served off of each search node and how much
>> processing power is required for the indexing, etc.
>>
>> Second, what all needs to be done to Nutch yet in order for it to be
>> able to handle billions of pages? Is there a general list of
>> requirements?
>>
>> Third, if nutch isn't capable of doing what we need, what is the
>> expected upper limit for it? Using the map/reduce version.
>>
>> Thanks,
>>
>> --
>> Ken van Mulder
>> Wavefire Technologies Corporation
>>
>> http://www.wavefire.com
>> 250.717.0200 (ext 113)
>>
>
>
>
--
Ken van Mulder
Wavefire Technologies Corporation
http://www.wavefire.com
250.717.0200 (ext 113)
Re: nutch questions
Posted by Stefan Groschupf <sg...@media-style.com>.
Ken,
may the user mailing list would be a better place for such questions.
The size of your index depends on you configuration(what kind of
index filter plugins you use)
You can say a document in the index needs 10KB plus the meta data
like date, content type or category of the page.
Storing the pages content took around 64KB for each page.
You also need to store a linkgraph and a list of known urls - web db.
I would say each 100 Mio document require 1 TB of storage.
Information about query speed can be found in the index, as a role of
thumb 4 GB of RAM can handle 20 queries per second by 2 Million
documents per box.
So in general you need many boxes, but the more expansive part of
such a project is bandwidth.
Nutch 0.8 works well, however you have to write some custom jobs to
get some standart jobs done, also storing index on the distributed
filesystem and search it from there is very very slow. Beside that
nutch has serious problems with spam detection in very large indexes.
HTH
Stefan
Am 09.12.2005 um 00:59 schrieb Ken van Mulder:
> Hey folks,
>
> We're looking at launching a search engine in the beginning of the
> new year that will eventually grow to being a multi-billion page
> index. Three questions:
>
> First, and most important for now, does anyone have any useful
> numbers for what the hardware requirements are to run such an
> engine? I have numbers for how fast I can get the crawler's
> working. But not for how many pages can be served off of each
> search node and how much processing power is required for the
> indexing, etc.
>
> Second, what all needs to be done to Nutch yet in order for it to
> be able to handle billions of pages? Is there a general list of
> requirements?
>
> Third, if nutch isn't capable of doing what we need, what is the
> expected upper limit for it? Using the map/reduce version.
>
> Thanks,
>
> --
> Ken van Mulder
> Wavefire Technologies Corporation
>
> http://www.wavefire.com
> 250.717.0200 (ext 113)
>