You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by fredericoagent <fr...@googlemail.com> on 2009/07/24 10:36:00 UTC
Server suggestion
If I want to setup nutch with lets say 400 million urls in the database.
Is it better to have a 4-5 super fast and loaded servers or have 12-15
smaller , cheaper servers.
By superfast I mean cpu is latest quad core or latest six core processor
with 6 Gigs Ram and 1. or 1.5 TB HD.
By cheap I mean something like a Xeon quad core 2.26 cpu with 3 Gig Ram and
500 Sata HD.
or if anyone can suggest a better spec ideal
--
View this message in context: http://www.nabble.com/Server-suggestion-tp24640718p24640718.html
Sent from the Nutch - Dev mailing list archive at Nabble.com.
Re: Server suggestion
Posted by Dennis Kubes <ku...@apache.org>.
My mistake, you're right. The last processing clusters we built were
using Xeon quad cores, not i7s. The i7s were search servers which
didn't need ecc memory. AFAICT, wikipedia is correct and the i7s don't
yet support ECC.
So my suggestion would be to stick with Xeon procs or something that
supports ECC for the processing clusters. I would never build a
processing cluster that doesn't have ECC memory. We spent a few weeks
when we first started trying to tracking down weird corruption checksum
bugs ultimately related to using non-ECC memory on a cluster.
Dennis
Doğacan Güney wrote:
> Hi Dennis,
>
> On Fri, Jul 24, 2009 at 16:46, Dennis Kubes<ku...@apache.org> wrote:
>>
>> fredericoagent wrote:
>>> If I want to setup nutch with lets say 400 million urls in the database.
>>>
>>> Is it better to have a 4-5 super fast and loaded servers or have 12-15
>>> smaller , cheaper servers.
>> More smaller servers. Make sure they are energy efficient though and have a
>> decent amount of Ram. If a server goes down, you aren't affected as much.
>>
>>> By superfast I mean cpu is latest quad core or latest six core processor
>>> with 6 Gigs Ram and 1. or 1.5 TB HD.
>>>
>>> By cheap I mean something like a Xeon quad core 2.26 cpu with 3 Gig Ram
>>> and
>>> 500 Sata HD.
>>>
>>>
>>> or if anyone can suggest a better spec ideal
>> Our first servers were 1Ghz (Yes really) running hadoop 0.04 way back when.
>> Our first production clusters were core2, 4G ECC, 1 750G hard drive. These
>> days been building i7 8-core, 12G ECC, 4T raid-5 machines with up to 8
>> disks, 2U for around 2200.00 each. If you are looking for a good server
>> builder check out swt.com. They are supermicro resellers and build solid
>> machines.
>>
>
> It suggests here:
>
> http://en.wikipedia.org/wiki/Core_i7#Drawbacks
>
> that core i7's do not support ECC rams. Have you ran into any issues or is WP
> wrong here?
>
>
>> Suggestions. Don't skimp on the hard drive, do at least 750G or more. Price
>> difference is negligible. Do at least 2G Ram, 4G is better, 8G is better
>> than that. You can get up to 12G on regular motherboards these days. After
>> that it gets much more expensive. Ao more recent processors, such as core2
>> or i7. They are more power efficient per processing unit. If you want a
>> really fast machine, do multiple disks in a raid-5 format.
>>
>> Dennis
>>
>
>
>
Re: Server suggestion
Posted by Doğacan Güney <do...@gmail.com>.
Hi Dennis,
On Fri, Jul 24, 2009 at 16:46, Dennis Kubes<ku...@apache.org> wrote:
>
>
> fredericoagent wrote:
>>
>> If I want to setup nutch with lets say 400 million urls in the database.
>>
>> Is it better to have a 4-5 super fast and loaded servers or have 12-15
>> smaller , cheaper servers.
>
> More smaller servers. Make sure they are energy efficient though and have a
> decent amount of Ram. If a server goes down, you aren't affected as much.
>
>>
>> By superfast I mean cpu is latest quad core or latest six core processor
>> with 6 Gigs Ram and 1. or 1.5 TB HD.
>>
>> By cheap I mean something like a Xeon quad core 2.26 cpu with 3 Gig Ram
>> and
>> 500 Sata HD.
>>
>>
>> or if anyone can suggest a better spec ideal
>
> Our first servers were 1Ghz (Yes really) running hadoop 0.04 way back when.
> Our first production clusters were core2, 4G ECC, 1 750G hard drive. These
> days been building i7 8-core, 12G ECC, 4T raid-5 machines with up to 8
> disks, 2U for around 2200.00 each. If you are looking for a good server
> builder check out swt.com. They are supermicro resellers and build solid
> machines.
>
It suggests here:
http://en.wikipedia.org/wiki/Core_i7#Drawbacks
that core i7's do not support ECC rams. Have you ran into any issues or is WP
wrong here?
> Suggestions. Don't skimp on the hard drive, do at least 750G or more. Price
> difference is negligible. Do at least 2G Ram, 4G is better, 8G is better
> than that. You can get up to 12G on regular motherboards these days. After
> that it gets much more expensive. Ao more recent processors, such as core2
> or i7. They are more power efficient per processing unit. If you want a
> really fast machine, do multiple disks in a raid-5 format.
>
> Dennis
>
--
Doğacan Güney
Re: Server suggestion
Posted by Dennis Kubes <ku...@apache.org>.
fredericoagent wrote:
> If I want to setup nutch with lets say 400 million urls in the database.
>
> Is it better to have a 4-5 super fast and loaded servers or have 12-15
> smaller , cheaper servers.
More smaller servers. Make sure they are energy efficient though and
have a decent amount of Ram. If a server goes down, you aren't affected
as much.
>
> By superfast I mean cpu is latest quad core or latest six core processor
> with 6 Gigs Ram and 1. or 1.5 TB HD.
>
> By cheap I mean something like a Xeon quad core 2.26 cpu with 3 Gig Ram and
> 500 Sata HD.
>
>
> or if anyone can suggest a better spec ideal
Our first servers were 1Ghz (Yes really) running hadoop 0.04 way back
when. Our first production clusters were core2, 4G ECC, 1 750G hard
drive. These days been building i7 8-core, 12G ECC, 4T raid-5 machines
with up to 8 disks, 2U for around 2200.00 each. If you are looking for
a good server builder check out swt.com. They are supermicro resellers
and build solid machines.
Suggestions. Don't skimp on the hard drive, do at least 750G or more.
Price difference is negligible. Do at least 2G Ram, 4G is better, 8G is
better than that. You can get up to 12G on regular motherboards these
days. After that it gets much more expensive. Ao more recent
processors, such as core2 or i7. They are more power efficient per
processing unit. If you want a really fast machine, do multiple disks
in a raid-5 format.
Dennis