You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Tadas Makčinskas <Ta...@bdc.lt> on 2013/04/16 12:13:48 UTC
HW infrastructure for Hadoop
We are thinking to distribute like 50 node cluster. And trying to figure out what would be a good HW infrastructure (Disks – I/O‘s, RAM, CPUs, network). I cannot actually come around any examples that people ran and found it working well and cost effectively.
If anybody could share their best considered infrastructure. Would be a tremendous help not trying to figure it out on our own.
Regards, Tadas
Re: HW infrastructure for Hadoop
Posted by Adam Smieszny <ad...@cloudera.com>.
There are also reference architectures available from a variety of hardware
vendors - the likes of Dell, HP, IBM, Cisco, and others. They often outline
a reasonable framework for disk/cpu/memory mix, and usually include some
description of network as well. If you have a preferred hardware vendor,
that would be another route to pursue.
Cheers,
Adam
On Tue, Apr 16, 2013 at 12:51 PM, Amal G Jose <am...@gmail.com> wrote:
> +1 for Hadoop operations.
> There is one document from Hortonworks that explain hadoop cluster
> infrastructure. That doc is a brand specific. But after referring the
> hadoop operations, we can refer this doc to get a clear overview.
>
>
> On Tue, Apr 16, 2013 at 4:50 PM, Bejoy Ks <be...@gmail.com> wrote:
>
>> +1 for "Hadoop Operations"
>>
>>
>> On Tue, Apr 16, 2013 at 3:57 PM, MARCOS MEDRADO RUBINELLI <
>> marcosm@buscapecompany.com> wrote:
>>
>>> Tadas,
>>>
>>> "Hadoop Operations" has pretty useful, up-to-date information. The
>>> chapter on hardware selection is available here:
>>> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
>>>
>>> Regards,
>>> Marcos
>>>
>>> Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
>>>
>>> We are thinking to distribute like 50 node cluster. And trying to
>>> figure out what would be a good HW infrastructure (Disks – I/O‘s, RAM,
>>> CPUs, network). I cannot actually come around any examples that people ran
>>> and found it working well and cost effectively. ****
>>>
>>> ** **
>>>
>>> If anybody could share their best considered infrastructure. Would be a
>>> tremendous help not trying to figure it out on our own.****
>>>
>>> ** **
>>>
>>> Regards, Tadas****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>>
>>>
>>
>
--
Adam Smieszny
Cloudera | Systems Engineer | http://www.linkedin.com/in/adamsmieszny
917.830.4156
Re: HW infrastructure for Hadoop
Posted by Adam Smieszny <ad...@cloudera.com>.
There are also reference architectures available from a variety of hardware
vendors - the likes of Dell, HP, IBM, Cisco, and others. They often outline
a reasonable framework for disk/cpu/memory mix, and usually include some
description of network as well. If you have a preferred hardware vendor,
that would be another route to pursue.
Cheers,
Adam
On Tue, Apr 16, 2013 at 12:51 PM, Amal G Jose <am...@gmail.com> wrote:
> +1 for Hadoop operations.
> There is one document from Hortonworks that explain hadoop cluster
> infrastructure. That doc is a brand specific. But after referring the
> hadoop operations, we can refer this doc to get a clear overview.
>
>
> On Tue, Apr 16, 2013 at 4:50 PM, Bejoy Ks <be...@gmail.com> wrote:
>
>> +1 for "Hadoop Operations"
>>
>>
>> On Tue, Apr 16, 2013 at 3:57 PM, MARCOS MEDRADO RUBINELLI <
>> marcosm@buscapecompany.com> wrote:
>>
>>> Tadas,
>>>
>>> "Hadoop Operations" has pretty useful, up-to-date information. The
>>> chapter on hardware selection is available here:
>>> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
>>>
>>> Regards,
>>> Marcos
>>>
>>> Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
>>>
>>> We are thinking to distribute like 50 node cluster. And trying to
>>> figure out what would be a good HW infrastructure (Disks – I/O‘s, RAM,
>>> CPUs, network). I cannot actually come around any examples that people ran
>>> and found it working well and cost effectively. ****
>>>
>>> ** **
>>>
>>> If anybody could share their best considered infrastructure. Would be a
>>> tremendous help not trying to figure it out on our own.****
>>>
>>> ** **
>>>
>>> Regards, Tadas****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>>
>>>
>>
>
--
Adam Smieszny
Cloudera | Systems Engineer | http://www.linkedin.com/in/adamsmieszny
917.830.4156
Re: HW infrastructure for Hadoop
Posted by Adam Smieszny <ad...@cloudera.com>.
There are also reference architectures available from a variety of hardware
vendors - the likes of Dell, HP, IBM, Cisco, and others. They often outline
a reasonable framework for disk/cpu/memory mix, and usually include some
description of network as well. If you have a preferred hardware vendor,
that would be another route to pursue.
Cheers,
Adam
On Tue, Apr 16, 2013 at 12:51 PM, Amal G Jose <am...@gmail.com> wrote:
> +1 for Hadoop operations.
> There is one document from Hortonworks that explain hadoop cluster
> infrastructure. That doc is a brand specific. But after referring the
> hadoop operations, we can refer this doc to get a clear overview.
>
>
> On Tue, Apr 16, 2013 at 4:50 PM, Bejoy Ks <be...@gmail.com> wrote:
>
>> +1 for "Hadoop Operations"
>>
>>
>> On Tue, Apr 16, 2013 at 3:57 PM, MARCOS MEDRADO RUBINELLI <
>> marcosm@buscapecompany.com> wrote:
>>
>>> Tadas,
>>>
>>> "Hadoop Operations" has pretty useful, up-to-date information. The
>>> chapter on hardware selection is available here:
>>> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
>>>
>>> Regards,
>>> Marcos
>>>
>>> Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
>>>
>>> We are thinking to distribute like 50 node cluster. And trying to
>>> figure out what would be a good HW infrastructure (Disks – I/O‘s, RAM,
>>> CPUs, network). I cannot actually come around any examples that people ran
>>> and found it working well and cost effectively. ****
>>>
>>> ** **
>>>
>>> If anybody could share their best considered infrastructure. Would be a
>>> tremendous help not trying to figure it out on our own.****
>>>
>>> ** **
>>>
>>> Regards, Tadas****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>>
>>>
>>
>
--
Adam Smieszny
Cloudera | Systems Engineer | http://www.linkedin.com/in/adamsmieszny
917.830.4156
Re: HW infrastructure for Hadoop
Posted by Adam Smieszny <ad...@cloudera.com>.
There are also reference architectures available from a variety of hardware
vendors - the likes of Dell, HP, IBM, Cisco, and others. They often outline
a reasonable framework for disk/cpu/memory mix, and usually include some
description of network as well. If you have a preferred hardware vendor,
that would be another route to pursue.
Cheers,
Adam
On Tue, Apr 16, 2013 at 12:51 PM, Amal G Jose <am...@gmail.com> wrote:
> +1 for Hadoop operations.
> There is one document from Hortonworks that explain hadoop cluster
> infrastructure. That doc is a brand specific. But after referring the
> hadoop operations, we can refer this doc to get a clear overview.
>
>
> On Tue, Apr 16, 2013 at 4:50 PM, Bejoy Ks <be...@gmail.com> wrote:
>
>> +1 for "Hadoop Operations"
>>
>>
>> On Tue, Apr 16, 2013 at 3:57 PM, MARCOS MEDRADO RUBINELLI <
>> marcosm@buscapecompany.com> wrote:
>>
>>> Tadas,
>>>
>>> "Hadoop Operations" has pretty useful, up-to-date information. The
>>> chapter on hardware selection is available here:
>>> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
>>>
>>> Regards,
>>> Marcos
>>>
>>> Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
>>>
>>> We are thinking to distribute like 50 node cluster. And trying to
>>> figure out what would be a good HW infrastructure (Disks – I/O‘s, RAM,
>>> CPUs, network). I cannot actually come around any examples that people ran
>>> and found it working well and cost effectively. ****
>>>
>>> ** **
>>>
>>> If anybody could share their best considered infrastructure. Would be a
>>> tremendous help not trying to figure it out on our own.****
>>>
>>> ** **
>>>
>>> Regards, Tadas****
>>>
>>> ** **
>>>
>>> ** **
>>>
>>>
>>>
>>
>
--
Adam Smieszny
Cloudera | Systems Engineer | http://www.linkedin.com/in/adamsmieszny
917.830.4156
Re: HW infrastructure for Hadoop
Posted by Amal G Jose <am...@gmail.com>.
+1 for Hadoop operations.
There is one document from Hortonworks that explain hadoop cluster
infrastructure. That doc is a brand specific. But after referring the
hadoop operations, we can refer this doc to get a clear overview.
On Tue, Apr 16, 2013 at 4:50 PM, Bejoy Ks <be...@gmail.com> wrote:
> +1 for "Hadoop Operations"
>
>
> On Tue, Apr 16, 2013 at 3:57 PM, MARCOS MEDRADO RUBINELLI <
> marcosm@buscapecompany.com> wrote:
>
>> Tadas,
>>
>> "Hadoop Operations" has pretty useful, up-to-date information. The
>> chapter on hardware selection is available here:
>> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
>>
>> Regards,
>> Marcos
>>
>> Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
>>
>> We are thinking to distribute like 50 node cluster. And trying to
>> figure out what would be a good HW infrastructure (Disks – I/O‘s, RAM,
>> CPUs, network). I cannot actually come around any examples that people ran
>> and found it working well and cost effectively. ****
>>
>> ** **
>>
>> If anybody could share their best considered infrastructure. Would be a
>> tremendous help not trying to figure it out on our own.****
>>
>> ** **
>>
>> Regards, Tadas****
>>
>> ** **
>>
>> ** **
>>
>>
>>
>
Re: HW infrastructure for Hadoop
Posted by Amal G Jose <am...@gmail.com>.
+1 for Hadoop operations.
There is one document from Hortonworks that explain hadoop cluster
infrastructure. That doc is a brand specific. But after referring the
hadoop operations, we can refer this doc to get a clear overview.
On Tue, Apr 16, 2013 at 4:50 PM, Bejoy Ks <be...@gmail.com> wrote:
> +1 for "Hadoop Operations"
>
>
> On Tue, Apr 16, 2013 at 3:57 PM, MARCOS MEDRADO RUBINELLI <
> marcosm@buscapecompany.com> wrote:
>
>> Tadas,
>>
>> "Hadoop Operations" has pretty useful, up-to-date information. The
>> chapter on hardware selection is available here:
>> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
>>
>> Regards,
>> Marcos
>>
>> Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
>>
>> We are thinking to distribute like 50 node cluster. And trying to
>> figure out what would be a good HW infrastructure (Disks – I/O‘s, RAM,
>> CPUs, network). I cannot actually come around any examples that people ran
>> and found it working well and cost effectively. ****
>>
>> ** **
>>
>> If anybody could share their best considered infrastructure. Would be a
>> tremendous help not trying to figure it out on our own.****
>>
>> ** **
>>
>> Regards, Tadas****
>>
>> ** **
>>
>> ** **
>>
>>
>>
>
Re: HW infrastructure for Hadoop
Posted by Amal G Jose <am...@gmail.com>.
+1 for Hadoop operations.
There is one document from Hortonworks that explain hadoop cluster
infrastructure. That doc is a brand specific. But after referring the
hadoop operations, we can refer this doc to get a clear overview.
On Tue, Apr 16, 2013 at 4:50 PM, Bejoy Ks <be...@gmail.com> wrote:
> +1 for "Hadoop Operations"
>
>
> On Tue, Apr 16, 2013 at 3:57 PM, MARCOS MEDRADO RUBINELLI <
> marcosm@buscapecompany.com> wrote:
>
>> Tadas,
>>
>> "Hadoop Operations" has pretty useful, up-to-date information. The
>> chapter on hardware selection is available here:
>> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
>>
>> Regards,
>> Marcos
>>
>> Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
>>
>> We are thinking to distribute like 50 node cluster. And trying to
>> figure out what would be a good HW infrastructure (Disks – I/O‘s, RAM,
>> CPUs, network). I cannot actually come around any examples that people ran
>> and found it working well and cost effectively. ****
>>
>> ** **
>>
>> If anybody could share their best considered infrastructure. Would be a
>> tremendous help not trying to figure it out on our own.****
>>
>> ** **
>>
>> Regards, Tadas****
>>
>> ** **
>>
>> ** **
>>
>>
>>
>
Re: HW infrastructure for Hadoop
Posted by Amal G Jose <am...@gmail.com>.
+1 for Hadoop operations.
There is one document from Hortonworks that explain hadoop cluster
infrastructure. That doc is a brand specific. But after referring the
hadoop operations, we can refer this doc to get a clear overview.
On Tue, Apr 16, 2013 at 4:50 PM, Bejoy Ks <be...@gmail.com> wrote:
> +1 for "Hadoop Operations"
>
>
> On Tue, Apr 16, 2013 at 3:57 PM, MARCOS MEDRADO RUBINELLI <
> marcosm@buscapecompany.com> wrote:
>
>> Tadas,
>>
>> "Hadoop Operations" has pretty useful, up-to-date information. The
>> chapter on hardware selection is available here:
>> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
>>
>> Regards,
>> Marcos
>>
>> Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
>>
>> We are thinking to distribute like 50 node cluster. And trying to
>> figure out what would be a good HW infrastructure (Disks – I/O‘s, RAM,
>> CPUs, network). I cannot actually come around any examples that people ran
>> and found it working well and cost effectively. ****
>>
>> ** **
>>
>> If anybody could share their best considered infrastructure. Would be a
>> tremendous help not trying to figure it out on our own.****
>>
>> ** **
>>
>> Regards, Tadas****
>>
>> ** **
>>
>> ** **
>>
>>
>>
>
Re: HW infrastructure for Hadoop
Posted by Bejoy Ks <be...@gmail.com>.
+1 for "Hadoop Operations"
On Tue, Apr 16, 2013 at 3:57 PM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Tadas,
>
> "Hadoop Operations" has pretty useful, up-to-date information. The chapter
> on hardware selection is available here:
> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
>
> Regards,
> Marcos
>
> Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
>
> We are thinking to distribute like 50 node cluster. And trying to figure
> out what would be a good HW infrastructure (Disks – I/O‘s, RAM, CPUs,
> network). I cannot actually come around any examples that people ran and
> found it working well and cost effectively. ****
>
> ** **
>
> If anybody could share their best considered infrastructure. Would be a
> tremendous help not trying to figure it out on our own.****
>
> ** **
>
> Regards, Tadas****
>
> ** **
>
> ** **
>
>
>
Re: HW infrastructure for Hadoop
Posted by Bejoy Ks <be...@gmail.com>.
+1 for "Hadoop Operations"
On Tue, Apr 16, 2013 at 3:57 PM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Tadas,
>
> "Hadoop Operations" has pretty useful, up-to-date information. The chapter
> on hardware selection is available here:
> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
>
> Regards,
> Marcos
>
> Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
>
> We are thinking to distribute like 50 node cluster. And trying to figure
> out what would be a good HW infrastructure (Disks – I/O‘s, RAM, CPUs,
> network). I cannot actually come around any examples that people ran and
> found it working well and cost effectively. ****
>
> ** **
>
> If anybody could share their best considered infrastructure. Would be a
> tremendous help not trying to figure it out on our own.****
>
> ** **
>
> Regards, Tadas****
>
> ** **
>
> ** **
>
>
>
Re: HW infrastructure for Hadoop
Posted by Bejoy Ks <be...@gmail.com>.
+1 for "Hadoop Operations"
On Tue, Apr 16, 2013 at 3:57 PM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Tadas,
>
> "Hadoop Operations" has pretty useful, up-to-date information. The chapter
> on hardware selection is available here:
> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
>
> Regards,
> Marcos
>
> Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
>
> We are thinking to distribute like 50 node cluster. And trying to figure
> out what would be a good HW infrastructure (Disks – I/O‘s, RAM, CPUs,
> network). I cannot actually come around any examples that people ran and
> found it working well and cost effectively. ****
>
> ** **
>
> If anybody could share their best considered infrastructure. Would be a
> tremendous help not trying to figure it out on our own.****
>
> ** **
>
> Regards, Tadas****
>
> ** **
>
> ** **
>
>
>
Re: HW infrastructure for Hadoop
Posted by Bejoy Ks <be...@gmail.com>.
+1 for "Hadoop Operations"
On Tue, Apr 16, 2013 at 3:57 PM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Tadas,
>
> "Hadoop Operations" has pretty useful, up-to-date information. The chapter
> on hardware selection is available here:
> http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
>
> Regards,
> Marcos
>
> Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
>
> We are thinking to distribute like 50 node cluster. And trying to figure
> out what would be a good HW infrastructure (Disks – I/O‘s, RAM, CPUs,
> network). I cannot actually come around any examples that people ran and
> found it working well and cost effectively. ****
>
> ** **
>
> If anybody could share their best considered infrastructure. Would be a
> tremendous help not trying to figure it out on our own.****
>
> ** **
>
> Regards, Tadas****
>
> ** **
>
> ** **
>
>
>
Re: HW infrastructure for Hadoop
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Tadas,
"Hadoop Operations" has pretty useful, up-to-date information. The chapter on hardware selection is available here: http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
Regards,
Marcos
Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
We are thinking to distribute like 50 node cluster. And trying to figure out what would be a good HW infrastructure (Disks – I/O‘s, RAM, CPUs, network). I cannot actually come around any examples that people ran and found it working well and cost effectively.
If anybody could share their best considered infrastructure. Would be a tremendous help not trying to figure it out on our own.
Regards, Tadas
Re: HW infrastructure for Hadoop
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Tadas,
"Hadoop Operations" has pretty useful, up-to-date information. The chapter on hardware selection is available here: http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
Regards,
Marcos
Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
We are thinking to distribute like 50 node cluster. And trying to figure out what would be a good HW infrastructure (Disks – I/O‘s, RAM, CPUs, network). I cannot actually come around any examples that people ran and found it working well and cost effectively.
If anybody could share their best considered infrastructure. Would be a tremendous help not trying to figure it out on our own.
Regards, Tadas
Re: HW infrastructure for Hadoop
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Tadas,
"Hadoop Operations" has pretty useful, up-to-date information. The chapter on hardware selection is available here: http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
Regards,
Marcos
Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
We are thinking to distribute like 50 node cluster. And trying to figure out what would be a good HW infrastructure (Disks – I/O‘s, RAM, CPUs, network). I cannot actually come around any examples that people ran and found it working well and cost effectively.
If anybody could share their best considered infrastructure. Would be a tremendous help not trying to figure it out on our own.
Regards, Tadas
Re: HW infrastructure for Hadoop
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Tadas,
"Hadoop Operations" has pretty useful, up-to-date information. The chapter on hardware selection is available here: http://my.safaribooksonline.com/book/databases/hadoop/9781449327279/4dot-planning-a-hadoop-cluster/id2760689
Regards,
Marcos
Em 16-04-2013 07:13, Tadas Makčinskas escreveu:
We are thinking to distribute like 50 node cluster. And trying to figure out what would be a good HW infrastructure (Disks – I/O‘s, RAM, CPUs, network). I cannot actually come around any examples that people ran and found it working well and cost effectively.
If anybody could share their best considered infrastructure. Would be a tremendous help not trying to figure it out on our own.
Regards, Tadas