You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Imran M Yousuf <im...@gmail.com> on 2010/12/17 06:03:53 UTC
[RFC] Deployment layout and server configurations
Hi,
I will go straight to the point, we are purchasing hardwares for going
live. Currently we are targeting a data capacity of ~5TB with a view
to increase capacity as the load increases. I have attached a
tentative schematic as attachment. Please give us feedback on the
network organization and additional tools as you see necessary.
* The HBase cluster will be in a separate 1Gbps network, connected
through a managed switch.
* Currently we do not have any MapReduce jobs, only Gets and
occasional Scans. All searches are done using Solr. Application layer
caching will also be in place.
* 6 DN+RS, each with 1TBX4 7.2k RPM Enterprise HDD, 12GB ECC RAM, 1 X
X3440/E5560 (4C/8T)
* 1 NN+MN RAID 1 with 400GB 15k SAS HDD, 12GB ECC RAM, 1 X X3440/E5560 (4C/8T)
* ZK - 500GB 7.2k RPM, 4GB ECC RAM, 1 X X3440/E5560 (4C/8T)
* ZK + Gangrila + Nagios - 500GB 7.2k RPM, 8GB ECC RAM, 1 X X3440/E5560 (4C/8T)
It would really be helpful for me if you would kindly help us on
whether these configurations are sufficient or not. The nature of the
application running is a E-Commerce PaaS initially. Later when we add
additional applications to the stack we will be able to increase the
capacity.
Regards,
--
Imran M Yousuf
Twitter: @imyousuf - http://twitter.com/imyousuf
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557
Re: [RFC] Deployment layout and server configurations
Posted by Imran M Yousuf <im...@gmail.com>.
Hi,
Please see my replies inline below, thanks a lot for the feedback.
On Fri, Dec 17, 2010 at 12:07 PM, Ted Dunning <td...@maprtech.com> wrote:
> On Thu, Dec 16, 2010 at 9:03 PM, Imran M Yousuf <im...@gmail.com> wrote:
>
>> * 1 NN+MN RAID 1 with 400GB 15k SAS HDD, 12GB ECC RAM, 1 X X3440/E5560
>> (4C/8T)
>>
>
> I don't think that the SAS drives will help you here.
>
Hmm, so will go with 7.2k RPM disks.
>
>> * ZK - 500GB 7.2k RPM, 4GB ECC RAM, 1 X X3440/E5560 (4C/8T)
>>
> * ZK + Gangrila + Nagios - 500GB 7.2k RPM, 8GB ECC RAM, 1 X X3440/E5560
>> (4C/8T)
>>
>
> You should have two drives on each of these machines and the might as well
> match the other drives.
>
> Why only two ZK machines? Normally it is recommended to have 3 or 5. The
> ZK load here should be very light so you might rather collocate ZK with
> other services a bit more.
I am tending to think to have one machince ZK+Gangrila+Nagios and
other 2 collocated with 2 of the RSs' as JG suggests.
>
> Is there any reason not to use 2T drives all round?
>
I was not sure whether 2T all around would be a good idea since I was
using only single 4C/8T processors. What do you think - single
processor for 2T would be ok?
> My own tendency would be to put 12GB all round so that all machines are
> interchangeable.
>
Good suggestion, never thought of it that way.
Thanks,
--
Imran M Yousuf
Twitter: @imyousuf - http://twitter.com/imyousuf
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557
Re: [RFC] Deployment layout and server configurations
Posted by Ted Dunning <td...@maprtech.com>.
On Thu, Dec 16, 2010 at 9:03 PM, Imran M Yousuf <im...@gmail.com> wrote:
> * 1 NN+MN RAID 1 with 400GB 15k SAS HDD, 12GB ECC RAM, 1 X X3440/E5560
> (4C/8T)
>
I don't think that the SAS drives will help you here.
> * ZK - 500GB 7.2k RPM, 4GB ECC RAM, 1 X X3440/E5560 (4C/8T)
>
* ZK + Gangrila + Nagios - 500GB 7.2k RPM, 8GB ECC RAM, 1 X X3440/E5560
> (4C/8T)
>
You should have two drives on each of these machines and the might as well
match the other drives.
Why only two ZK machines? Normally it is recommended to have 3 or 5. The
ZK load here should be very light so you might rather collocate ZK with
other services a bit more.
Is there any reason not to use 2T drives all round?
My own tendency would be to put 12GB all round so that all machines are
interchangeable.
Re: [RFC] Deployment layout and server configurations
Posted by Imran M Yousuf <im...@gmail.com>.
Hi Dave,
Thanks for your feedback.
On Fri, Dec 17, 2010 at 11:14 PM, Buttler, David <bu...@llnl.gov> wrote:
> For a small cluster like you have, I would be fine with putting all of the master pieces on one node:
> ZK, NN, HMaster, ganglia, nagios.
>
Thanks for the confirmation.
> You say you are going to put solr on other nodes -- that makes sense as solr can be very memory hungry, and you might not want to share 12 GB of RAM between solr and hbase. On the other hand, I have no problem with collocating solr and hbase on a large memory machine -- as long as there is no chance of swapping. The only problems come up when you run a heavy map/reduce job over the nodes and kill disk and network i/o -- that can slow solr down. But, as JD often recommends, it is better to keep your developers off of your production system and run your map/reduce jobs on a dedicated cluster (if you can afford the space/power/cooling/$ for multiple clusters).
>
Currently, we do not have any M/R Job to go with, but the plan is you
mentioned and JD recommends production servers will be off-limit for
M/R and for that we have a development cluster.
Regards,
Imran
> Dave
>
> -----Original Message-----
> From: Imran M Yousuf [mailto:imyousuf@gmail.com]
> Sent: Friday, December 17, 2010 4:40 AM
> To: user@hbase.apache.org
> Subject: Re: [RFC] Deployment layout and server configurations
>
> Hi JG :),
>
> On Fri, Dec 17, 2010 at 12:11 PM, Jonathan Gray <jg...@fb.com> wrote:
>> Hey Imran,
>>
>> This looks reasonable but it's hard to say without knowing what the read/write workload is like. You say all searches are done using Solr... will that also be hosted on these servers?
>>
>
> No Solr will not be collocated with them :), I am glad you remember me
> mentioning about it initially, I have attached the full System diagram
> in this email.
>
>> One thing. It looks like you have two servers for ZK? ZK should always be run in odd numbers (1, 3, 5, or 7). For a cluster of this size, you are already sensitive to a failure of the NN so you could just put ZK on the master node. It has RAID so you won't be sensitive to a single disk failure. Otherwise 3 nodes.
>
> Thanks for suggestion, how about if I put Gangrila and Nagios with the
> NN and MN as well?
>
> Regards,
>
>>
>> JG
>>
>>> -----Original Message-----
>>> From: Imran M Yousuf [mailto:imyousuf@gmail.com]
>>> Sent: Thursday, December 16, 2010 9:04 PM
>>> To: user@hbase.apache.org
>>> Subject: [RFC] Deployment layout and server configurations
>>>
>>> Hi,
>>>
>>> I will go straight to the point, we are purchasing hardwares for going live.
>>> Currently we are targeting a data capacity of ~5TB with a view to increase
>>> capacity as the load increases. I have attached a tentative schematic as
>>> attachment. Please give us feedback on the network organization and
>>> additional tools as you see necessary.
>>>
>>> * The HBase cluster will be in a separate 1Gbps network, connected through
>>> a managed switch.
>>> * Currently we do not have any MapReduce jobs, only Gets and occasional
>>> Scans. All searches are done using Solr. Application layer caching will also be
>>> in place.
>>> * 6 DN+RS, each with 1TBX4 7.2k RPM Enterprise HDD, 12GB ECC RAM, 1 X
>>> X3440/E5560 (4C/8T)
>>> * 1 NN+MN RAID 1 with 400GB 15k SAS HDD, 12GB ECC RAM, 1 X
>>> X3440/E5560 (4C/8T)
>>> * ZK - 500GB 7.2k RPM, 4GB ECC RAM, 1 X X3440/E5560 (4C/8T)
>>> * ZK + Gangrila + Nagios - 500GB 7.2k RPM, 8GB ECC RAM, 1 X X3440/E5560
>>> (4C/8T)
>>>
>>> It would really be helpful for me if you would kindly help us on whether
>>> these configurations are sufficient or not. The nature of the application
>>> running is a E-Commerce PaaS initially. Later when we add additional
>>> applications to the stack we will be able to increase the capacity.
>>>
>>> Regards,
>>>
>>> --
>>> Imran M Yousuf
>>> Twitter: @imyousuf - http://twitter.com/imyousuf
>>> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
>>> Mobile: +880-1711402557
>>
>
>
>
> --
> Imran M Yousuf
> Entrepreneur & CEO
> Smart IT Engineering Ltd.
> Dhaka, Bangladesh
> Twitter: @imyousuf - http://twitter.com/imyousuf
> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
> Mobile: +880-1711402557
>
--
Imran M Yousuf
Entrepreneur & CEO
Smart IT Engineering Ltd.
Dhaka, Bangladesh
Twitter: @imyousuf - http://twitter.com/imyousuf
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557
RE: [RFC] Deployment layout and server configurations
Posted by "Buttler, David" <bu...@llnl.gov>.
For a small cluster like you have, I would be fine with putting all of the master pieces on one node:
ZK, NN, HMaster, ganglia, nagios.
You say you are going to put solr on other nodes -- that makes sense as solr can be very memory hungry, and you might not want to share 12 GB of RAM between solr and hbase. On the other hand, I have no problem with collocating solr and hbase on a large memory machine -- as long as there is no chance of swapping. The only problems come up when you run a heavy map/reduce job over the nodes and kill disk and network i/o -- that can slow solr down. But, as JD often recommends, it is better to keep your developers off of your production system and run your map/reduce jobs on a dedicated cluster (if you can afford the space/power/cooling/$ for multiple clusters).
Dave
-----Original Message-----
From: Imran M Yousuf [mailto:imyousuf@gmail.com]
Sent: Friday, December 17, 2010 4:40 AM
To: user@hbase.apache.org
Subject: Re: [RFC] Deployment layout and server configurations
Hi JG :),
On Fri, Dec 17, 2010 at 12:11 PM, Jonathan Gray <jg...@fb.com> wrote:
> Hey Imran,
>
> This looks reasonable but it's hard to say without knowing what the read/write workload is like. You say all searches are done using Solr... will that also be hosted on these servers?
>
No Solr will not be collocated with them :), I am glad you remember me
mentioning about it initially, I have attached the full System diagram
in this email.
> One thing. It looks like you have two servers for ZK? ZK should always be run in odd numbers (1, 3, 5, or 7). For a cluster of this size, you are already sensitive to a failure of the NN so you could just put ZK on the master node. It has RAID so you won't be sensitive to a single disk failure. Otherwise 3 nodes.
Thanks for suggestion, how about if I put Gangrila and Nagios with the
NN and MN as well?
Regards,
>
> JG
>
>> -----Original Message-----
>> From: Imran M Yousuf [mailto:imyousuf@gmail.com]
>> Sent: Thursday, December 16, 2010 9:04 PM
>> To: user@hbase.apache.org
>> Subject: [RFC] Deployment layout and server configurations
>>
>> Hi,
>>
>> I will go straight to the point, we are purchasing hardwares for going live.
>> Currently we are targeting a data capacity of ~5TB with a view to increase
>> capacity as the load increases. I have attached a tentative schematic as
>> attachment. Please give us feedback on the network organization and
>> additional tools as you see necessary.
>>
>> * The HBase cluster will be in a separate 1Gbps network, connected through
>> a managed switch.
>> * Currently we do not have any MapReduce jobs, only Gets and occasional
>> Scans. All searches are done using Solr. Application layer caching will also be
>> in place.
>> * 6 DN+RS, each with 1TBX4 7.2k RPM Enterprise HDD, 12GB ECC RAM, 1 X
>> X3440/E5560 (4C/8T)
>> * 1 NN+MN RAID 1 with 400GB 15k SAS HDD, 12GB ECC RAM, 1 X
>> X3440/E5560 (4C/8T)
>> * ZK - 500GB 7.2k RPM, 4GB ECC RAM, 1 X X3440/E5560 (4C/8T)
>> * ZK + Gangrila + Nagios - 500GB 7.2k RPM, 8GB ECC RAM, 1 X X3440/E5560
>> (4C/8T)
>>
>> It would really be helpful for me if you would kindly help us on whether
>> these configurations are sufficient or not. The nature of the application
>> running is a E-Commerce PaaS initially. Later when we add additional
>> applications to the stack we will be able to increase the capacity.
>>
>> Regards,
>>
>> --
>> Imran M Yousuf
>> Twitter: @imyousuf - http://twitter.com/imyousuf
>> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
>> Mobile: +880-1711402557
>
--
Imran M Yousuf
Entrepreneur & CEO
Smart IT Engineering Ltd.
Dhaka, Bangladesh
Twitter: @imyousuf - http://twitter.com/imyousuf
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557
Re: [RFC] Deployment layout and server configurations
Posted by Imran M Yousuf <im...@gmail.com>.
Hi JG :),
On Fri, Dec 17, 2010 at 12:11 PM, Jonathan Gray <jg...@fb.com> wrote:
> Hey Imran,
>
> This looks reasonable but it's hard to say without knowing what the read/write workload is like. You say all searches are done using Solr... will that also be hosted on these servers?
>
No Solr will not be collocated with them :), I am glad you remember me
mentioning about it initially, I have attached the full System diagram
in this email.
> One thing. It looks like you have two servers for ZK? ZK should always be run in odd numbers (1, 3, 5, or 7). For a cluster of this size, you are already sensitive to a failure of the NN so you could just put ZK on the master node. It has RAID so you won't be sensitive to a single disk failure. Otherwise 3 nodes.
Thanks for suggestion, how about if I put Gangrila and Nagios with the
NN and MN as well?
Regards,
>
> JG
>
>> -----Original Message-----
>> From: Imran M Yousuf [mailto:imyousuf@gmail.com]
>> Sent: Thursday, December 16, 2010 9:04 PM
>> To: user@hbase.apache.org
>> Subject: [RFC] Deployment layout and server configurations
>>
>> Hi,
>>
>> I will go straight to the point, we are purchasing hardwares for going live.
>> Currently we are targeting a data capacity of ~5TB with a view to increase
>> capacity as the load increases. I have attached a tentative schematic as
>> attachment. Please give us feedback on the network organization and
>> additional tools as you see necessary.
>>
>> * The HBase cluster will be in a separate 1Gbps network, connected through
>> a managed switch.
>> * Currently we do not have any MapReduce jobs, only Gets and occasional
>> Scans. All searches are done using Solr. Application layer caching will also be
>> in place.
>> * 6 DN+RS, each with 1TBX4 7.2k RPM Enterprise HDD, 12GB ECC RAM, 1 X
>> X3440/E5560 (4C/8T)
>> * 1 NN+MN RAID 1 with 400GB 15k SAS HDD, 12GB ECC RAM, 1 X
>> X3440/E5560 (4C/8T)
>> * ZK - 500GB 7.2k RPM, 4GB ECC RAM, 1 X X3440/E5560 (4C/8T)
>> * ZK + Gangrila + Nagios - 500GB 7.2k RPM, 8GB ECC RAM, 1 X X3440/E5560
>> (4C/8T)
>>
>> It would really be helpful for me if you would kindly help us on whether
>> these configurations are sufficient or not. The nature of the application
>> running is a E-Commerce PaaS initially. Later when we add additional
>> applications to the stack we will be able to increase the capacity.
>>
>> Regards,
>>
>> --
>> Imran M Yousuf
>> Twitter: @imyousuf - http://twitter.com/imyousuf
>> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
>> Mobile: +880-1711402557
>
--
Imran M Yousuf
Entrepreneur & CEO
Smart IT Engineering Ltd.
Dhaka, Bangladesh
Twitter: @imyousuf - http://twitter.com/imyousuf
Blog: http://imyousuf-tech.blogs.smartitengineering.com/
Mobile: +880-1711402557
RE: [RFC] Deployment layout and server configurations
Posted by Jonathan Gray <jg...@fb.com>.
Hey Imran,
This looks reasonable but it's hard to say without knowing what the read/write workload is like. You say all searches are done using Solr... will that also be hosted on these servers?
One thing. It looks like you have two servers for ZK? ZK should always be run in odd numbers (1, 3, 5, or 7). For a cluster of this size, you are already sensitive to a failure of the NN so you could just put ZK on the master node. It has RAID so you won't be sensitive to a single disk failure. Otherwise 3 nodes.
JG
> -----Original Message-----
> From: Imran M Yousuf [mailto:imyousuf@gmail.com]
> Sent: Thursday, December 16, 2010 9:04 PM
> To: user@hbase.apache.org
> Subject: [RFC] Deployment layout and server configurations
>
> Hi,
>
> I will go straight to the point, we are purchasing hardwares for going live.
> Currently we are targeting a data capacity of ~5TB with a view to increase
> capacity as the load increases. I have attached a tentative schematic as
> attachment. Please give us feedback on the network organization and
> additional tools as you see necessary.
>
> * The HBase cluster will be in a separate 1Gbps network, connected through
> a managed switch.
> * Currently we do not have any MapReduce jobs, only Gets and occasional
> Scans. All searches are done using Solr. Application layer caching will also be
> in place.
> * 6 DN+RS, each with 1TBX4 7.2k RPM Enterprise HDD, 12GB ECC RAM, 1 X
> X3440/E5560 (4C/8T)
> * 1 NN+MN RAID 1 with 400GB 15k SAS HDD, 12GB ECC RAM, 1 X
> X3440/E5560 (4C/8T)
> * ZK - 500GB 7.2k RPM, 4GB ECC RAM, 1 X X3440/E5560 (4C/8T)
> * ZK + Gangrila + Nagios - 500GB 7.2k RPM, 8GB ECC RAM, 1 X X3440/E5560
> (4C/8T)
>
> It would really be helpful for me if you would kindly help us on whether
> these configurations are sufficient or not. The nature of the application
> running is a E-Commerce PaaS initially. Later when we add additional
> applications to the stack we will be able to increase the capacity.
>
> Regards,
>
> --
> Imran M Yousuf
> Twitter: @imyousuf - http://twitter.com/imyousuf
> Blog: http://imyousuf-tech.blogs.smartitengineering.com/
> Mobile: +880-1711402557