You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Ninad Raut <hb...@gmail.com> on 2009/05/14 11:56:04 UTC
Keeping Compute Nodes seperate from the region server node-- pros and
cons
Hi,
I want to get a design perspective here as to what will be the advantages of
seperating region servers and compute node(to run mapreduce tasks)
Will seperating datanodes from computes node reduce the load on the servers
and avoid swapping problems?
Will this seperation make map reduce tasks less efficient , since we are
doing away with localization issues?
Regards,
Ninad
Re: Keeping Compute Nodes seperate from the region server node-- pros
and cons
Posted by Ninad Raut <hb...@gmail.com>.
Hi Andy,
I am using EC2 cluster with large server grade machines. Hense, availability
cannot be determined as the cluster nodes can change ip overtime.
Yes, I am interested in replication. What should be a ideal design in this
case?
On Fri, May 15, 2009 at 9:33 PM, Andrew Purtell <ap...@apache.org> wrote:
> Hi Ninad,
>
> I think scenario 1 is fine for your case, < 20 nodes up on EC2.
>
> Are you planning to deploy Hadoop+HBase clusters in more than one
> availability zone? Interested in or implementing replication between?
>
> Best regards,
>
> - Andy
>
>
>
>
> ________________________________
> From: Ninad Raut <hb...@gmail.com>
> To: hbase-user@hadoop.apache.org
> Cc: Ranjit Nair <ra...@germinait.com>
> Sent: Thursday, May 14, 2009 11:30:05 PM
> Subject: Re: Keeping Compute Nodes seperate from the region server node--
> pros and cons
>
> Hi Andy,
> Thanks for the tip.
> I have a EC2 cluster with 6 nodes. Each a server grade large instance. I
> have the mapred & regionservers running on all the nodes. Our deployment
> will not go beyond 20 clusters in the near future. What would you suggest
> me
> to have? Scenario 1 or 2 as u mentioned ?
>
> On Thu, May 14, 2009 at 10:44 PM, Andrew Purtell <apurtell@apache.org
> >wrote:
>
> > Hi Ninad,
> >
> > I think the answer depends on the anticipated scale of the deployment.
> >
> > For small clusters (up to a few racks, ~40 servers per rack) I don't
> think
> > there is any significant performance hit to separate storage and
> > computation. Presumably all servers will share the same large GigE switch
> --
> > or maybe a redundant L2 pair via bonded interfaces for fail over -- or a
> few
> > of them stacked with high speed interconnects. This would relieve the
> > storage nodes of RAM and CPU burden related to the computational tasks as
> > you are thinking, providing more headroom in exchange for some quite
> modest
> > performance penalty. (However, if your computation load is high and
> > therefore the nodes are overburdened and are not stable, there is no
> > alternative...) In the future this consideration might change if DFS
> clients
> > are given some capability to find blocks on local disk via some optimized
> > I/O path.
> >
> > In a large cluster there might well be significant performance impact. In
> a
> > common deployment scenario, there are rack-local switched fabrics and
> > another switched fabric for uplinks from the racks. So, a rack would have
> a
> > switched GigE backplane or similar, but inter-rack connections might be
> > single GigE uplinks, a ~40-to-1 reduction in capacity worst case; or
> maybe
> > 10 GigE uplinks, a ~10-1 reduction. Therefore it would be desirable to
> > distribute the computation into the racks where the data is located. When
> a
> > region is deployed to a region server the underlying blocks on DFS are
> not
> > immediately migrated, but always after a compaction -- a rewrite -- the
> > underlying blocks will be available on rack local data nodes, according
> to
> > my understanding of how DFS places replicas upon write. So, after a
> split,
> > daughter regions will have their blocks appropriately located in a timely
> > manner. For the rest I wonder if it would be beneficial to consider
> > scheduling major compaction more frequently than the 24 hour default for
> > datacenter scale deployments, something like 8 hours, and you might also
> > consider triggering a major compaction on important tables after cluster
> > (re)init. Region deployment in a system in steady state should have
> > relatively little churn so this will have the effect of optimizing block
> > placement for region store access.
> >
> > Submitted for your consideration,
> >
> > - Andy
> >
> >
> >
> >
> >
> >
> > ________________________________
> > From: Ninad Raut <hb...@gmail.com>
> > To: hbase-user <hb...@hadoop.apache.org>
> > Cc: Ranjit Nair <ra...@germinait.com>
> > Sent: Thursday, May 14, 2009 2:56:04 AM
> > Subject: Keeping Compute Nodes seperate from the region server node--
> pros
> > and cons
> >
> > Hi,
> > I want to get a design perspective here as to what will be the advantages
> > of
> > seperating region servers and compute node(to run mapreduce tasks)
> > Will seperating datanodes from computes node reduce the load on the
> servers
> > and avoid swapping problems?
> > Will this seperation make map reduce tasks less efficient , since we are
> > doing away with localization issues?
> > Regards,
> > Ninad
> >
> >
> >
> >
> >
>
>
>
>
>
Re: Keeping Compute Nodes seperate from the region server node-- pros and cons
Posted by Andrew Purtell <ap...@apache.org>.
Hi Ninad,
I think scenario 1 is fine for your case, < 20 nodes up on EC2.
Are you planning to deploy Hadoop+HBase clusters in more than one availability zone? Interested in or implementing replication between?
Best regards,
- Andy
________________________________
From: Ninad Raut <hb...@gmail.com>
To: hbase-user@hadoop.apache.org
Cc: Ranjit Nair <ra...@germinait.com>
Sent: Thursday, May 14, 2009 11:30:05 PM
Subject: Re: Keeping Compute Nodes seperate from the region server node-- pros and cons
Hi Andy,
Thanks for the tip.
I have a EC2 cluster with 6 nodes. Each a server grade large instance. I
have the mapred & regionservers running on all the nodes. Our deployment
will not go beyond 20 clusters in the near future. What would you suggest me
to have? Scenario 1 or 2 as u mentioned ?
On Thu, May 14, 2009 at 10:44 PM, Andrew Purtell <ap...@apache.org>wrote:
> Hi Ninad,
>
> I think the answer depends on the anticipated scale of the deployment.
>
> For small clusters (up to a few racks, ~40 servers per rack) I don't think
> there is any significant performance hit to separate storage and
> computation. Presumably all servers will share the same large GigE switch --
> or maybe a redundant L2 pair via bonded interfaces for fail over -- or a few
> of them stacked with high speed interconnects. This would relieve the
> storage nodes of RAM and CPU burden related to the computational tasks as
> you are thinking, providing more headroom in exchange for some quite modest
> performance penalty. (However, if your computation load is high and
> therefore the nodes are overburdened and are not stable, there is no
> alternative...) In the future this consideration might change if DFS clients
> are given some capability to find blocks on local disk via some optimized
> I/O path.
>
> In a large cluster there might well be significant performance impact. In a
> common deployment scenario, there are rack-local switched fabrics and
> another switched fabric for uplinks from the racks. So, a rack would have a
> switched GigE backplane or similar, but inter-rack connections might be
> single GigE uplinks, a ~40-to-1 reduction in capacity worst case; or maybe
> 10 GigE uplinks, a ~10-1 reduction. Therefore it would be desirable to
> distribute the computation into the racks where the data is located. When a
> region is deployed to a region server the underlying blocks on DFS are not
> immediately migrated, but always after a compaction -- a rewrite -- the
> underlying blocks will be available on rack local data nodes, according to
> my understanding of how DFS places replicas upon write. So, after a split,
> daughter regions will have their blocks appropriately located in a timely
> manner. For the rest I wonder if it would be beneficial to consider
> scheduling major compaction more frequently than the 24 hour default for
> datacenter scale deployments, something like 8 hours, and you might also
> consider triggering a major compaction on important tables after cluster
> (re)init. Region deployment in a system in steady state should have
> relatively little churn so this will have the effect of optimizing block
> placement for region store access.
>
> Submitted for your consideration,
>
> - Andy
>
>
>
>
>
>
> ________________________________
> From: Ninad Raut <hb...@gmail.com>
> To: hbase-user <hb...@hadoop.apache.org>
> Cc: Ranjit Nair <ra...@germinait.com>
> Sent: Thursday, May 14, 2009 2:56:04 AM
> Subject: Keeping Compute Nodes seperate from the region server node-- pros
> and cons
>
> Hi,
> I want to get a design perspective here as to what will be the advantages
> of
> seperating region servers and compute node(to run mapreduce tasks)
> Will seperating datanodes from computes node reduce the load on the servers
> and avoid swapping problems?
> Will this seperation make map reduce tasks less efficient , since we are
> doing away with localization issues?
> Regards,
> Ninad
>
>
>
>
>
Re: Keeping Compute Nodes seperate from the region server node-- pros
and cons
Posted by Ninad Raut <hb...@gmail.com>.
Hi Andy,
Thanks for the tip.
I have a EC2 cluster with 6 nodes. Each a server grade large instance. I
have the mapred & regionservers running on all the nodes. Our deployment
will not go beyond 20 clusters in the near future. What would you suggest me
to have? Scenario 1 or 2 as u mentioned ?
On Thu, May 14, 2009 at 10:44 PM, Andrew Purtell <ap...@apache.org>wrote:
> Hi Ninad,
>
> I think the answer depends on the anticipated scale of the deployment.
>
> For small clusters (up to a few racks, ~40 servers per rack) I don't think
> there is any significant performance hit to separate storage and
> computation. Presumably all servers will share the same large GigE switch --
> or maybe a redundant L2 pair via bonded interfaces for fail over -- or a few
> of them stacked with high speed interconnects. This would relieve the
> storage nodes of RAM and CPU burden related to the computational tasks as
> you are thinking, providing more headroom in exchange for some quite modest
> performance penalty. (However, if your computation load is high and
> therefore the nodes are overburdened and are not stable, there is no
> alternative...) In the future this consideration might change if DFS clients
> are given some capability to find blocks on local disk via some optimized
> I/O path.
>
> In a large cluster there might well be significant performance impact. In a
> common deployment scenario, there are rack-local switched fabrics and
> another switched fabric for uplinks from the racks. So, a rack would have a
> switched GigE backplane or similar, but inter-rack connections might be
> single GigE uplinks, a ~40-to-1 reduction in capacity worst case; or maybe
> 10 GigE uplinks, a ~10-1 reduction. Therefore it would be desirable to
> distribute the computation into the racks where the data is located. When a
> region is deployed to a region server the underlying blocks on DFS are not
> immediately migrated, but always after a compaction -- a rewrite -- the
> underlying blocks will be available on rack local data nodes, according to
> my understanding of how DFS places replicas upon write. So, after a split,
> daughter regions will have their blocks appropriately located in a timely
> manner. For the rest I wonder if it would be beneficial to consider
> scheduling
> major compaction more frequently than the 24 hour default for datacenter
> scale deployments, something like 8 hours, and you might also consider
> triggering a major compaction on important tables after cluster (re)init.
> Region deployment in a system in steady state should have relatively little
> churn so this will have the effect of optimizing block placement for region
> store access.
>
> Submitted for your consideration,
>
> - Andy
>
>
>
>
>
>
> ________________________________
> From: Ninad Raut <hb...@gmail.com>
> To: hbase-user <hb...@hadoop.apache.org>
> Cc: Ranjit Nair <ra...@germinait.com>
> Sent: Thursday, May 14, 2009 2:56:04 AM
> Subject: Keeping Compute Nodes seperate from the region server node-- pros
> and cons
>
> Hi,
> I want to get a design perspective here as to what will be the advantages
> of
> seperating region servers and compute node(to run mapreduce tasks)
> Will seperating datanodes from computes node reduce the load on the servers
> and avoid swapping problems?
> Will this seperation make map reduce tasks less efficient , since we are
> doing away with localization issues?
> Regards,
> Ninad
>
>
>
>
>
Re: Keeping Compute Nodes seperate from the region server node-- pros and cons
Posted by Andrew Purtell <ap...@apache.org>.
Hi Ninad,
I think the answer depends on the anticipated scale of the deployment.
For small clusters (up to a few racks, ~40 servers per rack) I don't think there is any significant performance hit to separate storage and computation. Presumably all servers will share the same large GigE switch -- or maybe a redundant L2 pair via bonded interfaces for fail over -- or a few of them stacked with high speed interconnects. This would relieve the storage nodes of RAM and CPU burden related to the computational tasks as you are thinking, providing more headroom in exchange for some quite modest performance penalty. (However, if your computation load is high and therefore the nodes are overburdened and are not stable, there is no alternative...) In the future this consideration might change if DFS clients are given some capability to find blocks on local disk via some optimized I/O path.
In a large cluster there might well be significant performance impact. In a common deployment scenario, there are rack-local switched fabrics and another switched fabric for uplinks from the racks. So, a rack would have a switched GigE backplane or similar, but inter-rack connections might be single GigE uplinks, a ~40-to-1 reduction in capacity worst case; or maybe 10 GigE uplinks, a ~10-1 reduction. Therefore it would be desirable to distribute the computation into the racks where the data is located. When a region is deployed to a region server the underlying blocks on DFS are not immediately migrated, but always after a compaction -- a rewrite -- the underlying blocks will be available on rack local data nodes, according to my understanding of how DFS places replicas upon write. So, after a split, daughter regions will have their blocks appropriately located in a timely manner. For the rest I wonder if it would be beneficial to consider scheduling
major compaction more frequently than the 24 hour default for datacenter scale deployments, something like 8 hours, and you might also consider triggering a major compaction on important tables after cluster (re)init. Region deployment in a system in steady state should have relatively little churn so this will have the effect of optimizing block placement for region store access.
Submitted for your consideration,
- Andy
________________________________
From: Ninad Raut <hb...@gmail.com>
To: hbase-user <hb...@hadoop.apache.org>
Cc: Ranjit Nair <ra...@germinait.com>
Sent: Thursday, May 14, 2009 2:56:04 AM
Subject: Keeping Compute Nodes seperate from the region server node-- pros and cons
Hi,
I want to get a design perspective here as to what will be the advantages of
seperating region servers and compute node(to run mapreduce tasks)
Will seperating datanodes from computes node reduce the load on the servers
and avoid swapping problems?
Will this seperation make map reduce tasks less efficient , since we are
doing away with localization issues?
Regards,
Ninad