You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Bharath Ravi <bh...@gmail.com> on 2011/10/16 03:53:36 UTC
Load balancing requests in HDFS
Hi all,
I have a question about how HDFS load balances requests for files/blocks:
HDFS currently distributes data blocks randomly, for balance.
However, if certain files/blocks are more popular than others, some nodes
might get an "unfair" number of requests.
Adding more replicas for these popular files might not help, unless HDFS
explicitly distributes requests fairly among the replicas.
Looking around, I saw a bit of discussion on this dating to 2009,
Is something like this is done by HDFS today?
http://www.mail-archive.com/core-user@hadoop.apache.org/msg04357.html
http://grokbase.com/p/hadoop.apache.org/common-dev/2009/02/re-hdfs-load-traffic-balancing/18en5o4z2j4b6eaqnemkp4eppd7q
Thanks a lot in advance!
--
Bharath Ravi
Re: Load balancing requests in HDFS
Posted by Bharath Ravi <bh...@gmail.com>.
That clarified it, thanks a lot!
I'm trying to write a load balancer that prioritises datanodes in terms of
load (in the last t minutes, say)
as well as proximity to the client, instead of just the proximity. That is,
if the closest datanode to the client for a block is facing
a load above some threshold, it picks the next closest, that is not too
heavily loaded.
Sounds like this could work well in conjunction with
ReplicationTargetChooser, since it handles write requests, which is
another side of the same coin.
Thanks, again!
On 19 October 2011 02:04, Uma Maheswara Rao G 72686 <ma...@huawei.com>wrote:
> ----- Original Message -----
> From: Bharath Ravi <bh...@gmail.com>
> Date: Wednesday, October 19, 2011 8:16 am
> Subject: Re: Load balancing requests in HDFS
> To: common-dev@hadoop.apache.org
>
> > Thanks a lot Steve!
> >
> > ReplicationTargetChooser seems to address load balancing for initially
> > placing/laying out data,
> > but it doesn't seem to do active load balancing for incoming
> > requests to a
> > datanode: or does it?
>
> For every request, ReplicationTargetChooser will check the good targets to
> write.
> ( space, traffic, threadcount on DN..etc). DNs will update their statistics
> by heartbeats. So, NN can check this before actually choosing the taget to
> write the Data.
> Hope, this clarifies your doubt.
>
> >
> > Also, would you know if there are statistics on how effective
> > over-replication is for throughput gain?
> > Basically, although one might add more replicas, are they actually
> > usedeffectively to serve incoming requests?
> here over-replication means uping the replication factor, is it?
>
> >
> > On 18 October 2011 12:37, Steve Loughran <st...@apache.org> wrote:
> >
> > > On 16/10/11 02:53, Bharath Ravi wrote:
> > >
> > >> Hi all,
> > >>
> > >> I have a question about how HDFS load balances requests for
> > files/blocks:>>
> > >> HDFS currently distributes data blocks randomly, for balance.
> > >> However, if certain files/blocks are more popular than others,
> > some nodes
> > >> might get an "unfair" number of requests.
> > >> Adding more replicas for these popular files might not help,
> > unless HDFS
> > >> explicitly distributes requests fairly among the replicas.
> > >>
> > >
> > > Have a look at the ReplicationTargetChooser class; it does take
> > datanode> load into account, though it's concern is distribution
> > for data
> > > availability, not performance.
> > >
> > > The standard technique for popular files -including MR job JAR
> > files- is to
> > > over-replicate. One problem: how to determine what is popular
> > without adding
> > > more load on the namenode
> > >
> >
> >
> >
> > --
> > Bharath Ravi
> >
>
> Regards,
> Uma
>
--
Bharath Ravi
Re: Load balancing requests in HDFS
Posted by Uma Maheswara Rao G 72686 <ma...@huawei.com>.
----- Original Message -----
From: Bharath Ravi <bh...@gmail.com>
Date: Wednesday, October 19, 2011 8:16 am
Subject: Re: Load balancing requests in HDFS
To: common-dev@hadoop.apache.org
> Thanks a lot Steve!
>
> ReplicationTargetChooser seems to address load balancing for initially
> placing/laying out data,
> but it doesn't seem to do active load balancing for incoming
> requests to a
> datanode: or does it?
For every request, ReplicationTargetChooser will check the good targets to write.
( space, traffic, threadcount on DN..etc). DNs will update their statistics by heartbeats. So, NN can check this before actually choosing the taget to write the Data.
Hope, this clarifies your doubt.
>
> Also, would you know if there are statistics on how effective
> over-replication is for throughput gain?
> Basically, although one might add more replicas, are they actually
> usedeffectively to serve incoming requests?
here over-replication means uping the replication factor, is it?
>
> On 18 October 2011 12:37, Steve Loughran <st...@apache.org> wrote:
>
> > On 16/10/11 02:53, Bharath Ravi wrote:
> >
> >> Hi all,
> >>
> >> I have a question about how HDFS load balances requests for
> files/blocks:>>
> >> HDFS currently distributes data blocks randomly, for balance.
> >> However, if certain files/blocks are more popular than others,
> some nodes
> >> might get an "unfair" number of requests.
> >> Adding more replicas for these popular files might not help,
> unless HDFS
> >> explicitly distributes requests fairly among the replicas.
> >>
> >
> > Have a look at the ReplicationTargetChooser class; it does take
> datanode> load into account, though it's concern is distribution
> for data
> > availability, not performance.
> >
> > The standard technique for popular files -including MR job JAR
> files- is to
> > over-replicate. One problem: how to determine what is popular
> without adding
> > more load on the namenode
> >
>
>
>
> --
> Bharath Ravi
>
Regards,
Uma
Re: Load balancing requests in HDFS
Posted by Bharath Ravi <bh...@gmail.com>.
Thanks a lot Steve!
ReplicationTargetChooser seems to address load balancing for initially
placing/laying out data,
but it doesn't seem to do active load balancing for incoming requests to a
datanode: or does it?
Also, would you know if there are statistics on how effective
over-replication is for throughput gain?
Basically, although one might add more replicas, are they actually used
effectively to serve incoming requests?
On 18 October 2011 12:37, Steve Loughran <st...@apache.org> wrote:
> On 16/10/11 02:53, Bharath Ravi wrote:
>
>> Hi all,
>>
>> I have a question about how HDFS load balances requests for files/blocks:
>>
>> HDFS currently distributes data blocks randomly, for balance.
>> However, if certain files/blocks are more popular than others, some nodes
>> might get an "unfair" number of requests.
>> Adding more replicas for these popular files might not help, unless HDFS
>> explicitly distributes requests fairly among the replicas.
>>
>
> Have a look at the ReplicationTargetChooser class; it does take datanode
> load into account, though it's concern is distribution for data
> availability, not performance.
>
> The standard technique for popular files -including MR job JAR files- is to
> over-replicate. One problem: how to determine what is popular without adding
> more load on the namenode
>
--
Bharath Ravi
Re: Load balancing requests in HDFS
Posted by Steve Loughran <st...@apache.org>.
On 16/10/11 02:53, Bharath Ravi wrote:
> Hi all,
>
> I have a question about how HDFS load balances requests for files/blocks:
>
> HDFS currently distributes data blocks randomly, for balance.
> However, if certain files/blocks are more popular than others, some nodes
> might get an "unfair" number of requests.
> Adding more replicas for these popular files might not help, unless HDFS
> explicitly distributes requests fairly among the replicas.
Have a look at the ReplicationTargetChooser class; it does take datanode
load into account, though it's concern is distribution for data
availability, not performance.
The standard technique for popular files -including MR job JAR files- is
to over-replicate. One problem: how to determine what is popular without
adding more load on the namenode
Re: Load balancing requests in HDFS
Posted by Bharath Ravi <bh...@gmail.com>.
Thanks a lot Harsh!
I'll look into this jira issue!
On 16 October 2011 08:15, Harsh J <ha...@cloudera.com> wrote:
> Bharath,
>
> You may want to take a look at
> http://issues.apache.org/jira/browse/HDFS-782 and continue the
> discussion on that.
>
> On Sun, Oct 16, 2011 at 7:23 AM, Bharath Ravi <bh...@gmail.com>
> wrote:
> > Hi all,
> >
> > I have a question about how HDFS load balances requests for files/blocks:
> >
> > HDFS currently distributes data blocks randomly, for balance.
> > However, if certain files/blocks are more popular than others, some nodes
> > might get an "unfair" number of requests.
> > Adding more replicas for these popular files might not help, unless HDFS
> > explicitly distributes requests fairly among the replicas.
> >
> > Looking around, I saw a bit of discussion on this dating to 2009,
> > Is something like this is done by HDFS today?
> >
> > http://www.mail-archive.com/core-user@hadoop.apache.org/msg04357.html
> >
> http://grokbase.com/p/hadoop.apache.org/common-dev/2009/02/re-hdfs-load-traffic-balancing/18en5o4z2j4b6eaqnemkp4eppd7q
> >
> > Thanks a lot in advance!
> > --
> > Bharath Ravi
> >
>
>
>
> --
> Harsh J
>
--
Bharath Ravi
Re: Load balancing requests in HDFS
Posted by Harsh J <ha...@cloudera.com>.
Bharath,
You may want to take a look at
http://issues.apache.org/jira/browse/HDFS-782 and continue the
discussion on that.
On Sun, Oct 16, 2011 at 7:23 AM, Bharath Ravi <bh...@gmail.com> wrote:
> Hi all,
>
> I have a question about how HDFS load balances requests for files/blocks:
>
> HDFS currently distributes data blocks randomly, for balance.
> However, if certain files/blocks are more popular than others, some nodes
> might get an "unfair" number of requests.
> Adding more replicas for these popular files might not help, unless HDFS
> explicitly distributes requests fairly among the replicas.
>
> Looking around, I saw a bit of discussion on this dating to 2009,
> Is something like this is done by HDFS today?
>
> http://www.mail-archive.com/core-user@hadoop.apache.org/msg04357.html
> http://grokbase.com/p/hadoop.apache.org/common-dev/2009/02/re-hdfs-load-traffic-balancing/18en5o4z2j4b6eaqnemkp4eppd7q
>
> Thanks a lot in advance!
> --
> Bharath Ravi
>
--
Harsh J