You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Konstantin Shvachko <sh...@gmail.com> on 2018/12/22 01:18:08 UTC
[Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
Reads from Standby
Obviously +1 from me.
With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.
Thanks,
--Konst
On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943
> for Consistent Reads from Standby Node. The feature is intended to scale
> read RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
Re: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
Reads from Standby
Posted by Konstantin Shvachko <sh...@gmail.com>.
I just merged the feature branch into trunk.
Look for [SBN read] prefix in commits.
Thank you everybody for working on this.
--Konstantin
On Fri, Dec 21, 2018 at 6:34 PM Brahma Reddy Battula <
brahmareddy.battula@huawei.com> wrote:
> My late +1. Really it's useful feature.. Great work.
>
> -----Original Message-----
> From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
> Sent: Saturday, December 22, 2018 6:48 AM
> To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <
> hdfs-dev@hadoop.apache.org>
> Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
> Reads from Standby
>
> Obviously +1 from me.
>
> With four binding +1s, two non-binding +1s, and no -1s this vote passes.
> Thank you folks for working on the feature and for voting.
> Will do the merge in bit.
>
> Thanks,
> --Konst
>
> On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> > Hi Hadoop developers,
> >
> > I would like to propose to merge to trunk the feature branch
> > HDFS-12943 for Consistent Reads from Standby Node. The feature is
> > intended to scale read RPC workloads. On large clusters reads comprise
> > 95% of all RPCs to the NameNode. We should be able to accommodate
> > higher overall RPC workloads (up to 4x by some estimates) by adding
> multiple ObserverNodes.
> >
> > The main functionality has been implemented see sub-tasks of HDFS-12943.
> > We followed up with the test plan. Testing was done on two independent
> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > We ran standard HDFS commands, MR jobs, admin commands including
> > manual failover.
> > We know of one cluster running this feature in production.
> >
> > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > added documentation for the new feature, and fixed a few other jiras.
> >
> > I attached a unified patch to the umbrella jira for the review.
> > Please vote on this thread. The vote will run for 7 days until Wed Dec
> 21.
> >
> > Thanks,
> > --Konstantin
> >
>
Re: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
Reads from Standby
Posted by Konstantin Shvachko <sh...@gmail.com>.
I just merged the feature branch into trunk.
Look for [SBN read] prefix in commits.
Thank you everybody for working on this.
--Konstantin
On Fri, Dec 21, 2018 at 6:34 PM Brahma Reddy Battula <
brahmareddy.battula@huawei.com> wrote:
> My late +1. Really it's useful feature.. Great work.
>
> -----Original Message-----
> From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
> Sent: Saturday, December 22, 2018 6:48 AM
> To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <
> hdfs-dev@hadoop.apache.org>
> Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
> Reads from Standby
>
> Obviously +1 from me.
>
> With four binding +1s, two non-binding +1s, and no -1s this vote passes.
> Thank you folks for working on the feature and for voting.
> Will do the merge in bit.
>
> Thanks,
> --Konst
>
> On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> > Hi Hadoop developers,
> >
> > I would like to propose to merge to trunk the feature branch
> > HDFS-12943 for Consistent Reads from Standby Node. The feature is
> > intended to scale read RPC workloads. On large clusters reads comprise
> > 95% of all RPCs to the NameNode. We should be able to accommodate
> > higher overall RPC workloads (up to 4x by some estimates) by adding
> multiple ObserverNodes.
> >
> > The main functionality has been implemented see sub-tasks of HDFS-12943.
> > We followed up with the test plan. Testing was done on two independent
> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > We ran standard HDFS commands, MR jobs, admin commands including
> > manual failover.
> > We know of one cluster running this feature in production.
> >
> > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > added documentation for the new feature, and fixed a few other jiras.
> >
> > I attached a unified patch to the umbrella jira for the review.
> > Please vote on this thread. The vote will run for 7 days until Wed Dec
> 21.
> >
> > Thanks,
> > --Konstantin
> >
>
Re: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
Reads from Standby
Posted by Konstantin Shvachko <sh...@gmail.com>.
I just merged the feature branch into trunk.
Look for [SBN read] prefix in commits.
Thank you everybody for working on this.
--Konstantin
On Fri, Dec 21, 2018 at 6:34 PM Brahma Reddy Battula <
brahmareddy.battula@huawei.com> wrote:
> My late +1. Really it's useful feature.. Great work.
>
> -----Original Message-----
> From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
> Sent: Saturday, December 22, 2018 6:48 AM
> To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <
> hdfs-dev@hadoop.apache.org>
> Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
> Reads from Standby
>
> Obviously +1 from me.
>
> With four binding +1s, two non-binding +1s, and no -1s this vote passes.
> Thank you folks for working on the feature and for voting.
> Will do the merge in bit.
>
> Thanks,
> --Konst
>
> On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> > Hi Hadoop developers,
> >
> > I would like to propose to merge to trunk the feature branch
> > HDFS-12943 for Consistent Reads from Standby Node. The feature is
> > intended to scale read RPC workloads. On large clusters reads comprise
> > 95% of all RPCs to the NameNode. We should be able to accommodate
> > higher overall RPC workloads (up to 4x by some estimates) by adding
> multiple ObserverNodes.
> >
> > The main functionality has been implemented see sub-tasks of HDFS-12943.
> > We followed up with the test plan. Testing was done on two independent
> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > We ran standard HDFS commands, MR jobs, admin commands including
> > manual failover.
> > We know of one cluster running this feature in production.
> >
> > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > added documentation for the new feature, and fixed a few other jiras.
> >
> > I attached a unified patch to the umbrella jira for the review.
> > Please vote on this thread. The vote will run for 7 days until Wed Dec
> 21.
> >
> > Thanks,
> > --Konstantin
> >
>
Re: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
Reads from Standby
Posted by Konstantin Shvachko <sh...@gmail.com>.
I just merged the feature branch into trunk.
Look for [SBN read] prefix in commits.
Thank you everybody for working on this.
--Konstantin
On Fri, Dec 21, 2018 at 6:34 PM Brahma Reddy Battula <
brahmareddy.battula@huawei.com> wrote:
> My late +1. Really it's useful feature.. Great work.
>
> -----Original Message-----
> From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
> Sent: Saturday, December 22, 2018 6:48 AM
> To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <
> hdfs-dev@hadoop.apache.org>
> Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
> Reads from Standby
>
> Obviously +1 from me.
>
> With four binding +1s, two non-binding +1s, and no -1s this vote passes.
> Thank you folks for working on the feature and for voting.
> Will do the merge in bit.
>
> Thanks,
> --Konst
>
> On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> > Hi Hadoop developers,
> >
> > I would like to propose to merge to trunk the feature branch
> > HDFS-12943 for Consistent Reads from Standby Node. The feature is
> > intended to scale read RPC workloads. On large clusters reads comprise
> > 95% of all RPCs to the NameNode. We should be able to accommodate
> > higher overall RPC workloads (up to 4x by some estimates) by adding
> multiple ObserverNodes.
> >
> > The main functionality has been implemented see sub-tasks of HDFS-12943.
> > We followed up with the test plan. Testing was done on two independent
> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > We ran standard HDFS commands, MR jobs, admin commands including
> > manual failover.
> > We know of one cluster running this feature in production.
> >
> > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > added documentation for the new feature, and fixed a few other jiras.
> >
> > I attached a unified patch to the umbrella jira for the review.
> > Please vote on this thread. The vote will run for 7 days until Wed Dec
> 21.
> >
> > Thanks,
> > --Konstantin
> >
>
RE: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk -
Consistent Reads from Standby
Posted by Brahma Reddy Battula <br...@huawei.com>.
My late +1. Really it's useful feature.. Great work.
-----Original Message-----
From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
Sent: Saturday, December 22, 2018 6:48 AM
To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <hd...@hadoop.apache.org>
Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby
Obviously +1 from me.
With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.
Thanks,
--Konst
On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch
> HDFS-12943 for Consistent Reads from Standby Node. The feature is
> intended to scale read RPC workloads. On large clusters reads comprise
> 95% of all RPCs to the NameNode. We should be able to accommodate
> higher overall RPC workloads (up to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including
> manual failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
RE: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk -
Consistent Reads from Standby
Posted by Brahma Reddy Battula <br...@huawei.com>.
My late +1. Really it's useful feature.. Great work.
-----Original Message-----
From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
Sent: Saturday, December 22, 2018 6:48 AM
To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <hd...@hadoop.apache.org>
Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby
Obviously +1 from me.
With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.
Thanks,
--Konst
On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch
> HDFS-12943 for Consistent Reads from Standby Node. The feature is
> intended to scale read RPC workloads. On large clusters reads comprise
> 95% of all RPCs to the NameNode. We should be able to accommodate
> higher overall RPC workloads (up to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including
> manual failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
RE: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk -
Consistent Reads from Standby
Posted by Brahma Reddy Battula <br...@huawei.com>.
My late +1. Really it's useful feature.. Great work.
-----Original Message-----
From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
Sent: Saturday, December 22, 2018 6:48 AM
To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <hd...@hadoop.apache.org>
Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby
Obviously +1 from me.
With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.
Thanks,
--Konst
On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch
> HDFS-12943 for Consistent Reads from Standby Node. The feature is
> intended to scale read RPC workloads. On large clusters reads comprise
> 95% of all RPCs to the NameNode. We should be able to accommodate
> higher overall RPC workloads (up to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including
> manual failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
RE: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk -
Consistent Reads from Standby
Posted by Brahma Reddy Battula <br...@huawei.com>.
My late +1. Really it's useful feature.. Great work.
-----Original Message-----
From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
Sent: Saturday, December 22, 2018 6:48 AM
To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <hd...@hadoop.apache.org>
Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby
Obviously +1 from me.
With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.
Thanks,
--Konst
On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch
> HDFS-12943 for Consistent Reads from Standby Node. The feature is
> intended to scale read RPC workloads. On large clusters reads comprise
> 95% of all RPCs to the NameNode. We should be able to accommodate
> higher overall RPC workloads (up to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including
> manual failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>