You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Konstantin Shvachko <sh...@gmail.com> on 2018/12/15 02:16:23 UTC
[VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby
Hi Hadoop developers,
I would like to propose to merge to trunk the feature branch HDFS-12943 for
Consistent Reads from Standby Node. The feature is intended to scale read
RPC workloads. On large clusters reads comprise 95% of all RPCs to the
NameNode. We should be able to accommodate higher overall RPC workloads (up
to 4x by some estimates) by adding multiple ObserverNodes.
The main functionality has been implemented see sub-tasks of HDFS-12943.
We followed up with the test plan. Testing was done on two independent
clusters (see HDFS-14058 and HDFS-14059) with security enabled.
We ran standard HDFS commands, MR jobs, admin commands including manual
failover.
We know of one cluster running this feature in production.
Since the previous vote we addressed Daryn's concern (see HDFS-13873),
added documentation for the new feature, and fixed a few other jiras.
I attached a unified patch to the umbrella jira for the review.
Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
Thanks,
--Konstantin
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Chen Liang <va...@gmail.com>.
+1
Thanks Konstantin for driving the merge vote!
I have been working on the development and testing of the feature. It has
been running for several weeks on our cluster with ~100 nodes, which has HA
and Kerberos enabled. I have been able to run several different MapReduce
jobs and HDFS benchmarks (see HDFS-14058 for more detail). I feel confident
that the feature now has the completed functionality and is ready for merge
into trunk.
Chen
Konstantin Shvachko <sh...@gmail.com> 于2018年12月14日周五 下午6:24写道：
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
Re: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
Reads from Standby
Posted by Konstantin Shvachko <sh...@gmail.com>.
I just merged the feature branch into trunk.
Look for [SBN read] prefix in commits.
Thank you everybody for working on this.
--Konstantin
On Fri, Dec 21, 2018 at 6:34 PM Brahma Reddy Battula <
brahmareddy.battula@huawei.com> wrote:
> My late +1. Really it's useful feature.. Great work.
>
> -----Original Message-----
> From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
> Sent: Saturday, December 22, 2018 6:48 AM
> To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <
> hdfs-dev@hadoop.apache.org>
> Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
> Reads from Standby
>
> Obviously +1 from me.
>
> With four binding +1s, two non-binding +1s, and no -1s this vote passes.
> Thank you folks for working on the feature and for voting.
> Will do the merge in bit.
>
> Thanks,
> --Konst
>
> On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> > Hi Hadoop developers,
> >
> > I would like to propose to merge to trunk the feature branch
> > HDFS-12943 for Consistent Reads from Standby Node. The feature is
> > intended to scale read RPC workloads. On large clusters reads comprise
> > 95% of all RPCs to the NameNode. We should be able to accommodate
> > higher overall RPC workloads (up to 4x by some estimates) by adding
> multiple ObserverNodes.
> >
> > The main functionality has been implemented see sub-tasks of HDFS-12943.
> > We followed up with the test plan. Testing was done on two independent
> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > We ran standard HDFS commands, MR jobs, admin commands including
> > manual failover.
> > We know of one cluster running this feature in production.
> >
> > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > added documentation for the new feature, and fixed a few other jiras.
> >
> > I attached a unified patch to the umbrella jira for the review.
> > Please vote on this thread. The vote will run for 7 days until Wed Dec
> 21.
> >
> > Thanks,
> > --Konstantin
> >
>
Re: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
Reads from Standby
Posted by Konstantin Shvachko <sh...@gmail.com>.
I just merged the feature branch into trunk.
Look for [SBN read] prefix in commits.
Thank you everybody for working on this.
--Konstantin
On Fri, Dec 21, 2018 at 6:34 PM Brahma Reddy Battula <
brahmareddy.battula@huawei.com> wrote:
> My late +1. Really it's useful feature.. Great work.
>
> -----Original Message-----
> From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
> Sent: Saturday, December 22, 2018 6:48 AM
> To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <
> hdfs-dev@hadoop.apache.org>
> Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
> Reads from Standby
>
> Obviously +1 from me.
>
> With four binding +1s, two non-binding +1s, and no -1s this vote passes.
> Thank you folks for working on the feature and for voting.
> Will do the merge in bit.
>
> Thanks,
> --Konst
>
> On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> > Hi Hadoop developers,
> >
> > I would like to propose to merge to trunk the feature branch
> > HDFS-12943 for Consistent Reads from Standby Node. The feature is
> > intended to scale read RPC workloads. On large clusters reads comprise
> > 95% of all RPCs to the NameNode. We should be able to accommodate
> > higher overall RPC workloads (up to 4x by some estimates) by adding
> multiple ObserverNodes.
> >
> > The main functionality has been implemented see sub-tasks of HDFS-12943.
> > We followed up with the test plan. Testing was done on two independent
> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > We ran standard HDFS commands, MR jobs, admin commands including
> > manual failover.
> > We know of one cluster running this feature in production.
> >
> > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > added documentation for the new feature, and fixed a few other jiras.
> >
> > I attached a unified patch to the umbrella jira for the review.
> > Please vote on this thread. The vote will run for 7 days until Wed Dec
> 21.
> >
> > Thanks,
> > --Konstantin
> >
>
Re: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
Reads from Standby
Posted by Konstantin Shvachko <sh...@gmail.com>.
I just merged the feature branch into trunk.
Look for [SBN read] prefix in commits.
Thank you everybody for working on this.
--Konstantin
On Fri, Dec 21, 2018 at 6:34 PM Brahma Reddy Battula <
brahmareddy.battula@huawei.com> wrote:
> My late +1. Really it's useful feature.. Great work.
>
> -----Original Message-----
> From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
> Sent: Saturday, December 22, 2018 6:48 AM
> To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <
> hdfs-dev@hadoop.apache.org>
> Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
> Reads from Standby
>
> Obviously +1 from me.
>
> With four binding +1s, two non-binding +1s, and no -1s this vote passes.
> Thank you folks for working on the feature and for voting.
> Will do the merge in bit.
>
> Thanks,
> --Konst
>
> On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> > Hi Hadoop developers,
> >
> > I would like to propose to merge to trunk the feature branch
> > HDFS-12943 for Consistent Reads from Standby Node. The feature is
> > intended to scale read RPC workloads. On large clusters reads comprise
> > 95% of all RPCs to the NameNode. We should be able to accommodate
> > higher overall RPC workloads (up to 4x by some estimates) by adding
> multiple ObserverNodes.
> >
> > The main functionality has been implemented see sub-tasks of HDFS-12943.
> > We followed up with the test plan. Testing was done on two independent
> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > We ran standard HDFS commands, MR jobs, admin commands including
> > manual failover.
> > We know of one cluster running this feature in production.
> >
> > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > added documentation for the new feature, and fixed a few other jiras.
> >
> > I attached a unified patch to the umbrella jira for the review.
> > Please vote on this thread. The vote will run for 7 days until Wed Dec
> 21.
> >
> > Thanks,
> > --Konstantin
> >
>
Re: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
Reads from Standby
Posted by Konstantin Shvachko <sh...@gmail.com>.
I just merged the feature branch into trunk.
Look for [SBN read] prefix in commits.
Thank you everybody for working on this.
--Konstantin
On Fri, Dec 21, 2018 at 6:34 PM Brahma Reddy Battula <
brahmareddy.battula@huawei.com> wrote:
> My late +1. Really it's useful feature.. Great work.
>
> -----Original Message-----
> From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
> Sent: Saturday, December 22, 2018 6:48 AM
> To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <
> hdfs-dev@hadoop.apache.org>
> Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
> Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
> Reads from Standby
>
> Obviously +1 from me.
>
> With four binding +1s, two non-binding +1s, and no -1s this vote passes.
> Thank you folks for working on the feature and for voting.
> Will do the merge in bit.
>
> Thanks,
> --Konst
>
> On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> > Hi Hadoop developers,
> >
> > I would like to propose to merge to trunk the feature branch
> > HDFS-12943 for Consistent Reads from Standby Node. The feature is
> > intended to scale read RPC workloads. On large clusters reads comprise
> > 95% of all RPCs to the NameNode. We should be able to accommodate
> > higher overall RPC workloads (up to 4x by some estimates) by adding
> multiple ObserverNodes.
> >
> > The main functionality has been implemented see sub-tasks of HDFS-12943.
> > We followed up with the test plan. Testing was done on two independent
> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > We ran standard HDFS commands, MR jobs, admin commands including
> > manual failover.
> > We know of one cluster running this feature in production.
> >
> > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > added documentation for the new feature, and fixed a few other jiras.
> >
> > I attached a unified patch to the umbrella jira for the review.
> > Please vote on this thread. The vote will run for 7 days until Wed Dec
> 21.
> >
> > Thanks,
> > --Konstantin
> >
>
RE: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk -
Consistent Reads from Standby
Posted by Brahma Reddy Battula <br...@huawei.com>.
My late +1. Really it's useful feature.. Great work.
-----Original Message-----
From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
Sent: Saturday, December 22, 2018 6:48 AM
To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <hd...@hadoop.apache.org>
Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby
Obviously +1 from me.
With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.
Thanks,
--Konst
On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch
> HDFS-12943 for Consistent Reads from Standby Node. The feature is
> intended to scale read RPC workloads. On large clusters reads comprise
> 95% of all RPCs to the NameNode. We should be able to accommodate
> higher overall RPC workloads (up to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including
> manual failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
RE: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk -
Consistent Reads from Standby
Posted by Brahma Reddy Battula <br...@huawei.com>.
My late +1. Really it's useful feature.. Great work.
-----Original Message-----
From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
Sent: Saturday, December 22, 2018 6:48 AM
To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <hd...@hadoop.apache.org>
Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby
Obviously +1 from me.
With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.
Thanks,
--Konst
On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch
> HDFS-12943 for Consistent Reads from Standby Node. The feature is
> intended to scale read RPC workloads. On large clusters reads comprise
> 95% of all RPCs to the NameNode. We should be able to accommodate
> higher overall RPC workloads (up to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including
> manual failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
RE: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk -
Consistent Reads from Standby
Posted by Brahma Reddy Battula <br...@huawei.com>.
My late +1. Really it's useful feature.. Great work.
-----Original Message-----
From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
Sent: Saturday, December 22, 2018 6:48 AM
To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <hd...@hadoop.apache.org>
Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby
Obviously +1 from me.
With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.
Thanks,
--Konst
On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch
> HDFS-12943 for Consistent Reads from Standby Node. The feature is
> intended to scale read RPC workloads. On large clusters reads comprise
> 95% of all RPCs to the NameNode. We should be able to accommodate
> higher overall RPC workloads (up to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including
> manual failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
RE: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk -
Consistent Reads from Standby
Posted by Brahma Reddy Battula <br...@huawei.com>.
My late +1. Really it's useful feature.. Great work.
-----Original Message-----
From: Konstantin Shvachko [mailto:shv.hadoop@gmail.com]
Sent: Saturday, December 22, 2018 6:48 AM
To: Hadoop Common <co...@hadoop.apache.org>; hdfs-dev <hd...@hadoop.apache.org>
Cc: mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby
Obviously +1 from me.
With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.
Thanks,
--Konst
On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch
> HDFS-12943 for Consistent Reads from Standby Node. The feature is
> intended to scale read RPC workloads. On large clusters reads comprise
> 95% of all RPCs to the NameNode. We should be able to accommodate
> higher overall RPC workloads (up to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including
> manual failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
[Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
Reads from Standby
Posted by Konstantin Shvachko <sh...@gmail.com>.
Obviously +1 from me.
With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.
Thanks,
--Konst
On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943
> for Consistent Reads from Standby Node. The feature is intended to scale
> read RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Chen Liang <va...@gmail.com>.
+1
Thanks Konstantin for driving the merge vote!
I have been working on the development and testing of the feature. It has
been running for several weeks on our cluster with ~100 nodes, which has HA
and Kerberos enabled. I have been able to run several different MapReduce
jobs and HDFS benchmarks (see HDFS-14058 for more detail). I feel confident
that the feature now has the completed functionality and is ready for merge
into trunk.
Chen
Konstantin Shvachko <sh...@gmail.com> 于2018年12月14日周五 下午6:24写道：
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
[Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
Reads from Standby
Posted by Konstantin Shvachko <sh...@gmail.com>.
Obviously +1 from me.
With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.
Thanks,
--Konst
On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943
> for Consistent Reads from Standby Node. The feature is intended to scale
> read RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Chao Sun <su...@apache.org>.
+1 (no-binding)
At Uber we've deployed a subset of features covered in this JIRA in
production at multiple data centers, and it's being running great for a
while now. We are seeing huge benefits in terms of scaling our NameNode
throughput and providing better SLA guarantees for applications such as
Presto. We are very much looking forward to try out the full features once
this is merged into trunk.
Chao
On Tue, Dec 18, 2018 at 5:29 PM Jonathan Hung <jy...@gmail.com> wrote:
> +1!
>
> Jonathan Hung
>
>
> On Sat, Dec 15, 2018 at 8:26 AM Zhe Zhang <zh...@apache.org> wrote:
>
> > +1
> >
> > Thanks for addressing concerns from the previous vote.
> >
> > On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >
> > > Hi Hadoop developers,
> > >
> > > I would like to propose to merge to trunk the feature branch HDFS-12943
> > for
> > > Consistent Reads from Standby Node. The feature is intended to scale
> read
> > > RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> > > NameNode. We should be able to accommodate higher overall RPC workloads
> > (up
> > > to 4x by some estimates) by adding multiple ObserverNodes.
> > >
> > > The main functionality has been implemented see sub-tasks of
> HDFS-12943.
> > > We followed up with the test plan. Testing was done on two independent
> > > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > > We ran standard HDFS commands, MR jobs, admin commands including manual
> > > failover.
> > > We know of one cluster running this feature in production.
> > >
> > > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > > added documentation for the new feature, and fixed a few other jiras.
> > >
> > > I attached a unified patch to the umbrella jira for the review.
> > > Please vote on this thread. The vote will run for 7 days until Wed Dec
> > 21.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > --
> > Zhe Zhang
> > Apache Hadoop Committer
> > http://zhe-thoughts.github.io/about/ | @oldcap
> >
>
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Chao Sun <su...@apache.org>.
+1 (no-binding)
At Uber we've deployed a subset of features covered in this JIRA in
production at multiple data centers, and it's being running great for a
while now. We are seeing huge benefits in terms of scaling our NameNode
throughput and providing better SLA guarantees for applications such as
Presto. We are very much looking forward to try out the full features once
this is merged into trunk.
Chao
On Tue, Dec 18, 2018 at 5:29 PM Jonathan Hung <jy...@gmail.com> wrote:
> +1!
>
> Jonathan Hung
>
>
> On Sat, Dec 15, 2018 at 8:26 AM Zhe Zhang <zh...@apache.org> wrote:
>
> > +1
> >
> > Thanks for addressing concerns from the previous vote.
> >
> > On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >
> > > Hi Hadoop developers,
> > >
> > > I would like to propose to merge to trunk the feature branch HDFS-12943
> > for
> > > Consistent Reads from Standby Node. The feature is intended to scale
> read
> > > RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> > > NameNode. We should be able to accommodate higher overall RPC workloads
> > (up
> > > to 4x by some estimates) by adding multiple ObserverNodes.
> > >
> > > The main functionality has been implemented see sub-tasks of
> HDFS-12943.
> > > We followed up with the test plan. Testing was done on two independent
> > > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > > We ran standard HDFS commands, MR jobs, admin commands including manual
> > > failover.
> > > We know of one cluster running this feature in production.
> > >
> > > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > > added documentation for the new feature, and fixed a few other jiras.
> > >
> > > I attached a unified patch to the umbrella jira for the review.
> > > Please vote on this thread. The vote will run for 7 days until Wed Dec
> > 21.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > --
> > Zhe Zhang
> > Apache Hadoop Committer
> > http://zhe-thoughts.github.io/about/ | @oldcap
> >
>
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Chao Sun <su...@apache.org>.
+1 (no-binding)
At Uber we've deployed a subset of features covered in this JIRA in
production at multiple data centers, and it's being running great for a
while now. We are seeing huge benefits in terms of scaling our NameNode
throughput and providing better SLA guarantees for applications such as
Presto. We are very much looking forward to try out the full features once
this is merged into trunk.
Chao
On Tue, Dec 18, 2018 at 5:29 PM Jonathan Hung <jy...@gmail.com> wrote:
> +1!
>
> Jonathan Hung
>
>
> On Sat, Dec 15, 2018 at 8:26 AM Zhe Zhang <zh...@apache.org> wrote:
>
> > +1
> >
> > Thanks for addressing concerns from the previous vote.
> >
> > On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >
> > > Hi Hadoop developers,
> > >
> > > I would like to propose to merge to trunk the feature branch HDFS-12943
> > for
> > > Consistent Reads from Standby Node. The feature is intended to scale
> read
> > > RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> > > NameNode. We should be able to accommodate higher overall RPC workloads
> > (up
> > > to 4x by some estimates) by adding multiple ObserverNodes.
> > >
> > > The main functionality has been implemented see sub-tasks of
> HDFS-12943.
> > > We followed up with the test plan. Testing was done on two independent
> > > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > > We ran standard HDFS commands, MR jobs, admin commands including manual
> > > failover.
> > > We know of one cluster running this feature in production.
> > >
> > > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > > added documentation for the new feature, and fixed a few other jiras.
> > >
> > > I attached a unified patch to the umbrella jira for the review.
> > > Please vote on this thread. The vote will run for 7 days until Wed Dec
> > 21.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > --
> > Zhe Zhang
> > Apache Hadoop Committer
> > http://zhe-thoughts.github.io/about/ | @oldcap
> >
>
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Chao Sun <su...@apache.org>.
+1 (no-binding)
At Uber we've deployed a subset of features covered in this JIRA in
production at multiple data centers, and it's being running great for a
while now. We are seeing huge benefits in terms of scaling our NameNode
throughput and providing better SLA guarantees for applications such as
Presto. We are very much looking forward to try out the full features once
this is merged into trunk.
Chao
On Tue, Dec 18, 2018 at 5:29 PM Jonathan Hung <jy...@gmail.com> wrote:
> +1!
>
> Jonathan Hung
>
>
> On Sat, Dec 15, 2018 at 8:26 AM Zhe Zhang <zh...@apache.org> wrote:
>
> > +1
> >
> > Thanks for addressing concerns from the previous vote.
> >
> > On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >
> > > Hi Hadoop developers,
> > >
> > > I would like to propose to merge to trunk the feature branch HDFS-12943
> > for
> > > Consistent Reads from Standby Node. The feature is intended to scale
> read
> > > RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> > > NameNode. We should be able to accommodate higher overall RPC workloads
> > (up
> > > to 4x by some estimates) by adding multiple ObserverNodes.
> > >
> > > The main functionality has been implemented see sub-tasks of
> HDFS-12943.
> > > We followed up with the test plan. Testing was done on two independent
> > > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > > We ran standard HDFS commands, MR jobs, admin commands including manual
> > > failover.
> > > We know of one cluster running this feature in production.
> > >
> > > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > > added documentation for the new feature, and fixed a few other jiras.
> > >
> > > I attached a unified patch to the umbrella jira for the review.
> > > Please vote on this thread. The vote will run for 7 days until Wed Dec
> > 21.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > --
> > Zhe Zhang
> > Apache Hadoop Committer
> > http://zhe-thoughts.github.io/about/ | @oldcap
> >
>
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Jonathan Hung <jy...@gmail.com>.
+1!
Jonathan Hung
On Sat, Dec 15, 2018 at 8:26 AM Zhe Zhang <zh...@apache.org> wrote:
> +1
>
> Thanks for addressing concerns from the previous vote.
>
> On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> > Hi Hadoop developers,
> >
> > I would like to propose to merge to trunk the feature branch HDFS-12943
> for
> > Consistent Reads from Standby Node. The feature is intended to scale read
> > RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> > NameNode. We should be able to accommodate higher overall RPC workloads
> (up
> > to 4x by some estimates) by adding multiple ObserverNodes.
> >
> > The main functionality has been implemented see sub-tasks of HDFS-12943.
> > We followed up with the test plan. Testing was done on two independent
> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > We ran standard HDFS commands, MR jobs, admin commands including manual
> > failover.
> > We know of one cluster running this feature in production.
> >
> > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > added documentation for the new feature, and fixed a few other jiras.
> >
> > I attached a unified patch to the umbrella jira for the review.
> > Please vote on this thread. The vote will run for 7 days until Wed Dec
> 21.
> >
> > Thanks,
> > --Konstantin
> >
> --
> Zhe Zhang
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Jonathan Hung <jy...@gmail.com>.
+1!
Jonathan Hung
On Sat, Dec 15, 2018 at 8:26 AM Zhe Zhang <zh...@apache.org> wrote:
> +1
>
> Thanks for addressing concerns from the previous vote.
>
> On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> > Hi Hadoop developers,
> >
> > I would like to propose to merge to trunk the feature branch HDFS-12943
> for
> > Consistent Reads from Standby Node. The feature is intended to scale read
> > RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> > NameNode. We should be able to accommodate higher overall RPC workloads
> (up
> > to 4x by some estimates) by adding multiple ObserverNodes.
> >
> > The main functionality has been implemented see sub-tasks of HDFS-12943.
> > We followed up with the test plan. Testing was done on two independent
> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > We ran standard HDFS commands, MR jobs, admin commands including manual
> > failover.
> > We know of one cluster running this feature in production.
> >
> > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > added documentation for the new feature, and fixed a few other jiras.
> >
> > I attached a unified patch to the umbrella jira for the review.
> > Please vote on this thread. The vote will run for 7 days until Wed Dec
> 21.
> >
> > Thanks,
> > --Konstantin
> >
> --
> Zhe Zhang
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Jonathan Hung <jy...@gmail.com>.
+1!
Jonathan Hung
On Sat, Dec 15, 2018 at 8:26 AM Zhe Zhang <zh...@apache.org> wrote:
> +1
>
> Thanks for addressing concerns from the previous vote.
>
> On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> > Hi Hadoop developers,
> >
> > I would like to propose to merge to trunk the feature branch HDFS-12943
> for
> > Consistent Reads from Standby Node. The feature is intended to scale read
> > RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> > NameNode. We should be able to accommodate higher overall RPC workloads
> (up
> > to 4x by some estimates) by adding multiple ObserverNodes.
> >
> > The main functionality has been implemented see sub-tasks of HDFS-12943.
> > We followed up with the test plan. Testing was done on two independent
> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > We ran standard HDFS commands, MR jobs, admin commands including manual
> > failover.
> > We know of one cluster running this feature in production.
> >
> > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > added documentation for the new feature, and fixed a few other jiras.
> >
> > I attached a unified patch to the umbrella jira for the review.
> > Please vote on this thread. The vote will run for 7 days until Wed Dec
> 21.
> >
> > Thanks,
> > --Konstantin
> >
> --
> Zhe Zhang
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Erik Krogen <ek...@linkedin.com>.
+1 (non-binding) from me
I've been a part of the development of this feature throughout its existence, and have reviewed most of the code committed to the branch. The implementation has been conducted carefully and thoughtfully, and tested at a variety of sites. This feature has no potential negative impact if not enabled, so it does not present a risk to those who do not with to use it. I'm very much looking forward to seeing this be merged, and seeing how its usage can help to enable the community to scale the RPC workload of HDFS beyond what it can support today.
Erik
________________________________
From: Zhe Zhang <zh...@apache.org>
Sent: Saturday, December 15, 2018 8:26:14 AM
To: Konstantin Shvachko
Cc: Hadoop Common; hdfs-dev; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby
+1
Thanks for addressing concerns from the previous vote.
On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
--
Zhe Zhang
Apache Hadoop Committer
https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fzhe-thoughts.github.io%2Fabout%2F&data=02%7C01%7Cekrogen%40linkedin.com%7C641c95c0108143d5648708d662aa1374%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636804879950644875&sdata=Pez0NcOVzTI%2BpsvXXvlPzRTJq3SE2efD65zB5krtLak%3D&reserved=0 | @oldcap
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Jonathan Hung <jy...@gmail.com>.
+1!
Jonathan Hung
On Sat, Dec 15, 2018 at 8:26 AM Zhe Zhang <zh...@apache.org> wrote:
> +1
>
> Thanks for addressing concerns from the previous vote.
>
> On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> > Hi Hadoop developers,
> >
> > I would like to propose to merge to trunk the feature branch HDFS-12943
> for
> > Consistent Reads from Standby Node. The feature is intended to scale read
> > RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> > NameNode. We should be able to accommodate higher overall RPC workloads
> (up
> > to 4x by some estimates) by adding multiple ObserverNodes.
> >
> > The main functionality has been implemented see sub-tasks of HDFS-12943.
> > We followed up with the test plan. Testing was done on two independent
> > clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> > We ran standard HDFS commands, MR jobs, admin commands including manual
> > failover.
> > We know of one cluster running this feature in production.
> >
> > Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> > added documentation for the new feature, and fixed a few other jiras.
> >
> > I attached a unified patch to the umbrella jira for the review.
> > Please vote on this thread. The vote will run for 7 days until Wed Dec
> 21.
> >
> > Thanks,
> > --Konstantin
> >
> --
> Zhe Zhang
> Apache Hadoop Committer
> http://zhe-thoughts.github.io/about/ | @oldcap
>
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Erik Krogen <ek...@linkedin.com>.
+1 (non-binding) from me
I've been a part of the development of this feature throughout its existence, and have reviewed most of the code committed to the branch. The implementation has been conducted carefully and thoughtfully, and tested at a variety of sites. This feature has no potential negative impact if not enabled, so it does not present a risk to those who do not with to use it. I'm very much looking forward to seeing this be merged, and seeing how its usage can help to enable the community to scale the RPC workload of HDFS beyond what it can support today.
Erik
________________________________
From: Zhe Zhang <zh...@apache.org>
Sent: Saturday, December 15, 2018 8:26:14 AM
To: Konstantin Shvachko
Cc: Hadoop Common; hdfs-dev; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby
+1
Thanks for addressing concerns from the previous vote.
On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
--
Zhe Zhang
Apache Hadoop Committer
https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fzhe-thoughts.github.io%2Fabout%2F&data=02%7C01%7Cekrogen%40linkedin.com%7C641c95c0108143d5648708d662aa1374%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636804879950644875&sdata=Pez0NcOVzTI%2BpsvXXvlPzRTJq3SE2efD65zB5krtLak%3D&reserved=0 | @oldcap
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Zhe Zhang <zh...@apache.org>.
+1
Thanks for addressing concerns from the previous vote.
On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
--
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Zhe Zhang <zh...@apache.org>.
+1
Thanks for addressing concerns from the previous vote.
On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
--
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Zhe Zhang <zh...@apache.org>.
+1
Thanks for addressing concerns from the previous vote.
On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
--
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap
[Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
Reads from Standby
Posted by Konstantin Shvachko <sh...@gmail.com>.
Obviously +1 from me.
With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.
Thanks,
--Konst
On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943
> for Consistent Reads from Standby Node. The feature is intended to scale
> read RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
Re: [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads
from Standby
Posted by Zhe Zhang <zh...@apache.org>.
+1
Thanks for addressing concerns from the previous vote.
On Fri, Dec 14, 2018 at 6:24 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943 for
> Consistent Reads from Standby Node. The feature is intended to scale read
> RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>
--
Zhe Zhang
Apache Hadoop Committer
http://zhe-thoughts.github.io/about/ | @oldcap
[Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent
Reads from Standby
Posted by Konstantin Shvachko <sh...@gmail.com>.
Obviously +1 from me.
With four binding +1s, two non-binding +1s, and no -1s this vote passes.
Thank you folks for working on the feature and for voting.
Will do the merge in bit.
Thanks,
--Konst
On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hi Hadoop developers,
>
> I would like to propose to merge to trunk the feature branch HDFS-12943
> for Consistent Reads from Standby Node. The feature is intended to scale
> read RPC workloads. On large clusters reads comprise 95% of all RPCs to the
> NameNode. We should be able to accommodate higher overall RPC workloads (up
> to 4x by some estimates) by adding multiple ObserverNodes.
>
> The main functionality has been implemented see sub-tasks of HDFS-12943.
> We followed up with the test plan. Testing was done on two independent
> clusters (see HDFS-14058 and HDFS-14059) with security enabled.
> We ran standard HDFS commands, MR jobs, admin commands including manual
> failover.
> We know of one cluster running this feature in production.
>
> Since the previous vote we addressed Daryn's concern (see HDFS-13873),
> added documentation for the new feature, and fixed a few other jiras.
>
> I attached a unified patch to the umbrella jira for the review.
> Please vote on this thread. The vote will run for 7 days until Wed Dec 21.
>
> Thanks,
> --Konstantin
>