You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ozone.apache.org by Duong Nguyen <du...@apache.org> on 2023/05/20 00:05:24 UTC
[DISCUSS] Proposal to merge Ozone feature branch HDDS-7733-Symmetric-Tokens to master
Dear Ozone Devs,
I would like to start this discussion thread for the proposal to merge
HDDS-7733-Symmetric-Tokens to master.
This feature branch contains the implementation to replace the costly token
signature generation using asymmetric (RSA) keys with symmetric key
algorithms, like HMAC with SHA256. Symmetric key algorithms bring a
much better performance and are the natural fit for Ozone token use case.
Yet, they require building a mechanism to generate, store, distribute, and
renew symmetric secret keys. That requirement is not trivial and has to be
split into smaller tasks that cannot be shipped individually. That is
the reason why the implementation of HDDS-7733
<https://issues.apache.org/jira/browse/HDDS-7733> happens in a separate
feature branch.
HDDS-7733 is not a new feature but an internal redesign for Ozone tokens to
improve OM performance/Ozone scalability. Apart from the existing
integration and acceptance tests which should already cover the usage of
tokens pretty well, we also added E2E test cases to cover the edge cases
related to the symmetric secret keys life-cycle, as per HDDS-8003
<https://issues.apache.org/jira/browse/HDDS-8003>.
More information can be found on the wiki page:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255070328
If there are no objections to the merge, we could start the official vote.
Thanks,
Duong
Re: [DISCUSS] Proposal to merge Ozone feature branch HDDS-7733-Symmetric-Tokens to master
Posted by Duong Nguyen <du...@cloudera.com.INVALID>.
Hi Sumit,
Thanks for reviewing this. Regarding your query:
1. Impact on existing/old clients: the block token is just a plain byte
array from the client's perspective. It's up to OM's and Datanode's logic
to create, serialize, deserialize and verify the token. This change is
compatible with older client versions.
2. Each generated SecretKey is packaged with the algorithm name and other
auditing data before getting stored and distributed over the network. OM
and Datanodes only rely on the algorithm packaged with the SecretKey to
sign/verify tokens. A change in the configured algorithm doesn't impact the
keys that are already generated.
Thanks,
Duong
On Sun, May 21, 2023 at 9:20 PM Sumit Agrawal
<su...@cloudera.com.invalid> wrote:
> Hi Duong,
>
> Thanks for working on this. I have gone to design docs and it looks good.
>
> Few query:
> 1. Upgrade from existing block token with async algo to symmetric
> algorithm:
> - any impact to existing/old clients?
>
> 2. When using block token configuration with HMACWithSha256 and later
> changed to HMACWithSha1, and restart SCM to take config in effect.
> - do this have any impact to existing token and DNs running with old and
> new token?
> Just to know if algorithm is upgraded in system, how do this have impact or
> do need any further changes to support this.
>
>
> Regards
> Sumit
>
> On Sat, May 20, 2023 at 5:35 AM Duong Nguyen <du...@apache.org> wrote:
>
> > Dear Ozone Devs,
> >
> > I would like to start this discussion thread for the proposal to merge
> > HDDS-7733-Symmetric-Tokens to master.
> >
> > This feature branch contains the implementation to replace the costly
> token
> > signature generation using asymmetric (RSA) keys with symmetric key
> > algorithms, like HMAC with SHA256. Symmetric key algorithms bring a
> > much better performance and are the natural fit for Ozone token use case.
> > Yet, they require building a mechanism to generate, store, distribute,
> and
> > renew symmetric secret keys. That requirement is not trivial and has to
> be
> > split into smaller tasks that cannot be shipped individually. That is
> > the reason why the implementation of HDDS-7733
> > <https://issues.apache.org/jira/browse/HDDS-7733> happens in a separate
> > feature branch.
> >
> > HDDS-7733 is not a new feature but an internal redesign for Ozone tokens
> to
> > improve OM performance/Ozone scalability. Apart from the existing
> > integration and acceptance tests which should already cover the usage of
> > tokens pretty well, we also added E2E test cases to cover the edge cases
> > related to the symmetric secret keys life-cycle, as per HDDS-8003
> > <https://issues.apache.org/jira/browse/HDDS-8003>.
> >
> > More information can be found on the wiki page:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255070328
> >
> > If there are no objections to the merge, we could start the official
> vote.
> >
> > Thanks,
> > Duong
> >
>
>
> --
> *Sumit Agrawal* | Senior Staff Engineer
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
>
Re: [DISCUSS] Proposal to merge Ozone feature branch HDDS-7733-Symmetric-Tokens to master
Posted by Duong Nguyen <du...@cloudera.com.INVALID>.
Hi Anu,
Thanks for reviewing the proposal.
Today, any actor joining the Ratis ring has to call an RPC the SCM leader
to add to the Raft group. In secure mode (Kerberos enabled), the RPC has a
layer to authenticate the client user during connection establishment. This
prevents unknown principals from joining the Raft group.
Thanks,
Duong
On Mon, May 22, 2023 at 11:27 AM anu engineer <an...@gmail.com>
wrote:
> I had a slightly orthogonal question: Do we need any special
> authentication mechanism today to join a Ratis ring ?
> It is probably not a significant question -- and the answer is that we
> are secure -- it was not obvious for me from reading code. Hence the
> question.
>
>
> Here is a possible attack vector:
>
> 1. Assume that it is possible to join the Ratis ring by simply
> pointing to the Ratis group -- say you have a ring, and all I need to
> do to join is some config file change -- or kill a machine and do an
> ARP spoofing.
> 2. At that point, will updateKeys send the keys to an attacker ?
> 4. In other words, does putting the Symmetric keys into a replication
> mode even make sense.
> 5. When SCM fails over, it can simply send its id across or ID in all
> packets -- and the clients can negotiate a key ?
>
> Thanks
> Anu
>
> On Sun, May 21, 2023 at 9:20 PM Sumit Agrawal
> <su...@cloudera.com.invalid> wrote:
> >
> > Hi Duong,
> >
> > Thanks for working on this. I have gone to design docs and it looks good.
> >
> > Few query:
> > 1. Upgrade from existing block token with async algo to symmetric
> > algorithm:
> > - any impact to existing/old clients?
> >
> > 2. When using block token configuration with HMACWithSha256 and later
> > changed to HMACWithSha1, and restart SCM to take config in effect.
> > - do this have any impact to existing token and DNs running with old and
> > new token?
> > Just to know if algorithm is upgraded in system, how do this have impact
> or
> > do need any further changes to support this.
> >
> >
> > Regards
> > Sumit
> >
> > On Sat, May 20, 2023 at 5:35 AM Duong Nguyen <du...@apache.org> wrote:
> >
> > > Dear Ozone Devs,
> > >
> > > I would like to start this discussion thread for the proposal to merge
> > > HDDS-7733-Symmetric-Tokens to master.
> > >
> > > This feature branch contains the implementation to replace the costly
> token
> > > signature generation using asymmetric (RSA) keys with symmetric key
> > > algorithms, like HMAC with SHA256. Symmetric key algorithms bring a
> > > much better performance and are the natural fit for Ozone token use
> case.
> > > Yet, they require building a mechanism to generate, store, distribute,
> and
> > > renew symmetric secret keys. That requirement is not trivial and has
> to be
> > > split into smaller tasks that cannot be shipped individually. That is
> > > the reason why the implementation of HDDS-7733
> > > <https://issues.apache.org/jira/browse/HDDS-7733> happens in a
> separate
> > > feature branch.
> > >
> > > HDDS-7733 is not a new feature but an internal redesign for Ozone
> tokens to
> > > improve OM performance/Ozone scalability. Apart from the existing
> > > integration and acceptance tests which should already cover the usage
> of
> > > tokens pretty well, we also added E2E test cases to cover the edge
> cases
> > > related to the symmetric secret keys life-cycle, as per HDDS-8003
> > > <https://issues.apache.org/jira/browse/HDDS-8003>.
> > >
> > > More information can be found on the wiki page:
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255070328
> > >
> > > If there are no objections to the merge, we could start the official
> vote.
> > >
> > > Thanks,
> > > Duong
> > >
> >
> >
> > --
> > *Sumit Agrawal* | Senior Staff Engineer
> > cloudera.com <https://www.cloudera.com>
> > [image: Cloudera] <https://www.cloudera.com/>
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera
> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > ------------------------------
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@ozone.apache.org
> For additional commands, e-mail: dev-help@ozone.apache.org
>
>
Re: [DISCUSS] Proposal to merge Ozone feature branch HDDS-7733-Symmetric-Tokens to master
Posted by anu engineer <an...@gmail.com>.
I had a slightly orthogonal question: Do we need any special
authentication mechanism today to join a Ratis ring ?
It is probably not a significant question -- and the answer is that we
are secure -- it was not obvious for me from reading code. Hence the
question.
Here is a possible attack vector:
1. Assume that it is possible to join the Ratis ring by simply
pointing to the Ratis group -- say you have a ring, and all I need to
do to join is some config file change -- or kill a machine and do an
ARP spoofing.
2. At that point, will updateKeys send the keys to an attacker ?
4. In other words, does putting the Symmetric keys into a replication
mode even make sense.
5. When SCM fails over, it can simply send its id across or ID in all
packets -- and the clients can negotiate a key ?
Thanks
Anu
On Sun, May 21, 2023 at 9:20 PM Sumit Agrawal
<su...@cloudera.com.invalid> wrote:
>
> Hi Duong,
>
> Thanks for working on this. I have gone to design docs and it looks good.
>
> Few query:
> 1. Upgrade from existing block token with async algo to symmetric
> algorithm:
> - any impact to existing/old clients?
>
> 2. When using block token configuration with HMACWithSha256 and later
> changed to HMACWithSha1, and restart SCM to take config in effect.
> - do this have any impact to existing token and DNs running with old and
> new token?
> Just to know if algorithm is upgraded in system, how do this have impact or
> do need any further changes to support this.
>
>
> Regards
> Sumit
>
> On Sat, May 20, 2023 at 5:35 AM Duong Nguyen <du...@apache.org> wrote:
>
> > Dear Ozone Devs,
> >
> > I would like to start this discussion thread for the proposal to merge
> > HDDS-7733-Symmetric-Tokens to master.
> >
> > This feature branch contains the implementation to replace the costly token
> > signature generation using asymmetric (RSA) keys with symmetric key
> > algorithms, like HMAC with SHA256. Symmetric key algorithms bring a
> > much better performance and are the natural fit for Ozone token use case.
> > Yet, they require building a mechanism to generate, store, distribute, and
> > renew symmetric secret keys. That requirement is not trivial and has to be
> > split into smaller tasks that cannot be shipped individually. That is
> > the reason why the implementation of HDDS-7733
> > <https://issues.apache.org/jira/browse/HDDS-7733> happens in a separate
> > feature branch.
> >
> > HDDS-7733 is not a new feature but an internal redesign for Ozone tokens to
> > improve OM performance/Ozone scalability. Apart from the existing
> > integration and acceptance tests which should already cover the usage of
> > tokens pretty well, we also added E2E test cases to cover the edge cases
> > related to the symmetric secret keys life-cycle, as per HDDS-8003
> > <https://issues.apache.org/jira/browse/HDDS-8003>.
> >
> > More information can be found on the wiki page:
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255070328
> >
> > If there are no objections to the merge, we could start the official vote.
> >
> > Thanks,
> > Duong
> >
>
>
> --
> *Sumit Agrawal* | Senior Staff Engineer
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> ------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ozone.apache.org
For additional commands, e-mail: dev-help@ozone.apache.org
Re: [DISCUSS] Proposal to merge Ozone feature branch HDDS-7733-Symmetric-Tokens to master
Posted by Sumit Agrawal <su...@cloudera.com.INVALID>.
Hi Duong,
Thanks for working on this. I have gone to design docs and it looks good.
Few query:
1. Upgrade from existing block token with async algo to symmetric
algorithm:
- any impact to existing/old clients?
2. When using block token configuration with HMACWithSha256 and later
changed to HMACWithSha1, and restart SCM to take config in effect.
- do this have any impact to existing token and DNs running with old and
new token?
Just to know if algorithm is upgraded in system, how do this have impact or
do need any further changes to support this.
Regards
Sumit
On Sat, May 20, 2023 at 5:35 AM Duong Nguyen <du...@apache.org> wrote:
> Dear Ozone Devs,
>
> I would like to start this discussion thread for the proposal to merge
> HDDS-7733-Symmetric-Tokens to master.
>
> This feature branch contains the implementation to replace the costly token
> signature generation using asymmetric (RSA) keys with symmetric key
> algorithms, like HMAC with SHA256. Symmetric key algorithms bring a
> much better performance and are the natural fit for Ozone token use case.
> Yet, they require building a mechanism to generate, store, distribute, and
> renew symmetric secret keys. That requirement is not trivial and has to be
> split into smaller tasks that cannot be shipped individually. That is
> the reason why the implementation of HDDS-7733
> <https://issues.apache.org/jira/browse/HDDS-7733> happens in a separate
> feature branch.
>
> HDDS-7733 is not a new feature but an internal redesign for Ozone tokens to
> improve OM performance/Ozone scalability. Apart from the existing
> integration and acceptance tests which should already cover the usage of
> tokens pretty well, we also added E2E test cases to cover the edge cases
> related to the symmetric secret keys life-cycle, as per HDDS-8003
> <https://issues.apache.org/jira/browse/HDDS-8003>.
>
> More information can be found on the wiki page:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255070328
>
> If there are no objections to the merge, we could start the official vote.
>
> Thanks,
> Duong
>
--
*Sumit Agrawal* | Senior Staff Engineer
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
Re: [DISCUSS] Proposal to merge Ozone feature branch HDDS-7733-Symmetric-Tokens to master
Posted by Uma Maheswara Rao Gangumalla <um...@gmail.com>.
With my memory, for technical code changes, committer vote should be
binding.
Since this is branch merge and it is code change, I would expect at least 3
committer votes to pass (and no -1s).
Regards,
Uma
On Wed, May 31, 2023 at 11:18 AM Wei-Chiu Chuang <we...@apache.org> wrote:
> What does it take to pass a vote? Does it require PMCs or can any
> contributors can vote?
> I checked around and don't think there's an Apache org-wide bylaws
> dictating branch merge votes. And we don't have bylaws for Ozone project
> itself.
>
> On Wed, May 31, 2023 at 11:08 AM Uma Maheswara Rao Gangumalla <
> umagangumalla@gmail.com> wrote:
>
> > Thank you Duong for the update. I don't have any objections to it.
> > If no further objections, probably let's start a vote thread in a day or
> > two?
> >
> > Regards,
> > Uma
> >
> > On Tue, May 30, 2023 at 5:10 PM Duong Nguyen <duong@cloudera.com.invalid
> >
> > wrote:
> >
> > > Thanks for the review, Uma.
> > >
> > > I've updated the JIRA title to remove DFSIO. We only want to test OM
> pure
> > > performance and DFSIO is not mandatory.
> > >
> > > Thanks,
> > > Duong
> > >
> > > On Tue, May 30, 2023 at 11:33 AM Uma Maheswara Rao Gangumalla <
> > > umagangumalla@gmail.com> wrote:
> > >
> > > > Overall it looks good to me.
> > > > I have a quick question regarding the Performance section in the
> > > checklist.
> > > > Your referred JIRA is saying DFSIO, but I see results of Freon and
> > > > FSPerfTest. Are you planning to take the DFSIO test as well? Or you
> may
> > > > want to update your JIRA title to reflect the results you posted
> from?
> > > >
> > > > Regards,
> > > > Uma
> > > >
> > > > On Fri, May 19, 2023 at 5:05 PM Duong Nguyen <du...@apache.org>
> wrote:
> > > >
> > > > > Dear Ozone Devs,
> > > > >
> > > > > I would like to start this discussion thread for the proposal to
> > merge
> > > > > HDDS-7733-Symmetric-Tokens to master.
> > > > >
> > > > > This feature branch contains the implementation to replace the
> costly
> > > > token
> > > > > signature generation using asymmetric (RSA) keys with symmetric key
> > > > > algorithms, like HMAC with SHA256. Symmetric key algorithms bring a
> > > > > much better performance and are the natural fit for Ozone token use
> > > case.
> > > > > Yet, they require building a mechanism to generate, store,
> > distribute,
> > > > and
> > > > > renew symmetric secret keys. That requirement is not trivial and
> has
> > to
> > > > be
> > > > > split into smaller tasks that cannot be shipped individually. That
> is
> > > > > the reason why the implementation of HDDS-7733
> > > > > <https://issues.apache.org/jira/browse/HDDS-7733> happens in a
> > > separate
> > > > > feature branch.
> > > > >
> > > > > HDDS-7733 is not a new feature but an internal redesign for Ozone
> > > tokens
> > > > to
> > > > > improve OM performance/Ozone scalability. Apart from the existing
> > > > > integration and acceptance tests which should already cover the
> usage
> > > of
> > > > > tokens pretty well, we also added E2E test cases to cover the edge
> > > cases
> > > > > related to the symmetric secret keys life-cycle, as per HDDS-8003
> > > > > <https://issues.apache.org/jira/browse/HDDS-8003>.
> > > > >
> > > > > More information can be found on the wiki page:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255070328
> > > > >
> > > > > If there are no objections to the merge, we could start the
> official
> > > > vote.
> > > > >
> > > > > Thanks,
> > > > > Duong
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Proposal to merge Ozone feature branch HDDS-7733-Symmetric-Tokens to master
Posted by Sumit Agrawal <su...@cloudera.com.INVALID>.
Hi Doung/Weichiu
@anu, IMO, ARP spoofing is not possible,
- Secret key is shared to *registered DN*, who is registered to SCM
securely (similar to private certificate of DN generated by SCM)
- Secret key validity is 7 days (configurable), so reduce impact of leak of
secret key.
SCM to client to DN-- its hash value generated using secret key for token,
so it is not possible to generate the same hash with tampering.
It's agreed that the shared key is more vulnerable than the Private/Public
Key pair approach, but considering the performance as required, the secret
key is much better. And considering the existing mechanism of sharing
secret keys, it seems safe to continue.
It looks good to me.
binding +1
Regards
Sumit
On Thu, Jun 1, 2023 at 12:01 AM Uma Maheswara Rao Gangumalla <
umagangumalla@gmail.com> wrote:
> This is hadoop bylaw:
>
> -
>
> Code Change
>
> A change made to a codebase of the project and committed by a committer.
> This includes source code, documentation, website content, etc.
>
> Consensus approval of active committers, but with a minimum of one +1.
> The code can be committed after the first +1, unless the code change
> represents a merge from a branch, in which case three +1s are required.
>
> https://hadoop.apache.org/bylaws.html
>
> Regards,
> Uma
>
> On Wed, May 31, 2023 at 11:29 AM Siddharth Wagle <sw...@apache.org>
> wrote:
>
> > @Wei-Chiu Chuang <we...@cloudera.com> AFAIK we adopted Hadoop bylaws.
> So
> > it should be 3 committer +1s, right?
> >
> > - Sid
> >
> > On Wed, May 31, 2023 at 11:17 AM Wei-Chiu Chuang <we...@apache.org>
> > wrote:
> >
> > > What does it take to pass a vote? Does it require PMCs or can any
> > > contributors can vote?
> > > I checked around and don't think there's an Apache org-wide bylaws
> > > dictating branch merge votes. And we don't have bylaws for Ozone
> project
> > > itself.
> > >
> > > On Wed, May 31, 2023 at 11:08 AM Uma Maheswara Rao Gangumalla <
> > > umagangumalla@gmail.com> wrote:
> > >
> > > > Thank you Duong for the update. I don't have any objections to it.
> > > > If no further objections, probably let's start a vote thread in a day
> > or
> > > > two?
> > > >
> > > > Regards,
> > > > Uma
> > > >
> > > > On Tue, May 30, 2023 at 5:10 PM Duong Nguyen
> > <duong@cloudera.com.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Thanks for the review, Uma.
> > > > >
> > > > > I've updated the JIRA title to remove DFSIO. We only want to test
> OM
> > > pure
> > > > > performance and DFSIO is not mandatory.
> > > > >
> > > > > Thanks,
> > > > > Duong
> > > > >
> > > > > On Tue, May 30, 2023 at 11:33 AM Uma Maheswara Rao Gangumalla <
> > > > > umagangumalla@gmail.com> wrote:
> > > > >
> > > > > > Overall it looks good to me.
> > > > > > I have a quick question regarding the Performance section in the
> > > > > checklist.
> > > > > > Your referred JIRA is saying DFSIO, but I see results of Freon
> and
> > > > > > FSPerfTest. Are you planning to take the DFSIO test as well? Or
> you
> > > may
> > > > > > want to update your JIRA title to reflect the results you posted
> > > from?
> > > > > >
> > > > > > Regards,
> > > > > > Uma
> > > > > >
> > > > > > On Fri, May 19, 2023 at 5:05 PM Duong Nguyen <du...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Dear Ozone Devs,
> > > > > > >
> > > > > > > I would like to start this discussion thread for the proposal
> to
> > > > merge
> > > > > > > HDDS-7733-Symmetric-Tokens to master.
> > > > > > >
> > > > > > > This feature branch contains the implementation to replace the
> > > costly
> > > > > > token
> > > > > > > signature generation using asymmetric (RSA) keys with symmetric
> > key
> > > > > > > algorithms, like HMAC with SHA256. Symmetric key algorithms
> > bring a
> > > > > > > much better performance and are the natural fit for Ozone token
> > use
> > > > > case.
> > > > > > > Yet, they require building a mechanism to generate, store,
> > > > distribute,
> > > > > > and
> > > > > > > renew symmetric secret keys. That requirement is not trivial
> and
> > > has
> > > > to
> > > > > > be
> > > > > > > split into smaller tasks that cannot be shipped individually.
> > That
> > > is
> > > > > > > the reason why the implementation of HDDS-7733
> > > > > > > <https://issues.apache.org/jira/browse/HDDS-7733> happens in a
> > > > > separate
> > > > > > > feature branch.
> > > > > > >
> > > > > > > HDDS-7733 is not a new feature but an internal redesign for
> Ozone
> > > > > tokens
> > > > > > to
> > > > > > > improve OM performance/Ozone scalability. Apart from the
> existing
> > > > > > > integration and acceptance tests which should already cover the
> > > usage
> > > > > of
> > > > > > > tokens pretty well, we also added E2E test cases to cover the
> > edge
> > > > > cases
> > > > > > > related to the symmetric secret keys life-cycle, as per
> HDDS-8003
> > > > > > > <https://issues.apache.org/jira/browse/HDDS-8003>.
> > > > > > >
> > > > > > > More information can be found on the wiki page:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255070328
> > > > > > >
> > > > > > > If there are no objections to the merge, we could start the
> > > official
> > > > > > vote.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Duong
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
*Sumit Agrawal* | Senior Staff Engineer
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
------------------------------
Re: [DISCUSS] Proposal to merge Ozone feature branch HDDS-7733-Symmetric-Tokens to master
Posted by Uma Maheswara Rao Gangumalla <um...@gmail.com>.
This is hadoop bylaw:
-
Code Change
A change made to a codebase of the project and committed by a committer.
This includes source code, documentation, website content, etc.
Consensus approval of active committers, but with a minimum of one +1.
The code can be committed after the first +1, unless the code change
represents a merge from a branch, in which case three +1s are required.
https://hadoop.apache.org/bylaws.html
Regards,
Uma
On Wed, May 31, 2023 at 11:29 AM Siddharth Wagle <sw...@apache.org> wrote:
> @Wei-Chiu Chuang <we...@cloudera.com> AFAIK we adopted Hadoop bylaws. So
> it should be 3 committer +1s, right?
>
> - Sid
>
> On Wed, May 31, 2023 at 11:17 AM Wei-Chiu Chuang <we...@apache.org>
> wrote:
>
> > What does it take to pass a vote? Does it require PMCs or can any
> > contributors can vote?
> > I checked around and don't think there's an Apache org-wide bylaws
> > dictating branch merge votes. And we don't have bylaws for Ozone project
> > itself.
> >
> > On Wed, May 31, 2023 at 11:08 AM Uma Maheswara Rao Gangumalla <
> > umagangumalla@gmail.com> wrote:
> >
> > > Thank you Duong for the update. I don't have any objections to it.
> > > If no further objections, probably let's start a vote thread in a day
> or
> > > two?
> > >
> > > Regards,
> > > Uma
> > >
> > > On Tue, May 30, 2023 at 5:10 PM Duong Nguyen
> <duong@cloudera.com.invalid
> > >
> > > wrote:
> > >
> > > > Thanks for the review, Uma.
> > > >
> > > > I've updated the JIRA title to remove DFSIO. We only want to test OM
> > pure
> > > > performance and DFSIO is not mandatory.
> > > >
> > > > Thanks,
> > > > Duong
> > > >
> > > > On Tue, May 30, 2023 at 11:33 AM Uma Maheswara Rao Gangumalla <
> > > > umagangumalla@gmail.com> wrote:
> > > >
> > > > > Overall it looks good to me.
> > > > > I have a quick question regarding the Performance section in the
> > > > checklist.
> > > > > Your referred JIRA is saying DFSIO, but I see results of Freon and
> > > > > FSPerfTest. Are you planning to take the DFSIO test as well? Or you
> > may
> > > > > want to update your JIRA title to reflect the results you posted
> > from?
> > > > >
> > > > > Regards,
> > > > > Uma
> > > > >
> > > > > On Fri, May 19, 2023 at 5:05 PM Duong Nguyen <du...@apache.org>
> > wrote:
> > > > >
> > > > > > Dear Ozone Devs,
> > > > > >
> > > > > > I would like to start this discussion thread for the proposal to
> > > merge
> > > > > > HDDS-7733-Symmetric-Tokens to master.
> > > > > >
> > > > > > This feature branch contains the implementation to replace the
> > costly
> > > > > token
> > > > > > signature generation using asymmetric (RSA) keys with symmetric
> key
> > > > > > algorithms, like HMAC with SHA256. Symmetric key algorithms
> bring a
> > > > > > much better performance and are the natural fit for Ozone token
> use
> > > > case.
> > > > > > Yet, they require building a mechanism to generate, store,
> > > distribute,
> > > > > and
> > > > > > renew symmetric secret keys. That requirement is not trivial and
> > has
> > > to
> > > > > be
> > > > > > split into smaller tasks that cannot be shipped individually.
> That
> > is
> > > > > > the reason why the implementation of HDDS-7733
> > > > > > <https://issues.apache.org/jira/browse/HDDS-7733> happens in a
> > > > separate
> > > > > > feature branch.
> > > > > >
> > > > > > HDDS-7733 is not a new feature but an internal redesign for Ozone
> > > > tokens
> > > > > to
> > > > > > improve OM performance/Ozone scalability. Apart from the existing
> > > > > > integration and acceptance tests which should already cover the
> > usage
> > > > of
> > > > > > tokens pretty well, we also added E2E test cases to cover the
> edge
> > > > cases
> > > > > > related to the symmetric secret keys life-cycle, as per HDDS-8003
> > > > > > <https://issues.apache.org/jira/browse/HDDS-8003>.
> > > > > >
> > > > > > More information can be found on the wiki page:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255070328
> > > > > >
> > > > > > If there are no objections to the merge, we could start the
> > official
> > > > > vote.
> > > > > >
> > > > > > Thanks,
> > > > > > Duong
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Proposal to merge Ozone feature branch HDDS-7733-Symmetric-Tokens to master
Posted by Siddharth Wagle <sw...@apache.org>.
@Wei-Chiu Chuang <we...@cloudera.com> AFAIK we adopted Hadoop bylaws. So
it should be 3 committer +1s, right?
- Sid
On Wed, May 31, 2023 at 11:17 AM Wei-Chiu Chuang <we...@apache.org> wrote:
> What does it take to pass a vote? Does it require PMCs or can any
> contributors can vote?
> I checked around and don't think there's an Apache org-wide bylaws
> dictating branch merge votes. And we don't have bylaws for Ozone project
> itself.
>
> On Wed, May 31, 2023 at 11:08 AM Uma Maheswara Rao Gangumalla <
> umagangumalla@gmail.com> wrote:
>
> > Thank you Duong for the update. I don't have any objections to it.
> > If no further objections, probably let's start a vote thread in a day or
> > two?
> >
> > Regards,
> > Uma
> >
> > On Tue, May 30, 2023 at 5:10 PM Duong Nguyen <duong@cloudera.com.invalid
> >
> > wrote:
> >
> > > Thanks for the review, Uma.
> > >
> > > I've updated the JIRA title to remove DFSIO. We only want to test OM
> pure
> > > performance and DFSIO is not mandatory.
> > >
> > > Thanks,
> > > Duong
> > >
> > > On Tue, May 30, 2023 at 11:33 AM Uma Maheswara Rao Gangumalla <
> > > umagangumalla@gmail.com> wrote:
> > >
> > > > Overall it looks good to me.
> > > > I have a quick question regarding the Performance section in the
> > > checklist.
> > > > Your referred JIRA is saying DFSIO, but I see results of Freon and
> > > > FSPerfTest. Are you planning to take the DFSIO test as well? Or you
> may
> > > > want to update your JIRA title to reflect the results you posted
> from?
> > > >
> > > > Regards,
> > > > Uma
> > > >
> > > > On Fri, May 19, 2023 at 5:05 PM Duong Nguyen <du...@apache.org>
> wrote:
> > > >
> > > > > Dear Ozone Devs,
> > > > >
> > > > > I would like to start this discussion thread for the proposal to
> > merge
> > > > > HDDS-7733-Symmetric-Tokens to master.
> > > > >
> > > > > This feature branch contains the implementation to replace the
> costly
> > > > token
> > > > > signature generation using asymmetric (RSA) keys with symmetric key
> > > > > algorithms, like HMAC with SHA256. Symmetric key algorithms bring a
> > > > > much better performance and are the natural fit for Ozone token use
> > > case.
> > > > > Yet, they require building a mechanism to generate, store,
> > distribute,
> > > > and
> > > > > renew symmetric secret keys. That requirement is not trivial and
> has
> > to
> > > > be
> > > > > split into smaller tasks that cannot be shipped individually. That
> is
> > > > > the reason why the implementation of HDDS-7733
> > > > > <https://issues.apache.org/jira/browse/HDDS-7733> happens in a
> > > separate
> > > > > feature branch.
> > > > >
> > > > > HDDS-7733 is not a new feature but an internal redesign for Ozone
> > > tokens
> > > > to
> > > > > improve OM performance/Ozone scalability. Apart from the existing
> > > > > integration and acceptance tests which should already cover the
> usage
> > > of
> > > > > tokens pretty well, we also added E2E test cases to cover the edge
> > > cases
> > > > > related to the symmetric secret keys life-cycle, as per HDDS-8003
> > > > > <https://issues.apache.org/jira/browse/HDDS-8003>.
> > > > >
> > > > > More information can be found on the wiki page:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255070328
> > > > >
> > > > > If there are no objections to the merge, we could start the
> official
> > > > vote.
> > > > >
> > > > > Thanks,
> > > > > Duong
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Proposal to merge Ozone feature branch HDDS-7733-Symmetric-Tokens to master
Posted by Wei-Chiu Chuang <we...@apache.org>.
What does it take to pass a vote? Does it require PMCs or can any
contributors can vote?
I checked around and don't think there's an Apache org-wide bylaws
dictating branch merge votes. And we don't have bylaws for Ozone project
itself.
On Wed, May 31, 2023 at 11:08 AM Uma Maheswara Rao Gangumalla <
umagangumalla@gmail.com> wrote:
> Thank you Duong for the update. I don't have any objections to it.
> If no further objections, probably let's start a vote thread in a day or
> two?
>
> Regards,
> Uma
>
> On Tue, May 30, 2023 at 5:10 PM Duong Nguyen <du...@cloudera.com.invalid>
> wrote:
>
> > Thanks for the review, Uma.
> >
> > I've updated the JIRA title to remove DFSIO. We only want to test OM pure
> > performance and DFSIO is not mandatory.
> >
> > Thanks,
> > Duong
> >
> > On Tue, May 30, 2023 at 11:33 AM Uma Maheswara Rao Gangumalla <
> > umagangumalla@gmail.com> wrote:
> >
> > > Overall it looks good to me.
> > > I have a quick question regarding the Performance section in the
> > checklist.
> > > Your referred JIRA is saying DFSIO, but I see results of Freon and
> > > FSPerfTest. Are you planning to take the DFSIO test as well? Or you may
> > > want to update your JIRA title to reflect the results you posted from?
> > >
> > > Regards,
> > > Uma
> > >
> > > On Fri, May 19, 2023 at 5:05 PM Duong Nguyen <du...@apache.org> wrote:
> > >
> > > > Dear Ozone Devs,
> > > >
> > > > I would like to start this discussion thread for the proposal to
> merge
> > > > HDDS-7733-Symmetric-Tokens to master.
> > > >
> > > > This feature branch contains the implementation to replace the costly
> > > token
> > > > signature generation using asymmetric (RSA) keys with symmetric key
> > > > algorithms, like HMAC with SHA256. Symmetric key algorithms bring a
> > > > much better performance and are the natural fit for Ozone token use
> > case.
> > > > Yet, they require building a mechanism to generate, store,
> distribute,
> > > and
> > > > renew symmetric secret keys. That requirement is not trivial and has
> to
> > > be
> > > > split into smaller tasks that cannot be shipped individually. That is
> > > > the reason why the implementation of HDDS-7733
> > > > <https://issues.apache.org/jira/browse/HDDS-7733> happens in a
> > separate
> > > > feature branch.
> > > >
> > > > HDDS-7733 is not a new feature but an internal redesign for Ozone
> > tokens
> > > to
> > > > improve OM performance/Ozone scalability. Apart from the existing
> > > > integration and acceptance tests which should already cover the usage
> > of
> > > > tokens pretty well, we also added E2E test cases to cover the edge
> > cases
> > > > related to the symmetric secret keys life-cycle, as per HDDS-8003
> > > > <https://issues.apache.org/jira/browse/HDDS-8003>.
> > > >
> > > > More information can be found on the wiki page:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255070328
> > > >
> > > > If there are no objections to the merge, we could start the official
> > > vote.
> > > >
> > > > Thanks,
> > > > Duong
> > > >
> > >
> >
>
Re: [DISCUSS] Proposal to merge Ozone feature branch HDDS-7733-Symmetric-Tokens to master
Posted by Uma Maheswara Rao Gangumalla <um...@gmail.com>.
Thank you Duong for the update. I don't have any objections to it.
If no further objections, probably let's start a vote thread in a day or
two?
Regards,
Uma
On Tue, May 30, 2023 at 5:10 PM Duong Nguyen <du...@cloudera.com.invalid>
wrote:
> Thanks for the review, Uma.
>
> I've updated the JIRA title to remove DFSIO. We only want to test OM pure
> performance and DFSIO is not mandatory.
>
> Thanks,
> Duong
>
> On Tue, May 30, 2023 at 11:33 AM Uma Maheswara Rao Gangumalla <
> umagangumalla@gmail.com> wrote:
>
> > Overall it looks good to me.
> > I have a quick question regarding the Performance section in the
> checklist.
> > Your referred JIRA is saying DFSIO, but I see results of Freon and
> > FSPerfTest. Are you planning to take the DFSIO test as well? Or you may
> > want to update your JIRA title to reflect the results you posted from?
> >
> > Regards,
> > Uma
> >
> > On Fri, May 19, 2023 at 5:05 PM Duong Nguyen <du...@apache.org> wrote:
> >
> > > Dear Ozone Devs,
> > >
> > > I would like to start this discussion thread for the proposal to merge
> > > HDDS-7733-Symmetric-Tokens to master.
> > >
> > > This feature branch contains the implementation to replace the costly
> > token
> > > signature generation using asymmetric (RSA) keys with symmetric key
> > > algorithms, like HMAC with SHA256. Symmetric key algorithms bring a
> > > much better performance and are the natural fit for Ozone token use
> case.
> > > Yet, they require building a mechanism to generate, store, distribute,
> > and
> > > renew symmetric secret keys. That requirement is not trivial and has to
> > be
> > > split into smaller tasks that cannot be shipped individually. That is
> > > the reason why the implementation of HDDS-7733
> > > <https://issues.apache.org/jira/browse/HDDS-7733> happens in a
> separate
> > > feature branch.
> > >
> > > HDDS-7733 is not a new feature but an internal redesign for Ozone
> tokens
> > to
> > > improve OM performance/Ozone scalability. Apart from the existing
> > > integration and acceptance tests which should already cover the usage
> of
> > > tokens pretty well, we also added E2E test cases to cover the edge
> cases
> > > related to the symmetric secret keys life-cycle, as per HDDS-8003
> > > <https://issues.apache.org/jira/browse/HDDS-8003>.
> > >
> > > More information can be found on the wiki page:
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255070328
> > >
> > > If there are no objections to the merge, we could start the official
> > vote.
> > >
> > > Thanks,
> > > Duong
> > >
> >
>
Re: [DISCUSS] Proposal to merge Ozone feature branch HDDS-7733-Symmetric-Tokens to master
Posted by Duong Nguyen <du...@cloudera.com.INVALID>.
Thanks for the review, Uma.
I've updated the JIRA title to remove DFSIO. We only want to test OM pure
performance and DFSIO is not mandatory.
Thanks,
Duong
On Tue, May 30, 2023 at 11:33 AM Uma Maheswara Rao Gangumalla <
umagangumalla@gmail.com> wrote:
> Overall it looks good to me.
> I have a quick question regarding the Performance section in the checklist.
> Your referred JIRA is saying DFSIO, but I see results of Freon and
> FSPerfTest. Are you planning to take the DFSIO test as well? Or you may
> want to update your JIRA title to reflect the results you posted from?
>
> Regards,
> Uma
>
> On Fri, May 19, 2023 at 5:05 PM Duong Nguyen <du...@apache.org> wrote:
>
> > Dear Ozone Devs,
> >
> > I would like to start this discussion thread for the proposal to merge
> > HDDS-7733-Symmetric-Tokens to master.
> >
> > This feature branch contains the implementation to replace the costly
> token
> > signature generation using asymmetric (RSA) keys with symmetric key
> > algorithms, like HMAC with SHA256. Symmetric key algorithms bring a
> > much better performance and are the natural fit for Ozone token use case.
> > Yet, they require building a mechanism to generate, store, distribute,
> and
> > renew symmetric secret keys. That requirement is not trivial and has to
> be
> > split into smaller tasks that cannot be shipped individually. That is
> > the reason why the implementation of HDDS-7733
> > <https://issues.apache.org/jira/browse/HDDS-7733> happens in a separate
> > feature branch.
> >
> > HDDS-7733 is not a new feature but an internal redesign for Ozone tokens
> to
> > improve OM performance/Ozone scalability. Apart from the existing
> > integration and acceptance tests which should already cover the usage of
> > tokens pretty well, we also added E2E test cases to cover the edge cases
> > related to the symmetric secret keys life-cycle, as per HDDS-8003
> > <https://issues.apache.org/jira/browse/HDDS-8003>.
> >
> > More information can be found on the wiki page:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255070328
> >
> > If there are no objections to the merge, we could start the official
> vote.
> >
> > Thanks,
> > Duong
> >
>
Re: [DISCUSS] Proposal to merge Ozone feature branch HDDS-7733-Symmetric-Tokens to master
Posted by Uma Maheswara Rao Gangumalla <um...@gmail.com>.
Overall it looks good to me.
I have a quick question regarding the Performance section in the checklist.
Your referred JIRA is saying DFSIO, but I see results of Freon and
FSPerfTest. Are you planning to take the DFSIO test as well? Or you may
want to update your JIRA title to reflect the results you posted from?
Regards,
Uma
On Fri, May 19, 2023 at 5:05 PM Duong Nguyen <du...@apache.org> wrote:
> Dear Ozone Devs,
>
> I would like to start this discussion thread for the proposal to merge
> HDDS-7733-Symmetric-Tokens to master.
>
> This feature branch contains the implementation to replace the costly token
> signature generation using asymmetric (RSA) keys with symmetric key
> algorithms, like HMAC with SHA256. Symmetric key algorithms bring a
> much better performance and are the natural fit for Ozone token use case.
> Yet, they require building a mechanism to generate, store, distribute, and
> renew symmetric secret keys. That requirement is not trivial and has to be
> split into smaller tasks that cannot be shipped individually. That is
> the reason why the implementation of HDDS-7733
> <https://issues.apache.org/jira/browse/HDDS-7733> happens in a separate
> feature branch.
>
> HDDS-7733 is not a new feature but an internal redesign for Ozone tokens to
> improve OM performance/Ozone scalability. Apart from the existing
> integration and acceptance tests which should already cover the usage of
> tokens pretty well, we also added E2E test cases to cover the edge cases
> related to the symmetric secret keys life-cycle, as per HDDS-8003
> <https://issues.apache.org/jira/browse/HDDS-8003>.
>
> More information can be found on the wiki page:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255070328
>
> If there are no objections to the merge, we could start the official vote.
>
> Thanks,
> Duong
>