You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "张铎(Duo Zhang)" <pa...@gmail.com> on 2024/01/21 14:09:38 UTC

[DISCUSS] About RpcConnectionRegistry and security(authentication)

RpcConnectionRegistry was introduced in 2.5.0, and since it reduces
the load of zookeeper, I believe it has already been used by lots of
users.
We have already known that it can not work well with token(digest)
based authentication, and have already filed HBASE-25051 for it. and
recently, when fixing HBASE-28316, we found out that it could also be
broken if we choose to use different server principals for master and
region server, see HBASE-28321 for more details.
Since this is a very important feature and should have been widely
used among our users, I think we should fix these issues ASAP.
I've already worked on HBASE-25051 for a while and provided a workable
solution[1], so this time I took a deep thought and put up a design
doc[2], to address both HBASE-25051 and HBASE-28321.

Please take a look. Any comments here or on the design doc are welcomed.

Thanks.

1. https://github.com/apache/hbase/pull/5631
2. https://docs.google.com/document/d/1Cu-qzAdBGyBKM07aQP06RM0oeFSLPGtQFWuV_TDyBNg/edit?usp=sharing

Re: [DISCUSS] About RpcConnectionRegistry and security(authentication)

Posted by Charles Connell <cc...@hubspot.com.INVALID>.
I think that's good!

On Tue, Jan 23, 2024 at 10:51 PM 张铎(Duo Zhang) <pa...@gmail.com>
wrote:

> Thanks for the suggestion.
>
> We can do this check, in HBase we only have two roles to connect,
> either master or region server, the only problem here is that we do
> not know whether it is a master or region server, so after getting the
> server principal, we could check whether it matches the master pattern
> or the region server pattern, if both are not matched, we will fail
> the connection setup.
>
> WDYT?
>
> Thanks.
>
> Charles Connell <cc...@hubspot.com.invalid> 于2024年1月24日周三 08:40写道:
> >
> > Hi folks,
> >
> > I have experience enabling secure mode (Kerberos) for the HDFS layer on
> all
> > of Hubspot's HBase clusters. The pattern in the Hadoop project is for
> > clients to know what principal, or principal pattern, to expect a server
> to
> > present. For example, see `dfs.namenode.kerberos.principal.pattern` on
> >
> https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
> <https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml>
> .
> > Clients check the server's principal against the expected string or
> pattern
> > to make sure it matches. Ideally, using Kerberos would provide a
> > cryptographically verified guarantee to clients that they are connecting
> to
> > the right server and not an imposter who might have hijacked the right
> > server's domain name. I recommend that you add a property that the HBase
> > client checks against the principal name that the server presents during
> > the connection negotiation. If it doesn't match, the connection should
> > fail. As a backwards compatibility measure, the pattern can default to
> "*",
> > so it's really only used by users who care about this level of security.
> To
> > be honest, Hubspot does not use this security feature very much, but
> since
> > it seems to be standard in Hadoop, it would be nice to have it in HBase.
> >
> > Charles
> >
> > On Tue, Jan 23, 2024 at 9:00 AM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
> >
> > > Thanks Bryan.
> > >
> > > If no other concerns, let me at least re-implement the PR for
> > > HBASE-25051 based on the approach proposed here.
> > >
> > > Bryan Beaudreault <bb...@apache.org> 于2024年1月23日周二 21:40写道:
> > > >
> > > > Thanks for pulling this together Duo. I'll take a closer look at this
> > > after
> > > > I finish up the 2.6.0 release.
> > > >
> > > > To me the only possibly controversial part is:
> > > >
> > > > > For HBASE-28321, it should be part of our rpc negotiation, where
> the
> > > > server should return its server principal to the client, to let the
> > > client
> > > > acquire the necessary ticket for authentication. For me, I do not
> think
> > > > this will increase the security risk.
> > > >
> > > > I'm not a security or kerberos expert, so can't really speak to
> whether
> > > > that's safe without some deeper investigation. If we can get clarity
> on
> > > > that, then the rest seems mostly good.
> > > >
> > > > If anyone with kerberos security experience has opinions on that
> quote,
> > > > that'd be a big help to keep this moving forward.
> > > >
> > > > On Sun, Jan 21, 2024 at 9:10 AM 张铎(Duo Zhang) <palomino219@gmail.com
> >
> > > wrote:
> > > >
> > > > > RpcConnectionRegistry was introduced in 2.5.0, and since it reduces
> > > > > the load of zookeeper, I believe it has already been used by lots
> of
> > > > > users.
> > > > > We have already known that it can not work well with token(digest)
> > > > > based authentication, and have already filed HBASE-25051 for it.
> and
> > > > > recently, when fixing HBASE-28316, we found out that it could also
> be
> > > > > broken if we choose to use different server principals for master
> and
> > > > > region server, see HBASE-28321 for more details.
> > > > > Since this is a very important feature and should have been widely
> > > > > used among our users, I think we should fix these issues ASAP.
> > > > > I've already worked on HBASE-25051 for a while and provided a
> workable
> > > > > solution[1], so this time I took a deep thought and put up a design
> > > > > doc[2], to address both HBASE-25051 and HBASE-28321.
> > > > >
> > > > > Please take a look. Any comments here or on the design doc are
> > > welcomed.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > 1. https://github.com/apache/hbase/pull/5631
> <https://github.com/apache/hbase/pull/5631>
> > > <https://github.com/apache/hbase/pull/5631
> <https://github.com/apache/hbase/pull/5631>
> >
> > > > > 2.
> > > > >
> > >
> https://docs.google.com/document/d/1Cu-qzAdBGyBKM07aQP06RM0oeFSLPGtQFWuV_TDyBNg/edit?usp=sharing
> <https://docs.google.com/document/d/1Cu-qzAdBGyBKM07aQP06RM0oeFSLPGtQFWuV_TDyBNg/edit?usp=sharing>
> > > <
> https://docs.google.com/document/d/1Cu-qzAdBGyBKM07aQP06RM0oeFSLPGtQFWuV_TDyBNg/edit?usp=sharing
> <https://docs.google.com/document/d/1Cu-qzAdBGyBKM07aQP06RM0oeFSLPGtQFWuV_TDyBNg/edit?usp=sharing>
> >
> > > > >
> > >
>

Re: [DISCUSS] About RpcConnectionRegistry and security(authentication)

Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
Thanks for the suggestion.

We can do this check, in HBase we only have two roles to connect,
either master or region server, the only problem here is that we do
not know whether it is a master or region server, so after getting the
server principal, we could check whether it matches the master pattern
or the region server pattern, if both are not matched, we will fail
the connection setup.

WDYT?

Thanks.

Charles Connell <cc...@hubspot.com.invalid> 于2024年1月24日周三 08:40写道:
>
> Hi folks,
>
> I have experience enabling secure mode (Kerberos) for the HDFS layer on all
> of Hubspot's HBase clusters. The pattern in the Hadoop project is for
> clients to know what principal, or principal pattern, to expect a server to
> present. For example, see `dfs.namenode.kerberos.principal.pattern` on
> https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml.
> Clients check the server's principal against the expected string or pattern
> to make sure it matches. Ideally, using Kerberos would provide a
> cryptographically verified guarantee to clients that they are connecting to
> the right server and not an imposter who might have hijacked the right
> server's domain name. I recommend that you add a property that the HBase
> client checks against the principal name that the server presents during
> the connection negotiation. If it doesn't match, the connection should
> fail. As a backwards compatibility measure, the pattern can default to "*",
> so it's really only used by users who care about this level of security. To
> be honest, Hubspot does not use this security feature very much, but since
> it seems to be standard in Hadoop, it would be nice to have it in HBase.
>
> Charles
>
> On Tue, Jan 23, 2024 at 9:00 AM 张铎(Duo Zhang) <pa...@gmail.com> wrote:
>
> > Thanks Bryan.
> >
> > If no other concerns, let me at least re-implement the PR for
> > HBASE-25051 based on the approach proposed here.
> >
> > Bryan Beaudreault <bb...@apache.org> 于2024年1月23日周二 21:40写道:
> > >
> > > Thanks for pulling this together Duo. I'll take a closer look at this
> > after
> > > I finish up the 2.6.0 release.
> > >
> > > To me the only possibly controversial part is:
> > >
> > > > For HBASE-28321, it should be part of our rpc negotiation, where the
> > > server should return its server principal to the client, to let the
> > client
> > > acquire the necessary ticket for authentication. For me, I do not think
> > > this will increase the security risk.
> > >
> > > I'm not a security or kerberos expert, so can't really speak to whether
> > > that's safe without some deeper investigation. If we can get clarity on
> > > that, then the rest seems mostly good.
> > >
> > > If anyone with kerberos security experience has opinions on that quote,
> > > that'd be a big help to keep this moving forward.
> > >
> > > On Sun, Jan 21, 2024 at 9:10 AM 张铎(Duo Zhang) <pa...@gmail.com>
> > wrote:
> > >
> > > > RpcConnectionRegistry was introduced in 2.5.0, and since it reduces
> > > > the load of zookeeper, I believe it has already been used by lots of
> > > > users.
> > > > We have already known that it can not work well with token(digest)
> > > > based authentication, and have already filed HBASE-25051 for it. and
> > > > recently, when fixing HBASE-28316, we found out that it could also be
> > > > broken if we choose to use different server principals for master and
> > > > region server, see HBASE-28321 for more details.
> > > > Since this is a very important feature and should have been widely
> > > > used among our users, I think we should fix these issues ASAP.
> > > > I've already worked on HBASE-25051 for a while and provided a workable
> > > > solution[1], so this time I took a deep thought and put up a design
> > > > doc[2], to address both HBASE-25051 and HBASE-28321.
> > > >
> > > > Please take a look. Any comments here or on the design doc are
> > welcomed.
> > > >
> > > > Thanks.
> > > >
> > > > 1. https://github.com/apache/hbase/pull/5631
> > <https://github.com/apache/hbase/pull/5631>
> > > > 2.
> > > >
> > https://docs.google.com/document/d/1Cu-qzAdBGyBKM07aQP06RM0oeFSLPGtQFWuV_TDyBNg/edit?usp=sharing
> > <https://docs.google.com/document/d/1Cu-qzAdBGyBKM07aQP06RM0oeFSLPGtQFWuV_TDyBNg/edit?usp=sharing>
> > > >
> >

Re: [DISCUSS] About RpcConnectionRegistry and security(authentication)

Posted by Charles Connell <cc...@hubspot.com.INVALID>.
Hi folks,

I have experience enabling secure mode (Kerberos) for the HDFS layer on all
of Hubspot's HBase clusters. The pattern in the Hadoop project is for
clients to know what principal, or principal pattern, to expect a server to
present. For example, see `dfs.namenode.kerberos.principal.pattern` on
https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml.
Clients check the server's principal against the expected string or pattern
to make sure it matches. Ideally, using Kerberos would provide a
cryptographically verified guarantee to clients that they are connecting to
the right server and not an imposter who might have hijacked the right
server's domain name. I recommend that you add a property that the HBase
client checks against the principal name that the server presents during
the connection negotiation. If it doesn't match, the connection should
fail. As a backwards compatibility measure, the pattern can default to "*",
so it's really only used by users who care about this level of security. To
be honest, Hubspot does not use this security feature very much, but since
it seems to be standard in Hadoop, it would be nice to have it in HBase.

Charles

On Tue, Jan 23, 2024 at 9:00 AM 张铎(Duo Zhang) <pa...@gmail.com> wrote:

> Thanks Bryan.
>
> If no other concerns, let me at least re-implement the PR for
> HBASE-25051 based on the approach proposed here.
>
> Bryan Beaudreault <bb...@apache.org> 于2024年1月23日周二 21:40写道:
> >
> > Thanks for pulling this together Duo. I'll take a closer look at this
> after
> > I finish up the 2.6.0 release.
> >
> > To me the only possibly controversial part is:
> >
> > > For HBASE-28321, it should be part of our rpc negotiation, where the
> > server should return its server principal to the client, to let the
> client
> > acquire the necessary ticket for authentication. For me, I do not think
> > this will increase the security risk.
> >
> > I'm not a security or kerberos expert, so can't really speak to whether
> > that's safe without some deeper investigation. If we can get clarity on
> > that, then the rest seems mostly good.
> >
> > If anyone with kerberos security experience has opinions on that quote,
> > that'd be a big help to keep this moving forward.
> >
> > On Sun, Jan 21, 2024 at 9:10 AM 张铎(Duo Zhang) <pa...@gmail.com>
> wrote:
> >
> > > RpcConnectionRegistry was introduced in 2.5.0, and since it reduces
> > > the load of zookeeper, I believe it has already been used by lots of
> > > users.
> > > We have already known that it can not work well with token(digest)
> > > based authentication, and have already filed HBASE-25051 for it. and
> > > recently, when fixing HBASE-28316, we found out that it could also be
> > > broken if we choose to use different server principals for master and
> > > region server, see HBASE-28321 for more details.
> > > Since this is a very important feature and should have been widely
> > > used among our users, I think we should fix these issues ASAP.
> > > I've already worked on HBASE-25051 for a while and provided a workable
> > > solution[1], so this time I took a deep thought and put up a design
> > > doc[2], to address both HBASE-25051 and HBASE-28321.
> > >
> > > Please take a look. Any comments here or on the design doc are
> welcomed.
> > >
> > > Thanks.
> > >
> > > 1. https://github.com/apache/hbase/pull/5631
> <https://github.com/apache/hbase/pull/5631>
> > > 2.
> > >
> https://docs.google.com/document/d/1Cu-qzAdBGyBKM07aQP06RM0oeFSLPGtQFWuV_TDyBNg/edit?usp=sharing
> <https://docs.google.com/document/d/1Cu-qzAdBGyBKM07aQP06RM0oeFSLPGtQFWuV_TDyBNg/edit?usp=sharing>
> > >
>

Re: [DISCUSS] About RpcConnectionRegistry and security(authentication)

Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
Thanks Bryan.

If no other concerns, let me at least re-implement the PR for
HBASE-25051 based on the approach proposed here.

Bryan Beaudreault <bb...@apache.org> 于2024年1月23日周二 21:40写道:
>
> Thanks for pulling this together Duo. I'll take a closer look at this after
> I finish up the 2.6.0 release.
>
> To me the only possibly controversial part is:
>
> > For HBASE-28321, it should be part of our rpc negotiation, where the
> server should return its server principal to the client, to let the client
> acquire the necessary ticket for authentication. For me, I do not think
> this will increase the security risk.
>
> I'm not a security or kerberos expert, so can't really speak to whether
> that's safe without some deeper investigation. If we can get clarity on
> that, then the rest seems mostly good.
>
> If anyone with kerberos security experience has opinions on that quote,
> that'd be a big help to keep this moving forward.
>
> On Sun, Jan 21, 2024 at 9:10 AM 张铎(Duo Zhang) <pa...@gmail.com> wrote:
>
> > RpcConnectionRegistry was introduced in 2.5.0, and since it reduces
> > the load of zookeeper, I believe it has already been used by lots of
> > users.
> > We have already known that it can not work well with token(digest)
> > based authentication, and have already filed HBASE-25051 for it. and
> > recently, when fixing HBASE-28316, we found out that it could also be
> > broken if we choose to use different server principals for master and
> > region server, see HBASE-28321 for more details.
> > Since this is a very important feature and should have been widely
> > used among our users, I think we should fix these issues ASAP.
> > I've already worked on HBASE-25051 for a while and provided a workable
> > solution[1], so this time I took a deep thought and put up a design
> > doc[2], to address both HBASE-25051 and HBASE-28321.
> >
> > Please take a look. Any comments here or on the design doc are welcomed.
> >
> > Thanks.
> >
> > 1. https://github.com/apache/hbase/pull/5631
> > 2.
> > https://docs.google.com/document/d/1Cu-qzAdBGyBKM07aQP06RM0oeFSLPGtQFWuV_TDyBNg/edit?usp=sharing
> >

Re: [DISCUSS] About RpcConnectionRegistry and security(authentication)

Posted by Bryan Beaudreault <bb...@apache.org>.
Thanks for pulling this together Duo. I'll take a closer look at this after
I finish up the 2.6.0 release.

To me the only possibly controversial part is:

> For HBASE-28321, it should be part of our rpc negotiation, where the
server should return its server principal to the client, to let the client
acquire the necessary ticket for authentication. For me, I do not think
this will increase the security risk.

I'm not a security or kerberos expert, so can't really speak to whether
that's safe without some deeper investigation. If we can get clarity on
that, then the rest seems mostly good.

If anyone with kerberos security experience has opinions on that quote,
that'd be a big help to keep this moving forward.

On Sun, Jan 21, 2024 at 9:10 AM 张铎(Duo Zhang) <pa...@gmail.com> wrote:

> RpcConnectionRegistry was introduced in 2.5.0, and since it reduces
> the load of zookeeper, I believe it has already been used by lots of
> users.
> We have already known that it can not work well with token(digest)
> based authentication, and have already filed HBASE-25051 for it. and
> recently, when fixing HBASE-28316, we found out that it could also be
> broken if we choose to use different server principals for master and
> region server, see HBASE-28321 for more details.
> Since this is a very important feature and should have been widely
> used among our users, I think we should fix these issues ASAP.
> I've already worked on HBASE-25051 for a while and provided a workable
> solution[1], so this time I took a deep thought and put up a design
> doc[2], to address both HBASE-25051 and HBASE-28321.
>
> Please take a look. Any comments here or on the design doc are welcomed.
>
> Thanks.
>
> 1. https://github.com/apache/hbase/pull/5631
> 2.
> https://docs.google.com/document/d/1Cu-qzAdBGyBKM07aQP06RM0oeFSLPGtQFWuV_TDyBNg/edit?usp=sharing
>