You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Bryan Beaudreault <bb...@hubspot.com.INVALID> on 2022/10/03 17:58:17 UTC
[DISCUSS] Release criteria for native TLS support in HBase RPC
Hi all,
The core work for TLS in the HBase RPC is mostly complete. With what's been
committed so far, one can connect end-to-end with TLS between client/server
and server/server. By default, we also enable mTLS (clients and servers
validate the certificate and hostname at handshake).
Here is a list of all TLS related work so far (finished and remaining):
https://issues.apache.org/jira/browse/HBASE-26666?jql=project%20%3D%20HBASE%20AND%20labels%20%3D%20tls
As we now have the basic functionality done, I wanted to discuss what the
release criteria should be. We had originally discussed releasing this in
2.6.0, which Andrew proposed tentatively planning for mid-December.
Beyond the code being well tested with unit tests, I've also deployed this
end-to-end in a basic test cluster in my company's environment. I deployed
it to an existing cluster in a rolling fashion based on the steps described
in Andor's documentation [1]. I will be out most of October, but when I
return in November I hope to start deploying this on some production
clusters after backporting to our main fork.
What else would people like to see before including in a release, and would
anyone be willing to give some testing a try themselves?
Re: [DISCUSS] Release criteria for native TLS support in HBase RPC
Posted by Andor Molnar <an...@apache.org>.
Hi Andrew / Bryan,
I can contribute to SASL vs TLS too, since we're running Kerberos-based
clusters in production which we would like to migrate to TLS and JWT
authentication.
This is going to be started some time this month and focusing on the
encryption part at the first place, because we probably won't use TLS
client auth.
Our new authentication scheme (JWT) will come to production in
November, so encryption + authentication will be tested with that vs
the current Kerberos setup.
To wrap up, the following benchmarks will be conducted in the following
2 months:
1. Kerberos with QoP auth-conf (encrypted) vs Kerberos auth (no
encryption + TLS
2. Kerberos auth-conf vs JWT + TLS
Regards,
Andor
On Thu, 2022-10-06 at 09:25 -0700, Andrew Purtell wrote:
> I can certainly contribute SASL vs TLS comparison as microbenchmarks
> and
> single system chaos testing. I have a VM image set up with Kerberos
> and
> related services to provide repeatable results, something I did
> recently
> for HBASE-27097.
>
> We also have in house tooling to deploy into k8s pods in a secure
> configuration that provides the SASL QoP I'd be looking for, so
> perhaps I
> can borrow some team bandwidth to try out an open source RC, but
> can't
> commit to that one.
>
>
> On Thu, Oct 6, 2022 at 4:39 AM Bryan Beaudreault
> <bb...@hubspot.com.invalid> wrote:
>
> > Thanks Andrew. I can probably provide some performance numbers for
> > the
> > following:
> >
> > 1. Comparing no protection vs native TLS. We would expect a
> > regression, and
> > might be able to compare that regression to what is known about
> > SASL.
> > 2. Comparing native TLS to my company's setup which uses haproxy as
> > a proxy
> > between client/server for SSL termination. This may not be
> > interesting
> > to everyone, but I will be doing it anyway so I could provide it.
> >
> > I can similarly try running ITBLL against TLS and no protection. We
> > aren't
> > set up for that in-house (especially expect issues getting the
> > killers to
> > work), but can try.
> >
> > We run java 11 exclusively on the server side, and a mix of java 11
> > and 17
> > on the client side.
> >
> > I do not have access to a SASL setup, and part of the point here is
> > that
> > SASL is so complicated to setup that TLS is an attractive
> > alternative. I
> > don't think I'll be able to provide a baseline for that, but maybe
> > someone else can? In terms of that requirement, I think if TLS is
> > considered performant and reliable enough, one could decide which
> > to use
> > based on how they want to setup their environment. Both are off by
> > default,
> > so it's a decision for them to evaluate based on the needs and
> > existing
> > ecosystem they're working in.
> >
> > I won't be doing any of this until November, and will be juggling a
> > few
> > other things at that time. Hopefully I should have something mid-
> > november.
> >
> > On Wed, Oct 5, 2022 at 9:00 PM Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > The test environment should be set up to support both SASL RPC
> > > with QoP
> > of
> > > "privacy" ("auth-conf") and TLS RPC. I believe we are going to
> > > want to
> > see
> > > most of all the differences between operation with SASL RPC and
> > > TLS RPC.
> > > These options both provide authentication and encryption. You'll
> > > probably
> > > also want to bench on Java 11 to capture the typical user
> > > experience.
> > >
> > > The top consideration is going to be stability: For all the
> > > testers out
> > > there, what is the difference in error rates between SASL and
> > > TLS? Is
> > there
> > > any difference? A related concern would be communication issues
> > > in
> > abnormal
> > > conditions. Should manifest as inability to recover from the
> > > usual chaos
> > > test, i.e. IntegrationTestBigLinkedList with slowDeterministic
> > > and/or
> > > serverKilling policy.
> > >
> > > The runner up is performance. For all the testers out there, if
> > > it would
> > be
> > > possible to share high level throughput and latency measures and
> > > their
> > > differences, that would be great. More detail in depth would be
> > > helpful
> > > too, such as per operation breakdown, if you have the numbers,
> > > but that
> > is
> > > not critical. My personal expectation is TLS RPC will perform
> > > better as
> > TLS
> > > has the benefit of modern investment.
> > >
> > > On Mon, Oct 3, 2022 at 11:43 AM Andor Molnar <an...@apache.org>
> > > wrote:
> > >
> > > > Hi Bryan,
> > > >
> > > > We (Cloudera) also planning to start deploying TLS-based
> > > > clusters to
> > > > production in early November. We'll do feature validation and
> > > > perf
> > > > benchmarks from our private fork in October.
> > > >
> > > > Regards,
> > > > Andor
> > > >
> > > >
> > > >
> > > > On Mon, 2022-10-03 at 13:58 -0400, Bryan Beaudreault wrote:
> > > > > Hi all,
> > > > >
> > > > > The core work for TLS in the HBase RPC is mostly complete.
> > > > > With
> > > > > what's been
> > > > > committed so far, one can connect end-to-end with TLS between
> > > > > client/server
> > > > > and server/server. By default, we also enable mTLS (clients
> > > > > and
> > > > > servers
> > > > > validate the certificate and hostname at handshake).
> > > > >
> > > > > Here is a list of all TLS related work so far (finished and
> > > > > remaining):
> > > > >
> > https://issues.apache.org/jira/browse/HBASE-26666?jql=project%20%3D%20HBASE%20AND%20labels%20%3D%20tls
> > > <
> > https://issues.apache.org/jira/browse/HBASE-26666?jql=project%20%3D%20HBASE%20AND%20labels%20%3D%20tls
> > > > > As we now have the basic functionality done, I wanted to
> > > > > discuss what
> > > > > the
> > > > > release criteria should be. We had originally discussed
> > > > > releasing
> > > > > this in
> > > > > 2.6.0, which Andrew proposed tentatively planning for mid-
> > > > > December.
> > > > >
> > > > > Beyond the code being well tested with unit tests, I've also
> > > > > deployed
> > > > > this
> > > > > end-to-end in a basic test cluster in my company's
> > > > > environment. I
> > > > > deployed
> > > > > it to an existing cluster in a rolling fashion based on the
> > > > > steps
> > > > > described
> > > > > in Andor's documentation [1]. I will be out most of October,
> > > > > but when
> > > > > I
> > > > > return in November I hope to start deploying this on some
> > > > > production
> > > > > clusters after backporting to our main fork.
> > > > >
> > > > > What else would people like to see before including in a
> > > > > release, and
> > > > > would
> > > > > anyone be willing to give some testing a try themselves?
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Unrest, ignorance distilled, nihilistic imbeciles -
> > > It's what we’ve earned
> > > Welcome, apocalypse, what’s taken you so long?
> > > Bring us the fitting end that we’ve been counting on
> > > - A23, Welcome, Apocalypse
> > >
>
>
Re: [DISCUSS] Release criteria for native TLS support in HBase RPC
Posted by Andrew Purtell <ap...@apache.org>.
I can certainly contribute SASL vs TLS comparison as microbenchmarks and
single system chaos testing. I have a VM image set up with Kerberos and
related services to provide repeatable results, something I did recently
for HBASE-27097.
We also have in house tooling to deploy into k8s pods in a secure
configuration that provides the SASL QoP I'd be looking for, so perhaps I
can borrow some team bandwidth to try out an open source RC, but can't
commit to that one.
On Thu, Oct 6, 2022 at 4:39 AM Bryan Beaudreault
<bb...@hubspot.com.invalid> wrote:
> Thanks Andrew. I can probably provide some performance numbers for the
> following:
>
> 1. Comparing no protection vs native TLS. We would expect a regression, and
> might be able to compare that regression to what is known about SASL.
> 2. Comparing native TLS to my company's setup which uses haproxy as a proxy
> between client/server for SSL termination. This may not be interesting
> to everyone, but I will be doing it anyway so I could provide it.
>
> I can similarly try running ITBLL against TLS and no protection. We aren't
> set up for that in-house (especially expect issues getting the killers to
> work), but can try.
>
> We run java 11 exclusively on the server side, and a mix of java 11 and 17
> on the client side.
>
> I do not have access to a SASL setup, and part of the point here is that
> SASL is so complicated to setup that TLS is an attractive alternative. I
> don't think I'll be able to provide a baseline for that, but maybe
> someone else can? In terms of that requirement, I think if TLS is
> considered performant and reliable enough, one could decide which to use
> based on how they want to setup their environment. Both are off by default,
> so it's a decision for them to evaluate based on the needs and existing
> ecosystem they're working in.
>
> I won't be doing any of this until November, and will be juggling a few
> other things at that time. Hopefully I should have something mid-november.
>
> On Wed, Oct 5, 2022 at 9:00 PM Andrew Purtell <ap...@apache.org> wrote:
>
> > The test environment should be set up to support both SASL RPC with QoP
> of
> > "privacy" ("auth-conf") and TLS RPC. I believe we are going to want to
> see
> > most of all the differences between operation with SASL RPC and TLS RPC.
> > These options both provide authentication and encryption. You'll probably
> > also want to bench on Java 11 to capture the typical user experience.
> >
> > The top consideration is going to be stability: For all the testers out
> > there, what is the difference in error rates between SASL and TLS? Is
> there
> > any difference? A related concern would be communication issues in
> abnormal
> > conditions. Should manifest as inability to recover from the usual chaos
> > test, i.e. IntegrationTestBigLinkedList with slowDeterministic and/or
> > serverKilling policy.
> >
> > The runner up is performance. For all the testers out there, if it would
> be
> > possible to share high level throughput and latency measures and their
> > differences, that would be great. More detail in depth would be helpful
> > too, such as per operation breakdown, if you have the numbers, but that
> is
> > not critical. My personal expectation is TLS RPC will perform better as
> TLS
> > has the benefit of modern investment.
> >
> > On Mon, Oct 3, 2022 at 11:43 AM Andor Molnar <an...@apache.org> wrote:
> >
> > > Hi Bryan,
> > >
> > > We (Cloudera) also planning to start deploying TLS-based clusters to
> > > production in early November. We'll do feature validation and perf
> > > benchmarks from our private fork in October.
> > >
> > > Regards,
> > > Andor
> > >
> > >
> > >
> > > On Mon, 2022-10-03 at 13:58 -0400, Bryan Beaudreault wrote:
> > > > Hi all,
> > > >
> > > > The core work for TLS in the HBase RPC is mostly complete. With
> > > > what's been
> > > > committed so far, one can connect end-to-end with TLS between
> > > > client/server
> > > > and server/server. By default, we also enable mTLS (clients and
> > > > servers
> > > > validate the certificate and hostname at handshake).
> > > >
> > > > Here is a list of all TLS related work so far (finished and
> > > > remaining):
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/HBASE-26666?jql=project%20%3D%20HBASE%20AND%20labels%20%3D%20tls
> > <
> https://issues.apache.org/jira/browse/HBASE-26666?jql=project%20%3D%20HBASE%20AND%20labels%20%3D%20tls
> >
> > > >
> > > > As we now have the basic functionality done, I wanted to discuss what
> > > > the
> > > > release criteria should be. We had originally discussed releasing
> > > > this in
> > > > 2.6.0, which Andrew proposed tentatively planning for mid-December.
> > > >
> > > > Beyond the code being well tested with unit tests, I've also deployed
> > > > this
> > > > end-to-end in a basic test cluster in my company's environment. I
> > > > deployed
> > > > it to an existing cluster in a rolling fashion based on the steps
> > > > described
> > > > in Andor's documentation [1]. I will be out most of October, but when
> > > > I
> > > > return in November I hope to start deploying this on some production
> > > > clusters after backporting to our main fork.
> > > >
> > > > What else would people like to see before including in a release, and
> > > > would
> > > > anyone be willing to give some testing a try themselves?
> > >
> > >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Unrest, ignorance distilled, nihilistic imbeciles -
> > It's what we’ve earned
> > Welcome, apocalypse, what’s taken you so long?
> > Bring us the fitting end that we’ve been counting on
> > - A23, Welcome, Apocalypse
> >
>
--
Best regards,
Andrew
Unrest, ignorance distilled, nihilistic imbeciles -
It's what we’ve earned
Welcome, apocalypse, what’s taken you so long?
Bring us the fitting end that we’ve been counting on
- A23, Welcome, Apocalypse
Re: [DISCUSS] Release criteria for native TLS support in HBase RPC
Posted by Bryan Beaudreault <bb...@hubspot.com.INVALID>.
Thanks Andrew. I can probably provide some performance numbers for the
following:
1. Comparing no protection vs native TLS. We would expect a regression, and
might be able to compare that regression to what is known about SASL.
2. Comparing native TLS to my company's setup which uses haproxy as a proxy
between client/server for SSL termination. This may not be interesting
to everyone, but I will be doing it anyway so I could provide it.
I can similarly try running ITBLL against TLS and no protection. We aren't
set up for that in-house (especially expect issues getting the killers to
work), but can try.
We run java 11 exclusively on the server side, and a mix of java 11 and 17
on the client side.
I do not have access to a SASL setup, and part of the point here is that
SASL is so complicated to setup that TLS is an attractive alternative. I
don't think I'll be able to provide a baseline for that, but maybe
someone else can? In terms of that requirement, I think if TLS is
considered performant and reliable enough, one could decide which to use
based on how they want to setup their environment. Both are off by default,
so it's a decision for them to evaluate based on the needs and existing
ecosystem they're working in.
I won't be doing any of this until November, and will be juggling a few
other things at that time. Hopefully I should have something mid-november.
On Wed, Oct 5, 2022 at 9:00 PM Andrew Purtell <ap...@apache.org> wrote:
> The test environment should be set up to support both SASL RPC with QoP of
> "privacy" ("auth-conf") and TLS RPC. I believe we are going to want to see
> most of all the differences between operation with SASL RPC and TLS RPC.
> These options both provide authentication and encryption. You'll probably
> also want to bench on Java 11 to capture the typical user experience.
>
> The top consideration is going to be stability: For all the testers out
> there, what is the difference in error rates between SASL and TLS? Is there
> any difference? A related concern would be communication issues in abnormal
> conditions. Should manifest as inability to recover from the usual chaos
> test, i.e. IntegrationTestBigLinkedList with slowDeterministic and/or
> serverKilling policy.
>
> The runner up is performance. For all the testers out there, if it would be
> possible to share high level throughput and latency measures and their
> differences, that would be great. More detail in depth would be helpful
> too, such as per operation breakdown, if you have the numbers, but that is
> not critical. My personal expectation is TLS RPC will perform better as TLS
> has the benefit of modern investment.
>
> On Mon, Oct 3, 2022 at 11:43 AM Andor Molnar <an...@apache.org> wrote:
>
> > Hi Bryan,
> >
> > We (Cloudera) also planning to start deploying TLS-based clusters to
> > production in early November. We'll do feature validation and perf
> > benchmarks from our private fork in October.
> >
> > Regards,
> > Andor
> >
> >
> >
> > On Mon, 2022-10-03 at 13:58 -0400, Bryan Beaudreault wrote:
> > > Hi all,
> > >
> > > The core work for TLS in the HBase RPC is mostly complete. With
> > > what's been
> > > committed so far, one can connect end-to-end with TLS between
> > > client/server
> > > and server/server. By default, we also enable mTLS (clients and
> > > servers
> > > validate the certificate and hostname at handshake).
> > >
> > > Here is a list of all TLS related work so far (finished and
> > > remaining):
> > >
> >
> https://issues.apache.org/jira/browse/HBASE-26666?jql=project%20%3D%20HBASE%20AND%20labels%20%3D%20tls
> <https://issues.apache.org/jira/browse/HBASE-26666?jql=project%20%3D%20HBASE%20AND%20labels%20%3D%20tls>
> > >
> > > As we now have the basic functionality done, I wanted to discuss what
> > > the
> > > release criteria should be. We had originally discussed releasing
> > > this in
> > > 2.6.0, which Andrew proposed tentatively planning for mid-December.
> > >
> > > Beyond the code being well tested with unit tests, I've also deployed
> > > this
> > > end-to-end in a basic test cluster in my company's environment. I
> > > deployed
> > > it to an existing cluster in a rolling fashion based on the steps
> > > described
> > > in Andor's documentation [1]. I will be out most of October, but when
> > > I
> > > return in November I hope to start deploying this on some production
> > > clusters after backporting to our main fork.
> > >
> > > What else would people like to see before including in a release, and
> > > would
> > > anyone be willing to give some testing a try themselves?
> >
> >
>
> --
> Best regards,
> Andrew
>
> Unrest, ignorance distilled, nihilistic imbeciles -
> It's what we’ve earned
> Welcome, apocalypse, what’s taken you so long?
> Bring us the fitting end that we’ve been counting on
> - A23, Welcome, Apocalypse
>
Re: [DISCUSS] Release criteria for native TLS support in HBase RPC
Posted by Andrew Purtell <ap...@apache.org>.
The test environment should be set up to support both SASL RPC with QoP of
"privacy" ("auth-conf") and TLS RPC. I believe we are going to want to see
most of all the differences between operation with SASL RPC and TLS RPC.
These options both provide authentication and encryption. You'll probably
also want to bench on Java 11 to capture the typical user experience.
The top consideration is going to be stability: For all the testers out
there, what is the difference in error rates between SASL and TLS? Is there
any difference? A related concern would be communication issues in abnormal
conditions. Should manifest as inability to recover from the usual chaos
test, i.e. IntegrationTestBigLinkedList with slowDeterministic and/or
serverKilling policy.
The runner up is performance. For all the testers out there, if it would be
possible to share high level throughput and latency measures and their
differences, that would be great. More detail in depth would be helpful
too, such as per operation breakdown, if you have the numbers, but that is
not critical. My personal expectation is TLS RPC will perform better as TLS
has the benefit of modern investment.
On Mon, Oct 3, 2022 at 11:43 AM Andor Molnar <an...@apache.org> wrote:
> Hi Bryan,
>
> We (Cloudera) also planning to start deploying TLS-based clusters to
> production in early November. We'll do feature validation and perf
> benchmarks from our private fork in October.
>
> Regards,
> Andor
>
>
>
> On Mon, 2022-10-03 at 13:58 -0400, Bryan Beaudreault wrote:
> > Hi all,
> >
> > The core work for TLS in the HBase RPC is mostly complete. With
> > what's been
> > committed so far, one can connect end-to-end with TLS between
> > client/server
> > and server/server. By default, we also enable mTLS (clients and
> > servers
> > validate the certificate and hostname at handshake).
> >
> > Here is a list of all TLS related work so far (finished and
> > remaining):
> >
> https://issues.apache.org/jira/browse/HBASE-26666?jql=project%20%3D%20HBASE%20AND%20labels%20%3D%20tls
> >
> > As we now have the basic functionality done, I wanted to discuss what
> > the
> > release criteria should be. We had originally discussed releasing
> > this in
> > 2.6.0, which Andrew proposed tentatively planning for mid-December.
> >
> > Beyond the code being well tested with unit tests, I've also deployed
> > this
> > end-to-end in a basic test cluster in my company's environment. I
> > deployed
> > it to an existing cluster in a rolling fashion based on the steps
> > described
> > in Andor's documentation [1]. I will be out most of October, but when
> > I
> > return in November I hope to start deploying this on some production
> > clusters after backporting to our main fork.
> >
> > What else would people like to see before including in a release, and
> > would
> > anyone be willing to give some testing a try themselves?
>
>
--
Best regards,
Andrew
Unrest, ignorance distilled, nihilistic imbeciles -
It's what we’ve earned
Welcome, apocalypse, what’s taken you so long?
Bring us the fitting end that we’ve been counting on
- A23, Welcome, Apocalypse
Re: [DISCUSS] Release criteria for native TLS support in HBase RPC
Posted by Andor Molnar <an...@apache.org>.
Hi Bryan,
We (Cloudera) also planning to start deploying TLS-based clusters to
production in early November. We'll do feature validation and perf
benchmarks from our private fork in October.
Regards,
Andor
On Mon, 2022-10-03 at 13:58 -0400, Bryan Beaudreault wrote:
> Hi all,
>
> The core work for TLS in the HBase RPC is mostly complete. With
> what's been
> committed so far, one can connect end-to-end with TLS between
> client/server
> and server/server. By default, we also enable mTLS (clients and
> servers
> validate the certificate and hostname at handshake).
>
> Here is a list of all TLS related work so far (finished and
> remaining):
> https://issues.apache.org/jira/browse/HBASE-26666?jql=project%20%3D%20HBASE%20AND%20labels%20%3D%20tls
>
> As we now have the basic functionality done, I wanted to discuss what
> the
> release criteria should be. We had originally discussed releasing
> this in
> 2.6.0, which Andrew proposed tentatively planning for mid-December.
>
> Beyond the code being well tested with unit tests, I've also deployed
> this
> end-to-end in a basic test cluster in my company's environment. I
> deployed
> it to an existing cluster in a rolling fashion based on the steps
> described
> in Andor's documentation [1]. I will be out most of October, but when
> I
> return in November I hope to start deploying this on some production
> clusters after backporting to our main fork.
>
> What else would people like to see before including in a release, and
> would
> anyone be willing to give some testing a try themselves?