You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Sanjay Deshmukh <sa...@gmail.com> on 2023/06/05 18:11:14 UTC
Impact from Enabling TLS
We've recently been required to enable TLS between our Accumulo clients and
the tablet servers. We've got it working, but we're experiencing a
significant performance impact. I'm running jstack on our client processes
and consistently seeing a number of Accumulo client threads in
sun.security.ssl.SSLSocketImpl.performInitialHandshake. This implies to me
that the Accumulo client is not reusing TLS sessions, and instead
establishing a new session for each connection. Am I reading
this correctly? Is there a way to configure the Accumulo client to reuse
TLS sessions?
Re: Impact from Enabling TLS
Posted by Christopher <ct...@apache.org>.
Connector is mostly stateless, so its lifecycle doesn't really matter.
The name "Connector" was specifically used instead of "Connection" to
try to disabuse users of the false notion that the object is related
to any active network connections. Connector is a "how to connect"
object, rather than a stateful "Connected" or "Connection" object. It
barely carries anything more than "connection string"-type information
(credentials, location of remote server endpoint). There is an initial
authentication check each time you construct a new Connector, which
could be an annoyance. But, if you can live with that, you can create
as many short-lived ones as you like. The issue with the handshakes
will be at a deeper level, with the RPC connection pools having too
small a size and being exhausted too quickly, or holding connections
that time out too quickly, or Thrift or Java is not being efficient
with stuff at an even lower layer than our connection pools.
In 2.x, we do have a more stateful client object, the AccumuloClient,
which is closeable. But this also does not maintain active
connections. It merely tries to provide a way to manage the lifecycle
of the client-side thread pools and caches needed for all the
operations the client is asked to do. For Connector, all those
resources are stored in global static JVM state and are harder to
close, but it also meant you were always reusing them, even if you
created many Connectors. In AccumuloClient, you can just close the
client to clean these resources up, so you may want to keep
AccumuloClient objects around longer than you would if they were
Connectors. But that's not going to impact your observations on 1.10.
Increasing general.rpc.timeout in your client.conf might help with the
handshakes you are seeing. I thought that option allowed the
connections in the RPC connection pool to live longer, but this is a
long shot... I'm unlikely to be correct on this, but you may want to
experiment with it anyway.
On Mon, Jun 5, 2023 at 10:39 PM Sanjay Deshmukh <sa...@gmail.com> wrote:
>
> I guess I should clarify. What’s the intended lifecycle of a Scanner or BatchScanner vs a Connector? Is it intended to have a single long lived Connector from which a Scanner or BatchScanner is created and closed per request? Or is a Scanner or BatchScanner (or a pool of them) meant to be long lived and serve multiple subsequent requests?
>
> Sent from my iPhone
>
> On Jun 5, 2023, at 15:14, Christopher <ct...@apache.org> wrote:
>
>
> In Accumulo code, we do try to reuse thrift connections, but I'm not sure if the thrift code itself is reusing TLS sessions or not.
>
> Keep in mind that Accumulo is a distributed system. Many of these handshakes might be going to different servers in a cluster. Are you able to tell if that is the case? It might be possible to increase session timeouts or the number of cached sessions, but the best case is that you'll still need to do a handshake with each server your client talks to.
>
> It might also help if you told us the version of Accumulo you are using, and how you are using the client code (like, are you reusing client objects). If you can share a bit of the jstack with us, that might also be helpful if we need to troubleshoot further.
>
> On Mon, Jun 5, 2023, 14:12 Sanjay Deshmukh <sa...@gmail.com> wrote:
>>
>> We've recently been required to enable TLS between our Accumulo clients and the tablet servers. We've got it working, but we're experiencing a significant performance impact. I'm running jstack on our client processes and consistently seeing a number of Accumulo client threads in sun.security.ssl.SSLSocketImpl.performInitialHandshake. This implies to me that the Accumulo client is not reusing TLS sessions, and instead establishing a new session for each connection. Am I reading this correctly? Is there a way to configure the Accumulo client to reuse TLS sessions?
Re: Impact from Enabling TLS
Posted by Sanjay Deshmukh <sa...@gmail.com>.
I guess I should clarify. What’s the intended lifecycle of a Scanner or
BatchScanner vs a Connector? Is it intended to have a single long lived
Connector from which a Scanner or BatchScanner is created and closed per
request? Or is a Scanner or BatchScanner (or a pool of them) meant to be long
lived and serve multiple subsequent requests?
Sent from my iPhone
> On Jun 5, 2023, at 15:14, Christopher <ct...@apache.org> wrote:
>
>
>
>
> In Accumulo code, we do try to reuse thrift connections, but I'm not sure if
> the thrift code itself is reusing TLS sessions or not.
>
>
>
>
> Keep in mind that Accumulo is a distributed system. Many of these handshakes
> might be going to different servers in a cluster. Are you able to tell if
> that is the case? It might be possible to increase session timeouts or the
> number of cached sessions, but the best case is that you'll still need to do
> a handshake with each server your client talks to.
>
>
>
>
> It might also help if you told us the version of Accumulo you are using, and
> how you are using the client code (like, are you reusing client objects). If
> you can share a bit of the jstack with us, that might also be helpful if we
> need to troubleshoot further.
>
>
>
> On Mon, Jun 5, 2023, 14:12 Sanjay Deshmukh
> <[sanjayd@gmail.com](mailto:sanjayd@gmail.com)> wrote:
>
>
>> We've recently been required to enable TLS between our Accumulo clients and
the tablet servers. We've got it working, but we're experiencing a significant
performance impact. I'm running jstack on our client processes and
consistently seeing a number of Accumulo client threads in
sun.security.ssl.SSLSocketImpl.performInitialHandshake. This implies to me
that the Accumulo client is not reusing TLS sessions, and instead establishing
a new session for each connection. Am I reading this correctly? Is there a way
to configure the Accumulo client to reuse TLS sessions?
Re: Impact from Enabling TLS
Posted by Sanjay Deshmukh <sa...@gmail.com>.
I don’t think so. We’re actually still on Java 8, and that bug was introduced
in Java 9.
> On Jun 5, 2023, at 16:31, dlmarion@comcast.net wrote:
>
>
>
>
> Does this apply? <https://bugs.openjdk.org/browse/JDK-8190917>
>
>
>
> **From:** Sanjay Deshmukh <sa...@gmail.com>
> **Sent:** Monday, June 5, 2023 3:28 PM
> **To:** user@accumulo.apache.org
> **Subject:** Re: Impact from Enabling TLS
>
>
>
> Sure thing. We're running Accumulo 1.10.1. Our configuration has 15 tablet
> servers and a client pool of 4 threads that could be doing scans or batch
> scans at any given time. I wish I could copy over the exact jstack but
> unfortunately it's coming from an offline network, but I do see the same
> tablet servers in the thread names repeatedly (e.g. "batch scanner X- 4
> looking up 1 ranges at tabletservername:9997", implying it it's not an
> infrequent use problem. In some cases I even see multiple threads trying to
> handshake with the same tablet server. The jstack here seems to go from
> TabletServerbatchReaderIterator.doLookup down through TFramedTransport.flush
> and into SSLSocketImpl.performInitialHandshake.
>
>
>
> The lifecycle here is that each of these client threads is using a single
> shared instance of Connector, and instantiating a new Scanner or
> BatchScanner per use, and closing it when that use is done. There are many
> of these uses over time.
>
>
>
> On Mon, Jun 5, 2023 at 3:14 PM Christopher
> <[ctubbsii@apache.org](mailto:ctubbsii@apache.org)> wrote:
>
>> In Accumulo code, we do try to reuse thrift connections, but I'm not sure
if the thrift code itself is reusing TLS sessions or not.
>>
>>
>>
>> Keep in mind that Accumulo is a distributed system. Many of these
handshakes might be going to different servers in a cluster. Are you able to
tell if that is the case? It might be possible to increase session timeouts or
the number of cached sessions, but the best case is that you'll still need to
do a handshake with each server your client talks to.
>>
>>
>>
>> It might also help if you told us the version of Accumulo you are using,
and how you are using the client code (like, are you reusing client objects).
If you can share a bit of the jstack with us, that might also be helpful if we
need to troubleshoot further.
>>
>> On Mon, Jun 5, 2023, 14:12 Sanjay Deshmukh
<[sanjayd@gmail.com](mailto:sanjayd@gmail.com)> wrote:
>>
>>> We've recently been required to enable TLS between our Accumulo clients
and the tablet servers. We've got it working, but we're experiencing a
significant performance impact. I'm running jstack on our client processes and
consistently seeing a number of Accumulo client threads in
sun.security.ssl.SSLSocketImpl.performInitialHandshake. This implies to me
that the Accumulo client is not reusing TLS sessions, and instead establishing
a new session for each connection. Am I reading this correctly? Is there a way
to configure the Accumulo client to reuse TLS sessions?
>
>
>
>
>
>
> \--
>
> Sanjay Deshmukh
> [sanjayd@gmail.com](mailto:sanjayd@gmail.com)
RE: Impact from Enabling TLS
Posted by dl...@comcast.net.
Does this apply? https://bugs.openjdk.org/browse/JDK-8190917
From: Sanjay Deshmukh <sa...@gmail.com>
Sent: Monday, June 5, 2023 3:28 PM
To: user@accumulo.apache.org
Subject: Re: Impact from Enabling TLS
Sure thing. We're running Accumulo 1.10.1. Our configuration has 15 tablet servers and a client pool of 4 threads that could be doing scans or batch scans at any given time. I wish I could copy over the exact jstack but unfortunately it's coming from an offline network, but I do see the same tablet servers in the thread names repeatedly (e.g. "batch scanner X- 4 looking up 1 ranges at tabletservername:9997", implying it it's not an infrequent use problem. In some cases I even see multiple threads trying to handshake with the same tablet server. The jstack here seems to go from TabletServerbatchReaderIterator.doLookup down through TFramedTransport.flush and into SSLSocketImpl.performInitialHandshake.
The lifecycle here is that each of these client threads is using a single shared instance of Connector, and instantiating a new Scanner or BatchScanner per use, and closing it when that use is done. There are many of these uses over time.
On Mon, Jun 5, 2023 at 3:14 PM Christopher <ctubbsii@apache.org <ma...@apache.org> > wrote:
In Accumulo code, we do try to reuse thrift connections, but I'm not sure if the thrift code itself is reusing TLS sessions or not.
Keep in mind that Accumulo is a distributed system. Many of these handshakes might be going to different servers in a cluster. Are you able to tell if that is the case? It might be possible to increase session timeouts or the number of cached sessions, but the best case is that you'll still need to do a handshake with each server your client talks to.
It might also help if you told us the version of Accumulo you are using, and how you are using the client code (like, are you reusing client objects). If you can share a bit of the jstack with us, that might also be helpful if we need to troubleshoot further.
On Mon, Jun 5, 2023, 14:12 Sanjay Deshmukh <sanjayd@gmail.com <ma...@gmail.com> > wrote:
We've recently been required to enable TLS between our Accumulo clients and the tablet servers. We've got it working, but we're experiencing a significant performance impact. I'm running jstack on our client processes and consistently seeing a number of Accumulo client threads in sun.security.ssl.SSLSocketImpl.performInitialHandshake. This implies to me that the Accumulo client is not reusing TLS sessions, and instead establishing a new session for each connection. Am I reading this correctly? Is there a way to configure the Accumulo client to reuse TLS sessions?
--
Sanjay Deshmukh
sanjayd@gmail.com <ma...@gmail.com>
Re: Impact from Enabling TLS
Posted by Sanjay Deshmukh <sa...@gmail.com>.
Sure thing. We're running Accumulo 1.10.1. Our configuration has 15 tablet
servers and a client pool of 4 threads that could be doing scans or batch
scans at any given time. I wish I could copy over the exact jstack but
unfortunately it's coming from an offline network, but I do see the same
tablet servers in the thread names repeatedly (e.g. "batch scanner X- 4
looking up 1 ranges at tabletservername:9997", implying it it's not an
infrequent use problem. In some cases I even see multiple threads trying to
handshake with the same tablet server. The jstack here seems to go from
TabletServerbatchReaderIterator.doLookup down through
TFramedTransport.flush and into SSLSocketImpl.performInitialHandshake.
The lifecycle here is that each of these client threads is using a single
shared instance of Connector, and instantiating a new Scanner or
BatchScanner per use, and closing it when that use is done. There are many
of these uses over time.
On Mon, Jun 5, 2023 at 3:14 PM Christopher <ct...@apache.org> wrote:
> In Accumulo code, we do try to reuse thrift connections, but I'm not sure
> if the thrift code itself is reusing TLS sessions or not.
>
> Keep in mind that Accumulo is a distributed system. Many of these
> handshakes might be going to different servers in a cluster. Are you able
> to tell if that is the case? It might be possible to increase session
> timeouts or the number of cached sessions, but the best case is that you'll
> still need to do a handshake with each server your client talks to.
>
> It might also help if you told us the version of Accumulo you are using,
> and how you are using the client code (like, are you reusing client
> objects). If you can share a bit of the jstack with us, that might also be
> helpful if we need to troubleshoot further.
>
> On Mon, Jun 5, 2023, 14:12 Sanjay Deshmukh <sa...@gmail.com> wrote:
>
>> We've recently been required to enable TLS between our Accumulo clients
>> and the tablet servers. We've got it working, but we're experiencing a
>> significant performance impact. I'm running jstack on our client processes
>> and consistently seeing a number of Accumulo client threads in
>> sun.security.ssl.SSLSocketImpl.performInitialHandshake. This implies to me
>> that the Accumulo client is not reusing TLS sessions, and instead
>> establishing a new session for each connection. Am I reading
>> this correctly? Is there a way to configure the Accumulo client to reuse
>> TLS sessions?
>>
>
--
Sanjay Deshmukh
sanjayd@gmail.com
Re: Impact from Enabling TLS
Posted by Christopher <ct...@apache.org>.
In Accumulo code, we do try to reuse thrift connections, but I'm not sure
if the thrift code itself is reusing TLS sessions or not.
Keep in mind that Accumulo is a distributed system. Many of these
handshakes might be going to different servers in a cluster. Are you able
to tell if that is the case? It might be possible to increase session
timeouts or the number of cached sessions, but the best case is that you'll
still need to do a handshake with each server your client talks to.
It might also help if you told us the version of Accumulo you are using,
and how you are using the client code (like, are you reusing client
objects). If you can share a bit of the jstack with us, that might also be
helpful if we need to troubleshoot further.
On Mon, Jun 5, 2023, 14:12 Sanjay Deshmukh <sa...@gmail.com> wrote:
> We've recently been required to enable TLS between our Accumulo clients
> and the tablet servers. We've got it working, but we're experiencing a
> significant performance impact. I'm running jstack on our client processes
> and consistently seeing a number of Accumulo client threads in
> sun.security.ssl.SSLSocketImpl.performInitialHandshake. This implies to me
> that the Accumulo client is not reusing TLS sessions, and instead
> establishing a new session for each connection. Am I reading
> this correctly? Is there a way to configure the Accumulo client to reuse
> TLS sessions?
>