You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Bryan Beaudreault (Jira)" <ji...@apache.org> on 2023/08/14 14:37:00 UTC
[jira] [Comment Edited] (HBASE-27947) RegionServer OOM under load when TLS is enabled
[ https://issues.apache.org/jira/browse/HBASE-27947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753817#comment-17753817 ]
Bryan Beaudreault edited comment on HBASE-27947 at 8/14/23 2:36 PM:
--------------------------------------------------------------------
I had some time this afternoon and got a POC going of my idea above. It results in a massive improvement in throughput, virtually eliminating the contention in PoolArena. With this change, our netty-based SSL throughput is about onpar with haproxy (for this test at least). I just attached another flamegraph, running with the optimization.
I haven't tried contributing to the netty project before, but this improvement is good enough that I'm going to give it a try. I will post a link to the git issue here once it's ready (next 1-2 days).
was (Author: bbeaudreault):
I had some time this afternoon and got a POC going of my idea above. It results in a massive improvement in throughput, virtually eliminating the contention in PoolArena. With this change, our netty-based SSL actually has higher throughput than haproxy (for this test at least). I just attached another flamegraph, running with the optimization.
I haven't tried contributing to the netty project before, but this improvement is good enough that I'm going to give it a try. I will post a link to the git issue here once it's ready (next 1-2 days).
> RegionServer OOM under load when TLS is enabled
> -----------------------------------------------
>
> Key: HBASE-27947
> URL: https://issues.apache.org/jira/browse/HBASE-27947
> Project: HBase
> Issue Type: Bug
> Components: rpc
> Affects Versions: 2.6.0
> Reporter: Bryan Beaudreault
> Assignee: Bryan Beaudreault
> Priority: Critical
> Attachments: ssl-disabled-flamegraph.html, ssl-enabled-flamegraph.html, ssl-enabled-optimized.html
>
>
> We are rolling out the server side TLS settings to all of our QA clusters. This has mostly gone fine, except on 1 cluster. Most clusters, including this one have a sampled {{nettyDirectMemory}} usage of about 30-100mb. This cluster tends to get bursts of traffic, in which case it would typically jump to 400-500mb. Again this is sampled, so it could have been higher than that. When we enabled SSL on this cluster, we started seeing bursts up to at least 4gb. This exceeded our {{{}-XX:MaxDirectMemorySize{}}}, which caused OOM's and general chaos on the cluster.
>
> We've gotten it under control a little bit by setting {{-Dorg.apache.hbase.thirdparty.io.netty.maxDirectMemory}} and {{{}-Dorg.apache.hbase.thirdparty.io.netty.tryReflectionSetAccessible{}}}. We've set netty's maxDirectMemory to be approx equal to ({{{}-XX:MaxDirectMemorySize - BucketCacheSize - ReservoirSize{}}}). Now we are seeing netty's own OutOfDirectMemoryError, which is still causing pain for clients but at least insulates the other components of the regionserver.
>
> We're still digging into exactly why this is happening. The cluster clearly has a bad access pattern, but it doesn't seem like SSL should increase the memory footprint by 5-10x like we're seeing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)