You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Markus Dlugi (JIRA)" <ji...@apache.org> on 2017/08/31 07:39:00 UTC
[jira] [Comment Edited] (CASSANDRA-13754) FastThreadLocal leaks
memory
[ https://issues.apache.org/jira/browse/CASSANDRA-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148580#comment-16148580 ]
Markus Dlugi edited comment on CASSANDRA-13754 at 8/31/17 7:38 AM:
-------------------------------------------------------------------
[~jjirsa], we are experiencing the same issue on Cassandra 3.11.0 and Netty 4.0.44. We have some custom triggers which create additional rows when {{INSERT}} s on specific CFs are executed. When inserting a lot of data, the Cassandra would constantly crash with {{OutOfMemoryError}} s. We analyzed a heap dump and came to the same conclusion, which is that the instances of {{FastThreadLocalThread}} were responsible for this behaviour. As a quick workaround, we added a call to {{FastThreadLocal.removeAll()}} to our triggers, which alleviates the problem for us.
It seems that there already was an issue once with {{FastThreadLocal}} s, as described in CASSANDRA-13033, but apparently the change by [~snazy] in CASSANDRA-13034 resurfaced the issue. For a proper fix, I think all {{FastThreadLocalThread}} instances should call {{FastThreadLocal.removeAll()}} when they are done with their work. A quick glance at the code indicates that this affects the classes {{CompressedInputStream}} , {{StreamingInboundHandler}} , {{NamedThreadFactory}} and {{SEPWorker}}.
was (Author: markusdlugi):
[~jjirsa], we are experiencing the same issue on Cassandra 3.11.0 and Netty 4.0.44. We have some custom triggers which create additional rows when {{INSERT}}s on specific CFs are executed. When inserting a lot of data, the Cassandra would constantly crash with {{OutOfMemoryError}}s. We analyzed a heap dump and came to the same conclusion, which is that the instances of {{FastThreadLocalThread}} were responsible for this behaviour. As a quick workaround, we added a call to {{FastThreadLocal.removeAll()}} to our triggers, which alleviates the problem for us.
It seems that there already was an issue once with {{FastThreadLocal}}s, as described in CASSANDRA-13033, but apparently the change by [~snazy] in CASSANDRA-13034 resurfaced the issue. For a proper fix, I think all {{FastThreadLocalThread}} instances should call {{FastThreadLocal.removeAll()}} when they are done with their work. A quick glance at the code indicates that this affects the classes {{CompressedInputStream}}, {{StreamingInboundHandler}}, {{NamedThreadFactory}} and {{SEPWorker}}.
> FastThreadLocal leaks memory
> ----------------------------
>
> Key: CASSANDRA-13754
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13754
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.11.0, Netty 4.0.44.Final, OpenJDK 8u141-b15
> Reporter: Eric Evans
> Fix For: 3.11.1
>
>
> After a chronic bout of {{OutOfMemoryError}} in our development environment, a heap analysis is showing that more than 10G of our 12G heaps are consumed by the {{threadLocals}} members (instances of {{java.lang.ThreadLocalMap}}) of various {{io.netty.util.concurrent.FastThreadLocalThread}} instances. Reverting [cecbe17|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54] fixes the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org