You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Eric Evans (JIRA)" <ji...@apache.org> on 2017/08/08 14:57:00 UTC
[jira] [Commented] (CASSANDRA-13006) Disable automatic heap dumps
on OOM error
[ https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118421#comment-16118421 ]
Eric Evans commented on CASSANDRA-13006:
----------------------------------------
I think this behavior (invoking {{jmap}} on OOM) is a pretty serious violation to the Element of Least-surprise. We already provide mechanisms for passing arguments to the JVM, and TTBMK, all of them provide some means for dropping a heap dump on out-of-memory.
It definitely caught me be surprise. We carried over {{-XX:+HeapDumpOnOutOfMemoryError}} from our 2.2.x environment, only to have Cassandra and the JVM racing to create a dump of the same name.
Additionally, something about all of this is buggy, because on more than one occasion we've had Cassandra fork-bombing {{jmap}} processes
{noformat}
● cassandra-b.service - distributed storage system for structured data
Loaded: loaded (/lib/systemd/system/cassandra-b.service; static)
Active: active (running) since Sat 2017-08-05 22:32:07 UTC; 23h ago
Main PID: 25025 (java)
CGroup: /system.slice/cassandra-b.service
├─ 9213 jmap -histo 25025
├─ 9214 jmap -histo 25025
├─ 9284 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9285 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9388 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9453 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9519 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9520 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9733 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9735 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─ 9736 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─14835 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─14836 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─14837 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─14839 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─14841 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─14844 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18932 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18933 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18934 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18935 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18936 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18937 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18938 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18939 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18940 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18942 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18943 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18944 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
├─18945 jmap -dump:format=b,file=/srv/cassandra-b/cassandra-1501972327-pid24937.hprof 25025
[ ... ]
{noformat}
IMO, the sanest strategy here would be to leave the creation of heap dumps to the JVM.
> Disable automatic heap dumps on OOM error
> -----------------------------------------
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration
> Reporter: anmols
> Assignee: Benjamin Lerer
> Priority: Minor
> Fix For: 3.0.9
>
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by default if the process encountered an OOM error. These heap dumps are stored in the Apache Cassandra home directory unless configured otherwise (see [Cassandra Support Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps] for this feature).
>
> The creation and storage of heap dumps aides debugging and investigative workflows, but is not be desirable for a production environment where these heap dumps may occupy a large amount of disk space and require manual intervention for cleanups.
>
> Managing heap dumps on out of memory errors and configuring the paths for these heap dumps are available as JVM options in JVM. The current behavior conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError.
>
> A patch can be proposed here that would make the heap dump on OOM error honor the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM option.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org