You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Joseph Lynch (JIRA)" <ji...@apache.org> on 2019/07/16 21:37:00 UTC
[jira] [Comment Edited] (CASSANDRA-15214) OOMs caught and not
rethrown
[ https://issues.apache.org/jira/browse/CASSANDRA-15214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16886475#comment-16886475 ]
Joseph Lynch edited comment on CASSANDRA-15214 at 7/16/19 9:36 PM:
-------------------------------------------------------------------
We've (Netlfix) found handling OOMs to be generally hard to do correctly in all the various Java codebases we have so we built an agent solution which attaches to the JVM in [https://github.com/Netflix-Skunkworks/jvmquake]. I think the only reason that we couldn't just directly include that in C* is because it's a C JVMTI agent instead of a Java one, but perhaps we could just solve this with some documentation and making it really easy to include agents (which is useful regardless)? I can also spend some time and see if I can make it a java agent instead of a c one.
The following is the patch for supporting easy pluggable agents for C*:
{noformat}
diff --git a/conf/cassandra-env.sh b/conf/cassandra-env.sh
index d6c48be0a3..92061db3ab 100644
--- a/conf/cassandra-env.sh
+++ b/conf/cassandra-env.sh
@@ -134,6 +134,29 @@ do
JVM_OPTS="$JVM_OPTS $opt"
done
+# Pull in any agents present in CASSANDRA_HOME
+for agent_file in ${CASSANDRA_HOME}/agents/*.jar; do
+ if [ -e "${agent_file}" ]; then
+ base_file="${agent_file%.jar}"
+ if [ -s "${base_file}.options" ]; then
+ options=`cat ${base_file}.options`
+ agent_file="${agent_file}=${options}"
+ fi
+ JVM_OPTS="$JVM_OPTS -javaagent:${agent_file}"
+ fi
+done
+
+for agent_file in ${CASSANDRA_HOME}/agents/*.so; do
+ if [ -e "${agent_file}" ]; then
+ base_file="${agent_file%.so}"
+ if [ -s "${base_file}.options" ]; then
+ options=`cat ${base_file}.options`
+ agent_file="${agent_file}=${options}"
+ fi
+ JVM_OPTS="$JVM_OPTS -agentpath:${agent_file}"
+ fi
+done
{noformat}
Then we can just drop agents into the {{CASSANDRA_HOME/agents}} folder and they are loaded automatically by Cassandra. From a security perspective this is identical to "drop a jar".
was (Author: jolynch):
We've (Netlfix) found handling OOMs to be generally hard to do correctly in all the various Java codebases we have so we built an agent solution which attaches to the JVM in [https://github.com/Netflix-Skunkworks/jvmquake]. I think the only reason that we couldn't just directly include that in C* is because it's a C JVMTI agent instead of a Java one, but perhaps we could just solve this with some documentation and making it really easy to include agents (which is useful regardless)?
The following is the patch for supporting easy pluggable agents for C*:
{noformat}
diff --git a/conf/cassandra-env.sh b/conf/cassandra-env.sh
index d6c48be0a3..92061db3ab 100644
--- a/conf/cassandra-env.sh
+++ b/conf/cassandra-env.sh
@@ -134,6 +134,29 @@ do
JVM_OPTS="$JVM_OPTS $opt"
done
+# Pull in any agents present in CASSANDRA_HOME
+for agent_file in ${CASSANDRA_HOME}/agents/*.jar; do
+ if [ -e "${agent_file}" ]; then
+ base_file="${agent_file%.jar}"
+ if [ -s "${base_file}.options" ]; then
+ options=`cat ${base_file}.options`
+ agent_file="${agent_file}=${options}"
+ fi
+ JVM_OPTS="$JVM_OPTS -javaagent:${agent_file}"
+ fi
+done
+
+for agent_file in ${CASSANDRA_HOME}/agents/*.so; do
+ if [ -e "${agent_file}" ]; then
+ base_file="${agent_file%.so}"
+ if [ -s "${base_file}.options" ]; then
+ options=`cat ${base_file}.options`
+ agent_file="${agent_file}=${options}"
+ fi
+ JVM_OPTS="$JVM_OPTS -agentpath:${agent_file}"
+ fi
+done
{noformat}
Then we can just drop agents into the {{CASSANDRA_HOME/agents}} folder and they are loaded automatically by Cassandra. From a security perspective this is identical to "drop a jar".
> OOMs caught and not rethrown
> ----------------------------
>
> Key: CASSANDRA-15214
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15214
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Client, Messaging/Internode
> Reporter: Benedict
> Priority: Normal
> Fix For: 4.0
>
>
> Netty (at least, and perhaps elsewhere in Executors) catches all exceptions, so presently there is no way to ensure that an OOM reaches the JVM handler to trigger a crash/heapdump.
> It may be that the simplest most consistent way to do this would be to have a single thread spawned at startup that waits for any exceptions we must propagate to the Runtime.
> We could probably submit a patch upstream to Netty, but for a guaranteed future proof approach, it may be worth paying the cost of a single thread.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org