You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@cassandra.apache.org by GitBox <gi...@apache.org> on 2020/11/04 22:58:09 UTC
[GitHub] [cassandra] dcapwell commented on a change in pull request #761: CASSANDRA-15214: Force throw heap space OutOfMemory error
dcapwell commented on a change in pull request #761:
URL: https://github.com/apache/cassandra/pull/761#discussion_r517678215
##########
File path: src/java/org/apache/cassandra/utils/JVMStabilityInspector.java
##########
@@ -125,7 +120,28 @@ public static void inspectThrowable(Throwable t, boolean propagateOutOfMemory, C
killer.killCurrentJVM(t);
if (t.getCause() != null)
- inspectThrowable(t.getCause(), propagateOutOfMemory, fn);
+ inspectThrowable(t.getCause(), fn);
+ }
+
+ /**
+ * Intentionally produce a heap space OOM upon seeing a Direct buffer memory OOM.
+ * Direct buffer OOM cannot trigger JVM OOM error related options,
+ * e.g. OnOutOfMemoryError, HeapDumpOnOutOfMemoryError, etc.
+ */
+ private static void forceHeapSpaceOomMaybe(OutOfMemoryError oom)
+ {
+ // See the oom thrown from java.nio.Bits.reserveMemory.
+ if (oom.getMessage() != null && !oom.getMessage().equals("Direct buffer memory"))
Review comment:
can simplify to the below as string equality checks null, so don't need a null check first.
```
if (!"Direct buffer memory".equals(oom.getMessage()))
```
##########
File path: src/java/org/apache/cassandra/utils/JVMStabilityInspector.java
##########
@@ -125,7 +120,28 @@ public static void inspectThrowable(Throwable t, boolean propagateOutOfMemory, C
killer.killCurrentJVM(t);
if (t.getCause() != null)
- inspectThrowable(t.getCause(), propagateOutOfMemory, fn);
+ inspectThrowable(t.getCause(), fn);
+ }
+
+ /**
+ * Intentionally produce a heap space OOM upon seeing a Direct buffer memory OOM.
+ * Direct buffer OOM cannot trigger JVM OOM error related options,
+ * e.g. OnOutOfMemoryError, HeapDumpOnOutOfMemoryError, etc.
+ */
+ private static void forceHeapSpaceOomMaybe(OutOfMemoryError oom)
+ {
+ // See the oom thrown from java.nio.Bits.reserveMemory.
+ if (oom.getMessage() != null && !oom.getMessage().equals("Direct buffer memory"))
+ {
+ return;
+ }
+ logger.warn("Force heap space OutOfMemoryError in the presence of", oom);
+ while (true)
+ {
+ // java.util.AbstractCollection.MAX_ARRAY_SIZE is defined as Integer.MAX_VALUE - 8
+ // so Integer.MAX_VALUE / 2 should be a large enough and safe size to request.
+ long[] ignored = new long[Integer.MAX_VALUE / 2];
Review comment:
it may be good to collect the references in a list to make sure GC isn't able to free right away.
```
List<long[]> ignored = new ArrayList<>();
while (true)
{
ignored.add(new long[Integer.MAX_VALUE / 2]);
}
```
##########
File path: src/java/org/apache/cassandra/utils/JVMStabilityInspector.java
##########
@@ -125,7 +120,28 @@ public static void inspectThrowable(Throwable t, boolean propagateOutOfMemory, C
killer.killCurrentJVM(t);
if (t.getCause() != null)
- inspectThrowable(t.getCause(), propagateOutOfMemory, fn);
+ inspectThrowable(t.getCause(), fn);
+ }
+
+ /**
+ * Intentionally produce a heap space OOM upon seeing a Direct buffer memory OOM.
+ * Direct buffer OOM cannot trigger JVM OOM error related options,
+ * e.g. OnOutOfMemoryError, HeapDumpOnOutOfMemoryError, etc.
+ */
+ private static void forceHeapSpaceOomMaybe(OutOfMemoryError oom)
+ {
+ // See the oom thrown from java.nio.Bits.reserveMemory.
+ if (oom.getMessage() != null && !oom.getMessage().equals("Direct buffer memory"))
+ {
+ return;
+ }
+ logger.warn("Force heap space OutOfMemoryError in the presence of", oom);
Review comment:
thinking should be `error`
##########
File path: src/java/org/apache/cassandra/utils/JVMStabilityInspector.java
##########
@@ -125,7 +120,28 @@ public static void inspectThrowable(Throwable t, boolean propagateOutOfMemory, C
killer.killCurrentJVM(t);
if (t.getCause() != null)
- inspectThrowable(t.getCause(), propagateOutOfMemory, fn);
+ inspectThrowable(t.getCause(), fn);
+ }
+
+ /**
+ * Intentionally produce a heap space OOM upon seeing a Direct buffer memory OOM.
+ * Direct buffer OOM cannot trigger JVM OOM error related options,
+ * e.g. OnOutOfMemoryError, HeapDumpOnOutOfMemoryError, etc.
+ */
Review comment:
would be good to link to `CASSANDRA-15214`, something like
```
* see CASSANDRA-15214 for more details
```
##########
File path: src/java/org/apache/cassandra/utils/JVMStabilityInspector.java
##########
@@ -125,7 +120,28 @@ public static void inspectThrowable(Throwable t, boolean propagateOutOfMemory, C
killer.killCurrentJVM(t);
if (t.getCause() != null)
- inspectThrowable(t.getCause(), propagateOutOfMemory, fn);
+ inspectThrowable(t.getCause(), fn);
+ }
+
+ /**
+ * Intentionally produce a heap space OOM upon seeing a Direct buffer memory OOM.
+ * Direct buffer OOM cannot trigger JVM OOM error related options,
+ * e.g. OnOutOfMemoryError, HeapDumpOnOutOfMemoryError, etc.
+ */
+ private static void forceHeapSpaceOomMaybe(OutOfMemoryError oom)
+ {
+ // See the oom thrown from java.nio.Bits.reserveMemory.
+ if (oom.getMessage() != null && !oom.getMessage().equals("Direct buffer memory"))
+ {
+ return;
+ }
+ logger.warn("Force heap space OutOfMemoryError in the presence of", oom);
+ while (true)
+ {
+ // java.util.AbstractCollection.MAX_ARRAY_SIZE is defined as Integer.MAX_VALUE - 8
+ // so Integer.MAX_VALUE / 2 should be a large enough and safe size to request.
+ long[] ignored = new long[Integer.MAX_VALUE / 2];
Review comment:
Also, in case the while true loop never breaks (such as JVM gets smart enough to know this is not needed and avoids the allocation), we should probably start logging an error saying that this trick isn't doing what we hoped.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@cassandra.apache.org
For additional commands, e-mail: pr-help@cassandra.apache.org