You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Maxim Muzafarov (Jira)" <ji...@apache.org> on 2022/08/16 14:05:00 UTC
[jira] [Assigned] (IGNITE-16136) System Thread pool starvation and out of memory
[ https://issues.apache.org/jira/browse/IGNITE-16136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maxim Muzafarov reassigned IGNITE-16136:
----------------------------------------
Assignee: Maxim Muzafarov
> System Thread pool starvation and out of memory
> -----------------------------------------------
>
> Key: IGNITE-16136
> URL: https://issues.apache.org/jira/browse/IGNITE-16136
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 2.7.6
> Reporter: David Albrecht
> Assignee: Maxim Muzafarov
> Priority: Critical
> Labels: ise
> Fix For: 2.14
>
> Attachments: image-2021-12-15-21-13-43-775.png, image-2021-12-15-21-17-47-652.png
>
>
> We are experiencing thread pool starvations and after some time out of memory exceptions in some of our ignite client nodes while the server node seems to be running without any problems. It seems like all sys threads are stuck when calling MarshallerContextImpl.getClassName. Which in turn leads to a growing worker queue.
>
> First warnings regarding the thread pool starvation:
> {code:java}
> 10.12.21 11:22:34.603 [WARN ] IgniteKernal.warning(127): Possible thread pool starvation detected (no task completed in last 30000ms, is system thread pool size large enough?)
> 10.12.21 11:27:34.654 [WARN ] IgniteKernal.warning(127): Possible thread pool starvation detected (no task completed in last 30000ms, is system thread pool size large enough?)
> 10.12.21 11:32:34.713 [WARN ] IgniteKernal.warning(127): Possible thread pool starvation detected (no task completed in last 30000ms, is system thread pool size large enough?)
> 10.12.21 11:37:34.764 [WARN ] IgniteKernal.warning(127): Possible thread pool starvation detected (no task completed in last 30000ms, is system thread pool size large enough?)
> 10.12.21 11:42:34.796 [WARN ] IgniteKernal.warning(127): Possible thread pool starvation detected (no task completed in last 30000ms, is system thread pool size large enough?)
> 10.12.21 11:47:34.839 [WARN ] IgniteKernal.warning(127): Possible thread pool starvation detected (no task completed in last 30000ms, is system thread pool size large enough?)
> {code}
> Out of memory error leading to a crash of the application:
> {code}
> Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "https-openssl-nio-16443-ClientPoller"
> Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ajp-nio-16009-ClientPoller"
> 11-Dec-2021 03:07:24.446 SEVERE [Catalina-utility-1] org.apache.coyote.AbstractProtocol.startAsyncTimeout Error processing async timeouts
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
> {code}
> The queue full of messages:
> !image-2021-12-15-21-17-47-652.png!
> It seems like all sys threads are stuck while waiting at:
> {code}
> sys-#170
> at jdk.internal.misc.Unsafe.park(ZJ)V (Native Method)
> at java.util.concurrent.locks.LockSupport.park()V (LockSupport.java:323)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(Z)Ljava/lang/Object; (GridFutureAdapter.java:178)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.get()Ljava/lang/Object; (GridFutureAdapter.java:141)
> at org.apache.ignite.internal.MarshallerContextImpl.getClassName(BI)Ljava/lang/String; (MarshallerContextImpl.java:379)
> at org.apache.ignite.internal.MarshallerContextImpl.getClass(ILjava/lang/ClassLoader;)Ljava/lang/Class; (MarshallerContextImpl.java:344)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshallerUtils.classDescriptor(Ljava/util/concurrent/ConcurrentMap;ILjava/lang/ClassLoader;Lorg/apache/ignite/marshaller/MarshallerContext;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedMarshallerIdMapper;)Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor; (OptimizedMarshallerUtils.java:264)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object; (OptimizedObjectInputStream.java:341)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object; (OptimizedObjectInputStream.java:198)
> at java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object; (ObjectInputStream.java:484)
> at java.io.ObjectInputStream.readObject()Ljava/lang/Object; (ObjectInputStream.java:451)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(Ljava/lang/Object;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor$ClassFields;)V (OptimizedObjectInputStream.java:519)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(Ljava/lang/Class;Ljava/util/List;Ljava/lang/reflect/Method;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor$Fields;)Ljava/lang/Object; (OptimizedObjectInputStream.java:611)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(Lorg/apache/ignite/internal/marshaller/optimized/OptimizedObjectInputStream;)Ljava/lang/Object; (OptimizedClassDescriptor.java:954)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object; (OptimizedObjectInputStream.java:346)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object; (OptimizedObjectInputStream.java:198)
> at java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object; (ObjectInputStream.java:484)
> at java.io.ObjectInputStream.readObject()Ljava/lang/Object; (ObjectInputStream.java:451)
> at org.apache.ignite.internal.GridEventConsumeHandler$EventWrapper.readExternal(Ljava/io/ObjectInput;)V (GridEventConsumeHandler.java:558)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(Ljava/lang/reflect/Constructor;Ljava/lang/reflect/Method;)Ljava/lang/Object; (OptimizedObjectInputStream.java:555)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(Lorg/apache/ignite/internal/marshaller/optimized/OptimizedObjectInputStream;)Ljava/lang/Object; (OptimizedClassDescriptor.java:949)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object; (OptimizedObjectInputStream.java:346)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object; (OptimizedObjectInputStream.java:198)
> at java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object; (ObjectInputStream.java:484)
> at java.io.ObjectInputStream.readObject()Ljava/lang/Object; (ObjectInputStream.java:451)
> at java.util.concurrent.ConcurrentLinkedDeque.readObject(Ljava/io/ObjectInputStream;)V (ConcurrentLinkedDeque.java:1588)
> at jdk.internal.reflect.GeneratedMethodAccessor268.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source)
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Method.java:566)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(Ljava/lang/Class;Ljava/util/List;Ljava/lang/reflect/Method;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor$Fields;)Ljava/lang/Object; (OptimizedObjectInputStream.java:604)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(Lorg/apache/ignite/internal/marshaller/optimized/OptimizedObjectInputStream;)Ljava/lang/Object; (OptimizedClassDescriptor.java:954)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object; (OptimizedObjectInputStream.java:346)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object; (OptimizedObjectInputStream.java:198)
> at java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object; (ObjectInputStream.java:484)
> at java.io.ObjectInputStream.readObject()Ljava/lang/Object; (ObjectInputStream.java:451)
> at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(Ljava/io/InputStream;Ljava/lang/ClassLoader;)Ljava/lang/Object; (OptimizedMarshaller.java:228)
> at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(Ljava/io/InputStream;Ljava/lang/ClassLoader;)Ljava/lang/Object; (AbstractNodeNameAwareMarshaller.java:94)
> at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(Lorg/apache/ignite/internal/binary/streams/BinaryInputStream;Lorg/apache/ignite/internal/binary/BinaryContext;Ljava/lang/ClassLoader;)Ljava/lang/Object; (BinaryUtils.java:1762)
> at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0()Ljava/lang/Object; (BinaryReaderExImpl.java:1965)
> at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize()Ljava/lang/Object; (BinaryReaderExImpl.java:1717)
> at org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(I)Ljava/lang/Object; (BinaryReaderExImpl.java:1985)
> at org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(Ljava/lang/Object;Lorg/apache/ignite/internal/binary/BinaryReaderExImpl;)V (BinaryFieldAccessor.java:703)
> at org.apache.ignite.internal.binary.BinaryFieldAccessor.read(Ljava/lang/Object;Lorg/apache/ignite/internal/binary/BinaryReaderExImpl;)V (BinaryFieldAccessor.java:188)
> at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(Lorg/apache/ignite/internal/binary/BinaryReaderExImpl;)Ljava/lang/Object; (BinaryClassDescriptor.java:875)
> at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0()Ljava/lang/Object; (BinaryReaderExImpl.java:1765)
> at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize()Ljava/lang/Object; (BinaryReaderExImpl.java:1717)
> at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize([BLjava/lang/ClassLoader;)Ljava/lang/Object; (GridBinaryMarshaller.java:313)
> at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0([BLjava/lang/ClassLoader;)Ljava/lang/Object; (BinaryMarshaller.java:102)
> at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal([BLjava/lang/ClassLoader;)Ljava/lang/Object; (AbstractNodeNameAwareMarshaller.java:82)
> at org.apache.ignite.internal.util.IgniteUtils.unmarshal(Lorg/apache/ignite/marshaller/Marshaller;[BLjava/lang/ClassLoader;)Ljava/lang/Object; (IgniteUtils.java:10168)
> at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$7.onMessage(Ljava/util/UUID;Ljava/lang/Object;B)V (GridContinuousProcessor.java:269)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(Ljava/lang/Byte;Lorg/apache/ignite/internal/managers/communication/GridMessageListener;Ljava/util/UUID;Ljava/lang/Object;)V (GridIoManager.java:1569)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(Lorg/apache/ignite/internal/managers/communication/GridIoMessage;Ljava/util/UUID;)V (GridIoManager.java:1197)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(Lorg/apache/ignite/internal/managers/communication/GridIoManager;Lorg/apache/ignite/internal/managers/communication/GridIoMessage;Ljava/util/UUID;)V (GridIoManager.java:127)
> at org.apache.ignite.internal.managers.communication.GridIoManager$9.run()V (GridIoManager.java:1093)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1128)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:628)
> at java.lang.Thread.run()V (Thread.java:829)
> {code}
> Screenshot of sys threads stacktrace:
> !image-2021-12-15-21-13-43-775.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)