You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "garima.j" <ga...@ril.com> on 2019/05/23 09:18:53 UTC
JVM Halt on Null Pointer Exception in GridDhtTxAbstractEnlistFuture
Hello,
We have a 3 node cluster in production for Apache Ignite version 2.7. RAM
128GB. A Spark streaming service (with thick Ignite clients) writes data
into Ignite cache in a transaction (first get and then put).
Now, 3 spark clients stopped and hence NODE_FAILED events were received. My
Ignite crashed on one node and had the below stack trace :
[2019-05-23T13:57:04,976][WARN ][sys-stripe-5-#6][lock] Received near enlist
request from unknown node (will ignore) [txId=GridCacheVersion
[topVer=169659586, order=1558471024158, nodeOrder=23],
node=1be3bce3-7220-45bc-9863-4f16d97ea22b]
[2019-05-23T13:57:04,977][ERROR][sys-stripe-5-#6][GridCacheIoManager] Failed
processing message [senderId=1be3bce3-7220-45bc-9863-4f16d97ea22b,
msg=GridNearTxEnlistRequest [threadId=5872,
futId=c3170abca61-33b3ea8d-0a3e-44cb-83e6-032a37a9eed1, clientFirst=false,
miniId=1, subjId=1be3bce3-7220-45bc-9863-4f16d97ea22b,
topVer=AffinityTopologyVersion [topVer=101, minorTopVer=0],
lockVer=GridCacheVersion [topVer=169659586, order=1558471024158,
nodeOrder=23], mvccSnapshot=MvccSnapshotResponse [futId=1221240,
crdVer=1558179485875, cntr=110485182, opCntr=1, txs=[101051367, 110485176],
cleanupVer=101051361, tracking=0], timeout=5000, txTimeout=5000,
taskNameHash=0, op=UPSERT, needRes=false]]
java.lang.NullPointerException: null
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.<init>(GridDhtTxAbstractEnlistFuture.java:237)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxEnlistFuture.<init>(GridDhtTxEnlistFuture.java:84)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processNearTxEnlistRequest(GridDhtTransactionalCacheAdapter.java:2061)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$900(GridDhtTransactionalCacheAdapter.java:112)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:229)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:227)
~[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
[ignite-core-2.7.0.jar:2.7.0]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
[ignite-core-2.7.0.jar:2.7.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
[2019-05-23T13:57:05,043][ERROR][sys-stripe-5-#6][] Critical system error
detected. Will be handled accordingly to configured handler
[hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
failureCtx=FailureContext [type=CRITICAL_ERROR,
err=java.lang.NullPointerException]]
java.lang.NullPointerException: null
Please help and let me know why this failure happened.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: JVM Halt on Null Pointer Exception in GridDhtTxAbstractEnlistFuture
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Do you happen to use SQL MVCC?
https://apacheignite.readme.io/docs/multiversion-concurrency-control#section-multiversion-concurrency-control
Looks like you've stumbled on a previously unknown issue. Can you please
file a ticket against Apache Ignite JIRA, with some description of steps to
reproduce?
Regards,
--
Ilya Kasnacheev
чт, 23 мая 2019 г. в 12:19, garima.j <ga...@ril.com>:
> Hello,
>
> We have a 3 node cluster in production for Apache Ignite version 2.7. RAM
> 128GB. A Spark streaming service (with thick Ignite clients) writes data
> into Ignite cache in a transaction (first get and then put).
> Now, 3 spark clients stopped and hence NODE_FAILED events were received. My
> Ignite crashed on one node and had the below stack trace :
>
> [2019-05-23T13:57:04,976][WARN ][sys-stripe-5-#6][lock] Received near
> enlist
> request from unknown node (will ignore) [txId=GridCacheVersion
> [topVer=169659586, order=1558471024158, nodeOrder=23],
> node=1be3bce3-7220-45bc-9863-4f16d97ea22b]
> [2019-05-23T13:57:04,977][ERROR][sys-stripe-5-#6][GridCacheIoManager]
> Failed
> processing message [senderId=1be3bce3-7220-45bc-9863-4f16d97ea22b,
> msg=GridNearTxEnlistRequest [threadId=5872,
> futId=c3170abca61-33b3ea8d-0a3e-44cb-83e6-032a37a9eed1, clientFirst=false,
> miniId=1, subjId=1be3bce3-7220-45bc-9863-4f16d97ea22b,
> topVer=AffinityTopologyVersion [topVer=101, minorTopVer=0],
> lockVer=GridCacheVersion [topVer=169659586, order=1558471024158,
> nodeOrder=23], mvccSnapshot=MvccSnapshotResponse [futId=1221240,
> crdVer=1558179485875, cntr=110485182, opCntr=1, txs=[101051367, 110485176],
> cleanupVer=101051361, tracking=0], timeout=5000, txTimeout=5000,
> taskNameHash=0, op=UPSERT, needRes=false]]
> java.lang.NullPointerException: null
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.<init>(GridDhtTxAbstractEnlistFuture.java:237)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxEnlistFuture.<init>(GridDhtTxEnlistFuture.java:84)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processNearTxEnlistRequest(GridDhtTransactionalCacheAdapter.java:2061)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$900(GridDhtTransactionalCacheAdapter.java:112)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:229)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:227)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
> [ignite-core-2.7.0.jar:2.7.0]
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> [ignite-core-2.7.0.jar:2.7.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
> [2019-05-23T13:57:05,043][ERROR][sys-stripe-5-#6][] Critical system error
> detected. Will be handled accordingly to configured handler
> [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
> failureCtx=FailureContext [type=CRITICAL_ERROR,
> err=java.lang.NullPointerException]]
> java.lang.NullPointerException: null
>
> Please help and let me know why this failure happened.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: JVM Halt on Null Pointer Exception in GridDhtTxAbstractEnlistFuture
Posted by Denis Magda <dm...@apache.org>.
Hey,
Please try to disable the critical failures handler as written here:
http://apache-ignite-users.70518.x6.nabble.com/Error-Running-Gridgain-s-LoadCaches-java-application-td28276.html
Let us know if it works.
-
Denis
On Thu, May 23, 2019 at 2:19 AM garima.j <ga...@ril.com> wrote:
> Hello,
>
> We have a 3 node cluster in production for Apache Ignite version 2.7. RAM
> 128GB. A Spark streaming service (with thick Ignite clients) writes data
> into Ignite cache in a transaction (first get and then put).
> Now, 3 spark clients stopped and hence NODE_FAILED events were received. My
> Ignite crashed on one node and had the below stack trace :
>
> [2019-05-23T13:57:04,976][WARN ][sys-stripe-5-#6][lock] Received near
> enlist
> request from unknown node (will ignore) [txId=GridCacheVersion
> [topVer=169659586, order=1558471024158, nodeOrder=23],
> node=1be3bce3-7220-45bc-9863-4f16d97ea22b]
> [2019-05-23T13:57:04,977][ERROR][sys-stripe-5-#6][GridCacheIoManager]
> Failed
> processing message [senderId=1be3bce3-7220-45bc-9863-4f16d97ea22b,
> msg=GridNearTxEnlistRequest [threadId=5872,
> futId=c3170abca61-33b3ea8d-0a3e-44cb-83e6-032a37a9eed1, clientFirst=false,
> miniId=1, subjId=1be3bce3-7220-45bc-9863-4f16d97ea22b,
> topVer=AffinityTopologyVersion [topVer=101, minorTopVer=0],
> lockVer=GridCacheVersion [topVer=169659586, order=1558471024158,
> nodeOrder=23], mvccSnapshot=MvccSnapshotResponse [futId=1221240,
> crdVer=1558179485875, cntr=110485182, opCntr=1, txs=[101051367, 110485176],
> cleanupVer=101051361, tracking=0], timeout=5000, txTimeout=5000,
> taskNameHash=0, op=UPSERT, needRes=false]]
> java.lang.NullPointerException: null
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.<init>(GridDhtTxAbstractEnlistFuture.java:237)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxEnlistFuture.<init>(GridDhtTxEnlistFuture.java:84)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processNearTxEnlistRequest(GridDhtTransactionalCacheAdapter.java:2061)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$900(GridDhtTransactionalCacheAdapter.java:112)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:229)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:227)
> ~[ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> [ignite-core-2.7.0.jar:2.7.0]
> at
>
> org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
> [ignite-core-2.7.0.jar:2.7.0]
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> [ignite-core-2.7.0.jar:2.7.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
> [2019-05-23T13:57:05,043][ERROR][sys-stripe-5-#6][] Critical system error
> detected. Will be handled accordingly to configured handler
> [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
> super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]],
> failureCtx=FailureContext [type=CRITICAL_ERROR,
> err=java.lang.NullPointerException]]
> java.lang.NullPointerException: null
>
> Please help and let me know why this failure happened.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>