You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Ilya Kasnacheev <il...@gmail.com> on 2018/09/25 15:10:36 UTC
Re: java.lang.NullPointerException in GridDhtPartitionsExchangeFuture
Hello!
It's hard to say without reviewing logs, but it seems that there's some
inconsistency with regards to cache metadata on nodes.
Regards,
--
Ilya Kasnacheev
вт, 25 сент. 2018 г. в 0:13, HEWA WIDANA GAMAGE, SUBASH <
Subash.HewaWidanaGamage@fmr.com>:
> Hi all,
>
> We use Ignite 1.9.
>
>
>
> We could see this in our logs. All we do is cache.get() , cache.put()
> operations. With this log being seen, is it possible for cache.put or
> ignite.getOrCreateCache() method calling threads be blocked forever ?
> (unfortunately we couldn’t get a thread dump to prove that, but from
> application logs, it looks like it).
>
>
>
> java.lang.NullPointerException: null
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.updatePartitionSingleMap(GridDhtPartitionsExchangeFuture.java:1446)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processMessage(GridDhtPartitionsExchangeFuture.java:1199)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$100(GridDhtPartitionsExchangeFuture.java:86)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$1.apply(GridDhtPartitionsExchangeFuture.java:1167)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$1.apply(GridDhtPartitionsExchangeFuture.java:1155)
>
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
>
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:228)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onReceive(GridDhtPartitionsExchangeFuture.java:1155)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processSinglePartitionUpdate(GridCachePartitionExchangeManager.java:1304)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1200(GridCachePartitionExchangeManager.java:116)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:310)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:308)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:1992)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:1974)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:827)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:369)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:293)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:95)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:238)
>
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1222)
>
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:850)
>
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:108)
>
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$7.run(GridIoManager.java:790)
>
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:428)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
Re: java.lang.NullPointerException in GridDhtPartitionsExchangeFuture
Posted by Павлухин Иван <vo...@gmail.com>.
Hi Subash,
Correct me if I am wrong, but in current Ignite version persistence is not
mandatory and it is even disabled by default. With disabled persistence
nothing will be written to disk. 2.x version storage architecture was
developed as an improvement of the previous one. Ignite 2.x uses off-heap
memory, but it is vanilla RAM local to JVM process (managed by Ignite
instead of automatic JVM management). Several benefits of off-heap are
mentioned in [1].
[1]
https://apacheignite.readme.io/docs/durable-memory#section-in-memory-features
2018-09-27 14:45 GMT+03:00 Ilya Kasnacheev <il...@gmail.com>:
> Hello!
>
> It's hard to answer these questions without thorough review of logs and
> 1.9 code, and I doubt anyone will volunteer to do that since 1.x branch
> does not see any new development in Apache Ignite.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 26 сент. 2018 г. в 22:36, HEWA WIDANA GAMAGE, SUBASH <
> Subash.HewaWidanaGamage@fmr.com>:
>
>> Hi Kasnacheev,
>>
>> Thank you very much for the response..
>>
>>
>>
>> We use v1.9, because 2.x uses mandatory ignite native persistence(local
>> disk) along with durable memory management(RAM), and no option for a **java
>> heap only** cache storage. We wanted to keep away from storing anything in
>> disk. Hence using 1.9 Ignite.
>>
>> Can you please clarify “I believe that cache in question is no longer
>> consistent between nodes on metadata level” ?
>>
>>
>>
>> 1. Like what could have caused & under what conditions this
>> inconsistent state between nodes on cache metadata occurred ?
>>
>>
>>
>> 2. And does 2.x fixes the problem?( Is there a way you can suggest
>> to reproduce this issue, so that we can know for sure 2.x fixes the problem)
>>
>>
>>
>> It appeared from our logs that the cache.put threads hangs forever after
>> this error. Can that be possible with this?
>>
>>
>>
>>
>>
>>
>>
>> *From:* Ilya Kasnacheev [mailto:ilya.kasnacheev@gmail.com]
>> *Sent:* Wednesday, September 26, 2018 12:31 PM
>> *To:* user@ignite.apache.org
>> *Subject:* Re: java.lang.NullPointerException in
>> GridDhtPartitionsExchangeFuture
>>
>>
>> This email is from an external source - exercise caution regarding links
>> and attachments.
>>
>> Hello!
>>
>>
>>
>> This is a bad error message. I believe that cache in question is no
>> longer consistent between nodes on metadata level, and you can't fix that
>> without full restart or at least dropping and recreating the cache.
>>
>> I don't think that you will get much support from community on 1.x since
>> the focus has shifted to 2.x. Have you considered upgrading?
>>
>>
>>
>> Regards,
>>
>> --
>>
>> Ilya Kasnacheev
>>
>>
>>
>>
>>
>> ср, 26 сент. 2018 г. в 18:50, HEWA WIDANA GAMAGE, SUBASH <
>> Subash.HewaWidanaGamage@fmr.com>:
>>
>> This is the only single error from ignite happened after JVM startup.
>> Looks like I only posted the exception stack trace. Here’s the message and
>> from which thread it logged.
>>
>>
>>
>> level: ERROR
>>
>>
>>
>> logger: org.apache.ignite.internal.processors.cache.GridCacheIoManager
>>
>>
>>
>>
>> message: Failed processing message
>> [senderId=57ee6544-e0b3-45cc-bdb5-f3fd37d7db1e, msg=GridDhtPartitionsSingleMessage
>> [parts=null, partCntrs=null, client=false, compress=false, super=
>> GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId
>> [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0],
>> nodeId=57ee6544, evt=NODE_JOINED], lastVer=GridCacheVersion [topVer=0,
>> time=0, order=1536996016410, nodeOrder=0], flags=0, super=GridCacheMessage
>> [msgId=1, depInfo=null, err=null, skipPrepare=false, cacheId=0,
>> cacheId=0]]]] [IM_GROUP=fsy-pi-dt-ssam ]
>>
>>
>>
>> thread: sys-stripe-2-#3%null%
>>
>>
>>
>> *From:* Ilya Kasnacheev [mailto:ilya.kasnacheev@gmail.com]
>> *Sent:* Tuesday, September 25, 2018 11:11 AM
>> *To:* user@ignite.apache.org
>> *Subject:* Re: java.lang.NullPointerException in
>> GridDhtPartitionsExchangeFuture
>>
>>
>> This email is from an external source - exercise caution regarding links
>> and attachments.
>>
>> Hello!
>>
>>
>>
>> It's hard to say without reviewing logs, but it seems that there's some
>> inconsistency with regards to cache metadata on nodes.
>>
>>
>>
>> Regards,
>>
>> --
>>
>> Ilya Kasnacheev
>>
>>
>>
>>
>>
>> вт, 25 сент. 2018 г. в 0:13, HEWA WIDANA GAMAGE, SUBASH <
>> Subash.HewaWidanaGamage@fmr.com>:
>>
>> Hi all,
>>
>> We use Ignite 1.9.
>>
>>
>>
>> We could see this in our logs. All we do is cache.get() , cache.put()
>> operations. With this log being seen, is it possible for cache.put or
>> ignite.getOrCreateCache() method calling threads be blocked forever ?
>> (unfortunately we couldn’t get a thread dump to prove that, but from
>> application logs, it looks like it).
>>
>>
>>
>> java.lang.NullPointerException: null
>>
>> at org.apache.ignite.internal.
>> processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFutur
>> e.updatePartitionSingleMap(GridDhtPartitionsExchangeFuture.java:1446)
>>
>> at org.apache.ignite.internal.
>> processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFutur
>> e.processMessage(GridDhtPartitionsExchangeFuture.java:1199)
>>
>> at org.apache.ignite.internal.
>> processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFutur
>> e.access$100(GridDhtPartitionsExchangeFuture.java:86)
>>
>> at org.apache.ignite.internal.
>> processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFutur
>> e$1.apply(GridDhtPartitionsExchangeFuture.java:1167)
>>
>> at org.apache.ignite.internal.
>> processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFutur
>> e$1.apply(GridDhtPartitionsExchangeFuture.java:1155)
>>
>> at org.apache.ignite.internal.
>> util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
>>
>> at org.apache.ignite.internal.
>> util.future.GridFutureAdapter.listen(GridFutureAdapter.java:228)
>>
>> at org.apache.ignite.internal.
>> processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFutur
>> e.onReceive(GridDhtPartitionsExchangeFuture.java:1155)
>>
>> at org.apache.ignite.internal.processors.cache.
>> GridCachePartitionExchangeManager.processSinglePartitionUpdate(
>> GridCachePartitionExchangeManager.java:1304)
>>
>> at org.apache.ignite.internal.processors.cache.
>> GridCachePartitionExchangeManager.access$1200(
>> GridCachePartitionExchangeManager.java:116)
>>
>> at org.apache.ignite.internal.processors.cache.
>> GridCachePartitionExchangeManager$2.onMessage(
>> GridCachePartitionExchangeManager.java:310)
>>
>> at org.apache.ignite.internal.processors.cache.
>> GridCachePartitionExchangeManager$2.onMessage(
>> GridCachePartitionExchangeManager.java:308)
>>
>> at org.apache.ignite.internal.processors.cache.
>> GridCachePartitionExchangeManager$MessageHandler.apply(
>> GridCachePartitionExchangeManager.java:1992)
>>
>> at org.apache.ignite.internal.processors.cache.
>> GridCachePartitionExchangeManager$MessageHandler.apply(
>> GridCachePartitionExchangeManager.java:1974)
>>
>> at org.apache.ignite.internal.processors.cache.
>> GridCacheIoManager.processMessage(GridCacheIoManager.java:827)
>>
>> at org.apache.ignite.internal.processors.cache.
>> GridCacheIoManager.onMessage0(GridCacheIoManager.java:369)
>>
>> at org.apache.ignite.internal.processors.cache.
>> GridCacheIoManager.handleMessage(GridCacheIoManager.java:293)
>>
>> at org.apache.ignite.internal.processors.cache.
>> GridCacheIoManager.access$000(GridCacheIoManager.java:95)
>>
>> at org.apache.ignite.internal.processors.cache.
>> GridCacheIoManager$1.onMessage(GridCacheIoManager.java:238)
>>
>> at org.apache.ignite.internal.managers.communication.
>> GridIoManager.invokeListener(GridIoManager.java:1222)
>>
>> at org.apache.ignite.internal.managers.communication.
>> GridIoManager.processRegularMessage0(GridIoManager.java:850)
>>
>> at org.apache.ignite.internal.managers.communication.
>> GridIoManager.access$2100(GridIoManager.java:108)
>>
>> at org.apache.ignite.internal.managers.communication.
>> GridIoManager$7.run(GridIoManager.java:790)
>>
>> at org.apache.ignite.internal.
>> util.StripedExecutor$Stripe.run(StripedExecutor.java:428)
>>
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>>
>>
--
Best regards,
Ivan Pavlukhin
Re: java.lang.NullPointerException in GridDhtPartitionsExchangeFuture
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
It's hard to answer these questions without thorough review of logs and 1.9
code, and I doubt anyone will volunteer to do that since 1.x branch does
not see any new development in Apache Ignite.
Regards,
--
Ilya Kasnacheev
ср, 26 сент. 2018 г. в 22:36, HEWA WIDANA GAMAGE, SUBASH <
Subash.HewaWidanaGamage@fmr.com>:
> Hi Kasnacheev,
>
> Thank you very much for the response..
>
>
>
> We use v1.9, because 2.x uses mandatory ignite native persistence(local
> disk) along with durable memory management(RAM), and no option for a **java
> heap only** cache storage. We wanted to keep away from storing anything in
> disk. Hence using 1.9 Ignite.
>
> Can you please clarify “I believe that cache in question is no longer
> consistent between nodes on metadata level” ?
>
>
>
> 1. Like what could have caused & under what conditions this
> inconsistent state between nodes on cache metadata occurred ?
>
>
>
> 2. And does 2.x fixes the problem?( Is there a way you can suggest
> to reproduce this issue, so that we can know for sure 2.x fixes the problem)
>
>
>
> It appeared from our logs that the cache.put threads hangs forever after
> this error. Can that be possible with this?
>
>
>
>
>
>
>
> *From:* Ilya Kasnacheev [mailto:ilya.kasnacheev@gmail.com]
> *Sent:* Wednesday, September 26, 2018 12:31 PM
> *To:* user@ignite.apache.org
> *Subject:* Re: java.lang.NullPointerException in
> GridDhtPartitionsExchangeFuture
>
>
> This email is from an external source - exercise caution regarding links
> and attachments.
>
> Hello!
>
>
>
> This is a bad error message. I believe that cache in question is no longer
> consistent between nodes on metadata level, and you can't fix that without
> full restart or at least dropping and recreating the cache.
>
> I don't think that you will get much support from community on 1.x since
> the focus has shifted to 2.x. Have you considered upgrading?
>
>
>
> Regards,
>
> --
>
> Ilya Kasnacheev
>
>
>
>
>
> ср, 26 сент. 2018 г. в 18:50, HEWA WIDANA GAMAGE, SUBASH <
> Subash.HewaWidanaGamage@fmr.com>:
>
> This is the only single error from ignite happened after JVM startup.
> Looks like I only posted the exception stack trace. Here’s the message and
> from which thread it logged.
>
>
>
> level: ERROR
>
>
>
> logger:
> org.apache.ignite.internal.processors.cache.GridCacheIoManager
>
>
>
> message: Failed processing message
> [senderId=57ee6544-e0b3-45cc-bdb5-f3fd37d7db1e,
> msg=GridDhtPartitionsSingleMessage [parts=null, partCntrs=null,
> client=false, compress=false, super=GridDhtPartitionsAbstractMessage
> [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=13, minorTopVer=0], nodeId=57ee6544, evt=NODE_JOINED],
> lastVer=GridCacheVersion [topVer=0, time=0, order=1536996016410,
> nodeOrder=0], flags=0, super=GridCacheMessage [msgId=1, depInfo=null,
> err=null, skipPrepare=false, cacheId=0, cacheId=0]]]]
> [IM_GROUP=fsy-pi-dt-ssam ]
>
>
>
> thread: sys-stripe-2-#3%null%
>
>
>
> *From:* Ilya Kasnacheev [mailto:ilya.kasnacheev@gmail.com]
> *Sent:* Tuesday, September 25, 2018 11:11 AM
> *To:* user@ignite.apache.org
> *Subject:* Re: java.lang.NullPointerException in
> GridDhtPartitionsExchangeFuture
>
>
> This email is from an external source - exercise caution regarding links
> and attachments.
>
> Hello!
>
>
>
> It's hard to say without reviewing logs, but it seems that there's some
> inconsistency with regards to cache metadata on nodes.
>
>
>
> Regards,
>
> --
>
> Ilya Kasnacheev
>
>
>
>
>
> вт, 25 сент. 2018 г. в 0:13, HEWA WIDANA GAMAGE, SUBASH <
> Subash.HewaWidanaGamage@fmr.com>:
>
> Hi all,
>
> We use Ignite 1.9.
>
>
>
> We could see this in our logs. All we do is cache.get() , cache.put()
> operations. With this log being seen, is it possible for cache.put or
> ignite.getOrCreateCache() method calling threads be blocked forever ?
> (unfortunately we couldn’t get a thread dump to prove that, but from
> application logs, it looks like it).
>
>
>
> java.lang.NullPointerException: null
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.updatePartitionSingleMap(GridDhtPartitionsExchangeFuture.java:1446)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processMessage(GridDhtPartitionsExchangeFuture.java:1199)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$100(GridDhtPartitionsExchangeFuture.java:86)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$1.apply(GridDhtPartitionsExchangeFuture.java:1167)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$1.apply(GridDhtPartitionsExchangeFuture.java:1155)
>
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
>
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:228)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onReceive(GridDhtPartitionsExchangeFuture.java:1155)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processSinglePartitionUpdate(GridCachePartitionExchangeManager.java:1304)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1200(GridCachePartitionExchangeManager.java:116)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:310)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:308)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:1992)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:1974)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:827)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:369)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:293)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:95)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:238)
>
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1222)
>
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:850)
>
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:108)
>
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$7.run(GridIoManager.java:790)
>
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:428)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
>
RE: java.lang.NullPointerException in GridDhtPartitionsExchangeFuture
Posted by "HEWA WIDANA GAMAGE, SUBASH" <Su...@fmr.com>.
Hi Kasnacheev,
Thank you very much for the response..
We use v1.9, because 2.x uses mandatory ignite native persistence(local disk) along with durable memory management(RAM), and no option for a **java heap only** cache storage. We wanted to keep away from storing anything in disk. Hence using 1.9 Ignite.
Can you please clarify “I believe that cache in question is no longer consistent between nodes on metadata level” ?
1. Like what could have caused & under what conditions this inconsistent state between nodes on cache metadata occurred ?
2. And does 2.x fixes the problem?( Is there a way you can suggest to reproduce this issue, so that we can know for sure 2.x fixes the problem)
It appeared from our logs that the cache.put threads hangs forever after this error. Can that be possible with this?
From: Ilya Kasnacheev [mailto:ilya.kasnacheev@gmail.com]
Sent: Wednesday, September 26, 2018 12:31 PM
To: user@ignite.apache.org
Subject: Re: java.lang.NullPointerException in GridDhtPartitionsExchangeFuture
This email is from an external source - exercise caution regarding links and attachments.
Hello!
This is a bad error message. I believe that cache in question is no longer consistent between nodes on metadata level, and you can't fix that without full restart or at least dropping and recreating the cache.
I don't think that you will get much support from community on 1.x since the focus has shifted to 2.x. Have you considered upgrading?
Regards,
--
Ilya Kasnacheev
ср, 26 сент. 2018 г. в 18:50, HEWA WIDANA GAMAGE, SUBASH <Su...@fmr.com>>:
This is the only single error from ignite happened after JVM startup. Looks like I only posted the exception stack trace. Here’s the message and from which thread it logged.
level: ERROR
logger: org.apache.ignite.internal.processors.cache.GridCacheIoManager
message: Failed processing message [senderId=57ee6544-e0b3-45cc-bdb5-f3fd37d7db1e, msg=GridDhtPartitionsSingleMessage [parts=null, partCntrs=null, client=false, compress=false, super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], nodeId=57ee6544, evt=NODE_JOINED], lastVer=GridCacheVersion [topVer=0, time=0, order=1536996016410, nodeOrder=0], flags=0, super=GridCacheMessage [msgId=1, depInfo=null, err=null, skipPrepare=false, cacheId=0, cacheId=0]]]] [IM_GROUP=fsy-pi-dt-ssam ]
thread: sys-stripe-2-#3%null%
From: Ilya Kasnacheev [mailto:ilya.kasnacheev@gmail.com<ma...@gmail.com>]
Sent: Tuesday, September 25, 2018 11:11 AM
To: user@ignite.apache.org<ma...@ignite.apache.org>
Subject: Re: java.lang.NullPointerException in GridDhtPartitionsExchangeFuture
This email is from an external source - exercise caution regarding links and attachments.
Hello!
It's hard to say without reviewing logs, but it seems that there's some inconsistency with regards to cache metadata on nodes.
Regards,
--
Ilya Kasnacheev
вт, 25 сент. 2018 г. в 0:13, HEWA WIDANA GAMAGE, SUBASH <Su...@fmr.com>>:
Hi all,
We use Ignite 1.9.
We could see this in our logs. All we do is cache.get() , cache.put() operations. With this log being seen, is it possible for cache.put or ignite.getOrCreateCache() method calling threads be blocked forever ? (unfortunately we couldn’t get a thread dump to prove that, but from application logs, it looks like it).
java.lang.NullPointerException: null
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.updatePartitionSingleMap(GridDhtPartitionsExchangeFuture.java:1446)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processMessage(GridDhtPartitionsExchangeFuture.java:1199)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$100(GridDhtPartitionsExchangeFuture.java:86)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$1.apply(GridDhtPartitionsExchangeFuture.java:1167)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$1.apply(GridDhtPartitionsExchangeFuture.java:1155)
at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
at org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:228)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onReceive(GridDhtPartitionsExchangeFuture.java:1155)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processSinglePartitionUpdate(GridCachePartitionExchangeManager.java:1304)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1200(GridCachePartitionExchangeManager.java:116)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:310)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:308)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:1992)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:1974)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:827)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:369)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:293)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:95)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:238)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1222)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:850)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:108)
at org.apache.ignite.internal.managers.communication.GridIoManager$7.run(GridIoManager.java:790)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:428)
at java.lang.Thread.run(Thread.java:748)
Re: java.lang.NullPointerException in GridDhtPartitionsExchangeFuture
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
This is a bad error message. I believe that cache in question is no longer
consistent between nodes on metadata level, and you can't fix that without
full restart or at least dropping and recreating the cache.
I don't think that you will get much support from community on 1.x since
the focus has shifted to 2.x. Have you considered upgrading?
Regards,
--
Ilya Kasnacheev
ср, 26 сент. 2018 г. в 18:50, HEWA WIDANA GAMAGE, SUBASH <
Subash.HewaWidanaGamage@fmr.com>:
> This is the only single error from ignite happened after JVM startup.
> Looks like I only posted the exception stack trace. Here’s the message and
> from which thread it logged.
>
>
>
> level: ERROR
>
>
>
> logger:
> org.apache.ignite.internal.processors.cache.GridCacheIoManager
>
>
>
> message: Failed processing message
> [senderId=57ee6544-e0b3-45cc-bdb5-f3fd37d7db1e,
> msg=GridDhtPartitionsSingleMessage [parts=null, partCntrs=null,
> client=false, compress=false, super=GridDhtPartitionsAbstractMessage
> [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=13, minorTopVer=0], nodeId=57ee6544, evt=NODE_JOINED],
> lastVer=GridCacheVersion [topVer=0, time=0, order=1536996016410,
> nodeOrder=0], flags=0, super=GridCacheMessage [msgId=1, depInfo=null,
> err=null, skipPrepare=false, cacheId=0, cacheId=0]]]]
> [IM_GROUP=fsy-pi-dt-ssam ]
>
>
>
> thread: sys-stripe-2-#3%null%
>
>
>
> *From:* Ilya Kasnacheev [mailto:ilya.kasnacheev@gmail.com]
> *Sent:* Tuesday, September 25, 2018 11:11 AM
> *To:* user@ignite.apache.org
> *Subject:* Re: java.lang.NullPointerException in
> GridDhtPartitionsExchangeFuture
>
>
> This email is from an external source - exercise caution regarding links
> and attachments.
>
> Hello!
>
>
>
> It's hard to say without reviewing logs, but it seems that there's some
> inconsistency with regards to cache metadata on nodes.
>
>
>
> Regards,
>
> --
>
> Ilya Kasnacheev
>
>
>
>
>
> вт, 25 сент. 2018 г. в 0:13, HEWA WIDANA GAMAGE, SUBASH <
> Subash.HewaWidanaGamage@fmr.com>:
>
> Hi all,
>
> We use Ignite 1.9.
>
>
>
> We could see this in our logs. All we do is cache.get() , cache.put()
> operations. With this log being seen, is it possible for cache.put or
> ignite.getOrCreateCache() method calling threads be blocked forever ?
> (unfortunately we couldn’t get a thread dump to prove that, but from
> application logs, it looks like it).
>
>
>
> java.lang.NullPointerException: null
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.updatePartitionSingleMap(GridDhtPartitionsExchangeFuture.java:1446)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processMessage(GridDhtPartitionsExchangeFuture.java:1199)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$100(GridDhtPartitionsExchangeFuture.java:86)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$1.apply(GridDhtPartitionsExchangeFuture.java:1167)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$1.apply(GridDhtPartitionsExchangeFuture.java:1155)
>
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
>
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:228)
>
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onReceive(GridDhtPartitionsExchangeFuture.java:1155)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processSinglePartitionUpdate(GridCachePartitionExchangeManager.java:1304)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1200(GridCachePartitionExchangeManager.java:116)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:310)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:308)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:1992)
>
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:1974)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:827)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:369)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:293)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:95)
>
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:238)
>
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1222)
>
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:850)
>
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:108)
>
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$7.run(GridIoManager.java:790)
>
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:428)
>
> at java.lang.Thread.run(Thread.java:748)
>
>
>
>
RE: java.lang.NullPointerException in GridDhtPartitionsExchangeFuture
Posted by "HEWA WIDANA GAMAGE, SUBASH" <Su...@fmr.com>.
This is the only single error from ignite happened after JVM startup. Looks like I only posted the exception stack trace. Here’s the message and from which thread it logged.
level: ERROR
logger: org.apache.ignite.internal.processors.cache.GridCacheIoManager
message: Failed processing message [senderId=57ee6544-e0b3-45cc-bdb5-f3fd37d7db1e, msg=GridDhtPartitionsSingleMessage [parts=null, partCntrs=null, client=false, compress=false, super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], nodeId=57ee6544, evt=NODE_JOINED], lastVer=GridCacheVersion [topVer=0, time=0, order=1536996016410, nodeOrder=0], flags=0, super=GridCacheMessage [msgId=1, depInfo=null, err=null, skipPrepare=false, cacheId=0, cacheId=0]]]] [IM_GROUP=fsy-pi-dt-ssam ]
thread: sys-stripe-2-#3%null%
From: Ilya Kasnacheev [mailto:ilya.kasnacheev@gmail.com]
Sent: Tuesday, September 25, 2018 11:11 AM
To: user@ignite.apache.org
Subject: Re: java.lang.NullPointerException in GridDhtPartitionsExchangeFuture
This email is from an external source - exercise caution regarding links and attachments.
Hello!
It's hard to say without reviewing logs, but it seems that there's some inconsistency with regards to cache metadata on nodes.
Regards,
--
Ilya Kasnacheev
вт, 25 сент. 2018 г. в 0:13, HEWA WIDANA GAMAGE, SUBASH <Su...@fmr.com>>:
Hi all,
We use Ignite 1.9.
We could see this in our logs. All we do is cache.get() , cache.put() operations. With this log being seen, is it possible for cache.put or ignite.getOrCreateCache() method calling threads be blocked forever ? (unfortunately we couldn’t get a thread dump to prove that, but from application logs, it looks like it).
java.lang.NullPointerException: null
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.updatePartitionSingleMap(GridDhtPartitionsExchangeFuture.java:1446)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processMessage(GridDhtPartitionsExchangeFuture.java:1199)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$100(GridDhtPartitionsExchangeFuture.java:86)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$1.apply(GridDhtPartitionsExchangeFuture.java:1167)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$1.apply(GridDhtPartitionsExchangeFuture.java:1155)
at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:271)
at org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:228)
at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onReceive(GridDhtPartitionsExchangeFuture.java:1155)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processSinglePartitionUpdate(GridCachePartitionExchangeManager.java:1304)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1200(GridCachePartitionExchangeManager.java:116)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:310)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:308)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:1992)
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:1974)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:827)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:369)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:293)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:95)
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:238)
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1222)
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:850)
at org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:108)
at org.apache.ignite.internal.managers.communication.GridIoManager$7.run(GridIoManager.java:790)
at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:428)
at java.lang.Thread.run(Thread.java:748)