You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Sergey Sergeev <we...@gmail.com> on 2018/02/18 15:24:58 UTC
And again... Failed to get page IO instance (page content is corrupted)
Hi Igniters
I can't defeat the bug. It is reproducing on ignite version community 2.3.0
and on trial professional 2.3.2 editions.
The Cache has configuration like:
final CacheConfiguration<Pk, Message> cacheConfiguration
= new CacheConfiguration<>(getEntityClass().getSimpleName());
cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
cacheConfiguration.setBackups(0);
cacheConfiguration.setRebalanceMode(CacheRebalanceMode.ASYNC);
cacheConfiguration.setDataRegionName("messagePartSentRegion");
cacheConfiguration.setEagerTtl(true);
cacheConfiguration.setManagementEnabled(true);
cacheConfiguration.setStatisticsEnabled(true);
IgniteCache<Pk, Message> cache = ignite.getOrCreateCache(cacheConfiguration)
.withExpiryPolicy(new CreatedExpiryPolicy(new Duration(DAYS, 7)));
The data storage has configuration:
...
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="walMode" value="LOG_ONLY" />
<property name="writeThrottlingEnabled" value="true" />
<property name="metricsEnabled" value="true" />
<property name="defaultDataRegionConfiguration">
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="metricsEnabled" value="true" />
<property name="persistenceEnabled" value="true" />
<!-- 2Gb maximum size -->
<property name="maxSize" value="#{2L * 1024 * 1024 * 1024}"/>
</bean>
</property>
<property name="dataRegionConfigurations">
<list>
...
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="metricsEnabled" value="true" />
<property name="name" value="messagePartSentRegion" />
*<property name="persistenceEnabled" value="true" />*
<!-- 2Gb maximum size -->
<property name="maxSize" value="#{2L * 1024 * 1024
* 1024}"/>
</bean>
...
</list>
</property>
</bean>
</property>
...
After 7 days, I've exception:
ERROR GridCacheIoManager - Failed to process message
[senderId=e9e71d4c-e1dc-4647-8fd4-72fb281907a4, messageType=class
o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicFullUpdateRequest]
org.apache.ignite.IgniteException: Runtime failure on bounds:
[lower=null, upper=PendingRow []]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:954)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:933)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:833)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessageProcessed(GridCacheIoManager.java:1099)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1072)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
~[ignite-core-2.3.2.jar:2.3.2]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.lang.IllegalStateException: Failed to get page IO
instance (page content is corrupted)
at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4441)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:910)
~[ignite-core-2.3.2.jar:2.3.2]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942)
~[ignite-core-2.3.2.jar:2.3.2]
... 17 more
The description of such a problem:
http://apache-ignite-users.70518.x6.nabble.com/Partition-eviction-failed-this-can-cause-grid-hang-Caused-by-java-lang-IllegalStateException-Failed--td19122.html
Issues:
https://issues.apache.org/jira/browse/IGNITE-6423
https://issues.apache.org/jira/browse/IGNITE-7278
And... When do you plan to release the 2.4.0 version? May be fix?
--
Best regards,
Sergey S. Sergeev
exit(0);
Re: And again... Failed to get page IO instance (page content is corrupted)
Posted by Andrey Mashenkov <an...@gmail.com>.
Sergey, Arseny,
We've fixed a number of similar issues in 2.4 Ignite version.
However, this one, ExpiryPolicy with persistence still buggy. Here is a
ticket for it [1].
The ticket has a PR that should fix this issue, but it broke backward
compatibility
as TTLManager, who responsible for expired entries eviction, can't restore
data saved by previous Ignite versions.
So, TTL manager will not be able to evict persisted entries in background
after restart until entry will be requested.
[1] https://issues.apache.org/jira/browse/IGNITE-5874
On Sun, Feb 18, 2018 at 6:24 PM, Sergey Sergeev <we...@gmail.com>
wrote:
> Hi Igniters
>
> I can't defeat the bug. It is reproducing on ignite version community
> 2.3.0 and on trial professional 2.3.2 editions.
>
> The Cache has configuration like:
>
> final CacheConfiguration<Pk, Message> cacheConfiguration
> = new CacheConfiguration<>(getEntityClass().getSimpleName());
>
> cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
> cacheConfiguration.setBackups(0);
> cacheConfiguration.setRebalanceMode(CacheRebalanceMode.ASYNC);
> cacheConfiguration.setDataRegionName("messagePartSentRegion");
> cacheConfiguration.setEagerTtl(true);
> cacheConfiguration.setManagementEnabled(true);
> cacheConfiguration.setStatisticsEnabled(true);
>
> IgniteCache<Pk, Message> cache = ignite.getOrCreateCache(cacheConfiguration)
> .withExpiryPolicy(new CreatedExpiryPolicy(new Duration(DAYS, 7)));
>
> The data storage has configuration:
> ...
> <property name="dataStorageConfiguration">
> <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
> <property name="walMode" value="LOG_ONLY" />
> <property name="writeThrottlingEnabled" value="true" />
> <property name="metricsEnabled" value="true" />
> <property name="defaultDataRegionConfiguration">
> <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property name="metricsEnabled" value="true" />
> <property name="persistenceEnabled" value="true" />
> <!-- 2Gb maximum size -->
> <property name="maxSize" value="#{2L * 1024 * 1024 * 1024}"/>
> </bean>
> </property>
> <property name="dataRegionConfigurations">
> <list>
> ...
>
> <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property name="metricsEnabled" value="true" />
> <property name="name" value="messagePartSentRegion" />
> *<property name="persistenceEnabled" value="true" />*
> <!-- 2Gb maximum size -->
> <property name="maxSize" value="#{2L * 1024 * 1024 * 1024}"/>
> </bean>
> ...
> </list>
> </property>
> </bean>
> </property>
> ...
>
> After 7 days, I've exception:
>
> ERROR GridCacheIoManager - Failed to process message [senderId=e9e71d4c-e1dc-4647-8fd4-72fb281907a4, messageType=class o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicFullUpdateRequest]
> org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null, upper=PendingRow []]
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessageProcessed(GridCacheIoManager.java:1099) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1072) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) ~[ignite-core-2.3.2.jar:2.3.2]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
> Caused by: java.lang.IllegalStateException: Failed to get page IO instance (page content is corrupted)
> at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4441) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:910) ~[ignite-core-2.3.2.jar:2.3.2]
> at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.2.jar:2.3.2]
> ... 17 more
>
>
> The description of such a problem:
> http://apache-ignite-users.70518.x6.nabble.com/Partition-
> eviction-failed-this-can-cause-grid-hang-Caused-by-java-lang-
> IllegalStateException-Failed--td19122.html
>
> Issues:
> https://issues.apache.org/jira/browse/IGNITE-6423
> https://issues.apache.org/jira/browse/IGNITE-7278
>
> And... When do you plan to release the 2.4.0 version? May be fix?
>
>
> --
> Best regards,
> Sergey S. Sergeev
> exit(0);
>
--
Best regards,
Andrey V. Mashenkov
Re: And again... Failed to get page IO instance (page content is corrupted)
Posted by Sergey Sergeev <we...@gmail.com>.
Hi, Arseny
We are using ext4 with defaults,noatime,lazytime options on Virtuozzo Linux
release 7.3.
We have refactored parts of logic and abandoned the use of any expired
policies((
On Thu, Mar 15, 2018 at 7:34 PM, Arseny Kovalchuk <
arseny.kovalchuk@synesis.ru> wrote:
> Hi guys.
>
> I've got a reproducer that can be related. See comments
> http://apache-ignite-users.70518.x6.nabble.com/Partition-eviction-failed-
> this-can-cause-grid-hang-Caused-by-java-lang-
> IllegalStateException-Failed--tp19122p20524.html
>
> Sergey Sergeev, just for reference, what kind of file system do you use
> with Ignite's persistence?
>
>
> Arseny Kovalchuk
>
> Senior Software Engineer at Synesis
> skype: arseny.kovalchuk
> mobile: +375 (29) 666-16-16
> LinkedIn Profile <http://www.linkedin.com/in/arsenykovalchuk/en>
>
> On 9 March 2018 at 12:31, Sergey Sergeev <we...@gmail.com> wrote:
>
>> Hi Mikhail,
>>
>> Unfortunately, the problem has repeated itself on ignite-core-2.3.3
>>
>> 27.02.18 00:27:55 ERROR GridCacheIoManager - Failed to process message
>> [senderId=8f99c887-cd4b-4c38-a649-ca430040d535, messageType=class
>> o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtom
>> icUpdateResponse]
>> org.apache.ignite.IgniteException: Runtime failure on bounds:
>> [lower=null, upper=PendingRow []]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
>> apManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.
>> *GridCacheTtlManager.expire*(GridCacheTtlManager.java:197)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheUtils.u
>> nwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheIoManag
>> er.onMessageProcessed(GridCacheIoManager.java:1099)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheIoManag
>> er.processMessage(GridCacheIoManager.java:1072)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheIoManag
>> er.onMessage0(GridCacheIoManager.java:579) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheIoManag
>> er.handleMessage(GridCacheIoManager.java:378)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheIoManag
>> er.handleMessage(GridCacheIoManager.java:304)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheIoManag
>> er.access$100(GridCacheIoManager.java:99) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheIoManag
>> er$1.onMessage(GridCacheIoManager.java:293)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.managers.communication.GridIoMana
>> ger.invokeListener(GridIoManager.java:1555)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.managers.communication.GridIoMana
>> ger.processRegularMessage0(GridIoManager.java:1183)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.managers.communication.GridIoMana
>> ger.access$4200(GridIoManager.java:126) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.managers.communication.GridIoMana
>> ger$9.run(GridIoManager.java:1090) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
>> Caused by: java.lang.IllegalStateException: Failed to get page IO
>> instance (page content is corrupted)
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .io.IOVersions.forVersion(IOVersions.java:83)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.Cach
>> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.Cach
>> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.tree.PendingEntr
>> iesTree.getRow(PendingEntriesTree.java:118)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.tree.PendingEntr
>> iesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree$ForwardCursor.init(BPlusTree.java:4441)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.findLowerUnbounded(BPlusTree.java:910)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3]
>> ... 17 more
>>
>>
>> And if we are reading from cache...
>>
>>
>> 27.02.18 00:27:56 ERROR MessagePartProcessingHandler - error processing
>> incoming messagePartDeliveryReceipt=MessagePartDeliveryReceipt{id=null,
>> abonent=7****, externalId='7586F73B'}
>> org.springframework.dao.DataAccessResourceFailureException: Runtime
>> failure on bounds: [lower=null, upper=PendingRow []]; nested exception is
>> class org.apache.ignite.IgniteException: Runtime failure on bounds:
>> [lower=null, upper=PendingRow []]
>> at com.a1s.sdp.data.dao.ignite.IgniteExceptionTranslator.transl
>> ateExceptionIfPossible(IgniteExceptionTranslator.java:35)
>> ~[core-data-1.5-SNAPSHOT.jar:3752]
>> at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:41)
>> ~[core-data-1.5-SNAPSHOT.jar:3752]
>> at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.fi
>> ndOne(BaseIgniteCassandraRepository.java:136)
>> ~[core-data-1.5-SNAPSHOT.jar:3752]
>> at com.a1s.sdp.common.service.TransportServiceImpl.processMessa
>> gePartDelivered(TransportServiceImpl.java:213)
>> ~[core-common-1.5-SNAPSHOT.jar:3752]
>> at com.a1s.sdp.common.service.TransportServiceImpl.messagePartD
>> elivered(TransportServiceImpl.java:202) ~[core-common-1.5-SNAPSHOT.jar
>> :3752]
>> at com.a1s.sdp.common.transport.handler.MessagePartProcessingHa
>> ndler.lambda$channelReadMessagePartDeliveryReceipt$1(MessagePartProcessingHandler.java:91)
>> ~[core-common-1.5-SNAPSHOT.jar:3752]
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> ~[?:1.8.0_151]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> ~[?:1.8.0_151]
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> ~[?:1.8.0_151]
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> ~[?:1.8.0_151]
>> at java.lang.Thread.run(Thread.java:748) [?:*1.8.0_151*]
>> Caused by: org.apache.ignite.IgniteException: *Runtime failure on
>> bounds: [lower=null, upper=PendingRow []]*
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.
>> *IgniteCacheOffheapManagerImpl.**expire*(IgniteCacheOffheapManagerImpl.java:979)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheTtlMana
>> ger.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheUtils.u
>> nwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheGateway
>> .leaveNoLock(GridCacheGateway.java:240) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheGateway
>> .leave(GridCacheGateway.java:225) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GatewayProtected
>> CacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GatewayProtected
>> CacheProxy.get(GatewayProtectedCacheProxy.java:608)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.la
>> mbda$findOne$7(BaseIgniteCassandraRepository.java:136)
>> ~[core-data-1.5-SNAPSHOT.jar:3752]
>> at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39)
>> ~[core-data-1.5-SNAPSHOT.jar:3752]
>> ... 9 more
>> Caused by: java.lang.IllegalStateException: Failed to get page IO
>> instance (page content is corrupted)
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .io.IOVersions.forVersion(IOVersions.java:83)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.Cach
>> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.Cach
>> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.tree.PendingEntr
>> iesTree.getRow(PendingEntriesTree.java:118)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.tree.PendingEntr
>> iesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree$ForwardCursor.init(BPlusTree.java:4441)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.findLowerUnbounded(BPlusTree.java:910)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.
>> *IgniteCacheOffheapManagerImpl.**expire*(IgniteCacheOffheapManagerImpl.java:979)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheTtlMana
>> ger.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheUtils.u
>> nwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheGateway
>> .leaveNoLock(GridCacheGateway.java:240) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GridCacheGateway
>> .leave(GridCacheGateway.java:225) ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GatewayProtected
>> CacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at org.apache.ignite.internal.processors.cache.GatewayProtected
>> CacheProxy.get(GatewayProtectedCacheProxy.java:608)
>> ~[ignite-core-2.3.3.jar:2.3.3]
>> at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.la
>> mbda$findOne$7(BaseIgniteCassandraRepository.java:136)
>> ~[core-data-1.5-SNAPSHOT.jar:3752]
>> at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39)
>> ~[core-data-1.5-SNAPSHOT.jar:3752]
>> ... 9 more
>>
>>
>> The attachment (https://yadi.sk/d/u7fSK8Y53SpCpA) contains a part of
>> cache obtained from node of cluster - /var/ignite/***/db/node00-66fd
>> 86f5-4e2d-4f1b-8fe1-993bf5937d99/cache-MessagePartSent.
>> This cache contains objects of class MessagePartSent:
>>
>> @Table("message_part_sent_external")
>> public final class MessagePartSent {
>> @Id
>> private MessagePartSentPk id;
>> private MessagePartPk partId;
>>
>> public MessagePartSentPk getId() {
>> return id;
>> }
>>
>> public void setId(MessagePartSentPk id) {
>> this.id = id;
>> }
>>
>> public MessagePartPk getPartId() {
>> return partId;
>> }
>>
>> public void setPartId(MessagePartPk partId) {
>> this.partId = partId;
>> }
>>
>> @Override
>> public String toString() {
>> final StringBuilder sb = new StringBuilder("MessagePartSent{");
>> sb.append("id=").append(id);
>> sb.append(", partId=").append(partId);
>> sb.append('}');
>> return sb.toString();
>> }
>>
>> public static final class MessagePartSentPk implements Serializable {
>> private Integer linkId;
>> private String externalId;
>>
>> public MessagePartSentPk() {
>> //
>> }
>>
>> public MessagePartSentPk(Integer linkId, String externalId) {
>> this.linkId = linkId;
>> this.externalId = externalId;
>> }
>>
>> public Integer getLinkId() {
>> return linkId;
>> }
>>
>> public void setLinkId(Integer linkId) {
>> this.linkId = linkId;
>> }
>>
>> public String getExternalId() {
>> return externalId;
>> }
>>
>> public void setExternalId(String externalId) {
>> this.externalId = externalId;
>> }
>>
>> @Override
>> public boolean equals(Object o) {
>> if (this == o) return true;
>> if (!(o instanceof MessagePartSentPk)) return false;
>>
>> final MessagePartSentPk that = (MessagePartSentPk) o;
>>
>> if (externalId != null ? !externalId.equals(that.externalId)
>> : that.externalId != null) return false;
>> if (linkId != null ? !linkId.equals(that.linkId) :
>> that.linkId != null) return false;
>>
>> return true;
>> }
>>
>> @Override
>> public int hashCode() {
>> int result = linkId != null ? linkId.hashCode() : 0;
>> result = 31 * result + (externalId != null ?
>> externalId.hashCode() : 0);
>> return result;
>> }
>>
>> @Override
>> public String toString() {
>> final StringBuilder sb = new StringBuilder("MessagePartSent
>> Pk{");
>> sb.append("linkId=").append(linkId);
>> sb.append(", externalId='").append(externalId).append('\'');
>> sb.append('}');
>> return sb.toString();
>> }
>> }
>>
>> }
>>
>>
>> public final class MessagePartPk implements Serializable {
>> private UUID transactionId;
>> private Short partNumber;
>>
>> public MessagePartPk() {
>> //default
>> }
>>
>> public MessagePartPk(UUID transactionId, Short partNumber) {
>> this.transactionId = transactionId;
>> this.partNumber = partNumber;
>> }
>>
>> public UUID getTransactionId() {
>> return transactionId;
>> }
>>
>> public void setTransactionId(UUID transactionId) {
>> this.transactionId = transactionId;
>> }
>>
>> public Short getPartNumber() {
>> return partNumber;
>> }
>>
>> public void setPartNumber(Short partNumber) {
>> this.partNumber = partNumber;
>> }
>>
>> @Override
>> public boolean equals(Object o) {
>> if (this == o) return true;
>> if (!(o instanceof MessagePartPk)) return false;
>>
>> final MessagePartPk messagePartPk = (MessagePartPk) o;
>>
>> if (partNumber != null ? !partNumber.equals(messagePartPk.partNumber)
>> :
>> messagePartPk.partNumber != null) return false;
>> if (transactionId != null ? !transactionId.equals(messagePartPk.transactionId)
>> :
>> messagePartPk.transactionId != null)
>> return false;
>>
>> return true;
>> }
>>
>> @Override
>> public int hashCode() {
>> int result = transactionId != null ? transactionId.hashCode() : 0;
>> result = 31 * result + (partNumber != null ?
>> partNumber.hashCode() : 0);
>> return result;
>> }
>>
>> @Override
>> public String toString() {
>> final StringBuilder sb = new StringBuilder("MessagePartPk{");
>> sb.append("transactionId=").append(transactionId);
>> sb.append(", partNumber=").append(partNumber);
>> sb.append('}');
>> return sb.toString();
>> }
>>
>> On Thu, Mar 8, 2018 at 11:03 PM, Dmitriy Setrakyan <dsetrakyan@apache.org
>> > wrote:
>>
>>> Hi Sergey,
>>>
>>> The 2.4 release is about to be voted for. You can use the RC1 in the
>>> mean time:
>>> http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-A
>>> pache-Ignite-2-4-0-RC1-td27687.html
>>>
>>> D.
>>>
>>> On Mon, Feb 19, 2018 at 6:43 AM, Mikhail <mi...@gmail.com>
>>> wrote:
>>>
>>>> Hi Sergey,
>>>>
>>>> The release of 2.4 should be soon, in a week or couple, however,
>>>> there's no
>>>> strong schedule for Apache releases.
>>>>
>>>> Could you please share a reproducer for the issue? Might be you can
>>>> share a
>>>> storage on which the issue can be reproduced?
>>>>
>>>> Thanks,
>>>> Mike.
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>
>>>
>>>
>>
>>
>> --
>> Best regards,
>> Sergey S. Sergeev (aka SSSerj)
>> Notes: https://twitter.com/ssserj
>> exit(0);
>>
>
>
--
Best regards,
Sergey S. Sergeev (aka SSSerj)
exit(0);
Re: And again... Failed to get page IO instance (page content is corrupted)
Posted by Arseny Kovalchuk <ar...@synesis.ru>.
Hi guys.
I've got a reproducer that can be related. See comments
http://apache-ignite-users.70518.x6.nabble.com/Partition-eviction-failed-this-can-cause-grid-hang-Caused-by-java-lang-IllegalStateException-Failed--tp19122p20524.html
Sergey Sergeev, just for reference, what kind of file system do you use
with Ignite's persistence?
Arseny Kovalchuk
Senior Software Engineer at Synesis
skype: arseny.kovalchuk
mobile: +375 (29) 666-16-16
LinkedIn Profile <http://www.linkedin.com/in/arsenykovalchuk/en>
On 9 March 2018 at 12:31, Sergey Sergeev <we...@gmail.com> wrote:
> Hi Mikhail,
>
> Unfortunately, the problem has repeated itself on ignite-core-2.3.3
>
> 27.02.18 00:27:55 ERROR GridCacheIoManager - Failed to process message
> [senderId=8f99c887-cd4b-4c38-a649-ca430040d535, messageType=class
> o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtom
> icUpdateResponse]
> org.apache.ignite.IgniteException: Runtime failure on bounds:
> [lower=null, upper=PendingRow []]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
> apManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.
> *GridCacheTtlManager.expire*(GridCacheTtlManager.java:197)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheUtils.
> unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManag
> er.onMessageProcessed(GridCacheIoManager.java:1099)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManag
> er.processMessage(GridCacheIoManager.java:1072)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManag
> er.onMessage0(GridCacheIoManager.java:579) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManag
> er.handleMessage(GridCacheIoManager.java:378)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManag
> er.handleMessage(GridCacheIoManager.java:304)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManag
> er.access$100(GridCacheIoManager.java:99) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheIoManag
> er$1.onMessage(GridCacheIoManager.java:293) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.managers.communication.GridIoMana
> ger.invokeListener(GridIoManager.java:1555) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.managers.communication.GridIoMana
> ger.processRegularMessage0(GridIoManager.java:1183)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.managers.communication.GridIoMana
> ger.access$4200(GridIoManager.java:126) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.managers.communication.GridIoMana
> ger$9.run(GridIoManager.java:1090) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
> Caused by: java.lang.IllegalStateException: Failed to get page IO
> instance (page content is corrupted)
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .io.IOVersions.forVersion(IOVersions.java:83)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.Cach
> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.Cach
> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.tree.PendingEntr
> iesTree.getRow(PendingEntriesTree.java:118) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.tree.PendingEntr
> iesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .BPlusTree$ForwardCursor.init(BPlusTree.java:4441)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .BPlusTree.findLowerUnbounded(BPlusTree.java:910)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3]
> ... 17 more
>
>
> And if we are reading from cache...
>
>
> 27.02.18 00:27:56 ERROR MessagePartProcessingHandler - error processing
> incoming messagePartDeliveryReceipt=MessagePartDeliveryReceipt{id=null,
> abonent=7****, externalId='7586F73B'}
> org.springframework.dao.DataAccessResourceFailureException: Runtime
> failure on bounds: [lower=null, upper=PendingRow []]; nested exception is
> class org.apache.ignite.IgniteException: Runtime failure on bounds:
> [lower=null, upper=PendingRow []]
> at com.a1s.sdp.data.dao.ignite.IgniteExceptionTranslator.transl
> ateExceptionIfPossible(IgniteExceptionTranslator.java:35)
> ~[core-data-1.5-SNAPSHOT.jar:3752]
> at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:41)
> ~[core-data-1.5-SNAPSHOT.jar:3752]
> at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.fi
> ndOne(BaseIgniteCassandraRepository.java:136)
> ~[core-data-1.5-SNAPSHOT.jar:3752]
> at com.a1s.sdp.common.service.TransportServiceImpl.processMessa
> gePartDelivered(TransportServiceImpl.java:213)
> ~[core-common-1.5-SNAPSHOT.jar:3752]
> at com.a1s.sdp.common.service.TransportServiceImpl.messagePartD
> elivered(TransportServiceImpl.java:202) ~[core-common-1.5-SNAPSHOT.jar
> :3752]
> at com.a1s.sdp.common.transport.handler.MessagePartProcessingHa
> ndler.lambda$channelReadMessagePartDeliveryReceipt$1(MessagePartProcessingHandler.java:91)
> ~[core-common-1.5-SNAPSHOT.jar:3752]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_151]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_151]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_151]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_151]
> at java.lang.Thread.run(Thread.java:748) [?:*1.8.0_151*]
> Caused by: org.apache.ignite.IgniteException: *Runtime failure on bounds:
> [lower=null, upper=PendingRow []]*
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.
> *IgniteCacheOffheapManagerImpl.**expire*(IgniteCacheOffheapManagerImpl.java:979)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheTtlMana
> ger.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheUtils.
> unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheGateway
> .leaveNoLock(GridCacheGateway.java:240) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheGateway
> .leave(GridCacheGateway.java:225) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GatewayProtected
> CacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GatewayProtected
> CacheProxy.get(GatewayProtectedCacheProxy.java:608)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.la
> mbda$findOne$7(BaseIgniteCassandraRepository.java:136)
> ~[core-data-1.5-SNAPSHOT.jar:3752]
> at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39)
> ~[core-data-1.5-SNAPSHOT.jar:3752]
> ... 9 more
> Caused by: java.lang.IllegalStateException: Failed to get page IO
> instance (page content is corrupted)
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .io.IOVersions.forVersion(IOVersions.java:83)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.Cach
> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.Cach
> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.tree.PendingEntr
> iesTree.getRow(PendingEntriesTree.java:118) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.tree.PendingEntr
> iesTree.getRow(PendingEntriesTree.java:31) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .BPlusTree$ForwardCursor.init(BPlusTree.java:4441)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .BPlusTree.findLowerUnbounded(BPlusTree.java:910)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.persistence.tree
> .BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.
> *IgniteCacheOffheapManagerImpl.**expire*(IgniteCacheOffheapManagerImpl.java:979)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheTtlMana
> ger.expire(GridCacheTtlManager.java:197) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheUtils.
> unwindEvicts(GridCacheUtils.java:833) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheGateway
> .leaveNoLock(GridCacheGateway.java:240) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GridCacheGateway
> .leave(GridCacheGateway.java:225) ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GatewayProtected
> CacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at org.apache.ignite.internal.processors.cache.GatewayProtected
> CacheProxy.get(GatewayProtectedCacheProxy.java:608)
> ~[ignite-core-2.3.3.jar:2.3.3]
> at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.la
> mbda$findOne$7(BaseIgniteCassandraRepository.java:136)
> ~[core-data-1.5-SNAPSHOT.jar:3752]
> at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39)
> ~[core-data-1.5-SNAPSHOT.jar:3752]
> ... 9 more
>
>
> The attachment (https://yadi.sk/d/u7fSK8Y53SpCpA) contains a part of
> cache obtained from node of cluster - /var/ignite/***/db/node00-
> 66fd86f5-4e2d-4f1b-8fe1-993bf5937d99/cache-MessagePartSent.
> This cache contains objects of class MessagePartSent:
>
> @Table("message_part_sent_external")
> public final class MessagePartSent {
> @Id
> private MessagePartSentPk id;
> private MessagePartPk partId;
>
> public MessagePartSentPk getId() {
> return id;
> }
>
> public void setId(MessagePartSentPk id) {
> this.id = id;
> }
>
> public MessagePartPk getPartId() {
> return partId;
> }
>
> public void setPartId(MessagePartPk partId) {
> this.partId = partId;
> }
>
> @Override
> public String toString() {
> final StringBuilder sb = new StringBuilder("MessagePartSent{");
> sb.append("id=").append(id);
> sb.append(", partId=").append(partId);
> sb.append('}');
> return sb.toString();
> }
>
> public static final class MessagePartSentPk implements Serializable {
> private Integer linkId;
> private String externalId;
>
> public MessagePartSentPk() {
> //
> }
>
> public MessagePartSentPk(Integer linkId, String externalId) {
> this.linkId = linkId;
> this.externalId = externalId;
> }
>
> public Integer getLinkId() {
> return linkId;
> }
>
> public void setLinkId(Integer linkId) {
> this.linkId = linkId;
> }
>
> public String getExternalId() {
> return externalId;
> }
>
> public void setExternalId(String externalId) {
> this.externalId = externalId;
> }
>
> @Override
> public boolean equals(Object o) {
> if (this == o) return true;
> if (!(o instanceof MessagePartSentPk)) return false;
>
> final MessagePartSentPk that = (MessagePartSentPk) o;
>
> if (externalId != null ? !externalId.equals(that.externalId)
> : that.externalId != null) return false;
> if (linkId != null ? !linkId.equals(that.linkId) : that.linkId
> != null) return false;
>
> return true;
> }
>
> @Override
> public int hashCode() {
> int result = linkId != null ? linkId.hashCode() : 0;
> result = 31 * result + (externalId != null ?
> externalId.hashCode() : 0);
> return result;
> }
>
> @Override
> public String toString() {
> final StringBuilder sb = new StringBuilder("MessagePartSent
> Pk{");
> sb.append("linkId=").append(linkId);
> sb.append(", externalId='").append(externalId).append('\'');
> sb.append('}');
> return sb.toString();
> }
> }
>
> }
>
>
> public final class MessagePartPk implements Serializable {
> private UUID transactionId;
> private Short partNumber;
>
> public MessagePartPk() {
> //default
> }
>
> public MessagePartPk(UUID transactionId, Short partNumber) {
> this.transactionId = transactionId;
> this.partNumber = partNumber;
> }
>
> public UUID getTransactionId() {
> return transactionId;
> }
>
> public void setTransactionId(UUID transactionId) {
> this.transactionId = transactionId;
> }
>
> public Short getPartNumber() {
> return partNumber;
> }
>
> public void setPartNumber(Short partNumber) {
> this.partNumber = partNumber;
> }
>
> @Override
> public boolean equals(Object o) {
> if (this == o) return true;
> if (!(o instanceof MessagePartPk)) return false;
>
> final MessagePartPk messagePartPk = (MessagePartPk) o;
>
> if (partNumber != null ? !partNumber.equals(messagePartPk.partNumber)
> :
> messagePartPk.partNumber != null) return false;
> if (transactionId != null ? !transactionId.equals(messagePartPk.transactionId)
> :
> messagePartPk.transactionId != null)
> return false;
>
> return true;
> }
>
> @Override
> public int hashCode() {
> int result = transactionId != null ? transactionId.hashCode() : 0;
> result = 31 * result + (partNumber != null ? partNumber.hashCode()
> : 0);
> return result;
> }
>
> @Override
> public String toString() {
> final StringBuilder sb = new StringBuilder("MessagePartPk{");
> sb.append("transactionId=").append(transactionId);
> sb.append(", partNumber=").append(partNumber);
> sb.append('}');
> return sb.toString();
> }
>
> On Thu, Mar 8, 2018 at 11:03 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
>> Hi Sergey,
>>
>> The 2.4 release is about to be voted for. You can use the RC1 in the mean
>> time:
>> http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-
>> Apache-Ignite-2-4-0-RC1-td27687.html
>>
>> D.
>>
>> On Mon, Feb 19, 2018 at 6:43 AM, Mikhail <mi...@gmail.com>
>> wrote:
>>
>>> Hi Sergey,
>>>
>>> The release of 2.4 should be soon, in a week or couple, however, there's
>>> no
>>> strong schedule for Apache releases.
>>>
>>> Could you please share a reproducer for the issue? Might be you can
>>> share a
>>> storage on which the issue can be reproduced?
>>>
>>> Thanks,
>>> Mike.
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>
>>
>
>
> --
> Best regards,
> Sergey S. Sergeev (aka SSSerj)
> Notes: https://twitter.com/ssserj
> exit(0);
>
Re: And again... Failed to get page IO instance (page content is
corrupted)
Posted by "slava.koptilin" <sl...@gmail.com>.
Hello Oleksandr,
The ignite-2.6 branch is already stabilized and frozen.
I hope the voting will be opened today!
You can follow this topic to get updates on this.
http://apache-ignite-developers.2346864.n4.nabble.com/Ignite-2-6-emergency-release-suggestion-tc31340.html
Thanks!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: And again... Failed to get page IO instance (page content is corrupted)
Posted by Olexandr K <ol...@gmail.com>.
Hi guys,
are you still planning to release 2.6.0 as patch release for 2.5.0?
BR, Oleksandr
On Fri, Jun 29, 2018 at 11:43 AM, Andrey Mashenkov <
andrey.mashenkov@gmail.com> wrote:
> Hi Oleg,
>
> Yes, page corruption issues shouldn't happened when persistence is
> disabled.
> Please, let us know it you will face one.
>
>
> On Fri, Jun 29, 2018 at 1:56 AM Olexandr K <ol...@gmail.com>
> wrote:
>
>> Hi Andrey,
>>
>> Thanks for clarifying this.
>> We have just a single persistent cache and I reworked the code to get rid
>> of expiration policy.
>> All our non-persistent caches have expiration policy but this should not
>> be a problem, right?
>>
>> BR, Oleksandr
>>
>> On Thu, Jun 28, 2018 at 8:37 PM, Andrey Mashenkov <
>> andrey.mashenkov@gmail.com> wrote:
>>
>>> Hi Oleg,
>>>
>>> The issue you mentioned IGNITE-8659 [1] is caused by IGNITE-5874 [2]
>>> that will not a part of ignite-2.6 release.
>>> For now, 'ExpiryPolicy with persistence' is totally broken and all it's
>>> fixes are planned to the next 2.7 release.
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/IGNITE-8659
>>> [2] https://issues.apache.org/jira/browse/IGNITE-5874
>>>
>>> On Tue, Jun 26, 2018 at 11:26 PM Olexandr K <
>>> olexandr.kundirenko@gmail.com> wrote:
>>>
>>>> Hi Andrey,
>>>>
>>>> I see Fix version 2.7 in Jira: https://issues.apache.org/
>>>> jira/browse/IGNITE-8659
>>>> This is a critical bug.. bouncing of server node in not-a-right-time
>>>> causes a catastrophe.
>>>> This mean no availability in fact - I had to clean data folders to
>>>> start my cluster after that
>>>>
>>>> BR, Oleksandr
>>>>
>>>>
>>>> On Fri, Jun 22, 2018 at 4:06 PM, Andrey Mashenkov <
>>>> andrey.mashenkov@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> We've found and fixed few issues related to ExpiryPolicy usage.
>>>>> Most likely, your issue is [1] and it is planned to ignite 2.6 release.
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-8659
>>>>>
>>>>>
>>>>> On Fri, Jun 22, 2018 at 8:43 AM Olexandr K <
>>>>> olexandr.kundirenko@gmail.com> wrote:
>>>>>
>>>>>> Hi Team,
>>>>>>
>>>>>> Issue is still there in 2.5.0
>>>>>>
>>>>>> Steps to reproduce:
>>>>>> 1) start 2 servers + 2 clients topology
>>>>>> 2) start load testing on client nodes
>>>>>> 3) stop server 1
>>>>>> 4) start server 1
>>>>>> 5) stop server 1 again when rebalancing is in progress
>>>>>> => and we got data corrupted here, see error below
>>>>>> => we were not able to restart Ignite cluster after that and need to
>>>>>> perform data folders cleanup...
>>>>>>
>>>>>> 2018-06-21 11:28:01.684 [ttl-cleanup-worker-#43] ERROR - Critical
>>>>>> system error detected. Will be handled accordingly to configured handler
>>>>>> [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler,
>>>>>> failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
>>>>>> o.a.i.IgniteException: Runtime failure on bounds: [lower=null,
>>>>>> upper=PendingRow []]]]
>>>>>> org.apache.ignite.IgniteException: Runtime failure on bounds:
>>>>>> [lower=null, upper=PendingRow []]
>>>>>> at org.apache.ignite.internal.processors.cache.persistence.
>>>>>> tree.BPlusTree.find(BPlusTree.java:971)
>>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>>> at org.apache.ignite.internal.processors.cache.persistence.
>>>>>> tree.BPlusTree.find(BPlusTree.java:950)
>>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>>> at org.apache.ignite.internal.processors.cache.
>>>>>> IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024)
>>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>>> at org.apache.ignite.internal.processors.cache.
>>>>>> GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
>>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>>> at org.apache.ignite.internal.processors.cache.
>>>>>> GridCacheSharedTtlCleanupManager$CleanupWorker.body(
>>>>>> GridCacheSharedTtlCleanupManager.java:137)
>>>>>> [ignite-core-2.5.0.jar:2.5.0]
>>>>>> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>>>>> [ignite-core-2.5.0.jar:2.5.0]
>>>>>> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
>>>>>> Caused by: java.lang.IllegalStateException: Item not found: 2
>>>>>> at org.apache.ignite.internal.processors.cache.persistence.
>>>>>> tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
>>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>>> at org.apache.ignite.internal.processors.cache.persistence.
>>>>>> tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
>>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>>> at org.apache.ignite.internal.processors.cache.persistence.
>>>>>> tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
>>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>>> at org.apache.ignite.internal.processors.cache.persistence.
>>>>>> CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
>>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>>> at org.apache.ignite.internal.processors.cache.persistence.
>>>>>> CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
>>>>>> ~[ignite-core-2.5.0.j
>>>>>>
>>>>>> BR, Oleksandr
>>>>>>
>>>>>> On Thu, Jun 14, 2018 at 2:51 PM, Olexandr K <
>>>>>> olexandr.kundirenko@gmail.com> wrote:
>>>>>>
>>>>>>> Upgraded to 2.5.0 and didn't get such error so far..
>>>>>>> Thanks!
>>>>>>>
>>>>>>> On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <
>>>>>>> dkarachentsev@gridgain.com> wrote:
>>>>>>>
>>>>>>>> It would be better to upgrade to 2.5, where it is fixed.
>>>>>>>> But if you want to overcome this issue in your's version, you need
>>>>>>>> to add
>>>>>>>> ignite-indexing dependency to your classpath and configure SQL
>>>>>>>> indexes. For
>>>>>>>> example [1], just modify it to work with Spring in XML:
>>>>>>>> <property name="indexedTypes">
>>>>>>>> <list>
>>>>>>>> <value>org.your.KeyObject</value>
>>>>>>>> <value>org.your.ValueObject</value>
>>>>>>>> </list>
>>>>>>>> </property>
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://apacheignite-sql.readme.io/docs/schema-and-
>>>>>>>> indexes#section-registering-indexed-types
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> -Dmitry
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>> Andrey V. Mashenkov
>>>>>
>>>>
>>>>
>>>
>>> --
>>> Best regards,
>>> Andrey V. Mashenkov
>>>
>>
>>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
Re: And again... Failed to get page IO instance (page content is corrupted)
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Oleg,
Yes, page corruption issues shouldn't happened when persistence is disabled.
Please, let us know it you will face one.
On Fri, Jun 29, 2018 at 1:56 AM Olexandr K <ol...@gmail.com>
wrote:
> Hi Andrey,
>
> Thanks for clarifying this.
> We have just a single persistent cache and I reworked the code to get rid
> of expiration policy.
> All our non-persistent caches have expiration policy but this should not
> be a problem, right?
>
> BR, Oleksandr
>
> On Thu, Jun 28, 2018 at 8:37 PM, Andrey Mashenkov <
> andrey.mashenkov@gmail.com> wrote:
>
>> Hi Oleg,
>>
>> The issue you mentioned IGNITE-8659 [1] is caused by IGNITE-5874 [2] that
>> will not a part of ignite-2.6 release.
>> For now, 'ExpiryPolicy with persistence' is totally broken and all it's
>> fixes are planned to the next 2.7 release.
>>
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-8659
>> [2] https://issues.apache.org/jira/browse/IGNITE-5874
>>
>> On Tue, Jun 26, 2018 at 11:26 PM Olexandr K <
>> olexandr.kundirenko@gmail.com> wrote:
>>
>>> Hi Andrey,
>>>
>>> I see Fix version 2.7 in Jira:
>>> https://issues.apache.org/jira/browse/IGNITE-8659
>>> This is a critical bug.. bouncing of server node in not-a-right-time
>>> causes a catastrophe.
>>> This mean no availability in fact - I had to clean data folders to start
>>> my cluster after that
>>>
>>> BR, Oleksandr
>>>
>>>
>>> On Fri, Jun 22, 2018 at 4:06 PM, Andrey Mashenkov <
>>> andrey.mashenkov@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We've found and fixed few issues related to ExpiryPolicy usage.
>>>> Most likely, your issue is [1] and it is planned to ignite 2.6 release.
>>>>
>>>> [1] https://issues.apache.org/jira/browse/IGNITE-8659
>>>>
>>>>
>>>> On Fri, Jun 22, 2018 at 8:43 AM Olexandr K <
>>>> olexandr.kundirenko@gmail.com> wrote:
>>>>
>>>>> Hi Team,
>>>>>
>>>>> Issue is still there in 2.5.0
>>>>>
>>>>> Steps to reproduce:
>>>>> 1) start 2 servers + 2 clients topology
>>>>> 2) start load testing on client nodes
>>>>> 3) stop server 1
>>>>> 4) start server 1
>>>>> 5) stop server 1 again when rebalancing is in progress
>>>>> => and we got data corrupted here, see error below
>>>>> => we were not able to restart Ignite cluster after that and need to
>>>>> perform data folders cleanup...
>>>>>
>>>>> 2018-06-21 11:28:01.684 [ttl-cleanup-worker-#43] ERROR - Critical
>>>>> system error detected. Will be handled accordingly to configured handler
>>>>> [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler,
>>>>> failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
>>>>> o.a.i.IgniteException: Runtime failure on bounds: [lower=null,
>>>>> upper=PendingRow []]]]
>>>>> org.apache.ignite.IgniteException: Runtime failure on bounds:
>>>>> [lower=null, upper=PendingRow []]
>>>>> at
>>>>> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:971)
>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>> at
>>>>> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:950)
>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>> at
>>>>> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024)
>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>> at
>>>>> org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>> at
>>>>> org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137)
>>>>> [ignite-core-2.5.0.jar:2.5.0]
>>>>> at
>>>>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>>>> [ignite-core-2.5.0.jar:2.5.0]
>>>>> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
>>>>> Caused by: java.lang.IllegalStateException: Item not found: 2
>>>>> at
>>>>> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>> at
>>>>> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>> at
>>>>> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>> at
>>>>> org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
>>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>>> at
>>>>> org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
>>>>> ~[ignite-core-2.5.0.j
>>>>>
>>>>> BR, Oleksandr
>>>>>
>>>>> On Thu, Jun 14, 2018 at 2:51 PM, Olexandr K <
>>>>> olexandr.kundirenko@gmail.com> wrote:
>>>>>
>>>>>> Upgraded to 2.5.0 and didn't get such error so far..
>>>>>> Thanks!
>>>>>>
>>>>>> On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <
>>>>>> dkarachentsev@gridgain.com> wrote:
>>>>>>
>>>>>>> It would be better to upgrade to 2.5, where it is fixed.
>>>>>>> But if you want to overcome this issue in your's version, you need
>>>>>>> to add
>>>>>>> ignite-indexing dependency to your classpath and configure SQL
>>>>>>> indexes. For
>>>>>>> example [1], just modify it to work with Spring in XML:
>>>>>>> <property name="indexedTypes">
>>>>>>> <list>
>>>>>>> <value>org.your.KeyObject</value>
>>>>>>> <value>org.your.ValueObject</value>
>>>>>>> </list>
>>>>>>> </property>
>>>>>>>
>>>>>>> [1]
>>>>>>>
>>>>>>> https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-registering-indexed-types
>>>>>>>
>>>>>>> Thanks!
>>>>>>> -Dmitry
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Best regards,
>>>> Andrey V. Mashenkov
>>>>
>>>
>>>
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>
>
>
--
Best regards,
Andrey V. Mashenkov
Re: And again... Failed to get page IO instance (page content is corrupted)
Posted by Olexandr K <ol...@gmail.com>.
Hi Andrey,
Thanks for clarifying this.
We have just a single persistent cache and I reworked the code to get rid
of expiration policy.
All our non-persistent caches have expiration policy but this should not be
a problem, right?
BR, Oleksandr
On Thu, Jun 28, 2018 at 8:37 PM, Andrey Mashenkov <
andrey.mashenkov@gmail.com> wrote:
> Hi Oleg,
>
> The issue you mentioned IGNITE-8659 [1] is caused by IGNITE-5874 [2] that
> will not a part of ignite-2.6 release.
> For now, 'ExpiryPolicy with persistence' is totally broken and all it's
> fixes are planned to the next 2.7 release.
>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-8659
> [2] https://issues.apache.org/jira/browse/IGNITE-5874
>
> On Tue, Jun 26, 2018 at 11:26 PM Olexandr K <ol...@gmail.com>
> wrote:
>
>> Hi Andrey,
>>
>> I see Fix version 2.7 in Jira: https://issues.apache.org/
>> jira/browse/IGNITE-8659
>> This is a critical bug.. bouncing of server node in not-a-right-time
>> causes a catastrophe.
>> This mean no availability in fact - I had to clean data folders to start
>> my cluster after that
>>
>> BR, Oleksandr
>>
>>
>> On Fri, Jun 22, 2018 at 4:06 PM, Andrey Mashenkov <
>> andrey.mashenkov@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> We've found and fixed few issues related to ExpiryPolicy usage.
>>> Most likely, your issue is [1] and it is planned to ignite 2.6 release.
>>>
>>> [1] https://issues.apache.org/jira/browse/IGNITE-8659
>>>
>>>
>>> On Fri, Jun 22, 2018 at 8:43 AM Olexandr K <
>>> olexandr.kundirenko@gmail.com> wrote:
>>>
>>>> Hi Team,
>>>>
>>>> Issue is still there in 2.5.0
>>>>
>>>> Steps to reproduce:
>>>> 1) start 2 servers + 2 clients topology
>>>> 2) start load testing on client nodes
>>>> 3) stop server 1
>>>> 4) start server 1
>>>> 5) stop server 1 again when rebalancing is in progress
>>>> => and we got data corrupted here, see error below
>>>> => we were not able to restart Ignite cluster after that and need to
>>>> perform data folders cleanup...
>>>>
>>>> 2018-06-21 11:28:01.684 [ttl-cleanup-worker-#43] ERROR - Critical
>>>> system error detected. Will be handled accordingly to configured handler
>>>> [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler,
>>>> failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
>>>> o.a.i.IgniteException: Runtime failure on bounds: [lower=null,
>>>> upper=PendingRow []]]]
>>>> org.apache.ignite.IgniteException: Runtime failure on bounds:
>>>> [lower=null, upper=PendingRow []]
>>>> at org.apache.ignite.internal.processors.cache.persistence.
>>>> tree.BPlusTree.find(BPlusTree.java:971) ~[ignite-core-2.5.0.jar:2.5.0]
>>>> at org.apache.ignite.internal.processors.cache.persistence.
>>>> tree.BPlusTree.find(BPlusTree.java:950) ~[ignite-core-2.5.0.jar:2.5.0]
>>>> at org.apache.ignite.internal.processors.cache.
>>>> IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024)
>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>> at org.apache.ignite.internal.processors.cache.
>>>> GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>> at org.apache.ignite.internal.processors.cache.
>>>> GridCacheSharedTtlCleanupManager$CleanupWorker.body(
>>>> GridCacheSharedTtlCleanupManager.java:137)
>>>> [ignite-core-2.5.0.jar:2.5.0]
>>>> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>>> [ignite-core-2.5.0.jar:2.5.0]
>>>> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
>>>> Caused by: java.lang.IllegalStateException: Item not found: 2
>>>> at org.apache.ignite.internal.processors.cache.persistence.
>>>> tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>> at org.apache.ignite.internal.processors.cache.persistence.
>>>> tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>> at org.apache.ignite.internal.processors.cache.persistence.
>>>> tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>> at org.apache.ignite.internal.processors.cache.persistence.
>>>> CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
>>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>>> at org.apache.ignite.internal.processors.cache.persistence.
>>>> CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
>>>> ~[ignite-core-2.5.0.j
>>>>
>>>> BR, Oleksandr
>>>>
>>>> On Thu, Jun 14, 2018 at 2:51 PM, Olexandr K <
>>>> olexandr.kundirenko@gmail.com> wrote:
>>>>
>>>>> Upgraded to 2.5.0 and didn't get such error so far..
>>>>> Thanks!
>>>>>
>>>>> On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <
>>>>> dkarachentsev@gridgain.com> wrote:
>>>>>
>>>>>> It would be better to upgrade to 2.5, where it is fixed.
>>>>>> But if you want to overcome this issue in your's version, you need to
>>>>>> add
>>>>>> ignite-indexing dependency to your classpath and configure SQL
>>>>>> indexes. For
>>>>>> example [1], just modify it to work with Spring in XML:
>>>>>> <property name="indexedTypes">
>>>>>> <list>
>>>>>> <value>org.your.KeyObject</value>
>>>>>> <value>org.your.ValueObject</value>
>>>>>> </list>
>>>>>> </property>
>>>>>>
>>>>>> [1]
>>>>>> https://apacheignite-sql.readme.io/docs/schema-and-
>>>>>> indexes#section-registering-indexed-types
>>>>>>
>>>>>> Thanks!
>>>>>> -Dmitry
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> --
>>> Best regards,
>>> Andrey V. Mashenkov
>>>
>>
>>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
Re: And again... Failed to get page IO instance (page content is corrupted)
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi Oleg,
The issue you mentioned IGNITE-8659 [1] is caused by IGNITE-5874 [2] that
will not a part of ignite-2.6 release.
For now, 'ExpiryPolicy with persistence' is totally broken and all it's
fixes are planned to the next 2.7 release.
[1] https://issues.apache.org/jira/browse/IGNITE-8659
[2] https://issues.apache.org/jira/browse/IGNITE-5874
On Tue, Jun 26, 2018 at 11:26 PM Olexandr K <ol...@gmail.com>
wrote:
> Hi Andrey,
>
> I see Fix version 2.7 in Jira:
> https://issues.apache.org/jira/browse/IGNITE-8659
> This is a critical bug.. bouncing of server node in not-a-right-time
> causes a catastrophe.
> This mean no availability in fact - I had to clean data folders to start
> my cluster after that
>
> BR, Oleksandr
>
>
> On Fri, Jun 22, 2018 at 4:06 PM, Andrey Mashenkov <
> andrey.mashenkov@gmail.com> wrote:
>
>> Hi,
>>
>> We've found and fixed few issues related to ExpiryPolicy usage.
>> Most likely, your issue is [1] and it is planned to ignite 2.6 release.
>>
>> [1] https://issues.apache.org/jira/browse/IGNITE-8659
>>
>>
>> On Fri, Jun 22, 2018 at 8:43 AM Olexandr K <ol...@gmail.com>
>> wrote:
>>
>>> Hi Team,
>>>
>>> Issue is still there in 2.5.0
>>>
>>> Steps to reproduce:
>>> 1) start 2 servers + 2 clients topology
>>> 2) start load testing on client nodes
>>> 3) stop server 1
>>> 4) start server 1
>>> 5) stop server 1 again when rebalancing is in progress
>>> => and we got data corrupted here, see error below
>>> => we were not able to restart Ignite cluster after that and need to
>>> perform data folders cleanup...
>>>
>>> 2018-06-21 11:28:01.684 [ttl-cleanup-worker-#43] ERROR - Critical
>>> system error detected. Will be handled accordingly to configured handler
>>> [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler,
>>> failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
>>> o.a.i.IgniteException: Runtime failure on bounds: [lower=null,
>>> upper=PendingRow []]]]
>>> org.apache.ignite.IgniteException: Runtime failure on bounds:
>>> [lower=null, upper=PendingRow []]
>>> at
>>> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:971)
>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>> at
>>> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:950)
>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>> at
>>> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024)
>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>> at
>>> org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>> at
>>> org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137)
>>> [ignite-core-2.5.0.jar:2.5.0]
>>> at
>>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>>> [ignite-core-2.5.0.jar:2.5.0]
>>> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
>>> Caused by: java.lang.IllegalStateException: Item not found: 2
>>> at
>>> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>> at
>>> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>> at
>>> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>> at
>>> org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
>>> ~[ignite-core-2.5.0.jar:2.5.0]
>>> at
>>> org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
>>> ~[ignite-core-2.5.0.j
>>>
>>> BR, Oleksandr
>>>
>>> On Thu, Jun 14, 2018 at 2:51 PM, Olexandr K <
>>> olexandr.kundirenko@gmail.com> wrote:
>>>
>>>> Upgraded to 2.5.0 and didn't get such error so far..
>>>> Thanks!
>>>>
>>>> On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <
>>>> dkarachentsev@gridgain.com> wrote:
>>>>
>>>>> It would be better to upgrade to 2.5, where it is fixed.
>>>>> But if you want to overcome this issue in your's version, you need to
>>>>> add
>>>>> ignite-indexing dependency to your classpath and configure SQL
>>>>> indexes. For
>>>>> example [1], just modify it to work with Spring in XML:
>>>>> <property name="indexedTypes">
>>>>> <list>
>>>>> <value>org.your.KeyObject</value>
>>>>> <value>org.your.ValueObject</value>
>>>>> </list>
>>>>> </property>
>>>>>
>>>>> [1]
>>>>>
>>>>> https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-registering-indexed-types
>>>>>
>>>>> Thanks!
>>>>> -Dmitry
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>>
>>>>
>>>>
>>>
>>
>> --
>> Best regards,
>> Andrey V. Mashenkov
>>
>
>
--
Best regards,
Andrey V. Mashenkov
Re: And again... Failed to get page IO instance (page content is corrupted)
Posted by Olexandr K <ol...@gmail.com>.
Hi Andrey,
I see Fix version 2.7 in Jira:
https://issues.apache.org/jira/browse/IGNITE-8659
This is a critical bug.. bouncing of server node in not-a-right-time causes
a catastrophe.
This mean no availability in fact - I had to clean data folders to start my
cluster after that
BR, Oleksandr
On Fri, Jun 22, 2018 at 4:06 PM, Andrey Mashenkov <
andrey.mashenkov@gmail.com> wrote:
> Hi,
>
> We've found and fixed few issues related to ExpiryPolicy usage.
> Most likely, your issue is [1] and it is planned to ignite 2.6 release.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-8659
>
>
> On Fri, Jun 22, 2018 at 8:43 AM Olexandr K <ol...@gmail.com>
> wrote:
>
>> Hi Team,
>>
>> Issue is still there in 2.5.0
>>
>> Steps to reproduce:
>> 1) start 2 servers + 2 clients topology
>> 2) start load testing on client nodes
>> 3) stop server 1
>> 4) start server 1
>> 5) stop server 1 again when rebalancing is in progress
>> => and we got data corrupted here, see error below
>> => we were not able to restart Ignite cluster after that and need to
>> perform data folders cleanup...
>>
>> 2018-06-21 11:28:01.684 [ttl-cleanup-worker-#43] ERROR - Critical system
>> error detected. Will be handled accordingly to configured handler
>> [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler,
>> failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
>> o.a.i.IgniteException: Runtime failure on bounds: [lower=null,
>> upper=PendingRow []]]]
>> org.apache.ignite.IgniteException: Runtime failure on bounds:
>> [lower=null, upper=PendingRow []]
>> at org.apache.ignite.internal.processors.cache.persistence.
>> tree.BPlusTree.find(BPlusTree.java:971) ~[ignite-core-2.5.0.jar:2.5.0]
>> at org.apache.ignite.internal.processors.cache.persistence.
>> tree.BPlusTree.find(BPlusTree.java:950) ~[ignite-core-2.5.0.jar:2.5.0]
>> at org.apache.ignite.internal.processors.cache.
>> IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024)
>> ~[ignite-core-2.5.0.jar:2.5.0]
>> at org.apache.ignite.internal.processors.cache.
>> GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
>> ~[ignite-core-2.5.0.jar:2.5.0]
>> at org.apache.ignite.internal.processors.cache.
>> GridCacheSharedTtlCleanupManager$CleanupWorker.body(
>> GridCacheSharedTtlCleanupManager.java:137) [ignite-core-2.5.0.jar:2.5.0]
>> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>> [ignite-core-2.5.0.jar:2.5.0]
>> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
>> Caused by: java.lang.IllegalStateException: Item not found: 2
>> at org.apache.ignite.internal.processors.cache.persistence.
>> tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
>> ~[ignite-core-2.5.0.jar:2.5.0]
>> at org.apache.ignite.internal.processors.cache.persistence.
>> tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
>> ~[ignite-core-2.5.0.jar:2.5.0]
>> at org.apache.ignite.internal.processors.cache.persistence.
>> tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
>> ~[ignite-core-2.5.0.jar:2.5.0]
>> at org.apache.ignite.internal.processors.cache.persistence.
>> CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
>> ~[ignite-core-2.5.0.jar:2.5.0]
>> at org.apache.ignite.internal.processors.cache.persistence.
>> CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
>> ~[ignite-core-2.5.0.j
>>
>> BR, Oleksandr
>>
>> On Thu, Jun 14, 2018 at 2:51 PM, Olexandr K <
>> olexandr.kundirenko@gmail.com> wrote:
>>
>>> Upgraded to 2.5.0 and didn't get such error so far..
>>> Thanks!
>>>
>>> On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <
>>> dkarachentsev@gridgain.com> wrote:
>>>
>>>> It would be better to upgrade to 2.5, where it is fixed.
>>>> But if you want to overcome this issue in your's version, you need to
>>>> add
>>>> ignite-indexing dependency to your classpath and configure SQL indexes.
>>>> For
>>>> example [1], just modify it to work with Spring in XML:
>>>> <property name="indexedTypes">
>>>> <list>
>>>> <value>org.your.KeyObject</value>
>>>> <value>org.your.ValueObject</value>
>>>> </list>
>>>> </property>
>>>>
>>>> [1]
>>>> https://apacheignite-sql.readme.io/docs/schema-and-
>>>> indexes#section-registering-indexed-types
>>>>
>>>> Thanks!
>>>> -Dmitry
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>
>>>
>>>
>>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
Re: And again... Failed to get page IO instance (page content is corrupted)
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi,
We've found and fixed few issues related to ExpiryPolicy usage.
Most likely, your issue is [1] and it is planned to ignite 2.6 release.
[1] https://issues.apache.org/jira/browse/IGNITE-8659
On Fri, Jun 22, 2018 at 8:43 AM Olexandr K <ol...@gmail.com>
wrote:
> Hi Team,
>
> Issue is still there in 2.5.0
>
> Steps to reproduce:
> 1) start 2 servers + 2 clients topology
> 2) start load testing on client nodes
> 3) stop server 1
> 4) start server 1
> 5) stop server 1 again when rebalancing is in progress
> => and we got data corrupted here, see error below
> => we were not able to restart Ignite cluster after that and need to
> perform data folders cleanup...
>
> 2018-06-21 11:28:01.684 [ttl-cleanup-worker-#43] ERROR - Critical system
> error detected. Will be handled accordingly to configured handler
> [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler,
> failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
> o.a.i.IgniteException: Runtime failure on bounds: [lower=null,
> upper=PendingRow []]]]
> org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
> upper=PendingRow []]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:971)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:950)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137)
> [ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> [ignite-core-2.5.0.jar:2.5.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
> Caused by: java.lang.IllegalStateException: Item not found: 2
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
> ~[ignite-core-2.5.0.jar:2.5.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> ~[ignite-core-2.5.0.j
>
> BR, Oleksandr
>
> On Thu, Jun 14, 2018 at 2:51 PM, Olexandr K <olexandr.kundirenko@gmail.com
> > wrote:
>
>> Upgraded to 2.5.0 and didn't get such error so far..
>> Thanks!
>>
>> On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <
>> dkarachentsev@gridgain.com> wrote:
>>
>>> It would be better to upgrade to 2.5, where it is fixed.
>>> But if you want to overcome this issue in your's version, you need to add
>>> ignite-indexing dependency to your classpath and configure SQL indexes.
>>> For
>>> example [1], just modify it to work with Spring in XML:
>>> <property name="indexedTypes">
>>> <list>
>>> <value>org.your.KeyObject</value>
>>> <value>org.your.ValueObject</value>
>>> </list>
>>> </property>
>>>
>>> [1]
>>>
>>> https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-registering-indexed-types
>>>
>>> Thanks!
>>> -Dmitry
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>
>>
>
--
Best regards,
Andrey V. Mashenkov
Re: And again... Failed to get page IO instance (page content is corrupted)
Posted by Olexandr K <ol...@gmail.com>.
Hi Team,
Issue is still there in 2.5.0
Steps to reproduce:
1) start 2 servers + 2 clients topology
2) start load testing on client nodes
3) stop server 1
4) start server 1
5) stop server 1 again when rebalancing is in progress
=> and we got data corrupted here, see error below
=> we were not able to restart Ignite cluster after that and need to
perform data folders cleanup...
2018-06-21 11:28:01.684 [ttl-cleanup-worker-#43] ERROR - Critical system
error detected. Will be handled accordingly to configured handler
[hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler,
failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class
o.a.i.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]]]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:971)
~[ignite-core-2.5.0.jar:2.5.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:950)
~[ignite-core-2.5.0.jar:2.5.0]
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1024)
~[ignite-core-2.5.0.jar:2.5.0]
at
org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
~[ignite-core-2.5.0.jar:2.5.0]
at
org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137)
[ignite-core-2.5.0.jar:2.5.0]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
[ignite-core-2.5.0.jar:2.5.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Item not found: 2
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
~[ignite-core-2.5.0.jar:2.5.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
~[ignite-core-2.5.0.jar:2.5.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
~[ignite-core-2.5.0.jar:2.5.0]
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
~[ignite-core-2.5.0.jar:2.5.0]
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
~[ignite-core-2.5.0.j
BR, Oleksandr
On Thu, Jun 14, 2018 at 2:51 PM, Olexandr K <ol...@gmail.com>
wrote:
> Upgraded to 2.5.0 and didn't get such error so far..
> Thanks!
>
> On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <dkarachentsev@gridgain.com
> > wrote:
>
>> It would be better to upgrade to 2.5, where it is fixed.
>> But if you want to overcome this issue in your's version, you need to add
>> ignite-indexing dependency to your classpath and configure SQL indexes.
>> For
>> example [1], just modify it to work with Spring in XML:
>> <property name="indexedTypes">
>> <list>
>> <value>org.your.KeyObject</value>
>> <value>org.your.ValueObject</value>
>> </list>
>> </property>
>>
>> [1]
>> https://apacheignite-sql.readme.io/docs/schema-and-indexes#
>> section-registering-indexed-types
>>
>> Thanks!
>> -Dmitry
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>
Re: And again... Failed to get page IO instance (page content is corrupted)
Posted by Olexandr K <ol...@gmail.com>.
Upgraded to 2.5.0 and didn't get such error so far..
Thanks!
On Wed, Jun 13, 2018 at 4:58 PM, dkarachentsev <dk...@gridgain.com>
wrote:
> It would be better to upgrade to 2.5, where it is fixed.
> But if you want to overcome this issue in your's version, you need to add
> ignite-indexing dependency to your classpath and configure SQL indexes. For
> example [1], just modify it to work with Spring in XML:
> <property name="indexedTypes">
> <list>
> <value>org.your.KeyObject</value>
> <value>org.your.ValueObject</value>
> </list>
> </property>
>
> [1]
> https://apacheignite-sql.readme.io/docs/schema-and-
> indexes#section-registering-indexed-types
>
> Thanks!
> -Dmitry
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: And again... Failed to get page IO instance (page content is
corrupted)
Posted by dkarachentsev <dk...@gridgain.com>.
It would be better to upgrade to 2.5, where it is fixed.
But if you want to overcome this issue in your's version, you need to add
ignite-indexing dependency to your classpath and configure SQL indexes. For
example [1], just modify it to work with Spring in XML:
<property name="indexedTypes">
<list>
<value>org.your.KeyObject</value>
<value>org.your.ValueObject</value>
</list>
</property>
[1]
https://apacheignite-sql.readme.io/docs/schema-and-indexes#section-registering-indexed-types
Thanks!
-Dmitry
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: And again... Failed to get page IO instance (page content is corrupted)
Posted by Olexandr K <ol...@gmail.com>.
I'm using key/value API.
Should I define index for the key explicitly?
This sounds strange... Can you give a sample how I can do this in xml
please?
Here is one of my cache configurations.
Actually I'm storing UUID per UUID here: IgniteCache<UUID, UUID>
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="RefreshToken"/>
<property name="dataRegionName"
value="auth_durable_region"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="writeSynchronizationMode"
value="FULL_ASYNC"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="backups" value="0"/>
<property name="eagerTtl" value="true"/> <!-- true:
clean expired items in background, false: on next access -->
</bean>
On Wed, Jun 13, 2018 at 12:18 PM, Andrey Mashenkov <
andrey.mashenkov@gmail.com> wrote:
> Hi,
>
> Possibly, it is a bug in partition eviction optimization. Ignite can skip
> partition eviction procedure and remove partition instantly if there is no
> indexes.
>
> If it is so, you can try the latest ignite-2.5 version [1] or
> as a workaround you can add index via configuring QueryEntity or via
> setting cacheCfg.setIndexTypes().
>
>
> [1] https://ignite.apache.org/download.cgi#binaries
>
> On Wed, Jun 13, 2018 at 1:22 AM, Oleks K <ol...@gmail.com>
> wrote:
>
>> Hi guys,
>>
>> I got similar errors in 2.4.0
>>
>> First:
>>
>> org.apache.ignite.IgniteException: Runtime failure on bounds:
>> [lower=null,
>> upper=PendingRow []]
>> --> Caused by: java.lang.IllegalStateException: Failed to get page IO
>> instance (page content is corrupted)
>>
>> Then lots of:
>>
>> org.apache.ignite.IgniteException: Runtime failure on bounds
>> --> Caused by: java.lang.IllegalStateException: Item not found: 3
>>
>> This was reproduced when I started and stopped server nodes under the load
>> Topology: 2 server and 2 client nodes
>> Java: 1.8.0_162
>> OS: Windows Server 2012 R2 6.3 amd64
>>
>> Cache config:
>> <bean class="org.apache.ignite.configuration.CacheConfiguration">
>> <property name="name" value="RefreshToken"/>
>> <property name="dataRegionName"
>> value="auth_durable_region"/>
>> <property name="atomicityMode" value="ATOMIC"/>
>> <property name="writeSynchronizationMode"
>> value="FULL_ASYNC"/>
>> <property name="cacheMode" value="PARTITIONED"/>
>> <property name="backups" value="0"/>
>> <property name="eagerTtl" value="true"/>
>> </bean>
>>
>> Ignite team, can you comment on this please?
>> How critical is the issue? What is the impact?
>> Any workarounds? Fix planned?
>>
>> 2018-06-13 00:22:30.978 [exchange-worker-#42] INFO
>> org.apache.ignite.internal.processors.cache.distributed.dht.
>> preloader.GridDhtPartitionDemander
>> - Starting rebalancing [mode=ASYNC,
>> fromNode=bdddfe24-aab3-46fa-9452-efe933783adb, partitionsCount=787,
>> topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], updateSeq=12]
>> 2018-06-13 00:22:31.594 [ttl-cleanup-worker-#52] ERROR
>> org.apache.ignite.internal.processors.cache.GridCacheSharedT
>> tlCleanupManager
>> - Runtime error caught during grid runnable execution: GridWorker
>> [name=ttl-cleanup-worker, igniteInstanceName=null, finished=false,
>> hashCode=473353699, interrupted=false, runner=ttl-cleanup-worker-#52]
>> org.apache.ignite.IgniteException: Runtime failure on bounds:
>> [lower=null,
>> upper=PendingRow []]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.find(BPlusTree.java:963)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.find(BPlusTree.java:942)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
>> apManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:974)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.GridCacheTtlMana
>> ger.expire(GridCacheTtlManager.java:197)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.GridCacheSharedT
>> tlCleanupManager$CleanupWorker.body(GridCacheSh
>> aredTtlCleanupManager.java:129)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo
>> rker.java:110)
>> [ignite-core-2.4.0.jar:2.4.0]
>> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
>> Caused by: java.lang.IllegalStateException: Failed to get page IO
>> instance
>> (page content is corrupted)
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .io.IOVersions.forVersion(IOVersions.java:83)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .io.IOVersions.forPage(IOVersions.java:95)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.Cach
>> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.Cach
>> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.tree.PendingRow.
>> initKey(PendingRow.java:72)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.tree.PendingEntr
>> iesTree.getRow(PendingEntriesTree.java:118)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.tree.PendingEntr
>> iesTree.getRow(PendingEntriesTree.java:31)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4614)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree$ForwardCursor.init(BPlusTree.java:4516)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4455)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.findLowerUnbounded(BPlusTree.java:919)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.find(BPlusTree.java:951)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> ... 6 more
>>
>> 2018-06-13 00:23:35.596 [sys-stripe-6-#7] ERROR
>> org.apache.ignite.internal.processors.cache.GridCacheIoManager - Failed
>> to
>> process message [senderId=1dedb701-ffd1-4937-b202-7326367a92fb,
>> messageType=class
>> o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtom
>> icFullUpdateRequest]
>> org.apache.ignite.IgniteException: Runtime failure on bounds:
>> [lower=null,
>> upper=PendingRow []]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.find(BPlusTree.java:963)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.find(BPlusTree.java:942)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.IgniteCacheOffhe
>> apManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:974)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.GridCacheTtlMana
>> ger.expire(GridCacheTtlManager.java:197)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.GridCacheUtils.
>> unwindEvicts(GridCacheUtils.java:834)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.GridCacheIoManag
>> er.onMessageProcessed(GridCacheIoManager.java:1099)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.GridCacheIoManag
>> er.processMessage(GridCacheIoManager.java:1072)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.GridCacheIoManag
>> er.onMessage0(GridCacheIoManager.java:579)
>> [ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.GridCacheIoManag
>> er.handleMessage(GridCacheIoManager.java:378)
>> [ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.GridCacheIoManag
>> er.handleMessage(GridCacheIoManager.java:304)
>> [ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.GridCacheIoManag
>> er.access$100(GridCacheIoManager.java:99)
>> [ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.GridCacheIoManag
>> er$1.onMessage(GridCacheIoManager.java:293)
>> [ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.managers.communication.GridIoMana
>> ger.invokeListener(GridIoManager.java:1555)
>> [ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.managers.communication.GridIoMana
>> ger.processRegularMessage0(GridIoManager.java:1183)
>> [ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.managers.communication.GridIoMana
>> ger.access$4200(GridIoManager.java:126)
>> [ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.managers.communication.GridIoMana
>> ger$9.run(GridIoManager.java:1090)
>> [ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(
>> StripedExecutor.java:505)
>> [ignite-core-2.4.0.jar:2.4.0]
>> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
>> Caused by: java.lang.IllegalStateException: Item not found: 3
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.Cach
>> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.Cach
>> eDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.tree.PendingRow.
>> initKey(PendingRow.java:72)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.tree.PendingEntr
>> iesTree.getRow(PendingEntriesTree.java:118)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.tree.PendingEntr
>> iesTree.getRow(PendingEntriesTree.java:31)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4614)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree$ForwardCursor.init(BPlusTree.java:4516)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4455)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.findLowerUnbounded(BPlusTree.java:919)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> at
>> org.apache.ignite.internal.processors.cache.persistence.tree
>> .BPlusTree.find(BPlusTree.java:951)
>> ~[ignite-core-2.4.0.jar:2.4.0]
>> ... 17 more
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>
Re: And again... Failed to get page IO instance (page content is corrupted)
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi,
Possibly, it is a bug in partition eviction optimization. Ignite can skip
partition eviction procedure and remove partition instantly if there is no
indexes.
If it is so, you can try the latest ignite-2.5 version [1] or
as a workaround you can add index via configuring QueryEntity or via
setting cacheCfg.setIndexTypes().
[1] https://ignite.apache.org/download.cgi#binaries
On Wed, Jun 13, 2018 at 1:22 AM, Oleks K <ol...@gmail.com>
wrote:
> Hi guys,
>
> I got similar errors in 2.4.0
>
> First:
>
> org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
> upper=PendingRow []]
> --> Caused by: java.lang.IllegalStateException: Failed to get page IO
> instance (page content is corrupted)
>
> Then lots of:
>
> org.apache.ignite.IgniteException: Runtime failure on bounds
> --> Caused by: java.lang.IllegalStateException: Item not found: 3
>
> This was reproduced when I started and stopped server nodes under the load
> Topology: 2 server and 2 client nodes
> Java: 1.8.0_162
> OS: Windows Server 2012 R2 6.3 amd64
>
> Cache config:
> <bean class="org.apache.ignite.configuration.CacheConfiguration">
> <property name="name" value="RefreshToken"/>
> <property name="dataRegionName"
> value="auth_durable_region"/>
> <property name="atomicityMode" value="ATOMIC"/>
> <property name="writeSynchronizationMode"
> value="FULL_ASYNC"/>
> <property name="cacheMode" value="PARTITIONED"/>
> <property name="backups" value="0"/>
> <property name="eagerTtl" value="true"/>
> </bean>
>
> Ignite team, can you comment on this please?
> How critical is the issue? What is the impact?
> Any workarounds? Fix planned?
>
> 2018-06-13 00:22:30.978 [exchange-worker-#42] INFO
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.
> GridDhtPartitionDemander
> - Starting rebalancing [mode=ASYNC,
> fromNode=bdddfe24-aab3-46fa-9452-efe933783adb, partitionsCount=787,
> topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], updateSeq=12]
> 2018-06-13 00:22:31.594 [ttl-cleanup-worker-#52] ERROR
> org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManag
> er
> - Runtime error caught during grid runnable execution: GridWorker
> [name=ttl-cleanup-worker, igniteInstanceName=null, finished=false,
> hashCode=473353699, interrupted=false, runner=ttl-cleanup-worker-#52]
> org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
> upper=PendingRow []]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree.find(BPlusTree.java:963)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree.find(BPlusTree.java:942)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.
> expire(IgniteCacheOffheapManagerImpl.java:974)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(
> GridCacheTtlManager.java:197)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManag
> er$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:129)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> [ignite-core-2.4.0.jar:2.4.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
> Caused by: java.lang.IllegalStateException: Failed to get page IO instance
> (page content is corrupted)
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.io.IOVersions.forVersion(IOVersions.java:83)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.io.IOVersions.forPage(IOVersions.java:95)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.tree.
> PendingRow.initKey(PendingRow.java:72)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.tree.
> PendingEntriesTree.getRow(PendingEntriesTree.java:118)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.tree.
> PendingEntriesTree.getRow(PendingEntriesTree.java:31)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4614)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4516)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4455)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.
> findLowerUnbounded(BPlusTree.java:919)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree.find(BPlusTree.java:951)
> ~[ignite-core-2.4.0.jar:2.4.0]
> ... 6 more
>
> 2018-06-13 00:23:35.596 [sys-stripe-6-#7] ERROR
> org.apache.ignite.internal.processors.cache.GridCacheIoManager - Failed to
> process message [senderId=1dedb701-ffd1-4937-b202-7326367a92fb,
> messageType=class
> o.a.i.i.processors.cache.distributed.dht.atomic.
> GridNearAtomicFullUpdateRequest]
> org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
> upper=PendingRow []]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree.find(BPlusTree.java:963)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree.find(BPlusTree.java:942)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.
> expire(IgniteCacheOffheapManagerImpl.java:974)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(
> GridCacheTtlManager.java:197)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(
> GridCacheUtils.java:834)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.
> onMessageProcessed(GridCacheIoManager.java:1099)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.
> processMessage(GridCacheIoManager.java:1072)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(
> GridCacheIoManager.java:579)
> [ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.
> handleMessage(GridCacheIoManager.java:378)
> [ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.
> handleMessage(GridCacheIoManager.java:304)
> [ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(
> GridCacheIoManager.java:99)
> [ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.
> onMessage(GridCacheIoManager.java:293)
> [ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.managers.communication.
> GridIoManager.invokeListener(GridIoManager.java:1555)
> [ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.
> processRegularMessage0(GridIoManager.java:1183)
> [ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.managers.communication.
> GridIoManager.access$4200(GridIoManager.java:126)
> [ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.managers.communication.GridIoManager$9.run(
> GridIoManager.java:1090)
> [ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.util.StripedExecutor$Stripe.
> run(StripedExecutor.java:505)
> [ignite-core-2.4.0.jar:2.4.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
> Caused by: java.lang.IllegalStateException: Item not found: 3
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.io.AbstractDataPageIO.findIndirectItemIndex(
> AbstractDataPageIO.java:341)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.tree.
> PendingRow.initKey(PendingRow.java:72)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.tree.
> PendingEntriesTree.getRow(PendingEntriesTree.java:118)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.tree.
> PendingEntriesTree.getRow(PendingEntriesTree.java:31)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4614)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4516)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4455)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.
> findLowerUnbounded(BPlusTree.java:919)
> ~[ignite-core-2.4.0.jar:2.4.0]
> at
> org.apache.ignite.internal.processors.cache.persistence.
> tree.BPlusTree.find(BPlusTree.java:951)
> ~[ignite-core-2.4.0.jar:2.4.0]
> ... 17 more
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
Best regards,
Andrey V. Mashenkov
Re: And again... Failed to get page IO instance (page content is
corrupted)
Posted by Oleks K <ol...@gmail.com>.
Hi guys,
I got similar errors in 2.4.0
First:
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
--> Caused by: java.lang.IllegalStateException: Failed to get page IO
instance (page content is corrupted)
Then lots of:
org.apache.ignite.IgniteException: Runtime failure on bounds
--> Caused by: java.lang.IllegalStateException: Item not found: 3
This was reproduced when I started and stopped server nodes under the load
Topology: 2 server and 2 client nodes
Java: 1.8.0_162
OS: Windows Server 2012 R2 6.3 amd64
Cache config:
<bean class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="RefreshToken"/>
<property name="dataRegionName"
value="auth_durable_region"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="writeSynchronizationMode"
value="FULL_ASYNC"/>
<property name="cacheMode" value="PARTITIONED"/>
<property name="backups" value="0"/>
<property name="eagerTtl" value="true"/>
</bean>
Ignite team, can you comment on this please?
How critical is the issue? What is the impact?
Any workarounds? Fix planned?
2018-06-13 00:22:30.978 [exchange-worker-#42] INFO
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander
- Starting rebalancing [mode=ASYNC,
fromNode=bdddfe24-aab3-46fa-9452-efe933783adb, partitionsCount=787,
topology=AffinityTopologyVersion [topVer=5, minorTopVer=0], updateSeq=12]
2018-06-13 00:22:31.594 [ttl-cleanup-worker-#52] ERROR
org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager
- Runtime error caught during grid runnable execution: GridWorker
[name=ttl-cleanup-worker, igniteInstanceName=null, finished=false,
hashCode=473353699, interrupted=false, runner=ttl-cleanup-worker-#52]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:963)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:974)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:129)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
[ignite-core-2.4.0.jar:2.4.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Failed to get page IO instance
(page content is corrupted)
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4614)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4516)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4455)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:919)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:951)
~[ignite-core-2.4.0.jar:2.4.0]
... 6 more
2018-06-13 00:23:35.596 [sys-stripe-6-#7] ERROR
org.apache.ignite.internal.processors.cache.GridCacheIoManager - Failed to
process message [senderId=1dedb701-ffd1-4937-b202-7326367a92fb,
messageType=class
o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicFullUpdateRequest]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:963)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:942)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:974)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.unwindEvicts(GridCacheUtils.java:834)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessageProcessed(GridCacheIoManager.java:1099)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1072)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
[ignite-core-2.4.0.jar:2.4.0]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: java.lang.IllegalStateException: Item not found: 3
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.findIndirectItemIndex(AbstractDataPageIO.java:341)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.getDataOffset(AbstractDataPageIO.java:450)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.io.AbstractDataPageIO.readPayload(AbstractDataPageIO.java:492)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:150)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.tree.PendingRow.initKey(PendingRow.java:72)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:118)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.tree.PendingEntriesTree.getRow(PendingEntriesTree.java:31)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4614)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4516)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4455)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:919)
~[ignite-core-2.4.0.jar:2.4.0]
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:951)
~[ignite-core-2.4.0.jar:2.4.0]
... 17 more
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: And again... Failed to get page IO instance (page content is
corrupted)
Posted by "ilya.kasnacheev" <il...@gmail.com>.
Hello Sergey!
Cross-posting from a different thread, the underlying problem could be fixed
in 2.4:
https://issues.apache.org/jira/browse/IGNITE-7278
Regards,
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: And again... Failed to get page IO instance (page content is corrupted)
Posted by Sergey Sergeev <we...@gmail.com>.
Hi Mikhail,
Unfortunately, the problem has repeated itself on ignite-core-2.3.3
27.02.18 00:27:55 ERROR GridCacheIoManager - Failed to process message
[senderId=8f99c887-cd4b-4c38-a649-ca430040d535, messageType=class
o.a.i.i.processors.cache.distributed.dht.atomic.
GridNearAtomicUpdateResponse]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
*GridCacheTtlManager.expire*(GridCacheTtlManager.java:197)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
GridCacheUtils.unwindEvicts(GridCacheUtils.java:833)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.
onMessageProcessed(GridCacheIoManager.java:1099)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.
processMessage(GridCacheIoManager.java:1072) ~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.
handleMessage(GridCacheIoManager.java:378) ~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.
handleMessage(GridCacheIoManager.java:304) ~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
GridCacheIoManager.access$100(GridCacheIoManager.java:99)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.
onMessage(GridCacheIoManager.java:293) ~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.managers.communication.
GridIoManager.invokeListener(GridIoManager.java:1555)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.managers.communication.GridIoManager.
processRegularMessage0(GridIoManager.java:1183)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.managers.communication.
GridIoManager.access$4200(GridIoManager.java:126)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.managers.communication.
GridIoManager$9.run(GridIoManager.java:1090) ~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
~[ignite-core-2.3.3.jar:2.3.3]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.lang.IllegalStateException: Failed to get page IO instance
(page content is corrupted)
at org.apache.ignite.internal.processors.cache.persistence.
tree.io.IOVersions.forVersion(IOVersions.java:83)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
tree.io.IOVersions.forPage(IOVersions.java:95)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.tree.
PendingRow.initKey(PendingRow.java:72) ~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.tree.
PendingEntriesTree.getRow(PendingEntriesTree.java:118)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.tree.
PendingEntriesTree.getRow(PendingEntriesTree.java:31)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4441)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.findLowerUnbounded(BPlusTree.java:910)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3]
... 17 more
And if we are reading from cache...
27.02.18 00:27:56 ERROR MessagePartProcessingHandler - error processing
incoming messagePartDeliveryReceipt=MessagePartDeliveryReceipt{id=null,
abonent=7****, externalId='7586F73B'}
org.springframework.dao.DataAccessResourceFailureException: Runtime failure
on bounds: [lower=null, upper=PendingRow []]; nested exception is class
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
at com.a1s.sdp.data.dao.ignite.IgniteExceptionTranslator.
translateExceptionIfPossible(IgniteExceptionTranslator.java:35)
~[core-data-1.5-SNAPSHOT.jar:3752]
at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:41)
~[core-data-1.5-SNAPSHOT.jar:3752]
at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.
findOne(BaseIgniteCassandraRepository.java:136)
~[core-data-1.5-SNAPSHOT.jar:3752]
at com.a1s.sdp.common.service.TransportServiceImpl.
processMessagePartDelivered(TransportServiceImpl.java:213)
~[core-common-1.5-SNAPSHOT.jar:3752]
at com.a1s.sdp.common.service.TransportServiceImpl.
messagePartDelivered(TransportServiceImpl.java:202)
~[core-common-1.5-SNAPSHOT.jar:3752]
at com.a1s.sdp.common.transport.handler.
MessagePartProcessingHandler.lambda$channelReadMessagePartDeliveryReceipt$1(
MessagePartProcessingHandler.java:91) ~[core-common-1.5-SNAPSHOT.jar:3752]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_151]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:*1.8.0_151*]
Caused by: org.apache.ignite.IgniteException: *Runtime failure on bounds:
[lower=null, upper=PendingRow []]*
at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
*IgniteCacheOffheapManagerImpl.**expire*(IgniteCacheOffheapManagerImpl.java:979)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
GridCacheUtils.unwindEvicts(GridCacheUtils.java:833)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
GridCacheGateway.leaveNoLock(GridCacheGateway.java:240)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
GridCacheGateway.leave(GridCacheGateway.java:225)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
GatewayProtectedCacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:608)
~[ignite-core-2.3.3.jar:2.3.3]
at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.
lambda$findOne$7(BaseIgniteCassandraRepository.java:136)
~[core-data-1.5-SNAPSHOT.jar:3752]
at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39)
~[core-data-1.5-SNAPSHOT.jar:3752]
... 9 more
Caused by: java.lang.IllegalStateException: Failed to get page IO instance
(page content is corrupted)
at org.apache.ignite.internal.processors.cache.persistence.
tree.io.IOVersions.forVersion(IOVersions.java:83)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
tree.io.IOVersions.forPage(IOVersions.java:95)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.tree.
PendingRow.initKey(PendingRow.java:72) ~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.tree.
PendingEntriesTree.getRow(PendingEntriesTree.java:118)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.tree.
PendingEntriesTree.getRow(PendingEntriesTree.java:31)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4441)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.findLowerUnbounded(BPlusTree.java:910)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
*IgniteCacheOffheapManagerImpl.**expire*(IgniteCacheOffheapManagerImpl.java:979)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
GridCacheUtils.unwindEvicts(GridCacheUtils.java:833)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
GridCacheGateway.leaveNoLock(GridCacheGateway.java:240)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
GridCacheGateway.leave(GridCacheGateway.java:225)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
GatewayProtectedCacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510)
~[ignite-core-2.3.3.jar:2.3.3]
at org.apache.ignite.internal.processors.cache.
GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:608)
~[ignite-core-2.3.3.jar:2.3.3]
at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.
lambda$findOne$7(BaseIgniteCassandraRepository.java:136)
~[core-data-1.5-SNAPSHOT.jar:3752]
at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39)
~[core-data-1.5-SNAPSHOT.jar:3752]
... 9 more
The attachment (https://yadi.sk/d/u7fSK8Y53SpCpA) contains a part of cache
obtained from node of cluster - /var/ignite/***/db/
node00-66fd86f5-4e2d-4f1b-8fe1-993bf5937d99/cache-MessagePartSent.
This cache contains objects of class MessagePartSent:
@Table("message_part_sent_external")
public final class MessagePartSent {
@Id
private MessagePartSentPk id;
private MessagePartPk partId;
public MessagePartSentPk getId() {
return id;
}
public void setId(MessagePartSentPk id) {
this.id = id;
}
public MessagePartPk getPartId() {
return partId;
}
public void setPartId(MessagePartPk partId) {
this.partId = partId;
}
@Override
public String toString() {
final StringBuilder sb = new StringBuilder("MessagePartSent{");
sb.append("id=").append(id);
sb.append(", partId=").append(partId);
sb.append('}');
return sb.toString();
}
public static final class MessagePartSentPk implements Serializable {
private Integer linkId;
private String externalId;
public MessagePartSentPk() {
//
}
public MessagePartSentPk(Integer linkId, String externalId) {
this.linkId = linkId;
this.externalId = externalId;
}
public Integer getLinkId() {
return linkId;
}
public void setLinkId(Integer linkId) {
this.linkId = linkId;
}
public String getExternalId() {
return externalId;
}
public void setExternalId(String externalId) {
this.externalId = externalId;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof MessagePartSentPk)) return false;
final MessagePartSentPk that = (MessagePartSentPk) o;
if (externalId != null ? !externalId.equals(that.externalId) :
that.externalId != null) return false;
if (linkId != null ? !linkId.equals(that.linkId) : that.linkId
!= null) return false;
return true;
}
@Override
public int hashCode() {
int result = linkId != null ? linkId.hashCode() : 0;
result = 31 * result + (externalId != null ?
externalId.hashCode() : 0);
return result;
}
@Override
public String toString() {
final StringBuilder sb = new StringBuilder("
MessagePartSentPk{");
sb.append("linkId=").append(linkId);
sb.append(", externalId='").append(externalId).append('\'');
sb.append('}');
return sb.toString();
}
}
}
public final class MessagePartPk implements Serializable {
private UUID transactionId;
private Short partNumber;
public MessagePartPk() {
//default
}
public MessagePartPk(UUID transactionId, Short partNumber) {
this.transactionId = transactionId;
this.partNumber = partNumber;
}
public UUID getTransactionId() {
return transactionId;
}
public void setTransactionId(UUID transactionId) {
this.transactionId = transactionId;
}
public Short getPartNumber() {
return partNumber;
}
public void setPartNumber(Short partNumber) {
this.partNumber = partNumber;
}
@Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof MessagePartPk)) return false;
final MessagePartPk messagePartPk = (MessagePartPk) o;
if (partNumber != null ? !partNumber.equals(messagePartPk.partNumber)
:
messagePartPk.partNumber != null) return false;
if (transactionId != null ?
!transactionId.equals(messagePartPk.transactionId)
:
messagePartPk.transactionId != null)
return false;
return true;
}
@Override
public int hashCode() {
int result = transactionId != null ? transactionId.hashCode() : 0;
result = 31 * result + (partNumber != null ? partNumber.hashCode()
: 0);
return result;
}
@Override
public String toString() {
final StringBuilder sb = new StringBuilder("MessagePartPk{");
sb.append("transactionId=").append(transactionId);
sb.append(", partNumber=").append(partNumber);
sb.append('}');
return sb.toString();
}
On Thu, Mar 8, 2018 at 11:03 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:
> Hi Sergey,
>
> The 2.4 release is about to be voted for. You can use the RC1 in the mean
> time:
> http://apache-ignite-developers.2346864.n4.nabble.
> com/VOTE-Apache-Ignite-2-4-0-RC1-td27687.html
>
> D.
>
> On Mon, Feb 19, 2018 at 6:43 AM, Mikhail <mi...@gmail.com>
> wrote:
>
>> Hi Sergey,
>>
>> The release of 2.4 should be soon, in a week or couple, however, there's
>> no
>> strong schedule for Apache releases.
>>
>> Could you please share a reproducer for the issue? Might be you can share
>> a
>> storage on which the issue can be reproduced?
>>
>> Thanks,
>> Mike.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>
--
Best regards,
Sergey S. Sergeev (aka SSSerj)
Notes: https://twitter.com/ssserj
exit(0);
Re: And again... Failed to get page IO instance (page content is corrupted)
Posted by Dmitriy Setrakyan <ds...@apache.org>.
Hi Sergey,
The 2.4 release is about to be voted for. You can use the RC1 in the mean
time:
http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Apache-Ignite-2-4-0-RC1-td27687.html
D.
On Mon, Feb 19, 2018 at 6:43 AM, Mikhail <mi...@gmail.com>
wrote:
> Hi Sergey,
>
> The release of 2.4 should be soon, in a week or couple, however, there's no
> strong schedule for Apache releases.
>
> Could you please share a reproducer for the issue? Might be you can share a
> storage on which the issue can be reproduced?
>
> Thanks,
> Mike.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: And again... Failed to get page IO instance (page content is
corrupted)
Posted by Mikhail <mi...@gmail.com>.
Hi Sergey,
The release of 2.4 should be soon, in a week or couple, however, there's no
strong schedule for Apache releases.
Could you please share a reproducer for the issue? Might be you can share a
storage on which the issue can be reproduced?
Thanks,
Mike.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/