You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "Lo, Marcus " <ma...@citi.com> on 2021/06/09 06:16:56 UTC
Ignite crashed with CorruptedTreeException
Hi,
We have a 4 node ignite (v.2.10.0) cluster setup, running on redhat 7 with 60G non-heap cache and 10G heap, with persistence enabled. We find that all nodes would crash with CorruptedTreeException when we do the following:
1. Start all nodes as background process, and activate the cluster
2. Create a table and index via SQL:
CREATE TABLE IF NOT EXISTS LIMIT_VIEW_STATUS
(
viewId UUID,
batchId INT,
instance VARCHAR,
nodes VARCHAR,
status VARCHAR,
eqtgContext VARCHAR,
lastUpdateTime TIMESTAMP,
PRIMARY KEY (status, viewId)
)
with "template=RiskDataTemplate, affinity_key=viewId, cache_name=LimitViewStatusCache";
CREATE INDEX IF NOT EXISTS limit_view_status_batchid_instance_idx ON LIMIT_VIEW_STATUS (batchId,instance);
1. Insert data into the table concurrently by 5 thin clients.
2. Inactivate the cluster while thin clients are still inserting data.
3. Stop the nodes one by one by killing the ignite process (normal SIGTERM kill).
4. Restart all nodes one by one.
5. Insert data again into the table concurrently by 5 thin clients, and then all the nodes would crash.
Please find the attached for the config, log file (for one node) and zipped wal/pds files (for one node). I have changed the wal file size to 512Kb just to make the wal files small - the issue can be reproduced using default wal size (64Mb), or even very large wal size (slightly less than 2Gb).
Can anyone shed some light on the issue? Thanks.
Regards,
Marcus
Re: Ignite crashed with CorruptedTreeException
Posted by Maksim Timonin <ti...@gmail.com>.
Hi, Marcus!
Great! You're welcome.
On Wed, Jun 16, 2021 at 10:08 AM Lo, Marcus <ma...@citi.com> wrote:
> Hi Maksim,
>
>
>
> Thanks. We have tested the workaround and it mitigates the issue. 😊
>
>
>
> Regards,
>
> Marcus
>
>
>
> *From:* [gmail.com] Maksim Timonin <ti...@gmail.com>
> *Sent:* Thursday, June 10, 2021 5:52 PM
> *To:* user@ignite.apache.org
> *Subject:* Re: Ignite crashed with CorruptedTreeException
>
>
>
> Hi, Marcus! I've found a bug. It was already fixed within this ticket
> https://issues.apache.org/jira/browse/IGNITE-14451
> <https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/IGNITE-14451__;!!Jkho33Y!2lE2mKO1rPq_tr8J_acm_2Mv5oZTq6xaapgVJ-IHWfrlVJ4S-U7_ZtNwyLi9IA$>.
> So it will be available in Ignite 2.11.
>
>
>
> Reason is that you declare fields in your table in order (viewId, status)
> while PK has order (status, viewId). So, change the order in the table and
> it will be fine. This bug affects only Primary Key indexes, so it's safe to
> declare fields in different order in secondary indexes.
>
>
>
> On Thu, Jun 10, 2021 at 10:58 AM Maksim Timonin <ti...@gmail.com>
> wrote:
>
> hi, Marcus!
>
>
>
> Thank you for the reproducer, I've succeed in reproducing it. Create a
> ticket for that https://issues.apache.org/jira/browse/IGNITE-14869
> <https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/IGNITE-14869__;!!Jkho33Y!2lE2mKO1rPq_tr8J_acm_2Mv5oZTq6xaapgVJ-IHWfrlVJ4S-U7_ZtN-vi96WA$>
>
>
>
> I will return to you with a workaround, after finding a reason.
>
>
>
> On Thu, Jun 10, 2021 at 3:18 AM Lo, Marcus <ma...@citi.com> wrote:
>
> Hi Ivan, Maksim,
>
> Here is the reproducer:
>
> import org.apache.ignite.Ignition;
> import org.apache.ignite.binary.BinaryObject;
> import org.apache.ignite.binary.BinaryObjectBuilder;
> import org.apache.ignite.cache.QueryEntity;
> import org.apache.ignite.client.ClientCache;
> import org.apache.ignite.client.IgniteClient;
> import org.apache.ignite.configuration.BinaryConfiguration;
> import org.apache.ignite.configuration.ClientConfiguration;
> import org.junit.jupiter.api.Test;
>
> import java.sql.Timestamp;
> import java.time.Instant;
> import java.util.*;
> import java.util.stream.IntStream;
>
> import static java.util.stream.Collectors.toList;
>
> public class Reproducer {
>
> @Test
> public void reproduce() throws InterruptedException {
> ClientConfiguration config = constructIgniteThinClientConfig();
> IgniteClient ignite = Ignition.startClient(config);
>
> List<UUID> uuids = IntStream.range(0, 200).mapToObj((i) ->
> UUID.randomUUID()).collect(toList());
>
> while (true) {
> upsertLimitViewData(ignite, uuids);
> Thread.sleep(1000);
> }
> }
>
> private void upsertLimitViewData(IgniteClient ignite, List<UUID>
> uuids) {
> System.out.println("[" + Instant.now() + "] upserting data... " +
> Thread.currentThread().getName());
>
> ClientCache<Object, Object> cache =
> ignite.cache("LimitViewStatusCache").withKeepBinary();
> QueryEntity queryEntity =
> cache.getConfiguration().getQueryEntities()[0];
> BinaryObjectBuilder keyBuilder =
> ignite.binary().builder(queryEntity.getKeyType());
> BinaryObjectBuilder valueBuilder =
> ignite.binary().builder(queryEntity.getValueType());
> HashMap<Object, Object> valueMap = new HashMap<>();
>
> for (int i = 0; i < 200; i++) {
> BinaryObject key = keyBuilder
> .setField("viewId", uuids.get(i))
> .setField("status", "moo")
> .build();
>
> BinaryObject value = valueBuilder
> .setField("batchId", new Random().nextInt())
> .setField("instance", Integer.toString(new
> Random().nextInt()))
> .setField("nodes", Integer.toString(new
> Random().nextInt()))
> .setField("eqtgContext", Integer.toString(new
> Random().nextInt()))
> .setField("lastUpdateTime",
> Timestamp.from(Instant.now()))
> .build();
>
> valueMap.put(key, value);
> }
>
> cache.putAll(valueMap);
> }
>
> private ClientConfiguration constructIgniteThinClientConfig() {
> return
> new ClientConfiguration()
> .setAddresses("xxx:10800")
> .setPartitionAwarenessEnabled(false)
> .setBinaryConfiguration(new
> BinaryConfiguration().setCompactFooter(false))
> .setUserName("xxx")
> .setUserPassword("xxx");
> }
> }
>
>
> Regards,
> Marcus
>
> -----Original Message-----
> From: [External] Maksim Timonin <ti...@gmail.com>
> Sent: Thursday, June 10, 2021 12:31 AM
> To: user@ignite.apache.org
> Subject: Re: Ignite crashed with CorruptedTreeException
>
> Hi Marcus!
>
> Could you please provide a complete code that inserts data (either it is
> SQL, or cache put, which types do you use, etc.). I've tried to reproduce
> your case but failed.
>
> Thanks a lot!
>
>
>
> --
> Sent from:
> https://urldefense.com/v3/__http://apache-ignite-users.70518.x6.nabble.com/__;!!Jkho33Y!2v--HF_tnHWeR_0YefFDx-NcnoY3hkO-9G94IAXG23N6qzB_qz-rSYtuciav3A$
> <https://urldefense.com/v3/__http:/apache-ignite-users.70518.x6.nabble.com/__;!!Jkho33Y!2v--HF_tnHWeR_0YefFDx-NcnoY3hkO-9G94IAXG23N6qzB_qz-rSYtuciav3A$>
>
>
RE: Ignite crashed with CorruptedTreeException
Posted by "Lo, Marcus " <ma...@citi.com>.
Hi Maksim,
Thanks. We have tested the workaround and it mitigates the issue. 😊
Regards,
Marcus
From: [gmail.com] Maksim Timonin <ti...@gmail.com>
Sent: Thursday, June 10, 2021 5:52 PM
To: user@ignite.apache.org
Subject: Re: Ignite crashed with CorruptedTreeException
Hi, Marcus! I've found a bug. It was already fixed within this ticket https://issues.apache.org/jira/browse/IGNITE-14451<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/IGNITE-14451__;!!Jkho33Y!2lE2mKO1rPq_tr8J_acm_2Mv5oZTq6xaapgVJ-IHWfrlVJ4S-U7_ZtNwyLi9IA$>. So it will be available in Ignite 2.11.
Reason is that you declare fields in your table in order (viewId, status) while PK has order (status, viewId). So, change the order in the table and it will be fine. This bug affects only Primary Key indexes, so it's safe to declare fields in different order in secondary indexes.
On Thu, Jun 10, 2021 at 10:58 AM Maksim Timonin <ti...@gmail.com>> wrote:
hi, Marcus!
Thank you for the reproducer, I've succeed in reproducing it. Create a ticket for that https://issues.apache.org/jira/browse/IGNITE-14869<https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/IGNITE-14869__;!!Jkho33Y!2lE2mKO1rPq_tr8J_acm_2Mv5oZTq6xaapgVJ-IHWfrlVJ4S-U7_ZtN-vi96WA$>
I will return to you with a workaround, after finding a reason.
On Thu, Jun 10, 2021 at 3:18 AM Lo, Marcus <ma...@citi.com>> wrote:
Hi Ivan, Maksim,
Here is the reproducer:
import org.apache.ignite.Ignition;
import org.apache.ignite.binary.BinaryObject;
import org.apache.ignite.binary.BinaryObjectBuilder;
import org.apache.ignite.cache.QueryEntity;
import org.apache.ignite.client.ClientCache;
import org.apache.ignite.client.IgniteClient;
import org.apache.ignite.configuration.BinaryConfiguration;
import org.apache.ignite.configuration.ClientConfiguration;
import org.junit.jupiter.api.Test;
import java.sql.Timestamp;
import java.time.Instant;
import java.util.*;
import java.util.stream.IntStream;
import static java.util.stream.Collectors.toList;
public class Reproducer {
@Test
public void reproduce() throws InterruptedException {
ClientConfiguration config = constructIgniteThinClientConfig();
IgniteClient ignite = Ignition.startClient(config);
List<UUID> uuids = IntStream.range(0, 200).mapToObj((i) -> UUID.randomUUID()).collect(toList());
while (true) {
upsertLimitViewData(ignite, uuids);
Thread.sleep(1000);
}
}
private void upsertLimitViewData(IgniteClient ignite, List<UUID> uuids) {
System.out.println("[" + Instant.now() + "] upserting data... " + Thread.currentThread().getName());
ClientCache<Object, Object> cache = ignite.cache("LimitViewStatusCache").withKeepBinary();
QueryEntity queryEntity = cache.getConfiguration().getQueryEntities()[0];
BinaryObjectBuilder keyBuilder = ignite.binary().builder(queryEntity.getKeyType());
BinaryObjectBuilder valueBuilder = ignite.binary().builder(queryEntity.getValueType());
HashMap<Object, Object> valueMap = new HashMap<>();
for (int i = 0; i < 200; i++) {
BinaryObject key = keyBuilder
.setField("viewId", uuids.get(i))
.setField("status", "moo")
.build();
BinaryObject value = valueBuilder
.setField("batchId", new Random().nextInt())
.setField("instance", Integer.toString(new Random().nextInt()))
.setField("nodes", Integer.toString(new Random().nextInt()))
.setField("eqtgContext", Integer.toString(new Random().nextInt()))
.setField("lastUpdateTime", Timestamp.from(Instant.now()))
.build();
valueMap.put(key, value);
}
cache.putAll(valueMap);
}
private ClientConfiguration constructIgniteThinClientConfig() {
return
new ClientConfiguration()
.setAddresses("xxx:10800")
.setPartitionAwarenessEnabled(false)
.setBinaryConfiguration(new BinaryConfiguration().setCompactFooter(false))
.setUserName("xxx")
.setUserPassword("xxx");
}
}
Regards,
Marcus
-----Original Message-----
From: [External] Maksim Timonin <ti...@gmail.com>>
Sent: Thursday, June 10, 2021 12:31 AM
To: user@ignite.apache.org<ma...@ignite.apache.org>
Subject: Re: Ignite crashed with CorruptedTreeException
Hi Marcus!
Could you please provide a complete code that inserts data (either it is SQL, or cache put, which types do you use, etc.). I've tried to reproduce your case but failed.
Thanks a lot!
--
Sent from: https://urldefense.com/v3/__http://apache-ignite-users.70518.x6.nabble.com/__;!!Jkho33Y!2v--HF_tnHWeR_0YefFDx-NcnoY3hkO-9G94IAXG23N6qzB_qz-rSYtuciav3A$<https://urldefense.com/v3/__http:/apache-ignite-users.70518.x6.nabble.com/__;!!Jkho33Y!2v--HF_tnHWeR_0YefFDx-NcnoY3hkO-9G94IAXG23N6qzB_qz-rSYtuciav3A$>
Re: Ignite crashed with CorruptedTreeException
Posted by Maksim Timonin <ti...@gmail.com>.
Hi, Marcus! I've found a bug. It was already fixed within this ticket
https://issues.apache.org/jira/browse/IGNITE-14451. So it will be
available in Ignite 2.11.
Reason is that you declare fields in your table in order (viewId, status)
while PK has order (status, viewId). So, change the order in the table and
it will be fine. This bug affects only Primary Key indexes, so it's safe to
declare fields in different order in secondary indexes.
On Thu, Jun 10, 2021 at 10:58 AM Maksim Timonin <ti...@gmail.com>
wrote:
> hi, Marcus!
>
> Thank you for the reproducer, I've succeed in reproducing it. Create a
> ticket for that https://issues.apache.org/jira/browse/IGNITE-14869
>
> I will return to you with a workaround, after finding a reason.
>
> On Thu, Jun 10, 2021 at 3:18 AM Lo, Marcus <ma...@citi.com> wrote:
>
>> Hi Ivan, Maksim,
>>
>> Here is the reproducer:
>>
>> import org.apache.ignite.Ignition;
>> import org.apache.ignite.binary.BinaryObject;
>> import org.apache.ignite.binary.BinaryObjectBuilder;
>> import org.apache.ignite.cache.QueryEntity;
>> import org.apache.ignite.client.ClientCache;
>> import org.apache.ignite.client.IgniteClient;
>> import org.apache.ignite.configuration.BinaryConfiguration;
>> import org.apache.ignite.configuration.ClientConfiguration;
>> import org.junit.jupiter.api.Test;
>>
>> import java.sql.Timestamp;
>> import java.time.Instant;
>> import java.util.*;
>> import java.util.stream.IntStream;
>>
>> import static java.util.stream.Collectors.toList;
>>
>> public class Reproducer {
>>
>> @Test
>> public void reproduce() throws InterruptedException {
>> ClientConfiguration config = constructIgniteThinClientConfig();
>> IgniteClient ignite = Ignition.startClient(config);
>>
>> List<UUID> uuids = IntStream.range(0, 200).mapToObj((i) ->
>> UUID.randomUUID()).collect(toList());
>>
>> while (true) {
>> upsertLimitViewData(ignite, uuids);
>> Thread.sleep(1000);
>> }
>> }
>>
>> private void upsertLimitViewData(IgniteClient ignite, List<UUID>
>> uuids) {
>> System.out.println("[" + Instant.now() + "] upserting data... " +
>> Thread.currentThread().getName());
>>
>> ClientCache<Object, Object> cache =
>> ignite.cache("LimitViewStatusCache").withKeepBinary();
>> QueryEntity queryEntity =
>> cache.getConfiguration().getQueryEntities()[0];
>> BinaryObjectBuilder keyBuilder =
>> ignite.binary().builder(queryEntity.getKeyType());
>> BinaryObjectBuilder valueBuilder =
>> ignite.binary().builder(queryEntity.getValueType());
>> HashMap<Object, Object> valueMap = new HashMap<>();
>>
>> for (int i = 0; i < 200; i++) {
>> BinaryObject key = keyBuilder
>> .setField("viewId", uuids.get(i))
>> .setField("status", "moo")
>> .build();
>>
>> BinaryObject value = valueBuilder
>> .setField("batchId", new Random().nextInt())
>> .setField("instance", Integer.toString(new
>> Random().nextInt()))
>> .setField("nodes", Integer.toString(new
>> Random().nextInt()))
>> .setField("eqtgContext", Integer.toString(new
>> Random().nextInt()))
>> .setField("lastUpdateTime",
>> Timestamp.from(Instant.now()))
>> .build();
>>
>> valueMap.put(key, value);
>> }
>>
>> cache.putAll(valueMap);
>> }
>>
>> private ClientConfiguration constructIgniteThinClientConfig() {
>> return
>> new ClientConfiguration()
>> .setAddresses("xxx:10800")
>> .setPartitionAwarenessEnabled(false)
>> .setBinaryConfiguration(new
>> BinaryConfiguration().setCompactFooter(false))
>> .setUserName("xxx")
>> .setUserPassword("xxx");
>> }
>> }
>>
>>
>> Regards,
>> Marcus
>>
>> -----Original Message-----
>> From: [External] Maksim Timonin <ti...@gmail.com>
>> Sent: Thursday, June 10, 2021 12:31 AM
>> To: user@ignite.apache.org
>> Subject: Re: Ignite crashed with CorruptedTreeException
>>
>> Hi Marcus!
>>
>> Could you please provide a complete code that inserts data (either it is
>> SQL, or cache put, which types do you use, etc.). I've tried to reproduce
>> your case but failed.
>>
>> Thanks a lot!
>>
>>
>>
>> --
>> Sent from:
>> https://urldefense.com/v3/__http://apache-ignite-users.70518.x6.nabble.com/__;!!Jkho33Y!2v--HF_tnHWeR_0YefFDx-NcnoY3hkO-9G94IAXG23N6qzB_qz-rSYtuciav3A$
>>
>
Re: Ignite crashed with CorruptedTreeException
Posted by Maksim Timonin <ti...@gmail.com>.
hi, Marcus!
Thank you for the reproducer, I've succeed in reproducing it. Create a
ticket for that https://issues.apache.org/jira/browse/IGNITE-14869
I will return to you with a workaround, after finding a reason.
On Thu, Jun 10, 2021 at 3:18 AM Lo, Marcus <ma...@citi.com> wrote:
> Hi Ivan, Maksim,
>
> Here is the reproducer:
>
> import org.apache.ignite.Ignition;
> import org.apache.ignite.binary.BinaryObject;
> import org.apache.ignite.binary.BinaryObjectBuilder;
> import org.apache.ignite.cache.QueryEntity;
> import org.apache.ignite.client.ClientCache;
> import org.apache.ignite.client.IgniteClient;
> import org.apache.ignite.configuration.BinaryConfiguration;
> import org.apache.ignite.configuration.ClientConfiguration;
> import org.junit.jupiter.api.Test;
>
> import java.sql.Timestamp;
> import java.time.Instant;
> import java.util.*;
> import java.util.stream.IntStream;
>
> import static java.util.stream.Collectors.toList;
>
> public class Reproducer {
>
> @Test
> public void reproduce() throws InterruptedException {
> ClientConfiguration config = constructIgniteThinClientConfig();
> IgniteClient ignite = Ignition.startClient(config);
>
> List<UUID> uuids = IntStream.range(0, 200).mapToObj((i) ->
> UUID.randomUUID()).collect(toList());
>
> while (true) {
> upsertLimitViewData(ignite, uuids);
> Thread.sleep(1000);
> }
> }
>
> private void upsertLimitViewData(IgniteClient ignite, List<UUID>
> uuids) {
> System.out.println("[" + Instant.now() + "] upserting data... " +
> Thread.currentThread().getName());
>
> ClientCache<Object, Object> cache =
> ignite.cache("LimitViewStatusCache").withKeepBinary();
> QueryEntity queryEntity =
> cache.getConfiguration().getQueryEntities()[0];
> BinaryObjectBuilder keyBuilder =
> ignite.binary().builder(queryEntity.getKeyType());
> BinaryObjectBuilder valueBuilder =
> ignite.binary().builder(queryEntity.getValueType());
> HashMap<Object, Object> valueMap = new HashMap<>();
>
> for (int i = 0; i < 200; i++) {
> BinaryObject key = keyBuilder
> .setField("viewId", uuids.get(i))
> .setField("status", "moo")
> .build();
>
> BinaryObject value = valueBuilder
> .setField("batchId", new Random().nextInt())
> .setField("instance", Integer.toString(new
> Random().nextInt()))
> .setField("nodes", Integer.toString(new
> Random().nextInt()))
> .setField("eqtgContext", Integer.toString(new
> Random().nextInt()))
> .setField("lastUpdateTime",
> Timestamp.from(Instant.now()))
> .build();
>
> valueMap.put(key, value);
> }
>
> cache.putAll(valueMap);
> }
>
> private ClientConfiguration constructIgniteThinClientConfig() {
> return
> new ClientConfiguration()
> .setAddresses("xxx:10800")
> .setPartitionAwarenessEnabled(false)
> .setBinaryConfiguration(new
> BinaryConfiguration().setCompactFooter(false))
> .setUserName("xxx")
> .setUserPassword("xxx");
> }
> }
>
>
> Regards,
> Marcus
>
> -----Original Message-----
> From: [External] Maksim Timonin <ti...@gmail.com>
> Sent: Thursday, June 10, 2021 12:31 AM
> To: user@ignite.apache.org
> Subject: Re: Ignite crashed with CorruptedTreeException
>
> Hi Marcus!
>
> Could you please provide a complete code that inserts data (either it is
> SQL, or cache put, which types do you use, etc.). I've tried to reproduce
> your case but failed.
>
> Thanks a lot!
>
>
>
> --
> Sent from:
> https://urldefense.com/v3/__http://apache-ignite-users.70518.x6.nabble.com/__;!!Jkho33Y!2v--HF_tnHWeR_0YefFDx-NcnoY3hkO-9G94IAXG23N6qzB_qz-rSYtuciav3A$
>
RE: Ignite crashed with CorruptedTreeException
Posted by "Lo, Marcus " <ma...@citi.com>.
Hi Ivan, Maksim,
Here is the reproducer:
import org.apache.ignite.Ignition;
import org.apache.ignite.binary.BinaryObject;
import org.apache.ignite.binary.BinaryObjectBuilder;
import org.apache.ignite.cache.QueryEntity;
import org.apache.ignite.client.ClientCache;
import org.apache.ignite.client.IgniteClient;
import org.apache.ignite.configuration.BinaryConfiguration;
import org.apache.ignite.configuration.ClientConfiguration;
import org.junit.jupiter.api.Test;
import java.sql.Timestamp;
import java.time.Instant;
import java.util.*;
import java.util.stream.IntStream;
import static java.util.stream.Collectors.toList;
public class Reproducer {
@Test
public void reproduce() throws InterruptedException {
ClientConfiguration config = constructIgniteThinClientConfig();
IgniteClient ignite = Ignition.startClient(config);
List<UUID> uuids = IntStream.range(0, 200).mapToObj((i) -> UUID.randomUUID()).collect(toList());
while (true) {
upsertLimitViewData(ignite, uuids);
Thread.sleep(1000);
}
}
private void upsertLimitViewData(IgniteClient ignite, List<UUID> uuids) {
System.out.println("[" + Instant.now() + "] upserting data... " + Thread.currentThread().getName());
ClientCache<Object, Object> cache = ignite.cache("LimitViewStatusCache").withKeepBinary();
QueryEntity queryEntity = cache.getConfiguration().getQueryEntities()[0];
BinaryObjectBuilder keyBuilder = ignite.binary().builder(queryEntity.getKeyType());
BinaryObjectBuilder valueBuilder = ignite.binary().builder(queryEntity.getValueType());
HashMap<Object, Object> valueMap = new HashMap<>();
for (int i = 0; i < 200; i++) {
BinaryObject key = keyBuilder
.setField("viewId", uuids.get(i))
.setField("status", "moo")
.build();
BinaryObject value = valueBuilder
.setField("batchId", new Random().nextInt())
.setField("instance", Integer.toString(new Random().nextInt()))
.setField("nodes", Integer.toString(new Random().nextInt()))
.setField("eqtgContext", Integer.toString(new Random().nextInt()))
.setField("lastUpdateTime", Timestamp.from(Instant.now()))
.build();
valueMap.put(key, value);
}
cache.putAll(valueMap);
}
private ClientConfiguration constructIgniteThinClientConfig() {
return
new ClientConfiguration()
.setAddresses("xxx:10800")
.setPartitionAwarenessEnabled(false)
.setBinaryConfiguration(new BinaryConfiguration().setCompactFooter(false))
.setUserName("xxx")
.setUserPassword("xxx");
}
}
Regards,
Marcus
-----Original Message-----
From: [External] Maksim Timonin <ti...@gmail.com>
Sent: Thursday, June 10, 2021 12:31 AM
To: user@ignite.apache.org
Subject: Re: Ignite crashed with CorruptedTreeException
Hi Marcus!
Could you please provide a complete code that inserts data (either it is SQL, or cache put, which types do you use, etc.). I've tried to reproduce your case but failed.
Thanks a lot!
--
Sent from: https://urldefense.com/v3/__http://apache-ignite-users.70518.x6.nabble.com/__;!!Jkho33Y!2v--HF_tnHWeR_0YefFDx-NcnoY3hkO-9G94IAXG23N6qzB_qz-rSYtuciav3A$
Re: Ignite crashed with CorruptedTreeException
Posted by Maksim Timonin <ti...@gmail.com>.
Hi Marcus!
Could you please provide a complete code that inserts data (either it is
SQL, or cache put, which types do you use, etc.). I've tried to reproduce
your case but failed.
Thanks a lot!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite crashed with CorruptedTreeException
Posted by Ivan Daschinsky <iv...@gmail.com>.
I mean that you have accidentally insert string key, not UUID. During
insertion, ignite checks types of key during updating index B+ tree
and this check currently lead to CorruptedTreeException.
ср, 9 июн. 2021 г. в 17:37, Ivan Daschinsky <iv...@gmail.com>:
>
> Hi Marcus. Yes, it is a bug on ignite, but I suppose that you insert
> UUID instead of String
> Here is the actual cause of CorruptedTreeException:
>
> Caused by: java.lang.UnsupportedOperationException: Value type doesn't
> match: exp=20, act=13
> at org.apache.ignite.internal.processors.query.h2.database.inlinecolumn.AbstractInlineIndexColumn.ensureValueType(AbstractInlineIndexColumn.java:193)
>
> 20 -- is alias for UUID, 13 is alias for String. Could you please show
> your code that inserts data?
>
>
> ср, 9 июн. 2021 г. в 09:18, Lo, Marcus <ma...@citi.com>:
> >
> > Hi,
> >
> >
> >
> > We have a 4 node ignite (v.2.10.0) cluster setup, running on redhat 7 with 60G non-heap cache and 10G heap, with persistence enabled. We find that all nodes would crash with CorruptedTreeException when we do the following:
> >
> >
> >
> > Start all nodes as background process, and activate the cluster
> > Create a table and index via SQL:
> >
> > CREATE TABLE IF NOT EXISTS LIMIT_VIEW_STATUS
> >
> > (
> >
> > viewId UUID,
> >
> > batchId INT,
> >
> > instance VARCHAR,
> >
> > nodes VARCHAR,
> >
> > status VARCHAR,
> >
> > eqtgContext VARCHAR,
> >
> > lastUpdateTime TIMESTAMP,
> >
> > PRIMARY KEY (status, viewId)
> >
> > )
> >
> > with "template=RiskDataTemplate, affinity_key=viewId, cache_name=LimitViewStatusCache";
> >
> >
> >
> > CREATE INDEX IF NOT EXISTS limit_view_status_batchid_instance_idx ON LIMIT_VIEW_STATUS (batchId,instance);
> >
> > Insert data into the table concurrently by 5 thin clients.
> > Inactivate the cluster while thin clients are still inserting data.
> > Stop the nodes one by one by killing the ignite process (normal SIGTERM kill).
> > Restart all nodes one by one.
> > Insert data again into the table concurrently by 5 thin clients, and then all the nodes would crash.
> >
> >
> >
> > Please find the attached for the config, log file (for one node) and zipped wal/pds files (for one node). I have changed the wal file size to 512Kb just to make the wal files small – the issue can be reproduced using default wal size (64Mb), or even very large wal size (slightly less than 2Gb).
> >
> >
> >
> > Can anyone shed some light on the issue? Thanks.
> >
> >
> >
> > Regards,
> >
> > Marcus
> >
> >
>
>
>
> --
> Sincerely yours, Ivan Daschinskiy
--
Sincerely yours, Ivan Daschinskiy
Re: Ignite crashed with CorruptedTreeException
Posted by Ivan Daschinsky <iv...@gmail.com>.
Hi Marcus. Yes, it is a bug on ignite, but I suppose that you insert
UUID instead of String
Here is the actual cause of CorruptedTreeException:
Caused by: java.lang.UnsupportedOperationException: Value type doesn't
match: exp=20, act=13
at org.apache.ignite.internal.processors.query.h2.database.inlinecolumn.AbstractInlineIndexColumn.ensureValueType(AbstractInlineIndexColumn.java:193)
20 -- is alias for UUID, 13 is alias for String. Could you please show
your code that inserts data?
ср, 9 июн. 2021 г. в 09:18, Lo, Marcus <ma...@citi.com>:
>
> Hi,
>
>
>
> We have a 4 node ignite (v.2.10.0) cluster setup, running on redhat 7 with 60G non-heap cache and 10G heap, with persistence enabled. We find that all nodes would crash with CorruptedTreeException when we do the following:
>
>
>
> Start all nodes as background process, and activate the cluster
> Create a table and index via SQL:
>
> CREATE TABLE IF NOT EXISTS LIMIT_VIEW_STATUS
>
> (
>
> viewId UUID,
>
> batchId INT,
>
> instance VARCHAR,
>
> nodes VARCHAR,
>
> status VARCHAR,
>
> eqtgContext VARCHAR,
>
> lastUpdateTime TIMESTAMP,
>
> PRIMARY KEY (status, viewId)
>
> )
>
> with "template=RiskDataTemplate, affinity_key=viewId, cache_name=LimitViewStatusCache";
>
>
>
> CREATE INDEX IF NOT EXISTS limit_view_status_batchid_instance_idx ON LIMIT_VIEW_STATUS (batchId,instance);
>
> Insert data into the table concurrently by 5 thin clients.
> Inactivate the cluster while thin clients are still inserting data.
> Stop the nodes one by one by killing the ignite process (normal SIGTERM kill).
> Restart all nodes one by one.
> Insert data again into the table concurrently by 5 thin clients, and then all the nodes would crash.
>
>
>
> Please find the attached for the config, log file (for one node) and zipped wal/pds files (for one node). I have changed the wal file size to 512Kb just to make the wal files small – the issue can be reproduced using default wal size (64Mb), or even very large wal size (slightly less than 2Gb).
>
>
>
> Can anyone shed some light on the issue? Thanks.
>
>
>
> Regards,
>
> Marcus
>
>
--
Sincerely yours, Ivan Daschinskiy