You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by mahesh76private <ma...@gmail.com> on 2019/01/16 04:48:39 UTC
failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES
On 2.7, we are regularly seeing the below message and then the nodes stop.
[16:45:04,759][SEVERE][disco-event-worker-#63][] JVM will be halted
immediately due to the failure: [failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class o.a.i.IgniteCheckedException: Maximum number
of retries 1000 reached for Put operation (the tree may be corrupted).
Increase IGNITE_BPLUS_TREE_LOCK_RETRIES system property if you regularly see
this message (current value is 1000).]]
Can you please through some light on what this error is?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES
Posted by mahesh76private <ma...@gmail.com>.
OK.
We don't have 1000 users for causing any sort of concurrency related lock
ups.
So possible explanation is corruption of the tree, below is our scenario
-a-
We created a table with sql - in the create params KEY_TYPE= ".....",
VALUE_TYPE="..." are also set.
The intention is to retrieve data inserted into SQL table via Key/Value API
for some ML code using ignite framework.
-b-
Once the table is created, we created an index on primary key column.
Question: is creating an index column on primary key required? Internally, I
assume you would create an index.
-c-
we started inserting rows into table via SQL insert.
In this scenario, we got the LOCK_RETRIES issue
Hope this gives you some clue as to why the issue occurred? Can you please
address the question in b)
regards
Mahesh
.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES
Posted by Stanislav Lukyanov <st...@gmail.com>.
Yes.
You can also use an environment variable instead of the system property:
IGNITE_BPLUS_TREE_LOCK_RETRIES=100000 ignite.sh …
Stan
From: mahesh76private
Sent: 16 января 2019 г. 11:29
To: user@ignite.apache.org
Subject: RE: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES
how do I set it ?
should I boot ignite node (ignite.sh) with the following switch ?
java ... -DIGNITE_BPLUS_TREE_LOCK_RETRIES=100000
regards
Mahesh
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES
Posted by mahesh76private <ma...@gmail.com>.
how do I set it ?
should I boot ignite node (ignite.sh) with the following switch ?
java ... -DIGNITE_BPLUS_TREE_LOCK_RETRIES=100000
regards
Mahesh
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Index inline is actually a vital parameter that can make up for multiple
orders of performance difference. You can find threads about this in users
list archive.
However, you should definitely not set it to anything comparable to page
size.
Regards,
--
Ilya Kasnacheev
ср, 30 янв. 2019 г. в 07:30, mahesh76private <ma...@gmail.com>:
> On further debug, we realised we set index inline option to 2048..while
> creating index. This problem (LOCK_RETRIES issue) started only after that.
> Before, that we didn't face much of a problem with indexes.
>
> Also, another notable observation is setting index inline option to 2048
> also made the index huge, ran into 100s of Gig...
>
> Anyway, we are not using this option anymore.... Frankly, it seems a
> internal detail of ignite, and not sure why this is given as an option to
> configure for users.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES
Posted by "ilya.kasnacheev" <il...@gmail.com>.
Hello!
I have filed an usability ticket
https://issues.apache.org/jira/browse/IGNITE-11162
Regards,
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES
Posted by mahesh76private <ma...@gmail.com>.
On further debug, we realised we set index inline option to 2048..while
creating index. This problem (LOCK_RETRIES issue) started only after that.
Before, that we didn't face much of a problem with indexes.
Also, another notable observation is setting index inline option to 2048
also made the index huge, ran into 100s of Gig...
Anyway, we are not using this option anymore.... Frankly, it seems a
internal detail of ignite, and not sure why this is given as an option to
configure for users.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES
Posted by mahesh76private <ma...@gmail.com>.
We actually think while building index, the tree is getting corrupted.
Increasing LOCK_RETRIES didnt fix it too...
Please review the below link as well. Infact, we saw creating index was
getting into a infinite loop and throwing out errors at other clients.
http://apache-ignite-users.70518.x6.nabble.com/processDynamicIndexChange-exception-at-client-node-while-creating-index-td26770.html
My suspicion is this issue happens only when table is partitioned, no
affinity keys, and definitely spread across multiple nodes.
Please review the logs
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
This is highly unusual, I guess the default value for this property should
be far larger that anything attainable in practice.
Dmitriy, Alexey, can you please clarify?
Regards.
--
Ilya Kasnacheev
вс, 27 янв. 2019 г. в 21:16, mahesh76private <ma...@gmail.com>:
> Increasing the size of IGNITE_BPLUS_TREE_LOCK_RETRIES makes the issue go
> away.
> But the explanation that is occurs due to contention is somewhat not
> clear.
>
> In all cases, in occurred to be when I was creating an index. Feels like it
> is somehow related to the height of the tree... and this variable seems to
> limit the height.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
This is highly unusual, I guess the default value for this property should
be far larger that anything attainable in practice.
Dmitriy, Alexey, can you please clarify?
Regards.
--
Ilya Kasnacheev
вс, 27 янв. 2019 г. в 21:16, mahesh76private <ma...@gmail.com>:
> Increasing the size of IGNITE_BPLUS_TREE_LOCK_RETRIES makes the issue go
> away.
> But the explanation that is occurs due to contention is somewhat not
> clear.
>
> In all cases, in occurred to be when I was creating an index. Feels like it
> is somehow related to the height of the tree... and this variable seems to
> limit the height.
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
RE: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES
Posted by mahesh76private <ma...@gmail.com>.
Increasing the size of IGNITE_BPLUS_TREE_LOCK_RETRIES makes the issue go
away.
But the explanation that is occurs due to contention is somewhat not clear.
In all cases, in occurred to be when I was creating an index. Feels like it
is somehow related to the height of the tree... and this variable seems to
limit the height.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
RE: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES
Posted by Stanislav Lukyanov <st...@gmail.com>.
It means that Ignite couldn’t find the place it needed in a B+ tree in 1000 iterations.
It could mean either that there is a high contention on the tree (it changes a lot, and
one thread is unlucky and couldn’t keep up with the speed), or that the tree is corrupted.
Try to set a larger value to the IGNITE_BPLUS_TREE_LOCK_RETRIES property (e.g. 100000).
If you see still see the exception then it’s a corruption. If you don’t – it’s a contention.
Stan
From: mahesh76private
Sent: 16 января 2019 г. 7:48
To: user@ignite.apache.org
Subject: failure due to IGNITE_BPLUS_TREE_LOCK_RETRIES
On 2.7, we are regularly seeing the below message and then the nodes stop.
[16:45:04,759][SEVERE][disco-event-worker-#63][] JVM will be halted
immediately due to the failure: [failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class o.a.i.IgniteCheckedException: Maximum number
of retries 1000 reached for Put operation (the tree may be corrupted).
Increase IGNITE_BPLUS_TREE_LOCK_RETRIES system property if you regularly see
this message (current value is 1000).]]
Can you please through some light on what this error is?
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/