You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Denis Chudov (Jira)" <ji...@apache.org> on 2024/01/30 10:36:00 UTC
[jira] [Updated] (IGNITE-21394) TimeoutException in the listener of pending assignments change shouldn't fail the watch processor
[ https://issues.apache.org/jira/browse/IGNITE-21394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denis Chudov updated IGNITE-21394:
----------------------------------
Description:
*Motivation*
The handler of pending assignments event change ( TableManager#handleChangePendingAssignmentEvent() ) tries to do changePeerAsync after starting the partition and client. In order to know whether the calling of changePeerAsync is needed, it tries to get the current leader of corresponding raft group. This call of RaftGroupService#refreshAndGetLeaderWithTerm can fail with TimeoutException. For example, there is no known leader on the node that the GetLeader request is sent to, or that node is no more in the raft group, etc., and in the same time that node is the only known peer of the raft group: in these cases the GetLeader request will be constantly retried in hope to get a response with leader finally, when it's elected, but this can never happen. So, the TimeoutException is expected in this case.
This exception should be handled within the mentioned listener of pending assignments event change. otherwise it fails the watch processor, making it unable to handle the further meta storage updates (and making the node inoperable). This means that, most likely, the current node is not a leader of the raft group, and changePeers shouln't be done, or it has not caught up with the current assignments events, this means that some client requests for this node for this partition will fail, but the node will remain operable.
*Definition of done*
TimeoutException in the listener of pending assignments change doesn't fail the watch processor and doesn't lead to multiple exceptions like this:
{code:java}
[2024-01-29T22:00:58,658][ERROR][%isckvt_tmccd_3344%Raft-Group-Client-5][WatchProcessor] Error occurred when notifying safe time advanced callback
java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) ~[?:?]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
at org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:546) ~[ignite-raft-3.0.0-SNAPSHOT.jar:?]
at org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$handleErrorResponse$42(RaftGroupServiceImpl.java:635) ~[ignite-raft-3.0.0-SNAPSHOT.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.util.concurrent.TimeoutException
... 8 more{code}
Implementation notes
It can be reproduced in integration tests like ItSchemaChangeKvViewTest#testMergeChangesColumnDefault when there are 3 nodes starting, then a table with 25 partitions/1 replica created. During the table start the rebalance is possible, like this:
* a replication group is moved from node A to node B
* some node C tries to perform GetLeader, and has only node A in local peers
* node A thinks it is the only member of the replication group, and is not leader, sends "Unknown leader" response to C
* node C constatnly retries the request to node A.
was:
*Motivation*
The handler of pending assignments event change ( TableManager#handleChangePendingAssignmentEvent() ) tries to do changePeerAsync after starting the partition and client. In order to know whether the calling of changePeerAsync is needed, it tries to get the current leader of corresponding raft group. This call of RaftGroupService#refreshAndGetLeaderWithTerm can fail with TimeoutException. For example, there is no known leader on the node that the GetLeader request is sent to, or that node is no more in the raft group, etc., and in the same time that node is the only known peer of the raft group: in these cases the GetLeader request will be constantly retried in hope to get a response with leader finally, when it's elected, but this can never happen. So, the TimeoutException is expected in this case.
This exception should be handled within the mentioned listener of pending assignments event change. otherwise it fails the watch processor, making it unable to handle the further meta storage updates (and making the node inoperable). This means that, most likely, the current node is not a leader of the raft group, and changePeers shouln't be done, or it has not caught up with the current assignments events, this means that some client requests for this node for this partition will fail, but the node will remain operable.
*Definition of done*
TimeoutException in the listener of pending assignments change doesn't fail the watch processor and doesn't lead to multiple exceptions like this:
{code:java}
024-01-29T22:00:58,658][ERROR][%isckvt_tmccd_3344%Raft-Group-Client-5][WatchProcessor] Error occurred when notifying safe time advanced callback java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) ~[?:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?] at org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:546) ~[ignite-raft-3.0.0-SNAPSHOT.jar:?] at org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$handleErrorResponse$42(RaftGroupServiceImpl.java:635) ~[ignite-raft-3.0.0-SNAPSHOT.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?]Caused by: java.util.concurrent.TimeoutException ... 8 more {code}
Implementation notes
It can be reproduced in integration tests like ItSchemaChangeKvViewTest#testMergeChangesColumnDefault when there are 3 nodes starting, then a table with 25 partitions/1 replica created. During the table start the rebalance is possible, like this:
* a replication group is moved from node A to node B
* some node C tries to perform GetLeader, and has only node A in local peers
* node A thinks it is the only member of the replication group, and is not leader, sends "Unknown leader" response to C
* node C constatnly retries the request to node A.
> TimeoutException in the listener of pending assignments change shouldn't fail the watch processor
> -------------------------------------------------------------------------------------------------
>
> Key: IGNITE-21394
> URL: https://issues.apache.org/jira/browse/IGNITE-21394
> Project: Ignite
> Issue Type: Bug
> Reporter: Denis Chudov
> Priority: Major
> Labels: ignite-3
>
> *Motivation*
> The handler of pending assignments event change ( TableManager#handleChangePendingAssignmentEvent() ) tries to do changePeerAsync after starting the partition and client. In order to know whether the calling of changePeerAsync is needed, it tries to get the current leader of corresponding raft group. This call of RaftGroupService#refreshAndGetLeaderWithTerm can fail with TimeoutException. For example, there is no known leader on the node that the GetLeader request is sent to, or that node is no more in the raft group, etc., and in the same time that node is the only known peer of the raft group: in these cases the GetLeader request will be constantly retried in hope to get a response with leader finally, when it's elected, but this can never happen. So, the TimeoutException is expected in this case.
> This exception should be handled within the mentioned listener of pending assignments event change. otherwise it fails the watch processor, making it unable to handle the further meta storage updates (and making the node inoperable). This means that, most likely, the current node is not a leader of the raft group, and changePeers shouln't be done, or it has not caught up with the current assignments events, this means that some client requests for this node for this partition will fail, but the node will remain operable.
> *Definition of done*
> TimeoutException in the listener of pending assignments change doesn't fail the watch processor and doesn't lead to multiple exceptions like this:
> {code:java}
> [2024-01-29T22:00:58,658][ERROR][%isckvt_tmccd_3344%Raft-Group-Client-5][WatchProcessor] Error occurred when notifying safe time advanced callback
> java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException
> at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[?:?]
> at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) ~[?:?]
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632) ~[?:?]
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
> at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?]
> at org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:546) ~[ignite-raft-3.0.0-SNAPSHOT.jar:?]
> at org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$handleErrorResponse$42(RaftGroupServiceImpl.java:635) ~[ignite-raft-3.0.0-SNAPSHOT.jar:?]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> Caused by: java.util.concurrent.TimeoutException
> ... 8 more{code}
> Implementation notes
>
> It can be reproduced in integration tests like ItSchemaChangeKvViewTest#testMergeChangesColumnDefault when there are 3 nodes starting, then a table with 25 partitions/1 replica created. During the table start the rebalance is possible, like this:
> * a replication group is moved from node A to node B
> * some node C tries to perform GetLeader, and has only node A in local peers
> * node A thinks it is the only member of the replication group, and is not leader, sends "Unknown leader" response to C
> * node C constatnly retries the request to node A.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)