You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by " Kirill Sizov (Jira)" <ji...@apache.org> on 2024/12/06 11:08:00 UTC

[jira] [Comment Edited] (IGNITE-23675) WatchProcessor#notifyWatches:notificationFuture exception handling is missing

    [ https://issues.apache.org/jira/browse/IGNITE-23675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17903620#comment-17903620 ] 

 Kirill Sizov edited comment on IGNITE-23675 at 12/6/24 11:07 AM:
------------------------------------------------------------------

Ax example of the failure (TableManagerTest.testStoragesGetClearedInMiddleOfFailedPartitionStorageRebalance() fails):
{noformat}
successfully written to meta storage [tableId=4, assignments= [0=[Assignment [consistentId=node1, isPeer=true]]]].
[10:44:52] :         [:ignite-table:test] 
[10:44:52] :         [:ignite-table:test] TableManagerTest > testStoragesGetClearedInMiddleOfFailedPartitionStorageRebalance() STANDARD_ERROR
[10:44:52] :         [:ignite-table:test]     [2024-12-06T10:44:52,131][ERROR][%node1%metastorage-watch-executor-0][WatchProcessor] Failed to notify watches.
[10:44:52] :         [:ignite-table:test]     java.util.concurrent.CompletionException: java.lang.AssertionError: Causality token mismatch, expected 2, got 3
[10:44:52] :         [:ignite-table:test]         at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) ~[?:?]
[10:44:52] :         [:ignite-table:test]         at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319) ~[?:?]
[10:44:52] :         [:ignite-table:test]         at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1081) ~[?:?]
[10:44:52] :         [:ignite-table:test]         at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?]
[10:44:52] :         [:ignite-table:test]         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
[10:44:52] :         [:ignite-table:test]         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
[10:44:52] :         [:ignite-table:test]         at java.base/java.lang.Thread.run(Thread.java:834) [?:?]
[10:44:52] :         [:ignite-table:test]     Caused by: java.lang.AssertionError: Causality token mismatch, expected 2, got 3
[10:44:52] :         [:ignite-table:test]         at org.apache.ignite.internal.causality.IncrementalVersionedValue.completeInternal(IncrementalVersionedValue.java:266) ~[ignite-core-3.0.0-SNAPSHOT.jar:?]
[10:44:52] :         [:ignite-table:test]         at org.apache.ignite.internal.causality.IncrementalVersionedValue$1.onUpdate(IncrementalVersionedValue.java:112) ~[ignite-core-3.0.0-SNAPSHOT.jar:?]
[10:44:52] :         [:ignite-table:test]         at org.apache.ignite.internal.metastorage.server.WatchProcessor.notifyUpdateRevisionListeners(WatchProcessor.java:409) ~[ignite-metastorage-3.0.0-SNAPSHOT.jar:?]
[10:44:52] :         [:ignite-table:test]         at org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$notifyWatches$4(WatchProcessor.java:206) ~[ignite-metastorage-3.0.0-SNAPSHOT.jar:?]
[10:44:52] :         [:ignite-table:test]         at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) ~[?:?]
[10:44:52] :         [:ignite-table:test]         ... 4 more
[10:44:52] :         [:ignite-table:test]  {noformat}


was (Author: JIRAUSER301198):
Ax example of the failure:
{noformat}
successfully written to meta storage [tableId=4, assignments= [0=[Assignment [consistentId=node1, isPeer=true]]]].
[10:44:52] :         [:ignite-table:test] 
[10:44:52] :         [:ignite-table:test] TableManagerTest > testStoragesGetClearedInMiddleOfFailedPartitionStorageRebalance() STANDARD_ERROR
[10:44:52] :         [:ignite-table:test]     [2024-12-06T10:44:52,131][ERROR][%node1%metastorage-watch-executor-0][WatchProcessor] Failed to notify watches.
[10:44:52] :         [:ignite-table:test]     java.util.concurrent.CompletionException: java.lang.AssertionError: Causality token mismatch, expected 2, got 3
[10:44:52] :         [:ignite-table:test]         at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) ~[?:?]
[10:44:52] :         [:ignite-table:test]         at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319) ~[?:?]
[10:44:52] :         [:ignite-table:test]         at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1081) ~[?:?]
[10:44:52] :         [:ignite-table:test]         at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?]
[10:44:52] :         [:ignite-table:test]         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
[10:44:52] :         [:ignite-table:test]         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
[10:44:52] :         [:ignite-table:test]         at java.base/java.lang.Thread.run(Thread.java:834) [?:?]
[10:44:52] :         [:ignite-table:test]     Caused by: java.lang.AssertionError: Causality token mismatch, expected 2, got 3
[10:44:52] :         [:ignite-table:test]         at org.apache.ignite.internal.causality.IncrementalVersionedValue.completeInternal(IncrementalVersionedValue.java:266) ~[ignite-core-3.0.0-SNAPSHOT.jar:?]
[10:44:52] :         [:ignite-table:test]         at org.apache.ignite.internal.causality.IncrementalVersionedValue$1.onUpdate(IncrementalVersionedValue.java:112) ~[ignite-core-3.0.0-SNAPSHOT.jar:?]
[10:44:52] :         [:ignite-table:test]         at org.apache.ignite.internal.metastorage.server.WatchProcessor.notifyUpdateRevisionListeners(WatchProcessor.java:409) ~[ignite-metastorage-3.0.0-SNAPSHOT.jar:?]
[10:44:52] :         [:ignite-table:test]         at org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$notifyWatches$4(WatchProcessor.java:206) ~[ignite-metastorage-3.0.0-SNAPSHOT.jar:?]
[10:44:52] :         [:ignite-table:test]         at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072) ~[?:?]
[10:44:52] :         [:ignite-table:test]         ... 4 more
[10:44:52] :         [:ignite-table:test]  {noformat}

> WatchProcessor#notifyWatches:notificationFuture exception handling is missing
> -----------------------------------------------------------------------------
>
>                 Key: IGNITE-23675
>                 URL: https://issues.apache.org/jira/browse/IGNITE-23675
>             Project: Ignite
>          Issue Type: Bug
>            Reporter: Alexander Lapin
>            Priority: Major
>              Labels: ignite-3
>
> h3. Motivation
> Following assertions aren't logged
> {code:java}
>     private CompletableFuture<?> completeInternal(long causalityToken) {
>         synchronized (updateMutex) {
>             assert expectedToken == -1 || expectedToken == causalityToken
>                     : String.format("Causality token mismatch, expected %d, got %d", expectedToken, causalityToken);
>             assert causalityToken > lastCompleteToken : String.format(
>                     "Causality token must be greater than the last completed: [token=%s, lastCompleted=%s]", causalityToken,
>                     lastCompleteToken);
>             assert causalityToken > lastDeletedToken : String.format(
>                     "Causality token must be greater than the last deleted: [token=%s, lastDeleted=%s]", causalityToken,
>                     lastDeletedToken);
>  
> {code}
> and notificationFuture in WatchProcessor is hanged forever.
>  
> In order to fix that 
> {code:java}
>                 .whenCompleteAsync((unused, throwable) -> {
>                     if (throwable != null) {
>                         LOG.error("Failed to notify watches.", throwable);
>                         notifyFailureHandlerOnFirstFailureInNotificationChain(throwable);
>                     }
>                 }, watchExecutor)
> {code}
> or similar should be added to WatchProcessor#notificationFuture exception handling.
> That will highlight some [flaky|https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_RunAllTests/8617621?expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandBuildProblemsSection=true&expandCode+Inspection=true] tests that should be fixed - this is the biggest difficulty here.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)