You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by " Kirill Sizov (Jira)" <ji...@apache.org> on 2024/05/17 13:43:00 UTC
[jira] [Commented] (IGNITE-22207) Handling of AwaitPrimaryReplica request maight lead to the dedalock
[ https://issues.apache.org/jira/browse/IGNITE-22207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17847308#comment-17847308 ]
Kirill Sizov commented on IGNITE-22207:
----------------------------------------
There is a deadlock between {{ReplicaService.pendingInvokes}} and {{ReplicaManager.replicas}}
See threaddump:
{noformat}
"%sqllogic0%low-watermark-updater-0" #273 prio=5 os_prio=0 cpu=1503.23ms elapsed=2391.82s tid=0x00007fd3492b1000 nid=0x2527f6 waiting for monitor entry [0x00007fd289870000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.concurrent.ConcurrentHashMap.replaceNode(java.base@11.0.17/ConcurrentHashMap.java:1122)
- waiting to lock <0x000000071ab00178> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
at java.util.concurrent.ConcurrentHashMap.remove(java.base@11.0.17/ConcurrentHashMap.java:1552)
at org.apache.ignite.internal.replicator.ReplicaService.lambda$sendToReplica$3(ReplicaService.java:167)
at org.apache.ignite.internal.replicator.ReplicaService$$Lambda$3914/0x000000080117c840.apply(Unknown Source)
at java.util.concurrent.CompletableFuture.uniHandle(java.base@11.0.17/CompletableFuture.java:930)
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(java.base@11.0.17/CompletableFuture.java:907)
at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.17/CompletableFuture.java:506)
at java.util.concurrent.CompletableFuture.complete(java.base@11.0.17/CompletableFuture.java:2073)
at org.apache.ignite.internal.network.DefaultMessagingService.onInvokeResponse(DefaultMessagingService.java:524)
at org.apache.ignite.internal.network.DefaultMessagingService.send0(DefaultMessagingService.java:240)
at org.apache.ignite.internal.network.DefaultMessagingService.respond(DefaultMessagingService.java:183)
at org.apache.ignite.internal.network.DefaultMessagingService.respond(DefaultMessagingService.java:196)
at org.apache.ignite.internal.network.MessagingService.respond(MessagingService.java:142)
at org.apache.ignite.internal.replicator.ReplicaManager.lambda$handleReplicaRequest$2(ReplicaManager.java:301)
at org.apache.ignite.internal.replicator.ReplicaManager$$Lambda$3898/0x0000000801179040.accept(Unknown Source)
at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@11.0.17/CompletableFuture.java:859)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@11.0.17/CompletableFuture.java:837)
at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.17/CompletableFuture.java:506)
at java.util.concurrent.CompletableFuture.completeExceptionally(java.base@11.0.17/CompletableFuture.java:2088)
at org.apache.ignite.internal.replicator.ReplicaManager.lambda$stopReplicaInternal$17(ReplicaManager.java:630)
at org.apache.ignite.internal.replicator.ReplicaManager$$Lambda$4502/0x0000000801292040.apply(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.compute(java.base@11.0.17/ConcurrentHashMap.java:1932)
- locked <0x0000000719f589f8> (a java.util.concurrent.ConcurrentHashMap$Node)
at org.apache.ignite.internal.replicator.ReplicaManager.lambda$stopReplicaInternal$18(ReplicaManager.java:624)
at org.apache.ignite.internal.replicator.ReplicaManager$$Lambda$4501/0x0000000801291c40.accept(Unknown Source)
at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@11.0.17/CompletableFuture.java:859)
at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(java.base@11.0.17/CompletableFuture.java:883)
at java.util.concurrent.CompletableFuture.whenComplete(java.base@11.0.17/CompletableFuture.java:2251)
at org.apache.ignite.internal.replicator.ReplicaManager.stopReplicaInternal(ReplicaManager.java:612)
at org.apache.ignite.internal.replicator.ReplicaManager.stopReplica(ReplicaManager.java:595)
at org.apache.ignite.internal.table.distributed.TableManager.stopPartition(TableManager.java:2413)
at org.apache.ignite.internal.table.distributed.TableManager.destroyTableLocally(TableManager.java:1519)
at org.apache.ignite.internal.table.distributed.TableManager.lambda$onLwmChanged$22(TableManager.java:807)
at org.apache.ignite.internal.table.distributed.TableManager$$Lambda$3174/0x0000000800ffc040.apply(Unknown Source)
at java.util.stream.ReferencePipeline$3$1.accept(java.base@11.0.17/ReferencePipeline.java:195)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(java.base@11.0.17/ArrayList.java:1655)
at java.util.stream.AbstractPipeline.copyInto(java.base@11.0.17/AbstractPipeline.java:484)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@11.0.17/AbstractPipeline.java:474)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(java.base@11.0.17/ReduceOps.java:913)
at java.util.stream.AbstractPipeline.evaluate(java.base@11.0.17/AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(java.base@11.0.17/ReferencePipeline.java:578)
at org.apache.ignite.internal.table.distributed.TableManager.lambda$onLwmChanged$25(TableManager.java:808)
at org.apache.ignite.internal.table.distributed.TableManager$$Lambda$3173/0x0000000800ffb440.get(Unknown Source)
at org.apache.ignite.internal.util.IgniteUtils.inBusyLockAsync(IgniteUtils.java:890)
at org.apache.ignite.internal.table.distributed.TableManager.onLwmChanged(TableManager.java:803)
at org.apache.ignite.internal.table.distributed.TableManager.lambda$startAsync$5(TableManager.java:631)
at org.apache.ignite.internal.table.distributed.TableManager$$Lambda$1978/0x0000000800c24040.notify(Unknown Source)
at org.apache.ignite.internal.event.AbstractEventProducer.fireEvent(AbstractEventProducer.java:88)
at org.apache.ignite.internal.lowwatermark.LowWatermarkImpl.lambda$updateAndNotify$8(LowWatermarkImpl.java:306)
at org.apache.ignite.internal.lowwatermark.LowWatermarkImpl$$Lambda$3166/0x0000000800fecc40.apply(Unknown Source)
at java.util.concurrent.CompletableFuture$UniCompose.tryFire(java.base@11.0.17/CompletableFuture.java:1072)
at java.util.concurrent.CompletableFuture$Completion.run(java.base@11.0.17/CompletableFuture.java:478)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.17/Executors.java:515)
at java.util.concurrent.FutureTask.run(java.base@11.0.17/FutureTask.java:264)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.17/ScheduledThreadPoolExecutor.java:304)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.17/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.17/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.17/Thread.java:834)
{noformat}
and this:
{noformat}
"%sqllogic0%partition-operations-4" #485 prio=5 os_prio=0 cpu=9301.10ms elapsed=2387.08s tid=0x00007fd35899c000 nid=0x2528cc waiting for monitor entry [0x00007fd278711000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.concurrent.ConcurrentHashMap.compute(java.base@11.0.17/ConcurrentHashMap.java:1923)
- waiting to lock <0x0000000719f589f8> (a java.util.concurrent.ConcurrentHashMap$Node)
at org.apache.ignite.internal.replicator.ReplicaManager.handleReplicaRequest(ReplicaManager.java:293)
at org.apache.ignite.internal.replicator.ReplicaManager.onReplicaMessageReceived(ReplicaManager.java:255)
at org.apache.ignite.internal.replicator.ReplicaManager$$Lambda$751/0x00000008004fe840.onReceived(Unknown Source)
at org.apache.ignite.internal.network.DefaultMessagingService.sendToSelf(DefaultMessagingService.java:354)
at org.apache.ignite.internal.network.DefaultMessagingService.invoke0(DefaultMessagingService.java:287)
at org.apache.ignite.internal.network.DefaultMessagingService.invoke(DefaultMessagingService.java:215)
at org.apache.ignite.internal.network.wrapper.JumpToExecutorByConsistentIdAfterSend.invoke(JumpToExecutorByConsistentIdAfterSend.java:97)
at org.apache.ignite.internal.network.MessagingService.invoke(MessagingService.java:198)
at org.apache.ignite.internal.replicator.ReplicaService.lambda$sendToReplica$1(ReplicaService.java:158)
at org.apache.ignite.internal.replicator.ReplicaService$$Lambda$3884/0x0000000801172c40.apply(Unknown Source)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(java.base@11.0.17/ConcurrentHashMap.java:1705)
- locked <0x000000071ab00178> (a java.util.concurrent.ConcurrentHashMap$ReservationNode)
at org.apache.ignite.internal.replicator.ReplicaService.lambda$sendToReplica$4(ReplicaService.java:151)
at org.apache.ignite.internal.replicator.ReplicaService$$Lambda$3872/0x000000080116d440.accept(Unknown Source)
at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@11.0.17/CompletableFuture.java:859)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@11.0.17/CompletableFuture.java:837)
at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.17/CompletableFuture.java:506)
at java.util.concurrent.CompletableFuture.complete(java.base@11.0.17/CompletableFuture.java:2073)
at org.apache.ignite.internal.network.DefaultMessagingService.onInvokeResponse(DefaultMessagingService.java:524)
at org.apache.ignite.internal.network.DefaultMessagingService.send0(DefaultMessagingService.java:240)
at org.apache.ignite.internal.network.DefaultMessagingService.respond(DefaultMessagingService.java:183)
at org.apache.ignite.internal.network.DefaultMessagingService.respond(DefaultMessagingService.java:196)
at org.apache.ignite.internal.network.MessagingService.respond(MessagingService.java:142)
at org.apache.ignite.internal.replicator.ReplicaManager.sendReplicaUnavailableErrorResponse(ReplicaManager.java:813)
at org.apache.ignite.internal.replicator.ReplicaManager.handleReplicaRequest(ReplicaManager.java:326)
at org.apache.ignite.internal.replicator.ReplicaManager.lambda$onReplicaMessageReceived$1(ReplicaManager.java:253)
at org.apache.ignite.internal.replicator.ReplicaManager$$Lambda$3871/0x000000080116d040.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.17/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.17/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.17/Thread.java:834)
{noformat}
> Handling of AwaitPrimaryReplica request maight lead to the dedalock
> -------------------------------------------------------------------
>
> Key: IGNITE-22207
> URL: https://issues.apache.org/jira/browse/IGNITE-22207
> Project: Ignite
> Issue Type: Bug
> Reporter: Vladislav Pyatkov
> Priority: Major
> Labels: ignite-3
>
> h3. Motivation
> We handle ReplicaUnavailableException in ReplicaService to handle request transparency even if the replica had not been read at the time the request was sent. To do this, we send AwaitReplicaRequest as a handling of the exception and then process the origin request when the replica is guaranteed ready.
> In the event that the AwaitReplicaRequest is handled locally, we will do it synchronously in pendingInvokes.computeIfAbsent:
> {code}
> CompletableFuture<NetworkMessage> awaitReplicaFut = pendingInvokes.computeIfAbsent(
> targetNodeConsistentId,
> consistentId -> {
> AwaitReplicaRequest awaitReplicaReq = REPLICA_MESSAGES_FACTORY.awaitReplicaRequest()
> .groupId(req.groupId())
> .build();
> return messagingService.invoke(
> targetNodeConsistentId,
> awaitReplicaReq,
> replicationConfiguration.rpcTimeout().value()
> );
> }
> );
> {code}
> and then hands on handling of AwaitReplicaRequest, due to removing from pendingInvokes:
> {code}
> awaitReplicaFut.handle((response0, throwable0) -> {
> pendingInvokes.remove(targetNodeConsistentId, awaitReplicaFut);
> ...
> {code}
> h3. Definition of doe
> Take out of the network invokation from the map compute closure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)