[jira] [Commented] (IGNITE-24119) Multi-get in explicit RO transaction is broken

["Mikhail Efremov (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IGNITE-24119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18030693#comment-18030693 ] 

Mikhail Efremov commented on IGNITE-24119:
------------------------------------------

It's 100% reproducer, it works now over default zone (25 partitions and only 1 replica) that leads to disjunctive replicas distribution among 2 nodes. We may simplify it to 2 partitions with 1 replicas, so node 3344 will host partition 0 replica and node 3345 will host partition 1 replica both of the same table.

Under the hood (bad) we will get {{ReplicaUnavailableException}} that is produced inside {{ReplicaManager}} iif there no a corresponding partition replica or the replica's future wasn't completed somewhy (equals to not created-started). So if we'll do a breakpoint on {{replicaFut == null || !replicaFut.isDone()}} check we'll find the true stacktrace that leads to {{InternalTableImpl#getAll}} that calls {{partitionRowBatch.getValue().resultFuture = replicaSvc.invoke(recipientNode, request)}}. The most wonder thing there is the fact that {{recipientNode}} is the same for *ALL* batches that are relates to different partitions. How do we define this node? Up to the stack:

{code:java}
if (tx != null && tx.isReadOnly()) {
    assert !tx.implicit() : "implicit RO getAll not supported";

    BinaryRowEx firstRow = keyRows.iterator().next();

    return evaluateReadOnlyRecipientNode(partitionId(firstRow), tx.readTimestamp())
            .thenCompose(recipientNode -> getAll(keyRows, tx.readTimestamp(), tx.id(), tx.coordinatorId(), recipientNode));
}
{code}

Inside of {{evaluateReadOnlyRecipientNode}} we effectively do {{awaitPrimaryReplica}} that will return a primary replica for a partition of the only first row. If the row relates to 0th partition then all RO batch requests for *ALL* partitions will sent on node 3344, that doesn't contain any replica of partition 1, that will lead to {{ReplicaUnavailableException}}.

The solution is pretty strightforward: we have to call {{evaluateReadOnlyRecipientNode}} for every partition batch:

{code:java}
for (Int2ObjectMap.Entry<RowBatch> partitionRowBatch : rowBatchByPartitionId.int2ObjectEntrySet()) {
    int partitionId = partitionRowBatch.getIntKey();

    ReplicationGroupId replicationGroupId = targetReplicationGroupId(partitionId);

    ReadOnlyMultiRowPkReplicaRequest request = // ...

    partitionRowBatch.getValue().resultFuture = evaluateReadOnlyRecipientNode(partitionId, readTimestamp)
            .thenCompose(recipient -> replicaSvc.invoke(recipient, request));
 }
{code}

And the last, but not least: the reason why this issue wasn't triggered before because at least {{ItInternalTableTest}} are doing on 1 node cluster, so the scenario isn't working in non-distributive scenario. We can try to remove {{initialNodes}} overriding and find that 1/3 of tests in the class fails, even the scan operations.



> Multi-get in explicit RO transaction is broken
> ----------------------------------------------
>
>                 Key: IGNITE-24119
>                 URL: https://issues.apache.org/jira/browse/IGNITE-24119
>             Project: Ignite
>          Issue Type: Bug
>            Reporter: Roman Puchkovskiy
>            Assignee: Mikhail Efremov
>            Priority: Blocker
>              Labels: ignite-3, release-3.1.0-blocker
>         Attachments: ItMultiGetInExplicitReadOnlyTxTest.java
>
>
> Reproducer is attached. It demonstrates a simple scenario:
>  # Create a table in a 2-node cluster
>  # Put 100 keys (so that each of nodes gets at least one partition)
>  # Start an RO transaction
>  # Make a getAll() in that transaction passing all 100 keys
> After one minute, getAll() fails with the following:
> org.apache.ignite.tx.TransactionException: IGN-REP-3 TraceId:f526112d-d46a-490f-b932-f0b30e599454 Could not wait for the replica readiness due to timeout [replicaGroupId=TablePartitionIdMessageImpl [partitionId=0, tableId=16], req=ReadOnlyMultiRowPkReplicaRequestImpl]
>  
> at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
> at org.apache.ignite.internal.util.ExceptionUtils$1.copy(ExceptionUtils.java:838)
> at org.apache.ignite.internal.util.ExceptionUtils$ExceptionFactory.createCopy(ExceptionUtils.java:772)
> at org.apache.ignite.internal.util.ExceptionUtils.copyExceptionWithCause(ExceptionUtils.java:574)
> at org.apache.ignite.internal.util.ViewUtils.copyExceptionWithCauseIfPossible(ViewUtils.java:91)
> at org.apache.ignite.internal.util.ViewUtils.ensurePublicException(ViewUtils.java:71)
> at org.apache.ignite.internal.util.ViewUtils.sync(ViewUtils.java:54)
> at org.apache.ignite.internal.table.KeyValueViewImpl.getAll(KeyValueViewImpl.java:193)
> at org.apache.ignite.internal.table.PublicApiThreadingKeyValueView.lambda$getAll$6(PublicApiThreadingKeyValueView.java:87)
> at org.apache.ignite.internal.thread.PublicApiThreading.executeWithRole(PublicApiThreading.java:144)
> at org.apache.ignite.internal.thread.PublicApiThreading.execUserSyncOperation(PublicApiThreading.java:102)
> at org.apache.ignite.internal.table.PublicApiThreadingViewBase.executeSyncOp(PublicApiThreadingViewBase.java:107)
> at org.apache.ignite.internal.table.PublicApiThreadingKeyValueView.getAll(PublicApiThreadingKeyValueView.java:87)
> at org.apache.ignite.internal.restart.RestartProofKeyValueView.lambda$getAll$6(RestartProofKeyValueView.java:88)
> at org.apache.ignite.internal.restart.RestartProofApiObject.lambda$attached$0(RestartProofApiObject.java:46)
> at org.apache.ignite.internal.restart.IgniteAttachmentLock.attached(IgniteAttachmentLock.java:59)
> at org.apache.ignite.internal.restart.RestartProofApiObject.attached(RestartProofApiObject.java:46)
> at org.apache.ignite.internal.restart.RestartProofKeyValueView.getAll(RestartProofKeyValueView.java:88)
> at org.apache.ignite.internal.tx.readonly.ItMultiGetInExplicitReadOnlyTxTest.roTransactionNoticesGarbageCollectedTupleVersionsOnDataNodes(ItMultiGetInExplicitReadOnlyTxTest.java:54)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
> at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
> Caused by: org.apache.ignite.tx.TransactionException: IGN-REP-3 TraceId:f526112d-d46a-490f-b932-f0b30e599454 Could not wait for the replica readiness due to timeout [replicaGroupId=TablePartitionIdMessageImpl [partitionId=0, tableId=16], req=ReadOnlyMultiRowPkReplicaRequestImpl]
> at org.apache.ignite.internal.tx.TransactionExceptionMapperProvider.lambda$mappers$1(TransactionExceptionMapperProvider.java:44)
> at org.apache.ignite.internal.lang.IgniteExceptionMapper.map(IgniteExceptionMapper.java:61)
> at org.apache.ignite.internal.lang.IgniteExceptionMapperUtil.map(IgniteExceptionMapperUtil.java:195)
> at org.apache.ignite.internal.lang.IgniteExceptionMapperUtil.mapToPublicException(IgniteExceptionMapperUtil.java:125)
> at org.apache.ignite.internal.lang.IgniteExceptionMapperUtil.mapToPublicException(IgniteExceptionMapperUtil.java:88)
> at org.apache.ignite.internal.lang.IgniteExceptionMapperUtil.lambda$convertToPublicFuture$3(IgniteExceptionMapperUtil.java:178)
> at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
> at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)
> at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at org.apache.ignite.internal.replicator.ReplicaService.lambda$sendToReplicaRaw$5(ReplicaService.java:204)
> at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
> at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)
> at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.ignite.internal.replicator.exception.ReplicationTimeoutException: IGN-REP-3 TraceId:f526112d-d46a-490f-b932-f0b30e599454 Could not wait for the replica readiness due to timeout [replicaGroupId=TablePartitionIdMessageImpl [partitionId=0, tableId=16], req=ReadOnlyMultiRowPkReplicaRequestImpl]
> at org.apache.ignite.internal.util.ExceptionUtils.lambda$withCause$1(ExceptionUtils.java:433)
> at org.apache.ignite.internal.util.ExceptionUtils.withCauseInternal(ExceptionUtils.java:498)
> at org.apache.ignite.internal.util.ExceptionUtils.withCause(ExceptionUtils.java:433)
> ... 7 more
> Caused by: java.util.concurrent.TimeoutException
> at org.apache.ignite.internal.future.timeout.TimeoutWorker.body(TimeoutWorker.java:90)
> at org.apache.ignite.internal.util.worker.IgniteWorker.run(IgniteWorker.java:108)
> ... 1 more



--
This message was sent by Atlassian Jira
(v8.20.10#820010)