You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ashish Kumar (Jira)" <ji...@apache.org> on 2023/08/30 06:04:00 UTC
[jira] [Resolved] (HDDS-9057) Key Put command fails with "Failed to send request, message=cmdType: WriteChunk"

     [ https://issues.apache.org/jira/browse/HDDS-9057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ashish Kumar resolved HDDS-9057.
--------------------------------
    Resolution: Done

> Key Put command fails with "Failed to send request, message=cmdType: WriteChunk"
> --------------------------------------------------------------------------------
>
>                 Key: HDDS-9057
>                 URL: https://issues.apache.org/jira/browse/HDDS-9057
>             Project: Apache Ozone
>          Issue Type: Bug
>          Components: Ozone CLI
>            Reporter: Pratyush Bhatt
>            Assignee: Ashish Kumar
>            Priority: Major
>
> While running: 
> {noformat}
> ozone sh key put o3://ozone1/vol-test-shutdownonedatanodewhilewrite-1689877090/buck-test-shutdownonedatanodewhilewrite-1689877090/WriteOzoneFile_1689877098 /tmp/OzoneRestartTest/WriteOzoneFile_1689877098{noformat}
> The command fails with:
> {noformat}
> 23/07/20 18:23:48 ERROR impl.OrderedAsync: Failed to send request, message=cmdType: WriteChunk{noformat}
> Related log:
> {code:java}
> 023-07-20 18:18:30,300|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|23/07/20 18:18:30 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-xceiverclientmetrics.properties,hadoop-metrics2.properties
> 2023-07-20 18:18:30,371|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|23/07/20 18:18:30 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
> 2023-07-20 18:18:30,371|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|23/07/20 18:18:30 INFO impl.MetricsSystemImpl: XceiverClientMetrics metrics system started
> 2023-07-20 18:18:30,876|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|23/07/20 18:18:30 INFO metrics.MetricRegistries: Loaded MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl
> 2023-07-20 18:18:31,057|INFO|MainThread|cm_apilib.py:1128 - stopComponent()|Initiating stop of OZONE_DATANODE at host quasar-pibudx-7.quasar-pibudx.root.hwx.site
> 2023-07-20 18:18:31,113|INFO|MainThread|cm_apilib.py:1132 - stopComponent()|Command name = Stop , ID = 1546344731  
> 2023-07-20 18:18:31,540|INFO|MainThread|cm_apilib.py:1128 - stopComponent()|Initiating stop of OZONE_DATANODE at host quasar-pibudx-5.quasar-pibudx.root.hwx.site
> 2023-07-20 18:18:31,574|INFO|MainThread|cm_apilib.py:1132 - stopComponent()|Command name = Stop , ID = 1546344734  
> 2023-07-20 18:18:31,923|INFO|MainThread|cm_apilib.py:1128 - stopComponent()|Initiating stop of OZONE_DATANODE at host quasar-pibudx-9.quasar-pibudx.root.hwx.site
> 2023-07-20 18:18:31,961|INFO|MainThread|cm_apilib.py:1132 - stopComponent()|Command name = Stop , ID = 1546344737  
> 2023-07-20 18:18:32,323|INFO|MainThread|cm_apilib.py:1128 - stopComponent()|Initiating stop of OZONE_DATANODE at host quasar-pibudx-1.quasar-pibudx.root.hwx.site
> 2023-07-20 18:18:32,360|INFO|MainThread|cm_apilib.py:1132 - stopComponent()|Command name = Stop , ID = 1546344741  
> 2023-07-20 18:23:48,792|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|23/07/20 18:23:48 ERROR impl.OrderedAsync: Failed to send request, message=cmdType: WriteChunk
> 2023-07-20 18:23:48,792|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|traceID: ""
> 2023-07-20 18:23:48,792|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,792|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|datanodeUuid: "17e7afb2-6bdb-4eb3-abf7-99c411ef4a56"
> 2023-07-20 18:23:48,793|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|writeChunk {
> 2023-07-20 18:23:48,793|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockID {
> 2023-07-20 18:23:48,793|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,793|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|localID: 111677748019230008
> 2023-07-20 18:23:48,793|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockCommitSequenceId: 0
> 2023-07-20 18:23:48,793|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|replicaIndex: 0
> 2023-07-20 18:23:48,793|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,793|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkData {
> 2023-07-20 18:23:48,793|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_4"
> 2023-07-20 18:23:48,793|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 12582912
> 2023-07-20 18:23:48,793|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,794|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,794|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,794|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,794|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,794|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,794|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,794|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,794|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,794|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,794|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,795|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|encodedToken: "WQoGaHJ0X3FhEiZjb25JRDogMTEwMDEgbG9jSUQ6IDExMTY3Nzc0ODAxOTIzMDAwOBiq8rLNlzEoASgCMICAgIABOhYI8pD5uvbmvoviARC0pKf82Lq1wqUBIEGPFTvxZP_g89BUJQ_Sr7RU9mjkII0acy7aS0bMLyxuEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDExMDAxIGxvY0lEOiAxMTE2Nzc3NDgwMTkyMzAwMDg"
> 2023-07-20 18:23:48,795|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|, data.size=4194304
> 2023-07-20 18:23:48,795|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|java.util.concurrent.CompletionException: org.apache.ratis.protocol.exceptions.RaftRetryFailureException: Failed RaftClientRequest:client-18BAE915ECAE->17e7afb2-6bdb-4eb3-abf7-99c411ef4a56@group-15BC114E0F45, cid=4, seq=5*, RW, cmdType: WriteChunk
> 2023-07-20 18:23:48,795|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|traceID: ""
> 2023-07-20 18:23:48,795|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,795|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|datanodeUuid: "17e7afb2-6bdb-4eb3-abf7-99c411ef4a56"
> 2023-07-20 18:23:48,795|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|writeChunk {
> 2023-07-20 18:23:48,795|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockID {
> 2023-07-20 18:23:48,795|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,795|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|localID: 111677748019230008
> 2023-07-20 18:23:48,795|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockCommitSequenceId: 0
> 2023-07-20 18:23:48,796|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|replicaIndex: 0
> 2023-07-20 18:23:48,796|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,796|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkData {
> 2023-07-20 18:23:48,796|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_4"
> 2023-07-20 18:23:48,796|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 12582912
> 2023-07-20 18:23:48,796|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,796|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,796|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,796|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,796|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,796|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,796|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,797|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,797|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,797|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,797|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,797|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|encodedToken: "WQoGaHJ0X3FhEiZjb25JRDogMTEwMDEgbG9jSUQ6IDExMTY3Nzc0ODAxOTIzMDAwOBiq8rLNlzEoASgCMICAgIABOhYI8pD5uvbmvoviARC0pKf82Lq1wqUBIEGPFTvxZP_g89BUJQ_Sr7RU9mjkII0acy7aS0bMLyxuEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDExMDAxIGxvY0lEOiAxMTE2Nzc3NDgwMTkyMzAwMDg"
> 2023-07-20 18:23:48,797|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|, data.size=4194304 for 31 attempts with RequestTypeDependentRetryPolicy{WRITE->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7}), WATCH->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->NoRetry})}
> 2023-07-20 18:23:48,797|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> 2023-07-20 18:23:48,797|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> 2023-07-20 18:23:48,797|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
> 2023-07-20 18:23:48,797|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> 2023-07-20 18:23:48,797|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2023-07-20 18:23:48,798|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2023-07-20 18:23:48,798|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.fail(OrderedAsync.java:104)
> 2023-07-20 18:23:48,798|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.util.SlidingWindow$Client.fail(SlidingWindow.java:392)
> 2023-07-20 18:23:48,798|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.failAllAsyncRequests(OrderedAsync.java:148)
> 2023-07-20 18:23:48,798|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.handleAsyncRetryFailure(OrderedAsync.java:152)
> 2023-07-20 18:23:48,798|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequest$10(OrderedAsync.java:262)
> 2023-07-20 18:23:48,798|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
> 2023-07-20 18:23:48,798|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
> 2023-07-20 18:23:48,798|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2023-07-20 18:23:48,798|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2023-07-20 18:23:48,799|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.completeReplyExceptionally(GrpcClientProtocolClient.java:394)
> 2023-07-20 18:23:48,799|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.access$000(GrpcClientProtocolClient.java:300)
> 2023-07-20 18:23:48,799|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:331)
> 2023-07-20 18:23:48,799|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
> 2023-07-20 18:23:48,799|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2023-07-20 18:23:48,799|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2023-07-20 18:23:48,799|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.metrics.intercept.client.MetricClientCallListener.onClose(MetricClientCallListener.java:47)
> 2023-07-20 18:23:48,799|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:471)
> 2023-07-20 18:23:48,799|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:435)
> 2023-07-20 18:23:48,799|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:468)
> 2023-07-20 18:23:48,799|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563)
> 2023-07-20 18:23:48,799|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
> 2023-07-20 18:23:48,800|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744)
> 2023-07-20 18:23:48,800|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
> 2023-07-20 18:23:48,800|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2023-07-20 18:23:48,800|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
> 2023-07-20 18:23:48,800|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2023-07-20 18:23:48,800|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2023-07-20 18:23:48,800|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.lang.Thread.run(Thread.java:750)
> 2023-07-20 18:23:48,800|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.protocol.exceptions.RaftRetryFailureException: Failed RaftClientRequest:client-18BAE915ECAE->17e7afb2-6bdb-4eb3-abf7-99c411ef4a56@group-15BC114E0F45, cid=4, seq=5*, RW, cmdType: WriteChunk
> 2023-07-20 18:23:48,800|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|traceID: ""
> 2023-07-20 18:23:48,800|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,800|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|datanodeUuid: "17e7afb2-6bdb-4eb3-abf7-99c411ef4a56"
> 2023-07-20 18:23:48,800|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|writeChunk {
> 2023-07-20 18:23:48,800|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockID {
> 2023-07-20 18:23:48,801|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,801|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|localID: 111677748019230008
> 2023-07-20 18:23:48,801|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockCommitSequenceId: 0
> 2023-07-20 18:23:48,801|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|replicaIndex: 0
> 2023-07-20 18:23:48,801|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,801|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkData {
> 2023-07-20 18:23:48,801|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_4"
> 2023-07-20 18:23:48,801|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 12582912
> 2023-07-20 18:23:48,801|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,801|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,801|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,801|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,801|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,801|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,801|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|encodedToken: "WQoGaHJ0X3FhEiZjb25JRDogMTEwMDEgbG9jSUQ6IDExMTY3Nzc0ODAxOTIzMDAwOBiq8rLNlzEoASgCMICAgIABOhYI8pD5uvbmvoviARC0pKf82Lq1wqUBIEGPFTvxZP_g89BUJQ_Sr7RU9mjkII0acy7aS0bMLyxuEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDExMDAxIGxvY0lEOiAxMTE2Nzc3NDgwMTkyMzAwMDg"
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|, data.size=4194304 for 31 attempts with RequestTypeDependentRetryPolicy{WRITE->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7}), WATCH->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->NoRetry})}
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.RaftClientImpl.noMoreRetries(RaftClientImpl.java:307)
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 25 more
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: java.io.IOException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:99)
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.GrpcUtil.unwrapIOException(GrpcUtil.java:159)
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:330)
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 16 more
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:539)
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 16 more
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /172.27.26.194:9858
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
> 2023-07-20 18:23:48,802|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687)
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489)
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.lang.Thread.run(Thread.java:750)
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|23/07/20 18:23:48 ERROR impl.OrderedAsync: Failed to send request, message=cmdType: PutBlock
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|traceID: ""
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|datanodeUuid: "17e7afb2-6bdb-4eb3-abf7-99c411ef4a56"
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|putBlock {
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockData {
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockID {
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|localID: 111677748019230008
> 2023-07-20 18:23:48,803|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockCommitSequenceId: 0
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|metadata {
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|key: "TYPE"
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|value: "KEY"
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunks {
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_1"
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 0
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "[;\033\033"
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\235mx\330"
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "~8U\335"
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "q\274\333\324"
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunks {
> 2023-07-20 18:23:48,804|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_2"
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 4194304
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "t\300\241\323"
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\233\371\310r"
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\272B\303\351"
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunks {
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_3"
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 8388608
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,805|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunks {
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_4"
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 12582912
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|eof: false
> 2023-07-20 18:23:48,806|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|encodedToken: "WQoGaHJ0X3FhEiZjb25JRDogMTEwMDEgbG9jSUQ6IDExMTY3Nzc0ODAxOTIzMDAwOBiq8rLNlzEoASgCMICAgIABOhYI8pD5uvbmvoviARC0pKf82Lq1wqUBIEGPFTvxZP_g89BUJQ_Sr7RU9mjkII0acy7aS0bMLyxuEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDExMDAxIGxvY0lEOiAxMTE2Nzc3NDgwMTkyMzAwMDg"
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|, data.size=0
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|java.util.concurrent.CompletionException: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-18BAE915ECAE->RAFT is closed.
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.fail(OrderedAsync.java:104)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.util.SlidingWindow$Client.alreadyClosed(SlidingWindow.java:406)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.util.SlidingWindow$Client.fail(SlidingWindow.java:394)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.failAllAsyncRequests(OrderedAsync.java:148)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.handleAsyncRetryFailure(OrderedAsync.java:152)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequest$10(OrderedAsync.java:262)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.completeReplyExceptionally(GrpcClientProtocolClient.java:394)
> 2023-07-20 18:23:48,807|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.access$000(GrpcClientProtocolClient.java:300)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:331)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.metrics.intercept.client.MetricClientCallListener.onClose(MetricClientCallListener.java:47)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:471)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:435)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:468)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.lang.Thread.run(Thread.java:750)
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-18BAE915ECAE->RAFT is closed.
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 28 more
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.protocol.exceptions.RaftRetryFailureException: Failed RaftClientRequest:client-18BAE915ECAE->17e7afb2-6bdb-4eb3-abf7-99c411ef4a56@group-15BC114E0F45, cid=4, seq=5*, RW, cmdType: WriteChunk
> 2023-07-20 18:23:48,808|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|traceID: ""
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|datanodeUuid: "17e7afb2-6bdb-4eb3-abf7-99c411ef4a56"
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|writeChunk {
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockID {
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|localID: 111677748019230008
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockCommitSequenceId: 0
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|replicaIndex: 0
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkData {
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_4"
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 12582912
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,809|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|encodedToken: "WQoGaHJ0X3FhEiZjb25JRDogMTEwMDEgbG9jSUQ6IDExMTY3Nzc0ODAxOTIzMDAwOBiq8rLNlzEoASgCMICAgIABOhYI8pD5uvbmvoviARC0pKf82Lq1wqUBIEGPFTvxZP_g89BUJQ_Sr7RU9mjkII0acy7aS0bMLyxuEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDExMDAxIGxvY0lEOiAxMTE2Nzc3NDgwMTkyMzAwMDg"
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|, data.size=4194304 for 31 attempts with RequestTypeDependentRetryPolicy{WRITE->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7}), WATCH->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->NoRetry})}
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.RaftClientImpl.noMoreRetries(RaftClientImpl.java:307)
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 25 more
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: java.io.IOException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:99)
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.GrpcUtil.unwrapIOException(GrpcUtil.java:159)
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:330)
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 16 more
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:539)
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 16 more
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /172.27.26.194:9858
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
> 2023-07-20 18:23:48,810|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687)
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489)
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.lang.Thread.run(Thread.java:750)
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|23/07/20 18:23:48 ERROR impl.OrderedAsync: Failed to send request, message=cmdType: WriteChunk
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|traceID: ""
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|datanodeUuid: "17e7afb2-6bdb-4eb3-abf7-99c411ef4a56"
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|writeChunk {
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockID {
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|localID: 111677748019230008
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockCommitSequenceId: 0
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|replicaIndex: 0
> 2023-07-20 18:23:48,811|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkData {
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_5"
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 16777216
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|encodedToken: "WQoGaHJ0X3FhEiZjb25JRDogMTEwMDEgbG9jSUQ6IDExMTY3Nzc0ODAxOTIzMDAwOBiq8rLNlzEoASgCMICAgIABOhYI8pD5uvbmvoviARC0pKf82Lq1wqUBIEGPFTvxZP_g89BUJQ_Sr7RU9mjkII0acy7aS0bMLyxuEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDExMDAxIGxvY0lEOiAxMTE2Nzc3NDgwMTkyMzAwMDg"
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|, data.size=4194304
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|java.util.concurrent.CompletionException: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-18BAE915ECAE->RAFT is closed.
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> 2023-07-20 18:23:48,812|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
> 2023-07-20 18:23:48,813|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> 2023-07-20 18:23:48,813|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2023-07-20 18:23:48,813|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2023-07-20 18:23:48,813|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.fail(OrderedAsync.java:104)
> 2023-07-20 18:23:48,813|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.util.SlidingWindow$Client.alreadyClosed(SlidingWindow.java:406)
> 2023-07-20 18:23:48,813|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.util.SlidingWindow$Client.fail(SlidingWindow.java:394)
> 2023-07-20 18:23:48,813|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.failAllAsyncRequests(OrderedAsync.java:148)
> 2023-07-20 18:23:48,813|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.handleAsyncRetryFailure(OrderedAsync.java:152)
> 2023-07-20 18:23:48,813|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequest$10(OrderedAsync.java:262)
> 2023-07-20 18:23:48,813|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
> 2023-07-20 18:23:48,813|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
> 2023-07-20 18:23:48,813|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2023-07-20 18:23:48,813|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.completeReplyExceptionally(GrpcClientProtocolClient.java:394)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.access$000(GrpcClientProtocolClient.java:300)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:331)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.metrics.intercept.client.MetricClientCallListener.onClose(MetricClientCallListener.java:47)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:471)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:435)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:468)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.lang.Thread.run(Thread.java:750)
> 2023-07-20 18:23:48,814|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-18BAE915ECAE->RAFT is closed.
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 28 more
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.protocol.exceptions.RaftRetryFailureException: Failed RaftClientRequest:client-18BAE915ECAE->17e7afb2-6bdb-4eb3-abf7-99c411ef4a56@group-15BC114E0F45, cid=4, seq=5*, RW, cmdType: WriteChunk
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|traceID: ""
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|datanodeUuid: "17e7afb2-6bdb-4eb3-abf7-99c411ef4a56"
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|writeChunk {
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockID {
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|localID: 111677748019230008
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockCommitSequenceId: 0
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|replicaIndex: 0
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkData {
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_4"
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 12582912
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,815|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|encodedToken: "WQoGaHJ0X3FhEiZjb25JRDogMTEwMDEgbG9jSUQ6IDExMTY3Nzc0ODAxOTIzMDAwOBiq8rLNlzEoASgCMICAgIABOhYI8pD5uvbmvoviARC0pKf82Lq1wqUBIEGPFTvxZP_g89BUJQ_Sr7RU9mjkII0acy7aS0bMLyxuEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDExMDAxIGxvY0lEOiAxMTE2Nzc3NDgwMTkyMzAwMDg"
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|, data.size=4194304 for 31 attempts with RequestTypeDependentRetryPolicy{WRITE->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7}), WATCH->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->NoRetry})}
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.RaftClientImpl.noMoreRetries(RaftClientImpl.java:307)
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 25 more
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: java.io.IOException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:99)
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.GrpcUtil.unwrapIOException(GrpcUtil.java:159)
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:330)
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 16 more
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:539)
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 16 more
> 2023-07-20 18:23:48,816|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /172.27.26.194:9858
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687)
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489)
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.lang.Thread.run(Thread.java:750)
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|23/07/20 18:23:48 ERROR impl.OrderedAsync: Failed to send request, message=cmdType: WriteChunk
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|traceID: ""
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|datanodeUuid: "17e7afb2-6bdb-4eb3-abf7-99c411ef4a56"
> 2023-07-20 18:23:48,817|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|writeChunk {
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockID {
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|localID: 111677748019230008
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockCommitSequenceId: 0
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|replicaIndex: 0
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkData {
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_6"
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 20971520
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,818|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|encodedToken: "WQoGaHJ0X3FhEiZjb25JRDogMTEwMDEgbG9jSUQ6IDExMTY3Nzc0ODAxOTIzMDAwOBiq8rLNlzEoASgCMICAgIABOhYI8pD5uvbmvoviARC0pKf82Lq1wqUBIEGPFTvxZP_g89BUJQ_Sr7RU9mjkII0acy7aS0bMLyxuEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDExMDAxIGxvY0lEOiAxMTE2Nzc3NDgwMTkyMzAwMDg"
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|, data.size=4194304
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|java.util.concurrent.CompletionException: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-18BAE915ECAE->RAFT is closed.
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.fail(OrderedAsync.java:104)
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.util.SlidingWindow$Client.alreadyClosed(SlidingWindow.java:406)
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.util.SlidingWindow$Client.fail(SlidingWindow.java:394)
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.failAllAsyncRequests(OrderedAsync.java:148)
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.handleAsyncRetryFailure(OrderedAsync.java:152)
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequest$10(OrderedAsync.java:262)
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2023-07-20 18:23:48,819|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.completeReplyExceptionally(GrpcClientProtocolClient.java:394)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.access$000(GrpcClientProtocolClient.java:300)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:331)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.metrics.intercept.client.MetricClientCallListener.onClose(MetricClientCallListener.java:47)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:471)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:435)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:468)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.lang.Thread.run(Thread.java:750)
> 2023-07-20 18:23:48,820|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-18BAE915ECAE->RAFT is closed.
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 28 more
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.protocol.exceptions.RaftRetryFailureException: Failed RaftClientRequest:client-18BAE915ECAE->17e7afb2-6bdb-4eb3-abf7-99c411ef4a56@group-15BC114E0F45, cid=4, seq=5*, RW, cmdType: WriteChunk
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|traceID: ""
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|datanodeUuid: "17e7afb2-6bdb-4eb3-abf7-99c411ef4a56"
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|writeChunk {
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockID {
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|localID: 111677748019230008
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockCommitSequenceId: 0
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|replicaIndex: 0
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkData {
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_4"
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 12582912
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,821|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|encodedToken: "WQoGaHJ0X3FhEiZjb25JRDogMTEwMDEgbG9jSUQ6IDExMTY3Nzc0ODAxOTIzMDAwOBiq8rLNlzEoASgCMICAgIABOhYI8pD5uvbmvoviARC0pKf82Lq1wqUBIEGPFTvxZP_g89BUJQ_Sr7RU9mjkII0acy7aS0bMLyxuEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDExMDAxIGxvY0lEOiAxMTE2Nzc3NDgwMTkyMzAwMDg"
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|, data.size=4194304 for 31 attempts with RequestTypeDependentRetryPolicy{WRITE->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7}), WATCH->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->NoRetry})}
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.RaftClientImpl.noMoreRetries(RaftClientImpl.java:307)
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 25 more
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: java.io.IOException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:99)
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.GrpcUtil.unwrapIOException(GrpcUtil.java:159)
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:330)
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 16 more
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:539)
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 16 more
> 2023-07-20 18:23:48,822|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /172.27.26.194:9858
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687)
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489)
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.lang.Thread.run(Thread.java:750)
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|23/07/20 18:23:48 ERROR impl.OrderedAsync: Failed to send request, message=cmdType: WriteChunk
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|traceID: ""
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|datanodeUuid: "17e7afb2-6bdb-4eb3-abf7-99c411ef4a56"
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|writeChunk {
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockID {
> 2023-07-20 18:23:48,823|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|localID: 111677748019230008
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockCommitSequenceId: 0
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|replicaIndex: 0
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkData {
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_7"
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 25165824
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|encodedToken: "WQoGaHJ0X3FhEiZjb25JRDogMTEwMDEgbG9jSUQ6IDExMTY3Nzc0ODAxOTIzMDAwOBiq8rLNlzEoASgCMICAgIABOhYI8pD5uvbmvoviARC0pKf82Lq1wqUBIEGPFTvxZP_g89BUJQ_Sr7RU9mjkII0acy7aS0bMLyxuEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDExMDAxIGxvY0lEOiAxMTE2Nzc3NDgwMTkyMzAwMDg"
> 2023-07-20 18:23:48,824|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|, data.size=4194304
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|java.util.concurrent.CompletionException: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-18BAE915ECAE->RAFT is closed.
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.fail(OrderedAsync.java:104)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.util.SlidingWindow$Client.alreadyClosed(SlidingWindow.java:406)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.util.SlidingWindow$Client.fail(SlidingWindow.java:394)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.failAllAsyncRequests(OrderedAsync.java:148)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.handleAsyncRetryFailure(OrderedAsync.java:152)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequest$10(OrderedAsync.java:262)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.completeReplyExceptionally(GrpcClientProtocolClient.java:394)
> 2023-07-20 18:23:48,825|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.access$000(GrpcClientProtocolClient.java:300)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:331)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.metrics.intercept.client.MetricClientCallListener.onClose(MetricClientCallListener.java:47)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:471)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:435)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:468)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.lang.Thread.run(Thread.java:750)
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-18BAE915ECAE->RAFT is closed.
> 2023-07-20 18:23:48,826|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 28 more
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.protocol.exceptions.RaftRetryFailureException: Failed RaftClientRequest:client-18BAE915ECAE->17e7afb2-6bdb-4eb3-abf7-99c411ef4a56@group-15BC114E0F45, cid=4, seq=5*, RW, cmdType: WriteChunk
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|traceID: ""
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|datanodeUuid: "17e7afb2-6bdb-4eb3-abf7-99c411ef4a56"
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|writeChunk {
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockID {
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|localID: 111677748019230008
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockCommitSequenceId: 0
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|replicaIndex: 0
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkData {
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_4"
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 12582912
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,827|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|encodedToken: "WQoGaHJ0X3FhEiZjb25JRDogMTEwMDEgbG9jSUQ6IDExMTY3Nzc0ODAxOTIzMDAwOBiq8rLNlzEoASgCMICAgIABOhYI8pD5uvbmvoviARC0pKf82Lq1wqUBIEGPFTvxZP_g89BUJQ_Sr7RU9mjkII0acy7aS0bMLyxuEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDExMDAxIGxvY0lEOiAxMTE2Nzc3NDgwMTkyMzAwMDg"
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|, data.size=4194304 for 31 attempts with RequestTypeDependentRetryPolicy{WRITE->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7}), WATCH->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->NoRetry})}
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.RaftClientImpl.noMoreRetries(RaftClientImpl.java:307)
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 25 more
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: java.io.IOException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:99)
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.GrpcUtil.unwrapIOException(GrpcUtil.java:159)
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:330)
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 16 more
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:539)
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 16 more
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /172.27.26.194:9858
> 2023-07-20 18:23:48,828|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687)
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489)
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.lang.Thread.run(Thread.java:750)
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|23/07/20 18:23:48 ERROR impl.OrderedAsync: Failed to send request, message=cmdType: WriteChunk
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|traceID: ""
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|datanodeUuid: "17e7afb2-6bdb-4eb3-abf7-99c411ef4a56"
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|writeChunk {
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockID {
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,829|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|localID: 111677748019230008
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockCommitSequenceId: 0
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|replicaIndex: 0
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkData {
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_8"
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 29360128
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|encodedToken: "WQoGaHJ0X3FhEiZjb25JRDogMTEwMDEgbG9jSUQ6IDExMTY3Nzc0ODAxOTIzMDAwOBiq8rLNlzEoASgCMICAgIABOhYI8pD5uvbmvoviARC0pKf82Lq1wqUBIEGPFTvxZP_g89BUJQ_Sr7RU9mjkII0acy7aS0bMLyxuEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDExMDAxIGxvY0lEOiAxMTE2Nzc3NDgwMTkyMzAwMDg"
> 2023-07-20 18:23:48,830|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|, data.size=4194304
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|java.util.concurrent.CompletionException: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-18BAE915ECAE->RAFT is closed.
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.fail(OrderedAsync.java:104)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.util.SlidingWindow$Client.alreadyClosed(SlidingWindow.java:406)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.util.SlidingWindow$Client.fail(SlidingWindow.java:394)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.failAllAsyncRequests(OrderedAsync.java:148)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.handleAsyncRetryFailure(OrderedAsync.java:152)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequest$10(OrderedAsync.java:262)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.completeReplyExceptionally(GrpcClientProtocolClient.java:394)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.access$000(GrpcClientProtocolClient.java:300)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:331)
> 2023-07-20 18:23:48,831|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.metrics.intercept.client.MetricClientCallListener.onClose(MetricClientCallListener.java:47)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:471)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:435)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:468)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.lang.Thread.run(Thread.java:750)
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-18BAE915ECAE->RAFT is closed.
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 28 more
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.protocol.exceptions.RaftRetryFailureException: Failed RaftClientRequest:client-18BAE915ECAE->17e7afb2-6bdb-4eb3-abf7-99c411ef4a56@group-15BC114E0F45, cid=4, seq=5*, RW, cmdType: WriteChunk
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|traceID: ""
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,832|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|datanodeUuid: "17e7afb2-6bdb-4eb3-abf7-99c411ef4a56"
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|writeChunk {
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockID {
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|localID: 111677748019230008
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockCommitSequenceId: 0
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|replicaIndex: 0
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkData {
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_4"
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 12582912
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,833|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|encodedToken: "WQoGaHJ0X3FhEiZjb25JRDogMTEwMDEgbG9jSUQ6IDExMTY3Nzc0ODAxOTIzMDAwOBiq8rLNlzEoASgCMICAgIABOhYI8pD5uvbmvoviARC0pKf82Lq1wqUBIEGPFTvxZP_g89BUJQ_Sr7RU9mjkII0acy7aS0bMLyxuEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDExMDAxIGxvY0lEOiAxMTE2Nzc3NDgwMTkyMzAwMDg"
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|, data.size=4194304 for 31 attempts with RequestTypeDependentRetryPolicy{WRITE->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7}), WATCH->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->NoRetry})}
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.RaftClientImpl.noMoreRetries(RaftClientImpl.java:307)
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 25 more
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: java.io.IOException: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:99)
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.GrpcUtil.unwrapIOException(GrpcUtil.java:159)
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:330)
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 16 more
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:539)
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 16 more
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /172.27.26.194:9858
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Errors.newConnectException0(Errors.java:155)
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:128)
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.unix.Socket.finishConnect(Socket.java:359)
> 2023-07-20 18:23:48,834|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:710)
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:687)
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489)
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.lang.Thread.run(Thread.java:750)
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|23/07/20 18:23:48 ERROR impl.OrderedAsync: Failed to send request, message=cmdType: PutBlock
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|traceID: ""
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|datanodeUuid: "17e7afb2-6bdb-4eb3-abf7-99c411ef4a56"
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|putBlock {
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockData {
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockID {
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|localID: 111677748019230008
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockCommitSequenceId: 0
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,835|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|metadata {
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|key: "TYPE"
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|value: "KEY"
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunks {
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_1"
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 0
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "[;\033\033"
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\235mx\330"
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "~8U\335"
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "q\274\333\324"
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunks {
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_2"
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 4194304
> 2023-07-20 18:23:48,836|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "t\300\241\323"
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\233\371\310r"
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\272B\303\351"
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunks {
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_3"
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 8388608
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,837|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunks {
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_4"
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 12582912
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunks {
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_5"
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 16777216
> 2023-07-20 18:23:48,838|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunks {
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_6"
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 20971520
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,839|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunks {
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_7"
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 25165824
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunks {
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_8"
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 29360128
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,840|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|eof: false
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|encodedToken: "WQoGaHJ0X3FhEiZjb25JRDogMTEwMDEgbG9jSUQ6IDExMTY3Nzc0ODAxOTIzMDAwOBiq8rLNlzEoASgCMICAgIABOhYI8pD5uvbmvoviARC0pKf82Lq1wqUBIEGPFTvxZP_g89BUJQ_Sr7RU9mjkII0acy7aS0bMLyxuEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDExMDAxIGxvY0lEOiAxMTE2Nzc3NDgwMTkyMzAwMDg"
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|, data.size=0
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|java.util.concurrent.CompletionException: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-18BAE915ECAE->RAFT is closed.
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2023-07-20 18:23:48,841|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.fail(OrderedAsync.java:104)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.util.SlidingWindow$Client.alreadyClosed(SlidingWindow.java:406)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.util.SlidingWindow$Client.fail(SlidingWindow.java:394)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.failAllAsyncRequests(OrderedAsync.java:148)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.handleAsyncRetryFailure(OrderedAsync.java:152)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequest$10(OrderedAsync.java:262)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.completeReplyExceptionally(GrpcClientProtocolClient.java:394)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.access$000(GrpcClientProtocolClient.java:300)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:331)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.grpc.metrics.intercept.client.MetricClientCallListener.onClose(MetricClientCallListener.java:47)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:471)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:435)
> 2023-07-20 18:23:48,842|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:468)
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563)
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744)
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at java.lang.Thread.run(Thread.java:750)
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-18BAE915ECAE->RAFT is closed.
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|... 28 more
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|Caused by: org.apache.ratis.protocol.exceptions.RaftRetryFailureException: Failed RaftClientRequest:client-18BAE915ECAE->17e7afb2-6bdb-4eb3-abf7-99c411ef4a56@group-15BC114E0F45, cid=4, seq=5*, RW, cmdType: WriteChunk
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|traceID: ""
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|datanodeUuid: "17e7afb2-6bdb-4eb3-abf7-99c411ef4a56"
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|writeChunk {
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockID {
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|containerID: 11001
> 2023-07-20 18:23:48,843|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|localID: 111677748019230008
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|blockCommitSequenceId: 0
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|replicaIndex: 0
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkData {
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|chunkName: "111677748019230008_chunk_4"
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|offset: 12582912
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|len: 4194304
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksumData {
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|type: CRC32
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|bytesPerChecksum: 1048576
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|checksums: "\2478\352\034"
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|}
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|encodedToken: "WQoGaHJ0X3FhEiZjb25JRDogMTEwMDEgbG9jSUQ6IDExMTY3Nzc0ODAxOTIzMDAwOBiq8rLNlzEoASgCMICAgIABOhYI8pD5uvbmvoviARC0pKf82Lq1wqUBIEGPFTvxZP_g89BUJQ_Sr7RU9mjkII0acy7aS0bMLyxuEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDExMDAxIGxvY0lEOiAxMTE2Nzc3NDgwMTkyMzAwMDg"
> 2023-07-20 18:23:48,844|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|, data.size=4194304 for 31 attempts with RequestTypeDependentRetryPolicy{WRITE->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7}), WATCH->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@4273b3f7, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->NoRetry})}
> 2023-07-20 18:23:48,845|INFO|Thread-89|machine.py:203 - run()||GUID=cca033d1-439d-4e7b-be9a-d686f8667be2|at org.apache {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org