You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "zhengdaoTech (via GitHub)" <gi...@apache.org> on 2023/05/16 03:46:14 UTC
[GitHub] [incubator-seatunnel] zhengdaoTech opened a new issue, #4760: [Bug] [checkpoint] org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - store checkpoint states failed.
zhengdaoTech opened a new issue, #4760:
URL: https://github.com/apache/incubator-seatunnel/issues/4760
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
use fakesource to mysql, set 1000 rows, finally mysql get 3000 rows
and run info report checkpoing error
### SeaTunnel Version
2.3.1
### SeaTunnel Config
```conf
env{
execution.parallelism = 1
job.mode = "BATCH"
}
source {
FakeSource {
result_table_name = "fake"
row.num = 1000
schema{
fields{
name = string
age = int
sex = boolean
}
}
}
}
sink{
jdbc {
url = "jdbc:mysql://localhost:3306/test"
driver = "com.mysql.cj.jdbc.Driver"
user = "root"
password = "123456."
query = "insert into fake_data(name, age, sex) values (?,?,?)"
}
}
```
### Running Command
```shell
./bin/seatunnel.sh --config ./config/fake_to_mysql.conf -e local
```
### Error Exception
```log
2023-05-16 11:32:02,661 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 9223372036854775807
2023-05-16 11:32:13,866 ERROR org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - store checkpoint states failed.
org.apache.seatunnel.engine.checkpoint.storage.exception.CheckpointStorageException: Failed to write checkpoint data, state: PipelineState(jobId=710695952876306433, pipelineId=1, checkpointId=9223372036854775807, states=[8, -127, -128, -128, -76, -62, -58, -71, -18, 9, 16, 1, 24, -1, -1, -1, -1, -1, -1, -1, -1, 127, 32, -64, -37, -11, -108, -126, 49, 40, 2, 48, -25, -78, -10, -108, -126, 49, 59, 11, 11, 10, 58, 65, 99, 116, 105, 111, 110, 83, 116, 97, 116, 101, 75, 101, 121, 32, 45, 32, 112, 105, 112, 101, 108, 105, 110, 101, 45, 49, 32, 91, 83, 111, 117, 114, 99, 101, 91, 48, 93, 45, 70, 97, 107, 101, 83, 111, 117, 114, 99, 101, 45, 100, 101, 102, 97, 117, 108, 116, 93, 12, 19, 11, 10, 58, 65, 99, 116, 105, 111, 110, 83, 116, 97, 116, 101, 75, 101, 121, 32, 45, 32, 112, 105, 112, 101, 108, 105, 110, 101, 45, 49, 32, 91, 83, 111, 117, 114, 99, 101, 91, 48, 93, 45, 70, 97, 107, 101, 83, 111, 117, 114, 99, 101, 45, 100, 101, 102, 97, 117, 108, 116, 93, 12, 19, 11, 11, 10, 58, 65, 99
, 116, 105, 111, 110, 83, 116, 97, 116, 101, 75, 101, 121, 32, 45, 32, 112, 105, 112, 101, 108, 105, 110, 101, 45, 49, 32, 91, 83, 111, 117, 114, 99, 101, 91, 48, 93, 45, 70, 97, 107, 101, 83, 111, 117, 114, 99, 101, 45, 100, 101, 102, 97, 117, 108, 116, 93, 12, 16, 0, 27, 28, 12, 20, 27, 11, 10, 58, 65, 99, 116, 105, 111, 110, 83, 116, 97, 116, 101, 75, 101, 121, 32, 45, 32, 112, 105, 112, 101, 108, 105, 110, 101, 45, 49, 32, 91, 83, 111, 117, 114, 99, 101, 91, 48, 93, 45, 70, 97, 107, 101, 83, 111, 117, 114, 99, 101, 45, 100, 101, 102, 97, 117, 108, 116, 93, 12, 16, -1, -1, -1, -1, -1, -1, -1, -1, -1, 1, 27, 10, -98, 2, -84, -19, 0, 5, 115, 114, 0, 68, 111, 114, 103, 46, 97, 112, 97, 99, 104, 101, 46, 115, 101, 97, 116, 117, 110, 110, 101, 108, 46, 99, 111, 110, 110, 101, 99, 116, 111, 114, 115, 46, 115, 101, 97, 116, 117, 110, 110, 101, 108, 46, 102, 97, 107, 101, 46, 115, 116, 97, 116, 101, 46, 70, 97, 107, 101, 83, 111, 117, 114, 99, 101, 83, 116, 97, 116, 101, 48, -46, -117, 5
6, 40, 48, 23, 101, 2, 0, 1, 76, 0, 14, 97, 115, 115, 105, 103, 110, 101, 100, 83, 112, 108, 105, 116, 115, 116, 0, 15, 76, 106, 97, 118, 97, 47, 117, 116, 105, 108, 47, 83, 101, 116, 59, 120, 112, 115, 114, 0, 17, 106, 97, 118, 97, 46, 117, 116, 105, 108, 46, 72, 97, 115, 104, 83, 101, 116, -70, 68, -123, -107, -106, -72, -73, 52, 3, 0, 0, 120, 112, 119, 12, 0, 0, 0, 16, 63, 64, 0, 0, 0, 0, 0, 1, 115, 114, 0, 69, 111, 114, 103, 46, 97, 112, 97, 99, 104, 101, 46, 115, 101, 97, 116, 117, 110, 110, 101, 108, 46, 99, 111, 110, 110, 101, 99, 116, 111, 114, 115, 46, 115, 101, 97, 116, 117, 110, 110, 101, 108, 46, 102, 97, 107, 101, 46, 115, 111, 117, 114, 99, 101, 46, 70, 97, 107, 101, 83, 111, 117, 114, 99, 101, 83, 112, 108, 105, 116, -59, 53, -37, -55, 84, 83, 120, -86, 2, 0, 2, 73, 0, 6, 114, 111, 119, 78, 117, 109, 73, 0, 7, 115, 112, 108, 105, 116, 73, 100, 120, 112, 0, 0, 3, -24, 0, 0, 0, 0, 120, 28, 28, 32, 1, 20, 12, 11, 11, 10, 50, 65, 99, 116, 105, 111, 110, 83, 116, 97, 116,
101, 75, 101, 121, 32, 45, 32, 112, 105, 112, 101, 108, 105, 110, 101, 45, 49, 32, 91, 83, 105, 110, 107, 91, 48, 93, 45, 74, 100, 98, 99, 45, 100, 101, 102, 97, 117, 108, 116, 93, 12, 19, 11, 10, 50, 65, 99, 116, 105, 111, 110, 83, 116, 97, 116, 101, 75, 101, 121, 32, 45, 32, 112, 105, 112, 101, 108, 105, 110, 101, 45, 49, 32, 91, 83, 105, 110, 107, 91, 48, 93, 45, 74, 100, 98, 99, 45, 100, 101, 102, 97, 117, 108, 116, 93, 12, 19, 11, 11, 10, 50, 65, 99, 116, 105, 111, 110, 83, 116, 97, 116, 101, 75, 101, 121, 32, 45, 32, 112, 105, 112, 101, 108, 105, 110, 101, 45, 49, 32, 91, 83, 105, 110, 107, 91, 48, 93, 45, 74, 100, 98, 99, 45, 100, 101, 102, 97, 117, 108, 116, 93, 12, 16, 0, 27, 28, 12, 20, 32, 1, 20, 12, 60, 67, 11, 8, 2, 19, 8, 2, 19, 11, 8, 0, 16, -127, -36, -11, -108, -126, 49, 24, 1, 32, 0, 12, 20, 27, 8, 1, 16, 0, 28, 32, 1, 20, 12, 11, 8, 4, 19, 8, 4, 19, 11, 8, 0, 16, -1, -37, -11, -108, -126, 49, 24, 0, 32, 0, 12, 20, 27, 8, 1, 16, 0, 28, 32, 1, 20, 12, 11, 8, 5, 19,
8, 5, 19, 11, 8, 0, 16, -26, -78, -10, -108, -126, 49, 24, 0, 32, 0, 12, 20, 27, 8, 1, 16, 0, 28, 32, 1, 20, 12, 68])
at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage.storeCheckPoint(HdfsStorage.java:110) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:611) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$7(CheckpointCoordinator.java:388) ~[seatunnel-starter.jar:2.3.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
at org.apache.seatunnel.engine.common.utils.PassiveCompletableFuture.internalComplete(PassiveCompletableFuture.java:70) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.common.utils.PassiveCompletableFuture.lambda$new$0(PassiveCompletableFuture.java:33) ~[seatunnel-starter.jar:2.3.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
at org.apache.seatunnel.engine.server.checkpoint.PendingCheckpoint.acknowledgeTask(PendingCheckpoint.java:147) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.acknowledgeTask(CheckpointCoordinator.java:581) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.acknowledgeTask(CheckpointManager.java:261) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.checkpoint.operation.TaskAcknowledgeOperation.run(TaskAcknowledgeOperation.java:81) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.run(OperationExecutorImpl.java:411) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.runOrExecute(OperationExecutorImpl.java:438) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeLocal(Invocation.java:601) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:580) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:541) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:241) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:61) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.utils.NodeEngineUtil.sendOperationToMasterNode(NodeEngineUtil.java:41) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.execution.TaskExecutionContext.sendToMaster(TaskExecutionContext.java:43) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.ack(SeaTunnelTask.java:335) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:196) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:59) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:71) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:51) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:52) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:73) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:161) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:78) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:526) [seatunnel-starter.jar:2.3.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.io.IOException: Mkdirs failed to create /tmp/seatunnel/checkpoint_snapshot/710695952876306433 (exists=false, cwd=file:/opt/seatunnel/apache-seatunnel-incubating-2.3.1)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:458) ~[seatunnel-hadoop3-3.1.4-uber-2.3.1-optional.jar:2.3.1]
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443) ~[seatunnel-hadoop3-3.1.4-uber-2.3.1-optional.jar:2.3.1]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1118) ~[seatunnel-hadoop3-3.1.4-uber-2.3.1-optional.jar:2.3.1]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1098) ~[seatunnel-hadoop3-3.1.4-uber-2.3.1-optional.jar:2.3.1]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:987) ~[seatunnel-hadoop3-3.1.4-uber-2.3.1-optional.jar:2.3.1]
at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage.storeCheckPoint(HdfsStorage.java:107) ~[seatunnel-starter.jar:2.3.1]
... 46 more
2023-05-16 11:32:13,881 ERROR org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - complete checkpoint failed
org.apache.seatunnel.engine.checkpoint.storage.exception.CheckpointStorageException: Failed to write checkpoint data, state: PipelineState(jobId=710695952876306433, pipelineId=1, checkpointId=9223372036854775807, states=[8, -127, -128, -128, -76, -62, -58, -71, -18, 9, 16, 1, 24, -1, -1, -1, -1, -1, -1, -1, -1, 127, 32, -64, -37, -11, -108, -126, 49, 40, 2, 48, -25, -78, -10, -108, -126, 49, 59, 11, 11, 10, 58, 65, 99, 116, 105, 111, 110, 83, 116, 97, 116, 101, 75, 101, 121, 32, 45, 32, 112, 105, 112, 101, 108, 105, 110, 101, 45, 49, 32, 91, 83, 111, 117, 114, 99, 101, 91, 48, 93, 45, 70, 97, 107, 101, 83, 111, 117, 114, 99, 101, 45, 100, 101, 102, 97, 117, 108, 116, 93, 12, 19, 11, 10, 58, 65, 99, 116, 105, 111, 110, 83, 116, 97, 116, 101, 75, 101, 121, 32, 45, 32, 112, 105, 112, 101, 108, 105, 110, 101, 45, 49, 32, 91, 83, 111, 117, 114, 99, 101, 91, 48, 93, 45, 70, 97, 107, 101, 83, 111, 117, 114, 99, 101, 45, 100, 101, 102, 97, 117, 108, 116, 93, 12, 19, 11, 11, 10, 58, 65, 99
, 116, 105, 111, 110, 83, 116, 97, 116, 101, 75, 101, 121, 32, 45, 32, 112, 105, 112, 101, 108, 105, 110, 101, 45, 49, 32, 91, 83, 111, 117, 114, 99, 101, 91, 48, 93, 45, 70, 97, 107, 101, 83, 111, 117, 114, 99, 101, 45, 100, 101, 102, 97, 117, 108, 116, 93, 12, 16, 0, 27, 28, 12, 20, 27, 11, 10, 58, 65, 99, 116, 105, 111, 110, 83, 116, 97, 116, 101, 75, 101, 121, 32, 45, 32, 112, 105, 112, 101, 108, 105, 110, 101, 45, 49, 32, 91, 83, 111, 117, 114, 99, 101, 91, 48, 93, 45, 70, 97, 107, 101, 83, 111, 117, 114, 99, 101, 45, 100, 101, 102, 97, 117, 108, 116, 93, 12, 16, -1, -1, -1, -1, -1, -1, -1, -1, -1, 1, 27, 10, -98, 2, -84, -19, 0, 5, 115, 114, 0, 68, 111, 114, 103, 46, 97, 112, 97, 99, 104, 101, 46, 115, 101, 97, 116, 117, 110, 110, 101, 108, 46, 99, 111, 110, 110, 101, 99, 116, 111, 114, 115, 46, 115, 101, 97, 116, 117, 110, 110, 101, 108, 46, 102, 97, 107, 101, 46, 115, 116, 97, 116, 101, 46, 70, 97, 107, 101, 83, 111, 117, 114, 99, 101, 83, 116, 97, 116, 101, 48, -46, -117, 5
6, 40, 48, 23, 101, 2, 0, 1, 76, 0, 14, 97, 115, 115, 105, 103, 110, 101, 100, 83, 112, 108, 105, 116, 115, 116, 0, 15, 76, 106, 97, 118, 97, 47, 117, 116, 105, 108, 47, 83, 101, 116, 59, 120, 112, 115, 114, 0, 17, 106, 97, 118, 97, 46, 117, 116, 105, 108, 46, 72, 97, 115, 104, 83, 101, 116, -70, 68, -123, -107, -106, -72, -73, 52, 3, 0, 0, 120, 112, 119, 12, 0, 0, 0, 16, 63, 64, 0, 0, 0, 0, 0, 1, 115, 114, 0, 69, 111, 114, 103, 46, 97, 112, 97, 99, 104, 101, 46, 115, 101, 97, 116, 117, 110, 110, 101, 108, 46, 99, 111, 110, 110, 101, 99, 116, 111, 114, 115, 46, 115, 101, 97, 116, 117, 110, 110, 101, 108, 46, 102, 97, 107, 101, 46, 115, 111, 117, 114, 99, 101, 46, 70, 97, 107, 101, 83, 111, 117, 114, 99, 101, 83, 112, 108, 105, 116, -59, 53, -37, -55, 84, 83, 120, -86, 2, 0, 2, 73, 0, 6, 114, 111, 119, 78, 117, 109, 73, 0, 7, 115, 112, 108, 105, 116, 73, 100, 120, 112, 0, 0, 3, -24, 0, 0, 0, 0, 120, 28, 28, 32, 1, 20, 12, 11, 11, 10, 50, 65, 99, 116, 105, 111, 110, 83, 116, 97, 116,
101, 75, 101, 121, 32, 45, 32, 112, 105, 112, 101, 108, 105, 110, 101, 45, 49, 32, 91, 83, 105, 110, 107, 91, 48, 93, 45, 74, 100, 98, 99, 45, 100, 101, 102, 97, 117, 108, 116, 93, 12, 19, 11, 10, 50, 65, 99, 116, 105, 111, 110, 83, 116, 97, 116, 101, 75, 101, 121, 32, 45, 32, 112, 105, 112, 101, 108, 105, 110, 101, 45, 49, 32, 91, 83, 105, 110, 107, 91, 48, 93, 45, 74, 100, 98, 99, 45, 100, 101, 102, 97, 117, 108, 116, 93, 12, 19, 11, 11, 10, 50, 65, 99, 116, 105, 111, 110, 83, 116, 97, 116, 101, 75, 101, 121, 32, 45, 32, 112, 105, 112, 101, 108, 105, 110, 101, 45, 49, 32, 91, 83, 105, 110, 107, 91, 48, 93, 45, 74, 100, 98, 99, 45, 100, 101, 102, 97, 117, 108, 116, 93, 12, 16, 0, 27, 28, 12, 20, 32, 1, 20, 12, 60, 67, 11, 8, 2, 19, 8, 2, 19, 11, 8, 0, 16, -127, -36, -11, -108, -126, 49, 24, 1, 32, 0, 12, 20, 27, 8, 1, 16, 0, 28, 32, 1, 20, 12, 11, 8, 4, 19, 8, 4, 19, 11, 8, 0, 16, -1, -37, -11, -108, -126, 49, 24, 0, 32, 0, 12, 20, 27, 8, 1, 16, 0, 28, 32, 1, 20, 12, 11, 8, 5, 19,
8, 5, 19, 11, 8, 0, 16, -26, -78, -10, -108, -126, 49, 24, 0, 32, 0, 12, 20, 27, 8, 1, 16, 0, 28, 32, 1, 20, 12, 68])
at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage.storeCheckPoint(HdfsStorage.java:110) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:611) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$7(CheckpointCoordinator.java:388) ~[seatunnel-starter.jar:2.3.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
at org.apache.seatunnel.engine.common.utils.PassiveCompletableFuture.internalComplete(PassiveCompletableFuture.java:70) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.common.utils.PassiveCompletableFuture.lambda$new$0(PassiveCompletableFuture.java:33) ~[seatunnel-starter.jar:2.3.1]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
at org.apache.seatunnel.engine.server.checkpoint.PendingCheckpoint.acknowledgeTask(PendingCheckpoint.java:147) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.acknowledgeTask(CheckpointCoordinator.java:581) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.acknowledgeTask(CheckpointManager.java:261) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.checkpoint.operation.TaskAcknowledgeOperation.run(TaskAcknowledgeOperation.java:81) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.run(OperationExecutorImpl.java:411) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.runOrExecute(OperationExecutorImpl.java:438) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeLocal(Invocation.java:601) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:580) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:541) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:241) ~[seatunnel-starter.jar:2.3.1]
at com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:61) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.utils.NodeEngineUtil.sendOperationToMasterNode(NodeEngineUtil.java:41) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.execution.TaskExecutionContext.sendToMaster(TaskExecutionContext.java:43) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.ack(SeaTunnelTask.java:335) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:196) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:59) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27) ~[seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:71) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:51) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:52) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:73) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:161) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:78) [seatunnel-starter.jar:2.3.1]
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:526) [seatunnel-starter.jar:2.3.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.io.IOException: Mkdirs failed to create /tmp/seatunnel/checkpoint_snapshot/710695952876306433 (exists=false, cwd=file:/opt/seatunnel/apache-seatunnel-incubating-2.3.1)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:458) ~[seatunnel-hadoop3-3.1.4-uber-2.3.1-optional.jar:2.3.1]
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443) ~[seatunnel-hadoop3-3.1.4-uber-2.3.1-optional.jar:2.3.1]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1118) ~[seatunnel-hadoop3-3.1.4-uber-2.3.1-optional.jar:2.3.1]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1098) ~[seatunnel-hadoop3-3.1.4-uber-2.3.1-optional.jar:2.3.1]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:987) ~[seatunnel-hadoop3-3.1.4-uber-2.3.1-optional.jar:2.3.1]
at org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage.storeCheckPoint(HdfsStorage.java:107) ~[seatunnel-starter.jar:2.3.1]
... 46 more
2023-05-16 11:32:13,883 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - start clean pending checkpoint cause CheckpointCoordinator inside have error.
2023-05-16 11:32:13,888 WARN org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (710695952876306433), Pipeline: [(1/1)] checkpoint have error, cancel the pipeline
```
### Flink or Spark Version
non
### Java or Scala Version
java -version
openjdk version "11.0.18" 2023-01-17
OpenJDK Runtime Environment (build 11.0.18+10-post-Ubuntu-0ubuntu122.04)
OpenJDK 64-Bit Server VM (build 11.0.18+10-post-Ubuntu-0ubuntu122.04, mixed mode, sharing)
### Screenshots
non
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4760: [Bug] [checkpoint] org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - store checkpoint states failed.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4760:
URL: https://github.com/apache/seatunnel/issues/4760#issuecomment-1593881066
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] closed issue #4760: [Bug] [checkpoint] org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - store checkpoint states failed.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #4760: [Bug] [checkpoint] org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - store checkpoint states failed.
URL: https://github.com/apache/seatunnel/issues/4760
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] liugddx commented on issue #4760: [Bug] [checkpoint] org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - store checkpoint states failed.
Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on issue #4760:
URL: https://github.com/apache/incubator-seatunnel/issues/4760#issuecomment-1549782702
You need to check if you have write access to the folder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #4760: [Bug] [checkpoint] org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - store checkpoint states failed.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #4760:
URL: https://github.com/apache/seatunnel/issues/4760#issuecomment-1603473064
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [checkpoint] org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - store checkpoint states failed. [seatunnel]
Posted by "JohnYehyo (via GitHub)" <gi...@apache.org>.
JohnYehyo commented on issue #4760:
URL: https://github.com/apache/seatunnel/issues/4760#issuecomment-1842220635
Check your tenants and then change the user of /tmp/seatunnel/checkpoint_snapshot to the corresponding tenant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org