You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "WarsenLiu (via GitHub)" <gi...@apache.org> on 2024/07/04 07:06:54 UTC
[I] [Bug] [checkpoint-storage] Checkpoints are only write on the first node and sometimes it may timeout. [seatunnel]
WarsenLiu opened a new issue, #7106:
URL: https://github.com/apache/seatunnel/issues/7106
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Build SeaTunnel Engine using three 8u32g servers, but sometimes there may be checkpoint timeout and checkpoints always write to the first node, resulting in too many inode.
### SeaTunnel Version
2.3.5
### SeaTunnel Config
```conf
seatunnel:
engine:
classloader-cache-mode: true
history-job-expire-minutes: 180
backup-count: 1
queue-type: blockingqueue
print-execution-info-interval: 60
print-job-metrics-info-interval: 60
slot-service:
dynamic-slot: true
checkpoint:
interval: 300000
timeout: 600000
storage:
type: hdfs
max-retained: 3
plugin-config:
namespace: /data/apache-seatunnel-2.3.5/checkpoint
# namespace: /tmp/seatunnel/checkpoint_snapshot
storage.type: hdfs
fs.defaultFS: file:///data/apache-seatunnel-2.3.5/
```
### Running Command
```shell
used ds:
env {
parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 60000
job.name = "z003"
}
source {
MySQL-CDC {
base-url = "jdbc:mysql://xxx:3306/xxx?autoReconnect=true"
username = "root"
password = "xxx"
table-names = ["xxx.xxx"]
startup.mode = "initial"
result_table_name = "source_table_2"
query = "select xxx from xxx"
}
}
transform {
Sql {
source_table_name = "source_table_2"
result_table_name = "target_table_2"
query = "select xxx from source_table_2"
}
Sql {
source_table_name = "target_table_2"
result_table_name = "target_table_log_2"
query = "select xxx from target_table_2"
}
}
sink {
Jdbc {
url = "jdbc:mysql://xxx:3306/xxx?autoReconnect=true"
driver= "com.mysql.cj.jdbc.Driver"
user = "root"
password = "xxx"
database = "xxx"
source_table_name = "target_table_2"
generate_sink_sql = true
table = "xxx"
batch_size = 10
primary_keys = ["xxx"]
}
Jdbc {
url = "jdbc:mysql://xxx:3306/xxx?autoReconnect=true"
driver= "com.mysql.cj.jdbc.Driver"
user = "root"
password = "xxx"
database = "xxx"
source_table_name = "target_table_log_2"
batch_size = 10
query = "insert into xxx(xxx) values(?) ON DUPLICATE KEY UPDATE field= VALUES(field);"
}
}
```
### Error Exception
```log
[INFO] 2024-07-04 13:37:16.704 +0800 - ->
2024-07-04 13:37:15,926 INFO org.apache.seatunnel.engine.client.job.ClientJobProxy - Job (861093691492663301) end with state FAILED
2024-07-04 13:37:15,927 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2024-07-04 13:37:15,936 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [10.60.162.14]:5801:aed286d6-1625-4ff4-91b6-34028db07da3, connection: ClientConnection{alive=false, connectionId=2, channel=NioChannel{/10.60.162.35:52531->/10.60.162.14:5801}, remoteAddress=[10.60.162.14]:5801, lastReadTime=2024-07-04 13:37:08.482, lastWriteTime=2024-07-04 13:37:08.481, closedTime=2024-07-04 13:37:15.932, connected server version=5.1}
2024-07-04 13:37:15,940 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [10.60.162.16]:5801:a3bbbfda-b0bc-4738-a148-a17337bdb588, connection: ClientConnection{alive=false, connectionId=3, channel=NioChannel{/10.60.162.35:47319->/10.60.162.16:5801}, remoteAddress=[10.60.162.16]:5801, lastReadTime=2024-07-04 13:37:13.484, lastWriteTime=2024-07-04 13:37:13.482, closedTime=2024-07-04 13:37:15.937, connected server version=5.1}
2024-07-04 13:37:15,942 INFO com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel] [5.1] Removed connection to endpoint: [10.60.162.31]:5801:b9abbcd8-ac93-41d8-9d85-25664ce23716, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/10.60.162.35:49005->/10.60.162.31:5801}, remoteAddress=[10.60.162.31]:5801, lastReadTime=2024-07-04 13:37:15.906, lastWriteTime=2024-07-04 13:37:13.328, closedTime=2024-07-04 13:37:15.940, connected server version=5.1}
2024-07-04 13:37:15,942 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2024-07-04 13:37:15,946 INFO com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2024-07-04 13:37:15,946 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed SeaTunnel client......
2024-07-04 13:37:15,946 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed metrics executor service ......
2024-07-04 13:37:15,946 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
===============================================================================
2024-07-04 13:37:15,947 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Fatal Error,
2024-07-04 13:37:15,947 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Please submit bug report in https://github.com/apache/seatunnel/issues
2024-07-04 13:37:15,947 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Reason:SeaTunnel job executed failed
2024-07-04 13:37:15,949 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Checkpoint expired before completing. Please increase checkpoint timeout in the seatunnel.yaml or jobConfig env.
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:274)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:590)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:194)
... 2 more
2024-07-04 13:37:15,949 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
===============================================================================
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Checkpoint expired before completing. Please increase checkpoint timeout in the seatunnel.yaml or jobConfig env.
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:274)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:590)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:194)
... 2 more
2024-07-04 13:37:15,951 INFO org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - run shutdown hook because get close signal
[INFO] 2024-07-04 13:37:16.706 +0800 - process has exited. execute path:/data/hubs/dolphinscheduler/tmp/exec/process/default/13823066283424/13831054200608_19/164485/311103, processId:560021 ,exitStatusCode:1 ,processWaitForStatus:true ,processExitValue:1
[INFO] 2024-07-04 13:37:16.707 +0800 - ***********************************************************************************************
[INFO] 2024-07-04 13:37:16.707 +0800 - ********************************* Finalize task instance ************************************
[INFO] 2024-07-04 13:37:16.707 +0800 - ***********************************************************************************************
[INFO] 2024-07-04 13:37:16.707 +0800 - Upload output files: [] successfully
[INFO] 2024-07-04 13:37:16.707 +0800 - Send task execute status: FAILURE to master : 10.60.162.35:1234
[INFO] 2024-07-04 13:37:16.708 +0800 - Remove the current task execute context from worker cache
[INFO] 2024-07-04 13:37:16.708 +0800 - The current execute mode isn't develop mode, will clear the task execute file: /data/hubs/dolphinscheduler/tmp/exec/process/default/13823066283424/13831054200608_19/164485/311103
[INFO] 2024-07-04 13:37:16.708 +0800 - Success clear the task execute file: /data/hubs/dolphinscheduler/tmp/exec/process/default/13823066283424/13831054200608_19/164485/311103
[INFO] 2024-07-04 13:37:16.708 +0800 - FINALIZE_SESSION
```
### Zeta or Flink or Spark Version
Zeta
### Java or Scala Version
java version "1.8.0_401"
Java(TM) SE Runtime Environment (build 1.8.0_401-b10)
Java HotSpot(TM) 64-Bit Server VM (build 25.401-b10, mixed mode)
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [checkpoint-storage] Checkpoints are only write on the first node and sometimes it may timeout. [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7106:
URL: https://github.com/apache/seatunnel/issues/7106#issuecomment-2267215465
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [checkpoint-storage] Checkpoints are only write on the first node and sometimes it may timeout. [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #7106: [Bug] [checkpoint-storage] Checkpoints are only write on the first node and sometimes it may timeout.
URL: https://github.com/apache/seatunnel/issues/7106
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [checkpoint-storage] Checkpoints are only write on the first node and sometimes it may timeout. [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #7106:
URL: https://github.com/apache/seatunnel/issues/7106#issuecomment-3064433394
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org