You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "inkkim (via GitHub)" <gi...@apache.org> on 2024/01/17 02:56:17 UTC
[I] [Bug] [Zeta] Aggregate commit error [seatunnel]
inkkim opened a new issue, #6229:
URL: https://github.com/apache/seatunnel/issues/6229
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
When I try to do JDBC source(Oracle, PostgreSQL) to SFTP sink(parquet type), I got this error.
While data source and sink are in progress, we try to save checkpoints at regular intervals, but the pending and wait states are repeated. It appears that an error occurred during the final aggregation process. This happens equally in both cluster mode and local mode.
```bash
root@ubuntu-2004:/tmp/seatunnel/seatunnel/799451662937751553/09396c8f64/T_799451662937751553_09396c8f64_0_1/NON_PARTITION# ls -alh
total 1.8G
drwxrwxr-x 2 ink ink 4.0K Jan 16 10:38 .
drwxrwxr-x 3 ink ink 4.0K Jan 16 10:35 ..
-rw-rw-r-- 1 ink ink 184M Jan 16 10:39 T_799451662937751553_09396c8f64_0_1_0.parquet
-rw-rw-r-- 1 ink ink 184M Jan 16 10:39 T_799451662937751553_09396c8f64_0_1_1.parquet
-rw-rw-r-- 1 ink ink 184M Jan 16 10:39 T_799451662937751553_09396c8f64_0_1_2.parquet
-rw-rw-r-- 1 ink ink 184M Jan 16 10:39 T_799451662937751553_09396c8f64_0_1_3.parquet
-rw-rw-r-- 1 ink ink 184M Jan 16 10:39 T_799451662937751553_09396c8f64_0_1_4.parquet
-rw-rw-r-- 1 ink ink 184M Jan 16 10:39 T_799451662937751553_09396c8f64_0_1_5.parquet
-rw-rw-r-- 1 ink ink 184M Jan 16 10:39 T_799451662937751553_09396c8f64_0_1_6.parquet
-rw-rw-r-- 1 ink ink 184M Jan 16 10:40 T_799451662937751553_09396c8f64_0_1_7.parquet
-rw-rw-r-- 1 ink ink 184M Jan 16 10:40 T_799451662937751553_09396c8f64_0_1_8.parquet
-rw-rw-r-- 1 ink ink 184M Jan 16 10:40 T_799451662937751553_09396c8f64_0_1_9.parquet
```
### SeaTunnel Version
SeaTunnel 2.3.3
### SeaTunnel Config
```conf
# /usr/local/seatunnel/config/seatunnel.yaml
seatunnel:
engine:
history-job-expire-minutes: 1440
backup-count: 1
queue-type: blockingqueue
print-execution-info-interval: 60
print-job-metrics-info-interval: 60
slot-service:
dynamic-slot: true
checkpoint:
interval: 10000
timeout: 600000
storage:
type: hdfs
max-retained: 3
plugin-config:
namespace: /dev/
storage.type: hdfs
fs.defaultFS: hdfs://namenode:8020 # Ensure that the directory has written permission
# /usr/local/seatunnel/config/v2.batch.config.jdbc-oracle-sftp-parquet
env {
# You can set SeaTunnel environment configuration here
execution.parallelism = 3
job.mode = "BATCH"
checkpoint.interval = 10000
#execution.checkpoint.interval = 10000
#execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
}
source {
# This is a example source plugin **only for test and demonstrate the feature source plugin**
Jdbc {
url = "jdbc:oracle:thin:@oracle-server:19070:XE"
driver = "oracle.jdbc.OracleDriver"
user = "user"
password = "StrongP@ssword"
query = "select * from USER.TEST_ITEM"
result_table_name = "TEST_ITEM"
}
}
sink {
SftpFile {
path = "/tmp/sink/oracle2parquet"
host = "sftp-host"
port = 22
user = ink
password = "StrongP@ssword"
file_format_type = "parquet"
source_table_name = "TEST_ITEM"
}
```
### Running Command
```shell
/usr/local/seatunnel/bin/seatunnel.sh --config /usr/local/seatunnel/config/v2.batch.config.jdbc-oracle-sftp-parquet
```
### Error Exception
```log
Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Checkpoint notify complete failed
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:255)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:251)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.notifyCompleted(CheckpointCoordinator.java:328)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:736)
at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.lambda$null$7(CheckpointCoordinator.java:480)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.CompletionException: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: Aggregate commit error.
at org.apache.seatunnel.engine.server.task.SinkAggregatedCommitterTask.notifyCheckpointComplete(SinkAggregatedCommitterTask.java:300)
at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointFinishedOperation.lambda$run$0(CheckpointFinishedOperation.java:91)
at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48)
at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointFinishedOperation.run(CheckpointFinishedOperation.java:81)
at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139)
at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.biRelay(CompletableFuture.java:1300)
at java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1284)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at com.hazelcast.spi.impl.AbstractInvocationFuture.onComplete(AbstractInvocationFuture.java:1243)
at com.hazelcast.spi.impl.AbstractInvocationFuture.complete0(AbstractInvocationFuture.java:1234)
at com.hazelcast.spi.impl.AbstractInvocationFuture.completeExceptionallyInternal(AbstractInvocationFuture.java:1223)
at com.hazelcast.spi.impl.operationservice.impl.Invocation.completeExceptionally(Invocation.java:680)
at com.hazelcast.spi.impl.operationservice.impl.Invocation.notifyThrowable(Invocation.java:386)
at com.hazelcast.spi.impl.operationservice.impl.Invocation.notifyError(Invocation.java:330)
at com.hazelcast.spi.impl.operationservice.impl.Invocation.sendResponse(Invocation.java:230)
at com.hazelcast.spi.impl.operationservice.Operation.sendResponse(Operation.java:483)
```
### Zeta or Flink or Spark Version
Zeta Cluster Mode (2 instances)
### Java or Scala Version
OpenJDK Runtime Environment (build 1.8.0_392-8u392-ga-1~20.04-b08)
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Zeta] Aggregate commit error [seatunnel]
Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 commented on issue #6229:
URL: https://github.com/apache/seatunnel/issues/6229#issuecomment-1899906575
From the job config, I found you don`t config the `partition_column` and `partition_num`. So the source will read all data as a single split. If the split can not read complete in `600000` ms, The checkpoint will timeout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Zeta] Aggregate commit error [seatunnel]
Posted by "inkkim (via GitHub)" <gi...@apache.org>.
inkkim commented on issue #6229:
URL: https://github.com/apache/seatunnel/issues/6229#issuecomment-1905348367
@EricJoy2048
Thanks for replying,
I tried configuring the JDBC source as you said, but it didn't work.
However, when I looked up your reply to another issue in the past, I saw an article saying that tmp_path needs to be set in the SFTP sink settings, so I set it up and it worked out well.
#5797
Thank you again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Zeta] Aggregate commit error [seatunnel]
Posted by "EricJoy2048 (via GitHub)" <gi...@apache.org>.
EricJoy2048 commented on issue #6229:
URL: https://github.com/apache/seatunnel/issues/6229#issuecomment-1899904649
I can't tell the reason of agg commiter failure from the logs. Are there any more logs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Zeta] Aggregate commit error [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6229:
URL: https://github.com/apache/seatunnel/issues/6229#issuecomment-1960556979
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Zeta] Aggregate commit error [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6229: [Bug] [Zeta] Aggregate commit error
URL: https://github.com/apache/seatunnel/issues/6229
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Zeta] Aggregate commit error [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6229:
URL: https://github.com/apache/seatunnel/issues/6229#issuecomment-1996176278
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org