You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "duanmuyh (via GitHub)" <gi...@apache.org> on 2024/01/18 05:59:09 UTC
[I] The collection rate of local files and HDFS files is very slow when doing [seatunnel]
duanmuyh opened a new issue, #6240:
URL: https://github.com/apache/seatunnel/issues/6240
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
Collecting local text files into Doris, and attempting to put the files into HDFS has a low processing speed.
### SeaTunnel Version
seatunnel 2.3.3
Doris 2.0.2
Java 1.8.0_333
### SeaTunnel Config
```conf
env {
# You can set flink configuration here
execution.parallelism = 10
job.mode = "BATCH"
}
source {
localFile {
file_format_type = "text"
path = "/home/hadoop/yangst/wudao_20240111.txt"
}
}
sink {
jdbc {
url = "jdbc:mysql://xxx.xx.xx:port/wudao?rewriteBatchedStatements=true"
driver = "com.mysql.cj.jdbc.Driver"
user = "root"
password = "xxxx"
query = "insert into wudao(line) values(?)"
}
}
```
### Running Command
```shell
nohup sh bin/seatunnel.sh -c config/wudao.config -e local > nohup.out 2>&1 &
```
### Error Exception
```log
Job Progress Information
***********************************************
Job Id : 799900012245942273
Read Count So Far : 35192
Write Count So Far : 26996
Average Read Count : 50/s
Average Write Count : 50/s
Last Statistic Time : 2024-01-17 15:35:05
Current Statistic Time : 2024-01-17 15:36:05
***********************************************
2024-01-17 15:37:02,846 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1]
***********************************************
CoordinatorService Thread Pool Status
***********************************************
activeCount : 1
corePoolSize : 0
maximumPoolSize : 2147483647
poolSize : 1
completedTaskCount : 67
taskCount : 68
***********************************************
2024-01-17 15:37:02,848 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1]
***********************************************
Job info detail
***********************************************
createdJobCount : 0
scheduledJobCount : 0
runningJobCount : 1
failingJobCount : 0
failedJobCount : 0
cancellingJobCount : 0
canceledJobCount : 0
finishedJobCount : 0
restartingJobCount : 0
suspendedJobCount : 0
reconcilingJobCount : 0
***********************************************
2024-01-17 15:37:05,614 INFO org.apache.seatunnel.engine.client.job.JobMetricsRunner -
***********************************************
Job Progress Information
***********************************************
Job Id : 799900012245942273
Read Count So Far : 37192
Write Count So Far : 28996
Average Read Count : 33/s
Average Write Count : 33/s
Last Statistic Time : 2024-01-17 15:36:05
Current Statistic Time : 2024-01-17 15:37:05
***********************************************
2024-01-17 15:38:02,845 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1]
***********************************************
CoordinatorService Thread Pool Status
***********************************************
activeCount : 1
corePoolSize : 0
maximumPoolSize : 2147483647
poolSize : 1
completedTaskCount : 67
taskCount : 68
***********************************************
2024-01-17 15:38:02,849 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1]
***********************************************
Job info detail
***********************************************
createdJobCount : 0
scheduledJobCount : 0
runningJobCount : 1
failingJobCount : 0
failedJobCount : 0
cancellingJobCount : 0
canceledJobCount : 0
finishedJobCount : 0
restartingJobCount : 0
suspendedJobCount : 0
reconcilingJobCount : 0
***********************************************
2024-01-17 15:38:05,613 INFO org.apache.seatunnel.engine.client.job.JobMetricsRunner -
***********************************************
Job Progress Information
***********************************************
Job Id : 799900012245942273
Read Count So Far : 38192
Write Count So Far : 29996
Average Read Count : 16/s
Average Write Count : 16/s
Last Statistic Time : 2024-01-17 15:37:05
Current Statistic Time : 2024-01-17 15:38:05
***********************************************
2024-01-17 15:39:02,846 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1]
***********************************************
CoordinatorService Thread Pool Status
***********************************************
activeCount : 1
corePoolSize : 0
maximumPoolSize : 2147483647
poolSize : 1
completedTaskCount : 67
taskCount : 68
***********************************************
2024-01-17 15:39:02,849 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1]
***********************************************
Job info detail
***********************************************
createdJobCount : 0
scheduledJobCount : 0
runningJobCount : 1
failingJobCount : 0
failedJobCount : 0
cancellingJobCount : 0
canceledJobCount : 0
finishedJobCount : 0
restartingJobCount : 0
suspendedJobCount : 0
reconcilingJobCount : 0
***********************************************
2024-01-17 15:39:05,613 INFO org.apache.seatunnel.engine.client.job.JobMetricsRunner -
***********************************************
Job Progress Information
***********************************************
Job Id : 799900012245942273
Read Count So Far : 41192
Write Count So Far : 32996
Average Read Count : 50/s
Average Write Count : 50/s
Last Statistic Time : 2024-01-17 15:38:05
Current Statistic Time : 2024-01-17 15:39:05
***********************************************
2024-01-17 15:39:45,040 INFO org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 4
2024-01-17 15:40:02,845 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1]
***********************************************
CoordinatorService Thread Pool Status
***********************************************
activeCount : 1
corePoolSize : 0
maximumPoolSize : 2147483647
poolSize : 4
completedTaskCount : 72
taskCount : 73
***********************************************
2024-01-17 15:40:02,847 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1]
***********************************************
Job info detail
***********************************************
createdJobCount : 0
scheduledJobCount : 0
runningJobCount : 1
failingJobCount : 0
failedJobCount : 0
cancellingJobCount : 0
canceledJobCount : 0
finishedJobCount : 0
restartingJobCount : 0
suspendedJobCount : 0
reconcilingJobCount : 0
***********************************************
2024-01-17 15:40:05,609 INFO org.apache.seatunnel.engine.client.job.JobMetricsRunner -
***********************************************
Job Progress Information
***********************************************
Job Id : 799900012245942273
Read Count So Far : 44192
Write Count So Far : 35996
Average Read Count : 50/s
Average Write Count : 50/s
Last Statistic Time : 2024-01-17 15:39:05
Current Statistic Time : 2024-01-17 15:40:05
***********************************************
```
### Zeta or Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Attempted to collect local articles and HDFS text files into Doris. The working speed is very slow whether it is collecting locally or from HDFS [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6240:
URL: https://github.com/apache/seatunnel/issues/6240#issuecomment-1992811355
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Attempted to collect local articles and HDFS text files into Doris. The working speed is very slow whether it is collecting locally or from HDFS [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6240:
URL: https://github.com/apache/seatunnel/issues/6240#issuecomment-2008392269
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Attempted to collect local articles and HDFS text files into Doris. The working speed is very slow whether it is collecting locally or from HDFS [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6240: Attempted to collect local articles and HDFS text files into Doris. The working speed is very slow whether it is collecting locally or from HDFS
URL: https://github.com/apache/seatunnel/issues/6240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org