You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "duanmuyh (via GitHub)" <gi...@apache.org> on 2024/01/18 05:59:09 UTC

[I] The collection rate of local files and HDFS files is very slow when doing [seatunnel]

duanmuyh opened a new issue, #6240:
URL: https://github.com/apache/seatunnel/issues/6240

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   Collecting local text files into Doris, and attempting to put the files into HDFS has a low processing speed.
   
   ### SeaTunnel Version
   
   seatunnel 2.3.3
   Doris 2.0.2
   Java 1.8.0_333
   
   ### SeaTunnel Config
   
   ```conf
   env {
     # You can set flink configuration here
     execution.parallelism = 10
     job.mode = "BATCH"
   }
   
   source {
     localFile {
     file_format_type = "text"
     path = "/home/hadoop/yangst/wudao_20240111.txt"
     }
   }
   
   sink {
     jdbc {
         url = "jdbc:mysql://xxx.xx.xx:port/wudao?rewriteBatchedStatements=true"
         driver = "com.mysql.cj.jdbc.Driver"
         user = "root"
         password = "xxxx"
         query = "insert into wudao(line) values(?)"
     }
   }
   ```
   
   
   ### Running Command
   
   ```shell
   nohup sh bin/seatunnel.sh -c config/wudao.config -e local > nohup.out 2>&1 &
   ```
   
   
   ### Error Exception
   
   ```log
   Job Progress Information
   ***********************************************
   Job Id                    :  799900012245942273
   Read Count So Far         :               35192
   Write Count So Far        :               26996
   Average Read Count        :                50/s
   Average Write Count       :                50/s
   Last Statistic Time       : 2024-01-17 15:35:05
   Current Statistic Time    : 2024-01-17 15:36:05
   ***********************************************
   
   2024-01-17 15:37:02,846 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] 
   ***********************************************
        CoordinatorService Thread Pool Status
   ***********************************************
   activeCount               :                   1
   corePoolSize              :                   0
   maximumPoolSize           :          2147483647
   poolSize                  :                   1
   completedTaskCount        :                  67
   taskCount                 :                  68
   ***********************************************
   
   2024-01-17 15:37:02,848 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] 
   ***********************************************
                   Job info detail
   ***********************************************
   createdJobCount           :                   0
   scheduledJobCount         :                   0
   runningJobCount           :                   1
   failingJobCount           :                   0
   failedJobCount            :                   0
   cancellingJobCount        :                   0
   canceledJobCount          :                   0
   finishedJobCount          :                   0
   restartingJobCount        :                   0
   suspendedJobCount         :                   0
   reconcilingJobCount       :                   0
   ***********************************************
   
   2024-01-17 15:37:05,614 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
   ***********************************************
              Job Progress Information
   ***********************************************
   Job Id                    :  799900012245942273
   Read Count So Far         :               37192
   Write Count So Far        :               28996
   Average Read Count        :                33/s
   Average Write Count       :                33/s
   Last Statistic Time       : 2024-01-17 15:36:05
   Current Statistic Time    : 2024-01-17 15:37:05
   ***********************************************
   
   2024-01-17 15:38:02,845 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] 
   ***********************************************
        CoordinatorService Thread Pool Status
   ***********************************************
   activeCount               :                   1
   corePoolSize              :                   0
   maximumPoolSize           :          2147483647
   poolSize                  :                   1
   completedTaskCount        :                  67
   taskCount                 :                  68
   ***********************************************
   
   2024-01-17 15:38:02,849 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] 
   ***********************************************
                   Job info detail
   ***********************************************
   createdJobCount           :                   0
   scheduledJobCount         :                   0
   runningJobCount           :                   1
   failingJobCount           :                   0
   failedJobCount            :                   0
   cancellingJobCount        :                   0
   canceledJobCount          :                   0
   finishedJobCount          :                   0
   restartingJobCount        :                   0
   suspendedJobCount         :                   0
   reconcilingJobCount       :                   0
   ***********************************************
   
   2024-01-17 15:38:05,613 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
   ***********************************************
              Job Progress Information
   ***********************************************
   Job Id                    :  799900012245942273
   Read Count So Far         :               38192
   Write Count So Far        :               29996
   Average Read Count        :                16/s
   Average Write Count       :                16/s
   Last Statistic Time       : 2024-01-17 15:37:05
   Current Statistic Time    : 2024-01-17 15:38:05
   ***********************************************
   
   2024-01-17 15:39:02,846 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] 
   ***********************************************
        CoordinatorService Thread Pool Status
   ***********************************************
   activeCount               :                   1
   corePoolSize              :                   0
   maximumPoolSize           :          2147483647
   poolSize                  :                   1
   completedTaskCount        :                  67
   taskCount                 :                  68
   ***********************************************
   
   2024-01-17 15:39:02,849 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] 
   ***********************************************
                   Job info detail
   ***********************************************
   createdJobCount           :                   0
   scheduledJobCount         :                   0
   runningJobCount           :                   1
   failingJobCount           :                   0
   failedJobCount            :                   0
   cancellingJobCount        :                   0
   canceledJobCount          :                   0
   finishedJobCount          :                   0
   restartingJobCount        :                   0
   suspendedJobCount         :                   0
   reconcilingJobCount       :                   0
   ***********************************************
   
   2024-01-17 15:39:05,613 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
   ***********************************************
              Job Progress Information
   ***********************************************
   Job Id                    :  799900012245942273
   Read Count So Far         :               41192
   Write Count So Far        :               32996
   Average Read Count        :                50/s
   Average Write Count       :                50/s
   Last Statistic Time       : 2024-01-17 15:38:05
   Current Statistic Time    : 2024-01-17 15:39:05
   ***********************************************
   
   2024-01-17 15:39:45,040 INFO  org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 4
   2024-01-17 15:40:02,845 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] 
   ***********************************************
        CoordinatorService Thread Pool Status
   ***********************************************
   activeCount               :                   1
   corePoolSize              :                   0
   maximumPoolSize           :          2147483647
   poolSize                  :                   4
   completedTaskCount        :                  72
   taskCount                 :                  73
   ***********************************************
   
   2024-01-17 15:40:02,847 INFO  org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-478968] [5.1] 
   ***********************************************
                   Job info detail
   ***********************************************
   createdJobCount           :                   0
   scheduledJobCount         :                   0
   runningJobCount           :                   1
   failingJobCount           :                   0
   failedJobCount            :                   0
   cancellingJobCount        :                   0
   canceledJobCount          :                   0
   finishedJobCount          :                   0
   restartingJobCount        :                   0
   suspendedJobCount         :                   0
   reconcilingJobCount       :                   0
   ***********************************************
   
   2024-01-17 15:40:05,609 INFO  org.apache.seatunnel.engine.client.job.JobMetricsRunner - 
   ***********************************************
              Job Progress Information
   ***********************************************
   Job Id                    :  799900012245942273
   Read Count So Far         :               44192
   Write Count So Far        :               35996
   Average Read Count        :                50/s
   Average Write Count       :                50/s
   Last Statistic Time       : 2024-01-17 15:39:05
   Current Statistic Time    : 2024-01-17 15:40:05
   ***********************************************
   ```
   
   
   ### Zeta or Flink or Spark Version
   
   _No response_
   
   ### Java or Scala Version
   
   _No response_
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Attempted to collect local articles and HDFS text files into Doris. The working speed is very slow whether it is collecting locally or from HDFS [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6240:
URL: https://github.com/apache/seatunnel/issues/6240#issuecomment-1992811355

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Attempted to collect local articles and HDFS text files into Doris. The working speed is very slow whether it is collecting locally or from HDFS [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6240:
URL: https://github.com/apache/seatunnel/issues/6240#issuecomment-2008392269

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Attempted to collect local articles and HDFS text files into Doris. The working speed is very slow whether it is collecting locally or from HDFS [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6240: Attempted to collect local articles and HDFS text files into Doris. The working speed is very slow whether it is collecting locally or from HDFS
URL: https://github.com/apache/seatunnel/issues/6240


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org