[GitHub] [incubator-seatunnel] Hisoka-X opened a new issue, #3182: [Bug] [Engine] Synchronizing massive data from mysql to hdfs sometime will failed.

[GitBox <gi...@apache.org>]

Hisoka-X opened a new issue, #3182:
URL: https://github.com/apache/incubator-seatunnel/issues/3182

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   When 30 million mysql row are synchronized to hdfs through 8 parallellism, writes will be completed, but the rename operation will not be completed, resulting in incomplete target path data. Use SeaTunnel Engine.
   
   ### SeaTunnel Version
   
   dev
   
   ### SeaTunnel Config
   
   ```conf
   env {
     # You can set engine configuration here
     job.mode = "BATCH"
     checkpoint.interval = 30000
     #execution.checkpoint.data-uri = "hdfs://localhost:9000/checkpoint"
   }
   
   source {
     # This is a example source plugin **only for test and demonstrate the feature source plugin**
     jdbc{
       url = "jdbc:mysql://node1:3306/test"
       driver = "com.mysql.cj.jdbc.Driver"
       user = "root"
       password = "password"
       query = "select * from type_source_table where id"
       partition_column = "id"
       parallelism = 8
     }
   }
   
   transform {
   }
   
   sink {
     HdfsFile {
       fs.defaultFS="hdfs://hadoop1:9000"
       path="/test/result/"
       field_delimiter="\t"
       row_delimiter="\n"
       file_name_expression="${transactionId}_${now}"
       file_format="text"
       filename_time_format="yyyy.MM.dd"
       is_enable_transaction=true
     }
   }
   ```
   
   
   ### Running Command
   
   ```shell
   ./bin/seatunnel.sh -c ../engine_mysql2hdfs.conf -e LOCAL
   ```
   
   
   ### Error Exception
   
   ```log
   none
   ```
   
   
   ### Flink or Spark Version
   
   _No response_
   
   ### Java or Scala Version
   
   _No response_
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-seatunnel] github-actions[bot] commented on issue #3182: [Bug] [Engine] Synchronizing massive data from mysql to hdfs sometime will failed.

[GitBox <gi...@apache.org>]

github-actions[bot] commented on issue #3182:
URL: https://github.com/apache/incubator-seatunnel/issues/3182#issuecomment-1326910025

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-seatunnel] EricJoy2048 closed issue #3182: [Bug] [Engine] Synchronizing massive data from mysql to hdfs sometime will failed.

[GitBox <gi...@apache.org>]

EricJoy2048 closed issue #3182: [Bug] [Engine] Synchronizing massive data from mysql to hdfs sometime will failed.
URL: https://github.com/apache/incubator-seatunnel/issues/3182


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org