You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "k8slight (via GitHub)" <gi...@apache.org> on 2023/07/11 03:41:36 UTC

[GitHub] [seatunnel] k8slight opened a new issue, #5058: [Bug] [HdfsFile/LocalFile to MYSQL] Write data repeatedly

k8slight opened a new issue, #5058:
URL: https://github.com/apache/seatunnel/issues/5058

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   read data from csv file by HdfsFile/LocalFile plugin, then write the data to mysql.
   the number of csv file lines is 357147, but the number of database data rows exceeded 357147
   ![image](https://github.com/apache/seatunnel/assets/56961637/3aa28ab6-65df-430c-b172-beaf233ec84c)
   ![image](https://github.com/apache/seatunnel/assets/56961637/d22bfa4d-3c43-4996-979b-b6703bd28c36)
   ![image](https://github.com/apache/seatunnel/assets/56961637/819638e2-c1e7-4ff7-8d34-994eb0ed17af)
   ![image](https://github.com/apache/seatunnel/assets/56961637/113b0ccc-8c82-4a90-b2fd-b98c1b30a3ed)
   
   
   
   ### SeaTunnel Version
   
   2.3.2
   
   ### SeaTunnel Config
   
   ```conf
   env {
       execution.parallelism = 1
       job.mode = "BATCH"
   }
   # 在source所属的块中配置数据源
   source {
   
   HdfsFile {
     schema {
       fields {
         user_id = int
         item_id = int
         category_id = int
         action = string
         timestamp = bigint
       }
     }
     path = "/data/source/UserBehavior01.csv"
     file_format_type = "csv"
     fs.defaultFS = "hdfs://mycluster"
     hdfs_site_path = "/opt/hadoop-2.10.2/etc/hadoop/hdfs-site.xml"
   }
   
   }
   # 在transform的块中声明转换插件
   transform {
    
   }
   
   # 在sink块中声明要输出到哪
   sink {
   
   jdbc {
       url = "jdbc:mysql://node5:3306/bigdata_ods?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull&allowMultiQueries=true&serverTimezone=GMT%2B8&useSSL=false"
       driver = "com.mysql.cj.jdbc.Driver"
       user = "****"
       password = "****"
       query = "insert into USER_BEHAVIOR values(?,?,?,?,?)"
   }
   
   }
   ```
   
   
   ### Running Command
   
   ```shell
   seatunnel.sh --config test.config --deploy-mode cluster
   ```
   
   
   ### Error Exception
   
   ```log
   no error
   ```
   
   
   ### Flink or Spark Version
   
   _No response_
   
   ### Java or Scala Version
   
   java version "1.8.0_361"
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [HdfsFile/LocalFile to MYSQL] Write data repeatedly [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #5058: [Bug] [HdfsFile/LocalFile to MYSQL] Write data repeatedly
URL: https://github.com/apache/seatunnel/issues/5058


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] k8slight commented on issue #5058: [Bug] [HdfsFile/LocalFile to MYSQL] Write data repeatedly

Posted by "k8slight (via GitHub)" <gi...@apache.org>.
k8slight commented on issue #5058:
URL: https://github.com/apache/seatunnel/issues/5058#issuecomment-1631722338

   [local] mode have no such problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [HdfsFile/LocalFile to MYSQL] Write data repeatedly [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5058:
URL: https://github.com/apache/seatunnel/issues/5058#issuecomment-1745929881

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] github-actions[bot] commented on issue #5058: [Bug] [HdfsFile/LocalFile to MYSQL] Write data repeatedly

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5058:
URL: https://github.com/apache/seatunnel/issues/5058#issuecomment-1675559550

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org