You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "k8slight (via GitHub)" <gi...@apache.org> on 2023/07/11 03:41:36 UTC
[GitHub] [seatunnel] k8slight opened a new issue, #5058: [Bug] [HdfsFile/LocalFile to MYSQL] Write data repeatedly
k8slight opened a new issue, #5058:
URL: https://github.com/apache/seatunnel/issues/5058
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
read data from csv file by HdfsFile/LocalFile plugin, then write the data to mysql.
the number of csv file lines is 357147, but the number of database data rows exceeded 357147




### SeaTunnel Version
2.3.2
### SeaTunnel Config
```conf
env {
execution.parallelism = 1
job.mode = "BATCH"
}
# 在source所属的块中配置数据源
source {
HdfsFile {
schema {
fields {
user_id = int
item_id = int
category_id = int
action = string
timestamp = bigint
}
}
path = "/data/source/UserBehavior01.csv"
file_format_type = "csv"
fs.defaultFS = "hdfs://mycluster"
hdfs_site_path = "/opt/hadoop-2.10.2/etc/hadoop/hdfs-site.xml"
}
}
# 在transform的块中声明转换插件
transform {
}
# 在sink块中声明要输出到哪
sink {
jdbc {
url = "jdbc:mysql://node5:3306/bigdata_ods?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull&allowMultiQueries=true&serverTimezone=GMT%2B8&useSSL=false"
driver = "com.mysql.cj.jdbc.Driver"
user = "****"
password = "****"
query = "insert into USER_BEHAVIOR values(?,?,?,?,?)"
}
}
```
### Running Command
```shell
seatunnel.sh --config test.config --deploy-mode cluster
```
### Error Exception
```log
no error
```
### Flink or Spark Version
_No response_
### Java or Scala Version
java version "1.8.0_361"
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [HdfsFile/LocalFile to MYSQL] Write data repeatedly [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #5058: [Bug] [HdfsFile/LocalFile to MYSQL] Write data repeatedly
URL: https://github.com/apache/seatunnel/issues/5058
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] k8slight commented on issue #5058: [Bug] [HdfsFile/LocalFile to MYSQL] Write data repeatedly
Posted by "k8slight (via GitHub)" <gi...@apache.org>.
k8slight commented on issue #5058:
URL: https://github.com/apache/seatunnel/issues/5058#issuecomment-1631722338
[local] mode have no such problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [HdfsFile/LocalFile to MYSQL] Write data repeatedly [seatunnel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5058:
URL: https://github.com/apache/seatunnel/issues/5058#issuecomment-1745929881
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] github-actions[bot] commented on issue #5058: [Bug] [HdfsFile/LocalFile to MYSQL] Write data repeatedly
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5058:
URL: https://github.com/apache/seatunnel/issues/5058#issuecomment-1675559550
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org