You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/12 00:54:53 UTC
[GitHub] [hudi] sunmingqiaa opened a new issue, #6375: [SUPPORT]丢数了!flink cdc to hudi cow
sunmingqiaa opened a new issue, #6375:
URL: https://github.com/apache/hudi/issues/6375
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1.
2.
3.
4.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :0.11.1
* Spark version :
* Hive version :2.11
* Hadoop version :3.0
* Storage (HDFS/S3/GCS..) :hdfs
* Running on Docker? (yes/no) :no
**Additional context**
Add any other context about the problem here.
'connector'='hudi',\n" +
'path'= %s, \n" +
'table.type' = 'COPY_ON_WRITE',\n" +
'hoodie.datasource.write.recordkey.field'= %s,\n"
'write.tasks'= %s,\n" +
'write.task.max.size'=%s,\n" +
'write.merge.max_memory'=%s,\n" +
'hoodie.cleaner.commits.retained'=%s,\n" +
'hive_sync.enable'= 'true',\n" +
'hive_sync.mode'= 'hms',\n" +
'hive_sync.metastore.uris'= %s,\n" +
'hive_sync.jdbc_url'= %s,\n" +
'hive_sync.table'= %s,\n" +
'hive_sync.db'= %s,\n" +
'hive_sync.username'= %s,\n" +
'hive_sync.password'= %s,\n" +
'write.rate.limit'= %s,\n" +
'hive_sync.support_timestamp'= 'false'\n" +
试过导入100万数据和1000万数据,最近几次导完数都会发现少一部分数据,重复试过几次,每次导完查询结果都不一样,丢的数量不一定,hive和flilnksql查的结果一样,flink程序无异常。但是前几天测的时候没有发现丢数,两次区别在于,本次设置了write.rate.limit=1000。
**Stacktrace**


```Add the stacktrace of the error.```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 closed issue #6375: [SUPPORT]丢数了!flink cdc to hudi cow
Posted by GitBox <gi...@apache.org>.
danny0405 closed issue #6375: [SUPPORT]丢数了!flink cdc to hudi cow
URL: https://github.com/apache/hudi/issues/6375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #6375: [SUPPORT]丢数了!flink cdc to hudi cow
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #6375:
URL: https://github.com/apache/hudi/issues/6375#issuecomment-1212632685
It's a known bug, we have fixed it in: https://github.com/apache/hudi/pull/6179,
You can use this patch branch: https://github.com/apache/hudi/pull/6182 instead
or wait for the release 0.12.0 which would comes out soon ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] sunmingqiaa commented on issue #6375: [SUPPORT]丢数了!flink cdc to hudi cow
Posted by GitBox <gi...@apache.org>.
sunmingqiaa commented on issue #6375:
URL: https://github.com/apache/hudi/issues/6375#issuecomment-1212634123
> It's a known bug, we have fixed it in: #6179, You can use this patch branch: #6182 instead or wait for the release 0.12.0 which is coming out very soon ~
ok,thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org