You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "baishaoisde (via GitHub)" <gi...@apache.org> on 2023/04/06 01:11:14 UTC
[GitHub] [doris-spark-connector] baishaoisde opened a new issue, #92: [Enhancement] Add a parameter that controls the number of StreamLoad tasks committed per partition
baishaoisde opened a new issue, #92:
URL: https://github.com/apache/doris-spark-connector/issues/92
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Description
If the amount of data in a partition is greater than INSERT_BARCH_SIZE, each task commits multiple StreamLoad tasks. If the task fails to retry, all data in the partition is recommitted to the StreamLoad task, as well as the data that was previously successfully written. Data duplication occurs.
当一个分区中的数据量大于参数INSERT_BARCH_SIZE时，每个task便会提交多个StreamLoad任务，如果任务发生失败重试，那么该分区的所有数据便会重新提交StreamLoad任务，对于之前成功写入的数据也会重新提交，造成数据重复。
我的建议是增加一个参数，如果开启则强制每个分区只提交一个StreamLoad，保证数据不会被重复提交。
### Solution
My suggestion is to add a parameter that, if enabled, forces only one StreamLoad per partition to ensure that data is not repeatedly committed.
我的建议是增加一个参数，如果开启则强制每个分区只提交一个StreamLoad，保证数据不会被重复提交。
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Enhancement] Add a parameter that controls the number of StreamLoad tasks committed per partition [doris-spark-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on issue #92:
URL: https://github.com/apache/doris-spark-connector/issues/92#issuecomment-3478778971
This has taken too long, I'll close this now. If you still have problems, you can create a new one. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Enhancement] Add a parameter that controls the number of StreamLoad tasks committed per partition [doris-spark-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba closed issue #92: [Enhancement] Add a parameter that controls the number of StreamLoad tasks committed per partition
URL: https://github.com/apache/doris-spark-connector/issues/92
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Enhancement] Add a parameter that controls the number of StreamLoad tasks committed per partition [doris-spark-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba commented on issue #92:
URL: https://github.com/apache/doris-spark-connector/issues/92#issuecomment-2221897134
Hello @baishaoisde , can turning on 2pc solve this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org