You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "gnehil (via GitHub)" <gi...@apache.org> on 2023/10/24 11:47:46 UTC
[PR] [improvement] batch load retry [doris-spark-connector]
gnehil opened a new pull request, #148:
URL: https://github.com/apache/doris-spark-connector/pull/148
# Problem Summary
In the current implementation, due to traversing data through iterators and sending data in a stream, batch loading does not implement batch retry. Task retry will be performed after the load request fails, and the cost of task retry is relatively high. For some scenarios such as network jitter, batch retry is a better way
## Proposed changes:
The implementation of this PR is to choose whether to retry batch loading when it fails through configuration.
The behavior is as follows:
Batch retry is enabled when sink.max-retries is set greater than 0,an iterator is implemented to cache batch data, and a count value is used to control batch writing.
When the batch fails, reset the count and set the flag bit, and re-read the current batch data from the cache array.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement] batch load retry [doris-spark-connector]
Posted by "JNSimba (via GitHub)" <gi...@apache.org>.
JNSimba merged PR #148:
URL: https://github.com/apache/doris-spark-connector/pull/148
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org