You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/06/30 15:25:12 UTC
[GitHub] [doris] englefly opened a new pull request, #10534: [hotfix] abort load task if pending data exceed limit in table sink
englefly opened a new pull request, #10534:
URL: https://github.com/apache/doris/pull/10534
# Proposed changes
abort load task if pending data exceed limit (500M) in table sink
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #10534: [hotfix](dev-1.0.1) abort load task if pending data exceed limit in table sink
Posted by GitBox <gi...@apache.org>.
morningman commented on PR #10534:
URL: https://github.com/apache/doris/pull/10534#issuecomment-1172909529
Resolve by #10552
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman closed pull request #10534: [hotfix](dev-1.0.1) abort load task if pending data exceed limit in table sink
Posted by GitBox <gi...@apache.org>.
morningman closed pull request #10534: [hotfix](dev-1.0.1) abort load task if pending data exceed limit in table sink
URL: https://github.com/apache/doris/pull/10534
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] caiconghui commented on pull request #10534: [hotfix](dev-1.0.1) abort load task if pending data exceed limit in table sink
Posted by GitBox <gi...@apache.org>.
caiconghui commented on PR #10534:
URL: https://github.com/apache/doris/pull/10534#issuecomment-1171868093
what‘s the problem for this PR to solve？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #10534: [hotfix](dev-1.0.1) abort load task if pending data exceed limit in table sink
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #10534:
URL: https://github.com/apache/doris/pull/10534#discussion_r911566343
##########
be/src/vec/sink/vtablet_sink.cpp:
##########
@@ -117,12 +119,12 @@ Status VOlapTableSink::send(RuntimeState* state, vectorized::Block* input_block)
_partition_to_tablet_map.clear();
}
- //if pending bytes is more than 500M, wait
+ //if pending bytes is more than 500M, abort task
constexpr size_t MAX_PENDING_BYTES = 500 * 1024 * 1024;
- if ( get_pending_bytes() > MAX_PENDING_BYTES){
- while(get_pending_bytes() < MAX_PENDING_BYTES){
- std::this_thread::sleep_for(std::chrono::microseconds(500));
- }
+ if (get_pending_bytes() > MAX_PENDING_BYTES) {
Review Comment:
Not a good fix.
If the data size in a single batch is larger than 500MB, user can not load data anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org