You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/06/23 03:55:37 UTC
[GitHub] [doris] englefly opened a new pull request, #10354: Cir234 draft
englefly opened a new pull request, #10354:
URL: https://github.com/apache/doris/pull/10354
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
yiguolei commented on code in PR #10354:
URL: https://github.com/apache/doris/pull/10354#discussion_r904737714
##########
be/src/vec/exec/volap_scan_node.cpp:
##########
@@ -228,8 +228,12 @@ void VOlapScanNode::scanner_thread(VOlapScanner* scanner) {
std::lock_guard<std::mutex> l(_free_blocks_lock);
_free_blocks.emplace_back(block);
} else {
+ // block should not exceed MAX_BLOCK_SIZE,
+ // in order to avoid generating supper huge block when reading big-wide-table
+ constexpr size_t MAX_BLOCK_SIZE = 1024 * 1024 * 100;
if (!blocks.empty() &&
- blocks.back()->rows() + block->rows() <= _runtime_state->batch_size()) {
+ blocks.back()->rows() + block->rows() <= _runtime_state->batch_size() &&
Review Comment:
这个地方可能不用
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #10354:
URL: https://github.com/apache/doris/pull/10354#discussion_r904799723
##########
be/src/vec/exec/volap_scan_node.cpp:
##########
@@ -228,8 +228,12 @@ void VOlapScanNode::scanner_thread(VOlapScanner* scanner) {
std::lock_guard<std::mutex> l(_free_blocks_lock);
_free_blocks.emplace_back(block);
} else {
+ // block should not exceed MAX_BLOCK_SIZE,
+ // in order to avoid generating supper huge block when reading big-wide-table
+ constexpr size_t MAX_BLOCK_SIZE = 1024 * 1024 * 100;
if (!blocks.empty() &&
- blocks.back()->rows() + block->rows() <= _runtime_state->batch_size()) {
+ blocks.back()->rows() + block->rows() <= _runtime_state->batch_size() &&
Review Comment:
将外层while循环的条件调整，遇到 raw_bytes_read >= raw_bytes_threshold 就推出循环，这样可以避免大宽表时生成巨大的block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #10354:
URL: https://github.com/apache/doris/pull/10354#discussion_r904780991
##########
be/src/exec/tablet_sink.cpp:
##########
@@ -315,9 +315,19 @@ Status NodeChannel::add_row(BlockRow& block_row, int64_t tablet_id) {
SCOPED_ATOMIC_TIMER(&_mem_exceeded_block_ns);
SleepFor(MonoDelta::FromMilliseconds(10));
}
-
+ constexpr size_t BATCH_SIZE_FOR_SEND = 2 * 1024 * 1024; //2M
auto row_no = _cur_batch->add_row();
- if (row_no == RowBatch::INVALID_ROW_INDEX) {
+ if (row_no == RowBatch::INVALID_ROW_INDEX || _cur_batch->total_byte_size() > BATCH_SIZE_FOR_SEND) {
+ //if pending bytes is more than 1G, wait
+ constexpr size_t MAX_PENDING_BYTES = 1073741824; //1G
+ if (_pending_batches_bytes>1073741824){
+ LOG(INFO)<<"NodeChannel pending more than 1G data, wait..";
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #10354:
URL: https://github.com/apache/doris/pull/10354#discussion_r904821634
##########
be/src/exec/tablet_sink.cpp:
##########
@@ -315,9 +315,19 @@ Status NodeChannel::add_row(BlockRow& block_row, int64_t tablet_id) {
SCOPED_ATOMIC_TIMER(&_mem_exceeded_block_ns);
SleepFor(MonoDelta::FromMilliseconds(10));
}
-
+ constexpr size_t BATCH_SIZE_FOR_SEND = 2 * 1024 * 1024; //2M
auto row_no = _cur_batch->add_row();
- if (row_no == RowBatch::INVALID_ROW_INDEX) {
+ if (row_no == RowBatch::INVALID_ROW_INDEX || _cur_batch->total_byte_size() > BATCH_SIZE_FOR_SEND) {
+ //if pending bytes is more than 1G, wait
+ constexpr size_t MAX_PENDING_BYTES = 1073741824; //1G
+ if (_pending_batches_bytes>1073741824){
Review Comment:
master版本可以
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #10354: [hotfix](dev-1.0.1) 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
yiguolei merged PR #10354:
URL: https://github.com/apache/doris/pull/10354
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #10354:
URL: https://github.com/apache/doris/pull/10354#issuecomment-1164279780
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
yiguolei commented on code in PR #10354:
URL: https://github.com/apache/doris/pull/10354#discussion_r904731393
##########
be/src/exec/tablet_sink.h:
##########
@@ -227,6 +227,11 @@ class NodeChannel {
_node_info.brpc_port);
}
+ size_t get_pending_bytes() {
+ size_t v = _pending_batches_bytes;
Review Comment:
这个不需要这么写
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
yiguolei commented on code in PR #10354:
URL: https://github.com/apache/doris/pull/10354#discussion_r904730730
##########
be/src/exec/tablet_sink.cpp:
##########
@@ -315,9 +315,19 @@ Status NodeChannel::add_row(BlockRow& block_row, int64_t tablet_id) {
SCOPED_ATOMIC_TIMER(&_mem_exceeded_block_ns);
SleepFor(MonoDelta::FromMilliseconds(10));
}
-
+ constexpr size_t BATCH_SIZE_FOR_SEND = 2 * 1024 * 1024; //2M
auto row_no = _cur_batch->add_row();
- if (row_no == RowBatch::INVALID_ROW_INDEX) {
+ if (row_no == RowBatch::INVALID_ROW_INDEX || _cur_batch->total_byte_size() > BATCH_SIZE_FOR_SEND) {
Review Comment:
使用tuple pool的size 来计算
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
yiguolei commented on code in PR #10354:
URL: https://github.com/apache/doris/pull/10354#discussion_r904722453
##########
be/src/exec/tablet_sink.cpp:
##########
@@ -315,9 +315,19 @@ Status NodeChannel::add_row(BlockRow& block_row, int64_t tablet_id) {
SCOPED_ATOMIC_TIMER(&_mem_exceeded_block_ns);
SleepFor(MonoDelta::FromMilliseconds(10));
}
-
+ constexpr size_t BATCH_SIZE_FOR_SEND = 2 * 1024 * 1024; //2M
auto row_no = _cur_batch->add_row();
- if (row_no == RowBatch::INVALID_ROW_INDEX) {
+ if (row_no == RowBatch::INVALID_ROW_INDEX || _cur_batch->total_byte_size() > BATCH_SIZE_FOR_SEND) {
+ //if pending bytes is more than 1G, wait
+ constexpr size_t MAX_PENDING_BYTES = 1073741824; //1G
Review Comment:
这个条件似乎不需要，因为在313 行，已经检查了pending_batches_bytes 了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #10354:
URL: https://github.com/apache/doris/pull/10354#discussion_r904799723
##########
be/src/vec/exec/volap_scan_node.cpp:
##########
@@ -228,8 +228,12 @@ void VOlapScanNode::scanner_thread(VOlapScanner* scanner) {
std::lock_guard<std::mutex> l(_free_blocks_lock);
_free_blocks.emplace_back(block);
} else {
+ // block should not exceed MAX_BLOCK_SIZE,
+ // in order to avoid generating supper huge block when reading big-wide-table
+ constexpr size_t MAX_BLOCK_SIZE = 1024 * 1024 * 100;
if (!blocks.empty() &&
- blocks.back()->rows() + block->rows() <= _runtime_state->batch_size()) {
+ blocks.back()->rows() + block->rows() <= _runtime_state->batch_size() &&
Review Comment:
将外层while循环的条件调整，遇到 raw_bytes_read >= raw_bytes_threshold 就退出循环，这样可以避免大宽表时生成巨大的block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on code in PR #10354:
URL: https://github.com/apache/doris/pull/10354#discussion_r904811579
##########
be/src/exec/tablet_sink.cpp:
##########
@@ -315,9 +315,19 @@ Status NodeChannel::add_row(BlockRow& block_row, int64_t tablet_id) {
SCOPED_ATOMIC_TIMER(&_mem_exceeded_block_ns);
SleepFor(MonoDelta::FromMilliseconds(10));
}
-
+ constexpr size_t BATCH_SIZE_FOR_SEND = 2 * 1024 * 1024; //2M
auto row_no = _cur_batch->add_row();
- if (row_no == RowBatch::INVALID_ROW_INDEX) {
+ if (row_no == RowBatch::INVALID_ROW_INDEX || _cur_batch->total_byte_size() > BATCH_SIZE_FOR_SEND) {
+ //if pending bytes is more than 1G, wait
+ constexpr size_t MAX_PENDING_BYTES = 1073741824; //1G
+ if (_pending_batches_bytes>1073741824){
Review Comment:
之后这里用 MemTracker 的值作为判断条件是不是更合适
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #10354:
URL: https://github.com/apache/doris/pull/10354#issuecomment-1164279814
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10354: [hotfix](dev-1.0.1) 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #10354:
URL: https://github.com/apache/doris/pull/10354#issuecomment-1165141735
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #10354:
URL: https://github.com/apache/doris/pull/10354#discussion_r904763760
##########
be/src/exec/tablet_sink.cpp:
##########
@@ -315,9 +315,19 @@ Status NodeChannel::add_row(BlockRow& block_row, int64_t tablet_id) {
SCOPED_ATOMIC_TIMER(&_mem_exceeded_block_ns);
SleepFor(MonoDelta::FromMilliseconds(10));
}
-
+ constexpr size_t BATCH_SIZE_FOR_SEND = 2 * 1024 * 1024; //2M
auto row_no = _cur_batch->add_row();
- if (row_no == RowBatch::INVALID_ROW_INDEX) {
+ if (row_no == RowBatch::INVALID_ROW_INDEX || _cur_batch->total_byte_size() > BATCH_SIZE_FOR_SEND) {
+ //if pending bytes is more than 1G, wait
+ constexpr size_t MAX_PENDING_BYTES = 1073741824; //1G
+ if (_pending_batches_bytes>1073741824){
+ LOG(INFO)<<"NodeChannel pending more than 1G data, wait..";
Review Comment:
别用 INFO，这会有很多日志
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #10354:
URL: https://github.com/apache/doris/pull/10354#discussion_r904821261
##########
be/src/runtime/tablets_channel.cpp:
##########
@@ -265,6 +266,7 @@ Status TabletsChannel::reduce_mem_usage(int64_t mem_limit) {
}
}
VLOG_CRITICAL << "flush " << counter << " memtables to reduce memory: " << sum;
+ LOG(INFO) << "before reduce: tabelts channle consume "<< mem_consumption();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #10354:
URL: https://github.com/apache/doris/pull/10354#discussion_r904770639
##########
be/src/exec/tablet_sink.cpp:
##########
@@ -315,9 +315,19 @@ Status NodeChannel::add_row(BlockRow& block_row, int64_t tablet_id) {
SCOPED_ATOMIC_TIMER(&_mem_exceeded_block_ns);
SleepFor(MonoDelta::FromMilliseconds(10));
}
-
+ constexpr size_t BATCH_SIZE_FOR_SEND = 2 * 1024 * 1024; //2M
auto row_no = _cur_batch->add_row();
- if (row_no == RowBatch::INVALID_ROW_INDEX) {
+ if (row_no == RowBatch::INVALID_ROW_INDEX || _cur_batch->total_byte_size() > BATCH_SIZE_FOR_SEND) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
englefly commented on code in PR #10354:
URL: https://github.com/apache/doris/pull/10354#discussion_r904771327
##########
be/src/exec/tablet_sink.h:
##########
@@ -227,6 +227,11 @@ class NodeChannel {
_node_info.brpc_port);
}
+ size_t get_pending_bytes() {
+ size_t v = _pending_batches_bytes;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #10354: [fix] 5 concurrent "insert...select..." OOM
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on code in PR #10354:
URL: https://github.com/apache/doris/pull/10354#discussion_r904803085
##########
be/src/runtime/tablets_channel.cpp:
##########
@@ -265,6 +266,7 @@ Status TabletsChannel::reduce_mem_usage(int64_t mem_limit) {
}
}
VLOG_CRITICAL << "flush " << counter << " memtables to reduce memory: " << sum;
+ LOG(INFO) << "before reduce: tabelts channle consume "<< mem_consumption();
Review Comment:
LOG(INFO) here may be frequent, consider deleting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org