You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xinyiZzz (via GitHub)" <gi...@apache.org> on 2023/12/10 13:33:32 UTC
[PR] [chore](scan) Add session veriable query_max_scan_bytes [doris]
xinyiZzz opened a new pull request, #28217:
URL: https://github.com/apache/doris/pull/28217
## Proposed changes
add Query largest scan bytes in single BE.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](scan) Add session veriable query_max_scan_bytes [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #28217:
URL: https://github.com/apache/doris/pull/28217#discussion_r1421753303
##########
be/src/vec/exec/scan/vscanner.cpp:
##########
@@ -96,6 +96,7 @@ Status VScanner::get_block(RuntimeState* state, Block* block, bool* eof) {
}
_num_rows_read += block->rows();
_num_byte_read += block->allocated_bytes();
+ RETURN_IF_ERROR(state->get_query_ctx()->add_scan_bytes(block->allocated_bytes()));
Review Comment:
这里，每10次，调用一下，我怕太频繁了
##########
be/src/vec/exec/scan/vscanner.cpp:
##########
@@ -96,6 +96,7 @@ Status VScanner::get_block(RuntimeState* state, Block* block, bool* eof) {
}
_num_rows_read += block->rows();
_num_byte_read += block->allocated_bytes();
+ RETURN_IF_ERROR(state->get_query_ctx()->add_scan_bytes(block->allocated_bytes()));
Review Comment:
这里，每10次，调用一下，我怕太频繁了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](scan) Add session veriable query_max_scan_bytes [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #28217: [chore](scan) Add session veriable query_max_scan_bytes
URL: https://github.com/apache/doris/pull/28217
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](scan) Add session veriable query_max_scan_bytes [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #28217:
URL: https://github.com/apache/doris/pull/28217#issuecomment-1848967263
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](scan) Add session veriable query_max_scan_bytes [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28217:
URL: https://github.com/apache/doris/pull/28217#issuecomment-1849015939
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 92071290ab1e33052be4e3584f9c8a5632af428d, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4697 4455 4474 4455
q2 368 149 156 149
q3 1460 1249 1230 1230
q4 1091 909 864 864
q5 3126 3122 3135 3122
q6 245 126 128 126
q7 990 491 475 475
q8 2207 2219 2182 2182
q9 6688 6693 6679 6679
q10 3640 3231 3262 3231
q11 327 211 205 205
q12 350 205 202 202
q13 4564 3792 3789 3789
q14 245 210 213 210
q15 570 521 538 521
q16 443 389 390 389
q17 996 561 545 545
q18 7536 7250 6918 6918
q19 2346 1267 1440 1267
q20 535 298 285 285
q21 3083 2655 2630 2630
q22 350 275 287 275
Total cold run time: 45857 ms
Total hot run time: 39749 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4410 4369 4379 4369
q2 268 167 164 164
q3 3535 3536 3525 3525
q4 2398 2392 2387 2387
q5 5755 5714 5719 5714
q6 240 120 120 120
q7 2402 1869 1877 1869
q8 3512 3491 3508 3491
q9 9039 8988 8951 8951
q10 3914 3963 3985 3963
q11 493 377 370 370
q12 757 580 596 580
q13 4284 3584 3544 3544
q14 288 257 255 255
q15 563 528 508 508
q16 494 466 477 466
q17 1859 1860 1847 1847
q18 8712 8332 8238 8238
q19 1737 1737 1748 1737
q20 2260 1939 1941 1939
q21 6519 6170 6130 6130
q22 494 413 403 403
Total cold run time: 63933 ms
Total hot run time: 60570 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](scan) Add session veriable query_max_scan_bytes [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #28217:
URL: https://github.com/apache/doris/pull/28217#issuecomment-1849010238
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](scan) Add session veriable query_max_scan_bytes [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28217:
URL: https://github.com/apache/doris/pull/28217#issuecomment-1848990239
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.98 seconds
stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17219721807 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](scan) Add session veriable query_max_scan_bytes [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #28217:
URL: https://github.com/apache/doris/pull/28217#issuecomment-1849010175
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](scan) Add session veriable query_max_scan_bytes [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28217:
URL: https://github.com/apache/doris/pull/28217#issuecomment-1848986465
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit da7eaa4c44b8a48237098513a1192ed9d02c73b3, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4712 4460 4497 4460
q2 359 157 158 157
q3 1462 1221 1184 1184
q4 1098 929 911 911
q5 3105 3125 3148 3125
q6 251 126 128 126
q7 977 493 475 475
q8 2194 2199 2181 2181
q9 6688 6633 6642 6633
q10 3183 3245 3269 3245
q11 331 208 192 192
q12 344 207 206 206
q13 4504 3798 3788 3788
q14 244 211 210 210
q15 564 524 524 524
q16 450 390 386 386
q17 1002 581 539 539
q18 7713 7339 7248 7248
q19 1502 1298 1461 1298
q20 519 309 288 288
q21 3069 2666 2693 2666
q22 343 276 276 276
Total cold run time: 44614 ms
Total hot run time: 40118 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4429 4421 4373 4373
q2 267 167 173 167
q3 3552 3550 3538 3538
q4 2405 2396 2397 2396
q5 5730 5736 5749 5736
q6 241 121 123 121
q7 2393 1875 1871 1871
q8 3505 3508 3513 3508
q9 9035 8964 9031 8964
q10 3879 3981 3971 3971
q11 492 404 384 384
q12 760 596 596 596
q13 4306 3584 3593 3584
q14 274 248 263 248
q15 573 514 515 514
q16 513 461 474 461
q17 1854 1866 1820 1820
q18 8792 8289 8407 8289
q19 1689 1731 1723 1723
q20 2249 1964 1936 1936
q21 6514 6184 6197 6184
q22 506 439 419 419
Total cold run time: 63958 ms
Total hot run time: 60803 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](scan) Add session veriable query_max_scan_bytes [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28217:
URL: https://github.com/apache/doris/pull/28217#issuecomment-2155713435
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](scan) Add session veriable query_max_scan_bytes [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28217:
URL: https://github.com/apache/doris/pull/28217#issuecomment-1849019212
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.07 seconds
stream load tsv: 580 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.4 seconds inserted 10000000 Rows, about 352K ops/s
storage size: 17219805946 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](scan) Add session veriable query_max_scan_bytes [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28217:
URL: https://github.com/apache/doris/pull/28217#discussion_r1421753087
##########
be/src/runtime/query_context.h:
##########
@@ -199,6 +204,17 @@ class QueryContext {
pipeline::Dependency* get_execution_dependency() { return _execution_dependency.get(); }
+ Status add_scan_bytes(uint64_t bytes) {
Review Comment:
warning: method 'add_scan_bytes' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status add_scan_bytes(uint64_t bytes) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org