You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "WinkerDu (via GitHub)" <gi...@apache.org> on 2023/11/16 10:52:43 UTC
[PR] [Enhance](fe) select BE local broker to scan Hive table when 'broker.name' in hms catalog is specified [doris]
WinkerDu opened a new pull request, #27122:
URL: https://github.com/apache/doris/pull/27122
## Proposed changes
Issue Number: close #xxx
Since #24830 introduce `broker.name` in hms catalog, data scan will run on specified brokers.
And [doris operator](https://github.com/selectdb/doris-operator) support BE and broker deployed in same pod, BE access local broker is the fastest approach to access data.
In previous logic, every inputSplit will select one BE to execute, then randomly select one broker for actual data access, BE and related broker are always located on separate K8S pod.
This pr optimizes the broker select strategy to prioritize BE-local broker when `broker.name` is specified in hms catalog.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhance](fe) select BE local broker to scan Hive table when 'broker.name' in hms catalog is specified [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #27122:
URL: https://github.com/apache/doris/pull/27122
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhance](fe) select BE local broker to scan Hive table when 'broker.name' in hms catalog is specified [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27122:
URL: https://github.com/apache/doris/pull/27122#issuecomment-1815003381
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.27 seconds
stream load tsv: 574 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17098515535 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhance](fe) select BE local broker to scan Hive table when 'broker.name' in hms catalog is specified [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #27122:
URL: https://github.com/apache/doris/pull/27122#issuecomment-1814800916
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhance](fe) select BE local broker to scan Hive table when 'broker.name' in hms catalog is specified [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27122:
URL: https://github.com/apache/doris/pull/27122#issuecomment-1814307599
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhance](fe) select BE local broker to scan Hive table when 'broker.name' in hms catalog is specified [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27122:
URL: https://github.com/apache/doris/pull/27122#issuecomment-1814307667
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhance](fe) select BE local broker to scan Hive table when 'broker.name' in hms catalog is specified [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27122:
URL: https://github.com/apache/doris/pull/27122#issuecomment-1814990793
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 2412be4ee8957adc536568796a0932daa2aa27a3, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4913 4640 4620 4620
q2 361 170 159 159
q3 2017 1941 1859 1859
q4 1377 1269 1255 1255
q5 3970 3948 3977 3948
q6 243 129 134 129
q7 1410 864 890 864
q8 2756 2778 2762 2762
q9 9868 9795 9535 9535
q10 3479 3526 3543 3526
q11 372 245 240 240
q12 440 297 303 297
q13 4536 3821 3784 3784
q14 312 295 298 295
q15 573 542 527 527
q16 675 588 578 578
q17 1131 961 931 931
q18 7875 7277 7342 7277
q19 1693 1681 1671 1671
q20 523 304 301 301
q21 4383 3963 4001 3963
q22 481 379 373 373
Total cold run time: 53388 ms
Total hot run time: 48894 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4570 4551 4547 4547
q2 347 220 235 220
q3 4010 4013 3984 3984
q4 2700 2678 2684 2678
q5 9724 9716 9690 9690
q6 243 126 125 125
q7 2609 2300 2289 2289
q8 4517 4468 4482 4468
q9 13207 13080 13153 13080
q10 4102 4212 4209 4209
q11 799 659 635 635
q12 990 824 822 822
q13 4254 3599 3554 3554
q14 384 341 346 341
q15 579 531 525 525
q16 723 696 665 665
q17 3884 3907 3897 3897
q18 9666 9086 9103 9086
q19 1803 1793 1760 1760
q20 2383 2072 2054 2054
q21 8748 8627 8556 8556
q22 895 816 791 791
Total cold run time: 81137 ms
Total hot run time: 77976 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhance](fe) select BE local broker to scan Hive table when 'broker.name' in hms catalog is specified [doris]
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on PR #27122:
URL: https://github.com/apache/doris/pull/27122#issuecomment-1814210905
run buildall.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhance](fe) select BE local broker to scan Hive table when 'broker.name' in hms catalog is specified [doris]
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on PR #27122:
URL: https://github.com/apache/doris/pull/27122#issuecomment-1815905515
run arm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhance](fe) select BE local broker to scan Hive table when 'broker.name' in hms catalog is specified [doris]
Posted by "WinkerDu (via GitHub)" <gi...@apache.org>.
WinkerDu commented on PR #27122:
URL: https://github.com/apache/doris/pull/27122#issuecomment-1815830532
run feut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org