You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/10/10 07:51:10 UTC
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
BiteTheDDDDt opened a new pull request, #13246:
URL: https://github.com/apache/doris/pull/13246
# Proposed changes
make runtime filter breaking change(https://github.com/apache/doris/pull/12856) support rolling upgrade.
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BiteTheDDDDt merged pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
Posted by GitBox <gi...@apache.org>.
BiteTheDDDDt merged PR #13246:
URL: https://github.com/apache/doris/pull/13246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #13246:
URL: https://github.com/apache/doris/pull/13246#issuecomment-1292946621
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 38.73 seconds
load time: 571 seconds
storage size: 17154631732 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221027121422_clickbench_pr_34553.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #13246:
URL: https://github.com/apache/doris/pull/13246#issuecomment-1286545896
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 39.12 seconds
load time: 568 seconds
storage size: 17154655330 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221021150451_clickbench_pr_32352.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #13246:
URL: https://github.com/apache/doris/pull/13246#issuecomment-1293064794
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 38.39 seconds
load time: 573 seconds
storage size: 17154644824 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221027144130_clickbench_pr_34634.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
Posted by GitBox <gi...@apache.org>.
morningman commented on code in PR #13246:
URL: https://github.com/apache/doris/pull/13246#discussion_r1002729158
##########
be/src/vec/core/block.cpp:
##########
@@ -64,7 +64,14 @@ Block::Block(const std::vector<SlotDescriptor*>& slots, size_t block_size) {
}
Block::Block(const PBlock& pblock) {
- CHECK(HeartbeatServer::check_be_exec_version(pblock.be_exec_version()));
+ int be_exec_version = 0;
+ if (pblock.has_be_exec_version()) {
+ be_exec_version = pblock.be_exec_version();
+ } else {
+ LOG(WARNING) << "pblock.has_be_exec_version()=false, maybe pblock not be serialized well, "
Review Comment:
Agreed, remove this log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhangstar333 commented on pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
Posted by GitBox <gi...@apache.org>.
zhangstar333 commented on PR #13246:
URL: https://github.com/apache/doris/pull/13246#issuecomment-1293035228
LGTM, and fixed tpcds_sf1_p1 q85
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #13246:
URL: https://github.com/apache/doris/pull/13246#issuecomment-1292116484
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 38.85 seconds
load time: 566 seconds
storage size: 17154655328 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221026221153_clickbench_pr_34442.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13246:
URL: https://github.com/apache/doris/pull/13246#issuecomment-1294371041
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
Posted by GitBox <gi...@apache.org>.
HappenLee commented on code in PR #13246:
URL: https://github.com/apache/doris/pull/13246#discussion_r995682550
##########
gensrc/thrift/PaloInternalService.thrift:
##########
@@ -177,6 +177,9 @@ struct TQueryOptions {
49: optional bool skip_delete_predicate = false
50: optional bool enable_new_shuffle_hash_method
+
+ //
Review Comment:
add a comment or delete the `//`
##########
be/src/vec/core/block.cpp:
##########
@@ -64,7 +64,14 @@ Block::Block(const std::vector<SlotDescriptor*>& slots, size_t block_size) {
}
Block::Block(const PBlock& pblock) {
- CHECK(HeartbeatServer::check_be_exec_version(pblock.be_exec_version()));
+ int be_exec_version = 0;
+ if (pblock.has_be_exec_version()) {
+ be_exec_version = pblock.be_exec_version();
+ } else {
+ LOG(WARNING) << "pblock.has_be_exec_version()=false, maybe pblock not be serialized well, "
Review Comment:
if old BE do not have the version, the log seems will print everywhere
##########
be/test/olap/bloom_filter_column_predicate_test.cpp:
##########
@@ -145,7 +143,8 @@ TEST_F(TestBloomFilterColumnPredicate, FLOAT_COLUMN_VEC) {
bloom_filter->insert_fixed_len((char*)values, offsets, 3);
ColumnPredicate* pred = BloomFilterColumnPredicateFactory::create_column_predicate(
- 0, bloom_filter, OLAP_FIELD_TYPE_FLOAT);
+ BeExecVersionManager::get_newest_version(), bloom_filter, OLAP_FIELD_TYPE_FLOAT,
Review Comment:
the first param is column id not the version ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #13246:
URL: https://github.com/apache/doris/pull/13246#issuecomment-1279685483
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221015153413_clickbench_pr_29739.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #13246:
URL: https://github.com/apache/doris/pull/13246#issuecomment-1288603751
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 38.23 seconds
load time: 577 seconds
storage size: 17154810742 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221024161348_clickbench_pr_33024.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #13246:
URL: https://github.com/apache/doris/pull/13246#issuecomment-1292016994
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 39.52 seconds
load time: 569 seconds
storage size: 17154655375 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221026211419_clickbench_pr_34353.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
Posted by GitBox <gi...@apache.org>.
HappenLee commented on code in PR #13246:
URL: https://github.com/apache/doris/pull/13246#discussion_r1007228697
##########
be/src/exprs/runtime_filter.cpp:
##########
@@ -452,14 +457,29 @@ class RuntimePredicateWrapper {
<< "Can not change to bloom filter because of runtime filter type is "
<< to_string(_filter_type);
_is_bloomfilter = true;
+ insert_to_bloom_filter(_bloomfilter_func.get());
+ // release in filter
+ _hybrid_set.reset(create_set(_column_return_type));
+ }
+
+ void insert_to_bloom_filter(BloomFilterFuncBase* bloom_filter) const {
if (_hybrid_set->size() > 0) {
+ bool use_batch = _state->enable_vectorized_exec() &&
Review Comment:
use_batch seems can be set in constructor. maybe better `_use_batch` or tmeplate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13246:
URL: https://github.com/apache/doris/pull/13246#issuecomment-1294370938
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13246: [Feature](runtime-filter) add runtime filter breaking change adapt
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #13246:
URL: https://github.com/apache/doris/pull/13246#issuecomment-1293130089
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 39.04 seconds
load time: 559 seconds
storage size: 17154644085 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221027075027_clickbench_pr_34650.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org