You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "BiteTheDDDDt (via GitHub)" <gi...@apache.org> on 2024/04/10 12:04:49 UTC
[PR] [WIP](top-n) support topn filter on vscan node [doris]
BiteTheDDDDt opened a new pull request, #33496:
URL: https://github.com/apache/doris/pull/33496
## Proposed changes
support topn filter on vscan node
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2063329292
TeamCity be ut coverage result:
Function Coverage: 35.40% (8912/25172)
Line Coverage: 27.12% (73229/270033)
Region Coverage: 26.25% (37829/144119)
Branch Coverage: 23.06% (19282/83600)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4e56e96c2cc4c9ae35966758a3853a007fbe4dbc_4e56e96c2cc4c9ae35966758a3853a007fbe4dbc/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2056543842
<details>
<summary>TPC-H: <b>Total hot run time: 38958 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 320b32406b467581e449ed527447f33f6cc42fb1, data reload: false
------ Round 1 ----------------------------------
q1 17694 4591 4243 4243
q2 2027 192 185 185
q3 10896 1304 1203 1203
q4 10622 894 830 830
q5 7585 2766 2699 2699
q6 224 137 135 135
q7 1039 655 623 623
q8 9475 2121 2088 2088
q9 8102 6755 6698 6698
q10 8735 3619 3520 3520
q11 466 238 237 237
q12 425 220 220 220
q13 18654 2904 2965 2904
q14 277 230 238 230
q15 527 501 475 475
q16 522 406 387 387
q17 994 698 689 689
q18 7447 6767 6807 6767
q19 6661 1526 1496 1496
q20 697 340 325 325
q21 3642 2707 2749 2707
q22 361 297 305 297
Total cold run time: 117072 ms
Total hot run time: 38958 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4351 4225 4217 4217
q2 358 259 280 259
q3 2928 2801 2776 2776
q4 1851 1587 1569 1569
q5 5322 5367 5366 5366
q6 207 126 127 126
q7 2251 1908 1872 1872
q8 3205 3421 3361 3361
q9 8605 8555 8626 8555
q10 3910 3688 3670 3670
q11 591 473 486 473
q12 753 594 613 594
q13 17723 2938 2925 2925
q14 315 267 262 262
q15 504 472 463 463
q16 474 432 419 419
q17 1798 1525 1459 1459
q18 7558 7623 7366 7366
q19 1650 1501 1565 1501
q20 1958 1726 1760 1726
q21 4975 4838 4762 4762
q22 548 442 454 442
Total cold run time: 71835 ms
Total hot run time: 54163 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2063390111
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #33496:
URL: https://github.com/apache/doris/pull/33496#discussion_r1560817526
##########
be/src/pipeline/exec/es_scan_operator.cpp:
##########
@@ -53,8 +53,8 @@ Status EsScanLocalState::_init_profile() {
return Status::OK();
}
-Status EsScanLocalState::_process_conjuncts() {
- RETURN_IF_ERROR(Base::_process_conjuncts());
+Status EsScanLocalState::_process_conjuncts(RuntimeState* state) {
Review Comment:
warning: method '_process_conjuncts' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/exec/es_scan_operator.h:53:
```diff
- Status _process_conjuncts(RuntimeState* state) override;
+ static Status _process_conjuncts(RuntimeState* state) override;
```
##########
be/src/pipeline/exec/scan_operator.cpp:
##########
@@ -163,7 +160,7 @@ Status ScanLocalState<Derived>::open(RuntimeState* state) {
}
template <typename Derived>
-Status ScanLocalState<Derived>::_normalize_conjuncts() {
+Status ScanLocalState<Derived>::_normalize_conjuncts(RuntimeState* state) {
Review Comment:
warning: function '_normalize_conjuncts' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status ScanLocalState<Derived>::_normalize_conjuncts(RuntimeState* state) {
^
```
<details>
<summary>Additional context</summary>
**be/src/pipeline/exec/scan_operator.cpp:162:** 98 lines including whitespace and comments (threshold 80)
```cpp
Status ScanLocalState<Derived>::_normalize_conjuncts(RuntimeState* state) {
^
```
</details>
##########
be/src/vec/exprs/vtopn_pred.h:
##########
@@ -0,0 +1,105 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <gen_cpp/types.pb.h>
Review Comment:
warning: 'gen_cpp/types.pb.h' file not found [clang-diagnostic-error]
```cpp
#include <gen_cpp/types.pb.h>
^
```
##########
be/src/pipeline/exec/file_scan_operator.cpp:
##########
@@ -105,8 +105,8 @@ Status FileScanLocalState::init(RuntimeState* state, LocalStateInfo& info) {
return Status::OK();
}
-Status FileScanLocalState::_process_conjuncts() {
- RETURN_IF_ERROR(ScanLocalState<FileScanLocalState>::_process_conjuncts());
+Status FileScanLocalState::_process_conjuncts(RuntimeState* state) {
Review Comment:
warning: method '_process_conjuncts' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/exec/file_scan_operator.h:51:
```diff
- Status _process_conjuncts(RuntimeState* state) override;
+ static Status _process_conjuncts(RuntimeState* state) override;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #33496:
URL: https://github.com/apache/doris/pull/33496#discussion_r1560619746
##########
be/src/vec/exprs/vtopn_value.h:
##########
@@ -0,0 +1,66 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <gen_cpp/types.pb.h>
Review Comment:
warning: 'gen_cpp/types.pb.h' file not found [clang-diagnostic-error]
```cpp
#include <gen_cpp/types.pb.h>
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2056435512
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2057195313
<details>
<summary>TPC-H: <b>Total hot run time: 38541 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fb743102dbfe334a173695e4685efba340801ab8, data reload: false
------ Round 1 ----------------------------------
q1 17621 4316 4231 4231
q2 2016 192 197 192
q3 10434 1194 1148 1148
q4 10185 771 733 733
q5 7508 2714 2638 2638
q6 217 133 130 130
q7 1034 599 578 578
q8 9223 2080 2054 2054
q9 7314 6586 6547 6547
q10 8603 3565 3518 3518
q11 465 237 228 228
q12 495 216 208 208
q13 18681 2916 2953 2916
q14 273 221 232 221
q15 531 481 507 481
q16 545 401 404 401
q17 968 757 685 685
q18 7424 6814 6767 6767
q19 7168 1528 1497 1497
q20 655 323 305 305
q21 3555 2769 2838 2769
q22 360 294 295 294
Total cold run time: 115275 ms
Total hot run time: 38541 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4345 4171 4185 4171
q2 374 264 267 264
q3 2965 2739 2735 2735
q4 1858 1634 1617 1617
q5 5366 5338 5335 5335
q6 212 125 120 120
q7 2200 1880 1873 1873
q8 3187 3357 3378 3357
q9 8619 8552 8702 8552
q10 4056 3899 3946 3899
q11 583 505 489 489
q12 835 621 621 621
q13 16514 3261 3124 3124
q14 337 294 294 294
q15 510 483 496 483
q16 520 464 473 464
q17 1822 1527 1473 1473
q18 8095 7813 7817 7813
q19 1667 1579 1603 1579
q20 2107 1874 1859 1859
q21 8136 4937 4900 4900
q22 544 479 464 464
Total cold run time: 74852 ms
Total hot run time: 55486 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2051865027
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #33496:
URL: https://github.com/apache/doris/pull/33496#discussion_r1576137765
##########
be/src/pipeline/exec/scan_operator.cpp:
##########
@@ -1300,6 +1301,27 @@ Status ScanLocalState<Derived>::_init_profile() {
return Status::OK();
}
+template <typename Derived>
+Status ScanLocalState<Derived>::_get_topn_filters(RuntimeState* state) {
+ for (auto id : get_topn_filter_source_node_ids()) {
+ const auto& pred = state->get_query_ctx()->get_runtime_predicate(id);
+ if (!pred.inited()) {
Review Comment:
DCHECK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #33496:
URL: https://github.com/apache/doris/pull/33496#discussion_r1575684545
##########
be/src/vec/exprs/vtopn_pred.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <gen_cpp/types.pb.h>
+
+#include "runtime/query_context.h"
+#include "runtime/runtime_predicate.h"
+#include "runtime/runtime_state.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr.h"
+#include "vec/exprs/vslot_ref.h"
+#include "vec/functions/simple_function_factory.h"
+#include "vec/utils/util.hpp"
+
+namespace doris::vectorized {
+
+// only used for dynamic topn filter
+class VTopNPred : public VExpr {
+ ENABLE_FACTORY_CREATOR(VTopNPred);
+
+public:
+ VTopNPred(const TExprNode& node, int source_node_id)
+ : VExpr(node),
+ _source_node_id(source_node_id),
+ _expr_name(fmt::format("VTopNPred(source_node_id={})", _source_node_id)) {}
+
+ static Status create_vtopn_pred(SlotDescriptor* slot_desc, int source_node_id,
+ vectorized::VExprSPtr& expr) {
+ TExprNode node;
+ node.__set_node_type(TExprNodeType::FUNCTION_CALL);
+ node.__set_type(create_type_desc(PrimitiveType::TYPE_BOOLEAN));
+ node.__set_is_nullable(slot_desc->is_nullable());
+ expr = vectorized::VTopNPred::create_shared(node, source_node_id);
+
+ expr->add_child(VSlotRef::create_shared(slot_desc));
+
+ return Status::OK();
+ }
+
+ Status prepare(RuntimeState* state, const RowDescriptor& desc, VExprContext* context) override {
+ _predicate = &state->get_query_ctx()->get_runtime_predicate(_source_node_id);
+ RETURN_IF_ERROR_OR_PREPARED(VExpr::prepare(state, desc, context));
+
+ ColumnsWithTypeAndName argument_template;
+ argument_template.emplace_back(nullptr, _children[0]->data_type(),
+ _children[0]->expr_name());
+ argument_template.emplace_back(nullptr, _children[0]->data_type(), "topn value");
+
+ _function = SimpleFunctionFactory::instance().get_function(
+ _predicate->is_asc() ? "le" : "ge", argument_template, _data_type,
+ state->be_exec_version());
+ if (!_function) {
+ return Status::InternalError("get function failed");
+ }
+ return Status::OK();
+ }
+
+ Status execute(VExprContext* context, Block* block, int* result_column_id) override {
+ if (!_predicate->has_value()) {
+ block->insert({create_always_true_column(block->rows(), _data_type->is_nullable()),
+ _data_type, _expr_name});
+ *result_column_id = block->columns() - 1;
+ return Status::OK();
+ }
+
+ Field field = _predicate->get_value();
+ auto column_ptr = _children[0]->data_type()->create_column_const(1, field);
+ size_t row_size = std::max(block->rows(), column_ptr->size());
+ int topn_value_id = VExpr::insert_param(
+ block, {column_ptr, _children[0]->data_type(), _expr_name}, row_size);
+
+ int slot_id = -1;
+ RETURN_IF_ERROR(_children[0]->execute(context, block, &slot_id));
+
+ std::vector<size_t> arguments = {(size_t)slot_id, (size_t)topn_value_id};
+
+ size_t num_columns_without_result = block->columns();
+ block->insert({nullptr, _data_type, _expr_name});
+ RETURN_IF_ERROR(_function->execute(nullptr, *block, arguments, num_columns_without_result,
+ block->rows(), false));
+ *result_column_id = num_columns_without_result;
+
+ if (is_nullable()) {
Review Comment:
maybe only null first need always not filter ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2065776282
TeamCity be ut coverage result:
Function Coverage: 35.39% (8918/25199)
Line Coverage: 27.14% (73281/270044)
Region Coverage: 26.24% (37823/144130)
Branch Coverage: 23.07% (19283/83568)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4e56e96c2cc4c9ae35966758a3853a007fbe4dbc_4e56e96c2cc4c9ae35966758a3853a007fbe4dbc/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2058238092
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2058342558
TeamCity be ut coverage result:
Function Coverage: 35.53% (8902/25058)
Line Coverage: 27.27% (73116/268136)
Region Coverage: 26.40% (37814/143259)
Branch Coverage: 23.18% (19270/83122)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e2bb682446cccef472b333cff330fe1f77019b95_e2bb682446cccef472b333cff330fe1f77019b95/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2066153503
<details>
<summary>TPC-H: <b>Total hot run time: 39870 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4e56e96c2cc4c9ae35966758a3853a007fbe4dbc, data reload: false
------ Round 1 ----------------------------------
q1 17623 5200 4259 4259
q2 2024 193 190 190
q3 10470 1154 1203 1154
q4 10218 748 820 748
q5 7523 2791 2627 2627
q6 219 134 136 134
q7 998 600 587 587
q8 9224 2078 2055 2055
q9 9099 6655 6585 6585
q10 8851 3747 3728 3728
q11 462 233 226 226
q12 454 225 217 217
q13 17767 2945 2934 2934
q14 264 227 241 227
q15 526 484 478 478
q16 521 379 376 376
q17 971 715 651 651
q18 8262 7540 7264 7264
q19 6368 1491 1507 1491
q20 686 318 309 309
q21 5007 3323 4033 3323
q22 371 307 333 307
Total cold run time: 117908 ms
Total hot run time: 39870 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4327 4208 4271 4208
q2 383 269 257 257
q3 2991 2757 2785 2757
q4 1901 1570 1607 1570
q5 5311 5300 5296 5296
q6 210 121 126 121
q7 2275 1829 1903 1829
q8 3243 3383 3398 3383
q9 8629 8747 8903 8747
q10 4147 3912 3936 3912
q11 608 513 504 504
q12 862 637 655 637
q13 16004 3221 3187 3187
q14 323 278 299 278
q15 511 495 477 477
q16 502 459 464 459
q17 1810 1523 1496 1496
q18 8220 7917 7666 7666
q19 1712 1558 1597 1558
q20 2043 1822 1819 1819
q21 7107 5107 4951 4951
q22 550 480 485 480
Total cold run time: 73669 ms
Total hot run time: 55592 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2062980971
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2062983802
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2068890652
<details>
<summary>TPC-H: <b>Total hot run time: 40008 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 11ac8a55c98395f3712489fed29383234acc3a71, data reload: false
------ Round 1 ----------------------------------
q1 17601 4578 4262 4262
q2 2016 196 191 191
q3 10441 1122 1183 1122
q4 10192 776 729 729
q5 7517 2736 2704 2704
q6 218 133 137 133
q7 1034 626 604 604
q8 9223 2087 2065 2065
q9 9067 6656 6575 6575
q10 8695 3760 3749 3749
q11 456 244 253 244
q12 523 225 221 221
q13 17757 2979 2982 2979
q14 266 231 240 231
q15 515 478 474 474
q16 529 382 388 382
q17 983 641 629 629
q18 7997 7481 7373 7373
q19 6475 1551 1521 1521
q20 652 316 305 305
q21 5093 3213 3322 3213
q22 369 309 302 302
Total cold run time: 117619 ms
Total hot run time: 40008 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4347 4231 4232 4231
q2 376 257 270 257
q3 3068 2841 2820 2820
q4 1912 1589 1612 1589
q5 5396 5374 5336 5336
q6 209 125 127 125
q7 2244 1845 1824 1824
q8 3236 3387 3346 3346
q9 8696 8673 8941 8673
q10 4102 4042 3871 3871
q11 613 497 501 497
q12 838 649 679 649
q13 16799 3160 3193 3160
q14 337 271 291 271
q15 514 482 494 482
q16 491 458 453 453
q17 1837 1575 1533 1533
q18 8264 8023 7665 7665
q19 1709 1615 1512 1512
q20 2070 1871 1848 1848
q21 5140 5025 5044 5025
q22 560 475 446 446
Total cold run time: 72758 ms
Total hot run time: 55613 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2056563758
<details>
<summary>TPC-DS: <b>Total hot run time: 183473 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 320b32406b467581e449ed527447f33f6cc42fb1, data reload: false
query1 898 1118 1103 1103
query2 7970 2622 2547 2547
query3 6672 200 205 200
query4 37437 21582 21258 21258
query5 4147 394 405 394
query6 227 180 189 180
query7 4042 286 297 286
query8 220 174 166 166
query9 5776 2296 2289 2289
query10 547 258 257 257
query11 14624 14229 14127 14127
query12 161 93 89 89
query13 991 348 343 343
query14 9129 6796 6733 6733
query15 203 172 180 172
query16 7196 250 254 250
query17 1697 564 566 564
query18 1513 275 272 272
query19 196 146 153 146
query20 95 88 88 88
query21 199 126 127 126
query22 4959 4780 4810 4780
query23 33701 33203 33156 33156
query24 12792 2870 2863 2863
query25 548 365 374 365
query26 1862 148 149 148
query27 3101 306 308 306
query28 7722 2032 2000 2000
query29 851 610 588 588
query30 300 145 151 145
query31 874 725 725 725
query32 87 52 52 52
query33 579 239 231 231
query34 881 471 487 471
query35 850 710 709 709
query36 1035 878 932 878
query37 276 67 68 67
query38 3332 3164 3152 3152
query39 1574 1536 1528 1528
query40 284 127 126 126
query41 44 40 40 40
query42 106 98 97 97
query43 607 534 552 534
query44 1229 714 716 714
query45 282 258 268 258
query46 1054 716 742 716
query47 1951 1859 1869 1859
query48 359 289 282 282
query49 1145 368 359 359
query50 748 374 370 370
query51 6554 6646 6599 6599
query52 100 91 86 86
query53 344 281 272 272
query54 255 227 213 213
query55 75 68 71 68
query56 245 220 217 217
query57 1192 1108 1127 1108
query58 228 202 208 202
query59 3363 3234 3178 3178
query60 252 230 233 230
query61 95 92 93 92
query62 650 438 429 429
query63 302 284 279 279
query64 4822 3993 3974 3974
query65 3139 3052 3041 3041
query66 1326 329 313 313
query67 15411 14876 14996 14876
query68 4425 540 542 540
query69 465 315 299 299
query70 1215 1185 1174 1174
query71 365 273 265 265
query72 6763 2620 2447 2447
query73 712 314 314 314
query74 6852 6518 6465 6465
query75 3426 2656 2643 2643
query76 2109 1085 1125 1085
query77 454 242 243 242
query78 10775 10276 10206 10206
query79 2540 520 512 512
query80 1911 418 417 417
query81 521 1073 214 214
query82 760 90 103 90
query83 275 173 172 172
query84 250 90 86 86
query85 1810 293 268 268
query86 503 305 298 298
query87 3523 3262 3262 3262
query88 5328 2284 2330 2284
query89 478 378 391 378
query90 2008 179 181 179
query91 128 96 99 96
query92 62 46 46 46
query93 3133 513 504 504
query94 1306 178 178 178
query95 387 299 291 291
query96 606 264 262 262
query97 3124 2914 2904 2904
query98 243 221 214 214
query99 1232 853 852 852
Total cold run time: 293115 ms
Total hot run time: 183473 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #33496:
URL: https://github.com/apache/doris/pull/33496#discussion_r1565957282
##########
be/src/pipeline/exec/scan_operator.cpp:
##########
@@ -164,7 +161,7 @@ Status ScanLocalState<Derived>::open(RuntimeState* state) {
}
template <typename Derived>
-Status ScanLocalState<Derived>::_normalize_conjuncts() {
+Status ScanLocalState<Derived>::_normalize_conjuncts(RuntimeState* state) {
Review Comment:
warning: function '_normalize_conjuncts' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status ScanLocalState<Derived>::_normalize_conjuncts(RuntimeState* state) {
^
```
<details>
<summary>Additional context</summary>
**be/src/pipeline/exec/scan_operator.cpp:163:** 98 lines including whitespace and comments (threshold 80)
```cpp
Status ScanLocalState<Derived>::_normalize_conjuncts(RuntimeState* state) {
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2052001591
<details>
<summary>TPC-H: <b>Total hot run time: 38584 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 320b32406b467581e449ed527447f33f6cc42fb1, data reload: false
------ Round 1 ----------------------------------
q1 17604 4416 4297 4297
q2 2023 188 181 181
q3 10472 1136 1202 1136
q4 10193 755 766 755
q5 7534 2759 2658 2658
q6 211 128 129 128
q7 1008 599 584 584
q8 9213 2063 2036 2036
q9 8002 6618 6542 6542
q10 8530 3571 3536 3536
q11 462 236 232 232
q12 503 217 209 209
q13 19077 2930 2987 2930
q14 280 227 234 227
q15 525 468 486 468
q16 518 372 375 372
q17 965 737 667 667
q18 7465 6966 6709 6709
q19 8056 1513 1495 1495
q20 684 305 301 301
q21 3510 2817 2833 2817
q22 362 304 304 304
Total cold run time: 117197 ms
Total hot run time: 38584 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4372 4229 4196 4196
q2 367 261 247 247
q3 2966 2729 2762 2729
q4 1866 1542 1628 1542
q5 5390 5380 5300 5300
q6 205 120 118 118
q7 2215 1876 1864 1864
q8 3186 3375 3307 3307
q9 8583 8595 8817 8595
q10 4082 3934 3927 3927
q11 616 511 504 504
q12 794 615 651 615
q13 17356 3145 3035 3035
q14 353 288 275 275
q15 509 482 485 482
q16 513 455 466 455
q17 1860 1524 1493 1493
q18 8192 7912 7791 7791
q19 1674 1575 1608 1575
q20 2047 1877 1818 1818
q21 6044 4867 4883 4867
q22 549 465 482 465
Total cold run time: 73739 ms
Total hot run time: 55200 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2066174536
<details>
<summary>TPC-DS: <b>Total hot run time: 188000 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4e56e96c2cc4c9ae35966758a3853a007fbe4dbc, data reload: false
query1 910 358 338 338
query2 6189 2713 2412 2412
query3 6649 214 209 209
query4 23868 21324 21149 21149
query5 3916 403 407 403
query6 275 181 176 176
query7 4586 301 281 281
query8 238 184 184 184
query9 8593 2321 2281 2281
query10 417 254 244 244
query11 14730 14195 14221 14195
query12 135 89 86 86
query13 1630 359 352 352
query14 9720 7405 7065 7065
query15 226 185 193 185
query16 8015 265 265 265
query17 1935 595 566 566
query18 2059 290 282 282
query19 200 151 161 151
query20 96 89 90 89
query21 200 128 132 128
query22 5084 4830 4891 4830
query23 33866 33074 33260 33074
query24 11272 3055 3126 3055
query25 658 411 407 407
query26 741 165 154 154
query27 2451 384 363 363
query28 5758 2142 2071 2071
query29 919 632 636 632
query30 243 161 155 155
query31 1031 746 772 746
query32 92 57 57 57
query33 679 263 257 257
query34 959 495 520 495
query35 864 706 715 706
query36 1108 947 913 913
query37 105 66 64 64
query38 3432 3284 3323 3284
query39 1614 1603 1596 1596
query40 180 121 125 121
query41 38 38 36 36
query42 107 103 97 97
query43 594 556 575 556
query44 1112 741 736 736
query45 282 273 273 273
query46 1114 754 768 754
query47 2026 1996 1966 1966
query48 388 307 305 305
query49 848 392 390 390
query50 772 398 399 398
query51 6936 6990 6841 6841
query52 106 87 95 87
query53 347 271 274 271
query54 298 219 222 219
query55 75 69 71 69
query56 237 215 222 215
query57 1246 1131 1169 1131
query58 216 196 193 193
query59 3400 3275 3186 3186
query60 271 228 224 224
query61 107 85 86 85
query62 593 456 445 445
query63 300 277 272 272
query64 8512 7134 7054 7054
query65 3064 3033 3044 3033
query66 748 329 327 327
query67 15422 15236 15030 15030
query68 6908 533 542 533
query69 532 332 302 302
query70 1258 1132 1088 1088
query71 1467 1257 1259 1257
query72 7859 2656 2422 2422
query73 740 318 323 318
query74 6861 6533 6340 6340
query75 3848 2662 2644 2644
query76 4676 940 1002 940
query77 637 261 259 259
query78 11075 10304 10127 10127
query79 10309 526 538 526
query80 1657 446 430 430
query81 519 213 219 213
query82 842 91 92 91
query83 202 166 166 166
query84 272 84 79 79
query85 1314 264 256 256
query86 476 294 297 294
query87 3567 3265 3362 3265
query88 5234 2328 2334 2328
query89 530 379 374 374
query90 1998 180 179 179
query91 127 96 97 96
query92 57 45 44 44
query93 7054 515 523 515
query94 1118 182 178 178
query95 405 306 299 299
query96 611 260 263 260
query97 3137 2930 2913 2913
query98 241 214 219 214
query99 1263 836 889 836
Total cold run time: 301596 ms
Total hot run time: 188000 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2073901161
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2057077672
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2057219965
<details>
<summary>TPC-DS: <b>Total hot run time: 184778 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fb743102dbfe334a173695e4685efba340801ab8, data reload: false
query1 893 354 343 343
query2 6757 2632 2556 2556
query3 6669 198 197 197
query4 23670 21296 21358 21296
query5 4162 398 407 398
query6 280 175 165 165
query7 4572 280 280 280
query8 219 167 170 167
query9 8402 2300 2284 2284
query10 416 247 248 247
query11 14728 14319 14191 14191
query12 138 94 86 86
query13 1636 365 353 353
query14 10304 7969 7898 7898
query15 285 183 186 183
query16 8222 263 251 251
query17 2085 602 561 561
query18 2098 268 264 264
query19 331 148 146 146
query20 88 84 84 84
query21 194 131 125 125
query22 4932 4781 4812 4781
query23 34009 33194 33355 33194
query24 10976 3175 3008 3008
query25 619 382 380 380
query26 1183 161 149 149
query27 2576 370 364 364
query28 7414 2080 2083 2080
query29 873 615 603 603
query30 262 156 155 155
query31 1014 758 768 758
query32 88 55 57 55
query33 757 259 255 255
query34 1152 497 494 494
query35 876 736 710 710
query36 1104 914 931 914
query37 130 71 67 67
query38 3439 3323 3378 3323
query39 1646 1570 1605 1570
query40 178 125 127 125
query41 40 39 41 39
query42 103 98 94 94
query43 579 542 563 542
query44 1247 741 746 741
query45 275 286 260 260
query46 1107 763 742 742
query47 2019 1928 1960 1928
query48 377 314 289 289
query49 863 387 377 377
query50 786 385 386 385
query51 6873 6729 6831 6729
query52 109 87 89 87
query53 341 278 269 269
query54 303 223 221 221
query55 74 69 67 67
query56 237 218 221 218
query57 1223 1097 1109 1097
query58 241 202 197 197
query59 3463 3361 3260 3260
query60 262 229 229 229
query61 88 87 89 87
query62 585 442 450 442
query63 306 273 277 273
query64 5139 4066 3928 3928
query65 3169 3026 3078 3026
query66 775 346 336 336
query67 15505 15016 14863 14863
query68 5138 520 532 520
query69 503 309 299 299
query70 1161 1114 1138 1114
query71 395 276 285 276
query72 6716 2735 2623 2623
query73 710 310 311 310
query74 6780 6523 6368 6368
query75 3384 2594 2588 2588
query76 2891 928 1000 928
query77 589 268 262 262
query78 10991 10194 10346 10194
query79 9153 514 508 508
query80 2117 440 440 440
query81 540 214 222 214
query82 1508 92 97 92
query83 311 163 179 163
query84 263 83 80 80
query85 1875 263 320 263
query86 457 299 288 288
query87 3470 3299 3300 3299
query88 5194 2307 2317 2307
query89 567 379 367 367
query90 1895 180 179 179
query91 123 93 97 93
query92 60 46 46 46
query93 6943 500 490 490
query94 1109 178 175 175
query95 381 292 304 292
query96 601 264 258 258
query97 3161 2929 2928 2928
query98 228 217 225 217
query99 1277 838 811 811
Total cold run time: 295957 ms
Total hot run time: 184778 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2071765267
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2066001331
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2068931027
TeamCity be ut coverage result:
Function Coverage: 35.34% (8915/25224)
Line Coverage: 27.09% (73312/270672)
Region Coverage: 26.22% (37867/144397)
Branch Coverage: 23.04% (19286/83698)
Coverage Report: http://coverage.selectdb-in.cc/coverage/11ac8a55c98395f3712489fed29383234acc3a71_11ac8a55c98395f3712489fed29383234acc3a71/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee merged PR #33496:
URL: https://github.com/apache/doris/pull/33496
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2066173720
TeamCity be ut coverage result:
Function Coverage: 35.40% (8922/25204)
Line Coverage: 27.13% (73332/270316)
Region Coverage: 26.26% (37864/144215)
Branch Coverage: 23.08% (19296/83602)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4e56e96c2cc4c9ae35966758a3853a007fbe4dbc_4e56e96c2cc4c9ae35966758a3853a007fbe4dbc/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2063077646
TeamCity be ut coverage result:
Function Coverage: 35.41% (8910/25165)
Line Coverage: 27.11% (73172/269926)
Region Coverage: 26.26% (37837/144085)
Branch Coverage: 23.07% (19280/83576)
Coverage Report: http://coverage.selectdb-in.cc/coverage/094100e686b182c04ae12fb74304246cd23a161d_094100e686b182c04ae12fb74304246cd23a161d/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2068915071
<details>
<summary>TPC-DS: <b>Total hot run time: 188907 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 11ac8a55c98395f3712489fed29383234acc3a71, data reload: false
query1 900 360 349 349
query2 6335 2627 2388 2388
query3 6662 210 200 200
query4 22542 21284 21318 21284
query5 4002 415 413 413
query6 265 188 179 179
query7 4581 314 289 289
query8 250 186 185 185
query9 8439 2346 2309 2309
query10 423 258 252 252
query11 14804 14158 14187 14158
query12 135 91 84 84
query13 1620 360 354 354
query14 10600 7769 8489 7769
query15 258 188 188 188
query16 8199 272 269 269
query17 1919 639 566 566
query18 2107 285 288 285
query19 341 163 160 160
query20 95 92 89 89
query21 203 135 139 135
query22 5054 4830 4770 4770
query23 33820 33208 33434 33208
query24 10938 3092 3197 3092
query25 627 407 422 407
query26 768 166 166 166
query27 2212 385 384 384
query28 5897 2092 2034 2034
query29 873 636 632 632
query30 217 154 162 154
query31 970 751 790 751
query32 92 51 55 51
query33 748 281 261 261
query34 1126 509 509 509
query35 859 735 730 730
query36 1092 948 923 923
query37 105 66 67 66
query38 3444 3343 3323 3323
query39 1638 1560 1593 1560
query40 174 132 131 131
query41 42 38 37 37
query42 105 98 101 98
query43 588 560 552 552
query44 1246 763 745 745
query45 295 256 266 256
query46 1133 770 753 753
query47 2039 1972 1933 1933
query48 384 297 294 294
query49 857 400 412 400
query50 780 408 404 404
query51 6949 6726 6768 6726
query52 105 86 98 86
query53 352 280 282 280
query54 316 235 228 228
query55 76 70 70 70
query56 239 229 225 225
query57 1206 1118 1113 1113
query58 215 198 195 195
query59 3468 3254 3050 3050
query60 256 232 231 231
query61 88 85 85 85
query62 597 451 440 440
query63 305 282 278 278
query64 8134 7239 7166 7166
query65 3105 3061 3045 3045
query66 745 326 337 326
query67 15287 14887 14841 14841
query68 5189 532 529 529
query69 463 304 303 303
query70 1213 1129 1205 1129
query71 1405 1271 1271 1271
query72 7724 2646 2431 2431
query73 706 317 319 317
query74 6766 6412 6379 6379
query75 3382 2702 2621 2621
query76 3354 1049 978 978
query77 407 260 268 260
query78 10852 10210 10215 10210
query79 5086 531 515 515
query80 1905 451 454 451
query81 519 225 220 220
query82 1141 97 101 97
query83 309 172 167 167
query84 267 91 84 84
query85 1727 268 270 268
query86 496 296 307 296
query87 3498 3218 3243 3218
query88 4643 2317 2305 2305
query89 495 381 379 379
query90 1923 185 185 185
query91 127 98 98 98
query92 63 47 48 47
query93 5493 504 514 504
query94 1069 182 175 175
query95 395 305 354 305
query96 591 260 253 253
query97 3148 2917 2935 2917
query98 235 212 216 212
query99 1256 873 869 869
Total cold run time: 290093 ms
Total hot run time: 188907 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2047369070
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #33496:
URL: https://github.com/apache/doris/pull/33496#discussion_r1575682167
##########
be/src/vec/exprs/vtopn_pred.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <gen_cpp/types.pb.h>
+
+#include "runtime/query_context.h"
+#include "runtime/runtime_predicate.h"
+#include "runtime/runtime_state.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr.h"
+#include "vec/exprs/vslot_ref.h"
+#include "vec/functions/simple_function_factory.h"
+#include "vec/utils/util.hpp"
+
+namespace doris::vectorized {
+
+// only used for dynamic topn filter
+class VTopNPred : public VExpr {
+ ENABLE_FACTORY_CREATOR(VTopNPred);
+
+public:
+ VTopNPred(const TExprNode& node, int source_node_id)
+ : VExpr(node),
+ _source_node_id(source_node_id),
+ _expr_name(fmt::format("VTopNPred(source_node_id={})", _source_node_id)) {}
+
+ static Status create_vtopn_pred(SlotDescriptor* slot_desc, int source_node_id,
+ vectorized::VExprSPtr& expr) {
+ TExprNode node;
+ node.__set_node_type(TExprNodeType::FUNCTION_CALL);
+ node.__set_type(create_type_desc(PrimitiveType::TYPE_BOOLEAN));
+ node.__set_is_nullable(slot_desc->is_nullable());
+ expr = vectorized::VTopNPred::create_shared(node, source_node_id);
+
+ expr->add_child(VSlotRef::create_shared(slot_desc));
Review Comment:
think about the logic? maybe here not need be a slot desc? can execute a func? ADD A TODO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #33496:
URL: https://github.com/apache/doris/pull/33496#discussion_r1575682167
##########
be/src/vec/exprs/vtopn_pred.h:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <gen_cpp/types.pb.h>
+
+#include "runtime/query_context.h"
+#include "runtime/runtime_predicate.h"
+#include "runtime/runtime_state.h"
+#include "vec/columns/columns_number.h"
+#include "vec/data_types/data_type.h"
+#include "vec/exprs/vexpr.h"
+#include "vec/exprs/vslot_ref.h"
+#include "vec/functions/simple_function_factory.h"
+#include "vec/utils/util.hpp"
+
+namespace doris::vectorized {
+
+// only used for dynamic topn filter
+class VTopNPred : public VExpr {
+ ENABLE_FACTORY_CREATOR(VTopNPred);
+
+public:
+ VTopNPred(const TExprNode& node, int source_node_id)
+ : VExpr(node),
+ _source_node_id(source_node_id),
+ _expr_name(fmt::format("VTopNPred(source_node_id={})", _source_node_id)) {}
+
+ static Status create_vtopn_pred(SlotDescriptor* slot_desc, int source_node_id,
+ vectorized::VExprSPtr& expr) {
+ TExprNode node;
+ node.__set_node_type(TExprNodeType::FUNCTION_CALL);
+ node.__set_type(create_type_desc(PrimitiveType::TYPE_BOOLEAN));
+ node.__set_is_nullable(slot_desc->is_nullable());
+ expr = vectorized::VTopNPred::create_shared(node, source_node_id);
+
+ expr->add_child(VSlotRef::create_shared(slot_desc));
Review Comment:
think about the logic? maybe here not need be a slot desc? can execute a func?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2052020446
<details>
<summary>TPC-DS: <b>Total hot run time: 184173 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 320b32406b467581e449ed527447f33f6cc42fb1, data reload: false
query1 878 1116 1115 1115
query2 7125 2597 2494 2494
query3 6664 203 201 201
query4 37150 21713 21405 21405
query5 4143 399 386 386
query6 229 181 173 173
query7 4053 277 276 276
query8 218 169 168 168
query9 5755 2282 2278 2278
query10 374 245 249 245
query11 14565 14280 14285 14280
query12 134 94 87 87
query13 994 358 356 356
query14 9078 6967 6678 6678
query15 192 169 172 169
query16 7203 263 247 247
query17 1679 566 538 538
query18 1488 293 268 268
query19 196 161 147 147
query20 94 86 87 86
query21 194 130 123 123
query22 5075 4861 4896 4861
query23 33739 33140 33468 33140
query24 11351 3068 3005 3005
query25 507 391 384 384
query26 1431 152 152 152
query27 2954 362 368 362
query28 7574 2118 2077 2077
query29 864 615 620 615
query30 265 159 154 154
query31 946 764 743 743
query32 60 51 52 51
query33 589 270 243 243
query34 1017 500 506 500
query35 844 717 725 717
query36 1070 963 915 915
query37 107 63 65 63
query38 3696 3564 3632 3564
query39 1630 1582 1542 1542
query40 175 126 130 126
query41 42 40 44 40
query42 104 94 96 94
query43 588 557 563 557
query44 1413 717 720 717
query45 292 270 285 270
query46 1066 732 746 732
query47 2071 1999 1971 1971
query48 377 311 306 306
query49 815 362 364 362
query50 769 390 385 385
query51 6878 6648 6688 6648
query52 95 95 87 87
query53 345 272 277 272
query54 253 220 223 220
query55 75 68 69 68
query56 233 213 214 213
query57 1194 1165 1155 1155
query58 232 209 204 204
query59 3380 3116 3273 3116
query60 251 238 238 238
query61 111 108 107 107
query62 626 455 423 423
query63 305 280 276 276
query64 4044 3943 4018 3943
query65 3050 3051 3024 3024
query66 734 324 330 324
query67 15728 14900 14744 14744
query68 7087 527 536 527
query69 580 305 308 305
query70 1224 1195 1199 1195
query71 470 273 269 269
query72 6779 2780 2634 2634
query73 813 317 311 311
query74 7131 6436 6420 6420
query75 3133 2361 2357 2357
query76 4044 1111 1149 1111
query77 630 246 245 245
query78 11062 10191 10097 10097
query79 8416 510 508 508
query80 2506 427 423 423
query81 515 215 212 212
query82 1542 90 91 90
query83 306 166 168 166
query84 266 84 81 81
query85 1089 266 309 266
query86 453 289 309 289
query87 3709 3548 3597 3548
query88 6016 2351 2279 2279
query89 540 371 364 364
query90 1871 179 176 176
query91 122 93 97 93
query92 64 46 46 46
query93 6943 497 498 497
query94 1048 177 180 177
query95 368 288 288 288
query96 618 268 259 259
query97 2665 2454 2488 2454
query98 234 216 214 214
query99 1202 866 842 842
Total cold run time: 304817 ms
Total hot run time: 184173 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2056611251
TeamCity be ut coverage result:
Function Coverage: 35.55% (8906/25054)
Line Coverage: 27.28% (73133/268057)
Region Coverage: 26.42% (37823/143170)
Branch Coverage: 23.18% (19269/83130)
Coverage Report: http://coverage.selectdb-in.cc/coverage/320b32406b467581e449ed527447f33f6cc42fb1_320b32406b467581e449ed527447f33f6cc42fb1/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2057238416
TeamCity be ut coverage result:
Function Coverage: 35.53% (8905/25062)
Line Coverage: 27.27% (73132/268197)
Region Coverage: 26.39% (37812/143306)
Branch Coverage: 23.17% (19268/83152)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fb743102dbfe334a173695e4685efba340801ab8_fb743102dbfe334a173695e4685efba340801ab8/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2058325622
<details>
<summary>TPC-DS: <b>Total hot run time: 188789 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e2bb682446cccef472b333cff330fe1f77019b95, data reload: false
query1 897 367 359 359
query2 6226 2709 2459 2459
query3 6664 212 207 207
query4 23737 21289 21227 21227
query5 4130 427 405 405
query6 267 179 201 179
query7 4586 290 288 288
query8 238 175 177 175
query9 8766 2327 2298 2298
query10 417 268 264 264
query11 15031 14253 14266 14253
query12 135 94 86 86
query13 1646 369 386 369
query14 10965 8709 8645 8645
query15 252 175 183 175
query16 8268 279 259 259
query17 1979 575 536 536
query18 2092 276 267 267
query19 333 150 150 150
query20 89 84 91 84
query21 203 122 126 122
query22 5005 4834 4813 4813
query23 33570 33270 33332 33270
query24 11093 3109 3090 3090
query25 608 386 391 386
query26 724 157 161 157
query27 2274 366 382 366
query28 5986 2089 2090 2089
query29 847 645 644 644
query30 267 164 156 156
query31 994 781 785 781
query32 99 51 50 50
query33 697 262 249 249
query34 1072 495 501 495
query35 854 728 743 728
query36 1088 915 972 915
query37 105 67 66 66
query38 3463 3369 3346 3346
query39 1120 1066 1006 1006
query40 181 129 128 128
query41 46 44 38 38
query42 105 102 101 101
query43 563 543 533 533
query44 1210 749 748 748
query45 291 263 258 258
query46 1116 758 745 745
query47 2094 2016 1949 1949
query48 379 294 308 294
query49 881 380 373 373
query50 786 393 395 393
query51 6858 6717 6757 6717
query52 102 86 90 86
query53 340 274 277 274
query54 306 228 251 228
query55 77 72 72 72
query56 235 224 221 221
query57 1193 1118 1103 1103
query58 226 200 202 200
query59 3560 3127 3342 3127
query60 292 233 240 233
query61 94 94 88 88
query62 616 443 434 434
query63 307 270 277 270
query64 8107 7212 7124 7124
query65 3071 3012 3040 3012
query66 750 328 334 328
query67 15376 14849 14938 14849
query68 5234 533 526 526
query69 486 322 311 311
query70 1253 1193 1230 1193
query71 393 275 273 273
query72 7759 2780 2575 2575
query73 705 323 324 323
query74 6813 6533 6395 6395
query75 3379 2662 2588 2588
query76 3332 974 1021 974
query77 381 273 280 273
query78 10912 10269 10408 10269
query79 8181 514 513 513
query80 1955 462 449 449
query81 543 223 219 219
query82 1348 100 96 96
query83 322 172 171 171
query84 265 90 85 85
query85 1719 316 315 315
query86 459 297 307 297
query87 3541 3280 3300 3280
query88 5271 2360 2373 2360
query89 484 372 389 372
query90 1919 187 185 185
query91 136 108 106 106
query92 62 50 53 50
query93 6313 512 495 495
query94 1099 189 188 188
query95 404 313 322 313
query96 622 269 262 262
query97 3124 2943 2942 2942
query98 232 223 223 223
query99 1221 828 871 828
Total cold run time: 295816 ms
Total hot run time: 188789 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2073901218
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2063847788
TeamCity be ut coverage result:
Function Coverage: 35.41% (8913/25172)
Line Coverage: 27.12% (73239/270036)
Region Coverage: 26.25% (37836/144119)
Branch Coverage: 23.06% (19281/83600)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4e56e96c2cc4c9ae35966758a3853a007fbe4dbc_4e56e96c2cc4c9ae35966758a3853a007fbe4dbc/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2068746236
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #33496:
URL: https://github.com/apache/doris/pull/33496#discussion_r1569954495
##########
be/src/pipeline/exec/scan_operator.cpp:
##########
@@ -164,7 +161,7 @@ Status ScanLocalState<Derived>::open(RuntimeState* state) {
}
template <typename Derived>
-Status ScanLocalState<Derived>::_normalize_conjuncts() {
+Status ScanLocalState<Derived>::_normalize_conjuncts(RuntimeState* state) {
Review Comment:
warning: function '_normalize_conjuncts' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status ScanLocalState<Derived>::_normalize_conjuncts(RuntimeState* state) {
^
```
<details>
<summary>Additional context</summary>
**be/src/pipeline/exec/scan_operator.cpp:163:** 100 lines including whitespace and comments (threshold 80)
```cpp
Status ScanLocalState<Derived>::_normalize_conjuncts(RuntimeState* state) {
^
```
</details>
##########
be/src/pipeline/exec/olap_scan_operator.h:
##########
@@ -83,6 +83,8 @@ class OlapScanLocalState final : public ScanLocalState<OlapScanLocalState> {
bool _storage_no_merge() override;
+ bool _push_down_topn() override { return true; }
Review Comment:
warning: method '_push_down_topn' can be made static [readability-convert-member-functions-to-static]
```suggestion
static bool _push_down_topn() override { return true; }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2051327945
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #33496:
URL: https://github.com/apache/doris/pull/33496#discussion_r1575616536
##########
be/src/vec/utils/util.hpp:
##########
@@ -169,6 +170,63 @@ inline std::string remove_suffix(const std::string& name, const std::string& suf
return name.substr(0, name.length() - suffix.length());
};
+inline ColumnPtr create_always_true_column(size_t size, bool is_nullable) {
+ auto res_data_column = ColumnUInt8::create(size, 1);
+ if (is_nullable) {
+ auto null_map = ColumnVector<UInt8>::create(size, 0);
+ return ColumnNullable::create(std::move(res_data_column), std::move(null_map));
+ }
+ return res_data_column;
+}
+
+// change null element to true element
+inline void change_null_to_true(ColumnPtr column, ColumnPtr argument = nullptr) {
+ if (is_column_const(*column)) {
+ change_null_to_true(assert_cast<const ColumnConst*>(column.get())->get_data_column_ptr());
+ } else if (column->is_nullable()) {
+ auto* nullable =
+ const_cast<ColumnNullable*>(assert_cast<const ColumnNullable*>(column.get()));
+ auto* __restrict data = assert_cast<ColumnUInt8*>(nullable->get_nested_column_ptr().get())
+ ->get_data()
+ .data();
+ auto* __restrict null_map = const_cast<uint8_t*>(nullable->get_null_map_data().data());
+ for (size_t i = 0; i < column->size(); ++i) {
Review Comment:
call `column->size()` in for loop cause not SIMD, not call the fun in for loop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2071944860
TeamCity be ut coverage result:
Function Coverage: 35.21% (8916/25320)
Line Coverage: 26.98% (73328/271792)
Region Coverage: 26.15% (37874/144827)
Branch Coverage: 22.97% (19289/83966)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b408e018feffb526ca39952be25746a9defb01c4_b408e018feffb526ca39952be25746a9defb01c4/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2051942411
TeamCity be ut coverage result:
Function Coverage: 35.57% (8903/25027)
Line Coverage: 27.30% (73095/267752)
Region Coverage: 26.43% (37812/143039)
Branch Coverage: 23.19% (19263/83078)
Coverage Report: http://coverage.selectdb-in.cc/coverage/320b32406b467581e449ed527447f33f6cc42fb1_320b32406b467581e449ed527447f33f6cc42fb1/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2047380804
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [WIP](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2058310865
<details>
<summary>TPC-H: <b>Total hot run time: 39973 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e2bb682446cccef472b333cff330fe1f77019b95, data reload: false
------ Round 1 ----------------------------------
q1 17599 4381 4232 4232
q2 2009 192 189 189
q3 10457 1203 1160 1160
q4 10194 736 772 736
q5 7522 2711 2646 2646
q6 215 131 133 131
q7 1023 601 584 584
q8 9219 2071 2035 2035
q9 9051 6678 6615 6615
q10 8822 3735 3695 3695
q11 484 228 231 228
q12 467 219 205 205
q13 17770 2979 2952 2952
q14 282 233 234 233
q15 522 492 485 485
q16 534 408 380 380
q17 974 657 668 657
q18 8125 7380 7411 7380
q19 5116 1555 1508 1508
q20 646 316 300 300
q21 5024 3336 3317 3317
q22 357 305 309 305
Total cold run time: 116412 ms
Total hot run time: 39973 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4445 4173 4208 4173
q2 370 272 273 272
q3 2976 2740 2791 2740
q4 1912 1583 1612 1583
q5 5387 5314 5327 5314
q6 212 122 125 122
q7 2238 1889 1873 1873
q8 3199 3376 3386 3376
q9 8605 8637 8900 8637
q10 4113 3980 3927 3927
q11 612 509 494 494
q12 817 671 676 671
q13 16943 3184 3147 3147
q14 316 285 295 285
q15 511 468 489 468
q16 510 437 447 437
q17 1814 1535 1508 1508
q18 8161 8017 7724 7724
q19 1714 1548 1522 1522
q20 2096 1828 1830 1828
q21 9242 5028 4949 4949
q22 549 493 459 459
Total cold run time: 76742 ms
Total hot run time: 55509 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2063166204
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2065693941
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2063447652
TeamCity be ut coverage result:
Function Coverage: 35.40% (8911/25172)
Line Coverage: 27.11% (73203/270033)
Region Coverage: 26.24% (37815/144119)
Branch Coverage: 23.06% (19274/83600)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4e56e96c2cc4c9ae35966758a3853a007fbe4dbc_4e56e96c2cc4c9ae35966758a3853a007fbe4dbc/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](top-n) support topn filter on vscan node [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #33496:
URL: https://github.com/apache/doris/pull/33496#issuecomment-2063794491
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org