You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "BiteTheDDDDt (via GitHub)" <gi...@apache.org> on 2026/04/17 09:29:32 UTC
[PR] Dev runtime partition prune [doris]
BiteTheDDDDt opened a new pull request, #62589:
URL: https://github.com/apache/doris/pull/62589
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4268930603
# BE Regression && UT Coverage Report
Increment line coverage `84.54% (257/304)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_141c78da888bc5cd8198ffebb607430c90015f0d_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_141c78da888bc5cd8198ffebb607430c90015f0d_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.71% (27513/37328) |
| Line Coverage | 57.47% (298118/518704) |
| Region Coverage | 54.54% (247670/454115) |
| Branch Coverage | 56.05% (107208/191288) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4277736731
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4287174053
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/6bb4482b83b38672c7368e51b346c7bea9dc9247_6bb4482b83b38672c7368e51b346c7bea9dc9247_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/6bb4482b83b38672c7368e51b346c7bea9dc9247_6bb4482b83b38672c7368e51b346c7bea9dc9247_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1843/2361) |
| Line Coverage | 64.77% (32998/50947) |
| Region Coverage | 65.27% (16374/25085) |
| Branch Coverage | 55.86% (8743/15652) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4505822322
# BE Regression && UT Coverage Report
Increment line coverage `75.28% (603/801)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.56% (27865/37883) |
| Line Coverage | 57.43% (302571/526849) |
| Region Coverage | 54.28% (253198/466433) |
| Branch Coverage | 55.80% (109792/196776) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3308463839
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,875 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ if (!tb.__isset.partition_id || !tb.__isset.slot_id) {
+ return Status::InternalError(
+ "Runtime filter partition boundary must contain partition_id and slot_id");
+ }
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ if (slot_it == slot_descs.end()) {
+ return Status::InternalError(
+ "Runtime filter partition boundary references unknown slot_id={}, "
+ "partition_id={}",
+ slot_id, tb.partition_id);
+ }
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ if (!is_list && !is_range) { \
+ return Status::InternalError( \
+ "Runtime filter partition boundary must be RANGE or LIST, " \
+ "partition_id={}, slot_id={}", \
+ tb.partition_id, slot_id); \
+ } \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ if (parsed_ok) {
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ } else {
+ return Status::InternalError(
+ "Runtime filter partition boundary has unsupported type, partition_id={}, "
+ "slot_id={}, type={}",
+ tb.partition_id, slot_id, slot_type->get_name());
+ }
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ if (!expr) {
+ return nullptr;
+ }
+ if (expr->is_slot_ref()) {
+ return assert_cast<const VSlotRef*>(expr);
+ }
+ const VSlotRef* found = nullptr;
+ for (const auto& child : expr->children()) {
+ const VSlotRef* c = find_unique_slot_ref(child.get());
+ if (c) {
+ if (found) {
+ return nullptr; // multiple slot refs, can't handle
+ }
+ found = c;
+ }
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+Status ParsedPartitionBoundaries::get_or_compute_projected_boundaries(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ const std::unordered_map<int64_t, TTargetExprMonotonicity::type>& partition_directions,
+ VExprContext* ctx, std::shared_ptr<const std::vector<ParsedBoundary>>* output) const {
+ {
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto it = _projected_boundaries_by_filter_id.find(filter_id);
+ if (it != _projected_boundaries_by_filter_id.end()) {
+ *output = it->second;
+ return Status::OK();
+ }
+ }
+
+ std::vector<ParsedBoundary> projected;
+ auto store_projected_boundaries = [&](std::vector<ParsedBoundary>&& boundaries) {
+ auto cached = std::make_shared<const std::vector<ParsedBoundary>>(std::move(boundaries));
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto [it, inserted] = _projected_boundaries_by_filter_id.emplace(filter_id, cached);
+ *output = inserted ? cached : it->second;
+ return Status::OK();
+ };
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::unordered_set<int64_t> boundary_partition_ids;
+ boundary_partition_ids.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ boundary_partition_ids.insert(boundary.partition_id);
+ }
+ for (const auto& [partition_id, direction] : partition_directions) {
+ if (direction == TTargetExprMonotonicity::NON_MONOTONIC) {
+ return Status::InternalError(
+ "Runtime filter partition pruning received NON_MONOTONIC partition "
+ "metadata, filter_id={}, partition_id={}",
+ filter_id, partition_id);
+ }
+ if (!boundary_partition_ids.contains(partition_id)) {
+ return Status::InternalError(
+ "Runtime filter partition pruning received monotonicity for a partition "
+ "without boundary, filter_id={}, partition_id={}, slot_id={}",
+ filter_id, partition_id, leaf_slot_id);
+ }
+ }
+
+ if (target_expr->is_slot_ref()) {
+ auto* slot_ref = assert_cast<VSlotRef*>(target_expr.get());
+ if (slot_ref->slot_id() != leaf_slot_id) {
+ return Status::InternalError(
+ "Runtime filter partition pruning SlotRef target mismatch, filter_id={}, "
+ "target_slot_id={}, leaf_slot_id={}",
+ filter_id, slot_ref->slot_id(), leaf_slot_id);
+ }
+ std::vector<ParsedBoundary> slot_boundaries;
+ slot_boundaries.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ auto direction_it = partition_directions.find(boundary.partition_id);
+ if (direction_it == partition_directions.end()) {
+ continue;
+ }
+ if (direction_it->second != TTargetExprMonotonicity::MONOTONIC_INCREASING) {
+ return Status::InternalError(
+ "Runtime filter partition pruning SlotRef target must use increasing "
+ "monotonicity, filter_id={}, partition_id={}, monotonicity={}",
+ filter_id, boundary.partition_id, direction_it->second);
+ }
+ slot_boundaries.emplace_back(boundary);
+ }
+ return store_projected_boundaries(std::move(slot_boundaries));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> selected_boundaries;
+ selected_boundaries.reserve(orig_boundaries.size());
+ for (size_t i = 0; i < orig_boundaries.size(); ++i) {
+ auto it = partition_directions.find(orig_boundaries[i].partition_id);
+ if (it != partition_directions.end() &&
+ it->second != TTargetExprMonotonicity::NON_MONOTONIC) {
+ selected_boundaries.emplace_back(i, it->second);
+ }
+ }
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> projectable_boundaries;
+ projectable_boundaries.reserve(selected_boundaries.size());
+ for (const auto& selected_boundary : selected_boundaries) {
+ const auto& boundary = orig_boundaries[selected_boundary.first];
+ if (!boundary.only_null && !boundary.contains_null) {
+ projectable_boundaries.emplace_back(selected_boundary);
+ }
+ }
+ selected_boundaries.swap(projectable_boundaries);
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ if (slot_type_it == _slot_data_types.end()) {
+ return Status::InternalError(
+ "Runtime filter partition pruning target slot has no boundary type, filter_id={}, "
+ "slot_id={}, target_expr={}",
+ filter_id, leaf_slot_id, target_expr->expr_name());
+ }
+
+ // LIST partitions require regrouping projected values per partition and
+ // should not be marked as expression-prunable by FE yet.
+ for (const auto& selected_boundary : selected_boundaries) {
+ bool is_list = false;
+ std::visit([&](const auto& cvr) { is_list = cvr.is_fixed_value_range(); },
+ orig_boundaries[selected_boundary.first].boundary_cvr);
+ if (is_list) {
+ return Status::InternalError(
+ "Runtime filter partition pruning expression target does not support LIST "
+ "partition boundaries, filter_id={}, partition_id={}, target_expr={}",
+ filter_id, orig_boundaries[selected_boundary.first].partition_id,
+ target_expr->expr_name());
+ }
+ }
+
+ const DataTypePtr& input_type = slot_type_it->second;
+ bool input_is_nullable = input_type->is_nullable();
+ DataTypePtr inner_type = input_is_nullable ? remove_nullable(input_type) : input_type;
+ PrimitiveType input_ptype = input_type->get_primitive_type();
+
+ size_t N = selected_boundaries.size();
+ std::vector<bool> lo_open(N, false);
+ std::vector<bool> hi_open(N, false);
+ std::vector<size_t> lo_result_row(N, 0);
+ std::vector<size_t> hi_result_row(N, 0);
+ size_t lo_row_count = 0;
+ size_t hi_row_count = 0;
+
+ // Build input blocks for lo and hi
+ Block lo_block;
+ Block hi_block;
+
+ // Pad columns 0..leaf_column_id-1 with placeholder columns so the leaf
+ // VSlotRef (whose column_id is leaf_column_id) reads our typed column.
+ auto add_dummy_columns = [&](Block& block, size_t row_count) {
+ for (int col_idx = 0; col_idx < leaf_column_id; ++col_idx) {
+ auto col = ColumnUInt8::create(row_count, 0);
+ block.insert({std::move(col), std::make_shared<DataTypeUInt8>(),
+ fmt::format("dummy_{}", col_idx)});
+ }
+ };
+
+ // Macro-dispatch on input PrimitiveType to build finite RANGE endpoint
+ // columns. Open endpoints are not executed through target_expr; they stay
+ // open after projection and swap sides for monotonic decreasing targets.
+ bool input_built = false;
+#define BUILD_INPUT_COLUMNS(INPUT_PT) \
+ case TYPE_##INPUT_PT: { \
+ using InCol = typename PrimitiveTypeTraits<TYPE_##INPUT_PT>::ColumnType; \
+ MutableColumnPtr lo_inner_base = inner_type->create_column(); \
+ MutableColumnPtr hi_inner_base = inner_type->create_column(); \
+ auto* lo_inner = assert_cast<InCol*>(lo_inner_base.get()); \
+ auto* hi_inner = assert_cast<InCol*>(hi_inner_base.get()); \
+ lo_inner->reserve(N); \
+ hi_inner->reserve(N); \
+ auto lo_nulls = ColumnUInt8::create(); \
+ auto hi_nulls = ColumnUInt8::create(); \
+ lo_nulls->reserve(N); \
+ hi_nulls->reserve(N); \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& boundary = orig_boundaries[selected_boundaries[i].first]; \
+ const auto* cvr = \
+ std::get_if<ColumnValueRange<TYPE_##INPUT_PT>>(&boundary.boundary_cvr); \
+ DCHECK(cvr != nullptr); \
+ DCHECK(!boundary.only_null); \
+ DCHECK(!boundary.contains_null); \
+ lo_open[i] = cvr->is_low_value_minimum(); \
+ hi_open[i] = cvr->is_high_value_maximum(); \
+ if (!lo_open[i]) { \
+ lo_result_row[i] = lo_row_count++; \
+ lo_inner->insert_value(cvr->get_range_min_value()); \
+ lo_nulls->get_data().push_back(0); \
+ } \
+ if (!hi_open[i]) { \
+ hi_result_row[i] = hi_row_count++; \
+ hi_inner->insert_value(cvr->get_range_max_value()); \
+ hi_nulls->get_data().push_back(0); \
+ } \
+ } \
+ add_dummy_columns(lo_block, lo_row_count); \
+ add_dummy_columns(hi_block, hi_row_count); \
+ if (input_is_nullable) { \
+ auto lo_col = ColumnNullable::create(std::move(lo_inner_base), std::move(lo_nulls)); \
+ auto hi_col = ColumnNullable::create(std::move(hi_inner_base), std::move(hi_nulls)); \
+ lo_block.insert({std::move(lo_col), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_col), input_type, "leaf_slot"}); \
+ } else { \
+ lo_block.insert({std::move(lo_inner_base), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_inner_base), input_type, "leaf_slot"}); \
+ } \
+ input_built = true; \
+ break; \
+ }
+
+ switch (input_ptype) {
+ BUILD_INPUT_COLUMNS(TINYINT)
+ BUILD_INPUT_COLUMNS(SMALLINT)
+ BUILD_INPUT_COLUMNS(INT)
+ BUILD_INPUT_COLUMNS(BIGINT)
+ BUILD_INPUT_COLUMNS(LARGEINT)
+ BUILD_INPUT_COLUMNS(FLOAT)
+ BUILD_INPUT_COLUMNS(DOUBLE)
+ BUILD_INPUT_COLUMNS(CHAR)
+ BUILD_INPUT_COLUMNS(DATE)
+ BUILD_INPUT_COLUMNS(DATETIME)
+ BUILD_INPUT_COLUMNS(DATEV2)
+ BUILD_INPUT_COLUMNS(DATETIMEV2)
+ BUILD_INPUT_COLUMNS(TIMESTAMPTZ)
+ BUILD_INPUT_COLUMNS(VARCHAR)
+ BUILD_INPUT_COLUMNS(STRING)
+ BUILD_INPUT_COLUMNS(DECIMAL32)
+ BUILD_INPUT_COLUMNS(DECIMAL64)
+ BUILD_INPUT_COLUMNS(DECIMAL128I)
+ BUILD_INPUT_COLUMNS(DECIMAL256)
+ BUILD_INPUT_COLUMNS(DECIMALV2)
+ BUILD_INPUT_COLUMNS(BOOLEAN)
+ BUILD_INPUT_COLUMNS(IPV4)
+ BUILD_INPUT_COLUMNS(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_INPUT_COLUMNS
+
+ if (!input_built) {
+ return Status::InternalError(
+ "Runtime filter partition pruning expression target has unsupported input type, "
+ "filter_id={}, input_type={}, target_expr={}",
+ filter_id, input_type->get_name(), target_expr->expr_name());
+ }
+
+ int lo_result_id = -1;
+ int hi_result_id = -1;
+ ColumnPtr lo_result_col;
+ ColumnPtr hi_result_col;
+ if (lo_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &lo_block, &lo_result_id));
+ if (lo_result_id < 0) {
+ return Status::InternalError(
+ "Runtime filter partition pruning failed to project lower boundary, "
+ "filter_id={}, target_expr={}",
+ filter_id, target_expr->expr_name());
+ }
+ lo_result_col = lo_block.get_by_position(lo_result_id).column;
+ }
+ if (hi_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &hi_block, &hi_result_id));
+ if (hi_result_id < 0) {
+ return Status::InternalError(
+ "Runtime filter partition pruning failed to project upper boundary, "
+ "filter_id={}, target_expr={}",
+ filter_id, target_expr->expr_name());
+ }
+ hi_result_col = hi_block.get_by_position(hi_result_id).column;
+ }
+ int out_precision = cast_set<int>(target_expr->data_type()->get_precision());
+ int out_scale = cast_set<int>(target_expr->data_type()->get_scale());
+ bool out_nullable = target_expr->data_type()->is_nullable();
+ PrimitiveType output_ptype = target_expr->data_type()->get_primitive_type();
+
+ projected.reserve(N);
+
+ bool output_built = false;
+#define BUILD_PROJECTED_CVR(OUTPUT_PT) \
+ case TYPE_##OUTPUT_PT: { \
+ using OutputCppType = typename PrimitiveTypeTraits<TYPE_##OUTPUT_PT>::CppType; \
+ auto projected_value = [](const ColumnPtr& column, size_t row) -> OutputCppType { \
+ Field field = (*column)[row]; \
+ return field.get<TYPE_##OUTPUT_PT>(); \
+ }; \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& orig_boundary = orig_boundaries[selected_boundaries[i].first]; \
+ TTargetExprMonotonicity::type direction = selected_boundaries[i].second; \
+ if ((!lo_open[i] && lo_result_col->is_null_at(lo_result_row[i])) || \
+ (!hi_open[i] && hi_result_col->is_null_at(hi_result_row[i]))) { \
+ continue; \
+ } \
+ ColumnValueRange<TYPE_##OUTPUT_PT> cvr("", out_nullable, out_precision, out_scale); \
+ if (direction == TTargetExprMonotonicity::MONOTONIC_DECREASING) { \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, hi_projected)); \
+ } \
+ if (!lo_open[i]) { \
Review Comment:
原始 RANGE 分区上界可能是开区间（<），投影后统一变成 FILTER_LESS_OR_EQUAL（<=）？这个是否可能有问题
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4551440400
# BE UT Coverage Report
Increment line coverage `62.92% (504/801)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/3d38dc5ab813ed88625762426d63d5324bfff4fe_3d38dc5ab813ed88625762426d63d5324bfff4fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/3d38dc5ab813ed88625762426d63d5324bfff4fe_3d38dc5ab813ed88625762426d63d5324bfff4fe/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.88% (20936/38857) |
| Line Coverage | 37.47% (198602/530087) |
| Region Coverage | 33.67% (156138/463736) |
| Branch Coverage | 34.72% (68307/196735) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3309562083
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,875 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ if (!tb.__isset.partition_id || !tb.__isset.slot_id) {
+ return Status::InternalError(
+ "Runtime filter partition boundary must contain partition_id and slot_id");
+ }
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ if (slot_it == slot_descs.end()) {
+ return Status::InternalError(
+ "Runtime filter partition boundary references unknown slot_id={}, "
+ "partition_id={}",
+ slot_id, tb.partition_id);
+ }
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ if (!is_list && !is_range) { \
+ return Status::InternalError( \
+ "Runtime filter partition boundary must be RANGE or LIST, " \
+ "partition_id={}, slot_id={}", \
+ tb.partition_id, slot_id); \
+ } \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ if (parsed_ok) {
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ } else {
+ return Status::InternalError(
+ "Runtime filter partition boundary has unsupported type, partition_id={}, "
+ "slot_id={}, type={}",
+ tb.partition_id, slot_id, slot_type->get_name());
+ }
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ if (!expr) {
+ return nullptr;
+ }
+ if (expr->is_slot_ref()) {
+ return assert_cast<const VSlotRef*>(expr);
+ }
+ const VSlotRef* found = nullptr;
+ for (const auto& child : expr->children()) {
+ const VSlotRef* c = find_unique_slot_ref(child.get());
+ if (c) {
+ if (found) {
+ return nullptr; // multiple slot refs, can't handle
+ }
+ found = c;
+ }
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+Status ParsedPartitionBoundaries::get_or_compute_projected_boundaries(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ const std::unordered_map<int64_t, TTargetExprMonotonicity::type>& partition_directions,
+ VExprContext* ctx, std::shared_ptr<const std::vector<ParsedBoundary>>* output) const {
+ {
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto it = _projected_boundaries_by_filter_id.find(filter_id);
+ if (it != _projected_boundaries_by_filter_id.end()) {
+ *output = it->second;
+ return Status::OK();
+ }
+ }
+
+ std::vector<ParsedBoundary> projected;
+ auto store_projected_boundaries = [&](std::vector<ParsedBoundary>&& boundaries) {
+ auto cached = std::make_shared<const std::vector<ParsedBoundary>>(std::move(boundaries));
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto [it, inserted] = _projected_boundaries_by_filter_id.emplace(filter_id, cached);
+ *output = inserted ? cached : it->second;
+ return Status::OK();
+ };
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::unordered_set<int64_t> boundary_partition_ids;
+ boundary_partition_ids.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ boundary_partition_ids.insert(boundary.partition_id);
+ }
+ for (const auto& [partition_id, direction] : partition_directions) {
+ if (direction == TTargetExprMonotonicity::NON_MONOTONIC) {
+ return Status::InternalError(
+ "Runtime filter partition pruning received NON_MONOTONIC partition "
+ "metadata, filter_id={}, partition_id={}",
+ filter_id, partition_id);
+ }
+ if (!boundary_partition_ids.contains(partition_id)) {
+ return Status::InternalError(
+ "Runtime filter partition pruning received monotonicity for a partition "
+ "without boundary, filter_id={}, partition_id={}, slot_id={}",
+ filter_id, partition_id, leaf_slot_id);
+ }
+ }
+
+ if (target_expr->is_slot_ref()) {
+ auto* slot_ref = assert_cast<VSlotRef*>(target_expr.get());
+ if (slot_ref->slot_id() != leaf_slot_id) {
+ return Status::InternalError(
+ "Runtime filter partition pruning SlotRef target mismatch, filter_id={}, "
+ "target_slot_id={}, leaf_slot_id={}",
+ filter_id, slot_ref->slot_id(), leaf_slot_id);
+ }
+ std::vector<ParsedBoundary> slot_boundaries;
+ slot_boundaries.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ auto direction_it = partition_directions.find(boundary.partition_id);
+ if (direction_it == partition_directions.end()) {
+ continue;
+ }
+ if (direction_it->second != TTargetExprMonotonicity::MONOTONIC_INCREASING) {
+ return Status::InternalError(
+ "Runtime filter partition pruning SlotRef target must use increasing "
+ "monotonicity, filter_id={}, partition_id={}, monotonicity={}",
+ filter_id, boundary.partition_id, direction_it->second);
+ }
+ slot_boundaries.emplace_back(boundary);
+ }
+ return store_projected_boundaries(std::move(slot_boundaries));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> selected_boundaries;
+ selected_boundaries.reserve(orig_boundaries.size());
+ for (size_t i = 0; i < orig_boundaries.size(); ++i) {
+ auto it = partition_directions.find(orig_boundaries[i].partition_id);
+ if (it != partition_directions.end() &&
+ it->second != TTargetExprMonotonicity::NON_MONOTONIC) {
+ selected_boundaries.emplace_back(i, it->second);
+ }
+ }
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> projectable_boundaries;
+ projectable_boundaries.reserve(selected_boundaries.size());
+ for (const auto& selected_boundary : selected_boundaries) {
+ const auto& boundary = orig_boundaries[selected_boundary.first];
+ if (!boundary.only_null && !boundary.contains_null) {
+ projectable_boundaries.emplace_back(selected_boundary);
+ }
+ }
+ selected_boundaries.swap(projectable_boundaries);
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ if (slot_type_it == _slot_data_types.end()) {
+ return Status::InternalError(
+ "Runtime filter partition pruning target slot has no boundary type, filter_id={}, "
+ "slot_id={}, target_expr={}",
+ filter_id, leaf_slot_id, target_expr->expr_name());
+ }
+
+ // LIST partitions require regrouping projected values per partition and
+ // should not be marked as expression-prunable by FE yet.
+ for (const auto& selected_boundary : selected_boundaries) {
+ bool is_list = false;
+ std::visit([&](const auto& cvr) { is_list = cvr.is_fixed_value_range(); },
+ orig_boundaries[selected_boundary.first].boundary_cvr);
+ if (is_list) {
+ return Status::InternalError(
+ "Runtime filter partition pruning expression target does not support LIST "
+ "partition boundaries, filter_id={}, partition_id={}, target_expr={}",
+ filter_id, orig_boundaries[selected_boundary.first].partition_id,
+ target_expr->expr_name());
+ }
+ }
+
+ const DataTypePtr& input_type = slot_type_it->second;
+ bool input_is_nullable = input_type->is_nullable();
+ DataTypePtr inner_type = input_is_nullable ? remove_nullable(input_type) : input_type;
+ PrimitiveType input_ptype = input_type->get_primitive_type();
+
+ size_t N = selected_boundaries.size();
+ std::vector<bool> lo_open(N, false);
+ std::vector<bool> hi_open(N, false);
+ std::vector<size_t> lo_result_row(N, 0);
+ std::vector<size_t> hi_result_row(N, 0);
+ size_t lo_row_count = 0;
+ size_t hi_row_count = 0;
+
+ // Build input blocks for lo and hi
+ Block lo_block;
+ Block hi_block;
+
+ // Pad columns 0..leaf_column_id-1 with placeholder columns so the leaf
+ // VSlotRef (whose column_id is leaf_column_id) reads our typed column.
+ auto add_dummy_columns = [&](Block& block, size_t row_count) {
+ for (int col_idx = 0; col_idx < leaf_column_id; ++col_idx) {
+ auto col = ColumnUInt8::create(row_count, 0);
+ block.insert({std::move(col), std::make_shared<DataTypeUInt8>(),
+ fmt::format("dummy_{}", col_idx)});
+ }
+ };
+
+ // Macro-dispatch on input PrimitiveType to build finite RANGE endpoint
+ // columns. Open endpoints are not executed through target_expr; they stay
+ // open after projection and swap sides for monotonic decreasing targets.
+ bool input_built = false;
+#define BUILD_INPUT_COLUMNS(INPUT_PT) \
+ case TYPE_##INPUT_PT: { \
+ using InCol = typename PrimitiveTypeTraits<TYPE_##INPUT_PT>::ColumnType; \
+ MutableColumnPtr lo_inner_base = inner_type->create_column(); \
+ MutableColumnPtr hi_inner_base = inner_type->create_column(); \
+ auto* lo_inner = assert_cast<InCol*>(lo_inner_base.get()); \
+ auto* hi_inner = assert_cast<InCol*>(hi_inner_base.get()); \
+ lo_inner->reserve(N); \
+ hi_inner->reserve(N); \
+ auto lo_nulls = ColumnUInt8::create(); \
+ auto hi_nulls = ColumnUInt8::create(); \
+ lo_nulls->reserve(N); \
+ hi_nulls->reserve(N); \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& boundary = orig_boundaries[selected_boundaries[i].first]; \
+ const auto* cvr = \
+ std::get_if<ColumnValueRange<TYPE_##INPUT_PT>>(&boundary.boundary_cvr); \
+ DCHECK(cvr != nullptr); \
+ DCHECK(!boundary.only_null); \
+ DCHECK(!boundary.contains_null); \
+ lo_open[i] = cvr->is_low_value_minimum(); \
+ hi_open[i] = cvr->is_high_value_maximum(); \
+ if (!lo_open[i]) { \
+ lo_result_row[i] = lo_row_count++; \
+ lo_inner->insert_value(cvr->get_range_min_value()); \
+ lo_nulls->get_data().push_back(0); \
+ } \
+ if (!hi_open[i]) { \
+ hi_result_row[i] = hi_row_count++; \
+ hi_inner->insert_value(cvr->get_range_max_value()); \
+ hi_nulls->get_data().push_back(0); \
+ } \
+ } \
+ add_dummy_columns(lo_block, lo_row_count); \
+ add_dummy_columns(hi_block, hi_row_count); \
+ if (input_is_nullable) { \
+ auto lo_col = ColumnNullable::create(std::move(lo_inner_base), std::move(lo_nulls)); \
+ auto hi_col = ColumnNullable::create(std::move(hi_inner_base), std::move(hi_nulls)); \
+ lo_block.insert({std::move(lo_col), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_col), input_type, "leaf_slot"}); \
+ } else { \
+ lo_block.insert({std::move(lo_inner_base), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_inner_base), input_type, "leaf_slot"}); \
+ } \
+ input_built = true; \
+ break; \
+ }
+
+ switch (input_ptype) {
+ BUILD_INPUT_COLUMNS(TINYINT)
+ BUILD_INPUT_COLUMNS(SMALLINT)
+ BUILD_INPUT_COLUMNS(INT)
+ BUILD_INPUT_COLUMNS(BIGINT)
+ BUILD_INPUT_COLUMNS(LARGEINT)
+ BUILD_INPUT_COLUMNS(FLOAT)
+ BUILD_INPUT_COLUMNS(DOUBLE)
+ BUILD_INPUT_COLUMNS(CHAR)
+ BUILD_INPUT_COLUMNS(DATE)
+ BUILD_INPUT_COLUMNS(DATETIME)
+ BUILD_INPUT_COLUMNS(DATEV2)
+ BUILD_INPUT_COLUMNS(DATETIMEV2)
+ BUILD_INPUT_COLUMNS(TIMESTAMPTZ)
+ BUILD_INPUT_COLUMNS(VARCHAR)
+ BUILD_INPUT_COLUMNS(STRING)
+ BUILD_INPUT_COLUMNS(DECIMAL32)
+ BUILD_INPUT_COLUMNS(DECIMAL64)
+ BUILD_INPUT_COLUMNS(DECIMAL128I)
+ BUILD_INPUT_COLUMNS(DECIMAL256)
+ BUILD_INPUT_COLUMNS(DECIMALV2)
+ BUILD_INPUT_COLUMNS(BOOLEAN)
+ BUILD_INPUT_COLUMNS(IPV4)
+ BUILD_INPUT_COLUMNS(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_INPUT_COLUMNS
+
+ if (!input_built) {
+ return Status::InternalError(
+ "Runtime filter partition pruning expression target has unsupported input type, "
+ "filter_id={}, input_type={}, target_expr={}",
+ filter_id, input_type->get_name(), target_expr->expr_name());
+ }
+
+ int lo_result_id = -1;
+ int hi_result_id = -1;
+ ColumnPtr lo_result_col;
+ ColumnPtr hi_result_col;
+ if (lo_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &lo_block, &lo_result_id));
+ if (lo_result_id < 0) {
+ return Status::InternalError(
+ "Runtime filter partition pruning failed to project lower boundary, "
+ "filter_id={}, target_expr={}",
+ filter_id, target_expr->expr_name());
+ }
+ lo_result_col = lo_block.get_by_position(lo_result_id).column;
+ }
+ if (hi_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &hi_block, &hi_result_id));
+ if (hi_result_id < 0) {
+ return Status::InternalError(
+ "Runtime filter partition pruning failed to project upper boundary, "
+ "filter_id={}, target_expr={}",
+ filter_id, target_expr->expr_name());
+ }
+ hi_result_col = hi_block.get_by_position(hi_result_id).column;
+ }
+ int out_precision = cast_set<int>(target_expr->data_type()->get_precision());
+ int out_scale = cast_set<int>(target_expr->data_type()->get_scale());
+ bool out_nullable = target_expr->data_type()->is_nullable();
+ PrimitiveType output_ptype = target_expr->data_type()->get_primitive_type();
+
+ projected.reserve(N);
+
+ bool output_built = false;
+#define BUILD_PROJECTED_CVR(OUTPUT_PT) \
+ case TYPE_##OUTPUT_PT: { \
+ using OutputCppType = typename PrimitiveTypeTraits<TYPE_##OUTPUT_PT>::CppType; \
+ auto projected_value = [](const ColumnPtr& column, size_t row) -> OutputCppType { \
+ Field field = (*column)[row]; \
+ return field.get<TYPE_##OUTPUT_PT>(); \
+ }; \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& orig_boundary = orig_boundaries[selected_boundaries[i].first]; \
+ TTargetExprMonotonicity::type direction = selected_boundaries[i].second; \
+ if ((!lo_open[i] && lo_result_col->is_null_at(lo_result_row[i])) || \
+ (!hi_open[i] && hi_result_col->is_null_at(hi_result_row[i]))) { \
+ continue; \
+ } \
+ ColumnValueRange<TYPE_##OUTPUT_PT> cvr("", out_nullable, out_precision, out_scale); \
+ if (direction == TTargetExprMonotonicity::MONOTONIC_DECREASING) { \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, hi_projected)); \
+ } \
+ if (!lo_open[i]) { \
Review Comment:
假设分区列是dt，原始RANGE分区是：
P1:dtin[2024-01-0100:00:00,2024-01-0200:00:00)
上界2024-01-0200:00:00是开区间，这一点没错。
但RFtarget不是dt本身，而是一个非严格单调表达式，例如按天向上取整：
target_expr=date_ceil(dt,'day')
这个函数是单调递增的，但不是严格递增。分区里的真实行可能有：
dt=2024-01-0112:00:00
它满足原始分区范围，因为它小于开上界2024-01-0200:00:00。但投影后：
date_ceil(2024-01-0112:00:00,'day')=2024-01-0200:00:00
也就是说，虽然原始上界是开的，投影结果里仍然可能出现target_expr(range_end)这个值。
如果BE把投影后的范围做成：
[date_ceil(start),date_ceil(end))--上界仍然用<
那就会变成不包含2024-01-0200:00:00。假如RFbuildside正好只有：
RF={2024-01-0200:00:00}
BE会认为投影分区范围和RF没交集，从而裁掉P1；但实际上P1里2024-01-0112:00:00这类行投影后能命中RF，所以这是误裁剪。
所以当前用<=target_expr(end)是一个安全的over-approx：可能多保留一点分区，但不会把可能命中的分区裁掉。只有当我们能证明targetexpr
是严格单调且开边界不会映射出端点值时，才可以安全地保留<。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4554975524
<details>
<summary>TPC-H: <b>Total hot run time: 31731 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 36cd8599c1b5eba6dcfb83d6c88457494678e987, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17726 4015 4014 4014
q2 q3 10828 1438 806 806
q4 4750 482 355 355
q5 9561 2340 2146 2146
q6 391 179 143 143
q7 956 809 644 644
q8 9578 1755 1641 1641
q9 7022 5055 4974 4974
q10 6471 2239 1911 1911
q11 451 279 246 246
q12 691 434 295 295
q13 18235 3379 2823 2823
q14 267 259 237 237
q15 q16 835 788 708 708
q17 937 966 941 941
q18 6997 5782 5757 5757
q19 1192 1205 1082 1082
q20 513 398 275 275
q21 5585 2570 2427 2427
q22 428 362 306 306
Total cold run time: 103414 ms
Total hot run time: 31731 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4336 4262 4267 4262
q2 q3 4549 4934 4343 4343
q4 2109 2217 1374 1374
q5 4485 4335 5538 4335
q6 253 189 147 147
q7 2109 1825 1616 1616
q8 2554 2267 2355 2267
q9 7967 8240 7991 7991
q10 4804 4743 4320 4320
q11 601 430 384 384
q12 745 758 556 556
q13 3244 3625 3006 3006
q14 304 313 297 297
q15 q16 862 744 694 694
q17 1360 1337 1329 1329
q18 7989 7186 7219 7186
q19 1143 1095 1091 1091
q20 2236 2218 1935 1935
q21 5279 4627 4519 4519
q22 535 455 409 409
Total cold run time: 57464 ms
Total hot run time: 52061 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4539357346
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/f779760a4db0a4e1f4605c63a8a9bc21f51af126_f779760a4db0a4e1f4605c63a8a9bc21f51af126_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/f779760a4db0a4e1f4605c63a8a9bc21f51af126_f779760a4db0a4e1f4605c63a8a9bc21f51af126_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1854/2375) |
| Line Coverage | 64.52% (33334/51665) |
| Region Coverage | 65.20% (16523/25343) |
| Branch Coverage | 55.74% (8837/15854) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4449720738
# FE UT Coverage Report
Increment line coverage ` 30.41% (90/296)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_572ab7793a850dc7d5ad35a35834f642fa194f82/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_572ab7793a850dc7d5ad35a35834f642fa194f82/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3241048647
##########
regression-test/suites/query_p0/runtime_filter/rf_partition_pruning_type_matrix.groovy:
##########
@@ -0,0 +1,219 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.action.ProfileAction
+
+suite("rf_partition_pruning_type_matrix", "nonConcurrent") {
+ sql "set enable_runtime_filter_prune=false"
+ sql "set enable_runtime_filter_partition_prune=true"
+ sql "set runtime_filter_wait_infinitely=true"
+ sql "set disable_join_reorder=true"
+ sql "set enable_profile=true"
+ sql "set profile_level=2"
+ sql "set parallel_pipeline_task_num=1"
+
+ def profileAction = new ProfileAction(context)
+ def rfPruningCounterNames = ["TotalPartitionsForRFPruning", "PartitionsPrunedByRuntimeFilter"]
+
+ def getProfileByToken = { String token ->
+ String profileContent = ""
+ for (int attempt = 0; attempt < 60; attempt++) {
+ List profileData = profileAction.getProfileList()
+ for (final def profileItem in profileData) {
+ if (profileItem["Sql Statement"].toString().contains(token)) {
+ profileContent = profileAction.getProfile(profileItem["Profile ID"].toString())
+ break
+ }
+ }
+ if (profileContent != "" && rfPruningCounterNames.every { profileContent.contains(it) }) {
+ break
+ }
+ Thread.sleep(500)
+ }
+ return profileContent
+ }
+
+ def extractCounterSum = { String profileText, String counterName ->
+ def values = (profileText =~ /-\s*${counterName}:\s*(\d+)/).collect { it[1].toLong() }
+ return values.isEmpty() ? 0L : values.sum()
+ }
+
+ def assertPruningProfile = { String queryBody, String rfType, long expectedTotal, long expectedPruned ->
+ def token = UUID.randomUUID().toString()
+ def querySql = """
+ SELECT /*+ SET_VAR(runtime_filter_type='${rfType}') */ "${token}", *
+ ${queryBody}
+ ORDER BY 2
+ """
+ sql querySql
+ def profile = getProfileByToken(token)
Review Comment:
This helper executes the join and then discards the result, so the type-matrix cases only prove that the profile counters are at least the expected values. A false-positive pruning bug can still pass here: for example, if an INT RANGE case prunes all four partitions instead of keeping the partition containing `dim_key = 150`, `PartitionsPrunedByRuntimeFilter >= 3` still succeeds while the query returns no matching fact row. Please have the helper assert the expected joined rows (or at least the expected fact ids) in addition to the pruning counters, so these cases catch over-pruning/data-loss regressions. This is distinct from the existing profile-polling/golden-file threads because the profile can be present and the counters can satisfy the current assertions while the query result is wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4426034929
# FE Regression Coverage Report
Increment line coverage ` 65.93% (178/270)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_b29520eb8c2fd176695ee6c263c89bab10446d4a_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_b29520eb8c2fd176695ee6c263c89bab10446d4a_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4534540271
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4286496306
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4286987764
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4277736436
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4278503350
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/bcfae3c60326b6ef46e5c8dfc68aaf68a3be4fdd_bcfae3c60326b6ef46e5c8dfc68aaf68a3be4fdd_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/bcfae3c60326b6ef46e5c8dfc68aaf68a3be4fdd_bcfae3c60326b6ef46e5c8dfc68aaf68a3be4fdd_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.02% (1842/2361) |
| Line Coverage | 64.60% (32919/50958) |
| Region Coverage | 65.22% (16326/25034) |
| Branch Coverage | 55.76% (8707/15616) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4539063251
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4553722777
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4551463359
# BE Regression && UT Coverage Report
Increment line coverage `75.28% (603/801)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_3d38dc5ab813ed88625762426d63d5324bfff4fe_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_3d38dc5ab813ed88625762426d63d5324bfff4fe_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.73% (28060/38058) |
| Line Coverage | 57.67% (304930/528726) |
| Region Coverage | 54.68% (255900/468024) |
| Branch Coverage | 56.11% (110795/197453) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4564331115
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3330576119
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,866 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ DORIS_CHECK(tb.__isset.partition_id);
+ DORIS_CHECK(tb.__isset.slot_id);
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ DORIS_CHECK(slot_it != slot_descs.end());
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ DORIS_CHECK(is_list || is_range); \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ bool range_has_null = false; \
+ if (is_nullable && !tb.__isset.range_start) { \
+ range_has_null = true; \
+ } \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } else if (is_nullable) { \
+ range_has_null = true; \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ if (range_has_null) { \
+ boundary.contains_null = true; \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ DORIS_CHECK(parsed_ok);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static bool find_unique_slot_ref_impl(const VExpr* expr, const VSlotRef** found) {
+ if (!expr) {
+ return true;
+ }
+ if (expr->is_slot_ref()) {
+ const auto* slot = assert_cast<const VSlotRef*>(expr);
+ if (*found == nullptr) {
+ *found = slot;
+ return true;
+ }
+ if ((*found)->slot_id() != slot->slot_id()) {
+ return false;
+ }
+ DORIS_CHECK((*found)->column_id() == slot->column_id());
+ return true;
+ }
+ for (const auto& child : expr->children()) {
+ if (!find_unique_slot_ref_impl(child.get(), found)) {
+ return false;
+ }
+ }
+ return true;
+}
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ const VSlotRef* found = nullptr;
+ if (!find_unique_slot_ref_impl(expr, &found)) {
+ return nullptr;
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+Status ParsedPartitionBoundaries::get_or_compute_projected_boundaries(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ const std::unordered_map<int64_t, TTargetExprMonotonicity::type>& partition_directions,
+ VExprContext* ctx, std::shared_ptr<const std::vector<ParsedBoundary>>* output) const {
+ {
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto it = _projected_boundaries_by_filter_id.find(filter_id);
+ if (it != _projected_boundaries_by_filter_id.end()) {
+ *output = it->second;
+ return Status::OK();
+ }
+ }
+
+ std::vector<ParsedBoundary> projected;
+ auto store_projected_boundaries = [&](std::vector<ParsedBoundary>&& boundaries) {
+ auto cached = std::make_shared<const std::vector<ParsedBoundary>>(std::move(boundaries));
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto [it, inserted] = _projected_boundaries_by_filter_id.emplace(filter_id, cached);
+ *output = inserted ? cached : it->second;
+ return Status::OK();
+ };
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::unordered_set<int64_t> boundary_partition_ids;
+ boundary_partition_ids.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ boundary_partition_ids.insert(boundary.partition_id);
+ }
+ for (const auto& [partition_id, direction] : partition_directions) {
+ DORIS_CHECK(direction != TTargetExprMonotonicity::NON_MONOTONIC);
+ DORIS_CHECK(boundary_partition_ids.contains(partition_id));
+ }
+
+ if (target_expr->is_slot_ref()) {
+ auto* slot_ref = assert_cast<VSlotRef*>(target_expr.get());
+ DORIS_CHECK(slot_ref->slot_id() == leaf_slot_id);
+ std::vector<ParsedBoundary> slot_boundaries;
+ slot_boundaries.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ auto direction_it = partition_directions.find(boundary.partition_id);
+ if (direction_it == partition_directions.end()) {
+ continue;
+ }
+ DORIS_CHECK(direction_it->second == TTargetExprMonotonicity::MONOTONIC_INCREASING);
+ slot_boundaries.emplace_back(boundary);
+ }
+ return store_projected_boundaries(std::move(slot_boundaries));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> selected_boundaries;
+ selected_boundaries.reserve(orig_boundaries.size());
+ for (size_t i = 0; i < orig_boundaries.size(); ++i) {
+ auto it = partition_directions.find(orig_boundaries[i].partition_id);
+ if (it != partition_directions.end() &&
+ it->second != TTargetExprMonotonicity::NON_MONOTONIC) {
+ selected_boundaries.emplace_back(i, it->second);
+ }
+ }
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> projectable_boundaries;
+ projectable_boundaries.reserve(selected_boundaries.size());
+ for (const auto& selected_boundary : selected_boundaries) {
+ const auto& boundary = orig_boundaries[selected_boundary.first];
+ if (!boundary.only_null && !boundary.contains_null) {
+ projectable_boundaries.emplace_back(selected_boundary);
+ }
+ }
+ selected_boundaries.swap(projectable_boundaries);
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ DORIS_CHECK(slot_type_it != _slot_data_types.end());
+
+ const DataTypePtr& input_type = slot_type_it->second;
+ bool input_is_nullable = input_type->is_nullable();
+ DataTypePtr inner_type = input_is_nullable ? remove_nullable(input_type) : input_type;
+ PrimitiveType input_ptype = input_type->get_primitive_type();
+
+ size_t N = selected_boundaries.size();
+ std::vector<bool> lo_open(N, false);
+ std::vector<bool> hi_open(N, false);
+ std::vector<bool> boundary_is_list(N, false);
+ std::vector<size_t> lo_result_row(N, 0);
+ std::vector<size_t> hi_result_row(N, 0);
+ std::vector<size_t> list_result_begin(N, 0);
+ std::vector<size_t> list_result_end(N, 0);
+ size_t lo_row_count = 0;
+ size_t hi_row_count = 0;
+ size_t list_row_count = 0;
+
+ // Build input blocks for RANGE endpoints and LIST values.
+ Block lo_block;
+ Block hi_block;
+ Block list_block;
+
+ // Pad columns 0..leaf_column_id-1 with placeholder columns so the leaf
+ // VSlotRef (whose column_id is leaf_column_id) reads our typed column.
+ auto add_dummy_columns = [&](Block& block, size_t row_count) {
+ for (int col_idx = 0; col_idx < leaf_column_id; ++col_idx) {
+ auto col = ColumnUInt8::create(row_count, 0);
+ block.insert({std::move(col), std::make_shared<DataTypeUInt8>(),
+ fmt::format("dummy_{}", col_idx)});
+ }
+ };
+
+ // Macro-dispatch on input PrimitiveType to build finite RANGE endpoint
+ // columns and LIST value columns. Unbounded RANGE endpoints are not executed
+ // through target_expr; they stay unbounded after projection and swap sides
+ // for monotonic decreasing targets.
+ bool input_built = false;
+#define BUILD_INPUT_COLUMNS(INPUT_PT) \
+ case TYPE_##INPUT_PT: { \
+ using InCol = typename PrimitiveTypeTraits<TYPE_##INPUT_PT>::ColumnType; \
+ MutableColumnPtr lo_inner_base = inner_type->create_column(); \
+ MutableColumnPtr hi_inner_base = inner_type->create_column(); \
+ MutableColumnPtr list_inner_base = inner_type->create_column(); \
+ auto* lo_inner = assert_cast<InCol*>(lo_inner_base.get()); \
+ auto* hi_inner = assert_cast<InCol*>(hi_inner_base.get()); \
+ auto* list_inner = assert_cast<InCol*>(list_inner_base.get()); \
+ lo_inner->reserve(N); \
+ hi_inner->reserve(N); \
+ list_inner->reserve(N); \
+ auto lo_nulls = ColumnUInt8::create(); \
+ auto hi_nulls = ColumnUInt8::create(); \
+ auto list_nulls = ColumnUInt8::create(); \
+ lo_nulls->reserve(N); \
+ hi_nulls->reserve(N); \
+ list_nulls->reserve(N); \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& boundary = orig_boundaries[selected_boundaries[i].first]; \
+ const auto* cvr = \
+ std::get_if<ColumnValueRange<TYPE_##INPUT_PT>>(&boundary.boundary_cvr); \
+ DCHECK(cvr != nullptr); \
+ DCHECK(!boundary.only_null); \
+ DCHECK(!boundary.contains_null); \
+ boundary_is_list[i] = cvr->is_fixed_value_range(); \
+ if (boundary_is_list[i]) { \
+ list_result_begin[i] = list_row_count; \
+ for (const auto& value : cvr->get_fixed_value_set()) { \
+ list_inner->insert_value(value); \
+ list_nulls->get_data().push_back(0); \
+ ++list_row_count; \
+ } \
+ list_result_end[i] = list_row_count; \
+ continue; \
+ } \
+ lo_open[i] = cvr->is_low_value_minimum(); \
+ hi_open[i] = cvr->is_high_value_maximum(); \
+ if (!lo_open[i]) { \
+ lo_result_row[i] = lo_row_count++; \
+ lo_inner->insert_value(cvr->get_range_min_value()); \
+ lo_nulls->get_data().push_back(0); \
+ } \
+ if (!hi_open[i]) { \
+ hi_result_row[i] = hi_row_count++; \
+ hi_inner->insert_value(cvr->get_range_max_value()); \
+ hi_nulls->get_data().push_back(0); \
+ } \
+ } \
+ add_dummy_columns(lo_block, lo_row_count); \
+ add_dummy_columns(hi_block, hi_row_count); \
+ add_dummy_columns(list_block, list_row_count); \
+ if (input_is_nullable) { \
+ auto lo_col = ColumnNullable::create(std::move(lo_inner_base), std::move(lo_nulls)); \
+ auto hi_col = ColumnNullable::create(std::move(hi_inner_base), std::move(hi_nulls)); \
+ auto list_col = \
+ ColumnNullable::create(std::move(list_inner_base), std::move(list_nulls)); \
+ lo_block.insert({std::move(lo_col), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_col), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_col), input_type, "leaf_slot"}); \
+ } else { \
+ lo_block.insert({std::move(lo_inner_base), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_inner_base), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_inner_base), input_type, "leaf_slot"}); \
+ } \
+ input_built = true; \
+ break; \
+ }
+
+ switch (input_ptype) {
+ BUILD_INPUT_COLUMNS(TINYINT)
+ BUILD_INPUT_COLUMNS(SMALLINT)
+ BUILD_INPUT_COLUMNS(INT)
+ BUILD_INPUT_COLUMNS(BIGINT)
+ BUILD_INPUT_COLUMNS(LARGEINT)
+ BUILD_INPUT_COLUMNS(FLOAT)
+ BUILD_INPUT_COLUMNS(DOUBLE)
+ BUILD_INPUT_COLUMNS(CHAR)
+ BUILD_INPUT_COLUMNS(DATE)
+ BUILD_INPUT_COLUMNS(DATETIME)
+ BUILD_INPUT_COLUMNS(DATEV2)
+ BUILD_INPUT_COLUMNS(DATETIMEV2)
+ BUILD_INPUT_COLUMNS(TIMESTAMPTZ)
+ BUILD_INPUT_COLUMNS(VARCHAR)
+ BUILD_INPUT_COLUMNS(STRING)
+ BUILD_INPUT_COLUMNS(DECIMAL32)
+ BUILD_INPUT_COLUMNS(DECIMAL64)
+ BUILD_INPUT_COLUMNS(DECIMAL128I)
+ BUILD_INPUT_COLUMNS(DECIMAL256)
+ BUILD_INPUT_COLUMNS(DECIMALV2)
+ BUILD_INPUT_COLUMNS(BOOLEAN)
+ BUILD_INPUT_COLUMNS(IPV4)
+ BUILD_INPUT_COLUMNS(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_INPUT_COLUMNS
+
+ DORIS_CHECK(input_built);
+
+ int lo_result_id = -1;
+ int hi_result_id = -1;
+ int list_result_id = -1;
+ ColumnPtr lo_result_col;
+ ColumnPtr hi_result_col;
+ ColumnPtr list_result_col;
+ if (lo_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &lo_block, &lo_result_id));
+ DORIS_CHECK(lo_result_id >= 0);
+ lo_result_col = lo_block.get_by_position(lo_result_id).column;
+ }
+ if (hi_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &hi_block, &hi_result_id));
+ DORIS_CHECK(hi_result_id >= 0);
+ hi_result_col = hi_block.get_by_position(hi_result_id).column;
+ }
+ if (list_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &list_block, &list_result_id));
+ DORIS_CHECK(list_result_id >= 0);
+ list_result_col = list_block.get_by_position(list_result_id).column;
+ }
+ int out_precision = cast_set<int>(target_expr->data_type()->get_precision());
+ int out_scale = cast_set<int>(target_expr->data_type()->get_scale());
+ bool out_nullable = target_expr->data_type()->is_nullable();
+ PrimitiveType output_ptype = target_expr->data_type()->get_primitive_type();
+
+ projected.reserve(N);
+
+ bool output_built = false;
+#define BUILD_PROJECTED_CVR(OUTPUT_PT) \
+ case TYPE_##OUTPUT_PT: { \
+ using OutputCppType = typename PrimitiveTypeTraits<TYPE_##OUTPUT_PT>::CppType; \
+ auto projected_value = [](const ColumnPtr& column, size_t row) -> OutputCppType { \
+ Field field = (*column)[row]; \
+ return field.get<TYPE_##OUTPUT_PT>(); \
+ }; \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& orig_boundary = orig_boundaries[selected_boundaries[i].first]; \
+ TTargetExprMonotonicity::type direction = selected_boundaries[i].second; \
+ if (boundary_is_list[i]) { \
+ auto cvr = ColumnValueRange<TYPE_##OUTPUT_PT>::create_empty_column_value_range( \
+ out_nullable, out_precision, out_scale); \
+ bool list_has_value = false; \
+ bool list_has_null = false; \
+ for (size_t row = list_result_begin[i]; row < list_result_end[i]; ++row) { \
+ if (list_result_col->is_null_at(row)) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ auto projected_list_value = projected_value(list_result_col, row); \
+ static_cast<void>(cvr.add_fixed_value(projected_list_value)); \
+ list_has_value = true; \
+ } \
+ if (!list_has_value && !list_has_null) { \
+ continue; \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.contains_null = list_has_null; \
+ projected_boundary.only_null = list_has_null && !list_has_value; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ continue; \
+ } \
+ if ((!lo_open[i] && lo_result_col->is_null_at(lo_result_row[i])) || \
+ (!hi_open[i] && hi_result_col->is_null_at(hi_result_row[i]))) { \
+ continue; \
+ } \
+ ColumnValueRange<TYPE_##OUTPUT_PT> cvr("", out_nullable, out_precision, out_scale); \
+ /* Use closed projected finite bounds as a conservative envelope for */ \
+ /* non-strict monotonic expressions such as date ceil/floor/cast. */ \
+ if (direction == TTargetExprMonotonicity::MONOTONIC_DECREASING) { \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, hi_projected)); \
+ } \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, lo_projected)); \
+ } \
+ } else { \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, lo_projected)); \
+ } \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, hi_projected)); \
+ } \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.only_null = orig_boundary.only_null; \
+ projected_boundary.contains_null = orig_boundary.contains_null; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ } \
+ output_built = true; \
+ break; \
+ }
+
+ switch (output_ptype) {
+ BUILD_PROJECTED_CVR(TINYINT)
+ BUILD_PROJECTED_CVR(SMALLINT)
+ BUILD_PROJECTED_CVR(INT)
+ BUILD_PROJECTED_CVR(BIGINT)
+ BUILD_PROJECTED_CVR(LARGEINT)
+ BUILD_PROJECTED_CVR(FLOAT)
+ BUILD_PROJECTED_CVR(DOUBLE)
+ BUILD_PROJECTED_CVR(CHAR)
+ BUILD_PROJECTED_CVR(DATE)
+ BUILD_PROJECTED_CVR(DATETIME)
+ BUILD_PROJECTED_CVR(DATEV2)
+ BUILD_PROJECTED_CVR(DATETIMEV2)
+ BUILD_PROJECTED_CVR(TIMESTAMPTZ)
+ BUILD_PROJECTED_CVR(VARCHAR)
+ BUILD_PROJECTED_CVR(STRING)
+ BUILD_PROJECTED_CVR(DECIMAL32)
+ BUILD_PROJECTED_CVR(DECIMAL64)
+ BUILD_PROJECTED_CVR(DECIMAL128I)
+ BUILD_PROJECTED_CVR(DECIMAL256)
+ BUILD_PROJECTED_CVR(DECIMALV2)
+ BUILD_PROJECTED_CVR(BOOLEAN)
+ BUILD_PROJECTED_CVR(IPV4)
+ BUILD_PROJECTED_CVR(IPV6)
+ default:
+ break;
+ }
+
+#undef BUILD_PROJECTED_CVR
+
+ DORIS_CHECK(output_built);
+
+ return store_projected_boundaries(std::move(projected));
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static SQLFilterOp convert_opcode_to_filter_op(TExprOpcode::type op) {
+ switch (op) {
+ case TExprOpcode::LE:
+ return FILTER_LESS_OR_EQUAL;
+ case TExprOpcode::LT:
+ return FILTER_LESS;
+ case TExprOpcode::GE:
+ return FILTER_LARGER_OR_EQUAL;
+ case TExprOpcode::GT:
+ return FILTER_LARGER;
+ default:
+ return FILTER_IN; // sentinel: caller should skip
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+void RuntimeFilterPartitionPruner::_try_prune_by_single_rf(
+ const std::vector<ParsedBoundary>& boundaries, const VExprSPtr& impl,
+ phmap::flat_hash_set<int64_t>& newly_pruned) {
+ // Pre-compute whether the RF "matches NULL" -- i.e. whether the RF would
+ // accept a row whose probe value is NULL. The signal differs by RF impl:
+ // * IN filter → HybridSet::contain_null()
+ // * Bloom filter → BloomFilterFuncBase::contain_null()
+ // * MinMax filter → encoded via the BinaryPredicate node_type:
+ // NULL_AWARE_BINARY_PRED means the build side had a
+ // NULL key and a null-safe equal join asked NULL to
+ // match NULL; plain BINARY_PRED never matches NULL.
+ // FilterBase::contain_null() already folds in `_null_aware`, so we only
+ // get a true result when the build side is actually null-aware AND
+ // produced a NULL value.
+ bool rf_contains_null = false;
+ if (auto hybrid_set = impl->get_set_func()) {
+ rf_contains_null = hybrid_set->contain_null();
+ } else if (impl->node_type() == TExprNodeType::BLOOM_PRED) {
+ auto bloom = impl->get_bloom_filter_func();
+ rf_contains_null = bloom && bloom->contain_null();
+ } else if (impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) {
+ // Min/Max RF built on a null-safe equal join. The literal child holds
+ // the min or max bound; the NULL semantic is conveyed by the node
+ // type itself (see create_vbin_predicate in runtime_filter/utils.cpp).
+ rf_contains_null = true;
+ }
+
+ std::optional<ColumnValueRangeType> rf_cvr;
+ if (!boundaries.empty()) {
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+ using CppType = typename CvrType::CppType;
+
+ auto hybrid_set = impl->get_set_func();
+ if (hybrid_set) {
+ auto typed_rf_cvr = CvrType::create_empty_column_value_range(
+ boundaries.front().is_nullable, boundary_cvr.precision(),
+ boundary_cvr.scale());
+ auto* iter = hybrid_set->begin();
+ while (iter->has_next()) {
+ const void* value = iter->get_value();
+ if (value) {
+ if constexpr (std::is_same_v<CppType, String>) {
+ // String HybridSets store StringRef*, but
+ // ColumnValueRange<String>::CppType is
+ // std::string. Construct from the bytes.
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else if constexpr (std::is_same_v<CppType, StringRef>) {
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else {
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ *reinterpret_cast<const CppType*>(value)));
+ }
+ }
+ iter->next();
+ }
+ rf_cvr.emplace(std::move(typed_rf_cvr));
+ } else if ((impl->node_type() == TExprNodeType::BINARY_PRED ||
+ impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) &&
+ impl->children().size() == 2 && impl->children()[1]->is_literal()) {
+ // MinMax filter: binary pred with literal bound
+ auto* literal = assert_cast<VLiteral*>(impl->children()[1].get());
+ auto col_ptr = literal->get_column_ptr();
+ auto data = col_ptr->get_data_at(0);
+ CppType val {};
+ if constexpr (std::is_same_v<CppType, String>) {
+ // get_data_at returns a StringRef pointing at the
+ // raw character bytes; ColumnValueRange<String>'s
+ // CppType is std::string, so construct one from
+ // the bytes rather than dereferencing as String.
+ val = CppType(data.data, data.size);
+ } else if constexpr (std::is_same_v<CppType, StringRef>) {
+ val = CppType(data.data, data.size);
+ } else {
+ val = *reinterpret_cast<const CppType*>(data.data);
+ }
+
+ SQLFilterOp op = convert_opcode_to_filter_op(impl->op());
+ if (op == FILTER_IN) {
+ return; // unrecognized opcode, skip
+ }
+
+ CvrType typed_rf_cvr(boundary_cvr.column_name(),
+ boundaries.front().is_nullable,
+ boundary_cvr.precision(), boundary_cvr.scale());
+ static_cast<void>(typed_rf_cvr.add_range(op, val));
+ rf_cvr.emplace(std::move(typed_rf_cvr));
+ }
+ },
+ boundaries.front().boundary_cvr);
+ }
+ if (!rf_cvr.has_value()) {
+ return;
+ }
+
+ for (const auto& pb : boundaries) {
+ if (_pruned_partition_ids.contains(pb.partition_id) ||
+ newly_pruned.contains(pb.partition_id)) {
+ continue;
+ }
+
+ // NULL handling:
+ // A partition row whose key is NULL matches the RF iff `rf_contains_null`.
+ // - only_null partition (rows are exclusively NULL): prunable iff !rf_contains_null.
+ // - mixed (NULL + concrete values): if rf_contains_null, NULL rows alone
+ // prevent pruning. Otherwise NULL rows can never match, so we ignore
+ // the NULL portion and let the regular non-NULL intersection decide.
+ if (pb.only_null) {
+ if (!rf_contains_null) {
+ newly_pruned.insert(pb.partition_id);
+ }
+ continue;
+ }
+ if (pb.contains_null && rf_contains_null) {
+ continue;
+ }
+
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+
+ const auto* typed_rf_cvr = std::get_if<CvrType>(&rf_cvr.value());
+ DORIS_CHECK(typed_rf_cvr != nullptr);
+
+ auto boundary_copy = boundary_cvr;
+ auto rf_cvr_copy = *typed_rf_cvr;
+ boundary_copy.intersection(rf_cvr_copy);
+ if (boundary_copy.is_empty_value_range()) {
+ newly_pruned.insert(pb.partition_id);
+ }
+ },
+ pb.boundary_cvr);
+ }
+}
+
+Status RuntimeFilterPartitionPruner::prune_by_runtime_filters(
+ const ParsedPartitionBoundaries& parsed, const VExprContextSPtrs& conjuncts,
+ const std::vector<TRuntimeFilterDesc>& rf_descs, int scan_node_id,
+ int64_t* newly_pruned_count) {
+ *newly_pruned_count = 0;
+ if (parsed.empty()) {
+ return Status::OK();
+ }
+ const auto& slot_to_boundaries = parsed.slot_to_boundaries();
+
+ // Build filter_id -> per-partition monotonicity maps.
+ std::unordered_map<int, std::unordered_map<int64_t, TTargetExprMonotonicity::type>>
+ filter_id_to_partition_monotonicity;
+ for (const auto& desc : rf_descs) {
+ if (desc.__isset.planId_to_partition_target_monotonicity) {
+ auto it = desc.planId_to_partition_target_monotonicity.find(scan_node_id);
+ if (it != desc.planId_to_partition_target_monotonicity.end()) {
+ auto& partition_monotonicity = filter_id_to_partition_monotonicity[desc.filter_id];
+ for (const auto& partition_entry : it->second) {
+ DORIS_CHECK(partition_entry.__isset.partition_id);
+ DORIS_CHECK(partition_entry.__isset.monotonicity);
+ DORIS_CHECK(partition_entry.monotonicity !=
+ TTargetExprMonotonicity::NON_MONOTONIC);
+ auto [_, inserted] = partition_monotonicity.emplace(
+ partition_entry.partition_id, partition_entry.monotonicity);
+ DORIS_CHECK(inserted);
+ }
+ }
+ }
+ }
+
+ // This function is serialized by _conjuncts_lock in the caller, so our reads
+ // of _pruned_partition_ids never race with our writes below. The only concurrent
+ // readers are is_partition_pruned() calls (under shared_lock), which are
+ // properly synchronized by the unique_lock we take when inserting.
+ phmap::flat_hash_set<int64_t> newly_pruned;
+
+ for (const auto& conjunct_ctx : conjuncts) {
+ VExprSPtr root = conjunct_ctx->root();
+ if (!root->is_rf_wrapper()) {
+ continue;
+ }
+
+ VExprSPtr impl = root->get_impl();
+ if (!impl) {
+ continue;
+ }
+
+ if (impl->children().empty()) {
+ continue;
+ }
+
+ auto* wrapper_root = assert_cast<RuntimeFilterExpr*>(root.get());
Review Comment:
static_cast, already check root->is_rf_wrapper()
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,866 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ DORIS_CHECK(tb.__isset.partition_id);
+ DORIS_CHECK(tb.__isset.slot_id);
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ DORIS_CHECK(slot_it != slot_descs.end());
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ DORIS_CHECK(is_list || is_range); \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ bool range_has_null = false; \
+ if (is_nullable && !tb.__isset.range_start) { \
+ range_has_null = true; \
+ } \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } else if (is_nullable) { \
+ range_has_null = true; \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ if (range_has_null) { \
+ boundary.contains_null = true; \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ DORIS_CHECK(parsed_ok);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static bool find_unique_slot_ref_impl(const VExpr* expr, const VSlotRef** found) {
+ if (!expr) {
+ return true;
+ }
+ if (expr->is_slot_ref()) {
+ const auto* slot = assert_cast<const VSlotRef*>(expr);
+ if (*found == nullptr) {
+ *found = slot;
+ return true;
+ }
+ if ((*found)->slot_id() != slot->slot_id()) {
+ return false;
+ }
+ DORIS_CHECK((*found)->column_id() == slot->column_id());
+ return true;
+ }
+ for (const auto& child : expr->children()) {
+ if (!find_unique_slot_ref_impl(child.get(), found)) {
+ return false;
+ }
+ }
+ return true;
+}
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ const VSlotRef* found = nullptr;
+ if (!find_unique_slot_ref_impl(expr, &found)) {
+ return nullptr;
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+Status ParsedPartitionBoundaries::get_or_compute_projected_boundaries(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ const std::unordered_map<int64_t, TTargetExprMonotonicity::type>& partition_directions,
+ VExprContext* ctx, std::shared_ptr<const std::vector<ParsedBoundary>>* output) const {
+ {
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto it = _projected_boundaries_by_filter_id.find(filter_id);
+ if (it != _projected_boundaries_by_filter_id.end()) {
+ *output = it->second;
+ return Status::OK();
+ }
+ }
+
+ std::vector<ParsedBoundary> projected;
+ auto store_projected_boundaries = [&](std::vector<ParsedBoundary>&& boundaries) {
+ auto cached = std::make_shared<const std::vector<ParsedBoundary>>(std::move(boundaries));
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto [it, inserted] = _projected_boundaries_by_filter_id.emplace(filter_id, cached);
+ *output = inserted ? cached : it->second;
+ return Status::OK();
+ };
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::unordered_set<int64_t> boundary_partition_ids;
+ boundary_partition_ids.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ boundary_partition_ids.insert(boundary.partition_id);
+ }
+ for (const auto& [partition_id, direction] : partition_directions) {
+ DORIS_CHECK(direction != TTargetExprMonotonicity::NON_MONOTONIC);
+ DORIS_CHECK(boundary_partition_ids.contains(partition_id));
+ }
+
+ if (target_expr->is_slot_ref()) {
+ auto* slot_ref = assert_cast<VSlotRef*>(target_expr.get());
+ DORIS_CHECK(slot_ref->slot_id() == leaf_slot_id);
+ std::vector<ParsedBoundary> slot_boundaries;
+ slot_boundaries.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ auto direction_it = partition_directions.find(boundary.partition_id);
+ if (direction_it == partition_directions.end()) {
+ continue;
+ }
+ DORIS_CHECK(direction_it->second == TTargetExprMonotonicity::MONOTONIC_INCREASING);
+ slot_boundaries.emplace_back(boundary);
+ }
+ return store_projected_boundaries(std::move(slot_boundaries));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> selected_boundaries;
+ selected_boundaries.reserve(orig_boundaries.size());
+ for (size_t i = 0; i < orig_boundaries.size(); ++i) {
+ auto it = partition_directions.find(orig_boundaries[i].partition_id);
+ if (it != partition_directions.end() &&
+ it->second != TTargetExprMonotonicity::NON_MONOTONIC) {
+ selected_boundaries.emplace_back(i, it->second);
+ }
+ }
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> projectable_boundaries;
+ projectable_boundaries.reserve(selected_boundaries.size());
+ for (const auto& selected_boundary : selected_boundaries) {
+ const auto& boundary = orig_boundaries[selected_boundary.first];
+ if (!boundary.only_null && !boundary.contains_null) {
+ projectable_boundaries.emplace_back(selected_boundary);
+ }
+ }
+ selected_boundaries.swap(projectable_boundaries);
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ DORIS_CHECK(slot_type_it != _slot_data_types.end());
+
+ const DataTypePtr& input_type = slot_type_it->second;
+ bool input_is_nullable = input_type->is_nullable();
+ DataTypePtr inner_type = input_is_nullable ? remove_nullable(input_type) : input_type;
+ PrimitiveType input_ptype = input_type->get_primitive_type();
+
+ size_t N = selected_boundaries.size();
+ std::vector<bool> lo_open(N, false);
+ std::vector<bool> hi_open(N, false);
+ std::vector<bool> boundary_is_list(N, false);
+ std::vector<size_t> lo_result_row(N, 0);
+ std::vector<size_t> hi_result_row(N, 0);
+ std::vector<size_t> list_result_begin(N, 0);
+ std::vector<size_t> list_result_end(N, 0);
+ size_t lo_row_count = 0;
+ size_t hi_row_count = 0;
+ size_t list_row_count = 0;
+
+ // Build input blocks for RANGE endpoints and LIST values.
+ Block lo_block;
+ Block hi_block;
+ Block list_block;
+
+ // Pad columns 0..leaf_column_id-1 with placeholder columns so the leaf
+ // VSlotRef (whose column_id is leaf_column_id) reads our typed column.
+ auto add_dummy_columns = [&](Block& block, size_t row_count) {
+ for (int col_idx = 0; col_idx < leaf_column_id; ++col_idx) {
+ auto col = ColumnUInt8::create(row_count, 0);
+ block.insert({std::move(col), std::make_shared<DataTypeUInt8>(),
+ fmt::format("dummy_{}", col_idx)});
+ }
+ };
+
+ // Macro-dispatch on input PrimitiveType to build finite RANGE endpoint
+ // columns and LIST value columns. Unbounded RANGE endpoints are not executed
+ // through target_expr; they stay unbounded after projection and swap sides
+ // for monotonic decreasing targets.
+ bool input_built = false;
+#define BUILD_INPUT_COLUMNS(INPUT_PT) \
+ case TYPE_##INPUT_PT: { \
+ using InCol = typename PrimitiveTypeTraits<TYPE_##INPUT_PT>::ColumnType; \
+ MutableColumnPtr lo_inner_base = inner_type->create_column(); \
+ MutableColumnPtr hi_inner_base = inner_type->create_column(); \
+ MutableColumnPtr list_inner_base = inner_type->create_column(); \
+ auto* lo_inner = assert_cast<InCol*>(lo_inner_base.get()); \
+ auto* hi_inner = assert_cast<InCol*>(hi_inner_base.get()); \
+ auto* list_inner = assert_cast<InCol*>(list_inner_base.get()); \
+ lo_inner->reserve(N); \
+ hi_inner->reserve(N); \
+ list_inner->reserve(N); \
+ auto lo_nulls = ColumnUInt8::create(); \
+ auto hi_nulls = ColumnUInt8::create(); \
+ auto list_nulls = ColumnUInt8::create(); \
+ lo_nulls->reserve(N); \
+ hi_nulls->reserve(N); \
+ list_nulls->reserve(N); \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& boundary = orig_boundaries[selected_boundaries[i].first]; \
+ const auto* cvr = \
+ std::get_if<ColumnValueRange<TYPE_##INPUT_PT>>(&boundary.boundary_cvr); \
+ DCHECK(cvr != nullptr); \
+ DCHECK(!boundary.only_null); \
+ DCHECK(!boundary.contains_null); \
+ boundary_is_list[i] = cvr->is_fixed_value_range(); \
+ if (boundary_is_list[i]) { \
+ list_result_begin[i] = list_row_count; \
+ for (const auto& value : cvr->get_fixed_value_set()) { \
+ list_inner->insert_value(value); \
+ list_nulls->get_data().push_back(0); \
+ ++list_row_count; \
+ } \
+ list_result_end[i] = list_row_count; \
+ continue; \
+ } \
+ lo_open[i] = cvr->is_low_value_minimum(); \
+ hi_open[i] = cvr->is_high_value_maximum(); \
+ if (!lo_open[i]) { \
+ lo_result_row[i] = lo_row_count++; \
+ lo_inner->insert_value(cvr->get_range_min_value()); \
+ lo_nulls->get_data().push_back(0); \
+ } \
+ if (!hi_open[i]) { \
+ hi_result_row[i] = hi_row_count++; \
+ hi_inner->insert_value(cvr->get_range_max_value()); \
+ hi_nulls->get_data().push_back(0); \
+ } \
+ } \
+ add_dummy_columns(lo_block, lo_row_count); \
+ add_dummy_columns(hi_block, hi_row_count); \
+ add_dummy_columns(list_block, list_row_count); \
+ if (input_is_nullable) { \
+ auto lo_col = ColumnNullable::create(std::move(lo_inner_base), std::move(lo_nulls)); \
+ auto hi_col = ColumnNullable::create(std::move(hi_inner_base), std::move(hi_nulls)); \
+ auto list_col = \
+ ColumnNullable::create(std::move(list_inner_base), std::move(list_nulls)); \
+ lo_block.insert({std::move(lo_col), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_col), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_col), input_type, "leaf_slot"}); \
+ } else { \
+ lo_block.insert({std::move(lo_inner_base), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_inner_base), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_inner_base), input_type, "leaf_slot"}); \
+ } \
+ input_built = true; \
+ break; \
+ }
+
+ switch (input_ptype) {
+ BUILD_INPUT_COLUMNS(TINYINT)
+ BUILD_INPUT_COLUMNS(SMALLINT)
+ BUILD_INPUT_COLUMNS(INT)
+ BUILD_INPUT_COLUMNS(BIGINT)
+ BUILD_INPUT_COLUMNS(LARGEINT)
+ BUILD_INPUT_COLUMNS(FLOAT)
+ BUILD_INPUT_COLUMNS(DOUBLE)
+ BUILD_INPUT_COLUMNS(CHAR)
+ BUILD_INPUT_COLUMNS(DATE)
+ BUILD_INPUT_COLUMNS(DATETIME)
+ BUILD_INPUT_COLUMNS(DATEV2)
+ BUILD_INPUT_COLUMNS(DATETIMEV2)
+ BUILD_INPUT_COLUMNS(TIMESTAMPTZ)
+ BUILD_INPUT_COLUMNS(VARCHAR)
+ BUILD_INPUT_COLUMNS(STRING)
+ BUILD_INPUT_COLUMNS(DECIMAL32)
+ BUILD_INPUT_COLUMNS(DECIMAL64)
+ BUILD_INPUT_COLUMNS(DECIMAL128I)
+ BUILD_INPUT_COLUMNS(DECIMAL256)
+ BUILD_INPUT_COLUMNS(DECIMALV2)
+ BUILD_INPUT_COLUMNS(BOOLEAN)
+ BUILD_INPUT_COLUMNS(IPV4)
+ BUILD_INPUT_COLUMNS(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_INPUT_COLUMNS
+
+ DORIS_CHECK(input_built);
+
+ int lo_result_id = -1;
+ int hi_result_id = -1;
+ int list_result_id = -1;
+ ColumnPtr lo_result_col;
+ ColumnPtr hi_result_col;
+ ColumnPtr list_result_col;
+ if (lo_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &lo_block, &lo_result_id));
+ DORIS_CHECK(lo_result_id >= 0);
+ lo_result_col = lo_block.get_by_position(lo_result_id).column;
+ }
+ if (hi_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &hi_block, &hi_result_id));
+ DORIS_CHECK(hi_result_id >= 0);
+ hi_result_col = hi_block.get_by_position(hi_result_id).column;
+ }
+ if (list_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &list_block, &list_result_id));
+ DORIS_CHECK(list_result_id >= 0);
+ list_result_col = list_block.get_by_position(list_result_id).column;
+ }
+ int out_precision = cast_set<int>(target_expr->data_type()->get_precision());
+ int out_scale = cast_set<int>(target_expr->data_type()->get_scale());
+ bool out_nullable = target_expr->data_type()->is_nullable();
+ PrimitiveType output_ptype = target_expr->data_type()->get_primitive_type();
+
+ projected.reserve(N);
+
+ bool output_built = false;
+#define BUILD_PROJECTED_CVR(OUTPUT_PT) \
+ case TYPE_##OUTPUT_PT: { \
+ using OutputCppType = typename PrimitiveTypeTraits<TYPE_##OUTPUT_PT>::CppType; \
+ auto projected_value = [](const ColumnPtr& column, size_t row) -> OutputCppType { \
+ Field field = (*column)[row]; \
+ return field.get<TYPE_##OUTPUT_PT>(); \
+ }; \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& orig_boundary = orig_boundaries[selected_boundaries[i].first]; \
+ TTargetExprMonotonicity::type direction = selected_boundaries[i].second; \
+ if (boundary_is_list[i]) { \
+ auto cvr = ColumnValueRange<TYPE_##OUTPUT_PT>::create_empty_column_value_range( \
+ out_nullable, out_precision, out_scale); \
+ bool list_has_value = false; \
+ bool list_has_null = false; \
+ for (size_t row = list_result_begin[i]; row < list_result_end[i]; ++row) { \
+ if (list_result_col->is_null_at(row)) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ auto projected_list_value = projected_value(list_result_col, row); \
+ static_cast<void>(cvr.add_fixed_value(projected_list_value)); \
+ list_has_value = true; \
+ } \
+ if (!list_has_value && !list_has_null) { \
+ continue; \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.contains_null = list_has_null; \
+ projected_boundary.only_null = list_has_null && !list_has_value; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ continue; \
+ } \
+ if ((!lo_open[i] && lo_result_col->is_null_at(lo_result_row[i])) || \
+ (!hi_open[i] && hi_result_col->is_null_at(hi_result_row[i]))) { \
+ continue; \
+ } \
+ ColumnValueRange<TYPE_##OUTPUT_PT> cvr("", out_nullable, out_precision, out_scale); \
+ /* Use closed projected finite bounds as a conservative envelope for */ \
+ /* non-strict monotonic expressions such as date ceil/floor/cast. */ \
+ if (direction == TTargetExprMonotonicity::MONOTONIC_DECREASING) { \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, hi_projected)); \
+ } \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, lo_projected)); \
+ } \
+ } else { \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, lo_projected)); \
+ } \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, hi_projected)); \
+ } \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.only_null = orig_boundary.only_null; \
+ projected_boundary.contains_null = orig_boundary.contains_null; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ } \
+ output_built = true; \
+ break; \
+ }
+
+ switch (output_ptype) {
+ BUILD_PROJECTED_CVR(TINYINT)
+ BUILD_PROJECTED_CVR(SMALLINT)
+ BUILD_PROJECTED_CVR(INT)
+ BUILD_PROJECTED_CVR(BIGINT)
+ BUILD_PROJECTED_CVR(LARGEINT)
+ BUILD_PROJECTED_CVR(FLOAT)
+ BUILD_PROJECTED_CVR(DOUBLE)
+ BUILD_PROJECTED_CVR(CHAR)
+ BUILD_PROJECTED_CVR(DATE)
+ BUILD_PROJECTED_CVR(DATETIME)
+ BUILD_PROJECTED_CVR(DATEV2)
+ BUILD_PROJECTED_CVR(DATETIMEV2)
+ BUILD_PROJECTED_CVR(TIMESTAMPTZ)
+ BUILD_PROJECTED_CVR(VARCHAR)
+ BUILD_PROJECTED_CVR(STRING)
+ BUILD_PROJECTED_CVR(DECIMAL32)
+ BUILD_PROJECTED_CVR(DECIMAL64)
+ BUILD_PROJECTED_CVR(DECIMAL128I)
+ BUILD_PROJECTED_CVR(DECIMAL256)
+ BUILD_PROJECTED_CVR(DECIMALV2)
+ BUILD_PROJECTED_CVR(BOOLEAN)
+ BUILD_PROJECTED_CVR(IPV4)
+ BUILD_PROJECTED_CVR(IPV6)
+ default:
+ break;
+ }
+
+#undef BUILD_PROJECTED_CVR
+
+ DORIS_CHECK(output_built);
+
+ return store_projected_boundaries(std::move(projected));
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static SQLFilterOp convert_opcode_to_filter_op(TExprOpcode::type op) {
+ switch (op) {
+ case TExprOpcode::LE:
+ return FILTER_LESS_OR_EQUAL;
+ case TExprOpcode::LT:
+ return FILTER_LESS;
+ case TExprOpcode::GE:
+ return FILTER_LARGER_OR_EQUAL;
+ case TExprOpcode::GT:
+ return FILTER_LARGER;
+ default:
+ return FILTER_IN; // sentinel: caller should skip
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+void RuntimeFilterPartitionPruner::_try_prune_by_single_rf(
+ const std::vector<ParsedBoundary>& boundaries, const VExprSPtr& impl,
+ phmap::flat_hash_set<int64_t>& newly_pruned) {
+ // Pre-compute whether the RF "matches NULL" -- i.e. whether the RF would
+ // accept a row whose probe value is NULL. The signal differs by RF impl:
+ // * IN filter → HybridSet::contain_null()
+ // * Bloom filter → BloomFilterFuncBase::contain_null()
+ // * MinMax filter → encoded via the BinaryPredicate node_type:
+ // NULL_AWARE_BINARY_PRED means the build side had a
+ // NULL key and a null-safe equal join asked NULL to
+ // match NULL; plain BINARY_PRED never matches NULL.
+ // FilterBase::contain_null() already folds in `_null_aware`, so we only
+ // get a true result when the build side is actually null-aware AND
+ // produced a NULL value.
+ bool rf_contains_null = false;
+ if (auto hybrid_set = impl->get_set_func()) {
+ rf_contains_null = hybrid_set->contain_null();
+ } else if (impl->node_type() == TExprNodeType::BLOOM_PRED) {
+ auto bloom = impl->get_bloom_filter_func();
+ rf_contains_null = bloom && bloom->contain_null();
+ } else if (impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) {
+ // Min/Max RF built on a null-safe equal join. The literal child holds
+ // the min or max bound; the NULL semantic is conveyed by the node
+ // type itself (see create_vbin_predicate in runtime_filter/utils.cpp).
+ rf_contains_null = true;
+ }
+
+ std::optional<ColumnValueRangeType> rf_cvr;
+ if (!boundaries.empty()) {
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+ using CppType = typename CvrType::CppType;
+
+ auto hybrid_set = impl->get_set_func();
+ if (hybrid_set) {
+ auto typed_rf_cvr = CvrType::create_empty_column_value_range(
+ boundaries.front().is_nullable, boundary_cvr.precision(),
+ boundary_cvr.scale());
+ auto* iter = hybrid_set->begin();
+ while (iter->has_next()) {
+ const void* value = iter->get_value();
+ if (value) {
+ if constexpr (std::is_same_v<CppType, String>) {
+ // String HybridSets store StringRef*, but
+ // ColumnValueRange<String>::CppType is
+ // std::string. Construct from the bytes.
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else if constexpr (std::is_same_v<CppType, StringRef>) {
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else {
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ *reinterpret_cast<const CppType*>(value)));
+ }
+ }
+ iter->next();
+ }
+ rf_cvr.emplace(std::move(typed_rf_cvr));
+ } else if ((impl->node_type() == TExprNodeType::BINARY_PRED ||
+ impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) &&
+ impl->children().size() == 2 && impl->children()[1]->is_literal()) {
+ // MinMax filter: binary pred with literal bound
+ auto* literal = assert_cast<VLiteral*>(impl->children()[1].get());
+ auto col_ptr = literal->get_column_ptr();
+ auto data = col_ptr->get_data_at(0);
+ CppType val {};
+ if constexpr (std::is_same_v<CppType, String>) {
+ // get_data_at returns a StringRef pointing at the
+ // raw character bytes; ColumnValueRange<String>'s
+ // CppType is std::string, so construct one from
+ // the bytes rather than dereferencing as String.
+ val = CppType(data.data, data.size);
+ } else if constexpr (std::is_same_v<CppType, StringRef>) {
+ val = CppType(data.data, data.size);
+ } else {
+ val = *reinterpret_cast<const CppType*>(data.data);
+ }
+
+ SQLFilterOp op = convert_opcode_to_filter_op(impl->op());
+ if (op == FILTER_IN) {
+ return; // unrecognized opcode, skip
+ }
+
+ CvrType typed_rf_cvr(boundary_cvr.column_name(),
+ boundaries.front().is_nullable,
+ boundary_cvr.precision(), boundary_cvr.scale());
+ static_cast<void>(typed_rf_cvr.add_range(op, val));
+ rf_cvr.emplace(std::move(typed_rf_cvr));
+ }
+ },
+ boundaries.front().boundary_cvr);
+ }
+ if (!rf_cvr.has_value()) {
+ return;
+ }
+
+ for (const auto& pb : boundaries) {
+ if (_pruned_partition_ids.contains(pb.partition_id) ||
+ newly_pruned.contains(pb.partition_id)) {
+ continue;
+ }
+
+ // NULL handling:
+ // A partition row whose key is NULL matches the RF iff `rf_contains_null`.
+ // - only_null partition (rows are exclusively NULL): prunable iff !rf_contains_null.
+ // - mixed (NULL + concrete values): if rf_contains_null, NULL rows alone
+ // prevent pruning. Otherwise NULL rows can never match, so we ignore
+ // the NULL portion and let the regular non-NULL intersection decide.
+ if (pb.only_null) {
+ if (!rf_contains_null) {
+ newly_pruned.insert(pb.partition_id);
+ }
+ continue;
+ }
+ if (pb.contains_null && rf_contains_null) {
+ continue;
+ }
+
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+
+ const auto* typed_rf_cvr = std::get_if<CvrType>(&rf_cvr.value());
+ DORIS_CHECK(typed_rf_cvr != nullptr);
+
+ auto boundary_copy = boundary_cvr;
+ auto rf_cvr_copy = *typed_rf_cvr;
+ boundary_copy.intersection(rf_cvr_copy);
+ if (boundary_copy.is_empty_value_range()) {
+ newly_pruned.insert(pb.partition_id);
+ }
+ },
+ pb.boundary_cvr);
+ }
+}
+
+Status RuntimeFilterPartitionPruner::prune_by_runtime_filters(
+ const ParsedPartitionBoundaries& parsed, const VExprContextSPtrs& conjuncts,
+ const std::vector<TRuntimeFilterDesc>& rf_descs, int scan_node_id,
+ int64_t* newly_pruned_count) {
+ *newly_pruned_count = 0;
+ if (parsed.empty()) {
+ return Status::OK();
+ }
+ const auto& slot_to_boundaries = parsed.slot_to_boundaries();
+
+ // Build filter_id -> per-partition monotonicity maps.
+ std::unordered_map<int, std::unordered_map<int64_t, TTargetExprMonotonicity::type>>
+ filter_id_to_partition_monotonicity;
+ for (const auto& desc : rf_descs) {
+ if (desc.__isset.planId_to_partition_target_monotonicity) {
+ auto it = desc.planId_to_partition_target_monotonicity.find(scan_node_id);
+ if (it != desc.planId_to_partition_target_monotonicity.end()) {
+ auto& partition_monotonicity = filter_id_to_partition_monotonicity[desc.filter_id];
+ for (const auto& partition_entry : it->second) {
+ DORIS_CHECK(partition_entry.__isset.partition_id);
+ DORIS_CHECK(partition_entry.__isset.monotonicity);
+ DORIS_CHECK(partition_entry.monotonicity !=
+ TTargetExprMonotonicity::NON_MONOTONIC);
+ auto [_, inserted] = partition_monotonicity.emplace(
+ partition_entry.partition_id, partition_entry.monotonicity);
+ DORIS_CHECK(inserted);
+ }
+ }
+ }
+ }
+
+ // This function is serialized by _conjuncts_lock in the caller, so our reads
+ // of _pruned_partition_ids never race with our writes below. The only concurrent
+ // readers are is_partition_pruned() calls (under shared_lock), which are
+ // properly synchronized by the unique_lock we take when inserting.
+ phmap::flat_hash_set<int64_t> newly_pruned;
+
+ for (const auto& conjunct_ctx : conjuncts) {
+ VExprSPtr root = conjunct_ctx->root();
+ if (!root->is_rf_wrapper()) {
+ continue;
+ }
+
+ VExprSPtr impl = root->get_impl();
+ if (!impl) {
+ continue;
+ }
+
+ if (impl->children().empty()) {
+ continue;
+ }
+
+ auto* wrapper_root = assert_cast<RuntimeFilterExpr*>(root.get());
Review Comment:
static_cast, already check root->is_rf_wrapper()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4494441476
# FE UT Coverage Report
Increment line coverage ` 30.98% (92/297)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4496646451
# BE UT Coverage Report
Increment line coverage `62.92% (504/801)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.58% (20758/38741) |
| Line Coverage | 37.28% (196810/527979) |
| Region Coverage | 33.55% (154907/461676) |
| Branch Coverage | 34.63% (67810/195795) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4592142245
# FE UT Coverage Report
Increment line coverage ` 31.37% (96/306)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_b82194ce0fb5ac216aeed034490645f04c19f4e5/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_b82194ce0fb5ac216aeed034490645f04c19f4e5/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4560149031
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4429384142
<details>
<summary>TPC-H: <b>Total hot run time: 30022 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2391ea7f407dcc42f8f4ddc3bcd8e0838059d15c, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17712 4007 4041 4007
q2 q3 10796 878 622 622
q4 4709 469 344 344
q5 8187 1339 1134 1134
q6 362 167 138 138
q7 933 970 748 748
q8 11213 1408 1273 1273
q9 7448 5645 5428 5428
q10 6447 2079 1812 1812
q11 481 265 261 261
q12 688 421 291 291
q13 18224 3421 2756 2756
q14 290 286 262 262
q15 q16 903 870 793 793
q17 996 1016 790 790
q18 6559 5837 5729 5729
q19 1207 1389 1105 1105
q20 538 410 275 275
q21 4876 2313 1913 1913
q22 438 385 341 341
Total cold run time: 103007 ms
Total hot run time: 30022 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4828 4733 4798 4733
q2 q3 4670 4795 4201 4201
q4 2161 2217 1402 1402
q5 5033 5057 5216 5057
q6 196 171 133 133
q7 2046 1817 1855 1817
q8 3355 3152 3165 3152
q9 8510 8482 8559 8482
q10 4515 4564 4265 4265
q11 601 438 414 414
q12 705 756 535 535
q13 3348 3609 2923 2923
q14 314 308 287 287
q15 q16 778 788 690 690
q17 1341 1320 1307 1307
q18 8002 7115 7186 7115
q19 1173 1151 1132 1132
q20 2216 2206 1940 1940
q21 6198 5470 4922 4922
q22 527 476 398 398
Total cold run time: 60517 ms
Total hot run time: 54905 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4429111858
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4429112570
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4447561911
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4450215166
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4505871031
# BE Regression && UT Coverage Report
Increment line coverage `75.28% (603/801)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.63% (27894/37883) |
| Line Coverage | 57.53% (303104/526849) |
| Region Coverage | 54.44% (253916/466433) |
| Branch Coverage | 55.92% (110043/196776) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4495241204
<details>
<summary>TPC-H: <b>Total hot run time: 31123 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6be29ffaf962c85e6c243c68a1c26cf413cdb0b5, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17610 3896 3926 3896
q2 q3 11000 1435 807 807
q4 4813 481 355 355
q5 10591 2271 2133 2133
q6 412 180 138 138
q7 956 792 641 641
q8 9611 1735 1657 1657
q9 7014 4961 5004 4961
q10 6433 2120 1792 1792
q11 449 278 239 239
q12 644 429 291 291
q13 18127 3410 2787 2787
q14 262 258 242 242
q15 q16 822 800 710 710
q17 959 870 898 870
q18 6909 5848 5581 5581
q19 1190 1218 1109 1109
q20 513 420 267 267
q21 5693 2728 2345 2345
q22 439 363 302 302
Total cold run time: 104447 ms
Total hot run time: 31123 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4191 4115 4099 4099
q2 q3 4571 4921 4339 4339
q4 2104 2224 1431 1431
q5 4404 4282 5228 4282
q6 250 194 144 144
q7 2065 1788 1549 1549
q8 2436 2133 2111 2111
q9 7933 7747 7780 7747
q10 4781 4491 4040 4040
q11 563 394 382 382
q12 760 728 522 522
q13 3290 3687 2992 2992
q14 289 293 291 291
q15 q16 730 749 661 661
q17 1338 1331 1315 1315
q18 7973 7200 6770 6770
q19 1110 1103 1140 1103
q20 2261 2223 1942 1942
q21 5326 4641 4480 4480
q22 559 464 435 435
Total cold run time: 56934 ms
Total hot run time: 50635 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4504715563
# FE UT Coverage Report
Increment line coverage ` 30.98% (92/297)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4449904600
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4572757109
# BE Regression && UT Coverage Report
Increment line coverage `84.56% (668/790)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_fb1122e0a26c0dc4fc1defe6bfb9b6b5da020008_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_fb1122e0a26c0dc4fc1defe6bfb9b6b5da020008_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.86% (28155/38121) |
| Line Coverage | 57.78% (306127/529792) |
| Region Coverage | 54.48% (256051/469957) |
| Branch Coverage | 56.00% (111235/198626) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4555068765
<details>
<summary>TPC-DS: <b>Total hot run time: 171516 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 36cd8599c1b5eba6dcfb83d6c88457494678e987, data reload: false
query5 4339 638 480 480
query6 456 210 183 183
query7 4970 566 308 308
query8 374 223 205 205
query9 8766 4174 4120 4120
query10 458 328 260 260
query11 5896 2440 2475 2440
query12 150 104 100 100
query13 1259 611 428 428
query14 6383 5451 5125 5125
query14_1 4440 4479 4437 4437
query15 204 201 176 176
query16 1014 452 456 452
query17 1092 702 563 563
query18 2608 479 345 345
query19 199 189 141 141
query20 115 105 103 103
query21 214 143 121 121
query22 13667 13548 13441 13441
query23 17389 16461 16231 16231
query23_1 16396 16346 16248 16248
query24 7524 1770 1317 1317
query24_1 1348 1338 1339 1338
query25 547 446 396 396
query26 1307 310 163 163
query27 2667 566 340 340
query28 4445 2045 2034 2034
query29 1086 599 491 491
query30 310 239 225 225
query31 1143 1071 968 968
query32 121 64 59 59
query33 499 325 259 259
query34 1185 1155 637 637
query35 758 789 703 703
query36 1403 1348 1256 1256
query37 156 119 94 94
query38 3212 3133 3035 3035
query39 934 911 890 890
query39_1 883 866 884 866
query40 217 125 104 104
query41 64 63 63 63
query42 95 95 93 93
query43 321 326 275 275
query44
query45 197 194 175 175
query46 1142 1228 749 749
query47 2387 2411 2256 2256
query48 416 424 307 307
query49 625 473 350 350
query50 977 354 249 249
query51 4394 4340 4282 4282
query52 92 89 77 77
query53 249 272 196 196
query54 266 224 198 198
query55 81 77 71 71
query56 240 231 229 229
query57 1431 1423 1327 1327
query58 244 220 203 203
query59 1552 1676 1442 1442
query60 299 252 237 237
query61 160 157 153 153
query62 697 663 590 590
query63 234 183 196 183
query64 2529 860 647 647
query65
query66 1735 468 343 343
query67 29740 29737 29460 29460
query68
query69 426 320 265 265
query70 982 959 929 929
query71 306 219 207 207
query72 2990 2719 2585 2585
query73 853 765 432 432
query74 5156 4993 4777 4777
query75 2701 2615 2253 2253
query76 2328 1155 828 828
query77 366 390 286 286
query78 12503 12247 11921 11921
query79 1298 1040 763 763
query80 566 515 422 422
query81 453 287 249 249
query82 240 159 131 131
query83 280 281 260 260
query84 261 145 117 117
query85 948 621 502 502
query86 336 317 289 289
query87 3388 3345 3242 3242
query88 3615 2749 2706 2706
query89 418 384 337 337
query90 2192 177 199 177
query91 177 173 143 143
query92 66 63 58 58
query93 1475 1542 895 895
query94 530 343 333 333
query95 687 385 348 348
query96 1091 790 358 358
query97 2713 2683 2566 2566
query98 212 208 206 206
query99 1196 1194 1035 1035
Total cold run time: 251467 ms
Total hot run time: 171516 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4541735618
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3310374444
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.h:
##########
@@ -0,0 +1,163 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <memory>
+#include <mutex>
+#include <shared_mutex>
+#include <unordered_map>
+#include <vector>
+
+#include "common/global_types.h"
+#include "common/status.h"
+#include "core/data_type/data_type.h"
+#include "exec/common/hash_table/phmap_fwd_decl.h"
+#include "exprs/vexpr_fwd.h"
+#include "storage/olap_scan_common.h"
+
+namespace doris {
+
+class SlotDescriptor;
+class VExprContext;
+struct TPartitionBoundary;
+struct TRuntimeFilterDesc;
+struct TTargetExprMonotonicity;
+
+// Parsed representation of one partition boundary for one slot column.
+struct ParsedBoundary {
+ int64_t partition_id = 0;
+ SlotId slot_id = 0;
+ bool is_nullable = false;
+ ColumnValueRangeType boundary_cvr;
+ // True if the partition's value set is exactly {NULL} (e.g. LIST
+ // partition whose only key is NULL). The CVR alone cannot encode
+ // "only NULL" -- it stays as the whole range with contain_null=true
+ // -- so we track it explicitly to enable accurate pruning.
+ bool only_null = false;
+ // True if the partition's value set includes NULL (covers both
+ // only-NULL and mixed LIST partitions). Tracked separately because
+ // ColumnValueRange::set_contain_null(true) destructively clears the
+ // fixed-value set, so we cannot stash the NULL flag inside the CVR
+ // alongside the concrete values.
+ bool contains_null = false;
+};
+
+// Immutable, fragment-shared parse result of TPartitionBoundary list.
+//
+// Parsing is expensive (constructs VLiteral per literal, materializes a
+// ColumnPtr, builds ColumnValueRange) and depends only on plan-time data.
+// All pipeline instances of the same fragment share one parse, performed
+// in OperatorX::prepare() which is single-threaded fragment setup.
+class ParsedPartitionBoundaries {
+public:
+ ParsedPartitionBoundaries() = default;
+
+ // Build the parse result from the thrift `boundaries` list. Caller must
+ // ensure this is invoked at most once per instance (OperatorX::prepare()
+ // is the natural call site).
+ Status parse(const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs);
+
+ bool empty() const { return _slot_to_boundaries.empty(); }
+ int64_t total_partitions() const { return _total_partition_count; }
+
+ const std::unordered_map<SlotId, std::vector<ParsedBoundary>>& slot_to_boundaries() const {
+ return _slot_to_boundaries;
+ }
+
+ // Lazily compute target-domain boundaries for a monotonic RF target.
+ // `target_expr` is `impl->children()[0]` of the RF wrapper (a sub-tree of
+ // the conjunct). `leaf_slot_id` is the unique VSlotRef leaf inside it
+ // (FE asserted target_expr has exactly one input slot). `leaf_column_id`
+ // is that slot ref's `column_id()` -- the position in the runtime block.
+ // Only partitions present in `partition_directions` are projected and each
+ // partition uses its own FE-proven local direction.
+ // `ctx` is the conjunct's VExprContext (used to execute the sub-expression).
+ //
+ // Direct SlotRef targets reuse the parsed partition boundaries. Expression
+ // targets project finite RANGE endpoints by executing `target_expr`.
+ //
+ // Returns an empty vector when no selected boundary can be projected (e.g.
+ // every candidate contains NULL). Unexpected FE/BE metadata mismatches
+ // return an error instead of disabling pruning silently.
+ // The shared_ptr keeps the computed vector alive even if another pipeline
+ // instance inserts into the shared map and triggers unordered_map rehash.
+ //
+ // Direction:
+ // MONOTONIC_INCREASING: projected lo and hi keep their roles
+ // MONOTONIC_DECREASING: swap (projected lo, hi) -> (hi, lo)
+ //
+ // Open endpoints (MINVALUE/MAXVALUE) stay open after projection and swap
+ // sides for monotonic decreasing targets. Boundaries containing NULL
+ // partition values, or finite endpoints that project to NULL, are omitted
+ // from the result so this RF conservatively leaves them unpruned.
+ Status get_or_compute_projected_boundaries(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ const std::unordered_map<int64_t, TTargetExprMonotonicity::type>& partition_directions,
Review Comment:
This header only forward-declares `struct TTargetExprMonotonicity` above, but this public signature uses the nested type `TTargetExprMonotonicity::type`. A nested type cannot be named through an incomplete forward declaration, and `runtime_filter_partition_pruner.cpp` includes this header before including `gen_cpp/PlanNodes_types.h`, so the new file can fail to compile as soon as the header is parsed. Please include the generated thrift definition in the header (or avoid exposing the nested thrift type in the header signature) instead of relying on a forward declaration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3330500997
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,866 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ DORIS_CHECK(tb.__isset.partition_id);
+ DORIS_CHECK(tb.__isset.slot_id);
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ DORIS_CHECK(slot_it != slot_descs.end());
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ DORIS_CHECK(is_list || is_range); \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
Review Comment:
如果不是nullable是不是规划错误要报错呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4590367114
# FE Regression Coverage Report
Increment line coverage ` 21.76% (158/726)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_586640a94f47548a444d49ee53d5474cf04e3ae0_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_586640a94f47548a444d49ee53d5474cf04e3ae0_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4267225083
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4441069036
# BE UT Coverage Report
Increment line coverage `1.15% (8/698)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/ca30c183fbab4ffab1f11b3d91b0561736c7c199_ca30c183fbab4ffab1f11b3d91b0561736c7c199/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/ca30c183fbab4ffab1f11b3d91b0561736c7c199_ca30c183fbab4ffab1f11b3d91b0561736c7c199/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.53% (20616/38513) |
| Line Coverage | 37.15% (194893/524665) |
| Region Coverage | 33.45% (152527/455984) |
| Branch Coverage | 34.27% (66511/194075) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4440437251
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3318305519
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,845 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ DORIS_CHECK(tb.__isset.partition_id);
+ DORIS_CHECK(tb.__isset.slot_id);
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ DORIS_CHECK(slot_it != slot_descs.end());
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ DORIS_CHECK(is_list || is_range); \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ DORIS_CHECK(parsed_ok);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ if (!expr) {
+ return nullptr;
+ }
+ if (expr->is_slot_ref()) {
+ return assert_cast<const VSlotRef*>(expr);
+ }
+ const VSlotRef* found = nullptr;
+ for (const auto& child : expr->children()) {
+ const VSlotRef* c = find_unique_slot_ref(child.get());
+ if (c) {
+ if (found) {
+ return nullptr; // multiple slot refs, can't handle
+ }
+ found = c;
+ }
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+Status ParsedPartitionBoundaries::get_or_compute_projected_boundaries(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ const std::unordered_map<int64_t, TTargetExprMonotonicity::type>& partition_directions,
+ VExprContext* ctx, std::shared_ptr<const std::vector<ParsedBoundary>>* output) const {
+ {
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto it = _projected_boundaries_by_filter_id.find(filter_id);
+ if (it != _projected_boundaries_by_filter_id.end()) {
+ *output = it->second;
+ return Status::OK();
+ }
+ }
+
+ std::vector<ParsedBoundary> projected;
+ auto store_projected_boundaries = [&](std::vector<ParsedBoundary>&& boundaries) {
+ auto cached = std::make_shared<const std::vector<ParsedBoundary>>(std::move(boundaries));
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto [it, inserted] = _projected_boundaries_by_filter_id.emplace(filter_id, cached);
+ *output = inserted ? cached : it->second;
+ return Status::OK();
+ };
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::unordered_set<int64_t> boundary_partition_ids;
+ boundary_partition_ids.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ boundary_partition_ids.insert(boundary.partition_id);
+ }
+ for (const auto& [partition_id, direction] : partition_directions) {
+ DORIS_CHECK(direction != TTargetExprMonotonicity::NON_MONOTONIC);
+ DORIS_CHECK(boundary_partition_ids.contains(partition_id));
+ }
+
+ if (target_expr->is_slot_ref()) {
+ auto* slot_ref = assert_cast<VSlotRef*>(target_expr.get());
+ DORIS_CHECK(slot_ref->slot_id() == leaf_slot_id);
+ std::vector<ParsedBoundary> slot_boundaries;
+ slot_boundaries.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ auto direction_it = partition_directions.find(boundary.partition_id);
+ if (direction_it == partition_directions.end()) {
+ continue;
+ }
+ DORIS_CHECK(direction_it->second == TTargetExprMonotonicity::MONOTONIC_INCREASING);
+ slot_boundaries.emplace_back(boundary);
+ }
+ return store_projected_boundaries(std::move(slot_boundaries));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> selected_boundaries;
+ selected_boundaries.reserve(orig_boundaries.size());
+ for (size_t i = 0; i < orig_boundaries.size(); ++i) {
+ auto it = partition_directions.find(orig_boundaries[i].partition_id);
+ if (it != partition_directions.end() &&
+ it->second != TTargetExprMonotonicity::NON_MONOTONIC) {
+ selected_boundaries.emplace_back(i, it->second);
+ }
+ }
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> projectable_boundaries;
+ projectable_boundaries.reserve(selected_boundaries.size());
+ for (const auto& selected_boundary : selected_boundaries) {
+ const auto& boundary = orig_boundaries[selected_boundary.first];
+ if (!boundary.only_null && !boundary.contains_null) {
+ projectable_boundaries.emplace_back(selected_boundary);
+ }
+ }
+ selected_boundaries.swap(projectable_boundaries);
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ DORIS_CHECK(slot_type_it != _slot_data_types.end());
+
+ const DataTypePtr& input_type = slot_type_it->second;
+ bool input_is_nullable = input_type->is_nullable();
+ DataTypePtr inner_type = input_is_nullable ? remove_nullable(input_type) : input_type;
+ PrimitiveType input_ptype = input_type->get_primitive_type();
+
+ size_t N = selected_boundaries.size();
+ std::vector<bool> lo_open(N, false);
+ std::vector<bool> hi_open(N, false);
+ std::vector<bool> boundary_is_list(N, false);
+ std::vector<size_t> lo_result_row(N, 0);
+ std::vector<size_t> hi_result_row(N, 0);
+ std::vector<size_t> list_result_begin(N, 0);
+ std::vector<size_t> list_result_end(N, 0);
+ size_t lo_row_count = 0;
+ size_t hi_row_count = 0;
+ size_t list_row_count = 0;
+
+ // Build input blocks for RANGE endpoints and LIST values.
+ Block lo_block;
+ Block hi_block;
+ Block list_block;
+
+ // Pad columns 0..leaf_column_id-1 with placeholder columns so the leaf
+ // VSlotRef (whose column_id is leaf_column_id) reads our typed column.
+ auto add_dummy_columns = [&](Block& block, size_t row_count) {
+ for (int col_idx = 0; col_idx < leaf_column_id; ++col_idx) {
+ auto col = ColumnUInt8::create(row_count, 0);
+ block.insert({std::move(col), std::make_shared<DataTypeUInt8>(),
+ fmt::format("dummy_{}", col_idx)});
+ }
+ };
+
+ // Macro-dispatch on input PrimitiveType to build finite RANGE endpoint
+ // columns and LIST value columns. Unbounded RANGE endpoints are not executed
+ // through target_expr; they stay unbounded after projection and swap sides
+ // for monotonic decreasing targets.
+ bool input_built = false;
+#define BUILD_INPUT_COLUMNS(INPUT_PT) \
+ case TYPE_##INPUT_PT: { \
+ using InCol = typename PrimitiveTypeTraits<TYPE_##INPUT_PT>::ColumnType; \
+ MutableColumnPtr lo_inner_base = inner_type->create_column(); \
+ MutableColumnPtr hi_inner_base = inner_type->create_column(); \
+ MutableColumnPtr list_inner_base = inner_type->create_column(); \
+ auto* lo_inner = assert_cast<InCol*>(lo_inner_base.get()); \
+ auto* hi_inner = assert_cast<InCol*>(hi_inner_base.get()); \
+ auto* list_inner = assert_cast<InCol*>(list_inner_base.get()); \
+ lo_inner->reserve(N); \
+ hi_inner->reserve(N); \
+ list_inner->reserve(N); \
+ auto lo_nulls = ColumnUInt8::create(); \
+ auto hi_nulls = ColumnUInt8::create(); \
+ auto list_nulls = ColumnUInt8::create(); \
+ lo_nulls->reserve(N); \
+ hi_nulls->reserve(N); \
+ list_nulls->reserve(N); \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& boundary = orig_boundaries[selected_boundaries[i].first]; \
+ const auto* cvr = \
+ std::get_if<ColumnValueRange<TYPE_##INPUT_PT>>(&boundary.boundary_cvr); \
+ DCHECK(cvr != nullptr); \
+ DCHECK(!boundary.only_null); \
+ DCHECK(!boundary.contains_null); \
+ boundary_is_list[i] = cvr->is_fixed_value_range(); \
+ if (boundary_is_list[i]) { \
+ list_result_begin[i] = list_row_count; \
+ for (const auto& value : cvr->get_fixed_value_set()) { \
+ list_inner->insert_value(value); \
+ list_nulls->get_data().push_back(0); \
+ ++list_row_count; \
+ } \
+ list_result_end[i] = list_row_count; \
+ continue; \
+ } \
+ lo_open[i] = cvr->is_low_value_minimum(); \
+ hi_open[i] = cvr->is_high_value_maximum(); \
+ if (!lo_open[i]) { \
+ lo_result_row[i] = lo_row_count++; \
+ lo_inner->insert_value(cvr->get_range_min_value()); \
+ lo_nulls->get_data().push_back(0); \
+ } \
+ if (!hi_open[i]) { \
+ hi_result_row[i] = hi_row_count++; \
+ hi_inner->insert_value(cvr->get_range_max_value()); \
+ hi_nulls->get_data().push_back(0); \
+ } \
+ } \
+ add_dummy_columns(lo_block, lo_row_count); \
+ add_dummy_columns(hi_block, hi_row_count); \
+ add_dummy_columns(list_block, list_row_count); \
+ if (input_is_nullable) { \
+ auto lo_col = ColumnNullable::create(std::move(lo_inner_base), std::move(lo_nulls)); \
+ auto hi_col = ColumnNullable::create(std::move(hi_inner_base), std::move(hi_nulls)); \
+ auto list_col = \
+ ColumnNullable::create(std::move(list_inner_base), std::move(list_nulls)); \
+ lo_block.insert({std::move(lo_col), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_col), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_col), input_type, "leaf_slot"}); \
+ } else { \
+ lo_block.insert({std::move(lo_inner_base), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_inner_base), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_inner_base), input_type, "leaf_slot"}); \
+ } \
+ input_built = true; \
+ break; \
+ }
+
+ switch (input_ptype) {
+ BUILD_INPUT_COLUMNS(TINYINT)
+ BUILD_INPUT_COLUMNS(SMALLINT)
+ BUILD_INPUT_COLUMNS(INT)
+ BUILD_INPUT_COLUMNS(BIGINT)
+ BUILD_INPUT_COLUMNS(LARGEINT)
+ BUILD_INPUT_COLUMNS(FLOAT)
+ BUILD_INPUT_COLUMNS(DOUBLE)
+ BUILD_INPUT_COLUMNS(CHAR)
+ BUILD_INPUT_COLUMNS(DATE)
+ BUILD_INPUT_COLUMNS(DATETIME)
+ BUILD_INPUT_COLUMNS(DATEV2)
+ BUILD_INPUT_COLUMNS(DATETIMEV2)
+ BUILD_INPUT_COLUMNS(TIMESTAMPTZ)
+ BUILD_INPUT_COLUMNS(VARCHAR)
+ BUILD_INPUT_COLUMNS(STRING)
+ BUILD_INPUT_COLUMNS(DECIMAL32)
+ BUILD_INPUT_COLUMNS(DECIMAL64)
+ BUILD_INPUT_COLUMNS(DECIMAL128I)
+ BUILD_INPUT_COLUMNS(DECIMAL256)
+ BUILD_INPUT_COLUMNS(DECIMALV2)
+ BUILD_INPUT_COLUMNS(BOOLEAN)
+ BUILD_INPUT_COLUMNS(IPV4)
+ BUILD_INPUT_COLUMNS(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_INPUT_COLUMNS
+
+ DORIS_CHECK(input_built);
+
+ int lo_result_id = -1;
+ int hi_result_id = -1;
+ int list_result_id = -1;
+ ColumnPtr lo_result_col;
+ ColumnPtr hi_result_col;
+ ColumnPtr list_result_col;
+ if (lo_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &lo_block, &lo_result_id));
+ DORIS_CHECK(lo_result_id >= 0);
+ lo_result_col = lo_block.get_by_position(lo_result_id).column;
+ }
+ if (hi_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &hi_block, &hi_result_id));
+ DORIS_CHECK(hi_result_id >= 0);
+ hi_result_col = hi_block.get_by_position(hi_result_id).column;
+ }
+ if (list_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &list_block, &list_result_id));
+ DORIS_CHECK(list_result_id >= 0);
+ list_result_col = list_block.get_by_position(list_result_id).column;
+ }
+ int out_precision = cast_set<int>(target_expr->data_type()->get_precision());
+ int out_scale = cast_set<int>(target_expr->data_type()->get_scale());
+ bool out_nullable = target_expr->data_type()->is_nullable();
+ PrimitiveType output_ptype = target_expr->data_type()->get_primitive_type();
+
+ projected.reserve(N);
+
+ bool output_built = false;
+#define BUILD_PROJECTED_CVR(OUTPUT_PT) \
+ case TYPE_##OUTPUT_PT: { \
+ using OutputCppType = typename PrimitiveTypeTraits<TYPE_##OUTPUT_PT>::CppType; \
+ auto projected_value = [](const ColumnPtr& column, size_t row) -> OutputCppType { \
+ Field field = (*column)[row]; \
+ return field.get<TYPE_##OUTPUT_PT>(); \
+ }; \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& orig_boundary = orig_boundaries[selected_boundaries[i].first]; \
+ TTargetExprMonotonicity::type direction = selected_boundaries[i].second; \
+ if (boundary_is_list[i]) { \
+ auto cvr = ColumnValueRange<TYPE_##OUTPUT_PT>::create_empty_column_value_range( \
+ out_nullable, out_precision, out_scale); \
+ bool list_has_value = false; \
+ bool list_has_null = false; \
+ for (size_t row = list_result_begin[i]; row < list_result_end[i]; ++row) { \
+ if (list_result_col->is_null_at(row)) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ auto projected_list_value = projected_value(list_result_col, row); \
+ static_cast<void>(cvr.add_fixed_value(projected_list_value)); \
+ list_has_value = true; \
+ } \
+ if (!list_has_value && !list_has_null) { \
+ continue; \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.contains_null = list_has_null; \
+ projected_boundary.only_null = list_has_null && !list_has_value; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ continue; \
+ } \
+ if ((!lo_open[i] && lo_result_col->is_null_at(lo_result_row[i])) || \
+ (!hi_open[i] && hi_result_col->is_null_at(hi_result_row[i]))) { \
+ continue; \
+ } \
+ ColumnValueRange<TYPE_##OUTPUT_PT> cvr("", out_nullable, out_precision, out_scale); \
+ /* Use closed projected finite bounds as a conservative envelope for */ \
+ /* non-strict monotonic expressions such as date ceil/floor/cast. */ \
+ if (direction == TTargetExprMonotonicity::MONOTONIC_DECREASING) { \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, hi_projected)); \
+ } \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, lo_projected)); \
+ } \
+ } else { \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, lo_projected)); \
+ } \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, hi_projected)); \
+ } \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.only_null = orig_boundary.only_null; \
+ projected_boundary.contains_null = orig_boundary.contains_null; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ } \
+ output_built = true; \
+ break; \
+ }
+
+ switch (output_ptype) {
+ BUILD_PROJECTED_CVR(TINYINT)
+ BUILD_PROJECTED_CVR(SMALLINT)
+ BUILD_PROJECTED_CVR(INT)
+ BUILD_PROJECTED_CVR(BIGINT)
+ BUILD_PROJECTED_CVR(LARGEINT)
+ BUILD_PROJECTED_CVR(FLOAT)
+ BUILD_PROJECTED_CVR(DOUBLE)
+ BUILD_PROJECTED_CVR(CHAR)
+ BUILD_PROJECTED_CVR(DATE)
+ BUILD_PROJECTED_CVR(DATETIME)
+ BUILD_PROJECTED_CVR(DATEV2)
+ BUILD_PROJECTED_CVR(DATETIMEV2)
+ BUILD_PROJECTED_CVR(TIMESTAMPTZ)
+ BUILD_PROJECTED_CVR(VARCHAR)
+ BUILD_PROJECTED_CVR(STRING)
+ BUILD_PROJECTED_CVR(DECIMAL32)
+ BUILD_PROJECTED_CVR(DECIMAL64)
+ BUILD_PROJECTED_CVR(DECIMAL128I)
+ BUILD_PROJECTED_CVR(DECIMAL256)
+ BUILD_PROJECTED_CVR(DECIMALV2)
+ BUILD_PROJECTED_CVR(BOOLEAN)
+ BUILD_PROJECTED_CVR(IPV4)
+ BUILD_PROJECTED_CVR(IPV6)
+ default:
+ break;
+ }
+
+#undef BUILD_PROJECTED_CVR
+
+ DORIS_CHECK(output_built);
+
+ return store_projected_boundaries(std::move(projected));
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static SQLFilterOp convert_opcode_to_filter_op(TExprOpcode::type op) {
+ switch (op) {
+ case TExprOpcode::LE:
+ return FILTER_LESS_OR_EQUAL;
+ case TExprOpcode::LT:
+ return FILTER_LESS;
+ case TExprOpcode::GE:
+ return FILTER_LARGER_OR_EQUAL;
+ case TExprOpcode::GT:
+ return FILTER_LARGER;
+ default:
+ return FILTER_IN; // sentinel: caller should skip
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+void RuntimeFilterPartitionPruner::_try_prune_by_single_rf(
+ const std::vector<ParsedBoundary>& boundaries, const VExprSPtr& impl,
+ phmap::flat_hash_set<int64_t>& newly_pruned) {
+ // Pre-compute whether the RF "matches NULL" -- i.e. whether the RF would
+ // accept a row whose probe value is NULL. The signal differs by RF impl:
+ // * IN filter → HybridSet::contain_null()
+ // * Bloom filter → BloomFilterFuncBase::contain_null()
+ // * MinMax filter → encoded via the BinaryPredicate node_type:
+ // NULL_AWARE_BINARY_PRED means the build side had a
+ // NULL key and a null-safe equal join asked NULL to
+ // match NULL; plain BINARY_PRED never matches NULL.
+ // FilterBase::contain_null() already folds in `_null_aware`, so we only
+ // get a true result when the build side is actually null-aware AND
+ // produced a NULL value.
+ bool rf_contains_null = false;
+ if (auto hybrid_set = impl->get_set_func()) {
+ rf_contains_null = hybrid_set->contain_null();
+ } else if (impl->node_type() == TExprNodeType::BLOOM_PRED) {
+ auto bloom = impl->get_bloom_filter_func();
+ rf_contains_null = bloom && bloom->contain_null();
+ } else if (impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) {
+ // Min/Max RF built on a null-safe equal join. The literal child holds
+ // the min or max bound; the NULL semantic is conveyed by the node
+ // type itself (see create_vbin_predicate in runtime_filter/utils.cpp).
+ rf_contains_null = true;
+ }
+
+ std::optional<ColumnValueRangeType> rf_cvr;
+ if (!boundaries.empty()) {
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+ using CppType = typename CvrType::CppType;
+
+ auto hybrid_set = impl->get_set_func();
+ if (hybrid_set) {
+ auto typed_rf_cvr = CvrType::create_empty_column_value_range(
+ boundaries.front().is_nullable, boundary_cvr.precision(),
+ boundary_cvr.scale());
+ auto* iter = hybrid_set->begin();
+ while (iter->has_next()) {
+ const void* value = iter->get_value();
+ if (value) {
+ if constexpr (std::is_same_v<CppType, String>) {
+ // String HybridSets store StringRef*, but
+ // ColumnValueRange<String>::CppType is
+ // std::string. Construct from the bytes.
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else if constexpr (std::is_same_v<CppType, StringRef>) {
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else {
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ *reinterpret_cast<const CppType*>(value)));
+ }
+ }
+ iter->next();
+ }
+ rf_cvr.emplace(std::move(typed_rf_cvr));
+ } else if ((impl->node_type() == TExprNodeType::BINARY_PRED ||
+ impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) &&
+ impl->children().size() == 2 && impl->children()[1]->is_literal()) {
+ // MinMax filter: binary pred with literal bound
+ auto* literal = assert_cast<VLiteral*>(impl->children()[1].get());
+ auto col_ptr = literal->get_column_ptr();
+ auto data = col_ptr->get_data_at(0);
+ CppType val {};
+ if constexpr (std::is_same_v<CppType, String>) {
+ // get_data_at returns a StringRef pointing at the
+ // raw character bytes; ColumnValueRange<String>'s
+ // CppType is std::string, so construct one from
+ // the bytes rather than dereferencing as String.
+ val = CppType(data.data, data.size);
+ } else if constexpr (std::is_same_v<CppType, StringRef>) {
+ val = CppType(data.data, data.size);
+ } else {
+ val = *reinterpret_cast<const CppType*>(data.data);
+ }
+
+ SQLFilterOp op = convert_opcode_to_filter_op(impl->op());
+ if (op == FILTER_IN) {
+ return; // unrecognized opcode, skip
+ }
+
+ CvrType typed_rf_cvr(boundary_cvr.column_name(),
+ boundaries.front().is_nullable,
+ boundary_cvr.precision(), boundary_cvr.scale());
+ static_cast<void>(typed_rf_cvr.add_range(op, val));
+ rf_cvr.emplace(std::move(typed_rf_cvr));
+ }
+ },
+ boundaries.front().boundary_cvr);
+ }
+ if (!rf_cvr.has_value()) {
+ return;
+ }
+
+ for (const auto& pb : boundaries) {
+ if (_pruned_partition_ids.contains(pb.partition_id) ||
+ newly_pruned.contains(pb.partition_id)) {
+ continue;
+ }
+
+ // NULL handling:
+ // A partition row whose key is NULL matches the RF iff `rf_contains_null`.
+ // - only_null partition (rows are exclusively NULL): prunable iff !rf_contains_null.
+ // - mixed (NULL + concrete values): if rf_contains_null, NULL rows alone
+ // prevent pruning. Otherwise NULL rows can never match, so we ignore
+ // the NULL portion and let the regular non-NULL intersection decide.
+ if (pb.only_null) {
+ if (!rf_contains_null) {
+ newly_pruned.insert(pb.partition_id);
+ }
+ continue;
+ }
+ if (pb.contains_null && rf_contains_null) {
+ continue;
+ }
+
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+
+ const auto* typed_rf_cvr = std::get_if<CvrType>(&rf_cvr.value());
+ DORIS_CHECK(typed_rf_cvr != nullptr);
+
+ auto boundary_copy = boundary_cvr;
+ auto rf_cvr_copy = *typed_rf_cvr;
+ boundary_copy.intersection(rf_cvr_copy);
+ if (boundary_copy.is_empty_value_range()) {
+ newly_pruned.insert(pb.partition_id);
+ }
+ },
+ pb.boundary_cvr);
+ }
+}
+
+Status RuntimeFilterPartitionPruner::prune_by_runtime_filters(
+ const ParsedPartitionBoundaries& parsed, const VExprContextSPtrs& conjuncts,
+ const std::vector<TRuntimeFilterDesc>& rf_descs, int scan_node_id,
+ int64_t* newly_pruned_count) {
+ *newly_pruned_count = 0;
+ if (parsed.empty()) {
+ return Status::OK();
+ }
+ const auto& slot_to_boundaries = parsed.slot_to_boundaries();
+
+ // Build filter_id -> per-partition monotonicity maps.
+ std::unordered_map<int, std::unordered_map<int64_t, TTargetExprMonotonicity::type>>
+ filter_id_to_partition_monotonicity;
+ for (const auto& desc : rf_descs) {
+ if (desc.__isset.planId_to_partition_target_monotonicity) {
+ auto it = desc.planId_to_partition_target_monotonicity.find(scan_node_id);
+ if (it != desc.planId_to_partition_target_monotonicity.end()) {
+ auto& partition_monotonicity = filter_id_to_partition_monotonicity[desc.filter_id];
+ for (const auto& partition_entry : it->second) {
+ DORIS_CHECK(partition_entry.__isset.partition_id);
+ DORIS_CHECK(partition_entry.__isset.monotonicity);
+ DORIS_CHECK(partition_entry.monotonicity !=
+ TTargetExprMonotonicity::NON_MONOTONIC);
+ auto [_, inserted] = partition_monotonicity.emplace(
+ partition_entry.partition_id, partition_entry.monotonicity);
+ DORIS_CHECK(inserted);
+ }
+ }
+ }
+ }
+
+ // This function is serialized by _conjuncts_lock in the caller, so our reads
+ // of _pruned_partition_ids never race with our writes below. The only concurrent
+ // readers are is_partition_pruned() calls (under shared_lock), which are
+ // properly synchronized by the unique_lock we take when inserting.
+ phmap::flat_hash_set<int64_t> newly_pruned;
+
+ for (const auto& conjunct_ctx : conjuncts) {
+ VExprSPtr root = conjunct_ctx->root();
+ if (!root->is_rf_wrapper()) {
+ continue;
+ }
+
+ VExprSPtr impl = root->get_impl();
+ if (!impl) {
+ continue;
+ }
+
+ if (impl->children().empty()) {
+ continue;
+ }
+
+ auto* wrapper_root = assert_cast<RuntimeFilterExpr*>(root.get());
+ int filter_id = wrapper_root->filter_id();
+
+ VExprSPtr target_subtree = impl->children()[0];
+
+ auto partition_mono_it = filter_id_to_partition_monotonicity.find(filter_id);
+ bool has_partition_mono = partition_mono_it != filter_id_to_partition_monotonicity.end() &&
+ !partition_mono_it->second.empty();
+ if (!has_partition_mono) {
+ continue;
+ }
+
+ const VSlotRef* leaf_slot = find_unique_slot_ref(target_subtree.get());
Review Comment:
FE currently accepts LIST-partition expression targets as long as the expression is deterministic and `findUniqueSlotRef()` resolves to one partition slot. That Java helper still returns the slot when the same slot appears more than once, for example a target like `concat(k, k)` or `k + k` on `PARTITION BY LIST(k)`. But the BE helper above returns `nullptr` as soon as it sees a second `VSlotRef`, even if it is the same slot, so this `DORIS_CHECK` aborts the query after FE has marked the filter as partition-prunable.
This is distinct from the existing classifier monotonicity thread: LIST pruning does not use monotonicity, and the failure here is an FE/BE acceptance mismatch that can crash instead of conservatively skipping pruning. Please either make FE reject repeated-slot expression targets for LIST pruning, or make the BE helper accept repeated references to the same slot id and add a regression case for that shape.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4570626122
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/32e14d6220221b6aa1e1cab5adf12ffaae3e9f5d_32e14d6220221b6aa1e1cab5adf12ffaae3e9f5d_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/32e14d6220221b6aa1e1cab5adf12ffaae3e9f5d_32e14d6220221b6aa1e1cab5adf12ffaae3e9f5d_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1854/2375) |
| Line Coverage | 64.54% (33342/51663) |
| Region Coverage | 65.24% (16535/25343) |
| Branch Coverage | 55.75% (8838/15854) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4571689394
<details>
<summary>TPC-H: <b>Total hot run time: 31522 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 32e14d6220221b6aa1e1cab5adf12ffaae3e9f5d, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17661 4002 4007 4002
q2 q3 10996 1447 836 836
q4 4805 483 350 350
q5 10690 2296 2142 2142
q6 402 181 141 141
q7 1024 779 656 656
q8 9605 1740 1688 1688
q9 7083 4986 4965 4965
q10 6547 2331 1905 1905
q11 440 280 246 246
q12 695 425 297 297
q13 18249 3487 2814 2814
q14 265 261 243 243
q15 q16 816 796 712 712
q17 947 945 879 879
q18 7048 5779 5571 5571
q19 1232 1197 1172 1172
q20 510 389 266 266
q21 5664 2565 2334 2334
q22 452 370 303 303
Total cold run time: 105131 ms
Total hot run time: 31522 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4360 4254 4290 4254
q2 q3 4546 4940 4331 4331
q4 2078 2210 1391 1391
q5 4743 4839 5114 4839
q6 241 184 140 140
q7 1988 1820 1604 1604
q8 2445 2179 2349 2179
q9 8078 7991 8192 7991
q10 4808 4830 4368 4368
q11 575 421 373 373
q12 756 757 538 538
q13 3300 3629 2966 2966
q14 308 305 268 268
q15 q16 739 741 640 640
q17 1361 1338 1326 1326
q18 8012 7152 6899 6899
q19 1132 1117 1099 1099
q20 2212 2240 1943 1943
q21 5277 4586 4422 4422
q22 531 468 409 409
Total cold run time: 57490 ms
Total hot run time: 51980 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4570295094
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4570403351
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt merged PR #62589:
URL: https://github.com/apache/doris/pull/62589
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4288149525
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3108771902
##########
regression-test/data/query_p0/runtime_filter/rf_partition_pruning.out:
##########
@@ -0,0 +1,106 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !range_int_in --
+1 10 a
+3 50 c
+5 90 e
+
+-- !range_int_minmax --
+1 10 a
+3 50 c
+5 90 e
+
+-- !range_date_in --
+1 2024-01-15 jan
+2 2024-02-20 feb
+3 2024-03-10 mar
+
+-- !range_date_minmax --
+1 2024-01-15 jan
+2 2024-02-20 feb
+3 2024-03-10 mar
+
+-- !list_str_in --
+1 Beijing a
+2 Beijing b
+3 Beijing c
+
+-- !list_int_in --
+1 1 a
+2 1 b
+5 3 e
+6 3 f
+
+-- !no_pruning --
+13 250 m
+18 350 r
+3 50 c
+8 150 h
+
+-- !range_int_agg --
+3 10 90
+
+-- !list_str_agg --
+Beijing 3
+
+-- !list_int_multi --
+3 2 c
+4 2 d
+7 4 g
+8 4 h
+
+-- !bigint_range_in --
+1 100 v0a
+11 250100 v5a
+12 275000 v5b
+2 25000 v0b
+
+-- !bigint_range_minmax --
+1 100 v0a
+11 250100 v5a
+12 275000 v5b
+2 25000 v0b
+
+-- !datetime_range_in --
+1 2024-02-15T10:00 jan
+
+-- !datetime_range_minmax --
+1 2024-02-15T10:00 jan
+
+-- !list_50_in --
+10 10 cat10
+25 25 cat25
+3 3 cat3
+40 40 cat40
+50 50 cat50
+
+-- !neg_range_in --
+7 -50 g
+8 -10 h
+
+-- !neg_range_minmax --
+7 -50 g
+8 -10 h
+
+-- !desc_dim_in --
+1 10 a
+3 50 c
+5 90 e
+
+-- !desc_dim_minmax --
+1 10 a
+3 50 c
+5 90 e
+
+-- !combined_rf --
+1 10 a
+3 50 c
+5 90 e
+
+-- !span_in --
+13 250 m
+3 50 c
+
+-- !span_minmax --
+13 250 m
+3 50 c
+
Review Comment:
The suite now defines five more ordered result checks, but this golden file still stops at `span_minmax`. The missing sections are `list_only_null_pruned`, `list_only_null_minmax`, `list_mixed_value_kept`, `list_mixed_nullsafe`, and `range_multi_first_col`, so `run-regression-test.sh -d query_p0 -s rf_partition_pruning` will fail at the first new `order_qt_*` case.
##########
regression-test/suites/query_p0/runtime_filter/rf_partition_pruning.groovy:
##########
@@ -0,0 +1,1120 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.apache.doris.regression.action.ProfileAction
+
+// Marked nonConcurrent: this suite asserts on FE query-profile counters
+// (TotalPartitionsForRFPruning / PartitionsPrunedByRuntimeFilter). Even
+// though each query embeds a unique UUID token to look up its own profile,
+// the FE profile list is bounded (max_query_profile_num) and shared across
+// sessions, so heavy parallel traffic could evict our profile before the
+// poller finds it. Running serially keeps the assertions deterministic.
+suite("rf_partition_pruning", "nonConcurrent") {
+ sql "set enable_runtime_filter_prune=true;"
+
+ // ---- Profile utilities ----
+ def profileAction = new ProfileAction(context)
+
+ def getProfileByToken = { String token ->
Review Comment:
This helper still turns `profile was not visible yet` into `total=0/pruned=0`: if the profile list does not contain the token within `15 * 500ms`, `getProfileByToken()` returns `""` and `extractCounterSum()` quietly reads that as zero. On a slow or loaded FE that makes the suite fail even though pruning worked. Please fail explicitly when the profile is still missing, or keep polling until it appears / the query is known to be gone.
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,392 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <utility>
+
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// Helper to extract a typed value from a ColumnPtr's first row.
+// Must be a template function so that `if constexpr` properly discards
+// the inapplicable branch during instantiation.
+template <PrimitiveType PT>
+static typename PrimitiveTypeTraits<PT>::CppType extract_value_from_column(const ColumnPtr& col) {
+ using CppType = typename PrimitiveTypeTraits<PT>::CppType;
+ auto data = col->get_data_at(0);
+ if constexpr (is_string_type(PT)) {
+ return CppType(data.data, data.size);
+ } else {
+ return *reinterpret_cast<const CppType*>(data.data);
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+void RuntimeFilterPartitionPruner::parse_boundaries(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ if (!tb.__isset.partition_id || !tb.__isset.slot_id) {
+ continue;
+ }
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ if (slot_it == slot_descs.end()) {
+ continue;
+ }
+ SlotDescriptor* slot = slot_it->second;
+ PrimitiveType ptype = slot->type()->get_primitive_type();
+ int precision = cast_set<int>(slot->type()->get_precision());
+ int scale = cast_set<int>(slot->type()->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ if (!is_list && !is_range) break; \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = \
+ [&](const TExprNode& node) -> std::optional<std::pair<CppType, ColumnPtr>> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ VLiteral literal(node); \
+ auto col_ptr = literal.get_column_ptr(); \
+ auto val = extract_value_from_column<TYPE_##NAME>(col_ptr); \
+ return std::make_optional(std::pair<CppType, ColumnPtr> {val, col_ptr}); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ auto& [val, col_ptr] = *parsed; \
+ boundary.literal_columns.push_back(std::move(col_ptr)); \
+ static_cast<void>(empty_cvr.add_fixed_value(val)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ auto& [val, col_ptr] = *parsed; \
+ boundary.literal_columns.push_back(std::move(col_ptr)); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, val)); \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ auto& [val, col_ptr] = *parsed; \
+ boundary.literal_columns.push_back(std::move(col_ptr)); \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, val)); \
+ } \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ if (parsed_ok) {
+ _partition_column_slot_ids.insert(slot_id);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_partition_column_slot_ids.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static SQLFilterOp convert_opcode_to_filter_op(TExprOpcode::type op) {
+ switch (op) {
+ case TExprOpcode::LE:
+ return FILTER_LESS_OR_EQUAL;
+ case TExprOpcode::LT:
+ return FILTER_LESS;
+ case TExprOpcode::GE:
+ return FILTER_LARGER_OR_EQUAL;
+ case TExprOpcode::GT:
+ return FILTER_LARGER;
+ default:
+ return FILTER_IN; // sentinel: caller should skip
+ }
+}
+
+void RuntimeFilterPartitionPruner::_try_prune_by_single_rf(
+ const VExprSPtr& impl, SlotId slot_id, phmap::flat_hash_set<int64_t>& newly_pruned) {
+ auto boundaries_it = _slot_to_boundaries.find(slot_id);
+ if (boundaries_it == _slot_to_boundaries.end()) {
+ return;
+ }
+
+ // Pre-compute whether the RF "matches NULL" -- i.e. whether the RF would
+ // accept a row whose probe value is NULL. The signal differs by RF impl:
+ // * IN filter → HybridSet::contain_null()
+ // * Bloom filter → BloomFilterFuncBase::contain_null()
+ // * MinMax filter → encoded via the BinaryPredicate node_type:
+ // NULL_AWARE_BINARY_PRED means the build side had a
+ // NULL key and a null-safe equal join asked NULL to
+ // match NULL; plain BINARY_PRED never matches NULL.
+ // FilterBase::contain_null() already folds in `_null_aware`, so we only
+ // get a true result when the build side is actually null-aware AND
+ // produced a NULL value.
+ bool rf_contains_null = false;
+ if (auto hybrid_set = impl->get_set_func()) {
+ rf_contains_null = hybrid_set->contain_null();
+ } else if (impl->node_type() == TExprNodeType::BLOOM_PRED) {
+ auto bloom = impl->get_bloom_filter_func();
+ rf_contains_null = bloom && bloom->contain_null();
+ } else if (impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) {
+ // Min/Max RF built on a null-safe equal join. The literal child holds
+ // the min or max bound; the NULL semantic is conveyed by the node
+ // type itself (see create_vbin_predicate in runtime_filter/utils.cpp).
+ rf_contains_null = true;
+ }
+
+ for (const auto& pb : boundaries_it->second) {
+ if (_pruned_partition_ids.contains(pb.partition_id) ||
+ newly_pruned.contains(pb.partition_id)) {
+ continue;
+ }
+
+ // NULL handling:
+ // A partition row whose key is NULL matches the RF iff `rf_contains_null`.
+ // - only_null partition (rows are exclusively NULL): prunable iff !rf_contains_null.
+ // - mixed (NULL + concrete values): if rf_contains_null, NULL rows alone
+ // prevent pruning. Otherwise NULL rows can never match, so we ignore
+ // the NULL portion and let the regular non-NULL intersection decide.
+ if (pb.only_null) {
+ if (!rf_contains_null) {
+ newly_pruned.insert(pb.partition_id);
+ }
+ continue;
+ }
+ if (pb.contains_null && rf_contains_null) {
+ continue;
+ }
+
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+ using CppType = typename CvrType::CppType;
+
+ auto hybrid_set = impl->get_set_func();
Review Comment:
`VBloomPredicate` falls through this pruning logic today. In the real Bloom case (`runtime_filter_type='BLOOM_FILTER'`, or `IN_OR_BLOOM_FILTER` after the wrapper switches past `runtime_filter_max_in_num`), `impl->node_type()==BLOOM_PRED`, `get_set_func()==nullptr`, and we never enter the MinMax branch either. The result is that the new OLAP partition-pruning path becomes a no-op for Bloom RFs, so LIST-partition scans lose pruning entirely once the filter stops being an IN set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4289197162
# FE Regression Coverage Report
Increment line coverage ` 76.34% (100/131)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_6f985006ec32cea727f90e22b3cd688c1a0626e1_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_6f985006ec32cea727f90e22b3cd688c1a0626e1_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4267408412
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/141c78da888bc5cd8198ffebb607430c90015f0d_141c78da888bc5cd8198ffebb607430c90015f0d_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/141c78da888bc5cd8198ffebb607430c90015f0d_141c78da888bc5cd8198ffebb607430c90015f0d_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.48% (1798/2291) |
| Line Coverage | 64.12% (32283/50345) |
| Region Coverage | 65.03% (16235/24967) |
| Branch Coverage | 55.54% (8675/15620) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4429681479
<details>
<summary>TPC-DS: <b>Total hot run time: 168934 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2391ea7f407dcc42f8f4ddc3bcd8e0838059d15c, data reload: false
query5 4318 631 492 492
query6 441 195 179 179
query7 4867 555 276 276
query8 359 218 204 204
query9 8810 4035 4040 4035
query10 434 315 253 253
query11 6189 2366 2156 2156
query12 159 105 98 98
query13 1276 607 430 430
query14 7404 5333 4963 4963
query14_1 4353 4313 4288 4288
query15 213 198 175 175
query16 1013 463 455 455
query17 1289 746 608 608
query18 2727 493 356 356
query19 321 188 150 150
query20 116 109 106 106
query21 214 146 122 122
query22 13721 13953 14546 13953
query23 17240 16572 16249 16249
query23_1 16330 16249 16245 16245
query24 8053 1817 1406 1406
query24_1 1419 1397 1364 1364
query25 551 453 391 391
query26 1275 312 168 168
query27 2537 621 327 327
query28 4318 1980 1957 1957
query29 1074 608 501 501
query30 300 235 197 197
query31 1097 1057 933 933
query32 101 58 57 57
query33 519 315 252 252
query34 1163 1175 636 636
query35 741 772 650 650
query36 1292 1317 1152 1152
query37 148 103 87 87
query38 3184 3096 3062 3062
query39 915 921 881 881
query39_1 856 901 874 874
query40 228 130 116 116
query41 62 60 61 60
query42 94 90 93 90
query43 310 313 271 271
query44
query45 195 188 171 171
query46 1075 1188 709 709
query47 2303 2275 2203 2203
query48 387 403 298 298
query49 620 504 379 379
query50 724 284 220 220
query51 4285 4295 4183 4183
query52 88 89 79 79
query53 248 259 200 200
query54 258 219 199 199
query55 75 77 68 68
query56 230 221 210 210
query57 1394 1368 1296 1296
query58 242 212 211 211
query59 1543 1611 1389 1389
query60 310 259 243 243
query61 162 152 155 152
query62 666 622 563 563
query63 229 184 188 184
query64 2478 811 658 658
query65
query66 1726 485 351 351
query67 30073 29925 29264 29264
query68
query69 420 311 262 262
query70 977 944 900 900
query71 303 225 215 215
query72 3038 2550 2375 2375
query73 864 769 412 412
query74 5085 4913 4751 4751
query75 2731 2638 2277 2277
query76 2282 1138 784 784
query77 364 380 292 292
query78 12876 13026 12423 12423
query79 1481 966 682 682
query80 742 493 410 410
query81 465 287 236 236
query82 574 167 128 128
query83 349 275 250 250
query84 264 142 114 114
query85 889 487 430 430
query86 404 289 277 277
query87 3412 3332 3168 3168
query88 3565 2683 2666 2666
query89 421 366 327 327
query90 1781 185 180 180
query91 178 196 142 142
query92 64 62 58 58
query93 977 967 553 553
query94 601 354 294 294
query95 687 364 423 364
query96 1027 794 322 322
query97 2679 2673 2535 2535
query98 213 210 209 209
query99 1087 1135 1005 1005
Total cold run time: 252526 ms
Total hot run time: 168934 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4432433257
<details>
<summary>TPC-DS: <b>Total hot run time: 170082 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e7a084166584e3ccdc5ae8b6bd5fc50d3c0cb832, data reload: false
query5 4330 630 481 481
query6 422 203 180 180
query7 4826 559 308 308
query8 364 223 215 215
query9 8808 4112 4098 4098
query10 456 308 254 254
query11 5925 2334 2155 2155
query12 152 101 95 95
query13 1258 620 431 431
query14 6408 5301 5016 5016
query14_1 4343 4360 4348 4348
query15 206 203 202 202
query16 1026 490 465 465
query17 1157 742 632 632
query18 2470 512 363 363
query19 216 189 153 153
query20 114 109 109 109
query21 218 152 138 138
query22 13622 13564 13311 13311
query23 17321 16272 16591 16272
query23_1 16345 16243 16202 16202
query24 7626 1823 1377 1377
query24_1 1379 1380 1382 1380
query25 588 442 385 385
query26 1309 310 158 158
query27 2656 582 331 331
query28 4402 1961 1948 1948
query29 1111 597 509 509
query30 305 235 197 197
query31 1091 1070 942 942
query32 104 55 54 54
query33 514 309 246 246
query34 1154 1151 636 636
query35 736 753 649 649
query36 1319 1300 1173 1173
query37 153 99 84 84
query38 3187 3094 3036 3036
query39 924 920 881 881
query39_1 880 872 888 872
query40 229 124 111 111
query41 63 61 58 58
query42 91 90 93 90
query43 316 314 270 270
query44
query45 192 183 177 177
query46 1100 1160 729 729
query47 2253 2250 2181 2181
query48 410 410 286 286
query49 632 482 384 384
query50 707 286 215 215
query51 4371 4273 4157 4157
query52 87 86 74 74
query53 246 265 183 183
query54 259 211 201 201
query55 77 75 74 74
query56 236 232 213 213
query57 1397 1372 1303 1303
query58 241 205 205 205
query59 1558 1656 1411 1411
query60 289 254 233 233
query61 161 159 168 159
query62 673 620 561 561
query63 230 192 190 190
query64 2581 800 636 636
query65
query66 1814 478 361 361
query67 30043 29275 29889 29275
query68
query69 435 307 286 286
query70 996 879 929 879
query71 328 231 212 212
query72 3111 2852 2544 2544
query73 826 794 442 442
query74 5088 4908 4745 4745
query75 2730 2661 2318 2318
query76 2302 1145 767 767
query77 375 398 308 308
query78 13055 13006 12289 12289
query79 1428 949 752 752
query80 1305 510 409 409
query81 524 276 237 237
query82 572 163 129 129
query83 346 282 259 259
query84 258 135 109 109
query85 954 493 441 441
query86 428 288 297 288
query87 3382 3313 3173 3173
query88 3566 2689 2651 2651
query89 428 373 321 321
query90 1984 187 182 182
query91 183 166 147 147
query92 63 61 59 59
query93 1016 970 577 577
query94 724 358 311 311
query95 686 375 344 344
query96 1066 815 328 328
query97 2654 2678 2530 2530
query98 211 214 203 203
query99 1090 1105 981 981
Total cold run time: 252299 ms
Total hot run time: 170082 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3224701222
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java:
##########
@@ -228,6 +230,18 @@ private void createLegacyRuntimeFilterFromGroup(List<RuntimeFilter> group,
Expr targetExpr = targetExprList.get(i);
origFilter.addTarget(new RuntimeFilterTarget(
scanNode, targetExpr, true, isLocalTarget));
+ RuntimeFilterPartitionPruneClassifier.Classification classification =
Review Comment:
This grouped-RF path still classifies `curTargetExpression` even when the legacy `targetExpr` was just wrapped in the type-adjustment `CastExpr` above. The single-filter path now tracks `adjustedAfterNonIdentity` and skips classification in that case, but grouped filters do not, so a grouped RF target like `date_trunc(part_col)` that needs an outer cast can still be marked partition-prunable based on the pre-cast Nereids expression while BE evaluates partition endpoints through the post-cast legacy expression. That can produce unsafe projected boundaries and false-prune partitions. Please mirror the single-filter guard here, or classify an equivalent expression that includes the inserted cast, and add grouped-RF coverage for this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3225527557
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java:
##########
@@ -228,6 +230,18 @@ private void createLegacyRuntimeFilterFromGroup(List<RuntimeFilter> group,
Expr targetExpr = targetExprList.get(i);
origFilter.addTarget(new RuntimeFilterTarget(
scanNode, targetExpr, true, isLocalTarget));
+ RuntimeFilterPartitionPruneClassifier.Classification classification =
+ RuntimeFilterPartitionPruneClassifier.classify(
+ targetExpr, nereidsTargetExprList.get(i), scanNode);
+ if (classification.canPrunePartitions()) {
+ origFilter.markTargetCanPrunePartitions(scanNode.getId());
+ }
+ if (classification.emitTargetMonotonicity()) {
+ origFilter.setTargetMonotonicity(scanNode.getId(),
+ classification.getMonotonicity());
+ }
+ origFilter.setTargetPartitionMonotonicity(
+ scanNode.getId(), classification.getPartitionMonotonicity());
Review Comment:
This partition-pruning metadata is keyed only by `scanNode.getId()`, but a grouped runtime filter can contain more than one target for the same `OlapScanNode` after RF propagation through equalities on that scan. In that case the later target overwrites the same scan-id entry here, while `TRuntimeFilterDesc.planId_to_target_expr` is also serialized as one expression per plan node. BE therefore cannot tell which target expression the monotonicity map belongs to; if one target is eligible and another same-scan target is not (or has different per-partition monotonicity), the scan can receive stale/incorrect pruning metadata for the expression it actually evaluates and false-prune partitions. Please either prevent grouped filters from contributing more than one partition-pruning target per scan node, or key the pruning metadata by target expression/slot as well as scan id and add a grouped same-scan regression case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4439827585
# BE UT Coverage Report
Increment line coverage `1.15% (8/698)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/2337ac55b6530b92cabafe14881c96951fab95c9_2337ac55b6530b92cabafe14881c96951fab95c9/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/2337ac55b6530b92cabafe14881c96951fab95c9_2337ac55b6530b92cabafe14881c96951fab95c9/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.53% (20614/38511) |
| Line Coverage | 37.14% (194862/524640) |
| Region Coverage | 33.44% (152486/455966) |
| Branch Coverage | 34.27% (66501/194063) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4438222914
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4494528283
<details>
<summary>TPC-H: <b>Total hot run time: 31269 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef2529ca3c35ebe1f89cb9fa4212f239fce43a97, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17674 3877 3848 3848
q2 q3 10764 1448 836 836
q4 4692 472 342 342
q5 7636 2270 2078 2078
q6 241 182 139 139
q7 951 786 648 648
q8 9349 1704 1584 1584
q9 5220 4944 4915 4915
q10 6378 2129 1783 1783
q11 455 274 243 243
q12 640 422 296 296
q13 18119 3433 2740 2740
q14 261 253 233 233
q15 q16 791 777 711 711
q17 913 955 1003 955
q18 6853 5686 5512 5512
q19 1169 1244 1112 1112
q20 505 408 276 276
q21 5535 2812 2699 2699
q22 460 384 319 319
Total cold run time: 98606 ms
Total hot run time: 31269 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5273 4562 4477 4477
q2 q3 4895 5265 4722 4722
q4 2153 2262 1422 1422
q5 4775 4727 4571 4571
q6 229 190 128 128
q7 1899 1819 1534 1534
q8 2399 2062 2018 2018
q9 7793 7548 7202 7202
q10 4491 4394 3999 3999
q11 534 385 351 351
q12 716 722 513 513
q13 3070 3338 2832 2832
q14 270 273 259 259
q15 q16 683 695 631 631
q17 1264 1246 1231 1231
q18 7202 6760 6883 6760
q19 1090 1093 1096 1093
q20 2215 2214 1931 1931
q21 5278 4678 4546 4546
q22 525 452 441 441
Total cold run time: 56754 ms
Total hot run time: 50661 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4496714900
# FE Regression Coverage Report
Increment line coverage ` 46.98% (218/464)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4560623923
<details>
<summary>TPC-H: <b>Total hot run time: 32177 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9af6867af36b4da0dec52406116ba2d158e2e2b9, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17676 4073 4023 4023
q2 q3 10847 1393 815 815
q4 4741 484 350 350
q5 9461 2369 2193 2193
q6 388 177 141 141
q7 968 811 653 653
q8 9866 1781 1583 1583
q9 7139 5034 5041 5034
q10 6489 2388 1937 1937
q11 452 281 246 246
q12 691 449 301 301
q13 18263 3448 2841 2841
q14 271 264 242 242
q15 q16 841 786 713 713
q17 1018 1006 976 976
q18 6904 5734 6192 5734
q19 1240 1309 1119 1119
q20 545 428 271 271
q21 6270 2854 2701 2701
q22 460 373 304 304
Total cold run time: 104530 ms
Total hot run time: 32177 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4812 4810 4793 4793
q2 q3 4961 5238 4614 4614
q4 2119 2205 1392 1392
q5 4895 4769 4660 4660
q6 230 178 134 134
q7 1884 1741 1541 1541
q8 2348 1953 1935 1935
q9 7416 7429 7435 7429
q10 4781 4667 4216 4216
q11 533 381 354 354
q12 745 757 529 529
q13 3020 3405 2773 2773
q14 269 279 259 259
q15 q16 689 702 620 620
q17 1279 1290 1271 1271
q18 7386 6808 6673 6673
q19 1110 1103 1122 1103
q20 2230 2252 1932 1932
q21 5293 4564 4463 4463
q22 512 480 406 406
Total cold run time: 56512 ms
Total hot run time: 51097 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4571773092
# BE UT Coverage Report
Increment line coverage `71.27% (563/790)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/32e14d6220221b6aa1e1cab5adf12ffaae3e9f5d_32e14d6220221b6aa1e1cab5adf12ffaae3e9f5d/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/32e14d6220221b6aa1e1cab5adf12ffaae3e9f5d_32e14d6220221b6aa1e1cab5adf12ffaae3e9f5d/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 54.02% (21024/38921) |
| Line Coverage | 37.61% (199772/531138) |
| Region Coverage | 33.79% (157296/465535) |
| Branch Coverage | 34.79% (68849/197896) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4571688715
<details>
<summary>TPC-H: <b>Total hot run time: 31614 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fb1122e0a26c0dc4fc1defe6bfb9b6b5da020008, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17666 4176 4235 4176
q2 q3 11001 1383 832 832
q4 4745 474 346 346
q5 9936 2333 2087 2087
q6 343 183 136 136
q7 982 809 641 641
q8 9593 1701 1653 1653
q9 7072 5035 5007 5007
q10 6502 2211 1899 1899
q11 447 277 247 247
q12 693 431 315 315
q13 18235 3350 2783 2783
q14 274 263 244 244
q15 q16 821 788 709 709
q17 990 981 1042 981
q18 6764 5781 5476 5476
q19 1157 1250 1065 1065
q20 523 407 280 280
q21 5815 2707 2430 2430
q22 445 356 307 307
Total cold run time: 104004 ms
Total hot run time: 31614 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4362 4329 4329 4329
q2 q3 4534 4989 4387 4387
q4 2088 2212 1386 1386
q5 4466 4573 5061 4573
q6 252 196 135 135
q7 2032 1830 1570 1570
q8 2480 2203 2108 2108
q9 8116 7911 8027 7911
q10 4782 4744 4305 4305
q11 586 410 380 380
q12 745 782 548 548
q13 3330 3671 2951 2951
q14 300 293 274 274
q15 q16 708 737 645 645
q17 1405 1323 1334 1323
q18 8004 7183 6990 6990
q19 1156 1108 1132 1108
q20 2231 2220 1964 1964
q21 5308 4631 4420 4420
q22 513 460 438 438
Total cold run time: 57398 ms
Total hot run time: 51745 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4589969678
<details>
<summary>TPC-H: <b>Total hot run time: 31958 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 586640a94f47548a444d49ee53d5474cf04e3ae0, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17884 4089 4041 4041
q2 q3 10829 1440 824 824
q4 4756 481 353 353
q5 9882 2279 2179 2179
q6 388 185 140 140
q7 995 783 634 634
q8 9617 1778 1771 1771
q9 7136 5013 5016 5013
q10 6546 2220 1944 1944
q11 466 276 255 255
q12 687 429 302 302
q13 18223 3437 2812 2812
q14 269 258 233 233
q15 q16 821 772 705 705
q17 933 956 942 942
q18 6849 5696 5682 5682
q19 1178 1379 1058 1058
q20 519 407 264 264
q21 5735 2529 2504 2504
q22 427 360 302 302
Total cold run time: 104140 ms
Total hot run time: 31958 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4366 4264 4285 4264
q2 q3 4555 4940 4338 4338
q4 2108 2226 1378 1378
q5 4469 4570 5344 4570
q6 256 197 147 147
q7 1999 1813 1612 1612
q8 2548 2205 2375 2205
q9 7936 8268 7987 7987
q10 4842 4760 4322 4322
q11 574 428 380 380
q12 752 763 566 566
q13 3216 3532 2971 2971
q14 297 315 358 315
q15 q16 736 758 676 676
q17 1359 1361 1393 1361
q18 7977 7494 7024 7024
q19 1142 1096 1109 1096
q20 2242 2239 1956 1956
q21 5321 4631 4448 4448
q22 532 480 422 422
Total cold run time: 57227 ms
Total hot run time: 52038 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4594105894
<details>
<summary>TPC-DS: <b>Total hot run time: 169272 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b82194ce0fb5ac216aeed034490645f04c19f4e5, data reload: false
query5 4318 648 485 485
query6 450 191 182 182
query7 4872 544 311 311
query8 384 220 199 199
query9 8795 4080 4044 4044
query10 445 313 253 253
query11 5867 2340 2144 2144
query12 152 103 103 103
query13 1292 613 444 444
query14 6411 5388 5113 5113
query14_1 4408 4432 4385 4385
query15 210 202 181 181
query16 1029 471 441 441
query17 1133 712 597 597
query18 2745 485 357 357
query19 210 190 151 151
query20 111 110 108 108
query21 222 150 133 133
query22 13637 13569 13341 13341
query23 17411 16536 16185 16185
query23_1 16398 16313 16264 16264
query24 7571 1781 1290 1290
query24_1 1298 1308 1317 1308
query25 554 437 393 393
query26 1325 316 174 174
query27 2577 548 347 347
query28 4429 2001 2027 2001
query29 1070 618 488 488
query30 306 234 203 203
query31 1132 1072 944 944
query32 109 60 58 58
query33 530 320 243 243
query34 1163 1172 646 646
query35 748 772 686 686
query36 1399 1363 1277 1277
query37 157 101 95 95
query38 3204 3123 3046 3046
query39 938 921 891 891
query39_1 889 876 878 876
query40 219 118 99 99
query41 64 66 61 61
query42 93 92 94 92
query43 317 321 279 279
query44
query45 216 195 181 181
query46 1099 1216 734 734
query47 2400 2380 2279 2279
query48 412 404 297 297
query49 626 470 366 366
query50 1032 354 258 258
query51 4301 4287 4235 4235
query52 89 88 77 77
query53 251 268 195 195
query54 296 219 193 193
query55 78 73 69 69
query56 232 221 217 217
query57 1455 1407 1332 1332
query58 248 217 211 211
query59 1571 1653 1400 1400
query60 275 247 226 226
query61 161 163 159 159
query62 696 663 575 575
query63 226 182 191 182
query64 2509 805 604 604
query65
query66 1742 456 342 342
query67 29765 29602 29566 29566
query68
query69 425 302 260 260
query70 965 948 927 927
query71 297 221 201 201
query72 3054 2704 2533 2533
query73 857 755 444 444
query74 5174 4953 4768 4768
query75 2647 2581 2240 2240
query76 2301 1171 790 790
query77 342 377 281 281
query78 12299 12293 11878 11878
query79 1264 1073 737 737
query80 554 471 399 399
query81 444 277 241 241
query82 235 157 116 116
query83 269 275 246 246
query84 264 140 112 112
query85 881 539 449 449
query86 345 304 297 297
query87 3356 3340 3173 3173
query88 3626 2768 2747 2747
query89 410 382 326 326
query90 2178 185 172 172
query91 178 168 139 139
query92 65 62 58 58
query93 1381 1433 879 879
query94 545 372 323 323
query95 709 375 440 375
query96 1059 797 347 347
query97 2673 2712 2563 2563
query98 214 209 204 204
query99 1166 1186 1038 1038
Total cold run time: 250943 ms
Total hot run time: 169272 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3330589016
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,866 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ DORIS_CHECK(tb.__isset.partition_id);
+ DORIS_CHECK(tb.__isset.slot_id);
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ DORIS_CHECK(slot_it != slot_descs.end());
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ DORIS_CHECK(is_list || is_range); \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ bool range_has_null = false; \
+ if (is_nullable && !tb.__isset.range_start) { \
+ range_has_null = true; \
+ } \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } else if (is_nullable) { \
+ range_has_null = true; \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ if (range_has_null) { \
+ boundary.contains_null = true; \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ DORIS_CHECK(parsed_ok);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static bool find_unique_slot_ref_impl(const VExpr* expr, const VSlotRef** found) {
+ if (!expr) {
+ return true;
+ }
+ if (expr->is_slot_ref()) {
+ const auto* slot = assert_cast<const VSlotRef*>(expr);
+ if (*found == nullptr) {
+ *found = slot;
+ return true;
+ }
+ if ((*found)->slot_id() != slot->slot_id()) {
+ return false;
+ }
+ DORIS_CHECK((*found)->column_id() == slot->column_id());
+ return true;
+ }
+ for (const auto& child : expr->children()) {
+ if (!find_unique_slot_ref_impl(child.get(), found)) {
+ return false;
+ }
+ }
+ return true;
+}
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ const VSlotRef* found = nullptr;
+ if (!find_unique_slot_ref_impl(expr, &found)) {
+ return nullptr;
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+Status ParsedPartitionBoundaries::get_or_compute_projected_boundaries(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ const std::unordered_map<int64_t, TTargetExprMonotonicity::type>& partition_directions,
+ VExprContext* ctx, std::shared_ptr<const std::vector<ParsedBoundary>>* output) const {
+ {
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto it = _projected_boundaries_by_filter_id.find(filter_id);
+ if (it != _projected_boundaries_by_filter_id.end()) {
+ *output = it->second;
+ return Status::OK();
+ }
+ }
+
+ std::vector<ParsedBoundary> projected;
+ auto store_projected_boundaries = [&](std::vector<ParsedBoundary>&& boundaries) {
+ auto cached = std::make_shared<const std::vector<ParsedBoundary>>(std::move(boundaries));
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto [it, inserted] = _projected_boundaries_by_filter_id.emplace(filter_id, cached);
+ *output = inserted ? cached : it->second;
+ return Status::OK();
+ };
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::unordered_set<int64_t> boundary_partition_ids;
+ boundary_partition_ids.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ boundary_partition_ids.insert(boundary.partition_id);
+ }
+ for (const auto& [partition_id, direction] : partition_directions) {
+ DORIS_CHECK(direction != TTargetExprMonotonicity::NON_MONOTONIC);
+ DORIS_CHECK(boundary_partition_ids.contains(partition_id));
+ }
+
+ if (target_expr->is_slot_ref()) {
+ auto* slot_ref = assert_cast<VSlotRef*>(target_expr.get());
+ DORIS_CHECK(slot_ref->slot_id() == leaf_slot_id);
+ std::vector<ParsedBoundary> slot_boundaries;
+ slot_boundaries.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ auto direction_it = partition_directions.find(boundary.partition_id);
+ if (direction_it == partition_directions.end()) {
+ continue;
+ }
+ DORIS_CHECK(direction_it->second == TTargetExprMonotonicity::MONOTONIC_INCREASING);
+ slot_boundaries.emplace_back(boundary);
+ }
+ return store_projected_boundaries(std::move(slot_boundaries));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> selected_boundaries;
+ selected_boundaries.reserve(orig_boundaries.size());
+ for (size_t i = 0; i < orig_boundaries.size(); ++i) {
+ auto it = partition_directions.find(orig_boundaries[i].partition_id);
+ if (it != partition_directions.end() &&
+ it->second != TTargetExprMonotonicity::NON_MONOTONIC) {
+ selected_boundaries.emplace_back(i, it->second);
+ }
+ }
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> projectable_boundaries;
+ projectable_boundaries.reserve(selected_boundaries.size());
+ for (const auto& selected_boundary : selected_boundaries) {
+ const auto& boundary = orig_boundaries[selected_boundary.first];
+ if (!boundary.only_null && !boundary.contains_null) {
+ projectable_boundaries.emplace_back(selected_boundary);
+ }
+ }
+ selected_boundaries.swap(projectable_boundaries);
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ DORIS_CHECK(slot_type_it != _slot_data_types.end());
+
+ const DataTypePtr& input_type = slot_type_it->second;
+ bool input_is_nullable = input_type->is_nullable();
+ DataTypePtr inner_type = input_is_nullable ? remove_nullable(input_type) : input_type;
+ PrimitiveType input_ptype = input_type->get_primitive_type();
+
+ size_t N = selected_boundaries.size();
+ std::vector<bool> lo_open(N, false);
+ std::vector<bool> hi_open(N, false);
+ std::vector<bool> boundary_is_list(N, false);
+ std::vector<size_t> lo_result_row(N, 0);
+ std::vector<size_t> hi_result_row(N, 0);
+ std::vector<size_t> list_result_begin(N, 0);
+ std::vector<size_t> list_result_end(N, 0);
+ size_t lo_row_count = 0;
+ size_t hi_row_count = 0;
+ size_t list_row_count = 0;
+
+ // Build input blocks for RANGE endpoints and LIST values.
+ Block lo_block;
+ Block hi_block;
+ Block list_block;
+
+ // Pad columns 0..leaf_column_id-1 with placeholder columns so the leaf
+ // VSlotRef (whose column_id is leaf_column_id) reads our typed column.
+ auto add_dummy_columns = [&](Block& block, size_t row_count) {
+ for (int col_idx = 0; col_idx < leaf_column_id; ++col_idx) {
+ auto col = ColumnUInt8::create(row_count, 0);
+ block.insert({std::move(col), std::make_shared<DataTypeUInt8>(),
+ fmt::format("dummy_{}", col_idx)});
+ }
+ };
+
+ // Macro-dispatch on input PrimitiveType to build finite RANGE endpoint
+ // columns and LIST value columns. Unbounded RANGE endpoints are not executed
+ // through target_expr; they stay unbounded after projection and swap sides
+ // for monotonic decreasing targets.
+ bool input_built = false;
+#define BUILD_INPUT_COLUMNS(INPUT_PT) \
+ case TYPE_##INPUT_PT: { \
+ using InCol = typename PrimitiveTypeTraits<TYPE_##INPUT_PT>::ColumnType; \
+ MutableColumnPtr lo_inner_base = inner_type->create_column(); \
+ MutableColumnPtr hi_inner_base = inner_type->create_column(); \
+ MutableColumnPtr list_inner_base = inner_type->create_column(); \
+ auto* lo_inner = assert_cast<InCol*>(lo_inner_base.get()); \
+ auto* hi_inner = assert_cast<InCol*>(hi_inner_base.get()); \
+ auto* list_inner = assert_cast<InCol*>(list_inner_base.get()); \
+ lo_inner->reserve(N); \
+ hi_inner->reserve(N); \
+ list_inner->reserve(N); \
+ auto lo_nulls = ColumnUInt8::create(); \
+ auto hi_nulls = ColumnUInt8::create(); \
+ auto list_nulls = ColumnUInt8::create(); \
+ lo_nulls->reserve(N); \
+ hi_nulls->reserve(N); \
+ list_nulls->reserve(N); \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& boundary = orig_boundaries[selected_boundaries[i].first]; \
+ const auto* cvr = \
+ std::get_if<ColumnValueRange<TYPE_##INPUT_PT>>(&boundary.boundary_cvr); \
+ DCHECK(cvr != nullptr); \
+ DCHECK(!boundary.only_null); \
+ DCHECK(!boundary.contains_null); \
+ boundary_is_list[i] = cvr->is_fixed_value_range(); \
+ if (boundary_is_list[i]) { \
+ list_result_begin[i] = list_row_count; \
+ for (const auto& value : cvr->get_fixed_value_set()) { \
+ list_inner->insert_value(value); \
+ list_nulls->get_data().push_back(0); \
+ ++list_row_count; \
+ } \
+ list_result_end[i] = list_row_count; \
+ continue; \
+ } \
+ lo_open[i] = cvr->is_low_value_minimum(); \
+ hi_open[i] = cvr->is_high_value_maximum(); \
+ if (!lo_open[i]) { \
+ lo_result_row[i] = lo_row_count++; \
+ lo_inner->insert_value(cvr->get_range_min_value()); \
+ lo_nulls->get_data().push_back(0); \
+ } \
+ if (!hi_open[i]) { \
+ hi_result_row[i] = hi_row_count++; \
+ hi_inner->insert_value(cvr->get_range_max_value()); \
+ hi_nulls->get_data().push_back(0); \
+ } \
+ } \
+ add_dummy_columns(lo_block, lo_row_count); \
+ add_dummy_columns(hi_block, hi_row_count); \
+ add_dummy_columns(list_block, list_row_count); \
+ if (input_is_nullable) { \
+ auto lo_col = ColumnNullable::create(std::move(lo_inner_base), std::move(lo_nulls)); \
+ auto hi_col = ColumnNullable::create(std::move(hi_inner_base), std::move(hi_nulls)); \
+ auto list_col = \
+ ColumnNullable::create(std::move(list_inner_base), std::move(list_nulls)); \
+ lo_block.insert({std::move(lo_col), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_col), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_col), input_type, "leaf_slot"}); \
+ } else { \
+ lo_block.insert({std::move(lo_inner_base), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_inner_base), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_inner_base), input_type, "leaf_slot"}); \
+ } \
+ input_built = true; \
+ break; \
+ }
+
+ switch (input_ptype) {
+ BUILD_INPUT_COLUMNS(TINYINT)
+ BUILD_INPUT_COLUMNS(SMALLINT)
+ BUILD_INPUT_COLUMNS(INT)
+ BUILD_INPUT_COLUMNS(BIGINT)
+ BUILD_INPUT_COLUMNS(LARGEINT)
+ BUILD_INPUT_COLUMNS(FLOAT)
+ BUILD_INPUT_COLUMNS(DOUBLE)
+ BUILD_INPUT_COLUMNS(CHAR)
+ BUILD_INPUT_COLUMNS(DATE)
+ BUILD_INPUT_COLUMNS(DATETIME)
+ BUILD_INPUT_COLUMNS(DATEV2)
+ BUILD_INPUT_COLUMNS(DATETIMEV2)
+ BUILD_INPUT_COLUMNS(TIMESTAMPTZ)
+ BUILD_INPUT_COLUMNS(VARCHAR)
+ BUILD_INPUT_COLUMNS(STRING)
+ BUILD_INPUT_COLUMNS(DECIMAL32)
+ BUILD_INPUT_COLUMNS(DECIMAL64)
+ BUILD_INPUT_COLUMNS(DECIMAL128I)
+ BUILD_INPUT_COLUMNS(DECIMAL256)
+ BUILD_INPUT_COLUMNS(DECIMALV2)
+ BUILD_INPUT_COLUMNS(BOOLEAN)
+ BUILD_INPUT_COLUMNS(IPV4)
+ BUILD_INPUT_COLUMNS(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_INPUT_COLUMNS
+
+ DORIS_CHECK(input_built);
+
+ int lo_result_id = -1;
+ int hi_result_id = -1;
+ int list_result_id = -1;
+ ColumnPtr lo_result_col;
+ ColumnPtr hi_result_col;
+ ColumnPtr list_result_col;
+ if (lo_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &lo_block, &lo_result_id));
+ DORIS_CHECK(lo_result_id >= 0);
+ lo_result_col = lo_block.get_by_position(lo_result_id).column;
+ }
+ if (hi_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &hi_block, &hi_result_id));
+ DORIS_CHECK(hi_result_id >= 0);
+ hi_result_col = hi_block.get_by_position(hi_result_id).column;
+ }
+ if (list_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &list_block, &list_result_id));
+ DORIS_CHECK(list_result_id >= 0);
+ list_result_col = list_block.get_by_position(list_result_id).column;
+ }
+ int out_precision = cast_set<int>(target_expr->data_type()->get_precision());
+ int out_scale = cast_set<int>(target_expr->data_type()->get_scale());
+ bool out_nullable = target_expr->data_type()->is_nullable();
+ PrimitiveType output_ptype = target_expr->data_type()->get_primitive_type();
+
+ projected.reserve(N);
+
+ bool output_built = false;
+#define BUILD_PROJECTED_CVR(OUTPUT_PT) \
+ case TYPE_##OUTPUT_PT: { \
+ using OutputCppType = typename PrimitiveTypeTraits<TYPE_##OUTPUT_PT>::CppType; \
+ auto projected_value = [](const ColumnPtr& column, size_t row) -> OutputCppType { \
+ Field field = (*column)[row]; \
+ return field.get<TYPE_##OUTPUT_PT>(); \
+ }; \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& orig_boundary = orig_boundaries[selected_boundaries[i].first]; \
+ TTargetExprMonotonicity::type direction = selected_boundaries[i].second; \
+ if (boundary_is_list[i]) { \
+ auto cvr = ColumnValueRange<TYPE_##OUTPUT_PT>::create_empty_column_value_range( \
+ out_nullable, out_precision, out_scale); \
+ bool list_has_value = false; \
+ bool list_has_null = false; \
+ for (size_t row = list_result_begin[i]; row < list_result_end[i]; ++row) { \
+ if (list_result_col->is_null_at(row)) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ auto projected_list_value = projected_value(list_result_col, row); \
+ static_cast<void>(cvr.add_fixed_value(projected_list_value)); \
+ list_has_value = true; \
+ } \
+ if (!list_has_value && !list_has_null) { \
+ continue; \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.contains_null = list_has_null; \
+ projected_boundary.only_null = list_has_null && !list_has_value; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ continue; \
+ } \
+ if ((!lo_open[i] && lo_result_col->is_null_at(lo_result_row[i])) || \
+ (!hi_open[i] && hi_result_col->is_null_at(hi_result_row[i]))) { \
+ continue; \
+ } \
+ ColumnValueRange<TYPE_##OUTPUT_PT> cvr("", out_nullable, out_precision, out_scale); \
+ /* Use closed projected finite bounds as a conservative envelope for */ \
+ /* non-strict monotonic expressions such as date ceil/floor/cast. */ \
+ if (direction == TTargetExprMonotonicity::MONOTONIC_DECREASING) { \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, hi_projected)); \
+ } \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, lo_projected)); \
+ } \
+ } else { \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, lo_projected)); \
+ } \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, hi_projected)); \
+ } \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.only_null = orig_boundary.only_null; \
+ projected_boundary.contains_null = orig_boundary.contains_null; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ } \
+ output_built = true; \
+ break; \
+ }
+
+ switch (output_ptype) {
+ BUILD_PROJECTED_CVR(TINYINT)
+ BUILD_PROJECTED_CVR(SMALLINT)
+ BUILD_PROJECTED_CVR(INT)
+ BUILD_PROJECTED_CVR(BIGINT)
+ BUILD_PROJECTED_CVR(LARGEINT)
+ BUILD_PROJECTED_CVR(FLOAT)
+ BUILD_PROJECTED_CVR(DOUBLE)
+ BUILD_PROJECTED_CVR(CHAR)
+ BUILD_PROJECTED_CVR(DATE)
+ BUILD_PROJECTED_CVR(DATETIME)
+ BUILD_PROJECTED_CVR(DATEV2)
+ BUILD_PROJECTED_CVR(DATETIMEV2)
+ BUILD_PROJECTED_CVR(TIMESTAMPTZ)
+ BUILD_PROJECTED_CVR(VARCHAR)
+ BUILD_PROJECTED_CVR(STRING)
+ BUILD_PROJECTED_CVR(DECIMAL32)
+ BUILD_PROJECTED_CVR(DECIMAL64)
+ BUILD_PROJECTED_CVR(DECIMAL128I)
+ BUILD_PROJECTED_CVR(DECIMAL256)
+ BUILD_PROJECTED_CVR(DECIMALV2)
+ BUILD_PROJECTED_CVR(BOOLEAN)
+ BUILD_PROJECTED_CVR(IPV4)
+ BUILD_PROJECTED_CVR(IPV6)
+ default:
+ break;
+ }
+
+#undef BUILD_PROJECTED_CVR
+
+ DORIS_CHECK(output_built);
+
+ return store_projected_boundaries(std::move(projected));
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static SQLFilterOp convert_opcode_to_filter_op(TExprOpcode::type op) {
+ switch (op) {
+ case TExprOpcode::LE:
+ return FILTER_LESS_OR_EQUAL;
+ case TExprOpcode::LT:
+ return FILTER_LESS;
+ case TExprOpcode::GE:
+ return FILTER_LARGER_OR_EQUAL;
+ case TExprOpcode::GT:
+ return FILTER_LARGER;
+ default:
+ return FILTER_IN; // sentinel: caller should skip
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+void RuntimeFilterPartitionPruner::_try_prune_by_single_rf(
+ const std::vector<ParsedBoundary>& boundaries, const VExprSPtr& impl,
+ phmap::flat_hash_set<int64_t>& newly_pruned) {
+ // Pre-compute whether the RF "matches NULL" -- i.e. whether the RF would
+ // accept a row whose probe value is NULL. The signal differs by RF impl:
+ // * IN filter → HybridSet::contain_null()
+ // * Bloom filter → BloomFilterFuncBase::contain_null()
+ // * MinMax filter → encoded via the BinaryPredicate node_type:
+ // NULL_AWARE_BINARY_PRED means the build side had a
+ // NULL key and a null-safe equal join asked NULL to
+ // match NULL; plain BINARY_PRED never matches NULL.
+ // FilterBase::contain_null() already folds in `_null_aware`, so we only
+ // get a true result when the build side is actually null-aware AND
+ // produced a NULL value.
+ bool rf_contains_null = false;
+ if (auto hybrid_set = impl->get_set_func()) {
+ rf_contains_null = hybrid_set->contain_null();
+ } else if (impl->node_type() == TExprNodeType::BLOOM_PRED) {
+ auto bloom = impl->get_bloom_filter_func();
+ rf_contains_null = bloom && bloom->contain_null();
+ } else if (impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) {
+ // Min/Max RF built on a null-safe equal join. The literal child holds
+ // the min or max bound; the NULL semantic is conveyed by the node
+ // type itself (see create_vbin_predicate in runtime_filter/utils.cpp).
+ rf_contains_null = true;
+ }
+
+ std::optional<ColumnValueRangeType> rf_cvr;
+ if (!boundaries.empty()) {
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+ using CppType = typename CvrType::CppType;
+
+ auto hybrid_set = impl->get_set_func();
+ if (hybrid_set) {
+ auto typed_rf_cvr = CvrType::create_empty_column_value_range(
+ boundaries.front().is_nullable, boundary_cvr.precision(),
+ boundary_cvr.scale());
+ auto* iter = hybrid_set->begin();
+ while (iter->has_next()) {
+ const void* value = iter->get_value();
+ if (value) {
+ if constexpr (std::is_same_v<CppType, String>) {
+ // String HybridSets store StringRef*, but
+ // ColumnValueRange<String>::CppType is
+ // std::string. Construct from the bytes.
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else if constexpr (std::is_same_v<CppType, StringRef>) {
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else {
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ *reinterpret_cast<const CppType*>(value)));
+ }
+ }
+ iter->next();
+ }
+ rf_cvr.emplace(std::move(typed_rf_cvr));
+ } else if ((impl->node_type() == TExprNodeType::BINARY_PRED ||
+ impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) &&
+ impl->children().size() == 2 && impl->children()[1]->is_literal()) {
+ // MinMax filter: binary pred with literal bound
+ auto* literal = assert_cast<VLiteral*>(impl->children()[1].get());
+ auto col_ptr = literal->get_column_ptr();
+ auto data = col_ptr->get_data_at(0);
+ CppType val {};
+ if constexpr (std::is_same_v<CppType, String>) {
+ // get_data_at returns a StringRef pointing at the
+ // raw character bytes; ColumnValueRange<String>'s
+ // CppType is std::string, so construct one from
+ // the bytes rather than dereferencing as String.
+ val = CppType(data.data, data.size);
+ } else if constexpr (std::is_same_v<CppType, StringRef>) {
+ val = CppType(data.data, data.size);
+ } else {
+ val = *reinterpret_cast<const CppType*>(data.data);
+ }
+
+ SQLFilterOp op = convert_opcode_to_filter_op(impl->op());
+ if (op == FILTER_IN) {
+ return; // unrecognized opcode, skip
+ }
+
+ CvrType typed_rf_cvr(boundary_cvr.column_name(),
+ boundaries.front().is_nullable,
+ boundary_cvr.precision(), boundary_cvr.scale());
+ static_cast<void>(typed_rf_cvr.add_range(op, val));
+ rf_cvr.emplace(std::move(typed_rf_cvr));
+ }
+ },
+ boundaries.front().boundary_cvr);
+ }
+ if (!rf_cvr.has_value()) {
+ return;
+ }
+
+ for (const auto& pb : boundaries) {
+ if (_pruned_partition_ids.contains(pb.partition_id) ||
+ newly_pruned.contains(pb.partition_id)) {
+ continue;
+ }
+
+ // NULL handling:
+ // A partition row whose key is NULL matches the RF iff `rf_contains_null`.
+ // - only_null partition (rows are exclusively NULL): prunable iff !rf_contains_null.
+ // - mixed (NULL + concrete values): if rf_contains_null, NULL rows alone
+ // prevent pruning. Otherwise NULL rows can never match, so we ignore
+ // the NULL portion and let the regular non-NULL intersection decide.
+ if (pb.only_null) {
+ if (!rf_contains_null) {
+ newly_pruned.insert(pb.partition_id);
+ }
+ continue;
+ }
+ if (pb.contains_null && rf_contains_null) {
+ continue;
+ }
+
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+
+ const auto* typed_rf_cvr = std::get_if<CvrType>(&rf_cvr.value());
+ DORIS_CHECK(typed_rf_cvr != nullptr);
+
+ auto boundary_copy = boundary_cvr;
+ auto rf_cvr_copy = *typed_rf_cvr;
+ boundary_copy.intersection(rf_cvr_copy);
+ if (boundary_copy.is_empty_value_range()) {
+ newly_pruned.insert(pb.partition_id);
+ }
+ },
+ pb.boundary_cvr);
+ }
+}
+
+Status RuntimeFilterPartitionPruner::prune_by_runtime_filters(
+ const ParsedPartitionBoundaries& parsed, const VExprContextSPtrs& conjuncts,
+ const std::vector<TRuntimeFilterDesc>& rf_descs, int scan_node_id,
+ int64_t* newly_pruned_count) {
+ *newly_pruned_count = 0;
+ if (parsed.empty()) {
+ return Status::OK();
+ }
+ const auto& slot_to_boundaries = parsed.slot_to_boundaries();
+
+ // Build filter_id -> per-partition monotonicity maps.
+ std::unordered_map<int, std::unordered_map<int64_t, TTargetExprMonotonicity::type>>
+ filter_id_to_partition_monotonicity;
+ for (const auto& desc : rf_descs) {
+ if (desc.__isset.planId_to_partition_target_monotonicity) {
+ auto it = desc.planId_to_partition_target_monotonicity.find(scan_node_id);
+ if (it != desc.planId_to_partition_target_monotonicity.end()) {
+ auto& partition_monotonicity = filter_id_to_partition_monotonicity[desc.filter_id];
+ for (const auto& partition_entry : it->second) {
+ DORIS_CHECK(partition_entry.__isset.partition_id);
+ DORIS_CHECK(partition_entry.__isset.monotonicity);
+ DORIS_CHECK(partition_entry.monotonicity !=
+ TTargetExprMonotonicity::NON_MONOTONIC);
+ auto [_, inserted] = partition_monotonicity.emplace(
+ partition_entry.partition_id, partition_entry.monotonicity);
+ DORIS_CHECK(inserted);
+ }
+ }
+ }
+ }
+
+ // This function is serialized by _conjuncts_lock in the caller, so our reads
+ // of _pruned_partition_ids never race with our writes below. The only concurrent
+ // readers are is_partition_pruned() calls (under shared_lock), which are
+ // properly synchronized by the unique_lock we take when inserting.
+ phmap::flat_hash_set<int64_t> newly_pruned;
+
+ for (const auto& conjunct_ctx : conjuncts) {
+ VExprSPtr root = conjunct_ctx->root();
+ if (!root->is_rf_wrapper()) {
+ continue;
+ }
+
+ VExprSPtr impl = root->get_impl();
+ if (!impl) {
+ continue;
+ }
+
+ if (impl->children().empty()) {
+ continue;
+ }
+
+ auto* wrapper_root = assert_cast<RuntimeFilterExpr*>(root.get());
+ int filter_id = wrapper_root->filter_id();
+
+ VExprSPtr target_subtree = impl->children()[0];
+
+ auto partition_mono_it = filter_id_to_partition_monotonicity.find(filter_id);
+ bool has_partition_mono = partition_mono_it != filter_id_to_partition_monotonicity.end() &&
+ !partition_mono_it->second.empty();
+ if (!has_partition_mono) {
Review Comment:
这里需要讨论一下，如果是list分区，这个单调的判断是不是没什么意义。list分区本身就是全算一遍的，不过这个可以后续来改进
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3330549387
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,866 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ DORIS_CHECK(tb.__isset.partition_id);
+ DORIS_CHECK(tb.__isset.slot_id);
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ DORIS_CHECK(slot_it != slot_descs.end());
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ DORIS_CHECK(is_list || is_range); \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ bool range_has_null = false; \
+ if (is_nullable && !tb.__isset.range_start) { \
+ range_has_null = true; \
+ } \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } else if (is_nullable) { \
+ range_has_null = true; \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ if (range_has_null) { \
+ boundary.contains_null = true; \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ DORIS_CHECK(parsed_ok);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static bool find_unique_slot_ref_impl(const VExpr* expr, const VSlotRef** found) {
+ if (!expr) {
+ return true;
+ }
+ if (expr->is_slot_ref()) {
+ const auto* slot = assert_cast<const VSlotRef*>(expr);
+ if (*found == nullptr) {
+ *found = slot;
+ return true;
+ }
+ if ((*found)->slot_id() != slot->slot_id()) {
+ return false;
+ }
+ DORIS_CHECK((*found)->column_id() == slot->column_id());
+ return true;
+ }
+ for (const auto& child : expr->children()) {
+ if (!find_unique_slot_ref_impl(child.get(), found)) {
+ return false;
+ }
+ }
+ return true;
+}
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ const VSlotRef* found = nullptr;
+ if (!find_unique_slot_ref_impl(expr, &found)) {
+ return nullptr;
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+Status ParsedPartitionBoundaries::get_or_compute_projected_boundaries(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ const std::unordered_map<int64_t, TTargetExprMonotonicity::type>& partition_directions,
+ VExprContext* ctx, std::shared_ptr<const std::vector<ParsedBoundary>>* output) const {
+ {
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto it = _projected_boundaries_by_filter_id.find(filter_id);
+ if (it != _projected_boundaries_by_filter_id.end()) {
+ *output = it->second;
+ return Status::OK();
+ }
+ }
+
+ std::vector<ParsedBoundary> projected;
+ auto store_projected_boundaries = [&](std::vector<ParsedBoundary>&& boundaries) {
+ auto cached = std::make_shared<const std::vector<ParsedBoundary>>(std::move(boundaries));
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto [it, inserted] = _projected_boundaries_by_filter_id.emplace(filter_id, cached);
+ *output = inserted ? cached : it->second;
+ return Status::OK();
+ };
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::unordered_set<int64_t> boundary_partition_ids;
+ boundary_partition_ids.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ boundary_partition_ids.insert(boundary.partition_id);
+ }
+ for (const auto& [partition_id, direction] : partition_directions) {
+ DORIS_CHECK(direction != TTargetExprMonotonicity::NON_MONOTONIC);
+ DORIS_CHECK(boundary_partition_ids.contains(partition_id));
+ }
+
+ if (target_expr->is_slot_ref()) {
+ auto* slot_ref = assert_cast<VSlotRef*>(target_expr.get());
+ DORIS_CHECK(slot_ref->slot_id() == leaf_slot_id);
+ std::vector<ParsedBoundary> slot_boundaries;
+ slot_boundaries.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ auto direction_it = partition_directions.find(boundary.partition_id);
+ if (direction_it == partition_directions.end()) {
+ continue;
+ }
+ DORIS_CHECK(direction_it->second == TTargetExprMonotonicity::MONOTONIC_INCREASING);
+ slot_boundaries.emplace_back(boundary);
+ }
+ return store_projected_boundaries(std::move(slot_boundaries));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> selected_boundaries;
+ selected_boundaries.reserve(orig_boundaries.size());
+ for (size_t i = 0; i < orig_boundaries.size(); ++i) {
+ auto it = partition_directions.find(orig_boundaries[i].partition_id);
+ if (it != partition_directions.end() &&
+ it->second != TTargetExprMonotonicity::NON_MONOTONIC) {
+ selected_boundaries.emplace_back(i, it->second);
+ }
+ }
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> projectable_boundaries;
+ projectable_boundaries.reserve(selected_boundaries.size());
+ for (const auto& selected_boundary : selected_boundaries) {
+ const auto& boundary = orig_boundaries[selected_boundary.first];
+ if (!boundary.only_null && !boundary.contains_null) {
+ projectable_boundaries.emplace_back(selected_boundary);
+ }
+ }
+ selected_boundaries.swap(projectable_boundaries);
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ DORIS_CHECK(slot_type_it != _slot_data_types.end());
+
+ const DataTypePtr& input_type = slot_type_it->second;
+ bool input_is_nullable = input_type->is_nullable();
+ DataTypePtr inner_type = input_is_nullable ? remove_nullable(input_type) : input_type;
+ PrimitiveType input_ptype = input_type->get_primitive_type();
+
+ size_t N = selected_boundaries.size();
+ std::vector<bool> lo_open(N, false);
+ std::vector<bool> hi_open(N, false);
+ std::vector<bool> boundary_is_list(N, false);
+ std::vector<size_t> lo_result_row(N, 0);
+ std::vector<size_t> hi_result_row(N, 0);
+ std::vector<size_t> list_result_begin(N, 0);
+ std::vector<size_t> list_result_end(N, 0);
+ size_t lo_row_count = 0;
+ size_t hi_row_count = 0;
+ size_t list_row_count = 0;
+
+ // Build input blocks for RANGE endpoints and LIST values.
+ Block lo_block;
+ Block hi_block;
+ Block list_block;
+
+ // Pad columns 0..leaf_column_id-1 with placeholder columns so the leaf
+ // VSlotRef (whose column_id is leaf_column_id) reads our typed column.
+ auto add_dummy_columns = [&](Block& block, size_t row_count) {
+ for (int col_idx = 0; col_idx < leaf_column_id; ++col_idx) {
+ auto col = ColumnUInt8::create(row_count, 0);
+ block.insert({std::move(col), std::make_shared<DataTypeUInt8>(),
+ fmt::format("dummy_{}", col_idx)});
+ }
+ };
+
+ // Macro-dispatch on input PrimitiveType to build finite RANGE endpoint
+ // columns and LIST value columns. Unbounded RANGE endpoints are not executed
+ // through target_expr; they stay unbounded after projection and swap sides
+ // for monotonic decreasing targets.
+ bool input_built = false;
+#define BUILD_INPUT_COLUMNS(INPUT_PT) \
+ case TYPE_##INPUT_PT: { \
+ using InCol = typename PrimitiveTypeTraits<TYPE_##INPUT_PT>::ColumnType; \
+ MutableColumnPtr lo_inner_base = inner_type->create_column(); \
+ MutableColumnPtr hi_inner_base = inner_type->create_column(); \
+ MutableColumnPtr list_inner_base = inner_type->create_column(); \
+ auto* lo_inner = assert_cast<InCol*>(lo_inner_base.get()); \
+ auto* hi_inner = assert_cast<InCol*>(hi_inner_base.get()); \
+ auto* list_inner = assert_cast<InCol*>(list_inner_base.get()); \
+ lo_inner->reserve(N); \
+ hi_inner->reserve(N); \
+ list_inner->reserve(N); \
+ auto lo_nulls = ColumnUInt8::create(); \
+ auto hi_nulls = ColumnUInt8::create(); \
+ auto list_nulls = ColumnUInt8::create(); \
+ lo_nulls->reserve(N); \
+ hi_nulls->reserve(N); \
+ list_nulls->reserve(N); \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& boundary = orig_boundaries[selected_boundaries[i].first]; \
+ const auto* cvr = \
+ std::get_if<ColumnValueRange<TYPE_##INPUT_PT>>(&boundary.boundary_cvr); \
+ DCHECK(cvr != nullptr); \
+ DCHECK(!boundary.only_null); \
+ DCHECK(!boundary.contains_null); \
+ boundary_is_list[i] = cvr->is_fixed_value_range(); \
+ if (boundary_is_list[i]) { \
+ list_result_begin[i] = list_row_count; \
+ for (const auto& value : cvr->get_fixed_value_set()) { \
+ list_inner->insert_value(value); \
+ list_nulls->get_data().push_back(0); \
+ ++list_row_count; \
+ } \
+ list_result_end[i] = list_row_count; \
+ continue; \
+ } \
+ lo_open[i] = cvr->is_low_value_minimum(); \
+ hi_open[i] = cvr->is_high_value_maximum(); \
+ if (!lo_open[i]) { \
+ lo_result_row[i] = lo_row_count++; \
+ lo_inner->insert_value(cvr->get_range_min_value()); \
+ lo_nulls->get_data().push_back(0); \
+ } \
+ if (!hi_open[i]) { \
+ hi_result_row[i] = hi_row_count++; \
+ hi_inner->insert_value(cvr->get_range_max_value()); \
+ hi_nulls->get_data().push_back(0); \
+ } \
+ } \
+ add_dummy_columns(lo_block, lo_row_count); \
+ add_dummy_columns(hi_block, hi_row_count); \
+ add_dummy_columns(list_block, list_row_count); \
+ if (input_is_nullable) { \
+ auto lo_col = ColumnNullable::create(std::move(lo_inner_base), std::move(lo_nulls)); \
+ auto hi_col = ColumnNullable::create(std::move(hi_inner_base), std::move(hi_nulls)); \
+ auto list_col = \
+ ColumnNullable::create(std::move(list_inner_base), std::move(list_nulls)); \
+ lo_block.insert({std::move(lo_col), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_col), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_col), input_type, "leaf_slot"}); \
+ } else { \
+ lo_block.insert({std::move(lo_inner_base), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_inner_base), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_inner_base), input_type, "leaf_slot"}); \
+ } \
+ input_built = true; \
+ break; \
+ }
+
+ switch (input_ptype) {
+ BUILD_INPUT_COLUMNS(TINYINT)
+ BUILD_INPUT_COLUMNS(SMALLINT)
+ BUILD_INPUT_COLUMNS(INT)
+ BUILD_INPUT_COLUMNS(BIGINT)
+ BUILD_INPUT_COLUMNS(LARGEINT)
+ BUILD_INPUT_COLUMNS(FLOAT)
+ BUILD_INPUT_COLUMNS(DOUBLE)
+ BUILD_INPUT_COLUMNS(CHAR)
+ BUILD_INPUT_COLUMNS(DATE)
+ BUILD_INPUT_COLUMNS(DATETIME)
+ BUILD_INPUT_COLUMNS(DATEV2)
+ BUILD_INPUT_COLUMNS(DATETIMEV2)
+ BUILD_INPUT_COLUMNS(TIMESTAMPTZ)
+ BUILD_INPUT_COLUMNS(VARCHAR)
+ BUILD_INPUT_COLUMNS(STRING)
+ BUILD_INPUT_COLUMNS(DECIMAL32)
+ BUILD_INPUT_COLUMNS(DECIMAL64)
+ BUILD_INPUT_COLUMNS(DECIMAL128I)
+ BUILD_INPUT_COLUMNS(DECIMAL256)
+ BUILD_INPUT_COLUMNS(DECIMALV2)
+ BUILD_INPUT_COLUMNS(BOOLEAN)
+ BUILD_INPUT_COLUMNS(IPV4)
+ BUILD_INPUT_COLUMNS(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_INPUT_COLUMNS
+
+ DORIS_CHECK(input_built);
+
+ int lo_result_id = -1;
+ int hi_result_id = -1;
+ int list_result_id = -1;
+ ColumnPtr lo_result_col;
+ ColumnPtr hi_result_col;
+ ColumnPtr list_result_col;
+ if (lo_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &lo_block, &lo_result_id));
+ DORIS_CHECK(lo_result_id >= 0);
+ lo_result_col = lo_block.get_by_position(lo_result_id).column;
+ }
+ if (hi_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &hi_block, &hi_result_id));
+ DORIS_CHECK(hi_result_id >= 0);
+ hi_result_col = hi_block.get_by_position(hi_result_id).column;
+ }
+ if (list_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &list_block, &list_result_id));
+ DORIS_CHECK(list_result_id >= 0);
+ list_result_col = list_block.get_by_position(list_result_id).column;
+ }
+ int out_precision = cast_set<int>(target_expr->data_type()->get_precision());
+ int out_scale = cast_set<int>(target_expr->data_type()->get_scale());
+ bool out_nullable = target_expr->data_type()->is_nullable();
+ PrimitiveType output_ptype = target_expr->data_type()->get_primitive_type();
+
+ projected.reserve(N);
+
+ bool output_built = false;
+#define BUILD_PROJECTED_CVR(OUTPUT_PT) \
+ case TYPE_##OUTPUT_PT: { \
+ using OutputCppType = typename PrimitiveTypeTraits<TYPE_##OUTPUT_PT>::CppType; \
+ auto projected_value = [](const ColumnPtr& column, size_t row) -> OutputCppType { \
+ Field field = (*column)[row]; \
+ return field.get<TYPE_##OUTPUT_PT>(); \
+ }; \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& orig_boundary = orig_boundaries[selected_boundaries[i].first]; \
+ TTargetExprMonotonicity::type direction = selected_boundaries[i].second; \
+ if (boundary_is_list[i]) { \
+ auto cvr = ColumnValueRange<TYPE_##OUTPUT_PT>::create_empty_column_value_range( \
+ out_nullable, out_precision, out_scale); \
+ bool list_has_value = false; \
+ bool list_has_null = false; \
+ for (size_t row = list_result_begin[i]; row < list_result_end[i]; ++row) { \
+ if (list_result_col->is_null_at(row)) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ auto projected_list_value = projected_value(list_result_col, row); \
+ static_cast<void>(cvr.add_fixed_value(projected_list_value)); \
+ list_has_value = true; \
+ } \
+ if (!list_has_value && !list_has_null) { \
+ continue; \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.contains_null = list_has_null; \
+ projected_boundary.only_null = list_has_null && !list_has_value; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ continue; \
+ } \
+ if ((!lo_open[i] && lo_result_col->is_null_at(lo_result_row[i])) || \
+ (!hi_open[i] && hi_result_col->is_null_at(hi_result_row[i]))) { \
+ continue; \
+ } \
+ ColumnValueRange<TYPE_##OUTPUT_PT> cvr("", out_nullable, out_precision, out_scale); \
+ /* Use closed projected finite bounds as a conservative envelope for */ \
+ /* non-strict monotonic expressions such as date ceil/floor/cast. */ \
+ if (direction == TTargetExprMonotonicity::MONOTONIC_DECREASING) { \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, hi_projected)); \
+ } \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, lo_projected)); \
+ } \
+ } else { \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, lo_projected)); \
+ } \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, hi_projected)); \
+ } \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.only_null = orig_boundary.only_null; \
+ projected_boundary.contains_null = orig_boundary.contains_null; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ } \
+ output_built = true; \
+ break; \
+ }
+
+ switch (output_ptype) {
+ BUILD_PROJECTED_CVR(TINYINT)
+ BUILD_PROJECTED_CVR(SMALLINT)
+ BUILD_PROJECTED_CVR(INT)
+ BUILD_PROJECTED_CVR(BIGINT)
+ BUILD_PROJECTED_CVR(LARGEINT)
+ BUILD_PROJECTED_CVR(FLOAT)
+ BUILD_PROJECTED_CVR(DOUBLE)
+ BUILD_PROJECTED_CVR(CHAR)
+ BUILD_PROJECTED_CVR(DATE)
+ BUILD_PROJECTED_CVR(DATETIME)
+ BUILD_PROJECTED_CVR(DATEV2)
+ BUILD_PROJECTED_CVR(DATETIMEV2)
+ BUILD_PROJECTED_CVR(TIMESTAMPTZ)
+ BUILD_PROJECTED_CVR(VARCHAR)
+ BUILD_PROJECTED_CVR(STRING)
+ BUILD_PROJECTED_CVR(DECIMAL32)
+ BUILD_PROJECTED_CVR(DECIMAL64)
+ BUILD_PROJECTED_CVR(DECIMAL128I)
+ BUILD_PROJECTED_CVR(DECIMAL256)
+ BUILD_PROJECTED_CVR(DECIMALV2)
+ BUILD_PROJECTED_CVR(BOOLEAN)
+ BUILD_PROJECTED_CVR(IPV4)
+ BUILD_PROJECTED_CVR(IPV6)
+ default:
+ break;
+ }
+
+#undef BUILD_PROJECTED_CVR
+
+ DORIS_CHECK(output_built);
+
+ return store_projected_boundaries(std::move(projected));
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static SQLFilterOp convert_opcode_to_filter_op(TExprOpcode::type op) {
+ switch (op) {
+ case TExprOpcode::LE:
+ return FILTER_LESS_OR_EQUAL;
+ case TExprOpcode::LT:
+ return FILTER_LESS;
+ case TExprOpcode::GE:
+ return FILTER_LARGER_OR_EQUAL;
+ case TExprOpcode::GT:
+ return FILTER_LARGER;
+ default:
+ return FILTER_IN; // sentinel: caller should skip
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+void RuntimeFilterPartitionPruner::_try_prune_by_single_rf(
+ const std::vector<ParsedBoundary>& boundaries, const VExprSPtr& impl,
+ phmap::flat_hash_set<int64_t>& newly_pruned) {
+ // Pre-compute whether the RF "matches NULL" -- i.e. whether the RF would
+ // accept a row whose probe value is NULL. The signal differs by RF impl:
+ // * IN filter → HybridSet::contain_null()
+ // * Bloom filter → BloomFilterFuncBase::contain_null()
+ // * MinMax filter → encoded via the BinaryPredicate node_type:
+ // NULL_AWARE_BINARY_PRED means the build side had a
+ // NULL key and a null-safe equal join asked NULL to
+ // match NULL; plain BINARY_PRED never matches NULL.
+ // FilterBase::contain_null() already folds in `_null_aware`, so we only
+ // get a true result when the build side is actually null-aware AND
+ // produced a NULL value.
+ bool rf_contains_null = false;
+ if (auto hybrid_set = impl->get_set_func()) {
+ rf_contains_null = hybrid_set->contain_null();
+ } else if (impl->node_type() == TExprNodeType::BLOOM_PRED) {
+ auto bloom = impl->get_bloom_filter_func();
+ rf_contains_null = bloom && bloom->contain_null();
+ } else if (impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) {
+ // Min/Max RF built on a null-safe equal join. The literal child holds
+ // the min or max bound; the NULL semantic is conveyed by the node
+ // type itself (see create_vbin_predicate in runtime_filter/utils.cpp).
+ rf_contains_null = true;
+ }
+
+ std::optional<ColumnValueRangeType> rf_cvr;
+ if (!boundaries.empty()) {
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+ using CppType = typename CvrType::CppType;
+
+ auto hybrid_set = impl->get_set_func();
+ if (hybrid_set) {
+ auto typed_rf_cvr = CvrType::create_empty_column_value_range(
+ boundaries.front().is_nullable, boundary_cvr.precision(),
+ boundary_cvr.scale());
+ auto* iter = hybrid_set->begin();
+ while (iter->has_next()) {
+ const void* value = iter->get_value();
+ if (value) {
+ if constexpr (std::is_same_v<CppType, String>) {
+ // String HybridSets store StringRef*, but
+ // ColumnValueRange<String>::CppType is
+ // std::string. Construct from the bytes.
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else if constexpr (std::is_same_v<CppType, StringRef>) {
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
Review Comment:
这个分支是什么情况会走到呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4594017466
<details>
<summary>TPC-H: <b>Total hot run time: 29888 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b82194ce0fb5ac216aeed034490645f04c19f4e5, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17653 4033 3997 3997
q2 q3 10875 1441 812 812
q4 4752 486 345 345
q5 8400 922 587 587
q6 337 182 138 138
q7 902 865 652 652
q8 11024 1637 1635 1635
q9 7156 4583 4515 4515
q10 6865 1907 1529 1529
q11 456 281 258 258
q12 656 433 292 292
q13 18114 3830 2821 2821
q14 265 261 249 249
q15 q16 814 777 703 703
q17 950 946 1016 946
q18 6770 5742 5771 5742
q19 1178 1367 1268 1268
q20 560 474 287 287
q21 5852 2861 2799 2799
q22 520 368 313 313
Total cold run time: 104099 ms
Total hot run time: 29888 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4840 4799 4744 4744
q2 q3 5015 5294 4634 4634
q4 2117 2198 1390 1390
q5 5071 4789 4587 4587
q6 227 176 130 130
q7 2016 1798 1546 1546
q8 2452 2135 2141 2135
q9 7864 7347 7406 7347
q10 4722 4684 4222 4222
q11 538 397 356 356
q12 733 744 540 540
q13 3042 3409 2857 2857
q14 278 279 266 266
q15 q16 681 698 600 600
q17 1328 1290 1276 1276
q18 7490 6880 6928 6880
q19 1106 1104 1092 1092
q20 2237 2212 2007 2007
q21 5366 4646 4504 4504
q22 528 487 438 438
Total cold run time: 57651 ms
Total hot run time: 51551 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4450386075
<details>
<summary>TPC-H: <b>Total hot run time: 29592 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c090ab6440346db9f2c052585ec7b6dc593d0df3, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17623 3889 3851 3851
q2 q3 10714 887 628 628
q4 4706 469 351 351
q5 7820 1351 1136 1136
q6 310 171 142 142
q7 962 948 765 765
q8 10795 1443 1342 1342
q9 6958 5410 5536 5410
q10 6334 2089 1805 1805
q11 503 269 254 254
q12 699 418 306 306
q13 18199 3417 2774 2774
q14 303 292 262 262
q15 q16 898 880 790 790
q17 1281 1076 678 678
q18 6481 5673 5566 5566
q19 1507 1234 1081 1081
q20 546 404 269 269
q21 4804 2277 1870 1870
q22 426 357 312 312
Total cold run time: 101869 ms
Total hot run time: 29592 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4246 4119 4118 4118
q2 q3 4650 4795 4177 4177
q4 2113 2185 1390 1390
q5 4990 4967 5329 4967
q6 329 202 161 161
q7 2138 1859 1718 1718
q8 3551 3264 3181 3181
q9 8570 8616 8561 8561
q10 4500 4502 4281 4281
q11 647 426 418 418
q12 734 748 527 527
q13 3454 3605 2983 2983
q14 328 313 287 287
q15 q16 790 790 707 707
q17 1373 1335 1304 1304
q18 8036 7126 7210 7126
q19 1177 1196 1191 1191
q20 2244 2280 1982 1982
q21 6214 5449 5205 5205
q22 570 548 437 437
Total cold run time: 60654 ms
Total hot run time: 54721 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4541734955
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3330534056
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,866 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ DORIS_CHECK(tb.__isset.partition_id);
+ DORIS_CHECK(tb.__isset.slot_id);
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ DORIS_CHECK(slot_it != slot_descs.end());
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ DORIS_CHECK(is_list || is_range); \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ bool range_has_null = false; \
+ if (is_nullable && !tb.__isset.range_start) { \
+ range_has_null = true; \
+ } \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } else if (is_nullable) { \
+ range_has_null = true; \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ if (range_has_null) { \
+ boundary.contains_null = true; \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ DORIS_CHECK(parsed_ok);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static bool find_unique_slot_ref_impl(const VExpr* expr, const VSlotRef** found) {
+ if (!expr) {
Review Comment:
什么情况下这个expr会是空指针呢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3330551591
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,866 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ DORIS_CHECK(tb.__isset.partition_id);
+ DORIS_CHECK(tb.__isset.slot_id);
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ DORIS_CHECK(slot_it != slot_descs.end());
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ DORIS_CHECK(is_list || is_range); \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ bool range_has_null = false; \
+ if (is_nullable && !tb.__isset.range_start) { \
+ range_has_null = true; \
+ } \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } else if (is_nullable) { \
+ range_has_null = true; \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ if (range_has_null) { \
+ boundary.contains_null = true; \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ DORIS_CHECK(parsed_ok);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static bool find_unique_slot_ref_impl(const VExpr* expr, const VSlotRef** found) {
+ if (!expr) {
+ return true;
+ }
+ if (expr->is_slot_ref()) {
+ const auto* slot = assert_cast<const VSlotRef*>(expr);
+ if (*found == nullptr) {
+ *found = slot;
+ return true;
+ }
+ if ((*found)->slot_id() != slot->slot_id()) {
+ return false;
+ }
+ DORIS_CHECK((*found)->column_id() == slot->column_id());
+ return true;
+ }
+ for (const auto& child : expr->children()) {
+ if (!find_unique_slot_ref_impl(child.get(), found)) {
+ return false;
+ }
+ }
+ return true;
+}
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ const VSlotRef* found = nullptr;
+ if (!find_unique_slot_ref_impl(expr, &found)) {
+ return nullptr;
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+Status ParsedPartitionBoundaries::get_or_compute_projected_boundaries(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ const std::unordered_map<int64_t, TTargetExprMonotonicity::type>& partition_directions,
+ VExprContext* ctx, std::shared_ptr<const std::vector<ParsedBoundary>>* output) const {
+ {
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto it = _projected_boundaries_by_filter_id.find(filter_id);
+ if (it != _projected_boundaries_by_filter_id.end()) {
+ *output = it->second;
+ return Status::OK();
+ }
+ }
+
+ std::vector<ParsedBoundary> projected;
+ auto store_projected_boundaries = [&](std::vector<ParsedBoundary>&& boundaries) {
+ auto cached = std::make_shared<const std::vector<ParsedBoundary>>(std::move(boundaries));
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto [it, inserted] = _projected_boundaries_by_filter_id.emplace(filter_id, cached);
+ *output = inserted ? cached : it->second;
+ return Status::OK();
+ };
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::unordered_set<int64_t> boundary_partition_ids;
+ boundary_partition_ids.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ boundary_partition_ids.insert(boundary.partition_id);
+ }
+ for (const auto& [partition_id, direction] : partition_directions) {
+ DORIS_CHECK(direction != TTargetExprMonotonicity::NON_MONOTONIC);
+ DORIS_CHECK(boundary_partition_ids.contains(partition_id));
+ }
+
+ if (target_expr->is_slot_ref()) {
+ auto* slot_ref = assert_cast<VSlotRef*>(target_expr.get());
+ DORIS_CHECK(slot_ref->slot_id() == leaf_slot_id);
+ std::vector<ParsedBoundary> slot_boundaries;
+ slot_boundaries.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ auto direction_it = partition_directions.find(boundary.partition_id);
+ if (direction_it == partition_directions.end()) {
+ continue;
+ }
+ DORIS_CHECK(direction_it->second == TTargetExprMonotonicity::MONOTONIC_INCREASING);
+ slot_boundaries.emplace_back(boundary);
+ }
+ return store_projected_boundaries(std::move(slot_boundaries));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> selected_boundaries;
+ selected_boundaries.reserve(orig_boundaries.size());
+ for (size_t i = 0; i < orig_boundaries.size(); ++i) {
+ auto it = partition_directions.find(orig_boundaries[i].partition_id);
+ if (it != partition_directions.end() &&
+ it->second != TTargetExprMonotonicity::NON_MONOTONIC) {
+ selected_boundaries.emplace_back(i, it->second);
+ }
+ }
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> projectable_boundaries;
+ projectable_boundaries.reserve(selected_boundaries.size());
+ for (const auto& selected_boundary : selected_boundaries) {
+ const auto& boundary = orig_boundaries[selected_boundary.first];
+ if (!boundary.only_null && !boundary.contains_null) {
+ projectable_boundaries.emplace_back(selected_boundary);
+ }
+ }
+ selected_boundaries.swap(projectable_boundaries);
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ DORIS_CHECK(slot_type_it != _slot_data_types.end());
+
+ const DataTypePtr& input_type = slot_type_it->second;
+ bool input_is_nullable = input_type->is_nullable();
+ DataTypePtr inner_type = input_is_nullable ? remove_nullable(input_type) : input_type;
+ PrimitiveType input_ptype = input_type->get_primitive_type();
+
+ size_t N = selected_boundaries.size();
+ std::vector<bool> lo_open(N, false);
+ std::vector<bool> hi_open(N, false);
+ std::vector<bool> boundary_is_list(N, false);
+ std::vector<size_t> lo_result_row(N, 0);
+ std::vector<size_t> hi_result_row(N, 0);
+ std::vector<size_t> list_result_begin(N, 0);
+ std::vector<size_t> list_result_end(N, 0);
+ size_t lo_row_count = 0;
+ size_t hi_row_count = 0;
+ size_t list_row_count = 0;
+
+ // Build input blocks for RANGE endpoints and LIST values.
+ Block lo_block;
+ Block hi_block;
+ Block list_block;
+
+ // Pad columns 0..leaf_column_id-1 with placeholder columns so the leaf
+ // VSlotRef (whose column_id is leaf_column_id) reads our typed column.
+ auto add_dummy_columns = [&](Block& block, size_t row_count) {
+ for (int col_idx = 0; col_idx < leaf_column_id; ++col_idx) {
+ auto col = ColumnUInt8::create(row_count, 0);
+ block.insert({std::move(col), std::make_shared<DataTypeUInt8>(),
+ fmt::format("dummy_{}", col_idx)});
+ }
+ };
+
+ // Macro-dispatch on input PrimitiveType to build finite RANGE endpoint
+ // columns and LIST value columns. Unbounded RANGE endpoints are not executed
+ // through target_expr; they stay unbounded after projection and swap sides
+ // for monotonic decreasing targets.
+ bool input_built = false;
+#define BUILD_INPUT_COLUMNS(INPUT_PT) \
+ case TYPE_##INPUT_PT: { \
+ using InCol = typename PrimitiveTypeTraits<TYPE_##INPUT_PT>::ColumnType; \
+ MutableColumnPtr lo_inner_base = inner_type->create_column(); \
+ MutableColumnPtr hi_inner_base = inner_type->create_column(); \
+ MutableColumnPtr list_inner_base = inner_type->create_column(); \
+ auto* lo_inner = assert_cast<InCol*>(lo_inner_base.get()); \
+ auto* hi_inner = assert_cast<InCol*>(hi_inner_base.get()); \
+ auto* list_inner = assert_cast<InCol*>(list_inner_base.get()); \
+ lo_inner->reserve(N); \
+ hi_inner->reserve(N); \
+ list_inner->reserve(N); \
+ auto lo_nulls = ColumnUInt8::create(); \
+ auto hi_nulls = ColumnUInt8::create(); \
+ auto list_nulls = ColumnUInt8::create(); \
+ lo_nulls->reserve(N); \
+ hi_nulls->reserve(N); \
+ list_nulls->reserve(N); \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& boundary = orig_boundaries[selected_boundaries[i].first]; \
+ const auto* cvr = \
+ std::get_if<ColumnValueRange<TYPE_##INPUT_PT>>(&boundary.boundary_cvr); \
+ DCHECK(cvr != nullptr); \
+ DCHECK(!boundary.only_null); \
+ DCHECK(!boundary.contains_null); \
+ boundary_is_list[i] = cvr->is_fixed_value_range(); \
+ if (boundary_is_list[i]) { \
+ list_result_begin[i] = list_row_count; \
+ for (const auto& value : cvr->get_fixed_value_set()) { \
+ list_inner->insert_value(value); \
+ list_nulls->get_data().push_back(0); \
+ ++list_row_count; \
+ } \
+ list_result_end[i] = list_row_count; \
+ continue; \
+ } \
+ lo_open[i] = cvr->is_low_value_minimum(); \
+ hi_open[i] = cvr->is_high_value_maximum(); \
+ if (!lo_open[i]) { \
+ lo_result_row[i] = lo_row_count++; \
+ lo_inner->insert_value(cvr->get_range_min_value()); \
+ lo_nulls->get_data().push_back(0); \
+ } \
+ if (!hi_open[i]) { \
+ hi_result_row[i] = hi_row_count++; \
+ hi_inner->insert_value(cvr->get_range_max_value()); \
+ hi_nulls->get_data().push_back(0); \
+ } \
+ } \
+ add_dummy_columns(lo_block, lo_row_count); \
+ add_dummy_columns(hi_block, hi_row_count); \
+ add_dummy_columns(list_block, list_row_count); \
+ if (input_is_nullable) { \
+ auto lo_col = ColumnNullable::create(std::move(lo_inner_base), std::move(lo_nulls)); \
+ auto hi_col = ColumnNullable::create(std::move(hi_inner_base), std::move(hi_nulls)); \
+ auto list_col = \
+ ColumnNullable::create(std::move(list_inner_base), std::move(list_nulls)); \
+ lo_block.insert({std::move(lo_col), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_col), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_col), input_type, "leaf_slot"}); \
+ } else { \
+ lo_block.insert({std::move(lo_inner_base), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_inner_base), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_inner_base), input_type, "leaf_slot"}); \
+ } \
+ input_built = true; \
+ break; \
+ }
+
+ switch (input_ptype) {
+ BUILD_INPUT_COLUMNS(TINYINT)
+ BUILD_INPUT_COLUMNS(SMALLINT)
+ BUILD_INPUT_COLUMNS(INT)
+ BUILD_INPUT_COLUMNS(BIGINT)
+ BUILD_INPUT_COLUMNS(LARGEINT)
+ BUILD_INPUT_COLUMNS(FLOAT)
+ BUILD_INPUT_COLUMNS(DOUBLE)
+ BUILD_INPUT_COLUMNS(CHAR)
+ BUILD_INPUT_COLUMNS(DATE)
+ BUILD_INPUT_COLUMNS(DATETIME)
+ BUILD_INPUT_COLUMNS(DATEV2)
+ BUILD_INPUT_COLUMNS(DATETIMEV2)
+ BUILD_INPUT_COLUMNS(TIMESTAMPTZ)
+ BUILD_INPUT_COLUMNS(VARCHAR)
+ BUILD_INPUT_COLUMNS(STRING)
+ BUILD_INPUT_COLUMNS(DECIMAL32)
+ BUILD_INPUT_COLUMNS(DECIMAL64)
+ BUILD_INPUT_COLUMNS(DECIMAL128I)
+ BUILD_INPUT_COLUMNS(DECIMAL256)
+ BUILD_INPUT_COLUMNS(DECIMALV2)
+ BUILD_INPUT_COLUMNS(BOOLEAN)
+ BUILD_INPUT_COLUMNS(IPV4)
+ BUILD_INPUT_COLUMNS(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_INPUT_COLUMNS
+
+ DORIS_CHECK(input_built);
+
+ int lo_result_id = -1;
+ int hi_result_id = -1;
+ int list_result_id = -1;
+ ColumnPtr lo_result_col;
+ ColumnPtr hi_result_col;
+ ColumnPtr list_result_col;
+ if (lo_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &lo_block, &lo_result_id));
+ DORIS_CHECK(lo_result_id >= 0);
+ lo_result_col = lo_block.get_by_position(lo_result_id).column;
+ }
+ if (hi_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &hi_block, &hi_result_id));
+ DORIS_CHECK(hi_result_id >= 0);
+ hi_result_col = hi_block.get_by_position(hi_result_id).column;
+ }
+ if (list_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &list_block, &list_result_id));
+ DORIS_CHECK(list_result_id >= 0);
+ list_result_col = list_block.get_by_position(list_result_id).column;
+ }
+ int out_precision = cast_set<int>(target_expr->data_type()->get_precision());
+ int out_scale = cast_set<int>(target_expr->data_type()->get_scale());
+ bool out_nullable = target_expr->data_type()->is_nullable();
+ PrimitiveType output_ptype = target_expr->data_type()->get_primitive_type();
+
+ projected.reserve(N);
+
+ bool output_built = false;
+#define BUILD_PROJECTED_CVR(OUTPUT_PT) \
+ case TYPE_##OUTPUT_PT: { \
+ using OutputCppType = typename PrimitiveTypeTraits<TYPE_##OUTPUT_PT>::CppType; \
+ auto projected_value = [](const ColumnPtr& column, size_t row) -> OutputCppType { \
+ Field field = (*column)[row]; \
+ return field.get<TYPE_##OUTPUT_PT>(); \
+ }; \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& orig_boundary = orig_boundaries[selected_boundaries[i].first]; \
+ TTargetExprMonotonicity::type direction = selected_boundaries[i].second; \
+ if (boundary_is_list[i]) { \
+ auto cvr = ColumnValueRange<TYPE_##OUTPUT_PT>::create_empty_column_value_range( \
+ out_nullable, out_precision, out_scale); \
+ bool list_has_value = false; \
+ bool list_has_null = false; \
+ for (size_t row = list_result_begin[i]; row < list_result_end[i]; ++row) { \
+ if (list_result_col->is_null_at(row)) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ auto projected_list_value = projected_value(list_result_col, row); \
+ static_cast<void>(cvr.add_fixed_value(projected_list_value)); \
+ list_has_value = true; \
+ } \
+ if (!list_has_value && !list_has_null) { \
+ continue; \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.contains_null = list_has_null; \
+ projected_boundary.only_null = list_has_null && !list_has_value; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ continue; \
+ } \
+ if ((!lo_open[i] && lo_result_col->is_null_at(lo_result_row[i])) || \
+ (!hi_open[i] && hi_result_col->is_null_at(hi_result_row[i]))) { \
+ continue; \
+ } \
+ ColumnValueRange<TYPE_##OUTPUT_PT> cvr("", out_nullable, out_precision, out_scale); \
+ /* Use closed projected finite bounds as a conservative envelope for */ \
+ /* non-strict monotonic expressions such as date ceil/floor/cast. */ \
+ if (direction == TTargetExprMonotonicity::MONOTONIC_DECREASING) { \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, hi_projected)); \
+ } \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, lo_projected)); \
+ } \
+ } else { \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, lo_projected)); \
+ } \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, hi_projected)); \
+ } \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.only_null = orig_boundary.only_null; \
+ projected_boundary.contains_null = orig_boundary.contains_null; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ } \
+ output_built = true; \
+ break; \
+ }
+
+ switch (output_ptype) {
+ BUILD_PROJECTED_CVR(TINYINT)
+ BUILD_PROJECTED_CVR(SMALLINT)
+ BUILD_PROJECTED_CVR(INT)
+ BUILD_PROJECTED_CVR(BIGINT)
+ BUILD_PROJECTED_CVR(LARGEINT)
+ BUILD_PROJECTED_CVR(FLOAT)
+ BUILD_PROJECTED_CVR(DOUBLE)
+ BUILD_PROJECTED_CVR(CHAR)
+ BUILD_PROJECTED_CVR(DATE)
+ BUILD_PROJECTED_CVR(DATETIME)
+ BUILD_PROJECTED_CVR(DATEV2)
+ BUILD_PROJECTED_CVR(DATETIMEV2)
+ BUILD_PROJECTED_CVR(TIMESTAMPTZ)
+ BUILD_PROJECTED_CVR(VARCHAR)
+ BUILD_PROJECTED_CVR(STRING)
+ BUILD_PROJECTED_CVR(DECIMAL32)
+ BUILD_PROJECTED_CVR(DECIMAL64)
+ BUILD_PROJECTED_CVR(DECIMAL128I)
+ BUILD_PROJECTED_CVR(DECIMAL256)
+ BUILD_PROJECTED_CVR(DECIMALV2)
+ BUILD_PROJECTED_CVR(BOOLEAN)
+ BUILD_PROJECTED_CVR(IPV4)
+ BUILD_PROJECTED_CVR(IPV6)
+ default:
+ break;
+ }
+
+#undef BUILD_PROJECTED_CVR
+
+ DORIS_CHECK(output_built);
+
+ return store_projected_boundaries(std::move(projected));
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static SQLFilterOp convert_opcode_to_filter_op(TExprOpcode::type op) {
+ switch (op) {
+ case TExprOpcode::LE:
+ return FILTER_LESS_OR_EQUAL;
+ case TExprOpcode::LT:
+ return FILTER_LESS;
+ case TExprOpcode::GE:
+ return FILTER_LARGER_OR_EQUAL;
+ case TExprOpcode::GT:
+ return FILTER_LARGER;
+ default:
+ return FILTER_IN; // sentinel: caller should skip
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+void RuntimeFilterPartitionPruner::_try_prune_by_single_rf(
+ const std::vector<ParsedBoundary>& boundaries, const VExprSPtr& impl,
+ phmap::flat_hash_set<int64_t>& newly_pruned) {
+ // Pre-compute whether the RF "matches NULL" -- i.e. whether the RF would
+ // accept a row whose probe value is NULL. The signal differs by RF impl:
+ // * IN filter → HybridSet::contain_null()
+ // * Bloom filter → BloomFilterFuncBase::contain_null()
+ // * MinMax filter → encoded via the BinaryPredicate node_type:
+ // NULL_AWARE_BINARY_PRED means the build side had a
+ // NULL key and a null-safe equal join asked NULL to
+ // match NULL; plain BINARY_PRED never matches NULL.
+ // FilterBase::contain_null() already folds in `_null_aware`, so we only
+ // get a true result when the build side is actually null-aware AND
+ // produced a NULL value.
+ bool rf_contains_null = false;
+ if (auto hybrid_set = impl->get_set_func()) {
+ rf_contains_null = hybrid_set->contain_null();
+ } else if (impl->node_type() == TExprNodeType::BLOOM_PRED) {
+ auto bloom = impl->get_bloom_filter_func();
+ rf_contains_null = bloom && bloom->contain_null();
+ } else if (impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) {
+ // Min/Max RF built on a null-safe equal join. The literal child holds
+ // the min or max bound; the NULL semantic is conveyed by the node
+ // type itself (see create_vbin_predicate in runtime_filter/utils.cpp).
+ rf_contains_null = true;
+ }
+
+ std::optional<ColumnValueRangeType> rf_cvr;
+ if (!boundaries.empty()) {
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+ using CppType = typename CvrType::CppType;
+
+ auto hybrid_set = impl->get_set_func();
+ if (hybrid_set) {
+ auto typed_rf_cvr = CvrType::create_empty_column_value_range(
+ boundaries.front().is_nullable, boundary_cvr.precision(),
+ boundary_cvr.scale());
+ auto* iter = hybrid_set->begin();
+ while (iter->has_next()) {
+ const void* value = iter->get_value();
+ if (value) {
+ if constexpr (std::is_same_v<CppType, String>) {
+ // String HybridSets store StringRef*, but
+ // ColumnValueRange<String>::CppType is
+ // std::string. Construct from the bytes.
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else if constexpr (std::is_same_v<CppType, StringRef>) {
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else {
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ *reinterpret_cast<const CppType*>(value)));
+ }
+ }
+ iter->next();
+ }
+ rf_cvr.emplace(std::move(typed_rf_cvr));
+ } else if ((impl->node_type() == TExprNodeType::BINARY_PRED ||
+ impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) &&
+ impl->children().size() == 2 && impl->children()[1]->is_literal()) {
+ // MinMax filter: binary pred with literal bound
+ auto* literal = assert_cast<VLiteral*>(impl->children()[1].get());
+ auto col_ptr = literal->get_column_ptr();
+ auto data = col_ptr->get_data_at(0);
+ CppType val {};
+ if constexpr (std::is_same_v<CppType, String>) {
+ // get_data_at returns a StringRef pointing at the
+ // raw character bytes; ColumnValueRange<String>'s
+ // CppType is std::string, so construct one from
+ // the bytes rather than dereferencing as String.
+ val = CppType(data.data, data.size);
+ } else if constexpr (std::is_same_v<CppType, StringRef>) {
+ val = CppType(data.data, data.size);
+ } else {
+ val = *reinterpret_cast<const CppType*>(data.data);
+ }
+
+ SQLFilterOp op = convert_opcode_to_filter_op(impl->op());
Review Comment:
这个不是TExprNodeType::BINARY_PRED，为什么会变成in了呢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4593476114
# FE Regression Coverage Report
Increment line coverage ` 48.07% (224/466)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_b82194ce0fb5ac216aeed034490645f04c19f4e5_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_b82194ce0fb5ac216aeed034490645f04c19f4e5_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3297174711
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java:
##########
@@ -195,12 +200,24 @@ private void createLegacyRuntimeFilterFromGroup(List<RuntimeFilter> group,
new RuntimeFilterExpressionTranslator(targetSlotRef);
targetExpr = curTargetExpression.accept(translator, ctx);
}
+ boolean adjustedAfterNonIdentity = false;
+ if (!src.getType().equals(targetExpr.getType()) && head.getType() != TRuntimeFilterType.BITMAP) {
+ adjustedAfterNonIdentity = !isIdentityTarget;
+ }
targetExpr = castTargetToSourceTypeIfNeeded(src, targetExpr, head.getType());
TupleId targetTupleId = targetSlotRef.getDesc().getParentId();
SlotId targetSlotId = targetSlotRef.getSlotId();
scanNodeList.add(scanNode);
targetExprList.add(targetExpr);
+ nereidsTargetExprList.add(curTargetExpression);
targetTupleIdMapList.add(ImmutableMap.of(targetTupleId, ImmutableList.of(targetSlotId)));
+ targetAdjustedAfterNonIdentityList.add(adjustedAfterNonIdentity);
+ int scanNodeId = scanNode.getId().asInt();
+ String targetExprSql = targetExpr.debugString();
Review Comment:
do not use debugstring in normal code flow, It not only has poor performance but also may throw exceptions. if u want generate sql from expr, you should use `ExprToSqlVisitor`
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterPartitionPruneClassifier.java:
##########
@@ -0,0 +1,295 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.glue.translator;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.FunctionCallExpr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotRef;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.ListPartitionItem;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.PartitionInfo;
+import org.apache.doris.catalog.PartitionItem;
+import org.apache.doris.catalog.PartitionKey;
+import org.apache.doris.catalog.PartitionType;
+import org.apache.doris.catalog.RangePartitionItem;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.Monotonic;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.planner.OlapScanNode;
+import org.apache.doris.planner.PlanNode;
+import org.apache.doris.thrift.TTargetExprMonotonicity;
+
+import com.google.common.collect.Range;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Classifies whether one runtime-filter target can safely drive BE-side partition pruning.
+ *
+ * <p>This is the single FE gate for RF partition pruning. It intentionally reasons about the
+ * final legacy target expression that will be sent to BE, so late-added casts and partition
+ * expression domains cannot bypass the safety checks.
+ */
+final class RuntimeFilterPartitionPruneClassifier {
+ private RuntimeFilterPartitionPruneClassifier() {
+ }
+
+ static Classification classify(Expr targetExpr, Expression nereidsTargetExpr, PlanNode scanNode) {
+ if (!(scanNode instanceof OlapScanNode)) {
+ return Classification.unsupported("target scan is not an OlapScanNode");
+ }
+
+ OlapScanNode olapScanNode = (OlapScanNode) scanNode;
+ OlapTable table = olapScanNode.getOlapTable();
+ if (table == null) {
+ return Classification.unsupported("target scan has no OlapTable");
+ }
+
+ PartitionInfo partitionInfo = table.getPartitionInfo();
+ PartitionType partType = partitionInfo.getType();
+ if (partType != PartitionType.RANGE && partType != PartitionType.LIST) {
+ return Classification.unsupported("partition type is not RANGE or LIST");
+ }
+ if (hasUnsupportedAutomaticPartitionExpression(partitionInfo)) {
+ return Classification.unsupported("automatic partition expression boundary is not modeled");
+ }
+
+ if (targetExpr instanceof SlotRef) {
+ SlotRef slotRef = (SlotRef) targetExpr;
+ if (!isPartitionColumnSlot(slotRef, partitionInfo.getPartitionColumns())) {
+ return Classification.unsupported("target SlotRef is not a partition column");
+ }
+ if (!hasSerializedBoundary(slotRef, partitionInfo, partType)) {
+ return Classification.unsupported("target SlotRef has no serialized partition boundary");
+ }
+ Map<Long, TTargetExprMonotonicity> partitionMonotonicity =
+ allSelectedPartitionsIncreasing(olapScanNode, partitionInfo);
+ if (partitionMonotonicity.isEmpty()) {
+ return Classification.unsupported("target SlotRef has no prunable selected partitions");
+ }
+ return Classification.supportedPartitions(slotRef, partitionMonotonicity);
+ }
+
+ SlotRef leafSlot = findUniqueSlotRef(targetExpr);
+ if (leafSlot == null || !isPartitionColumnSlot(leafSlot, partitionInfo.getPartitionColumns())) {
+ return Classification.unsupported("target expression is not rooted on one partition column");
+ }
+ if (partType != PartitionType.RANGE) {
+ return Classification.unsupported("local monotonicity is only supported for RANGE partition");
+ }
+ if (!hasSerializedBoundary(leafSlot, partitionInfo, partType)) {
+ return Classification.unsupported("target expression has no serialized partition boundary");
+ }
+
+ Map<Long, TTargetExprMonotonicity> partitionMonotonicity =
+ classifyLocalMonotonicity(nereidsTargetExpr, olapScanNode, partitionInfo, leafSlot);
+ if (partitionMonotonicity.isEmpty()) {
+ return Classification.unsupported("target expression is not monotonic on selected partitions");
+ }
+ return Classification.supportedPartitions(leafSlot, partitionMonotonicity);
+ }
+
+ private static boolean hasUnsupportedAutomaticPartitionExpression(PartitionInfo partitionInfo) {
+ if (!partitionInfo.enableAutomaticPartition()) {
+ return false;
+ }
+ for (Expr partitionExpr : partitionInfo.getPartitionExprs()) {
+ if (containsFunctionCall(partitionExpr)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static boolean containsFunctionCall(Expr expr) {
+ if (expr instanceof FunctionCallExpr) {
+ return true;
+ }
+ for (Expr child : expr.getChildren()) {
+ if (containsFunctionCall(child)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static boolean hasSerializedBoundary(SlotRef slotRef, PartitionInfo partitionInfo, PartitionType partType) {
+ if (partType != PartitionType.RANGE) {
+ return true;
+ }
+ List<Column> partitionColumns = partitionInfo.getPartitionColumns();
+ return !partitionColumns.isEmpty() && sameColumn(slotRef.getColumn(), partitionColumns.get(0));
+ }
+
+ private static boolean isPartitionColumnSlot(SlotRef slotRef, List<Column> partitionColumns) {
+ Column targetColumn = slotRef.getColumn();
+ if (targetColumn == null) {
+ return false;
+ }
+ for (Column partitionColumn : partitionColumns) {
+ if (sameColumn(targetColumn, partitionColumn)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static boolean sameColumn(Column targetColumn, Column partitionColumn) {
+ if (targetColumn == partitionColumn) {
+ return true;
+ }
+ int targetUniqueId = targetColumn.getUniqueId();
+ int partitionUniqueId = partitionColumn.getUniqueId();
+ if (targetUniqueId != Column.COLUMN_UNIQUE_ID_INIT_VALUE
+ && partitionUniqueId != Column.COLUMN_UNIQUE_ID_INIT_VALUE
+ && targetUniqueId == partitionUniqueId) {
+ return true;
+ }
+ return targetColumn.equals(partitionColumn);
+ }
+
+ private static SlotRef findUniqueSlotRef(Expr expr) {
+ if (expr instanceof SlotRef) {
+ return (SlotRef) expr;
+ }
+ SlotRef result = null;
+ for (Expr child : expr.getChildren()) {
+ SlotRef childSlot = findUniqueSlotRef(child);
+ if (childSlot == null) {
+ continue;
+ }
+ if (result != null && result.getSlotId().asInt() != childSlot.getSlotId().asInt()) {
+ return null;
+ }
+ result = childSlot;
+ }
+ return result;
+ }
+
+ private static Map<Long, TTargetExprMonotonicity> classifyLocalMonotonicity(
+ Expression nereidsTargetExpr, OlapScanNode scanNode, PartitionInfo partitionInfo, SlotRef leafSlot) {
+ Map<Long, TTargetExprMonotonicity> result = new HashMap<>();
+ if (!(nereidsTargetExpr instanceof Monotonic) || nereidsTargetExpr.getInputSlots().size() != 1) {
Review Comment:
getInputSlots return set. so if a expr has multi input with same slot, the getInputSlots().size() equals 1, but the expr maybe not Monotonicity for this slot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4439141217
<details>
<summary>TPC-DS: <b>Total hot run time: 169452 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2337ac55b6530b92cabafe14881c96951fab95c9, data reload: false
query5 4308 637 487 487
query6 430 204 175 175
query7 4835 554 299 299
query8 367 208 238 208
query9 8827 4151 4164 4151
query10 440 311 255 255
query11 5944 2356 2270 2270
query12 155 99 99 99
query13 1255 599 435 435
query14 6468 5324 5016 5016
query14_1 4351 4316 4283 4283
query15 202 201 178 178
query16 1041 461 441 441
query17 1145 756 624 624
query18 2727 486 351 351
query19 212 193 154 154
query20 111 108 104 104
query21 216 139 114 114
query22 13684 13858 14600 13858
query23 17394 16491 16177 16177
query23_1 16280 16243 16195 16195
query24 7527 1786 1362 1362
query24_1 1358 1360 1393 1360
query25 567 452 372 372
query26 1303 307 167 167
query27 2619 604 336 336
query28 4413 2034 2000 2000
query29 1055 608 489 489
query30 302 238 202 202
query31 1095 1053 934 934
query32 101 56 55 55
query33 533 323 245 245
query34 1186 1171 655 655
query35 738 772 662 662
query36 1319 1325 1167 1167
query37 148 101 84 84
query38 3193 3147 3097 3097
query39 917 915 899 899
query39_1 873 863 867 863
query40 218 136 109 109
query41 62 59 63 59
query42 91 90 89 89
query43 313 315 277 277
query44
query45 193 189 181 181
query46 1125 1234 708 708
query47 2264 2205 2248 2205
query48 401 428 310 310
query49 635 496 397 397
query50 728 300 213 213
query51 4293 4253 4175 4175
query52 88 86 73 73
query53 239 258 191 191
query54 255 208 189 189
query55 76 71 68 68
query56 225 223 207 207
query57 1398 1380 1265 1265
query58 231 222 210 210
query59 1568 1694 1410 1410
query60 288 260 235 235
query61 156 155 168 155
query62 675 618 555 555
query63 222 185 185 185
query64 2529 800 632 632
query65
query66 1709 477 357 357
query67 29470 29955 29233 29233
query68
query69 440 305 276 276
query70 930 948 968 948
query71 299 238 213 213
query72 2942 2753 2387 2387
query73 881 827 433 433
query74 5109 4914 4744 4744
query75 2747 2631 2300 2300
query76 2347 1180 750 750
query77 360 403 302 302
query78 12964 12933 12370 12370
query79 1491 1046 774 774
query80 1331 484 390 390
query81 527 278 242 242
query82 623 156 122 122
query83 332 265 253 253
query84 262 140 112 112
query85 900 492 425 425
query86 424 305 288 288
query87 3452 3275 3171 3171
query88 3775 2845 2822 2822
query89 423 372 326 326
query90 1958 186 179 179
query91 182 173 136 136
query92 59 63 52 52
query93 1133 958 573 573
query94 712 335 314 314
query95 682 372 424 372
query96 1077 830 349 349
query97 2659 2655 2506 2506
query98 211 203 200 200
query99 1095 1133 1005 1005
Total cold run time: 251965 ms
Total hot run time: 169452 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4433603193
# FE Regression Coverage Report
Increment line coverage ` 67.35% (198/294)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_e7a084166584e3ccdc5ae8b6bd5fc50d3c0cb832_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_e7a084166584e3ccdc5ae8b6bd5fc50d3c0cb832_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4432339457
<details>
<summary>TPC-H: <b>Total hot run time: 29820 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e7a084166584e3ccdc5ae8b6bd5fc50d3c0cb832, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17582 3892 3875 3875
q2 q3 10708 898 619 619
q4 4666 458 347 347
q5 7447 1331 1147 1147
q6 193 169 143 143
q7 915 952 761 761
q8 9315 1438 1290 1290
q9 5578 5360 5321 5321
q10 6282 2080 1798 1798
q11 478 275 261 261
q12 631 431 296 296
q13 18094 3313 2761 2761
q14 295 291 263 263
q15 q16 896 844 800 800
q17 928 1004 774 774
q18 6568 5768 5662 5662
q19 1170 1204 1098 1098
q20 541 408 274 274
q21 4458 2446 1986 1986
q22 506 405 344 344
Total cold run time: 97251 ms
Total hot run time: 29820 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4745 4756 4782 4756
q2 q3 4670 4818 4232 4232
q4 2141 2183 1401 1401
q5 5045 5045 5313 5045
q6 204 176 137 137
q7 2033 1808 1604 1604
q8 3405 3064 3136 3064
q9 8527 8469 8510 8469
q10 4583 4507 4260 4260
q11 604 434 403 403
q12 714 750 517 517
q13 3185 3603 3004 3004
q14 302 296 265 265
q15 q16 769 782 718 718
q17 1498 1317 1266 1266
q18 8046 7136 7278 7136
q19 1139 1162 1160 1160
q20 2232 2224 1946 1946
q21 6218 5510 4945 4945
q22 575 529 418 418
Total cold run time: 60635 ms
Total hot run time: 54746 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4448452198
<details>
<summary>TPC-H: <b>Total hot run time: 29653 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5bfeb4d44af5550f9edd89c84a3d05b18135d530, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17629 3907 3949 3907
q2 q3 10791 900 617 617
q4 4729 472 348 348
q5 8267 1343 1151 1151
q6 370 177 138 138
q7 940 957 768 768
q8 11249 1501 1268 1268
q9 7537 5388 5336 5336
q10 6421 2122 1847 1847
q11 473 260 254 254
q12 687 426 302 302
q13 18153 3269 2747 2747
q14 294 288 268 268
q15 q16 906 862 810 810
q17 942 1041 653 653
q18 6532 5742 5628 5628
q19 1171 1374 1098 1098
q20 541 422 287 287
q21 4729 2320 1918 1918
q22 435 367 308 308
Total cold run time: 102796 ms
Total hot run time: 29653 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4631 4573 4588 4573
q2 q3 4703 4818 4236 4236
q4 2122 2178 1385 1385
q5 4989 4998 5277 4998
q6 196 162 131 131
q7 2054 1798 1663 1663
q8 3497 3298 3166 3166
q9 8484 8390 8426 8390
q10 4508 4460 4273 4273
q11 627 422 401 401
q12 693 755 538 538
q13 3287 3637 2950 2950
q14 315 301 269 269
q15 q16 769 801 705 705
q17 1314 1290 1274 1274
q18 7952 7144 7160 7144
q19 1176 1141 1085 1085
q20 2192 2230 1941 1941
q21 6134 5392 4795 4795
q22 526 472 404 404
Total cold run time: 60169 ms
Total hot run time: 54321 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3108102326
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,371 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <utility>
+
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// Helper to extract a typed value from a ColumnPtr's first row.
+// Must be a template function so that `if constexpr` properly discards
+// the inapplicable branch during instantiation.
+template <PrimitiveType PT>
+static typename PrimitiveTypeTraits<PT>::CppType extract_value_from_column(const ColumnPtr& col) {
+ using CppType = typename PrimitiveTypeTraits<PT>::CppType;
+ auto data = col->get_data_at(0);
+ if constexpr (is_string_type(PT)) {
+ return CppType(data.data, data.size);
+ } else {
+ return *reinterpret_cast<const CppType*>(data.data);
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+void RuntimeFilterPartitionPruner::parse_boundaries(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ if (!tb.__isset.partition_id || !tb.__isset.slot_id) {
+ continue;
+ }
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ if (slot_it == slot_descs.end()) {
+ continue;
+ }
+ SlotDescriptor* slot = slot_it->second;
+ PrimitiveType ptype = slot->type()->get_primitive_type();
+ int precision = cast_set<int>(slot->type()->get_precision());
+ int scale = cast_set<int>(slot->type()->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ if (!is_list && !is_range) break; \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = \
+ [&](const TExprNode& node) -> std::optional<std::pair<CppType, ColumnPtr>> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ VLiteral literal(node); \
+ auto col_ptr = literal.get_column_ptr(); \
+ auto val = extract_value_from_column<TYPE_##NAME>(col_ptr); \
+ return std::make_optional(std::pair<CppType, ColumnPtr> {val, col_ptr}); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ auto& [val, col_ptr] = *parsed; \
+ boundary.literal_columns.push_back(std::move(col_ptr)); \
+ static_cast<void>(empty_cvr.add_fixed_value(val)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ cvr.set_contain_null(true); \
Review Comment:
When a LIST boundary contains both NULL and concrete values, `cvr.intersection(empty_cvr)` first materializes the concrete fixed-value set, but `set_contain_null(true)` immediately calls `set_empty_value_range()` and discards those fixed values. That turns a partition like `{NULL, 5}` into a NULL-only boundary, so `_try_prune_by_single_rf()` can later prune it even when the runtime filter contains `5`. Please preserve both the concrete values and the NULL marker here instead of resetting the CVR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4278332442
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4493991499
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4505745196
# BE Regression && UT Coverage Report
Increment line coverage `75.28% (603/801)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.50% (27845/37883) |
| Line Coverage | 57.40% (302409/526849) |
| Region Coverage | 54.25% (253050/466433) |
| Branch Coverage | 55.77% (109737/196776) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4507178536
<details>
<summary>TPC-DS: <b>Total hot run time: 168747 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6be29ffaf962c85e6c243c68a1c26cf413cdb0b5, data reload: false
query5 4335 663 518 518
query6 469 215 193 193
query7 4818 572 321 321
query8 379 227 226 226
query9 8841 4065 4063 4063
query10 485 332 273 273
query11 5940 2427 2218 2218
query12 157 102 98 98
query13 1350 566 431 431
query14 6462 5326 5003 5003
query14_1 4362 4361 4299 4299
query15 206 196 179 179
query16 999 496 464 464
query17 1150 718 603 603
query18 2505 489 375 375
query19 214 205 150 150
query20 115 106 107 106
query21 219 140 125 125
query22 13634 13531 13468 13468
query23 17293 16304 15929 15929
query23_1 16196 16084 16124 16084
query24 7518 1773 1342 1342
query24_1 1323 1339 1300 1300
query25 591 479 423 423
query26 1316 315 177 177
query27 2660 546 333 333
query28 4475 1962 1942 1942
query29 1109 634 526 526
query30 305 236 198 198
query31 1107 1049 945 945
query32 111 62 59 59
query33 524 321 246 246
query34 1167 1118 650 650
query35 733 777 677 677
query36 1322 1307 1187 1187
query37 151 106 93 93
query38 3216 3138 3063 3063
query39 928 941 919 919
query39_1 879 904 877 877
query40 223 122 101 101
query41 64 63 65 63
query42 92 96 96 96
query43 316 313 275 275
query44
query45 194 185 176 176
query46 1072 1218 749 749
query47 2198 2207 2217 2207
query48 400 421 296 296
query49 626 468 346 346
query50 1042 344 248 248
query51 4347 4276 4246 4246
query52 89 93 81 81
query53 247 267 193 193
query54 267 220 194 194
query55 82 74 70 70
query56 237 231 230 230
query57 1391 1363 1282 1282
query58 247 212 222 212
query59 1550 1594 1378 1378
query60 281 251 230 230
query61 158 162 157 157
query62 668 630 571 571
query63 232 192 191 191
query64 2574 790 608 608
query65
query66 1787 455 342 342
query67 30050 30035 29757 29757
query68
query69 442 317 272 272
query70 971 932 947 932
query71 304 229 208 208
query72 2900 2719 2438 2438
query73 880 768 419 419
query74 5067 4918 4725 4725
query75 2643 2561 2258 2258
query76 2329 1128 772 772
query77 354 378 285 285
query78 12155 12123 11684 11684
query79 1458 1087 770 770
query80 1277 489 386 386
query81 522 282 241 241
query82 606 156 122 122
query83 328 266 243 243
query84 257 143 108 108
query85 909 571 438 438
query86 430 305 294 294
query87 3342 3385 3178 3178
query88 3585 2666 2677 2666
query89 423 377 322 322
query90 1936 179 199 179
query91 181 176 146 146
query92 63 60 58 58
query93 1540 1463 868 868
query94 726 363 306 306
query95 689 468 346 346
query96 1009 756 356 356
query97 2679 2683 2537 2537
query98 216 209 209 209
query99 1137 1088 955 955
Total cold run time: 251840 ms
Total hot run time: 168747 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4563131394
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3310231274
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,875 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ if (!tb.__isset.partition_id || !tb.__isset.slot_id) {
+ return Status::InternalError(
+ "Runtime filter partition boundary must contain partition_id and slot_id");
+ }
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ if (slot_it == slot_descs.end()) {
+ return Status::InternalError(
+ "Runtime filter partition boundary references unknown slot_id={}, "
+ "partition_id={}",
+ slot_id, tb.partition_id);
+ }
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ if (!is_list && !is_range) { \
+ return Status::InternalError( \
+ "Runtime filter partition boundary must be RANGE or LIST, " \
+ "partition_id={}, slot_id={}", \
+ tb.partition_id, slot_id); \
+ } \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ if (parsed_ok) {
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ } else {
+ return Status::InternalError(
+ "Runtime filter partition boundary has unsupported type, partition_id={}, "
+ "slot_id={}, type={}",
+ tb.partition_id, slot_id, slot_type->get_name());
+ }
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ if (!expr) {
+ return nullptr;
+ }
+ if (expr->is_slot_ref()) {
+ return assert_cast<const VSlotRef*>(expr);
+ }
+ const VSlotRef* found = nullptr;
+ for (const auto& child : expr->children()) {
+ const VSlotRef* c = find_unique_slot_ref(child.get());
+ if (c) {
+ if (found) {
+ return nullptr; // multiple slot refs, can't handle
+ }
+ found = c;
+ }
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+Status ParsedPartitionBoundaries::get_or_compute_projected_boundaries(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ const std::unordered_map<int64_t, TTargetExprMonotonicity::type>& partition_directions,
+ VExprContext* ctx, std::shared_ptr<const std::vector<ParsedBoundary>>* output) const {
+ {
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto it = _projected_boundaries_by_filter_id.find(filter_id);
+ if (it != _projected_boundaries_by_filter_id.end()) {
+ *output = it->second;
+ return Status::OK();
+ }
+ }
+
+ std::vector<ParsedBoundary> projected;
+ auto store_projected_boundaries = [&](std::vector<ParsedBoundary>&& boundaries) {
+ auto cached = std::make_shared<const std::vector<ParsedBoundary>>(std::move(boundaries));
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto [it, inserted] = _projected_boundaries_by_filter_id.emplace(filter_id, cached);
+ *output = inserted ? cached : it->second;
+ return Status::OK();
+ };
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::unordered_set<int64_t> boundary_partition_ids;
+ boundary_partition_ids.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ boundary_partition_ids.insert(boundary.partition_id);
+ }
+ for (const auto& [partition_id, direction] : partition_directions) {
+ if (direction == TTargetExprMonotonicity::NON_MONOTONIC) {
+ return Status::InternalError(
+ "Runtime filter partition pruning received NON_MONOTONIC partition "
+ "metadata, filter_id={}, partition_id={}",
+ filter_id, partition_id);
+ }
+ if (!boundary_partition_ids.contains(partition_id)) {
+ return Status::InternalError(
+ "Runtime filter partition pruning received monotonicity for a partition "
+ "without boundary, filter_id={}, partition_id={}, slot_id={}",
+ filter_id, partition_id, leaf_slot_id);
+ }
+ }
+
+ if (target_expr->is_slot_ref()) {
+ auto* slot_ref = assert_cast<VSlotRef*>(target_expr.get());
+ if (slot_ref->slot_id() != leaf_slot_id) {
+ return Status::InternalError(
+ "Runtime filter partition pruning SlotRef target mismatch, filter_id={}, "
+ "target_slot_id={}, leaf_slot_id={}",
+ filter_id, slot_ref->slot_id(), leaf_slot_id);
+ }
+ std::vector<ParsedBoundary> slot_boundaries;
+ slot_boundaries.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ auto direction_it = partition_directions.find(boundary.partition_id);
+ if (direction_it == partition_directions.end()) {
+ continue;
+ }
+ if (direction_it->second != TTargetExprMonotonicity::MONOTONIC_INCREASING) {
+ return Status::InternalError(
+ "Runtime filter partition pruning SlotRef target must use increasing "
+ "monotonicity, filter_id={}, partition_id={}, monotonicity={}",
+ filter_id, boundary.partition_id, direction_it->second);
+ }
+ slot_boundaries.emplace_back(boundary);
+ }
+ return store_projected_boundaries(std::move(slot_boundaries));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> selected_boundaries;
+ selected_boundaries.reserve(orig_boundaries.size());
+ for (size_t i = 0; i < orig_boundaries.size(); ++i) {
+ auto it = partition_directions.find(orig_boundaries[i].partition_id);
+ if (it != partition_directions.end() &&
+ it->second != TTargetExprMonotonicity::NON_MONOTONIC) {
+ selected_boundaries.emplace_back(i, it->second);
+ }
+ }
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> projectable_boundaries;
+ projectable_boundaries.reserve(selected_boundaries.size());
+ for (const auto& selected_boundary : selected_boundaries) {
+ const auto& boundary = orig_boundaries[selected_boundary.first];
+ if (!boundary.only_null && !boundary.contains_null) {
+ projectable_boundaries.emplace_back(selected_boundary);
+ }
+ }
+ selected_boundaries.swap(projectable_boundaries);
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ if (slot_type_it == _slot_data_types.end()) {
+ return Status::InternalError(
+ "Runtime filter partition pruning target slot has no boundary type, filter_id={}, "
+ "slot_id={}, target_expr={}",
+ filter_id, leaf_slot_id, target_expr->expr_name());
+ }
+
+ // LIST partitions require regrouping projected values per partition and
+ // should not be marked as expression-prunable by FE yet.
+ for (const auto& selected_boundary : selected_boundaries) {
+ bool is_list = false;
+ std::visit([&](const auto& cvr) { is_list = cvr.is_fixed_value_range(); },
+ orig_boundaries[selected_boundary.first].boundary_cvr);
+ if (is_list) {
+ return Status::InternalError(
+ "Runtime filter partition pruning expression target does not support LIST "
Review Comment:
之前list没支持expr，现在顺便加上了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4555489946
# FE Regression Coverage Report
Increment line coverage ` 68.09% (224/329)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_36cd8599c1b5eba6dcfb83d6c88457494678e987_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_36cd8599c1b5eba6dcfb83d6c88457494678e987_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3309070083
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,875 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ if (!tb.__isset.partition_id || !tb.__isset.slot_id) {
+ return Status::InternalError(
+ "Runtime filter partition boundary must contain partition_id and slot_id");
+ }
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ if (slot_it == slot_descs.end()) {
+ return Status::InternalError(
+ "Runtime filter partition boundary references unknown slot_id={}, "
+ "partition_id={}",
+ slot_id, tb.partition_id);
+ }
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ if (!is_list && !is_range) { \
+ return Status::InternalError( \
+ "Runtime filter partition boundary must be RANGE or LIST, " \
+ "partition_id={}, slot_id={}", \
+ tb.partition_id, slot_id); \
+ } \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ if (parsed_ok) {
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ } else {
+ return Status::InternalError(
+ "Runtime filter partition boundary has unsupported type, partition_id={}, "
+ "slot_id={}, type={}",
+ tb.partition_id, slot_id, slot_type->get_name());
+ }
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ if (!expr) {
+ return nullptr;
+ }
+ if (expr->is_slot_ref()) {
+ return assert_cast<const VSlotRef*>(expr);
+ }
+ const VSlotRef* found = nullptr;
+ for (const auto& child : expr->children()) {
+ const VSlotRef* c = find_unique_slot_ref(child.get());
+ if (c) {
+ if (found) {
+ return nullptr; // multiple slot refs, can't handle
+ }
+ found = c;
+ }
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+Status ParsedPartitionBoundaries::get_or_compute_projected_boundaries(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ const std::unordered_map<int64_t, TTargetExprMonotonicity::type>& partition_directions,
+ VExprContext* ctx, std::shared_ptr<const std::vector<ParsedBoundary>>* output) const {
+ {
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto it = _projected_boundaries_by_filter_id.find(filter_id);
+ if (it != _projected_boundaries_by_filter_id.end()) {
+ *output = it->second;
+ return Status::OK();
+ }
+ }
+
+ std::vector<ParsedBoundary> projected;
+ auto store_projected_boundaries = [&](std::vector<ParsedBoundary>&& boundaries) {
+ auto cached = std::make_shared<const std::vector<ParsedBoundary>>(std::move(boundaries));
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto [it, inserted] = _projected_boundaries_by_filter_id.emplace(filter_id, cached);
+ *output = inserted ? cached : it->second;
+ return Status::OK();
+ };
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::unordered_set<int64_t> boundary_partition_ids;
+ boundary_partition_ids.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ boundary_partition_ids.insert(boundary.partition_id);
+ }
+ for (const auto& [partition_id, direction] : partition_directions) {
+ if (direction == TTargetExprMonotonicity::NON_MONOTONIC) {
+ return Status::InternalError(
+ "Runtime filter partition pruning received NON_MONOTONIC partition "
+ "metadata, filter_id={}, partition_id={}",
+ filter_id, partition_id);
+ }
+ if (!boundary_partition_ids.contains(partition_id)) {
+ return Status::InternalError(
+ "Runtime filter partition pruning received monotonicity for a partition "
+ "without boundary, filter_id={}, partition_id={}, slot_id={}",
+ filter_id, partition_id, leaf_slot_id);
+ }
+ }
+
+ if (target_expr->is_slot_ref()) {
+ auto* slot_ref = assert_cast<VSlotRef*>(target_expr.get());
+ if (slot_ref->slot_id() != leaf_slot_id) {
+ return Status::InternalError(
+ "Runtime filter partition pruning SlotRef target mismatch, filter_id={}, "
+ "target_slot_id={}, leaf_slot_id={}",
+ filter_id, slot_ref->slot_id(), leaf_slot_id);
+ }
+ std::vector<ParsedBoundary> slot_boundaries;
+ slot_boundaries.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ auto direction_it = partition_directions.find(boundary.partition_id);
+ if (direction_it == partition_directions.end()) {
+ continue;
+ }
+ if (direction_it->second != TTargetExprMonotonicity::MONOTONIC_INCREASING) {
+ return Status::InternalError(
+ "Runtime filter partition pruning SlotRef target must use increasing "
+ "monotonicity, filter_id={}, partition_id={}, monotonicity={}",
+ filter_id, boundary.partition_id, direction_it->second);
+ }
+ slot_boundaries.emplace_back(boundary);
+ }
+ return store_projected_boundaries(std::move(slot_boundaries));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> selected_boundaries;
+ selected_boundaries.reserve(orig_boundaries.size());
+ for (size_t i = 0; i < orig_boundaries.size(); ++i) {
+ auto it = partition_directions.find(orig_boundaries[i].partition_id);
+ if (it != partition_directions.end() &&
+ it->second != TTargetExprMonotonicity::NON_MONOTONIC) {
+ selected_boundaries.emplace_back(i, it->second);
+ }
+ }
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> projectable_boundaries;
+ projectable_boundaries.reserve(selected_boundaries.size());
+ for (const auto& selected_boundary : selected_boundaries) {
+ const auto& boundary = orig_boundaries[selected_boundary.first];
+ if (!boundary.only_null && !boundary.contains_null) {
+ projectable_boundaries.emplace_back(selected_boundary);
+ }
+ }
+ selected_boundaries.swap(projectable_boundaries);
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ if (slot_type_it == _slot_data_types.end()) {
+ return Status::InternalError(
+ "Runtime filter partition pruning target slot has no boundary type, filter_id={}, "
+ "slot_id={}, target_expr={}",
+ filter_id, leaf_slot_id, target_expr->expr_name());
+ }
+
+ // LIST partitions require regrouping projected values per partition and
+ // should not be marked as expression-prunable by FE yet.
+ for (const auto& selected_boundary : selected_boundaries) {
+ bool is_list = false;
+ std::visit([&](const auto& cvr) { is_list = cvr.is_fixed_value_range(); },
+ orig_boundaries[selected_boundary.first].boundary_cvr);
+ if (is_list) {
+ return Status::InternalError(
+ "Runtime filter partition pruning expression target does not support LIST "
Review Comment:
这个是单纯没实现还是什么情况
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4555425948
# BE Regression && UT Coverage Report
Increment line coverage `76.18% (662/869)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_36cd8599c1b5eba6dcfb83d6c88457494678e987_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_36cd8599c1b5eba6dcfb83d6c88457494678e987_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.73% (28080/38083) |
| Line Coverage | 57.67% (305229/529311) |
| Region Coverage | 54.50% (255612/469035) |
| Branch Coverage | 55.94% (110842/198130) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3115782627
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java:
##########
@@ -190,4 +199,66 @@ private org.apache.doris.planner.RuntimeFilter finalize(org.apache.doris.planner
origFilter.extractTargetsPosition();
return origFilter;
}
+
+ /**
+ * Classify whether a runtime-filter target expression can drive partition
+ * pruning on the given scan node, by checking that the target expression
+ * resolves (possibly through a chain of Nereids {@link Monotonic} functions)
+ * to a single partition column and tracking the cumulative direction.
+ *
+ * <p>This must run while the original Nereids {@code Expression} is still
+ * available; legacy planner only sees an {@code analysis.Expr} after
+ * translation and cannot redo the classification. The result is stored on
+ * the legacy {@code RuntimeFilter} and consumed at thrift-serialization time
+ * (and by {@code OlapScanNode.toThrift} to gate boundary emission).
+ *
+ * <p>Trust the {@code Monotonic} contract — implementers (DateTrunc, Year,
Review Comment:
This is a correctness issue. Several Nereids functions that implement Monotonic are only conditionally monotonic and rely on isMonotonic(lower, upper), for example from_unixtime, date_format, time_format, convert_tz, and yearweek. Because this classifier ignores that hook, a target like from_unixtime(part_col, %m) can still be marked monotonic and sent to BE. The BE then projects partition endpoints through the target expression and may false-prune partitions that cross a wrap boundary, for example a year-spanning partition whose month set is {12, 01, 02}. Please either consult the range-aware hook with the actual partition bounds before serializing monotonicity, or conservatively treat these shapes as NON_MONOTONIC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4288162491
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4287487790
# FE UT Coverage Report
Increment line coverage ` 25.95% (34/131)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_6bb4482b83b38672c7368e51b346c7bea9dc9247/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_6bb4482b83b38672c7368e51b346c7bea9dc9247/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4286988347
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4590021323
<details>
<summary>TPC-DS: <b>Total hot run time: 170798 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 586640a94f47548a444d49ee53d5474cf04e3ae0, data reload: false
query5 4324 637 481 481
query6 434 202 187 187
query7 4868 575 315 315
query8 366 221 208 208
query9 8789 4064 4059 4059
query10 460 325 268 268
query11 5904 2369 2197 2197
query12 158 100 99 99
query13 1263 596 420 420
query14 6444 5447 5064 5064
query14_1 4377 4353 4385 4353
query15 206 196 173 173
query16 1034 466 425 425
query17 1101 703 565 565
query18 2505 498 338 338
query19 198 184 142 142
query20 111 106 104 104
query21 224 137 119 119
query22 13662 13677 13575 13575
query23 17521 16481 16197 16197
query23_1 16323 16249 16353 16249
query24 7532 1790 1315 1315
query24_1 1307 1305 1339 1305
query25 600 468 408 408
query26 1302 333 174 174
query27 2656 540 359 359
query28 4473 2053 2091 2053
query29 1142 640 509 509
query30 323 242 207 207
query31 1141 1088 955 955
query32 119 65 62 62
query33 542 334 269 269
query34 1231 1135 657 657
query35 779 797 714 714
query36 1462 1433 1262 1262
query37 167 115 100 100
query38 3264 3172 3073 3073
query39 951 940 929 929
query39_1 882 890 901 890
query40 232 134 109 109
query41 71 69 69 69
query42 105 100 101 100
query43 342 334 289 289
query44
query45 211 190 192 190
query46 1140 1242 810 810
query47 2349 2338 2314 2314
query48 404 417 310 310
query49 659 492 375 375
query50 976 364 259 259
query51 4332 4329 4261 4261
query52 97 99 80 80
query53 258 283 195 195
query54 297 243 220 220
query55 83 82 76 76
query56 273 249 256 249
query57 1425 1429 1324 1324
query58 269 228 226 226
query59 1630 1719 1472 1472
query60 305 272 252 252
query61 178 223 156 156
query62 700 650 578 578
query63 222 186 192 186
query64 2565 791 622 622
query65
query66 1784 475 348 348
query67 30007 29918 29675 29675
query68
query69 454 313 273 273
query70 1012 978 956 956
query71 316 231 211 211
query72 2985 2703 2365 2365
query73 874 772 436 436
query74 5154 4940 4752 4752
query75 2738 2608 2245 2245
query76 2342 1223 823 823
query77 354 387 289 289
query78 12433 12443 11891 11891
query79 1474 1062 767 767
query80 596 492 395 395
query81 457 288 251 251
query82 567 157 125 125
query83 363 285 251 251
query84 265 142 113 113
query85 876 528 440 440
query86 371 313 270 270
query87 3394 3360 3188 3188
query88 3724 2741 2786 2741
query89 446 381 334 334
query90 1873 187 180 180
query91 179 170 142 142
query92 67 63 62 62
query93 1599 1568 889 889
query94 566 350 302 302
query95 714 377 344 344
query96 1010 780 352 352
query97 2688 2670 2557 2557
query98 214 210 216 210
query99 1185 1146 1024 1024
Total cold run time: 252861 ms
Total hot run time: 170798 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4428616760
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/2391ea7f407dcc42f8f4ddc3bcd8e0838059d15c_2391ea7f407dcc42f8f4ddc3bcd8e0838059d15c_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/2391ea7f407dcc42f8f4ddc3bcd8e0838059d15c_2391ea7f407dcc42f8f4ddc3bcd8e0838059d15c_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.05% (1849/2369) |
| Line Coverage | 64.72% (33221/51327) |
| Region Coverage | 65.24% (16440/25198) |
| Branch Coverage | 55.80% (8778/15732) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4440760469
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/ca30c183fbab4ffab1f11b3d91b0561736c7c199_ca30c183fbab4ffab1f11b3d91b0561736c7c199_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/ca30c183fbab4ffab1f11b3d91b0561736c7c199_ca30c183fbab4ffab1f11b3d91b0561736c7c199_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.03% (1850/2371) |
| Line Coverage | 64.68% (33223/51362) |
| Region Coverage | 65.23% (16446/25214) |
| Branch Coverage | 55.71% (8778/15758) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4540807328
<details>
<summary>TPC-H: <b>Total hot run time: 31741 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f779760a4db0a4e1f4605c63a8a9bc21f51af126, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17606 4087 4150 4087
q2 q3 10979 1409 876 876
q4 4817 481 351 351
q5 10399 2289 2173 2173
q6 375 180 138 138
q7 941 792 660 660
q8 9650 1872 1634 1634
q9 7055 5026 5207 5026
q10 6572 2203 1903 1903
q11 438 280 239 239
q12 671 427 302 302
q13 18204 3550 2842 2842
q14 270 260 244 244
q15 q16 827 779 711 711
q17 1002 888 972 888
q18 7298 5715 5568 5568
q19 1171 1210 1106 1106
q20 522 403 262 262
q21 5576 2635 2423 2423
q22 441 362 308 308
Total cold run time: 104814 ms
Total hot run time: 31741 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4430 4373 4322 4322
q2 q3 4589 4975 4362 4362
q4 2163 2251 1429 1429
q5 4565 5024 5087 5024
q6 238 185 141 141
q7 2026 1854 1607 1607
q8 2547 2329 2232 2232
q9 8132 8125 8316 8125
q10 4896 4840 4321 4321
q11 590 488 413 413
q12 764 788 569 569
q13 3404 3655 3059 3059
q14 315 295 275 275
q15 q16 738 753 688 688
q17 1414 1390 1376 1376
q18 8004 7261 6903 6903
q19 1160 1109 1090 1090
q20 2256 2264 1972 1972
q21 5427 4771 4657 4657
q22 535 462 426 426
Total cold run time: 58193 ms
Total hot run time: 52991 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4267225874
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4278836048
# BE UT Coverage Report
Increment line coverage `1.42% (5/353)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/bcfae3c60326b6ef46e5c8dfc68aaf68a3be4fdd_bcfae3c60326b6ef46e5c8dfc68aaf68a3be4fdd/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/bcfae3c60326b6ef46e5c8dfc68aaf68a3be4fdd_bcfae3c60326b6ef46e5c8dfc68aaf68a3be4fdd/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.19% (20328/38216) |
| Line Coverage | 36.74% (191335/520768) |
| Region Coverage | 33.02% (148832/450786) |
| Branch Coverage | 33.99% (65012/191278) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4279454767
# BE Regression && UT Coverage Report
Increment line coverage `81.59% (288/353)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_bcfae3c60326b6ef46e5c8dfc68aaf68a3be4fdd_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_bcfae3c60326b6ef46e5c8dfc68aaf68a3be4fdd_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.81% (27625/37429) |
| Line Coverage | 57.53% (298704/519189) |
| Region Coverage | 54.62% (248515/454969) |
| Branch Coverage | 56.06% (107555/191868) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3107948892
##########
be/src/exec/operator/olap_scan_operator.cpp:
##########
@@ -91,6 +95,9 @@ Status OlapScanLocalState::init(RuntimeState* state, LocalStateInfo& info) {
RETURN_IF_ERROR(Base::init(state, info));
RETURN_IF_ERROR(_sync_cloud_tablets(state));
Review Comment:
This new OLAP path bypasses the existing `enable_runtime_filter_partition_prune` session flag. `FileScanner` still gates runtime-filter partition pruning on `query_options().enable_runtime_filter_partition_prune`, but here we always parse partition boundaries and later run `_do_partition_pruning_by_rf()` unconditionally from `_process_conjuncts()` / `_on_runtime_filter_update()`. As a result, `SET enable_runtime_filter_partition_prune = false` no longer disables the optimization for OLAP scans, which is a user-visible behavior regression.
##########
fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java:
##########
@@ -1201,9 +1208,131 @@ protected void toThrift(TPlanNode msg) {
msg.olap_scan_node.setDistributeColumnIds(new ArrayList<>(distributionColumnIds));
+ // Populate partition boundaries for BE-side runtime filter partition pruning.
+ // Only serialize when this scan node actually has runtime filters, to avoid
+ // bloating thrift for tables with many partitions but no RF consumers.
+ if (!runtimeFilters.isEmpty()) {
+ setPartitionBoundaries(msg.olap_scan_node);
+ }
+
super.toThrift(msg);
}
+ private void setPartitionBoundaries(TOlapScanNode olapScanNode) {
+ PartitionInfo partitionInfo = olapTable.getPartitionInfo();
+ PartitionType partType = partitionInfo.getType();
+ if (partType != PartitionType.RANGE && partType != PartitionType.LIST) {
+ return;
+ }
+ List<Column> partColumns = partitionInfo.getPartitionColumns();
+ if (partColumns.isEmpty()) {
+ return;
+ }
+
+ // Build partition column name → slot ID mapping
+ Map<String, Integer> partColToSlotId = Maps.newHashMap();
+ for (SlotDescriptor slot : desc.getSlots()) {
+ if (slot.getColumn() == null) {
+ continue;
+ }
+ for (Column partCol : partColumns) {
+ if (slot.getColumn().getName().equalsIgnoreCase(partCol.getName())) {
+ partColToSlotId.put(partCol.getName(), slot.getId().asInt());
+ break;
+ }
+ }
+ }
+ if (partColToSlotId.isEmpty()) {
+ return;
+ }
+
+ List<TPartitionBoundary> boundaries = new ArrayList<>();
+ for (Long partitionId : selectedPartitionIds) {
+ PartitionItem item = partitionInfo.getItem(partitionId);
+ if (item == null) {
+ continue;
+ }
+ if (item instanceof RangePartitionItem) {
+ addRangeBoundaries(boundaries, partitionId, (RangePartitionItem) item,
+ partColumns, partColToSlotId);
+ } else if (item instanceof ListPartitionItem) {
+ addListBoundaries(boundaries, partitionId, (ListPartitionItem) item,
+ partColumns, partColToSlotId);
+ }
+ }
+ if (!boundaries.isEmpty()) {
+ olapScanNode.setPartitionBoundaries(boundaries);
+ }
+ }
+
+ private void addRangeBoundaries(List<TPartitionBoundary> boundaries, long partitionId,
+ RangePartitionItem rangeItem, List<Column> partColumns,
Review Comment:
This is not safe for multi-column RANGE partitions. Doris range partition membership is based on lexicographic `PartitionKey` comparison, not on an independent range over the first column. For example, a legal partition like `[(1, 1), (1, 5))` becomes a first-column boundary `[1, 1)`, so BE will see an empty interval and can prune the partition even when `k1 = 1` should still match rows inside it. Unless FE can serialize a provably correct projection, this optimization needs to be disabled for multi-column RANGE partitions.
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,306 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// Helper to extract a typed value from a ColumnPtr's first row.
+// Must be a template function so that `if constexpr` properly discards
+// the inapplicable branch during instantiation.
+template <PrimitiveType PT>
+static typename PrimitiveTypeTraits<PT>::CppType extract_value_from_column(const ColumnPtr& col) {
+ using CppType = typename PrimitiveTypeTraits<PT>::CppType;
+ auto data = col->get_data_at(0);
+ if constexpr (is_string_type(PT)) {
+ return CppType(data.data, data.size);
+ } else {
+ return *reinterpret_cast<const CppType*>(data.data);
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+void RuntimeFilterPartitionPruner::parse_boundaries(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ if (!tb.__isset.partition_id || !tb.__isset.slot_id) {
+ continue;
+ }
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ if (slot_it == slot_descs.end()) {
+ continue;
+ }
+ SlotDescriptor* slot = slot_it->second;
+ PrimitiveType ptype = slot->type()->get_primitive_type();
+ int precision = cast_set<int>(slot->type()->get_precision());
+ int scale = cast_set<int>(slot->type()->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ if (!is_list && !is_range) break; \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::pair<CppType, ColumnPtr> { \
Review Comment:
FE now emits `NULL_LITERAL` entries for nullable LIST partitions (`OlapScanNode.addListBoundaries()`), but this parser assumes every boundary literal can be converted into a concrete `CppType`. For a `NULL_LITERAL`, `VLiteral` builds a `ColumnNullable`; `get_data_at(0)` returns `{nullptr, 0}`, so the non-string branch dereferences a null pointer and the string branch collapses NULL to `""`. That makes nullable LIST partitions unsafe here. We need to propagate NULL through `ColumnValueRange::contain_null()` (or skip NULL boundaries for now) instead of treating it as an ordinary fixed value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4286497520
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4553767121
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4556146615
# FE UT Coverage Report
Increment line coverage ` 31.37% (96/306)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_36cd8599c1b5eba6dcfb83d6c88457494678e987/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_36cd8599c1b5eba6dcfb83d6c88457494678e987/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4494578659
<details>
<summary>TPC-DS: <b>Total hot run time: 169714 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ef2529ca3c35ebe1f89cb9fa4212f239fce43a97, data reload: false
query5 4330 661 512 512
query6 331 232 212 212
query7 4394 555 323 323
query8 344 239 225 225
query9 8844 4045 4084 4045
query10 455 347 319 319
query11 5812 2380 2265 2265
query12 188 133 130 130
query13 1344 594 434 434
query14 6071 5427 5107 5107
query14_1 4403 4409 4390 4390
query15 210 201 183 183
query16 1027 476 463 463
query17 1153 739 621 621
query18 2746 500 377 377
query19 226 214 174 174
query20 142 135 135 135
query21 216 141 124 124
query22 13699 13572 13413 13413
query23 17149 16341 16070 16070
query23_1 16167 16189 16210 16189
query24 7775 1789 1306 1306
query24_1 1318 1315 1313 1313
query25 596 500 454 454
query26 1315 336 179 179
query27 2682 539 361 361
query28 4340 2006 1989 1989
query29 1032 651 523 523
query30 316 255 208 208
query31 1146 1066 951 951
query32 91 80 78 78
query33 570 362 303 303
query34 1192 1123 634 634
query35 738 770 673 673
query36 1309 1321 1157 1157
query37 150 103 89 89
query38 3219 3120 3051 3051
query39 946 925 896 896
query39_1 883 877 862 862
query40 224 151 125 125
query41 66 65 63 63
query42 117 115 110 110
query43 315 328 294 294
query44
query45 212 207 194 194
query46 1089 1168 732 732
query47 2271 2337 2231 2231
query48 396 407 296 296
query49 633 498 382 382
query50 1003 360 251 251
query51 4387 4310 4215 4215
query52 107 105 97 97
query53 267 282 208 208
query54 319 274 255 255
query55 94 92 84 84
query56 307 306 317 306
query57 1398 1432 1300 1300
query58 295 270 276 270
query59 1536 1613 1400 1400
query60 314 330 307 307
query61 163 156 159 156
query62 687 623 567 567
query63 252 206 201 201
query64 2389 801 645 645
query65
query66 1708 485 356 356
query67 29372 29979 29867 29867
query68
query69 466 330 309 309
query70 1034 1013 1001 1001
query71 309 270 258 258
query72 2938 2727 2382 2382
query73 858 763 447 447
query74 5059 4904 4729 4729
query75 2669 2638 2272 2272
query76 2286 1130 764 764
query77 396 397 333 333
query78 12159 12278 11674 11674
query79 1408 1154 728 728
query80 641 566 475 475
query81 457 275 241 241
query82 1382 161 124 124
query83 357 267 255 255
query84 258 144 114 114
query85 891 552 454 454
query86 389 346 331 331
query87 3382 3437 3196 3196
query88 3526 2658 2690 2658
query89 432 398 340 340
query90 1941 179 186 179
query91 182 172 137 137
query92 80 77 76 76
query93 1437 1419 977 977
query94 553 332 317 317
query95 700 390 430 390
query96 1033 763 320 320
query97 2718 2717 2589 2589
query98 234 235 236 235
query99 1098 1100 974 974
Total cold run time: 252808 ms
Total hot run time: 169714 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4428348636
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4564687984
# FE Regression Coverage Report
Increment line coverage ` 64.50% (218/338)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_2ede6c8cafb5ff16b7121d7edac1018fc2e592e0_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_2ede6c8cafb5ff16b7121d7edac1018fc2e592e0_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4563546636
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/2ede6c8cafb5ff16b7121d7edac1018fc2e592e0_2ede6c8cafb5ff16b7121d7edac1018fc2e592e0_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/2ede6c8cafb5ff16b7121d7edac1018fc2e592e0_2ede6c8cafb5ff16b7121d7edac1018fc2e592e0_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1854/2375) |
| Line Coverage | 64.53% (33337/51665) |
| Region Coverage | 65.23% (16530/25343) |
| Branch Coverage | 55.75% (8838/15854) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4563736546
<details>
<summary>TPC-DS: <b>Total hot run time: 169925 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2ede6c8cafb5ff16b7121d7edac1018fc2e592e0, data reload: false
query5 4340 650 478 478
query6 443 219 205 205
query7 4833 592 305 305
query8 377 229 219 219
query9 8821 4076 4068 4068
query10 483 321 256 256
query11 5936 2597 2250 2250
query12 165 106 100 100
query13 1275 626 440 440
query14 6737 5445 5107 5107
query14_1 4433 4431 4438 4431
query15 210 197 193 193
query16 1034 482 451 451
query17 1206 729 608 608
query18 2729 490 365 365
query19 243 196 150 150
query20 112 107 110 107
query21 220 144 122 122
query22 13571 13566 13399 13399
query23 17405 16505 16161 16161
query23_1 16349 16249 16271 16249
query24 7450 1792 1340 1340
query24_1 1357 1328 1338 1328
query25 570 472 422 422
query26 1290 313 175 175
query27 2594 582 347 347
query28 4400 2008 2014 2008
query29 1108 637 503 503
query30 308 239 197 197
query31 1139 1084 968 968
query32 112 65 62 62
query33 525 370 251 251
query34 1189 1136 658 658
query35 754 774 687 687
query36 1392 1388 1237 1237
query37 159 106 93 93
query38 3212 3120 3081 3081
query39 936 935 894 894
query39_1 867 870 855 855
query40 217 122 100 100
query41 71 62 61 61
query42 95 95 92 92
query43 321 321 278 278
query44
query45 200 188 182 182
query46 1101 1212 767 767
query47 2343 2362 2237 2237
query48 389 438 325 325
query49 662 467 356 356
query50 973 364 272 272
query51 4314 4241 4260 4241
query52 88 86 77 77
query53 250 272 188 188
query54 266 224 214 214
query55 78 77 68 68
query56 257 245 210 210
query57 1445 1380 1332 1332
query58 242 219 215 215
query59 1647 1715 1436 1436
query60 303 247 237 237
query61 169 152 160 152
query62 689 648 588 588
query63 225 186 188 186
query64 2487 784 641 641
query65
query66 1744 471 371 371
query67 29720 29729 29571 29571
query68
query69 434 309 280 280
query70 960 940 966 940
query71 294 221 207 207
query72 3031 2752 2474 2474
query73 837 775 438 438
query74 5167 4971 4792 4792
query75 2680 2587 2242 2242
query76 2262 1145 791 791
query77 365 375 290 290
query78 12355 12323 11992 11992
query79 1475 1093 773 773
query80 1260 495 387 387
query81 499 284 239 239
query82 568 160 120 120
query83 369 291 251 251
query84 275 146 114 114
query85 956 549 451 451
query86 415 303 283 283
query87 3435 3359 3219 3219
query88 3619 2712 2702 2702
query89 447 376 325 325
query90 1845 182 179 179
query91 182 174 144 144
query92 69 66 59 59
query93 1533 1450 899 899
query94 693 356 310 310
query95 703 462 370 370
query96 1011 791 338 338
query97 2729 2694 2591 2591
query98 211 219 211 211
query99 1194 1171 1064 1064
Total cold run time: 252953 ms
Total hot run time: 169925 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4570966993
# FE UT Coverage Report
Increment line coverage ` 31.37% (96/306)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_32e14d6220221b6aa1e1cab5adf12ffaae3e9f5d/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_32e14d6220221b6aa1e1cab5adf12ffaae3e9f5d/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4571779736
<details>
<summary>TPC-DS: <b>Total hot run time: 171574 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fb1122e0a26c0dc4fc1defe6bfb9b6b5da020008, data reload: false
query5 4338 632 476 476
query6 448 200 190 190
query7 4866 583 316 316
query8 369 222 207 207
query9 8765 4027 4032 4027
query10 460 327 265 265
query11 5950 2585 2228 2228
query12 165 107 106 106
query13 1287 610 456 456
query14 6550 5434 5121 5121
query14_1 4438 4471 4351 4351
query15 206 197 177 177
query16 1016 454 435 435
query17 1101 690 554 554
query18 2626 476 340 340
query19 213 186 146 146
query20 114 109 107 107
query21 222 139 120 120
query22 13658 13608 13381 13381
query23 17386 16489 16339 16339
query23_1 16464 16334 16385 16334
query24 7717 1770 1319 1319
query24_1 1309 1339 1328 1328
query25 567 449 383 383
query26 1306 303 168 168
query27 2674 561 336 336
query28 4448 1999 2008 1999
query29 1088 605 500 500
query30 309 244 203 203
query31 1144 1080 955 955
query32 110 63 60 60
query33 535 344 265 265
query34 1192 1133 672 672
query35 764 797 708 708
query36 1415 1413 1223 1223
query37 156 107 91 91
query38 3228 3137 3110 3110
query39 933 916 914 914
query39_1 877 888 897 888
query40 234 127 110 110
query41 84 70 69 69
query42 100 96 96 96
query43 331 325 279 279
query44
query45 201 197 184 184
query46 1097 1178 752 752
query47 2427 2425 2305 2305
query48 402 441 312 312
query49 661 480 370 370
query50 971 360 275 275
query51 4392 4316 4271 4271
query52 92 92 80 80
query53 262 272 197 197
query54 288 234 219 219
query55 83 78 75 75
query56 252 240 246 240
query57 1459 1416 1340 1340
query58 251 232 228 228
query59 1605 1722 1479 1479
query60 300 265 255 255
query61 190 182 184 182
query62 699 654 585 585
query63 238 189 192 189
query64 2610 836 608 608
query65
query66 1752 469 345 345
query67 29810 29625 29656 29625
query68
query69 435 314 273 273
query70 956 973 965 965
query71 310 222 217 217
query72 3013 2714 2644 2644
query73 883 803 440 440
query74 5147 4962 4801 4801
query75 2704 2559 2231 2231
query76 2310 1149 788 788
query77 362 383 289 289
query78 12443 12602 11954 11954
query79 1506 1070 751 751
query80 1300 486 424 424
query81 524 282 239 239
query82 663 160 123 123
query83 331 288 251 251
query84 266 150 112 112
query85 944 530 453 453
query86 450 304 295 295
query87 3371 3357 3166 3166
query88 3708 2755 2804 2755
query89 432 381 333 333
query90 1920 183 187 183
query91 182 168 142 142
query92 67 63 61 61
query93 1637 1372 889 889
query94 705 355 314 314
query95 689 480 347 347
query96 1118 799 338 338
query97 2713 2700 2630 2630
query98 220 213 212 212
query99 1159 1157 1024 1024
Total cold run time: 254021 ms
Total hot run time: 171574 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4507080080
<details>
<summary>TPC-H: <b>Total hot run time: 31614 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6be29ffaf962c85e6c243c68a1c26cf413cdb0b5, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17631 3976 3952 3952
q2 q3 10748 1394 805 805
q4 4685 472 354 354
q5 7574 2317 2143 2143
q6 238 180 142 142
q7 961 778 649 649
q8 9369 1846 1672 1672
q9 5198 4956 4911 4911
q10 6404 2110 1809 1809
q11 446 281 244 244
q12 642 431 312 312
q13 18129 3430 2769 2769
q14 266 254 240 240
q15 q16 811 767 709 709
q17 935 853 957 853
q18 7129 5665 5616 5616
q19 1180 1234 1131 1131
q20 543 597 305 305
q21 6078 2786 2679 2679
q22 482 379 319 319
Total cold run time: 99449 ms
Total hot run time: 31614 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4730 4714 4751 4714
q2 q3 4882 5255 4639 4639
q4 2188 2248 1446 1446
q5 4859 4797 4755 4755
q6 226 184 137 137
q7 1859 1759 1577 1577
q8 2495 2134 2111 2111
q9 7250 7305 7269 7269
q10 4517 4415 4013 4013
q11 559 393 374 374
q12 718 735 524 524
q13 3041 3459 2846 2846
q14 271 284 256 256
q15 q16 678 704 624 624
q17 1312 1284 1286 1284
q18 7392 6973 6835 6835
q19 1136 1107 1105 1105
q20 2225 2223 1945 1945
q21 5500 4751 4722 4722
q22 522 484 434 434
Total cold run time: 56360 ms
Total hot run time: 51610 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4505245547
# BE Regression && UT Coverage Report
Increment line coverage `75.28% (603/801)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.50% (27894/37949) |
| Line Coverage | 57.49% (302778/526635) |
| Region Coverage | 54.40% (253555/466103) |
| Branch Coverage | 55.95% (109952/196522) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3220461861
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java:
##########
@@ -322,6 +336,18 @@ public void createLegacyRuntimeFilter(RuntimeFilter filter, PlanNode node, PlanT
Expr targetExpr = targetExprList.get(i);
origFilter.addTarget(new RuntimeFilterTarget(
scanNode, targetExpr, true, isLocalTarget));
+ RuntimeFilterPartitionPruneClassifier.Classification classification =
+ RuntimeFilterPartitionPruneClassifier.classify(
+ targetExpr, filter.getTargetExpressions().get(i), scanNode);
Review Comment:
The classifier is proving monotonicity from `filter.getTargetExpressions().get(i)`, but the BE will project partition boundaries through the final legacy `targetExpr` stored in the RF, including any `CastExpr` injected by the type-adjustment block above. That means a target expression that is locally monotonic before type adjustment can still be marked safe even though the actual expression evaluated in `ParsedPartitionBoundaries::get_or_compute_projection()` has one more cast on top that was not checked against the partition range.
Please either classify an equivalent Nereids expression that includes the inserted cast, or conservatively skip RF partition pruning whenever the adjustment at line 307 wraps a non-identity target. The current tests cover the bare-slot implicit-cast case, but not a locally monotonic non-slot target with an added outer cast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4421051778
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4421055083
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4422660576
<details>
<summary>TPC-H: <b>Total hot run time: 29730 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b29520eb8c2fd176695ee6c263c89bab10446d4a, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17704 4060 3972 3972
q2 q3 10788 893 615 615
q4 4708 460 357 357
q5 8237 1345 1148 1148
q6 347 174 140 140
q7 954 975 760 760
q8 11064 1456 1303 1303
q9 7476 5451 5307 5307
q10 6314 2088 1802 1802
q11 469 274 254 254
q12 684 417 290 290
q13 18192 3343 2765 2765
q14 293 282 262 262
q15 q16 896 868 782 782
q17 1013 1084 780 780
q18 6544 5684 5602 5602
q19 1205 1253 1123 1123
q20 525 395 274 274
q21 4830 2296 1881 1881
q22 429 355 313 313
Total cold run time: 102672 ms
Total hot run time: 29730 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4323 4210 4256 4210
q2 q3 4651 4786 4182 4182
q4 2156 2198 1413 1413
q5 4989 4969 5609 4969
q6 219 178 148 148
q7 2059 1835 1657 1657
q8 3444 3164 3170 3164
q9 8339 8540 8450 8450
q10 4501 4538 4254 4254
q11 661 422 412 412
q12 694 755 529 529
q13 3240 3600 2901 2901
q14 304 307 298 298
q15 q16 776 792 693 693
q17 1418 1369 1527 1369
q18 8050 7276 7155 7155
q19 1192 1180 1151 1151
q20 2282 2261 1996 1996
q21 6273 5636 5043 5043
q22 577 513 468 468
Total cold run time: 60148 ms
Total hot run time: 54462 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4268937326
# FE Regression Coverage Report
Increment line coverage ` 77.32% (75/97)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_141c78da888bc5cd8198ffebb607430c90015f0d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_141c78da888bc5cd8198ffebb607430c90015f0d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4267253429
OpenCode automated review failed and did not complete.
Error: Review step was skipped (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/24560451334
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4269319798
OpenCode automated review failed and did not complete.
Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/24567893050
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4278766869
# FE UT Coverage Report
Increment line coverage ` 22.33% (23/103)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_bcfae3c60326b6ef46e5c8dfc68aaf68a3be4fdd/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_bcfae3c60326b6ef46e5c8dfc68aaf68a3be4fdd/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4278332872
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3116865659
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,736 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vruntimefilter_wrapper.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+void ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ if (!tb.__isset.partition_id || !tb.__isset.slot_id) {
+ continue;
+ }
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ if (slot_it == slot_descs.end()) {
+ continue;
+ }
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ if (!is_list && !is_range) break; \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ if (parsed_ok) {
+ _partition_column_slot_ids.insert(slot_id);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_partition_column_slot_ids.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ if (!expr) {
+ return nullptr;
+ }
+ if (expr->is_slot_ref()) {
+ return assert_cast<const VSlotRef*>(expr);
+ }
+ const VSlotRef* found = nullptr;
+ for (const auto& child : expr->children()) {
+ const VSlotRef* c = find_unique_slot_ref(child.get());
+ if (c) {
+ if (found) {
+ return nullptr; // multiple slot refs, can't handle
+ }
+ found = c;
+ }
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+const std::vector<ParsedBoundary>* ParsedPartitionBoundaries::get_or_compute_projection(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ TTargetExprMonotonicity::type direction, VExprContext* ctx) const {
+ {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ auto it = _projection_cache.find(filter_id);
+ if (it != _projection_cache.end()) {
+ return &it->second;
+ }
+ }
+
+ // Build projection
+ std::vector<ParsedBoundary> projected;
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ if (slot_type_it == _slot_data_types.end()) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+
+ // LIST partitions: TODO(rf-partition-prune) -- projecting through a
+ // monotonic function preserves set membership but requires re-grouping
+ // the projected values back per-partition. Skip for now (caller treats
+ // empty cache as "no projection available").
+ {
+ bool any_list = false;
+ std::visit([&](const auto& cvr) { any_list = cvr.is_fixed_value_range(); },
+ orig_boundaries[0].boundary_cvr);
+ if (any_list) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+ }
+
+ const DataTypePtr& input_type = slot_type_it->second;
+ bool input_is_nullable = input_type->is_nullable();
+ DataTypePtr inner_type = input_is_nullable ? remove_nullable(input_type) : input_type;
+ PrimitiveType input_ptype = input_type->get_primitive_type();
+
+ size_t N = orig_boundaries.size();
+
+ // Track open bounds (low_value == TYPE_MIN or high_value == TYPE_MAX)
+ // externally so we can propagate null_in_projection regardless of whether
+ // the input column is nullable.
+ std::vector<bool> ext_null_lo(N, false);
+ std::vector<bool> ext_null_hi(N, false);
+
+ // Build input blocks for lo and hi
+ Block lo_block;
+ Block hi_block;
+
+ // Pad columns 0..leaf_column_id-1 with placeholder columns so the leaf
+ // VSlotRef (whose column_id is leaf_column_id) reads our typed column.
+ for (int col_idx = 0; col_idx < leaf_column_id; ++col_idx) {
+ auto add_dummy = [&](Block& blk) {
+ auto col = ColumnUInt8::create(N, 0);
+ blk.insert({std::move(col), std::make_shared<DataTypeUInt8>(),
+ fmt::format("dummy_{}", col_idx)});
+ };
+ add_dummy(lo_block);
+ add_dummy(hi_block);
+ }
+
+ // Macro-dispatch on input PrimitiveType to build the typed value column
+ // for each side. Inserts NULL (when input column is nullable) or default
+ // (when not) for boundaries that are only_null / open / type-mismatched;
+ // ext_null_lo/hi tracks these so we can mark them null_in_projection
+ // after executing the projection.
+ bool input_built = false;
+#define BUILD_INPUT_COLUMNS(INPUT_PT) \
+ case TYPE_##INPUT_PT: { \
+ using InCol = typename PrimitiveTypeTraits<TYPE_##INPUT_PT>::ColumnType; \
+ MutableColumnPtr lo_inner_base = inner_type->create_column(); \
+ MutableColumnPtr hi_inner_base = inner_type->create_column(); \
+ auto* lo_inner = assert_cast<InCol*>(lo_inner_base.get()); \
+ auto* hi_inner = assert_cast<InCol*>(hi_inner_base.get()); \
+ lo_inner->reserve(N); \
+ hi_inner->reserve(N); \
+ auto lo_nulls = ColumnUInt8::create(); \
+ auto hi_nulls = ColumnUInt8::create(); \
+ lo_nulls->reserve(N); \
+ hi_nulls->reserve(N); \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& boundary = orig_boundaries[i]; \
+ bool null_lo = boundary.only_null; \
+ bool null_hi = boundary.only_null; \
+ const auto* cvr = \
+ std::get_if<ColumnValueRange<TYPE_##INPUT_PT>>(&boundary.boundary_cvr); \
+ if (cvr == nullptr) { \
+ null_lo = null_hi = true; \
+ } else { \
+ if (cvr->is_low_value_minimum()) null_lo = true; \
+ if (cvr->is_high_value_maximum()) null_hi = true; \
+ } \
+ ext_null_lo[i] = null_lo; \
+ ext_null_hi[i] = null_hi; \
+ if (null_lo) { \
+ lo_inner->insert_default(); \
+ lo_nulls->get_data().push_back(1); \
+ } else { \
+ lo_inner->insert_value(cvr->get_range_min_value()); \
+ lo_nulls->get_data().push_back(0); \
+ } \
+ if (null_hi) { \
+ hi_inner->insert_default(); \
+ hi_nulls->get_data().push_back(1); \
+ } else { \
+ hi_inner->insert_value(cvr->get_range_max_value()); \
+ hi_nulls->get_data().push_back(0); \
+ } \
+ } \
+ if (input_is_nullable) { \
+ auto lo_col = ColumnNullable::create(std::move(lo_inner_base), std::move(lo_nulls)); \
+ auto hi_col = ColumnNullable::create(std::move(hi_inner_base), std::move(hi_nulls)); \
+ lo_block.insert({std::move(lo_col), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_col), input_type, "leaf_slot"}); \
+ } else { \
+ lo_block.insert({std::move(lo_inner_base), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_inner_base), input_type, "leaf_slot"}); \
+ } \
+ input_built = true; \
+ break; \
+ }
+
+ switch (input_ptype) {
+ BUILD_INPUT_COLUMNS(TINYINT)
+ BUILD_INPUT_COLUMNS(SMALLINT)
+ BUILD_INPUT_COLUMNS(INT)
+ BUILD_INPUT_COLUMNS(BIGINT)
+ BUILD_INPUT_COLUMNS(LARGEINT)
+ BUILD_INPUT_COLUMNS(FLOAT)
+ BUILD_INPUT_COLUMNS(DOUBLE)
+ BUILD_INPUT_COLUMNS(CHAR)
+ BUILD_INPUT_COLUMNS(DATE)
+ BUILD_INPUT_COLUMNS(DATETIME)
+ BUILD_INPUT_COLUMNS(DATEV2)
+ BUILD_INPUT_COLUMNS(DATETIMEV2)
+ BUILD_INPUT_COLUMNS(TIMESTAMPTZ)
+ BUILD_INPUT_COLUMNS(VARCHAR)
+ BUILD_INPUT_COLUMNS(STRING)
+ BUILD_INPUT_COLUMNS(DECIMAL32)
+ BUILD_INPUT_COLUMNS(DECIMAL64)
+ BUILD_INPUT_COLUMNS(DECIMAL128I)
+ BUILD_INPUT_COLUMNS(DECIMAL256)
+ BUILD_INPUT_COLUMNS(DECIMALV2)
+ BUILD_INPUT_COLUMNS(BOOLEAN)
+ BUILD_INPUT_COLUMNS(IPV4)
+ BUILD_INPUT_COLUMNS(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_INPUT_COLUMNS
+
+ if (!input_built) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+
+ int lo_result_id = -1;
+ int hi_result_id = -1;
+ auto status_lo = target_expr->execute(ctx, &lo_block, &lo_result_id);
+ auto status_hi = target_expr->execute(ctx, &hi_block, &hi_result_id);
+ if (!status_lo.ok() || !status_hi.ok() || lo_result_id < 0 || hi_result_id < 0) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+
+ const auto& lo_result_col = lo_block.get_by_position(lo_result_id).column;
+ const auto& hi_result_col = hi_block.get_by_position(hi_result_id).column;
+
+ int out_precision = cast_set<int>(target_expr->data_type()->get_precision());
+ int out_scale = cast_set<int>(target_expr->data_type()->get_scale());
+ bool out_nullable = target_expr->data_type()->is_nullable();
+ PrimitiveType output_ptype = target_expr->data_type()->get_primitive_type();
+
+ projected.resize(N);
+
+#define BUILD_PROJECTED_CVR(OUTPUT_PT) \
+ case TYPE_##OUTPUT_PT: { \
+ using OutputCppType = typename PrimitiveTypeTraits<TYPE_##OUTPUT_PT>::CppType; \
+ for (size_t i = 0; i < N; ++i) { \
+ projected[i].partition_id = orig_boundaries[i].partition_id; \
+ projected[i].slot_id = leaf_slot_id; \
+ projected[i].is_nullable = out_nullable; \
+ projected[i].only_null = orig_boundaries[i].only_null; \
+ projected[i].contains_null = orig_boundaries[i].contains_null; \
+ bool lo_is_null = ext_null_lo[i] || lo_result_col->is_null_at(i); \
+ bool hi_is_null = ext_null_hi[i] || hi_result_col->is_null_at(i); \
+ if (lo_is_null || hi_is_null) { \
+ projected[i].null_in_projection = true; \
+ ColumnValueRange<TYPE_##OUTPUT_PT> full_range("", out_nullable, out_precision, \
+ out_scale); \
+ projected[i].boundary_cvr = std::move(full_range); \
+ } else { \
+ Field lo_field = (*lo_result_col)[i]; \
+ Field hi_field = (*hi_result_col)[i]; \
+ OutputCppType lo_projected = lo_field.get<TYPE_##OUTPUT_PT>(); \
+ OutputCppType hi_projected = hi_field.get<TYPE_##OUTPUT_PT>(); \
+ ColumnValueRange<TYPE_##OUTPUT_PT> cvr("", out_nullable, out_precision, \
+ out_scale); \
+ if (direction == TTargetExprMonotonicity::MONOTONIC_DECREASING) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, hi_projected)); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, lo_projected)); \
+ } else { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, lo_projected)); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, hi_projected)); \
+ } \
+ projected[i].boundary_cvr = std::move(cvr); \
+ } \
+ } \
+ break; \
+ }
+
+ switch (output_ptype) {
+ BUILD_PROJECTED_CVR(TINYINT)
+ BUILD_PROJECTED_CVR(SMALLINT)
+ BUILD_PROJECTED_CVR(INT)
+ BUILD_PROJECTED_CVR(BIGINT)
+ BUILD_PROJECTED_CVR(LARGEINT)
+ BUILD_PROJECTED_CVR(FLOAT)
+ BUILD_PROJECTED_CVR(DOUBLE)
+ BUILD_PROJECTED_CVR(CHAR)
+ BUILD_PROJECTED_CVR(DATE)
+ BUILD_PROJECTED_CVR(DATETIME)
+ BUILD_PROJECTED_CVR(DATEV2)
+ BUILD_PROJECTED_CVR(DATETIMEV2)
+ BUILD_PROJECTED_CVR(TIMESTAMPTZ)
+ BUILD_PROJECTED_CVR(VARCHAR)
+ BUILD_PROJECTED_CVR(STRING)
+ BUILD_PROJECTED_CVR(DECIMAL32)
+ BUILD_PROJECTED_CVR(DECIMAL64)
+ BUILD_PROJECTED_CVR(DECIMAL128I)
+ BUILD_PROJECTED_CVR(DECIMAL256)
+ BUILD_PROJECTED_CVR(DECIMALV2)
+ BUILD_PROJECTED_CVR(BOOLEAN)
+ BUILD_PROJECTED_CVR(IPV4)
+ BUILD_PROJECTED_CVR(IPV6)
+ default:
+ break;
+ }
+
+#undef BUILD_PROJECTED_CVR
+
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static SQLFilterOp convert_opcode_to_filter_op(TExprOpcode::type op) {
+ switch (op) {
+ case TExprOpcode::LE:
+ return FILTER_LESS_OR_EQUAL;
+ case TExprOpcode::LT:
+ return FILTER_LESS;
+ case TExprOpcode::GE:
+ return FILTER_LARGER_OR_EQUAL;
+ case TExprOpcode::GT:
+ return FILTER_LARGER;
+ default:
+ return FILTER_IN; // sentinel: caller should skip
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+void RuntimeFilterPartitionPruner::_try_prune_by_single_rf(
+ const std::vector<ParsedBoundary>& boundaries, const VExprSPtr& impl,
+ phmap::flat_hash_set<int64_t>& newly_pruned) {
+ // Pre-compute whether the RF "matches NULL" -- i.e. whether the RF would
+ // accept a row whose probe value is NULL. The signal differs by RF impl:
+ // * IN filter → HybridSet::contain_null()
+ // * Bloom filter → BloomFilterFuncBase::contain_null()
+ // * MinMax filter → encoded via the BinaryPredicate node_type:
+ // NULL_AWARE_BINARY_PRED means the build side had a
+ // NULL key and a null-safe equal join asked NULL to
+ // match NULL; plain BINARY_PRED never matches NULL.
+ // FilterBase::contain_null() already folds in `_null_aware`, so we only
+ // get a true result when the build side is actually null-aware AND
+ // produced a NULL value.
+ bool rf_contains_null = false;
+ if (auto hybrid_set = impl->get_set_func()) {
+ rf_contains_null = hybrid_set->contain_null();
+ } else if (impl->node_type() == TExprNodeType::BLOOM_PRED) {
+ auto bloom = impl->get_bloom_filter_func();
+ rf_contains_null = bloom && bloom->contain_null();
+ } else if (impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) {
+ // Min/Max RF built on a null-safe equal join. The literal child holds
+ // the min or max bound; the NULL semantic is conveyed by the node
+ // type itself (see create_vbin_predicate in runtime_filter/utils.cpp).
+ rf_contains_null = true;
+ }
+
+ for (const auto& pb : boundaries) {
+ if (_pruned_partition_ids.contains(pb.partition_id) ||
+ newly_pruned.contains(pb.partition_id)) {
+ continue;
+ }
+
+ // Skip partitions with NULL projection conservatively
+ if (pb.null_in_projection) {
+ continue;
+ }
+
+ // NULL handling:
+ // A partition row whose key is NULL matches the RF iff `rf_contains_null`.
+ // - only_null partition (rows are exclusively NULL): prunable iff !rf_contains_null.
+ // - mixed (NULL + concrete values): if rf_contains_null, NULL rows alone
+ // prevent pruning. Otherwise NULL rows can never match, so we ignore
+ // the NULL portion and let the regular non-NULL intersection decide.
+ if (pb.only_null) {
+ if (!rf_contains_null) {
+ newly_pruned.insert(pb.partition_id);
+ }
+ continue;
+ }
+ if (pb.contains_null && rf_contains_null) {
+ continue;
+ }
+
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+ using CppType = typename CvrType::CppType;
+
+ auto hybrid_set = impl->get_set_func();
+ if (hybrid_set) {
+ // IN filter: build a fixed-value CVR from the HybridSet
+ auto rf_cvr = CvrType::create_empty_column_value_range(
+ pb.is_nullable, boundary_cvr.precision(), boundary_cvr.scale());
+ auto* iter = hybrid_set->begin();
+ while (iter->has_next()) {
+ const void* value = iter->get_value();
+ if (value) {
+ if constexpr (std::is_same_v<CppType, StringRef>) {
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else {
+ static_cast<void>(rf_cvr.add_fixed_value(
+ *reinterpret_cast<const CppType*>(value)));
+ }
+ }
+ iter->next();
+ }
+ auto boundary_copy = boundary_cvr;
+ boundary_copy.intersection(rf_cvr);
+ if (boundary_copy.is_empty_value_range()) {
+ newly_pruned.insert(pb.partition_id);
+ }
+ } else if ((impl->node_type() == TExprNodeType::BINARY_PRED ||
+ impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) &&
+ impl->children().size() == 2 && impl->children()[1]->is_literal()) {
+ // MinMax filter: binary pred with literal bound
+ auto* literal = assert_cast<VLiteral*>(impl->children()[1].get());
+ auto col_ptr = literal->get_column_ptr();
+ auto data = col_ptr->get_data_at(0);
+ CppType val {};
+ if constexpr (std::is_same_v<CppType, StringRef>) {
+ val = CppType(data.data, data.size);
+ } else {
+ val = *reinterpret_cast<const CppType*>(data.data);
Review Comment:
`PrimitiveTypeTraits<TYPE_CHAR/VARCHAR/STRING>::CppType` is `std::string`, not `StringRef` (`be/src/core/data_type/primitive_type.h`). In this MIN/MAX branch the literal comes from `create_literal()`, which builds a `ColumnConst(ColumnString)` (`be/src/exec/runtime_filter/runtime_filter_consumer.cpp`, `be/src/exprs/vliteral.cpp`, `be/src/core/data_type/data_type.cpp`), so `get_data_at(0)` here returns a `StringRef` pointing at the character bytes. Reinterpreting `data.data` as `const CppType*` on the `else` branch is therefore invalid for `CHAR`/`VARCHAR`/`STRING` and can read garbage or crash.
This path is reachable for string join keys because `RuntimeFilterConsumer` builds `BINARY_PRED` / `NULL_AWARE_BINARY_PRED` literals for `MIN_MAX` filters on those types. Please construct string-like values from the returned `StringRef` (for example `CppType(data.data, data.size)`) instead of dereferencing `data.data` as `CppType*`, and add a string `MIN_MAX` partition-pruning regression case so this stays covered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4433010755
# BE UT Coverage Report
Increment line coverage `1.15% (8/698)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/e7a084166584e3ccdc5ae8b6bd5fc50d3c0cb832_e7a084166584e3ccdc5ae8b6bd5fc50d3c0cb832/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/e7a084166584e3ccdc5ae8b6bd5fc50d3c0cb832_e7a084166584e3ccdc5ae8b6bd5fc50d3c0cb832/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.53% (20636/38548) |
| Line Coverage | 37.14% (194951/524884) |
| Region Coverage | 33.43% (152450/456017) |
| Branch Coverage | 34.24% (66453/194070) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4449420993
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/572ab7793a850dc7d5ad35a35834f642fa194f82_572ab7793a850dc7d5ad35a35834f642fa194f82_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/572ab7793a850dc7d5ad35a35834f642fa194f82_572ab7793a850dc7d5ad35a35834f642fa194f82_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.03% (1850/2371) |
| Line Coverage | 64.70% (33231/51362) |
| Region Coverage | 65.27% (16457/25214) |
| Branch Coverage | 55.76% (8786/15758) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4448527492
<details>
<summary>TPC-DS: <b>Total hot run time: 169889 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5bfeb4d44af5550f9edd89c84a3d05b18135d530, data reload: false
query5 4309 624 471 471
query6 440 199 179 179
query7 4829 564 289 289
query8 374 225 201 201
query9 8824 4109 4084 4084
query10 466 327 267 267
query11 5918 2393 2281 2281
query12 165 105 99 99
query13 1256 604 398 398
query14 6911 5395 5055 5055
query14_1 4397 4409 4443 4409
query15 201 198 178 178
query16 1020 471 437 437
query17 1134 711 623 623
query18 2709 470 351 351
query19 214 185 146 146
query20 114 107 105 105
query21 217 147 130 130
query22 13705 13640 13393 13393
query23 17119 16323 15983 15983
query23_1 16139 16112 16181 16112
query24 7536 1804 1391 1391
query24_1 1389 1400 1381 1381
query25 591 490 441 441
query26 1289 327 167 167
query27 2654 583 331 331
query28 4435 1997 1983 1983
query29 1102 633 519 519
query30 312 251 202 202
query31 1134 1074 952 952
query32 107 64 62 62
query33 529 344 265 265
query34 1179 1173 657 657
query35 769 771 689 689
query36 1282 1308 1133 1133
query37 154 110 93 93
query38 3238 3152 3056 3056
query39 939 932 921 921
query39_1 873 876 862 862
query40 239 138 126 126
query41 71 69 70 69
query42 98 99 95 95
query43 329 333 292 292
query44
query45 194 196 182 182
query46 1107 1218 754 754
query47 2297 2352 2154 2154
query48 409 417 320 320
query49 659 525 421 421
query50 714 293 228 228
query51 4334 4260 4158 4158
query52 91 92 79 79
query53 254 275 203 203
query54 289 252 220 220
query55 80 78 71 71
query56 253 235 236 235
query57 1393 1377 1278 1278
query58 256 226 231 226
query59 1602 1672 1442 1442
query60 296 255 245 245
query61 152 160 174 160
query62 666 666 569 569
query63 225 194 190 190
query64 2522 811 669 669
query65
query66 1760 492 370 370
query67 30173 30048 29895 29895
query68
query69 438 323 276 276
query70 975 977 903 903
query71 298 223 207 207
query72 2895 2791 2531 2531
query73 813 717 402 402
query74 5077 4955 4748 4748
query75 2760 2609 2299 2299
query76 2308 1131 786 786
query77 375 388 310 310
query78 12930 13050 12339 12339
query79 1433 1008 736 736
query80 1313 515 407 407
query81 523 280 243 243
query82 653 158 118 118
query83 355 272 246 246
query84 267 137 115 115
query85 931 511 428 428
query86 425 294 270 270
query87 3390 3371 3221 3221
query88 3532 2662 2663 2662
query89 426 371 327 327
query90 1918 183 195 183
query91 185 201 143 143
query92 66 63 53 53
query93 1120 987 588 588
query94 716 353 315 315
query95 689 475 348 348
query96 992 781 348 348
query97 2716 2642 2562 2562
query98 217 203 204 203
query99 1122 1094 969 969
Total cold run time: 252700 ms
Total hot run time: 169889 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3240798438
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterPartitionPruneClassifier.java:
##########
@@ -0,0 +1,281 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.glue.translator;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.analysis.FunctionCallExpr;
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.analysis.SlotRef;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.ListPartitionItem;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.PartitionInfo;
+import org.apache.doris.catalog.PartitionItem;
+import org.apache.doris.catalog.PartitionKey;
+import org.apache.doris.catalog.PartitionType;
+import org.apache.doris.catalog.RangePartitionItem;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.Monotonic;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.planner.OlapScanNode;
+import org.apache.doris.planner.PlanNode;
+import org.apache.doris.thrift.TTargetExprMonotonicity;
+
+import com.google.common.collect.Range;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Classifies whether one runtime-filter target can safely drive BE-side partition pruning.
+ *
+ * <p>This is the single FE gate for RF partition pruning. It intentionally reasons about the
+ * final legacy target expression that will be sent to BE, so late-added casts and partition
+ * expression domains cannot bypass the safety checks.
+ */
+final class RuntimeFilterPartitionPruneClassifier {
+ private RuntimeFilterPartitionPruneClassifier() {
+ }
+
+ static Classification classify(Expr targetExpr, Expression nereidsTargetExpr, PlanNode scanNode) {
+ if (!(scanNode instanceof OlapScanNode)) {
+ return Classification.unsupported("target scan is not an OlapScanNode");
+ }
+
+ OlapScanNode olapScanNode = (OlapScanNode) scanNode;
+ OlapTable table = olapScanNode.getOlapTable();
+ if (table == null) {
+ return Classification.unsupported("target scan has no OlapTable");
+ }
+
+ PartitionInfo partitionInfo = table.getPartitionInfo();
+ PartitionType partType = partitionInfo.getType();
+ if (partType != PartitionType.RANGE && partType != PartitionType.LIST) {
+ return Classification.unsupported("partition type is not RANGE or LIST");
+ }
+ if (hasUnsupportedAutomaticPartitionExpression(partitionInfo)) {
+ return Classification.unsupported("automatic partition expression boundary is not modeled");
+ }
+
+ if (targetExpr instanceof SlotRef) {
+ SlotRef slotRef = (SlotRef) targetExpr;
+ if (!isPartitionColumnSlot(slotRef, partitionInfo.getPartitionColumns())) {
Review Comment:
This accepts any partition column for RANGE targets, but `OlapScanNode.addRangeBoundaries()` serializes a `TPartitionBoundary` only for `partColumns.get(0)`. For a table such as `PARTITION BY RANGE(k1, k2)`, a runtime filter on `k2` reaches this branch, `RuntimeFilter.toThrift()` emits per-partition monotonicity for the scan, and `OlapScanNode` still sends only the `k1` boundary. BE then looks up `slot_to_boundaries` for the `k2` slot in `RuntimeFilterPartitionPruner::prune_by_runtime_filters()` and returns `InternalError("target slot is not a partition boundary slot")`, so a valid join on the second RANGE partition column fails when RF partition pruning is enabled. This is distinct from the earlier first-column multi-range projection thread: even with the conservative first-column projection, non-first RANGE partition targets need to be rejected here (or FE must serialize correct boundaries for that slot) and covered by a regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4495329806
<details>
<summary>TPC-DS: <b>Total hot run time: 168638 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6be29ffaf962c85e6c243c68a1c26cf413cdb0b5, data reload: false
query5 4341 632 490 490
query6 428 209 190 190
query7 4811 537 286 286
query8 390 223 207 207
query9 8809 4033 4034 4033
query10 436 310 263 263
query11 5987 2409 2248 2248
query12 159 105 101 101
query13 1270 651 394 394
query14 6369 5300 4984 4984
query14_1 4335 4291 4311 4291
query15 197 195 172 172
query16 989 450 420 420
query17 1089 703 574 574
query18 2428 480 352 352
query19 198 179 141 141
query20 113 106 102 102
query21 211 136 121 121
query22 13566 13541 13377 13377
query23 17296 16444 16024 16024
query23_1 16207 16223 16158 16158
query24 7527 1735 1318 1318
query24_1 1334 1310 1298 1298
query25 577 451 427 427
query26 1327 330 167 167
query27 2667 559 340 340
query28 4466 1975 1983 1975
query29 1113 642 525 525
query30 313 245 202 202
query31 1115 1071 976 976
query32 116 62 60 60
query33 543 329 265 265
query34 1174 1127 662 662
query35 755 773 673 673
query36 1330 1319 1126 1126
query37 152 108 92 92
query38 3206 3160 3043 3043
query39 933 921 918 918
query39_1 876 882 877 877
query40 228 127 105 105
query41 72 68 69 68
query42 96 99 97 97
query43 319 317 278 278
query44
query45 196 193 182 182
query46 1053 1250 748 748
query47 2284 2343 2131 2131
query48 386 419 290 290
query49 651 479 374 374
query50 978 343 249 249
query51 4344 4282 4184 4184
query52 90 90 77 77
query53 248 266 195 195
query54 288 247 217 217
query55 81 77 71 71
query56 246 250 239 239
query57 1423 1372 1288 1288
query58 252 230 233 230
query59 1518 1587 1407 1407
query60 304 264 282 264
query61 158 154 154 154
query62 657 621 562 562
query63 234 184 185 184
query64 2543 772 627 627
query65
query66 1789 468 340 340
query67 30094 29916 29928 29916
query68
query69 428 298 275 275
query70 964 946 938 938
query71 296 221 211 211
query72 2887 2688 2422 2422
query73 867 766 426 426
query74 5089 4929 4752 4752
query75 2649 2561 2260 2260
query76 2315 1113 750 750
query77 354 373 281 281
query78 12206 12269 11723 11723
query79 1403 1026 786 786
query80 1295 481 387 387
query81 515 280 242 242
query82 578 160 121 121
query83 325 281 251 251
query84 261 143 113 113
query85 921 514 471 471
query86 432 284 269 269
query87 3382 3428 3154 3154
query88 3509 2663 2643 2643
query89 433 370 323 323
query90 1922 178 179 178
query91 188 169 140 140
query92 61 64 54 54
query93 1457 1513 807 807
query94 734 345 303 303
query95 688 398 445 398
query96 1089 780 343 343
query97 2668 2662 2519 2519
query98 222 204 204 204
query99 1105 1092 974 974
Total cold run time: 251434 ms
Total hot run time: 168638 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4543796226
<details>
<summary>TPC-DS: <b>Total hot run time: 171358 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3d38dc5ab813ed88625762426d63d5324bfff4fe, data reload: false
query5 4338 638 527 527
query6 451 200 206 200
query7 4851 566 295 295
query8 371 222 206 206
query9 8818 4183 4158 4158
query10 456 306 276 276
query11 5870 2415 2232 2232
query12 158 105 100 100
query13 1274 611 442 442
query14 6482 5499 5165 5165
query14_1 4492 4457 4437 4437
query15 210 199 180 180
query16 1041 469 421 421
query17 1114 701 565 565
query18 2534 479 342 342
query19 205 180 137 137
query20 111 106 104 104
query21 213 136 119 119
query22 13685 13597 13508 13508
query23 17485 16548 16064 16064
query23_1 16330 16338 16259 16259
query24 7646 1794 1338 1338
query24_1 1317 1305 1320 1305
query25 554 442 390 390
query26 1324 331 173 173
query27 2652 591 333 333
query28 4507 2029 2002 2002
query29 1064 616 512 512
query30 300 245 191 191
query31 1126 1074 989 989
query32 113 62 58 58
query33 514 322 261 261
query34 1186 1167 660 660
query35 780 780 691 691
query36 1414 1411 1245 1245
query37 151 102 90 90
query38 3239 3127 3086 3086
query39 923 909 930 909
query39_1 874 874 894 874
query40 222 125 98 98
query41 74 66 64 64
query42 97 95 93 93
query43 326 325 273 273
query44
query45 201 187 177 177
query46 1124 1157 699 699
query47 2368 2435 2273 2273
query48 425 402 307 307
query49 638 471 362 362
query50 977 357 251 251
query51 4364 4302 4278 4278
query52 87 91 79 79
query53 255 275 193 193
query54 268 221 213 213
query55 82 82 70 70
query56 238 238 211 211
query57 1422 1404 1328 1328
query58 260 228 218 218
query59 1593 1701 1460 1460
query60 301 270 234 234
query61 189 187 191 187
query62 704 656 593 593
query63 234 192 197 192
query64 2664 827 690 690
query65
query66 1802 501 358 358
query67 29736 29681 29585 29585
query68
query69 441 312 284 284
query70 971 948 901 901
query71 312 234 217 217
query72 3233 2924 2462 2462
query73 845 753 430 430
query74 5126 4999 4792 4792
query75 2711 2594 2253 2253
query76 2300 1160 786 786
query77 369 383 271 271
query78 12469 12621 11864 11864
query79 1446 1085 792 792
query80 586 508 401 401
query81 454 282 251 251
query82 558 153 120 120
query83 353 281 255 255
query84 259 144 117 117
query85 874 561 486 486
query86 371 316 275 275
query87 3422 3351 3218 3218
query88 3682 2749 2769 2749
query89 430 377 330 330
query90 1960 171 178 171
query91 181 167 142 142
query92 65 63 58 58
query93 1469 1430 901 901
query94 538 348 279 279
query95 694 466 354 354
query96 1140 837 348 348
query97 2685 2676 2567 2567
query98 225 214 210 210
query99 1196 1140 978 978
Total cold run time: 252443 ms
Total hot run time: 171358 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4507327150
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3308916954
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,875 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ if (!tb.__isset.partition_id || !tb.__isset.slot_id) {
+ return Status::InternalError(
+ "Runtime filter partition boundary must contain partition_id and slot_id");
+ }
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ if (slot_it == slot_descs.end()) {
+ return Status::InternalError(
+ "Runtime filter partition boundary references unknown slot_id={}, "
+ "partition_id={}",
+ slot_id, tb.partition_id);
+ }
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ if (!is_list && !is_range) { \
+ return Status::InternalError( \
+ "Runtime filter partition boundary must be RANGE or LIST, " \
+ "partition_id={}, slot_id={}", \
+ tb.partition_id, slot_id); \
+ } \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ if (parsed_ok) {
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ } else {
+ return Status::InternalError(
+ "Runtime filter partition boundary has unsupported type, partition_id={}, "
+ "slot_id={}, type={}",
+ tb.partition_id, slot_id, slot_type->get_name());
+ }
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ if (!expr) {
+ return nullptr;
+ }
+ if (expr->is_slot_ref()) {
+ return assert_cast<const VSlotRef*>(expr);
+ }
+ const VSlotRef* found = nullptr;
+ for (const auto& child : expr->children()) {
+ const VSlotRef* c = find_unique_slot_ref(child.get());
+ if (c) {
+ if (found) {
+ return nullptr; // multiple slot refs, can't handle
+ }
+ found = c;
+ }
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+Status ParsedPartitionBoundaries::get_or_compute_projected_boundaries(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ const std::unordered_map<int64_t, TTargetExprMonotonicity::type>& partition_directions,
+ VExprContext* ctx, std::shared_ptr<const std::vector<ParsedBoundary>>* output) const {
+ {
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto it = _projected_boundaries_by_filter_id.find(filter_id);
+ if (it != _projected_boundaries_by_filter_id.end()) {
+ *output = it->second;
+ return Status::OK();
+ }
+ }
+
+ std::vector<ParsedBoundary> projected;
+ auto store_projected_boundaries = [&](std::vector<ParsedBoundary>&& boundaries) {
+ auto cached = std::make_shared<const std::vector<ParsedBoundary>>(std::move(boundaries));
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto [it, inserted] = _projected_boundaries_by_filter_id.emplace(filter_id, cached);
+ *output = inserted ? cached : it->second;
+ return Status::OK();
+ };
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::unordered_set<int64_t> boundary_partition_ids;
+ boundary_partition_ids.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ boundary_partition_ids.insert(boundary.partition_id);
+ }
+ for (const auto& [partition_id, direction] : partition_directions) {
+ if (direction == TTargetExprMonotonicity::NON_MONOTONIC) {
+ return Status::InternalError(
+ "Runtime filter partition pruning received NON_MONOTONIC partition "
+ "metadata, filter_id={}, partition_id={}",
+ filter_id, partition_id);
+ }
+ if (!boundary_partition_ids.contains(partition_id)) {
+ return Status::InternalError(
+ "Runtime filter partition pruning received monotonicity for a partition "
+ "without boundary, filter_id={}, partition_id={}, slot_id={}",
+ filter_id, partition_id, leaf_slot_id);
+ }
+ }
+
+ if (target_expr->is_slot_ref()) {
+ auto* slot_ref = assert_cast<VSlotRef*>(target_expr.get());
+ if (slot_ref->slot_id() != leaf_slot_id) {
+ return Status::InternalError(
+ "Runtime filter partition pruning SlotRef target mismatch, filter_id={}, "
+ "target_slot_id={}, leaf_slot_id={}",
+ filter_id, slot_ref->slot_id(), leaf_slot_id);
+ }
+ std::vector<ParsedBoundary> slot_boundaries;
+ slot_boundaries.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ auto direction_it = partition_directions.find(boundary.partition_id);
+ if (direction_it == partition_directions.end()) {
+ continue;
+ }
+ if (direction_it->second != TTargetExprMonotonicity::MONOTONIC_INCREASING) {
+ return Status::InternalError(
+ "Runtime filter partition pruning SlotRef target must use increasing "
+ "monotonicity, filter_id={}, partition_id={}, monotonicity={}",
+ filter_id, boundary.partition_id, direction_it->second);
+ }
+ slot_boundaries.emplace_back(boundary);
+ }
+ return store_projected_boundaries(std::move(slot_boundaries));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> selected_boundaries;
+ selected_boundaries.reserve(orig_boundaries.size());
+ for (size_t i = 0; i < orig_boundaries.size(); ++i) {
+ auto it = partition_directions.find(orig_boundaries[i].partition_id);
+ if (it != partition_directions.end() &&
+ it->second != TTargetExprMonotonicity::NON_MONOTONIC) {
+ selected_boundaries.emplace_back(i, it->second);
+ }
+ }
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> projectable_boundaries;
+ projectable_boundaries.reserve(selected_boundaries.size());
+ for (const auto& selected_boundary : selected_boundaries) {
+ const auto& boundary = orig_boundaries[selected_boundary.first];
+ if (!boundary.only_null && !boundary.contains_null) {
+ projectable_boundaries.emplace_back(selected_boundary);
+ }
+ }
+ selected_boundaries.swap(projectable_boundaries);
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ if (slot_type_it == _slot_data_types.end()) {
+ return Status::InternalError(
+ "Runtime filter partition pruning target slot has no boundary type, filter_id={}, "
+ "slot_id={}, target_expr={}",
+ filter_id, leaf_slot_id, target_expr->expr_name());
+ }
+
+ // LIST partitions require regrouping projected values per partition and
+ // should not be marked as expression-prunable by FE yet.
+ for (const auto& selected_boundary : selected_boundaries) {
+ bool is_list = false;
+ std::visit([&](const auto& cvr) { is_list = cvr.is_fixed_value_range(); },
+ orig_boundaries[selected_boundary.first].boundary_cvr);
+ if (is_list) {
+ return Status::InternalError(
+ "Runtime filter partition pruning expression target does not support LIST "
+ "partition boundaries, filter_id={}, partition_id={}, target_expr={}",
+ filter_id, orig_boundaries[selected_boundary.first].partition_id,
+ target_expr->expr_name());
+ }
+ }
+
+ const DataTypePtr& input_type = slot_type_it->second;
+ bool input_is_nullable = input_type->is_nullable();
+ DataTypePtr inner_type = input_is_nullable ? remove_nullable(input_type) : input_type;
+ PrimitiveType input_ptype = input_type->get_primitive_type();
+
+ size_t N = selected_boundaries.size();
+ std::vector<bool> lo_open(N, false);
+ std::vector<bool> hi_open(N, false);
+ std::vector<size_t> lo_result_row(N, 0);
+ std::vector<size_t> hi_result_row(N, 0);
+ size_t lo_row_count = 0;
+ size_t hi_row_count = 0;
+
+ // Build input blocks for lo and hi
+ Block lo_block;
+ Block hi_block;
+
+ // Pad columns 0..leaf_column_id-1 with placeholder columns so the leaf
+ // VSlotRef (whose column_id is leaf_column_id) reads our typed column.
+ auto add_dummy_columns = [&](Block& block, size_t row_count) {
+ for (int col_idx = 0; col_idx < leaf_column_id; ++col_idx) {
+ auto col = ColumnUInt8::create(row_count, 0);
+ block.insert({std::move(col), std::make_shared<DataTypeUInt8>(),
+ fmt::format("dummy_{}", col_idx)});
+ }
+ };
+
+ // Macro-dispatch on input PrimitiveType to build finite RANGE endpoint
+ // columns. Open endpoints are not executed through target_expr; they stay
+ // open after projection and swap sides for monotonic decreasing targets.
+ bool input_built = false;
+#define BUILD_INPUT_COLUMNS(INPUT_PT) \
+ case TYPE_##INPUT_PT: { \
+ using InCol = typename PrimitiveTypeTraits<TYPE_##INPUT_PT>::ColumnType; \
+ MutableColumnPtr lo_inner_base = inner_type->create_column(); \
+ MutableColumnPtr hi_inner_base = inner_type->create_column(); \
+ auto* lo_inner = assert_cast<InCol*>(lo_inner_base.get()); \
+ auto* hi_inner = assert_cast<InCol*>(hi_inner_base.get()); \
+ lo_inner->reserve(N); \
+ hi_inner->reserve(N); \
+ auto lo_nulls = ColumnUInt8::create(); \
+ auto hi_nulls = ColumnUInt8::create(); \
+ lo_nulls->reserve(N); \
+ hi_nulls->reserve(N); \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& boundary = orig_boundaries[selected_boundaries[i].first]; \
+ const auto* cvr = \
+ std::get_if<ColumnValueRange<TYPE_##INPUT_PT>>(&boundary.boundary_cvr); \
+ DCHECK(cvr != nullptr); \
+ DCHECK(!boundary.only_null); \
+ DCHECK(!boundary.contains_null); \
+ lo_open[i] = cvr->is_low_value_minimum(); \
+ hi_open[i] = cvr->is_high_value_maximum(); \
+ if (!lo_open[i]) { \
+ lo_result_row[i] = lo_row_count++; \
+ lo_inner->insert_value(cvr->get_range_min_value()); \
+ lo_nulls->get_data().push_back(0); \
+ } \
+ if (!hi_open[i]) { \
+ hi_result_row[i] = hi_row_count++; \
+ hi_inner->insert_value(cvr->get_range_max_value()); \
+ hi_nulls->get_data().push_back(0); \
+ } \
+ } \
+ add_dummy_columns(lo_block, lo_row_count); \
+ add_dummy_columns(hi_block, hi_row_count); \
+ if (input_is_nullable) { \
+ auto lo_col = ColumnNullable::create(std::move(lo_inner_base), std::move(lo_nulls)); \
+ auto hi_col = ColumnNullable::create(std::move(hi_inner_base), std::move(hi_nulls)); \
+ lo_block.insert({std::move(lo_col), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_col), input_type, "leaf_slot"}); \
+ } else { \
+ lo_block.insert({std::move(lo_inner_base), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_inner_base), input_type, "leaf_slot"}); \
+ } \
+ input_built = true; \
+ break; \
+ }
+
+ switch (input_ptype) {
+ BUILD_INPUT_COLUMNS(TINYINT)
+ BUILD_INPUT_COLUMNS(SMALLINT)
+ BUILD_INPUT_COLUMNS(INT)
+ BUILD_INPUT_COLUMNS(BIGINT)
+ BUILD_INPUT_COLUMNS(LARGEINT)
+ BUILD_INPUT_COLUMNS(FLOAT)
+ BUILD_INPUT_COLUMNS(DOUBLE)
+ BUILD_INPUT_COLUMNS(CHAR)
+ BUILD_INPUT_COLUMNS(DATE)
+ BUILD_INPUT_COLUMNS(DATETIME)
+ BUILD_INPUT_COLUMNS(DATEV2)
+ BUILD_INPUT_COLUMNS(DATETIMEV2)
+ BUILD_INPUT_COLUMNS(TIMESTAMPTZ)
+ BUILD_INPUT_COLUMNS(VARCHAR)
+ BUILD_INPUT_COLUMNS(STRING)
+ BUILD_INPUT_COLUMNS(DECIMAL32)
+ BUILD_INPUT_COLUMNS(DECIMAL64)
+ BUILD_INPUT_COLUMNS(DECIMAL128I)
+ BUILD_INPUT_COLUMNS(DECIMAL256)
+ BUILD_INPUT_COLUMNS(DECIMALV2)
+ BUILD_INPUT_COLUMNS(BOOLEAN)
+ BUILD_INPUT_COLUMNS(IPV4)
+ BUILD_INPUT_COLUMNS(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_INPUT_COLUMNS
+
+ if (!input_built) {
+ return Status::InternalError(
+ "Runtime filter partition pruning expression target has unsupported input type, "
+ "filter_id={}, input_type={}, target_expr={}",
+ filter_id, input_type->get_name(), target_expr->expr_name());
+ }
+
+ int lo_result_id = -1;
+ int hi_result_id = -1;
+ ColumnPtr lo_result_col;
+ ColumnPtr hi_result_col;
+ if (lo_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &lo_block, &lo_result_id));
+ if (lo_result_id < 0) {
+ return Status::InternalError(
+ "Runtime filter partition pruning failed to project lower boundary, "
+ "filter_id={}, target_expr={}",
+ filter_id, target_expr->expr_name());
+ }
+ lo_result_col = lo_block.get_by_position(lo_result_id).column;
+ }
+ if (hi_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &hi_block, &hi_result_id));
+ if (hi_result_id < 0) {
+ return Status::InternalError(
+ "Runtime filter partition pruning failed to project upper boundary, "
+ "filter_id={}, target_expr={}",
+ filter_id, target_expr->expr_name());
+ }
+ hi_result_col = hi_block.get_by_position(hi_result_id).column;
+ }
+ int out_precision = cast_set<int>(target_expr->data_type()->get_precision());
+ int out_scale = cast_set<int>(target_expr->data_type()->get_scale());
+ bool out_nullable = target_expr->data_type()->is_nullable();
+ PrimitiveType output_ptype = target_expr->data_type()->get_primitive_type();
+
+ projected.reserve(N);
+
+ bool output_built = false;
+#define BUILD_PROJECTED_CVR(OUTPUT_PT) \
+ case TYPE_##OUTPUT_PT: { \
+ using OutputCppType = typename PrimitiveTypeTraits<TYPE_##OUTPUT_PT>::CppType; \
+ auto projected_value = [](const ColumnPtr& column, size_t row) -> OutputCppType { \
+ Field field = (*column)[row]; \
+ return field.get<TYPE_##OUTPUT_PT>(); \
+ }; \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& orig_boundary = orig_boundaries[selected_boundaries[i].first]; \
+ TTargetExprMonotonicity::type direction = selected_boundaries[i].second; \
+ if ((!lo_open[i] && lo_result_col->is_null_at(lo_result_row[i])) || \
+ (!hi_open[i] && hi_result_col->is_null_at(hi_result_row[i]))) { \
+ continue; \
+ } \
+ ColumnValueRange<TYPE_##OUTPUT_PT> cvr("", out_nullable, out_precision, out_scale); \
+ if (direction == TTargetExprMonotonicity::MONOTONIC_DECREASING) { \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, hi_projected)); \
+ } \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, lo_projected)); \
+ } \
+ } else { \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, lo_projected)); \
+ } \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, hi_projected)); \
+ } \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.only_null = orig_boundary.only_null; \
+ projected_boundary.contains_null = orig_boundary.contains_null; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ } \
+ output_built = true; \
+ break; \
+ }
+
+ switch (output_ptype) {
+ BUILD_PROJECTED_CVR(TINYINT)
+ BUILD_PROJECTED_CVR(SMALLINT)
+ BUILD_PROJECTED_CVR(INT)
+ BUILD_PROJECTED_CVR(BIGINT)
+ BUILD_PROJECTED_CVR(LARGEINT)
+ BUILD_PROJECTED_CVR(FLOAT)
+ BUILD_PROJECTED_CVR(DOUBLE)
+ BUILD_PROJECTED_CVR(CHAR)
+ BUILD_PROJECTED_CVR(DATE)
+ BUILD_PROJECTED_CVR(DATETIME)
+ BUILD_PROJECTED_CVR(DATEV2)
+ BUILD_PROJECTED_CVR(DATETIMEV2)
+ BUILD_PROJECTED_CVR(TIMESTAMPTZ)
+ BUILD_PROJECTED_CVR(VARCHAR)
+ BUILD_PROJECTED_CVR(STRING)
+ BUILD_PROJECTED_CVR(DECIMAL32)
+ BUILD_PROJECTED_CVR(DECIMAL64)
+ BUILD_PROJECTED_CVR(DECIMAL128I)
+ BUILD_PROJECTED_CVR(DECIMAL256)
+ BUILD_PROJECTED_CVR(DECIMALV2)
+ BUILD_PROJECTED_CVR(BOOLEAN)
+ BUILD_PROJECTED_CVR(IPV4)
+ BUILD_PROJECTED_CVR(IPV6)
+ default:
+ break;
+ }
+
+#undef BUILD_PROJECTED_CVR
+
+ if (!output_built) {
+ return Status::InternalError(
+ "Runtime filter partition pruning expression target has unsupported output type, "
+ "filter_id={}, output_type={}, target_expr={}",
+ filter_id, target_expr->data_type()->get_name(), target_expr->expr_name());
+ }
+
+ return store_projected_boundaries(std::move(projected));
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static SQLFilterOp convert_opcode_to_filter_op(TExprOpcode::type op) {
+ switch (op) {
+ case TExprOpcode::LE:
+ return FILTER_LESS_OR_EQUAL;
+ case TExprOpcode::LT:
+ return FILTER_LESS;
+ case TExprOpcode::GE:
+ return FILTER_LARGER_OR_EQUAL;
+ case TExprOpcode::GT:
+ return FILTER_LARGER;
+ default:
+ return FILTER_IN; // sentinel: caller should skip
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+void RuntimeFilterPartitionPruner::_try_prune_by_single_rf(
+ const std::vector<ParsedBoundary>& boundaries, const VExprSPtr& impl,
+ phmap::flat_hash_set<int64_t>& newly_pruned) {
+ // Pre-compute whether the RF "matches NULL" -- i.e. whether the RF would
+ // accept a row whose probe value is NULL. The signal differs by RF impl:
+ // * IN filter → HybridSet::contain_null()
+ // * Bloom filter → BloomFilterFuncBase::contain_null()
+ // * MinMax filter → encoded via the BinaryPredicate node_type:
+ // NULL_AWARE_BINARY_PRED means the build side had a
+ // NULL key and a null-safe equal join asked NULL to
+ // match NULL; plain BINARY_PRED never matches NULL.
+ // FilterBase::contain_null() already folds in `_null_aware`, so we only
+ // get a true result when the build side is actually null-aware AND
+ // produced a NULL value.
+ bool rf_contains_null = false;
+ if (auto hybrid_set = impl->get_set_func()) {
+ rf_contains_null = hybrid_set->contain_null();
+ } else if (impl->node_type() == TExprNodeType::BLOOM_PRED) {
+ auto bloom = impl->get_bloom_filter_func();
+ rf_contains_null = bloom && bloom->contain_null();
+ } else if (impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) {
+ // Min/Max RF built on a null-safe equal join. The literal child holds
+ // the min or max bound; the NULL semantic is conveyed by the node
+ // type itself (see create_vbin_predicate in runtime_filter/utils.cpp).
+ rf_contains_null = true;
+ }
+
+ for (const auto& pb : boundaries) {
+ if (_pruned_partition_ids.contains(pb.partition_id) ||
+ newly_pruned.contains(pb.partition_id)) {
+ continue;
+ }
+
+ // NULL handling:
+ // A partition row whose key is NULL matches the RF iff `rf_contains_null`.
+ // - only_null partition (rows are exclusively NULL): prunable iff !rf_contains_null.
+ // - mixed (NULL + concrete values): if rf_contains_null, NULL rows alone
+ // prevent pruning. Otherwise NULL rows can never match, so we ignore
+ // the NULL portion and let the regular non-NULL intersection decide.
+ if (pb.only_null) {
+ if (!rf_contains_null) {
+ newly_pruned.insert(pb.partition_id);
+ }
+ continue;
+ }
+ if (pb.contains_null && rf_contains_null) {
+ continue;
+ }
+
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+ using CppType = typename CvrType::CppType;
+
+ auto hybrid_set = impl->get_set_func();
+ if (hybrid_set) {
Review Comment:
For循环能不能放在外面，这个runtime filter的这个range只需要生成一次，后续的这个每一个partition的都复用这个生成的来计算，而不是每一个partition算一次
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4473676379
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4474545969
<details>
<summary>TPC-DS: <b>Total hot run time: 168106 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ef2529ca3c35ebe1f89cb9fa4212f239fce43a97, data reload: false
query5 4352 651 488 488
query6 463 202 189 189
query7 4932 585 300 300
query8 373 223 209 209
query9 8800 4054 4084 4054
query10 456 313 267 267
query11 5891 2380 2179 2179
query12 161 106 103 103
query13 1258 592 430 430
query14 6427 5358 5035 5035
query14_1 4339 4381 4375 4375
query15 211 205 178 178
query16 1029 464 485 464
query17 1142 723 618 618
query18 2674 502 360 360
query19 212 189 151 151
query20 113 111 107 107
query21 231 149 126 126
query22 13677 13580 13413 13413
query23 17219 16378 15888 15888
query23_1 16128 16072 15998 15998
query24 7567 1796 1300 1300
query24_1 1317 1310 1331 1310
query25 578 488 406 406
query26 1299 355 179 179
query27 2616 555 336 336
query28 4452 1988 1955 1955
query29 1148 646 526 526
query30 307 250 206 206
query31 1127 1077 946 946
query32 108 62 65 62
query33 534 341 271 271
query34 1183 1140 651 651
query35 758 808 666 666
query36 1354 1297 1118 1118
query37 157 106 94 94
query38 3211 3134 3030 3030
query39 937 939 905 905
query39_1 900 868 869 868
query40 224 123 101 101
query41 66 68 64 64
query42 96 92 92 92
query43 321 319 274 274
query44
query45 196 182 177 177
query46 1047 1165 712 712
query47 2357 2349 2173 2173
query48 402 414 291 291
query49 635 457 367 367
query50 1032 361 244 244
query51 4294 4276 4374 4276
query52 89 90 78 78
query53 234 299 195 195
query54 272 219 203 203
query55 79 78 69 69
query56 240 221 221 221
query57 1392 1374 1310 1310
query58 244 238 214 214
query59 1569 1635 1430 1430
query60 287 253 240 240
query61 165 161 157 157
query62 670 647 556 556
query63 232 196 188 188
query64 2534 772 602 602
query65
query66 1742 469 341 341
query67 30155 29907 29826 29826
query68
query69 426 302 279 279
query70 971 987 934 934
query71 304 223 217 217
query72 3067 2748 2426 2426
query73 870 755 395 395
query74 5091 4933 4732 4732
query75 2651 2564 2220 2220
query76 2331 1188 786 786
query77 371 400 284 284
query78 12185 12184 11610 11610
query79 1386 1033 712 712
query80 1140 479 381 381
query81 506 279 243 243
query82 578 160 125 125
query83 359 274 248 248
query84 268 143 118 118
query85 939 533 473 473
query86 415 290 274 274
query87 3403 3418 3206 3206
query88 3625 2676 2642 2642
query89 446 375 322 322
query90 1824 190 192 190
query91 183 170 146 146
query92 63 63 61 61
query93 1486 1562 890 890
query94 633 372 294 294
query95 674 384 431 384
query96 1045 818 319 319
query97 2641 2679 2502 2502
query98 211 207 220 207
query99 1088 1111 980 980
Total cold run time: 251635 ms
Total hot run time: 168106 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4564615062
# BE Regression && UT Coverage Report
Increment line coverage `84.81% (653/770)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_2ede6c8cafb5ff16b7121d7edac1018fc2e592e0_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_2ede6c8cafb5ff16b7121d7edac1018fc2e592e0_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 72.20% (27496/38085) |
| Line Coverage | 55.56% (294085/529273) |
| Region Coverage | 52.38% (245813/469294) |
| Branch Coverage | 53.58% (106213/198248) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4560806209
# BE UT Coverage Report
Increment line coverage `66.47% (557/838)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/9af6867af36b4da0dec52406116ba2d158e2e2b9_9af6867af36b4da0dec52406116ba2d158e2e2b9/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/9af6867af36b4da0dec52406116ba2d158e2e2b9_9af6867af36b4da0dec52406116ba2d158e2e2b9/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.94% (20972/38883) |
| Line Coverage | 37.55% (199239/530647) |
| Region Coverage | 33.75% (156810/464641) |
| Branch Coverage | 34.75% (68597/197411) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4570295597
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4570404102
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4333359613
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3331437214
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,866 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ DORIS_CHECK(tb.__isset.partition_id);
+ DORIS_CHECK(tb.__isset.slot_id);
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ DORIS_CHECK(slot_it != slot_descs.end());
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ DORIS_CHECK(is_list || is_range); \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ bool range_has_null = false; \
+ if (is_nullable && !tb.__isset.range_start) { \
+ range_has_null = true; \
+ } \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } else if (is_nullable) { \
+ range_has_null = true; \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ if (range_has_null) { \
+ boundary.contains_null = true; \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ DORIS_CHECK(parsed_ok);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static bool find_unique_slot_ref_impl(const VExpr* expr, const VSlotRef** found) {
+ if (!expr) {
+ return true;
+ }
+ if (expr->is_slot_ref()) {
+ const auto* slot = assert_cast<const VSlotRef*>(expr);
+ if (*found == nullptr) {
+ *found = slot;
+ return true;
+ }
+ if ((*found)->slot_id() != slot->slot_id()) {
+ return false;
+ }
+ DORIS_CHECK((*found)->column_id() == slot->column_id());
+ return true;
+ }
+ for (const auto& child : expr->children()) {
+ if (!find_unique_slot_ref_impl(child.get(), found)) {
+ return false;
+ }
+ }
+ return true;
+}
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ const VSlotRef* found = nullptr;
+ if (!find_unique_slot_ref_impl(expr, &found)) {
+ return nullptr;
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+Status ParsedPartitionBoundaries::get_or_compute_projected_boundaries(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ const std::unordered_map<int64_t, TTargetExprMonotonicity::type>& partition_directions,
+ VExprContext* ctx, std::shared_ptr<const std::vector<ParsedBoundary>>* output) const {
+ {
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto it = _projected_boundaries_by_filter_id.find(filter_id);
+ if (it != _projected_boundaries_by_filter_id.end()) {
+ *output = it->second;
+ return Status::OK();
+ }
+ }
+
+ std::vector<ParsedBoundary> projected;
+ auto store_projected_boundaries = [&](std::vector<ParsedBoundary>&& boundaries) {
+ auto cached = std::make_shared<const std::vector<ParsedBoundary>>(std::move(boundaries));
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto [it, inserted] = _projected_boundaries_by_filter_id.emplace(filter_id, cached);
+ *output = inserted ? cached : it->second;
+ return Status::OK();
+ };
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::unordered_set<int64_t> boundary_partition_ids;
+ boundary_partition_ids.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ boundary_partition_ids.insert(boundary.partition_id);
+ }
+ for (const auto& [partition_id, direction] : partition_directions) {
+ DORIS_CHECK(direction != TTargetExprMonotonicity::NON_MONOTONIC);
+ DORIS_CHECK(boundary_partition_ids.contains(partition_id));
+ }
+
+ if (target_expr->is_slot_ref()) {
+ auto* slot_ref = assert_cast<VSlotRef*>(target_expr.get());
+ DORIS_CHECK(slot_ref->slot_id() == leaf_slot_id);
+ std::vector<ParsedBoundary> slot_boundaries;
+ slot_boundaries.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ auto direction_it = partition_directions.find(boundary.partition_id);
+ if (direction_it == partition_directions.end()) {
+ continue;
+ }
+ DORIS_CHECK(direction_it->second == TTargetExprMonotonicity::MONOTONIC_INCREASING);
+ slot_boundaries.emplace_back(boundary);
+ }
+ return store_projected_boundaries(std::move(slot_boundaries));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> selected_boundaries;
+ selected_boundaries.reserve(orig_boundaries.size());
+ for (size_t i = 0; i < orig_boundaries.size(); ++i) {
+ auto it = partition_directions.find(orig_boundaries[i].partition_id);
+ if (it != partition_directions.end() &&
+ it->second != TTargetExprMonotonicity::NON_MONOTONIC) {
+ selected_boundaries.emplace_back(i, it->second);
+ }
+ }
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> projectable_boundaries;
+ projectable_boundaries.reserve(selected_boundaries.size());
+ for (const auto& selected_boundary : selected_boundaries) {
+ const auto& boundary = orig_boundaries[selected_boundary.first];
+ if (!boundary.only_null && !boundary.contains_null) {
+ projectable_boundaries.emplace_back(selected_boundary);
+ }
+ }
+ selected_boundaries.swap(projectable_boundaries);
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ DORIS_CHECK(slot_type_it != _slot_data_types.end());
+
+ const DataTypePtr& input_type = slot_type_it->second;
+ bool input_is_nullable = input_type->is_nullable();
+ DataTypePtr inner_type = input_is_nullable ? remove_nullable(input_type) : input_type;
+ PrimitiveType input_ptype = input_type->get_primitive_type();
+
+ size_t N = selected_boundaries.size();
+ std::vector<bool> lo_open(N, false);
+ std::vector<bool> hi_open(N, false);
+ std::vector<bool> boundary_is_list(N, false);
+ std::vector<size_t> lo_result_row(N, 0);
+ std::vector<size_t> hi_result_row(N, 0);
+ std::vector<size_t> list_result_begin(N, 0);
+ std::vector<size_t> list_result_end(N, 0);
+ size_t lo_row_count = 0;
+ size_t hi_row_count = 0;
+ size_t list_row_count = 0;
+
+ // Build input blocks for RANGE endpoints and LIST values.
+ Block lo_block;
+ Block hi_block;
+ Block list_block;
+
+ // Pad columns 0..leaf_column_id-1 with placeholder columns so the leaf
+ // VSlotRef (whose column_id is leaf_column_id) reads our typed column.
+ auto add_dummy_columns = [&](Block& block, size_t row_count) {
+ for (int col_idx = 0; col_idx < leaf_column_id; ++col_idx) {
+ auto col = ColumnUInt8::create(row_count, 0);
+ block.insert({std::move(col), std::make_shared<DataTypeUInt8>(),
+ fmt::format("dummy_{}", col_idx)});
+ }
+ };
+
+ // Macro-dispatch on input PrimitiveType to build finite RANGE endpoint
+ // columns and LIST value columns. Unbounded RANGE endpoints are not executed
+ // through target_expr; they stay unbounded after projection and swap sides
+ // for monotonic decreasing targets.
+ bool input_built = false;
+#define BUILD_INPUT_COLUMNS(INPUT_PT) \
+ case TYPE_##INPUT_PT: { \
+ using InCol = typename PrimitiveTypeTraits<TYPE_##INPUT_PT>::ColumnType; \
+ MutableColumnPtr lo_inner_base = inner_type->create_column(); \
+ MutableColumnPtr hi_inner_base = inner_type->create_column(); \
+ MutableColumnPtr list_inner_base = inner_type->create_column(); \
+ auto* lo_inner = assert_cast<InCol*>(lo_inner_base.get()); \
+ auto* hi_inner = assert_cast<InCol*>(hi_inner_base.get()); \
+ auto* list_inner = assert_cast<InCol*>(list_inner_base.get()); \
+ lo_inner->reserve(N); \
+ hi_inner->reserve(N); \
+ list_inner->reserve(N); \
+ auto lo_nulls = ColumnUInt8::create(); \
+ auto hi_nulls = ColumnUInt8::create(); \
+ auto list_nulls = ColumnUInt8::create(); \
+ lo_nulls->reserve(N); \
+ hi_nulls->reserve(N); \
+ list_nulls->reserve(N); \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& boundary = orig_boundaries[selected_boundaries[i].first]; \
+ const auto* cvr = \
+ std::get_if<ColumnValueRange<TYPE_##INPUT_PT>>(&boundary.boundary_cvr); \
+ DCHECK(cvr != nullptr); \
+ DCHECK(!boundary.only_null); \
+ DCHECK(!boundary.contains_null); \
+ boundary_is_list[i] = cvr->is_fixed_value_range(); \
+ if (boundary_is_list[i]) { \
+ list_result_begin[i] = list_row_count; \
+ for (const auto& value : cvr->get_fixed_value_set()) { \
+ list_inner->insert_value(value); \
+ list_nulls->get_data().push_back(0); \
+ ++list_row_count; \
+ } \
+ list_result_end[i] = list_row_count; \
+ continue; \
+ } \
+ lo_open[i] = cvr->is_low_value_minimum(); \
+ hi_open[i] = cvr->is_high_value_maximum(); \
+ if (!lo_open[i]) { \
+ lo_result_row[i] = lo_row_count++; \
+ lo_inner->insert_value(cvr->get_range_min_value()); \
+ lo_nulls->get_data().push_back(0); \
+ } \
+ if (!hi_open[i]) { \
+ hi_result_row[i] = hi_row_count++; \
+ hi_inner->insert_value(cvr->get_range_max_value()); \
+ hi_nulls->get_data().push_back(0); \
+ } \
+ } \
+ add_dummy_columns(lo_block, lo_row_count); \
+ add_dummy_columns(hi_block, hi_row_count); \
+ add_dummy_columns(list_block, list_row_count); \
+ if (input_is_nullable) { \
+ auto lo_col = ColumnNullable::create(std::move(lo_inner_base), std::move(lo_nulls)); \
+ auto hi_col = ColumnNullable::create(std::move(hi_inner_base), std::move(hi_nulls)); \
+ auto list_col = \
+ ColumnNullable::create(std::move(list_inner_base), std::move(list_nulls)); \
+ lo_block.insert({std::move(lo_col), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_col), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_col), input_type, "leaf_slot"}); \
+ } else { \
+ lo_block.insert({std::move(lo_inner_base), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_inner_base), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_inner_base), input_type, "leaf_slot"}); \
+ } \
+ input_built = true; \
+ break; \
+ }
+
+ switch (input_ptype) {
+ BUILD_INPUT_COLUMNS(TINYINT)
+ BUILD_INPUT_COLUMNS(SMALLINT)
+ BUILD_INPUT_COLUMNS(INT)
+ BUILD_INPUT_COLUMNS(BIGINT)
+ BUILD_INPUT_COLUMNS(LARGEINT)
+ BUILD_INPUT_COLUMNS(FLOAT)
+ BUILD_INPUT_COLUMNS(DOUBLE)
+ BUILD_INPUT_COLUMNS(CHAR)
+ BUILD_INPUT_COLUMNS(DATE)
+ BUILD_INPUT_COLUMNS(DATETIME)
+ BUILD_INPUT_COLUMNS(DATEV2)
+ BUILD_INPUT_COLUMNS(DATETIMEV2)
+ BUILD_INPUT_COLUMNS(TIMESTAMPTZ)
+ BUILD_INPUT_COLUMNS(VARCHAR)
+ BUILD_INPUT_COLUMNS(STRING)
+ BUILD_INPUT_COLUMNS(DECIMAL32)
+ BUILD_INPUT_COLUMNS(DECIMAL64)
+ BUILD_INPUT_COLUMNS(DECIMAL128I)
+ BUILD_INPUT_COLUMNS(DECIMAL256)
+ BUILD_INPUT_COLUMNS(DECIMALV2)
+ BUILD_INPUT_COLUMNS(BOOLEAN)
+ BUILD_INPUT_COLUMNS(IPV4)
+ BUILD_INPUT_COLUMNS(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_INPUT_COLUMNS
+
+ DORIS_CHECK(input_built);
+
+ int lo_result_id = -1;
+ int hi_result_id = -1;
+ int list_result_id = -1;
+ ColumnPtr lo_result_col;
+ ColumnPtr hi_result_col;
+ ColumnPtr list_result_col;
+ if (lo_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &lo_block, &lo_result_id));
+ DORIS_CHECK(lo_result_id >= 0);
+ lo_result_col = lo_block.get_by_position(lo_result_id).column;
+ }
+ if (hi_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &hi_block, &hi_result_id));
+ DORIS_CHECK(hi_result_id >= 0);
+ hi_result_col = hi_block.get_by_position(hi_result_id).column;
+ }
+ if (list_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &list_block, &list_result_id));
+ DORIS_CHECK(list_result_id >= 0);
+ list_result_col = list_block.get_by_position(list_result_id).column;
+ }
+ int out_precision = cast_set<int>(target_expr->data_type()->get_precision());
+ int out_scale = cast_set<int>(target_expr->data_type()->get_scale());
+ bool out_nullable = target_expr->data_type()->is_nullable();
+ PrimitiveType output_ptype = target_expr->data_type()->get_primitive_type();
+
+ projected.reserve(N);
+
+ bool output_built = false;
+#define BUILD_PROJECTED_CVR(OUTPUT_PT) \
+ case TYPE_##OUTPUT_PT: { \
+ using OutputCppType = typename PrimitiveTypeTraits<TYPE_##OUTPUT_PT>::CppType; \
+ auto projected_value = [](const ColumnPtr& column, size_t row) -> OutputCppType { \
+ Field field = (*column)[row]; \
+ return field.get<TYPE_##OUTPUT_PT>(); \
+ }; \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& orig_boundary = orig_boundaries[selected_boundaries[i].first]; \
+ TTargetExprMonotonicity::type direction = selected_boundaries[i].second; \
+ if (boundary_is_list[i]) { \
+ auto cvr = ColumnValueRange<TYPE_##OUTPUT_PT>::create_empty_column_value_range( \
+ out_nullable, out_precision, out_scale); \
+ bool list_has_value = false; \
+ bool list_has_null = false; \
+ for (size_t row = list_result_begin[i]; row < list_result_end[i]; ++row) { \
+ if (list_result_col->is_null_at(row)) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ auto projected_list_value = projected_value(list_result_col, row); \
+ static_cast<void>(cvr.add_fixed_value(projected_list_value)); \
+ list_has_value = true; \
+ } \
+ if (!list_has_value && !list_has_null) { \
+ continue; \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.contains_null = list_has_null; \
+ projected_boundary.only_null = list_has_null && !list_has_value; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ continue; \
+ } \
+ if ((!lo_open[i] && lo_result_col->is_null_at(lo_result_row[i])) || \
+ (!hi_open[i] && hi_result_col->is_null_at(hi_result_row[i]))) { \
+ continue; \
+ } \
+ ColumnValueRange<TYPE_##OUTPUT_PT> cvr("", out_nullable, out_precision, out_scale); \
+ /* Use closed projected finite bounds as a conservative envelope for */ \
+ /* non-strict monotonic expressions such as date ceil/floor/cast. */ \
+ if (direction == TTargetExprMonotonicity::MONOTONIC_DECREASING) { \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, hi_projected)); \
+ } \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, lo_projected)); \
+ } \
+ } else { \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, lo_projected)); \
+ } \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, hi_projected)); \
+ } \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.only_null = orig_boundary.only_null; \
+ projected_boundary.contains_null = orig_boundary.contains_null; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ } \
+ output_built = true; \
+ break; \
+ }
+
+ switch (output_ptype) {
+ BUILD_PROJECTED_CVR(TINYINT)
+ BUILD_PROJECTED_CVR(SMALLINT)
+ BUILD_PROJECTED_CVR(INT)
+ BUILD_PROJECTED_CVR(BIGINT)
+ BUILD_PROJECTED_CVR(LARGEINT)
+ BUILD_PROJECTED_CVR(FLOAT)
+ BUILD_PROJECTED_CVR(DOUBLE)
+ BUILD_PROJECTED_CVR(CHAR)
+ BUILD_PROJECTED_CVR(DATE)
+ BUILD_PROJECTED_CVR(DATETIME)
+ BUILD_PROJECTED_CVR(DATEV2)
+ BUILD_PROJECTED_CVR(DATETIMEV2)
+ BUILD_PROJECTED_CVR(TIMESTAMPTZ)
+ BUILD_PROJECTED_CVR(VARCHAR)
+ BUILD_PROJECTED_CVR(STRING)
+ BUILD_PROJECTED_CVR(DECIMAL32)
+ BUILD_PROJECTED_CVR(DECIMAL64)
+ BUILD_PROJECTED_CVR(DECIMAL128I)
+ BUILD_PROJECTED_CVR(DECIMAL256)
+ BUILD_PROJECTED_CVR(DECIMALV2)
+ BUILD_PROJECTED_CVR(BOOLEAN)
+ BUILD_PROJECTED_CVR(IPV4)
+ BUILD_PROJECTED_CVR(IPV6)
+ default:
+ break;
+ }
+
+#undef BUILD_PROJECTED_CVR
+
+ DORIS_CHECK(output_built);
+
+ return store_projected_boundaries(std::move(projected));
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static SQLFilterOp convert_opcode_to_filter_op(TExprOpcode::type op) {
+ switch (op) {
+ case TExprOpcode::LE:
+ return FILTER_LESS_OR_EQUAL;
+ case TExprOpcode::LT:
+ return FILTER_LESS;
+ case TExprOpcode::GE:
+ return FILTER_LARGER_OR_EQUAL;
+ case TExprOpcode::GT:
+ return FILTER_LARGER;
+ default:
+ return FILTER_IN; // sentinel: caller should skip
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+void RuntimeFilterPartitionPruner::_try_prune_by_single_rf(
+ const std::vector<ParsedBoundary>& boundaries, const VExprSPtr& impl,
+ phmap::flat_hash_set<int64_t>& newly_pruned) {
+ // Pre-compute whether the RF "matches NULL" -- i.e. whether the RF would
+ // accept a row whose probe value is NULL. The signal differs by RF impl:
+ // * IN filter → HybridSet::contain_null()
+ // * Bloom filter → BloomFilterFuncBase::contain_null()
+ // * MinMax filter → encoded via the BinaryPredicate node_type:
+ // NULL_AWARE_BINARY_PRED means the build side had a
+ // NULL key and a null-safe equal join asked NULL to
+ // match NULL; plain BINARY_PRED never matches NULL.
+ // FilterBase::contain_null() already folds in `_null_aware`, so we only
+ // get a true result when the build side is actually null-aware AND
+ // produced a NULL value.
+ bool rf_contains_null = false;
+ if (auto hybrid_set = impl->get_set_func()) {
+ rf_contains_null = hybrid_set->contain_null();
+ } else if (impl->node_type() == TExprNodeType::BLOOM_PRED) {
+ auto bloom = impl->get_bloom_filter_func();
+ rf_contains_null = bloom && bloom->contain_null();
+ } else if (impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) {
+ // Min/Max RF built on a null-safe equal join. The literal child holds
+ // the min or max bound; the NULL semantic is conveyed by the node
+ // type itself (see create_vbin_predicate in runtime_filter/utils.cpp).
+ rf_contains_null = true;
+ }
+
+ std::optional<ColumnValueRangeType> rf_cvr;
+ if (!boundaries.empty()) {
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+ using CppType = typename CvrType::CppType;
+
+ auto hybrid_set = impl->get_set_func();
+ if (hybrid_set) {
+ auto typed_rf_cvr = CvrType::create_empty_column_value_range(
+ boundaries.front().is_nullable, boundary_cvr.precision(),
+ boundary_cvr.scale());
+ auto* iter = hybrid_set->begin();
+ while (iter->has_next()) {
+ const void* value = iter->get_value();
+ if (value) {
+ if constexpr (std::is_same_v<CppType, String>) {
+ // String HybridSets store StringRef*, but
+ // ColumnValueRange<String>::CppType is
+ // std::string. Construct from the bytes.
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else if constexpr (std::is_same_v<CppType, StringRef>) {
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
Review Comment:
删了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4591610750
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4494113441
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/fe71e9b789993be3f389260e0a34e371580cef7c_fe71e9b789993be3f389260e0a34e371580cef7c_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/fe71e9b789993be3f389260e0a34e371580cef7c_fe71e9b789993be3f389260e0a34e371580cef7c_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1854/2375) |
| Line Coverage | 64.51% (33327/51662) |
| Region Coverage | 65.22% (16527/25341) |
| Branch Coverage | 55.72% (8833/15852) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4504349159
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4426081842
# BE Regression && UT Coverage Report
Increment line coverage `43.45% (305/702)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_b29520eb8c2fd176695ee6c263c89bab10446d4a_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_b29520eb8c2fd176695ee6c263c89bab10446d4a_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.71% (27826/37749) |
| Line Coverage | 57.52% (301121/523489) |
| Region Coverage | 54.51% (251038/460506) |
| Branch Coverage | 55.80% (108707/194808) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4429604601
<details>
<summary>TPC-H: <b>Total hot run time: 29564 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2391ea7f407dcc42f8f4ddc3bcd8e0838059d15c, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17646 3922 3937 3922
q2 q3 10581 917 616 616
q4 4668 460 354 354
q5 7467 1334 1131 1131
q6 196 181 146 146
q7 929 950 766 766
q8 9301 1439 1250 1250
q9 5563 5369 5363 5363
q10 6301 2060 1792 1792
q11 486 272 264 264
q12 637 420 299 299
q13 18094 3323 2717 2717
q14 307 281 269 269
q15 q16 911 877 792 792
q17 1181 1066 702 702
q18 6487 5704 5622 5622
q19 1614 1302 1084 1084
q20 538 410 268 268
q21 4875 2268 1903 1903
q22 429 359 304 304
Total cold run time: 98211 ms
Total hot run time: 29564 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4282 4183 4198 4183
q2 q3 4605 4733 4191 4191
q4 2102 2176 1404 1404
q5 4969 4981 5285 4981
q6 202 174 138 138
q7 2009 1912 1680 1680
q8 3479 3199 3139 3139
q9 8467 8369 8349 8349
q10 4475 4563 4259 4259
q11 598 432 424 424
q12 715 776 530 530
q13 3365 3607 2863 2863
q14 316 312 272 272
q15 q16 764 782 699 699
q17 1377 1370 1337 1337
q18 8320 7169 7087 7087
q19 1164 1158 1152 1152
q20 2233 2267 1978 1978
q21 6196 5441 4942 4942
q22 550 574 476 476
Total cold run time: 60188 ms
Total hot run time: 54084 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4344994621
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3322030867
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,857 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ DORIS_CHECK(tb.__isset.partition_id);
+ DORIS_CHECK(tb.__isset.slot_id);
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ DORIS_CHECK(slot_it != slot_descs.end());
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ DORIS_CHECK(is_list || is_range); \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
Review Comment:
Nullable RANGE partition columns are still unsafe here. `allow_partition_column_nullable` is enabled by default for RANGE partitions, and `NullLiteral.compareLiteral()` sorts NULL below every concrete value, so a `VALUES LESS THAN (...)` first partition can contain NULL rows. FE will still classify a direct partition `SlotRef` as prunable and serialize this boundary with no `range_start`; BE builds only the non-NULL CVR and leaves `ParsedBoundary::contains_null` false. For a null-safe join such as `f.k <=> d.k` with an IN runtime filter whose build side contains only NULL, `rf_contains_null` is true but the first RANGE partition reaches the regular CVR intersection path, intersects an empty non-NULL RF value set, and gets pruned even though its NULL rows should match. Please record nullable MINVALUE-side RANGE boundaries as containing NULL (or conservatively skip RF partition pruning for nullable RANGE partition columns) and add a regression case for a nullable RANGE partition plu
s `<=>` runtime filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3302353005
##########
be/test/exec/runtime_filter/runtime_filter_partition_pruner_test.cpp:
##########
@@ -0,0 +1,450 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gtest/gtest.h>
+
+#include <memory>
+#include <type_traits>
+#include <unordered_map>
+#include <utility>
+#include <vector>
+
+#include "core/data_type/data_type_factory.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exec/runtime_filter/utils.h"
+#include "exprs/create_predicate_function.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vdirect_in_predicate.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+class IdentityWrapperExpr final : public VExpr {
+public:
+ explicit IdentityWrapperExpr(DataTypePtr type) : VExpr(std::move(type), false) {}
+
+ const std::string& expr_name() const override { return _expr_name; }
+
+ Status execute_column_impl(VExprContext* context, const Block* block, const Selector* selector,
+ size_t count, ColumnPtr& result_column) const override {
+ return get_child(0)->execute_column(context, block, selector, count, result_column);
+ }
+
+private:
+ std::string _expr_name = "identity_wrapper";
+};
+
+class RuntimeFilterPartitionPrunerTest : public testing::Test {
+protected:
+ static constexpr SlotId SLOT_ID = 1;
+
+ template <PrimitiveType PT>
+ using CppType = typename PrimitiveTypeTraits<PT>::CppType;
+
+ template <PrimitiveType PT>
+ TExprNode literal_node(const CppType<PT>& value, int precision = 0, int scale = 0) {
+ TExprNode node;
+ EXPECT_TRUE(create_texpr_literal_node<PT>(&value, &node, precision, scale).ok());
+ return node;
+ }
+
+ TExprNode null_node(PrimitiveType ptype, int precision = 0, int scale = 0) {
+ TExprNode node;
+ node.__set_node_type(TExprNodeType::NULL_LITERAL);
+ node.__set_type(create_type_desc(ptype, precision, scale));
+ return node;
+ }
+
+ template <PrimitiveType PT>
+ TPartitionBoundary list_boundary(int64_t partition_id, const std::vector<TExprNode>& values) {
+ TPartitionBoundary boundary;
+ boundary.__set_partition_id(partition_id);
+ boundary.__set_slot_id(SLOT_ID);
+ boundary.__set_list_values(values);
+ return boundary;
+ }
+
+ template <PrimitiveType PT>
+ TPartitionBoundary range_boundary(int64_t partition_id, const CppType<PT>& start,
+ const CppType<PT>& end, int precision = 0, int scale = 0) {
+ TPartitionBoundary boundary;
+ boundary.__set_partition_id(partition_id);
+ boundary.__set_slot_id(SLOT_ID);
+ boundary.__set_range_start(literal_node<PT>(start, precision, scale));
+ boundary.__set_range_end(literal_node<PT>(end, precision, scale));
+ return boundary;
+ }
+
+ template <PrimitiveType PT>
+ TPartitionBoundary lower_unbounded_range_boundary(int64_t partition_id, const CppType<PT>& end,
+ int precision = 0, int scale = 0) {
+ TPartitionBoundary boundary;
+ boundary.__set_partition_id(partition_id);
+ boundary.__set_slot_id(SLOT_ID);
+ boundary.__set_range_end(literal_node<PT>(end, precision, scale));
+ return boundary;
+ }
+
+ template <PrimitiveType PT>
+ TPartitionBoundary upper_unbounded_range_boundary(int64_t partition_id,
+ const CppType<PT>& start, int precision = 0,
+ int scale = 0) {
+ TPartitionBoundary boundary;
+ boundary.__set_partition_id(partition_id);
+ boundary.__set_slot_id(SLOT_ID);
+ boundary.__set_range_start(literal_node<PT>(start, precision, scale));
+ return boundary;
+ }
+
+ SlotDescriptor slot_desc(PrimitiveType ptype, bool nullable, int precision = 0, int scale = 0) {
+ SlotDescriptor slot;
+ slot._id = SLOT_ID;
+ slot._type = DataTypeFactory::instance().create_data_type(
+ create_type_desc(ptype, precision, scale), nullable);
+ slot._col_name = "part_col";
+ return slot;
+ }
+
+ std::unique_ptr<ParsedPartitionBoundaries> parse_boundaries(
+ PrimitiveType ptype, const std::vector<TPartitionBoundary>& boundaries,
+ bool nullable = false, int precision = 0, int scale = 0) {
+ auto slot = slot_desc(ptype, nullable, precision, scale);
+ phmap::flat_hash_map<int, SlotDescriptor*> slots;
+ slots[SLOT_ID] = &slot;
+ auto parsed = std::make_unique<ParsedPartitionBoundaries>();
+ EXPECT_TRUE(parsed->parse(boundaries, slots).ok());
+ return parsed;
+ }
+
+ template <PrimitiveType PT>
+ std::shared_ptr<HybridSetBase> in_filter(const CppType<PT>& value, bool contain_null = false) {
+ std::shared_ptr<HybridSetBase> set(create_set(PT, contain_null));
+ if constexpr (std::is_same_v<CppType<PT>, String>) {
+ StringRef ref(value.data(), value.size());
+ set->insert(&ref);
+ } else {
+ set->insert(&value);
+ }
+ if (contain_null) {
+ set->insert(static_cast<const void*>(nullptr));
+ }
+ return set;
+ }
+
+ template <PrimitiveType PT>
+ VExprSPtr in_predicate(const CppType<PT>& value, bool contain_null = false) {
+ TExprNode node;
+ node.__set_type(create_type_desc(PrimitiveType::TYPE_BOOLEAN));
+ node.__set_node_type(contain_null ? TExprNodeType::NULL_AWARE_IN_PRED
+ : TExprNodeType::IN_PRED);
+ node.in_predicate.__set_is_not_in(false);
+ node.__set_opcode(TExprOpcode::FILTER_IN);
+ node.__set_is_nullable(false);
+ return VDirectInPredicate::create_shared(node, in_filter<PT>(value, contain_null));
+ }
+
+ template <PrimitiveType PT>
+ VExprSPtr minmax_predicate_le(const CppType<PT>& value, const DataTypePtr& type) {
+ VExprSPtr pred;
+ TExprNode node;
+ EXPECT_TRUE(create_vbin_predicate(type, TExprOpcode::LE, pred, &node, false).ok());
+ VExprSPtr lhs;
+ VExprSPtr rhs;
+ EXPECT_TRUE(create_literal(type, &value, lhs).ok());
+ EXPECT_TRUE(create_literal(type, &value, rhs).ok());
+ pred->add_child(lhs);
+ pred->add_child(rhs);
+ return pred;
+ }
+
+ VExprSPtr slot_ref_expr(const DataTypePtr& type) {
+ TSlotRef slot_ref_thrift;
+ slot_ref_thrift.__set_slot_id(SLOT_ID);
+ slot_ref_thrift.__set_tuple_id(0);
+ TExprNode node;
+ node.__set_type(create_type_desc(type->get_primitive_type(), type->get_precision(),
+ type->get_scale()));
+ node.__set_node_type(TExprNodeType::SLOT_REF);
+ node.__set_num_children(0);
+ node.__set_slot_ref(slot_ref_thrift);
+ auto slot_ref = VSlotRef::create_shared(node);
+ slot_ref->set_slot_id(SLOT_ID);
+ slot_ref->set_column_id(0);
+ return slot_ref;
+ }
+
+ VExprSPtr identity_wrapper_expr(const DataTypePtr& type) {
+ auto expr = std::make_shared<IdentityWrapperExpr>(type);
+ expr->add_child(slot_ref_expr(type));
+ return expr;
+ }
+
+ TRuntimeFilterDesc rf_desc_with_monotonicity(int filter_id, int scan_node_id,
+ const std::vector<int64_t>& partition_ids) {
+ TRuntimeFilterDesc desc;
+ desc.__set_filter_id(filter_id);
+ std::vector<TPartitionTargetExprMonotonicity> entries;
+ entries.reserve(partition_ids.size());
+ for (int64_t partition_id : partition_ids) {
+ TPartitionTargetExprMonotonicity entry;
+ entry.__set_partition_id(partition_id);
+ entry.__set_monotonicity(TTargetExprMonotonicity::MONOTONIC_INCREASING);
+ entries.emplace_back(std::move(entry));
+ }
+ desc.__set_planId_to_partition_target_monotonicity({{scan_node_id, std::move(entries)}});
+ return desc;
+ }
+
+ template <PrimitiveType PT>
+ void assert_parse_and_prune_type(const CppType<PT>& keep_value, const CppType<PT>& prune_value,
+ int precision = 0, int scale = 0) {
+ auto boundaries = std::vector<TPartitionBoundary> {
+ list_boundary<PT>(1, {literal_node<PT>(keep_value, precision, scale)}),
+ list_boundary<PT>(2, {literal_node<PT>(prune_value, precision, scale)})};
+ auto parsed = parse_boundaries(PT, boundaries, false, precision, scale);
+ ASSERT_FALSE(parsed->empty());
+ ASSERT_EQ(parsed->total_partitions(), 2);
+ const auto& parsed_boundaries = parsed->slot_to_boundaries().at(SLOT_ID);
+
+ RuntimeFilterPartitionPruner in_pruner;
+ phmap::flat_hash_set<int64_t> in_pruned;
+ in_pruner._try_prune_by_single_rf(parsed_boundaries, in_predicate<PT>(keep_value),
Review Comment:
This test calls `_try_prune_by_single_rf()` directly, but that method is declared `private` in `RuntimeFilterPartitionPruner` (`be/src/exec/runtime_filter/runtime_filter_partition_pruner.h`). The same private helper is also called later in this file, so `run-be-ut` should fail at compile time before these cases can execute. Please either exercise the public `prune_by_runtime_filters()` API here (as the later public-path test does), or explicitly expose/friend a test-only helper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4597972782
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4560666855
<details>
<summary>TPC-DS: <b>Total hot run time: 171927 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9af6867af36b4da0dec52406116ba2d158e2e2b9, data reload: false
query5 4344 632 488 488
query6 450 207 193 193
query7 4836 569 337 337
query8 372 222 207 207
query9 8817 4225 4171 4171
query10 472 321 264 264
query11 5933 2457 2260 2260
query12 157 102 104 102
query13 1263 585 447 447
query14 6427 5500 5170 5170
query14_1 4535 4472 4530 4472
query15 201 197 177 177
query16 1049 458 442 442
query17 1096 710 579 579
query18 2724 481 348 348
query19 208 180 142 142
query20 112 107 109 107
query21 218 140 118 118
query22 13713 13674 13398 13398
query23 17413 16538 16257 16257
query23_1 16420 16359 16311 16311
query24 7507 1795 1334 1334
query24_1 1341 1309 1347 1309
query25 555 453 402 402
query26 1306 314 163 163
query27 2657 579 336 336
query28 4474 1996 2025 1996
query29 1053 599 468 468
query30 302 243 197 197
query31 1120 1089 960 960
query32 103 58 60 58
query33 512 325 258 258
query34 1204 1166 650 650
query35 759 798 694 694
query36 1427 1431 1218 1218
query37 152 106 89 89
query38 3210 3163 3034 3034
query39 939 916 918 916
query39_1 882 876 877 876
query40 214 125 101 101
query41 70 63 64 63
query42 97 97 94 94
query43 327 333 287 287
query44
query45 203 189 183 183
query46 1130 1232 751 751
query47 2348 2413 2240 2240
query48 416 406 297 297
query49 633 471 367 367
query50 1031 353 250 250
query51 4360 4268 4178 4178
query52 89 91 78 78
query53 244 267 194 194
query54 274 234 199 199
query55 82 77 72 72
query56 235 234 241 234
query57 1460 1419 1337 1337
query58 252 224 213 213
query59 1632 1689 1475 1475
query60 286 271 254 254
query61 194 210 155 155
query62 704 674 601 601
query63 232 185 191 185
query64 2543 804 634 634
query65
query66 1758 485 349 349
query67 29896 29759 29568 29568
query68
query69 432 309 274 274
query70 967 927 928 927
query71 296 213 211 211
query72 3043 2718 2454 2454
query73 856 747 434 434
query74 5168 4970 4838 4838
query75 2704 2621 2266 2266
query76 2331 1194 794 794
query77 365 392 297 297
query78 12383 12565 11909 11909
query79 1229 1038 743 743
query80 555 510 427 427
query81 452 287 243 243
query82 241 161 129 129
query83 282 289 260 260
query84 265 148 123 123
query85 936 637 532 532
query86 338 301 301 301
query87 3405 3349 3202 3202
query88 3701 2750 2745 2745
query89 424 371 331 331
query90 2146 185 181 181
query91 180 170 137 137
query92 62 60 60 60
query93 1503 1496 896 896
query94 552 336 309 309
query95 718 382 361 361
query96 1029 757 356 356
query97 2692 2698 2583 2583
query98 221 209 209 209
query99 1217 1156 1040 1040
Total cold run time: 252077 ms
Total hot run time: 171927 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4563657682
<details>
<summary>TPC-H: <b>Total hot run time: 32179 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2ede6c8cafb5ff16b7121d7edac1018fc2e592e0, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17518 4158 4101 4101
q2 q3 10666 1484 852 852
q4 4679 482 350 350
q5 7621 2352 2169 2169
q6 252 180 144 144
q7 964 788 643 643
q8 9463 1713 1550 1550
q9 5361 5060 5058 5058
q10 6418 2199 1893 1893
q11 441 280 248 248
q12 633 435 298 298
q13 18108 3452 2790 2790
q14 270 257 242 242
q15 q16 828 781 710 710
q17 995 965 968 965
q18 7007 5693 5595 5595
q19 1173 1256 1260 1256
q20 537 452 295 295
q21 5869 2886 2701 2701
q22 470 382 319 319
Total cold run time: 99273 ms
Total hot run time: 32179 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5175 4985 4963 4963
q2 q3 4909 5351 4744 4744
q4 2192 2237 1440 1440
q5 5188 4884 4812 4812
q6 241 187 134 134
q7 2063 1866 1641 1641
q8 2652 2336 2321 2321
q9 8182 7540 7564 7540
q10 4763 4689 4238 4238
q11 596 420 390 390
q12 737 748 536 536
q13 2980 3355 2815 2815
q14 277 282 262 262
q15 q16 689 707 630 630
q17 1336 1296 1292 1292
q18 7644 7042 6997 6997
q19 1132 1127 1136 1127
q20 2255 2234 1972 1972
q21 5422 4709 4624 4624
q22 529 461 441 441
Total cold run time: 58962 ms
Total hot run time: 52919 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4561189550
# BE Regression && UT Coverage Report
Increment line coverage `78.52% (658/838)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_9af6867af36b4da0dec52406116ba2d158e2e2b9_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_9af6867af36b4da0dec52406116ba2d158e2e2b9_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.86% (28128/38084) |
| Line Coverage | 57.80% (305957/529292) |
| Region Coverage | 54.72% (256664/469064) |
| Branch Coverage | 56.15% (111251/198137) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4561206733
# FE Regression Coverage Report
Increment line coverage ` 59.10% (224/379)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_9af6867af36b4da0dec52406116ba2d158e2e2b9_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_9af6867af36b4da0dec52406116ba2d158e2e2b9_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4571144792
# FE UT Coverage Report
Increment line coverage ` 31.37% (96/306)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_fb1122e0a26c0dc4fc1defe6bfb9b6b5da020008/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_fb1122e0a26c0dc4fc1defe6bfb9b6b5da020008/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4440179533
# BE Regression && UT Coverage Report
Increment line coverage `43.30% (304/702)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_2337ac55b6530b92cabafe14881c96951fab95c9_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_2337ac55b6530b92cabafe14881c96951fab95c9_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.69% (27793/37714) |
| Line Coverage | 57.59% (301348/523263) |
| Region Coverage | 54.74% (252002/460385) |
| Branch Coverage | 55.91% (108906/194791) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4439057855
<details>
<summary>TPC-H: <b>Total hot run time: 29785 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2337ac55b6530b92cabafe14881c96951fab95c9, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17641 4152 3973 3973
q2 q3 10714 888 607 607
q4 4666 461 346 346
q5 7452 1360 1142 1142
q6 191 169 142 142
q7 917 973 761 761
q8 9315 1391 1318 1318
q9 5603 5398 5342 5342
q10 6269 2100 1818 1818
q11 481 267 262 262
q12 626 422 292 292
q13 18065 3323 2760 2760
q14 294 288 263 263
q15 q16 871 852 783 783
q17 1026 1137 736 736
q18 6392 5738 5543 5543
q19 1159 1301 1129 1129
q20 519 398 273 273
q21 4090 2409 1968 1968
q22 492 428 327 327
Total cold run time: 96783 ms
Total hot run time: 29785 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4814 4758 4988 4758
q2 q3 4684 4816 4225 4225
q4 2157 2220 1422 1422
q5 4975 5069 5258 5069
q6 199 171 141 141
q7 2086 1781 1674 1674
q8 3354 3105 3072 3072
q9 8354 8391 8407 8391
q10 4505 4470 4254 4254
q11 612 426 380 380
q12 685 757 516 516
q13 3312 3692 2895 2895
q14 298 302 276 276
q15 q16 767 951 691 691
q17 1359 1298 1295 1295
q18 7931 7161 7199 7161
q19 1184 1167 1183 1167
q20 2300 2214 1923 1923
q21 6160 5399 4822 4822
q22 551 507 422 422
Total cold run time: 60287 ms
Total hot run time: 54554 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4440827738
<details>
<summary>TPC-H: <b>Total hot run time: 29532 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ca30c183fbab4ffab1f11b3d91b0561736c7c199, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17463 4060 3989 3989
q2 q3 10553 874 614 614
q4 4655 457 361 361
q5 7454 1320 1128 1128
q6 193 169 145 145
q7 928 958 758 758
q8 9331 1407 1277 1277
q9 5588 5379 5297 5297
q10 6257 2073 1798 1798
q11 475 267 257 257
q12 628 423 297 297
q13 18117 3344 2721 2721
q14 299 279 262 262
q15 q16 909 862 788 788
q17 1005 1024 786 786
q18 6509 5691 5571 5571
q19 1164 1243 1035 1035
q20 534 396 265 265
q21 4437 2266 1878 1878
q22 422 355 305 305
Total cold run time: 96921 ms
Total hot run time: 29532 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4187 4110 4115 4110
q2 q3 4633 4750 4161 4161
q4 2105 2178 1391 1391
q5 4977 4961 5306 4961
q6 185 163 130 130
q7 2020 1773 1727 1727
q8 3564 3193 3206 3193
q9 8493 8452 8378 8378
q10 4443 4458 4234 4234
q11 612 440 406 406
q12 697 741 549 549
q13 3213 3699 2935 2935
q14 326 308 274 274
q15 q16 755 787 702 702
q17 1372 1327 1325 1325
q18 8064 7274 7070 7070
q19 1139 1147 1173 1147
q20 2268 2220 1978 1978
q21 6158 5371 4853 4853
q22 556 525 433 433
Total cold run time: 59767 ms
Total hot run time: 53957 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4449156322
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3099229731
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,293 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// Helper to extract a typed value from a ColumnPtr's first row.
+// Must be a template function so that `if constexpr` properly discards
+// the inapplicable branch during instantiation.
+template <PrimitiveType PT>
+static typename PrimitiveTypeTraits<PT>::CppType extract_value_from_column(const ColumnPtr& col) {
+ using CppType = typename PrimitiveTypeTraits<PT>::CppType;
+ auto data = col->get_data_at(0);
+ if constexpr (is_string_type(PT)) {
+ return CppType(data.data, data.size);
+ } else {
+ return *reinterpret_cast<const CppType*>(data.data);
+ }
+}
+
+void RuntimeFilterPartitionPruner::parse_boundaries(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ if (!tb.__isset.partition_id || !tb.__isset.slot_id) {
+ continue;
+ }
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ if (slot_it == slot_descs.end()) {
+ continue;
+ }
+ SlotDescriptor* slot = slot_it->second;
+ PrimitiveType ptype = slot->type()->get_primitive_type();
+ int precision = cast_set<int>(slot->type()->get_precision());
+ int scale = cast_set<int>(slot->type()->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ if (!is_list && !is_range) break; \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::pair<CppType, ColumnPtr> { \
+ VLiteral literal(node); \
+ auto col_ptr = literal.get_column_ptr(); \
+ auto val = extract_value_from_column<TYPE_##NAME>(col_ptr); \
+ return {val, col_ptr}; \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ for (const auto& node : tb.list_values) { \
+ auto [val, col_ptr] = parse_texpr_node(node); \
+ boundary.literal_columns.push_back(std::move(col_ptr)); \
+ static_cast<void>(empty_cvr.add_fixed_value(val)); \
+ } \
+ cvr.intersection(empty_cvr); \
+ } else { \
+ if (tb.__isset.range_start) { \
+ auto [val, col_ptr] = parse_texpr_node(tb.range_start); \
+ boundary.literal_columns.push_back(std::move(col_ptr)); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, val)); \
+ } \
+ if (tb.__isset.range_end) { \
+ auto [val, col_ptr] = parse_texpr_node(tb.range_end); \
+ boundary.literal_columns.push_back(std::move(col_ptr)); \
+ static_cast<void>(cvr.add_range(FILTER_LESS, val)); \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ if (parsed_ok) {
+ _partition_column_slot_ids.insert(slot_id);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_partition_column_slot_ids.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+}
+
+int64_t RuntimeFilterPartitionPruner::prune_by_runtime_filters(
+ const VExprContextSPtrs& conjuncts) {
+ if (_partition_column_slot_ids.empty()) {
+ return 0;
+ }
+
+ phmap::flat_hash_set<int64_t> newly_pruned;
+
+ for (const auto& conjunct_ctx : conjuncts) {
+ VExprSPtr root = conjunct_ctx->root();
+ if (!root->is_rf_wrapper()) {
+ continue;
+ }
+
+ VExprSPtr impl = root->get_impl();
+ if (!impl) {
+ continue;
+ }
+
+ // Determine the target slot_id from the RF's probe expression.
+ // For IN filter: impl is VDirectInPredicate, children()[0] is VSlotRef
+ // For MinMax filter: impl is VectorizedFnCall (binary pred), children()[0] is VSlotRef
+ if (impl->children().empty() || !impl->children()[0]->is_slot_ref()) {
+ continue;
+ }
+ auto* slot_ref = assert_cast<VSlotRef*>(impl->children()[0].get());
+ SlotId slot_id = slot_ref->slot_id();
+ if (!_partition_column_slot_ids.contains(slot_id)) {
+ continue;
+ }
+
+ auto boundaries_it = _slot_to_boundaries.find(slot_id);
+ if (boundaries_it == _slot_to_boundaries.end()) {
+ continue;
+ }
+
+ for (const auto& pb : boundaries_it->second) {
+ if (_pruned_partition_ids.contains(pb.partition_id) ||
+ newly_pruned.contains(pb.partition_id)) {
+ continue;
+ }
Review Comment:
`prune_by_runtime_filters()` reads `_pruned_partition_ids` (e.g. `.contains()`) without holding `_prune_mutex`, but the same set can be written under the mutex at the end of the method and read concurrently by `is_partition_pruned()`/`pruned_partition_count()`. This is a data race because `phmap::flat_hash_set` is not thread-safe for concurrent read/write. Please guard *all* accesses to `_pruned_partition_ids` with the mutex (e.g. take a `shared_lock` for read checks or a `unique_lock` for the whole pruning pass).
##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -1386,6 +1418,18 @@ enum TMinMaxRuntimeFilterType {
MIN_MAX = 4
}
+// Monotonicity of a runtime filter's target expression, used by BE-side
+// partition pruning. For Range partitions, only MONOTONIC_INCREASING or
+// MONOTONIC_DECREASING target expressions allow safe boundary transformation
+// and pruning. List partitions can always be pruned regardless of monotonicity.
+// When the target expression is a plain SlotRef (identity), FE may omit this
+// field; BE treats an absent value as NON_MONOTONIC (conservative).
Review Comment:
The comments about `target_expr_monotonicity` conflict: the enum comment says FE may omit the field for a SlotRef and BE treats it as NON_MONOTONIC, but the `TRuntimeFilterDesc` comment later says FE should set MONOTONIC_INCREASING for a SlotRef. Please reconcile these docstrings so they reflect the intended behavior (and don’t encourage omitting a field that disables pruning).
##########
fe/fe-core/src/main/java/org/apache/doris/planner/RuntimeFilter.java:
##########
@@ -280,6 +285,36 @@ public TRuntimeFilterDesc toThrift() {
} else if (builderNode instanceof SetOperationNode) {
tFilter.setNullAware(true);
}
+
+ // Set target_expr_monotonicity for partition column identity path.
+ // When the RF target is a direct SlotRef on a partition column, the
+ // mapping from column value to partition is monotonic (identity), so
+ // the BE can use the RF values to prune partitions.
+ for (RuntimeFilterTarget target : targets) {
+ if (target.expr instanceof SlotRef && target.node instanceof OlapScanNode) {
+ SlotRef slotRef = (SlotRef) target.expr;
+ Column col = slotRef.getColumn();
+ if (col != null) {
+ OlapScanNode scanNode = (OlapScanNode) target.node;
+ OlapTable table = scanNode.getOlapTable();
+ PartitionType partType = table.getPartitionInfo().getType();
+ if (partType == PartitionType.RANGE || partType == PartitionType.LIST) {
+ for (Column partCol : table.getPartitionInfo().getPartitionColumns()) {
+ if (partCol.getName().equalsIgnoreCase(col.getName())) {
+ tFilter.setTargetExprMonotonicity(
+ TTargetExprMonotonicity.MONOTONIC_INCREASING);
+ break;
+ }
+ }
+ }
+ }
+ }
+ // Only check the first target; if there are multiple targets on
+ // different tables, monotonicity may differ. We conservatively
+ // set it based on the first target only.
+ break;
+ }
Review Comment:
The loop that sets `target_expr_monotonicity` always `break`s after the first target, even when the first target is not an `OlapScanNode` / partition-column `SlotRef`. That means the field may stay unset even if a later target is eligible, unnecessarily disabling partition pruning. Consider scanning targets until an eligible partition-column SlotRef is found (or explicitly selecting the intended target deterministically) before deciding to stop.
##########
regression-test/conf/regression-conf.groovy:
##########
@@ -24,8 +24,8 @@ defaultDb = "regression_test"
// init cmd like: select @@session.tx_read_only
// at each time we connect.
// add allowLoadLocalInfile so that the jdbc can execute mysql load data from client.
-jdbcUrl = "jdbc:mysql://127.0.0.1:9030/?useLocalSessionState=true&allowLoadLocalInfile=true&zeroDateTimeBehavior=round"
-targetJdbcUrl = "jdbc:mysql://127.0.0.1:9030/?useLocalSessionState=true&allowLoadLocalInfile=true&zeroDateTimeBehavior=round"
+jdbcUrl = "jdbc:mysql://127.0.0.1:9333/?useLocalSessionState=true&allowLoadLocalInfile=true&zeroDateTimeBehavior=round"
+targetJdbcUrl = "jdbc:mysql://127.0.0.1:9333/?useLocalSessionState=true&allowLoadLocalInfile=true&zeroDateTimeBehavior=round"
Review Comment:
`jdbcUrl` / `targetJdbcUrl` were changed from 9030 to 9333, but other related settings in this same config (e.g. `ccrDownstreamUrl`, `doris_port`, and `ccrDownstreamFeThriftAddress`) still use 9030. This makes the regression test config internally inconsistent and will likely break CI environments that expect default ports. Please revert these port changes or make them configurable (e.g. via env/property) while keeping repo defaults consistent.
##########
gensrc/thrift/PlanNodes.thrift:
##########
@@ -913,6 +938,13 @@ struct TOlapScanNode {
24: optional bool enable_mor_value_predicate_pushdown
// Read MOR table as DUP table: skip merge, skip delete sign
25: optional bool read_mor_as_dup
+ // Partition-id → tablet-id list mapping; used together with partition_boundaries
+ // for BE-side runtime filter partition pruning.
+ 26: optional map<Types.TPartitionId, list<Types.TTabletId>> partiton_to_tablets
+ // Partition boundary descriptors for BE-side runtime filter partition pruning.
+ // Only partitions that are candidates for pruning are included; partitions FE
+ // does not want pruned (e.g. default catch-all) are omitted from this list.
+ 27: optional list<TPartitionBoundary> partition_boundaries
Review Comment:
The field name `partiton_to_tablets` appears to be a typo (missing 'i' in 'partition'). Even though Thrift field names don't affect wire compatibility, this typo becomes part of generated code APIs and will be hard to undo later. Please rename it to `partition_to_tablets` before merging (keeping field id 26 unchanged).
##########
regression-test/conf/regression-conf.groovy:
##########
@@ -35,7 +35,7 @@ syncerAddress = "127.0.0.1:9190"
feSyncerUser = "root"
feSyncerPassword = ""
-feHttpAddress = "127.0.0.1:8030"
+feHttpAddress = "127.0.0.1:8333"
Review Comment:
`feHttpAddress` was changed from 8030 to 8333 while other regression-test defaults remain on standard ports. This will cause profile-related and HTTP-based regression tests to fail unless the environment is customized. Please keep the committed default `feHttpAddress` aligned with the standard FE HTTP port (or make it configurable without changing the default).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4277390461
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4277647155
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3108286524
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,374 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <utility>
+
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// Helper to extract a typed value from a ColumnPtr's first row.
+// Must be a template function so that `if constexpr` properly discards
+// the inapplicable branch during instantiation.
+template <PrimitiveType PT>
+static typename PrimitiveTypeTraits<PT>::CppType extract_value_from_column(const ColumnPtr& col) {
+ using CppType = typename PrimitiveTypeTraits<PT>::CppType;
+ auto data = col->get_data_at(0);
+ if constexpr (is_string_type(PT)) {
+ return CppType(data.data, data.size);
+ } else {
+ return *reinterpret_cast<const CppType*>(data.data);
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+void RuntimeFilterPartitionPruner::parse_boundaries(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ if (!tb.__isset.partition_id || !tb.__isset.slot_id) {
+ continue;
+ }
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ if (slot_it == slot_descs.end()) {
+ continue;
+ }
+ SlotDescriptor* slot = slot_it->second;
+ PrimitiveType ptype = slot->type()->get_primitive_type();
+ int precision = cast_set<int>(slot->type()->get_precision());
+ int scale = cast_set<int>(slot->type()->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ if (!is_list && !is_range) break; \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = \
+ [&](const TExprNode& node) -> std::optional<std::pair<CppType, ColumnPtr>> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ VLiteral literal(node); \
+ auto col_ptr = literal.get_column_ptr(); \
+ auto val = extract_value_from_column<TYPE_##NAME>(col_ptr); \
+ return std::make_optional(std::pair<CppType, ColumnPtr> {val, col_ptr}); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ auto& [val, col_ptr] = *parsed; \
+ boundary.literal_columns.push_back(std::move(col_ptr)); \
+ static_cast<void>(empty_cvr.add_fixed_value(val)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ auto& [val, col_ptr] = *parsed; \
+ boundary.literal_columns.push_back(std::move(col_ptr)); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, val)); \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ auto& [val, col_ptr] = *parsed; \
+ boundary.literal_columns.push_back(std::move(col_ptr)); \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, val)); \
+ } \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ if (parsed_ok) {
+ _partition_column_slot_ids.insert(slot_id);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_partition_column_slot_ids.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static SQLFilterOp convert_opcode_to_filter_op(TExprOpcode::type op) {
+ switch (op) {
+ case TExprOpcode::LE:
+ return FILTER_LESS_OR_EQUAL;
+ case TExprOpcode::LT:
+ return FILTER_LESS;
+ case TExprOpcode::GE:
+ return FILTER_LARGER_OR_EQUAL;
+ case TExprOpcode::GT:
+ return FILTER_LARGER;
+ default:
+ return FILTER_IN; // sentinel: caller should skip
+ }
+}
+
+void RuntimeFilterPartitionPruner::_try_prune_by_single_rf(
+ const VExprSPtr& impl, SlotId slot_id, phmap::flat_hash_set<int64_t>& newly_pruned) {
+ auto boundaries_it = _slot_to_boundaries.find(slot_id);
+ if (boundaries_it == _slot_to_boundaries.end()) {
+ return;
+ }
+
+ // Pre-compute whether the RF "matches NULL" -- i.e. whether the RF would
+ // accept a row whose probe value is NULL. This is encoded by the set's
+ // contain_null() (see FilterBase::contain_null = _null_aware && _contain_null).
+ // For BINARY_PRED predicates (=, <, >, <=, >=) NULL never compares true,
+ // so they never match a NULL probe row -- treat as not containing NULL.
+ auto hybrid_set_for_null = impl->get_set_func();
Review Comment:
`rf_contains_null` is only derived from `get_set_func()`, which means it works for direct-IN filters but stays false for the other null-aware RF implementations. In particular, `BLOOM_PRED` carries its NULL bit on `get_bloom_filter_func()->contain_null()`, and null-aware min/max filters arrive as `NULL_AWARE_BINARY_PRED` rather than `BINARY_PRED`. With the current logic, a LIST partition like `VALUES IN ((NULL))` is pruned by the `pb.only_null` branch for a null-safe `<=>` join as soon as the RF is realized as bloom or null-aware min/max, even though NULL probe rows should match and the partition must be kept.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4288723552
# BE UT Coverage Report
Increment line coverage `0.91% (6/658)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/6f985006ec32cea727f90e22b3cd688c1a0626e1_6f985006ec32cea727f90e22b3cd688c1a0626e1/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/6f985006ec32cea727f90e22b3cd688c1a0626e1_6f985006ec32cea727f90e22b3cd688c1a0626e1/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.26% (20358/38221) |
| Line Coverage | 36.80% (191822/521235) |
| Region Coverage | 32.99% (149152/452110) |
| Branch Coverage | 33.93% (65243/192281) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4589224686
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4589485613
# FE UT Coverage Report
Increment line coverage ` 31.37% (96/306)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_586640a94f47548a444d49ee53d5474cf04e3ae0/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_586640a94f47548a444d49ee53d5474cf04e3ae0/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4593403730
# BE Regression && UT Coverage Report
Increment line coverage `85.48% (677/792)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_b82194ce0fb5ac216aeed034490645f04c19f4e5_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_b82194ce0fb5ac216aeed034490645f04c19f4e5_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.89% (28188/38147) |
| Line Coverage | 57.90% (306849/529922) |
| Region Coverage | 54.90% (258055/470057) |
| Branch Coverage | 56.21% (111568/198490) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4591829542
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/b82194ce0fb5ac216aeed034490645f04c19f4e5_b82194ce0fb5ac216aeed034490645f04c19f4e5_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/b82194ce0fb5ac216aeed034490645f04c19f4e5_b82194ce0fb5ac216aeed034490645f04c19f4e5_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.34% (1906/2433) |
| Line Coverage | 64.77% (33982/52468) |
| Region Coverage | 65.28% (17516/26833) |
| Branch Coverage | 53.98% (9300/17230) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4590327122
# BE Regression && UT Coverage Report
Increment line coverage `85.48% (677/792)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_586640a94f47548a444d49ee53d5474cf04e3ae0_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_586640a94f47548a444d49ee53d5474cf04e3ae0_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.74% (28127/38142) |
| Line Coverage | 57.74% (305911/529827) |
| Region Coverage | 54.55% (256290/469837) |
| Branch Coverage | 56.01% (111148/198431) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3331436714
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,866 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ DORIS_CHECK(tb.__isset.partition_id);
+ DORIS_CHECK(tb.__isset.slot_id);
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ DORIS_CHECK(slot_it != slot_descs.end());
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ DORIS_CHECK(is_list || is_range); \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ bool range_has_null = false; \
+ if (is_nullable && !tb.__isset.range_start) { \
+ range_has_null = true; \
+ } \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } else if (is_nullable) { \
+ range_has_null = true; \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ if (range_has_null) { \
+ boundary.contains_null = true; \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ DORIS_CHECK(parsed_ok);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static bool find_unique_slot_ref_impl(const VExpr* expr, const VSlotRef** found) {
+ if (!expr) {
Review Comment:
改成check了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4589177856
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4589959994
# BE UT Coverage Report
Increment line coverage `71.09% (563/792)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/586640a94f47548a444d49ee53d5474cf04e3ae0_586640a94f47548a444d49ee53d5474cf04e3ae0/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/586640a94f47548a444d49ee53d5474cf04e3ae0_586640a94f47548a444d49ee53d5474cf04e3ae0/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 54.05% (21051/38950) |
| Line Coverage | 37.65% (199993/531209) |
| Region Coverage | 33.80% (157295/465432) |
| Branch Coverage | 34.80% (68799/197712) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4440924332
<details>
<summary>TPC-DS: <b>Total hot run time: 169090 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ca30c183fbab4ffab1f11b3d91b0561736c7c199, data reload: false
query5 4299 626 472 472
query6 427 203 179 179
query7 4819 572 281 281
query8 361 230 202 202
query9 8799 4043 4048 4043
query10 459 317 260 260
query11 5963 2328 2214 2214
query12 151 102 100 100
query13 1253 612 430 430
query14 7265 5302 4977 4977
query14_1 4288 4278 4299 4278
query15 211 200 175 175
query16 1027 469 474 469
query17 1312 765 638 638
query18 2725 499 363 363
query19 319 191 152 152
query20 114 107 113 107
query21 217 140 130 130
query22 13680 13991 14408 13991
query23 17372 16423 16078 16078
query23_1 16207 16199 16218 16199
query24 7723 1803 1411 1411
query24_1 1429 1360 1392 1360
query25 564 476 394 394
query26 1265 299 158 158
query27 2608 565 352 352
query28 4319 1946 1946 1946
query29 1080 611 521 521
query30 304 236 193 193
query31 1104 1050 936 936
query32 105 57 54 54
query33 504 320 255 255
query34 1131 1102 640 640
query35 735 755 654 654
query36 1282 1282 1110 1110
query37 144 104 88 88
query38 3176 3085 3041 3041
query39 939 916 874 874
query39_1 876 870 858 858
query40 236 127 112 112
query41 63 59 59 59
query42 90 90 90 90
query43 307 314 262 262
query44
query45 198 183 175 175
query46 1057 1218 707 707
query47 2309 2204 2170 2170
query48 406 419 295 295
query49 624 504 391 391
query50 700 278 216 216
query51 4240 4225 4216 4216
query52 85 85 74 74
query53 228 272 184 184
query54 254 214 195 195
query55 76 73 67 67
query56 236 220 220 220
query57 1380 1366 1296 1296
query58 237 216 213 213
query59 1511 1576 1387 1387
query60 317 255 243 243
query61 160 160 156 156
query62 671 616 565 565
query63 229 182 190 182
query64 2498 787 660 660
query65
query66 1750 482 353 353
query67 30151 29910 29223 29223
query68
query69 412 297 275 275
query70 993 985 945 945
query71 306 232 218 218
query72 3149 2899 2620 2620
query73 868 788 448 448
query74 5071 4893 4712 4712
query75 2749 2628 2291 2291
query76 2260 1132 759 759
query77 354 381 281 281
query78 12868 12904 12261 12261
query79 1432 944 747 747
query80 1343 493 403 403
query81 522 275 243 243
query82 619 161 123 123
query83 318 270 244 244
query84 256 140 111 111
query85 913 475 416 416
query86 418 293 275 275
query87 3413 3286 3128 3128
query88 3568 2685 2650 2650
query89 415 367 325 325
query90 1929 173 179 173
query91 179 171 137 137
query92 63 60 57 57
query93 1137 935 550 550
query94 719 321 295 295
query95 667 368 349 349
query96 1010 757 333 333
query97 2652 2678 2532 2532
query98 214 208 207 207
query99 1106 1101 970 970
Total cold run time: 252917 ms
Total hot run time: 169090 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4448034358
# FE UT Coverage Report
Increment line coverage ` 28.38% (84/296)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_5bfeb4d44af5550f9edd89c84a3d05b18135d530/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_5bfeb4d44af5550f9edd89c84a3d05b18135d530/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4450130663
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/c090ab6440346db9f2c052585ec7b6dc593d0df3_c090ab6440346db9f2c052585ec7b6dc593d0df3_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/c090ab6440346db9f2c052585ec7b6dc593d0df3_c090ab6440346db9f2c052585ec7b6dc593d0df3_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.03% (1850/2371) |
| Line Coverage | 64.69% (33224/51362) |
| Region Coverage | 65.24% (16449/25214) |
| Branch Coverage | 55.72% (8780/15758) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4474503045
<details>
<summary>TPC-H: <b>Total hot run time: 31285 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ef2529ca3c35ebe1f89cb9fa4212f239fce43a97, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17680 3996 3868 3868
q2 q3 10949 1453 859 859
q4 4839 475 349 349
q5 10672 2271 2116 2116
q6 398 180 139 139
q7 999 767 640 640
q8 9586 1698 1684 1684
q9 6893 4962 4942 4942
q10 6388 2125 1840 1840
q11 429 280 245 245
q12 645 423 293 293
q13 18123 3390 2754 2754
q14 267 247 233 233
q15 q16 820 775 709 709
q17 1009 1029 1004 1004
q18 6904 5727 5619 5619
q19 1185 1229 1094 1094
q20 517 416 280 280
q21 5543 2686 2313 2313
q22 454 373 304 304
Total cold run time: 104300 ms
Total hot run time: 31285 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4210 4140 4228 4140
q2 q3 4512 4910 4304 4304
q4 2105 2224 1406 1406
q5 4445 4331 5112 4331
q6 253 201 148 148
q7 1957 1782 1556 1556
q8 2509 2223 2231 2223
q9 7718 7714 7789 7714
q10 4792 4739 4090 4090
q11 567 433 382 382
q12 772 731 518 518
q13 3322 3685 3067 3067
q14 294 300 293 293
q15 q16 735 726 655 655
q17 1375 1400 1328 1328
q18 8077 7109 6764 6764
q19 1177 1085 1087 1085
q20 2235 2242 1947 1947
q21 5339 4636 4500 4500
q22 538 470 424 424
Total cold run time: 56932 ms
Total hot run time: 50875 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4450215622
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4268412666
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4277606248
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4563844786
# BE UT Coverage Report
Increment line coverage `71.69% (552/770)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/2ede6c8cafb5ff16b7121d7edac1018fc2e592e0_2ede6c8cafb5ff16b7121d7edac1018fc2e592e0/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/2ede6c8cafb5ff16b7121d7edac1018fc2e592e0_2ede6c8cafb5ff16b7121d7edac1018fc2e592e0/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 54.02% (21007/38884) |
| Line Coverage | 37.66% (199814/530627) |
| Region Coverage | 33.84% (157312/464877) |
| Branch Coverage | 34.83% (68805/197522) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3330634932
##########
be/src/exec/scan/scanner_scheduler.cpp:
##########
@@ -202,6 +202,9 @@ void ScannerScheduler::_scanner_scan(std::shared_ptr<ScannerContext> ctx,
<< rf_status.to_string();
}
+ // After processing late RFs, check if this scanner's partition was pruned.
+ if (!eos && scanner->check_partition_pruned()) { eos = true; }
Review Comment:
这个是不是应该先于199行被调用呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4494083348
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4540604665
# FE Regression Coverage Report
Increment line coverage ` 67.33% (204/303)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_f779760a4db0a4e1f4605c63a8a9bc21f51af126_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_f779760a4db0a4e1f4605c63a8a9bc21f51af126_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4540917267
<details>
<summary>TPC-DS: <b>Total hot run time: 170883 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f779760a4db0a4e1f4605c63a8a9bc21f51af126, data reload: false
query5 4323 656 502 502
query6 448 200 189 189
query7 4864 570 312 312
query8 386 211 206 206
query9 8828 4168 4147 4147
query10 456 311 260 260
query11 5937 2558 2250 2250
query12 174 108 106 106
query13 1271 624 457 457
query14 6559 5565 5232 5232
query14_1 4533 4504 4503 4503
query15 207 201 177 177
query16 1051 483 414 414
query17 1138 734 594 594
query18 2727 479 354 354
query19 221 189 153 153
query20 120 114 108 108
query21 214 140 125 125
query22 13685 13713 13402 13402
query23 17360 16682 16228 16228
query23_1 16465 16334 16474 16334
query24 7554 1787 1338 1338
query24_1 1382 1331 1356 1331
query25 591 472 415 415
query26 1325 316 173 173
query27 2621 559 350 350
query28 4451 2010 2036 2010
query29 1117 639 508 508
query30 310 238 209 209
query31 1129 1091 980 980
query32 110 67 61 61
query33 550 342 266 266
query34 1199 1170 660 660
query35 800 781 660 660
query36 1413 1427 1235 1235
query37 151 100 89 89
query38 3229 3164 3087 3087
query39 927 962 892 892
query39_1 874 884 877 877
query40 230 122 99 99
query41 65 63 63 63
query42 94 92 94 92
query43 319 326 286 286
query44
query45 197 190 175 175
query46 1112 1203 755 755
query47 2439 2385 2231 2231
query48 399 413 305 305
query49 620 476 348 348
query50 1042 367 254 254
query51 4344 4320 4223 4223
query52 89 88 79 79
query53 242 275 193 193
query54 291 223 195 195
query55 78 74 69 69
query56 224 228 258 228
query57 1432 1415 1339 1339
query58 242 219 216 216
query59 1641 1719 1534 1534
query60 291 252 225 225
query61 161 157 159 157
query62 688 652 591 591
query63 233 187 194 187
query64 2495 786 610 610
query65
query66 1750 478 337 337
query67 29882 29864 29727 29727
query68
query69 440 308 271 271
query70 986 942 942 942
query71 299 219 211 211
query72 2939 2778 2295 2295
query73 882 748 442 442
query74 5133 4974 4790 4790
query75 2694 2589 2259 2259
query76 2344 1153 759 759
query77 343 363 286 286
query78 12603 12527 11842 11842
query79 1462 1080 738 738
query80 1298 468 387 387
query81 522 282 243 243
query82 638 166 120 120
query83 349 276 246 246
query84 273 144 117 117
query85 945 546 447 447
query86 424 309 294 294
query87 3403 3380 3261 3261
query88 3657 2770 2757 2757
query89 433 382 338 338
query90 1958 187 185 185
query91 182 168 141 141
query92 62 65 55 55
query93 1558 1556 799 799
query94 735 334 296 296
query95 682 466 353 353
query96 1028 760 340 340
query97 2721 2724 2632 2632
query98 219 211 220 211
query99 1187 1147 1034 1034
Total cold run time: 254074 ms
Total hot run time: 170883 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4422754922
<details>
<summary>TPC-DS: <b>Total hot run time: 170147 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b29520eb8c2fd176695ee6c263c89bab10446d4a, data reload: false
query5 4342 648 493 493
query6 450 198 184 184
query7 4906 607 300 300
query8 366 221 204 204
query9 8797 4064 4088 4064
query10 449 331 270 270
query11 6135 2414 2240 2240
query12 167 111 103 103
query13 1269 605 421 421
query14 7355 5364 5051 5051
query14_1 4327 4321 4332 4321
query15 209 208 188 188
query16 1017 475 457 457
query17 1295 784 624 624
query18 2716 515 368 368
query19 317 196 149 149
query20 115 108 106 106
query21 220 146 124 124
query22 13837 14051 14629 14051
query23 17507 16536 16107 16107
query23_1 16289 16185 16299 16185
query24 7730 1892 1396 1396
query24_1 1383 1337 1363 1337
query25 557 466 393 393
query26 1299 315 162 162
query27 2550 630 317 317
query28 4306 1988 1974 1974
query29 1057 605 491 491
query30 307 236 195 195
query31 1111 1062 951 951
query32 108 58 62 58
query33 509 317 255 255
query34 1188 1147 636 636
query35 762 770 658 658
query36 1274 1300 1099 1099
query37 150 101 85 85
query38 3188 3123 3017 3017
query39 931 912 894 894
query39_1 901 865 896 865
query40 233 131 107 107
query41 64 62 59 59
query42 93 93 92 92
query43 316 331 273 273
query44
query45 195 184 171 171
query46 1113 1213 709 709
query47 2304 2276 2230 2230
query48 385 405 291 291
query49 634 511 394 394
query50 733 303 223 223
query51 4251 4240 4429 4240
query52 87 90 75 75
query53 245 280 192 192
query54 264 228 196 196
query55 78 72 70 70
query56 239 237 235 235
query57 1402 1393 1290 1290
query58 237 213 209 209
query59 1566 1606 1426 1426
query60 294 266 237 237
query61 160 170 156 156
query62 670 610 555 555
query63 229 189 200 189
query64 2485 799 667 667
query65
query66 1744 512 355 355
query67 30046 29993 29814 29814
query68
query69 428 334 288 288
query70 993 929 941 929
query71 306 230 213 213
query72 2985 2735 2431 2431
query73 829 793 414 414
query74 5122 4958 4730 4730
query75 2881 2648 2308 2308
query76 2304 1216 784 784
query77 384 400 294 294
query78 12887 12912 12464 12464
query79 1461 1016 752 752
query80 628 528 405 405
query81 450 286 246 246
query82 583 166 123 123
query83 363 279 256 256
query84 262 139 115 115
query85 853 544 508 508
query86 390 300 291 291
query87 3374 3335 3166 3166
query88 3618 2704 2669 2669
query89 435 377 329 329
query90 1860 194 185 185
query91 196 189 157 157
query92 66 66 60 60
query93 993 976 565 565
query94 567 379 337 337
query95 676 463 349 349
query96 1036 789 353 353
query97 2689 2645 2508 2508
query98 218 208 205 205
query99 1134 1107 953 953
Total cold run time: 252807 ms
Total hot run time: 170147 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4429463631
<details>
<summary>TPC-DS: <b>Total hot run time: 168108 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2391ea7f407dcc42f8f4ddc3bcd8e0838059d15c, data reload: false
query5 4332 630 473 473
query6 450 217 182 182
query7 4888 556 308 308
query8 369 238 219 219
query9 8816 4123 4108 4108
query10 453 313 269 269
query11 5912 2391 2260 2260
query12 158 112 101 101
query13 1273 608 407 407
query14 6881 5304 4996 4996
query14_1 4345 4328 4342 4328
query15 205 201 177 177
query16 1050 439 497 439
query17 1157 768 613 613
query18 2724 509 356 356
query19 240 185 148 148
query20 125 109 107 107
query21 226 147 123 123
query22 13653 13593 13410 13410
query23 17261 16443 15965 15965
query23_1 16098 16201 16188 16188
query24 7615 1760 1348 1348
query24_1 1368 1355 1354 1354
query25 576 453 409 409
query26 1284 305 168 168
query27 2617 623 328 328
query28 4389 1967 1913 1913
query29 1087 612 489 489
query30 306 246 197 197
query31 1108 1066 934 934
query32 115 70 63 63
query33 525 332 258 258
query34 1198 1165 626 626
query35 739 772 669 669
query36 1332 1363 1144 1144
query37 154 103 105 103
query38 3209 3130 2993 2993
query39 934 917 897 897
query39_1 882 884 875 875
query40 230 151 117 117
query41 69 66 64 64
query42 97 91 92 91
query43 317 317 279 279
query44
query45 193 185 182 182
query46 1101 1185 734 734
query47 2320 2308 2164 2164
query48 419 408 305 305
query49 635 489 403 403
query50 706 292 221 221
query51 4261 4249 4203 4203
query52 93 90 76 76
query53 238 275 196 196
query54 266 229 203 203
query55 78 75 71 71
query56 233 234 225 225
query57 1401 1391 1300 1300
query58 244 245 197 197
query59 1518 1631 1369 1369
query60 312 265 244 244
query61 161 155 154 154
query62 664 626 558 558
query63 223 180 187 180
query64 2550 829 652 652
query65
query66 1740 485 360 360
query67 30082 30049 29150 29150
query68
query69 415 308 274 274
query70 1025 948 939 939
query71 298 221 212 212
query72 2947 2732 2447 2447
query73 831 761 428 428
query74 5065 4954 4800 4800
query75 2730 2631 2299 2299
query76 2311 1125 756 756
query77 350 385 295 295
query78 12861 12857 12401 12401
query79 1296 1081 743 743
query80 570 513 396 396
query81 452 278 247 247
query82 231 163 127 127
query83 356 276 245 245
query84 271 140 111 111
query85 870 501 427 427
query86 339 290 289 289
query87 3337 3308 3152 3152
query88 3500 2643 2649 2643
query89 409 381 322 322
query90 2036 187 181 181
query91 183 169 140 140
query92 64 58 59 58
query93 940 971 565 565
query94 514 362 316 316
query95 669 399 356 356
query96 981 785 333 333
query97 2661 2676 2552 2552
query98 212 212 207 207
query99 1109 1092 1003 1003
Total cold run time: 250346 ms
Total hot run time: 168108 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4431447780
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4423693430
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/b29520eb8c2fd176695ee6c263c89bab10446d4a_b29520eb8c2fd176695ee6c263c89bab10446d4a_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/b29520eb8c2fd176695ee6c263c89bab10446d4a_b29520eb8c2fd176695ee6c263c89bab10446d4a_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.05% (1849/2369) |
| Line Coverage | 64.70% (33208/51327) |
| Region Coverage | 65.27% (16446/25198) |
| Branch Coverage | 55.81% (8780/15732) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4433541179
# BE Regression && UT Coverage Report
Increment line coverage `45.58% (320/702)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_e7a084166584e3ccdc5ae8b6bd5fc50d3c0cb832_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_e7a084166584e3ccdc5ae8b6bd5fc50d3c0cb832_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.76% (27847/37751) |
| Line Coverage | 57.56% (301344/523508) |
| Region Coverage | 54.57% (251237/460436) |
| Branch Coverage | 55.81% (108720/194798) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4289108407
# BE Regression && UT Coverage Report
Increment line coverage `47.73% (316/662)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_6f985006ec32cea727f90e22b3cd688c1a0626e1_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_6f985006ec32cea727f90e22b3cd688c1a0626e1_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.79% (26875/37436) |
| Line Coverage | 55.00% (285806/519640) |
| Region Coverage | 51.74% (236083/456269) |
| Branch Coverage | 52.96% (102142/192854) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4288649703
# FE UT Coverage Report
Increment line coverage ` 25.95% (34/131)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_6f985006ec32cea727f90e22b3cd688c1a0626e1/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_6f985006ec32cea727f90e22b3cd688c1a0626e1/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4543756152
<details>
<summary>TPC-H: <b>Total hot run time: 32071 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3d38dc5ab813ed88625762426d63d5324bfff4fe, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17620 4074 4057 4057
q2 q3 10947 1517 831 831
q4 4841 480 344 344
q5 10640 2405 2179 2179
q6 390 177 137 137
q7 999 772 661 661
q8 9576 1746 1680 1680
q9 7087 5036 5020 5020
q10 6466 2230 1889 1889
q11 448 296 247 247
q12 651 443 302 302
q13 18212 3968 2782 2782
q14 272 264 245 245
q15 q16 824 767 702 702
q17 970 983 1036 983
q18 6910 5684 6315 5684
q19 1257 1370 1079 1079
q20 536 414 282 282
q21 5863 2857 2659 2659
q22 461 363 308 308
Total cold run time: 104970 ms
Total hot run time: 32071 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5181 4820 4891 4820
q2 q3 4868 5293 4619 4619
q4 2196 2272 1455 1455
q5 4982 4743 4807 4743
q6 239 181 126 126
q7 1876 1598 1444 1444
q8 2301 2004 1973 1973
q9 7493 7530 7452 7452
q10 4768 4704 4269 4269
q11 561 388 357 357
q12 822 755 543 543
q13 3025 3377 2820 2820
q14 264 288 250 250
q15 q16 682 700 620 620
q17 1307 1281 1290 1281
q18 7388 6854 6867 6854
q19 1115 1118 1092 1092
q20 2231 2225 1960 1960
q21 5368 4637 4589 4589
q22 544 465 415 415
Total cold run time: 57211 ms
Total hot run time: 51682 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3309562083
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,875 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ if (!tb.__isset.partition_id || !tb.__isset.slot_id) {
+ return Status::InternalError(
+ "Runtime filter partition boundary must contain partition_id and slot_id");
+ }
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ if (slot_it == slot_descs.end()) {
+ return Status::InternalError(
+ "Runtime filter partition boundary references unknown slot_id={}, "
+ "partition_id={}",
+ slot_id, tb.partition_id);
+ }
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ if (!is_list && !is_range) { \
+ return Status::InternalError( \
+ "Runtime filter partition boundary must be RANGE or LIST, " \
+ "partition_id={}, slot_id={}", \
+ tb.partition_id, slot_id); \
+ } \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ if (parsed_ok) {
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ } else {
+ return Status::InternalError(
+ "Runtime filter partition boundary has unsupported type, partition_id={}, "
+ "slot_id={}, type={}",
+ tb.partition_id, slot_id, slot_type->get_name());
+ }
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ if (!expr) {
+ return nullptr;
+ }
+ if (expr->is_slot_ref()) {
+ return assert_cast<const VSlotRef*>(expr);
+ }
+ const VSlotRef* found = nullptr;
+ for (const auto& child : expr->children()) {
+ const VSlotRef* c = find_unique_slot_ref(child.get());
+ if (c) {
+ if (found) {
+ return nullptr; // multiple slot refs, can't handle
+ }
+ found = c;
+ }
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+Status ParsedPartitionBoundaries::get_or_compute_projected_boundaries(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ const std::unordered_map<int64_t, TTargetExprMonotonicity::type>& partition_directions,
+ VExprContext* ctx, std::shared_ptr<const std::vector<ParsedBoundary>>* output) const {
+ {
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto it = _projected_boundaries_by_filter_id.find(filter_id);
+ if (it != _projected_boundaries_by_filter_id.end()) {
+ *output = it->second;
+ return Status::OK();
+ }
+ }
+
+ std::vector<ParsedBoundary> projected;
+ auto store_projected_boundaries = [&](std::vector<ParsedBoundary>&& boundaries) {
+ auto cached = std::make_shared<const std::vector<ParsedBoundary>>(std::move(boundaries));
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto [it, inserted] = _projected_boundaries_by_filter_id.emplace(filter_id, cached);
+ *output = inserted ? cached : it->second;
+ return Status::OK();
+ };
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::unordered_set<int64_t> boundary_partition_ids;
+ boundary_partition_ids.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ boundary_partition_ids.insert(boundary.partition_id);
+ }
+ for (const auto& [partition_id, direction] : partition_directions) {
+ if (direction == TTargetExprMonotonicity::NON_MONOTONIC) {
+ return Status::InternalError(
+ "Runtime filter partition pruning received NON_MONOTONIC partition "
+ "metadata, filter_id={}, partition_id={}",
+ filter_id, partition_id);
+ }
+ if (!boundary_partition_ids.contains(partition_id)) {
+ return Status::InternalError(
+ "Runtime filter partition pruning received monotonicity for a partition "
+ "without boundary, filter_id={}, partition_id={}, slot_id={}",
+ filter_id, partition_id, leaf_slot_id);
+ }
+ }
+
+ if (target_expr->is_slot_ref()) {
+ auto* slot_ref = assert_cast<VSlotRef*>(target_expr.get());
+ if (slot_ref->slot_id() != leaf_slot_id) {
+ return Status::InternalError(
+ "Runtime filter partition pruning SlotRef target mismatch, filter_id={}, "
+ "target_slot_id={}, leaf_slot_id={}",
+ filter_id, slot_ref->slot_id(), leaf_slot_id);
+ }
+ std::vector<ParsedBoundary> slot_boundaries;
+ slot_boundaries.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ auto direction_it = partition_directions.find(boundary.partition_id);
+ if (direction_it == partition_directions.end()) {
+ continue;
+ }
+ if (direction_it->second != TTargetExprMonotonicity::MONOTONIC_INCREASING) {
+ return Status::InternalError(
+ "Runtime filter partition pruning SlotRef target must use increasing "
+ "monotonicity, filter_id={}, partition_id={}, monotonicity={}",
+ filter_id, boundary.partition_id, direction_it->second);
+ }
+ slot_boundaries.emplace_back(boundary);
+ }
+ return store_projected_boundaries(std::move(slot_boundaries));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> selected_boundaries;
+ selected_boundaries.reserve(orig_boundaries.size());
+ for (size_t i = 0; i < orig_boundaries.size(); ++i) {
+ auto it = partition_directions.find(orig_boundaries[i].partition_id);
+ if (it != partition_directions.end() &&
+ it->second != TTargetExprMonotonicity::NON_MONOTONIC) {
+ selected_boundaries.emplace_back(i, it->second);
+ }
+ }
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> projectable_boundaries;
+ projectable_boundaries.reserve(selected_boundaries.size());
+ for (const auto& selected_boundary : selected_boundaries) {
+ const auto& boundary = orig_boundaries[selected_boundary.first];
+ if (!boundary.only_null && !boundary.contains_null) {
+ projectable_boundaries.emplace_back(selected_boundary);
+ }
+ }
+ selected_boundaries.swap(projectable_boundaries);
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ if (slot_type_it == _slot_data_types.end()) {
+ return Status::InternalError(
+ "Runtime filter partition pruning target slot has no boundary type, filter_id={}, "
+ "slot_id={}, target_expr={}",
+ filter_id, leaf_slot_id, target_expr->expr_name());
+ }
+
+ // LIST partitions require regrouping projected values per partition and
+ // should not be marked as expression-prunable by FE yet.
+ for (const auto& selected_boundary : selected_boundaries) {
+ bool is_list = false;
+ std::visit([&](const auto& cvr) { is_list = cvr.is_fixed_value_range(); },
+ orig_boundaries[selected_boundary.first].boundary_cvr);
+ if (is_list) {
+ return Status::InternalError(
+ "Runtime filter partition pruning expression target does not support LIST "
+ "partition boundaries, filter_id={}, partition_id={}, target_expr={}",
+ filter_id, orig_boundaries[selected_boundary.first].partition_id,
+ target_expr->expr_name());
+ }
+ }
+
+ const DataTypePtr& input_type = slot_type_it->second;
+ bool input_is_nullable = input_type->is_nullable();
+ DataTypePtr inner_type = input_is_nullable ? remove_nullable(input_type) : input_type;
+ PrimitiveType input_ptype = input_type->get_primitive_type();
+
+ size_t N = selected_boundaries.size();
+ std::vector<bool> lo_open(N, false);
+ std::vector<bool> hi_open(N, false);
+ std::vector<size_t> lo_result_row(N, 0);
+ std::vector<size_t> hi_result_row(N, 0);
+ size_t lo_row_count = 0;
+ size_t hi_row_count = 0;
+
+ // Build input blocks for lo and hi
+ Block lo_block;
+ Block hi_block;
+
+ // Pad columns 0..leaf_column_id-1 with placeholder columns so the leaf
+ // VSlotRef (whose column_id is leaf_column_id) reads our typed column.
+ auto add_dummy_columns = [&](Block& block, size_t row_count) {
+ for (int col_idx = 0; col_idx < leaf_column_id; ++col_idx) {
+ auto col = ColumnUInt8::create(row_count, 0);
+ block.insert({std::move(col), std::make_shared<DataTypeUInt8>(),
+ fmt::format("dummy_{}", col_idx)});
+ }
+ };
+
+ // Macro-dispatch on input PrimitiveType to build finite RANGE endpoint
+ // columns. Open endpoints are not executed through target_expr; they stay
+ // open after projection and swap sides for monotonic decreasing targets.
+ bool input_built = false;
+#define BUILD_INPUT_COLUMNS(INPUT_PT) \
+ case TYPE_##INPUT_PT: { \
+ using InCol = typename PrimitiveTypeTraits<TYPE_##INPUT_PT>::ColumnType; \
+ MutableColumnPtr lo_inner_base = inner_type->create_column(); \
+ MutableColumnPtr hi_inner_base = inner_type->create_column(); \
+ auto* lo_inner = assert_cast<InCol*>(lo_inner_base.get()); \
+ auto* hi_inner = assert_cast<InCol*>(hi_inner_base.get()); \
+ lo_inner->reserve(N); \
+ hi_inner->reserve(N); \
+ auto lo_nulls = ColumnUInt8::create(); \
+ auto hi_nulls = ColumnUInt8::create(); \
+ lo_nulls->reserve(N); \
+ hi_nulls->reserve(N); \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& boundary = orig_boundaries[selected_boundaries[i].first]; \
+ const auto* cvr = \
+ std::get_if<ColumnValueRange<TYPE_##INPUT_PT>>(&boundary.boundary_cvr); \
+ DCHECK(cvr != nullptr); \
+ DCHECK(!boundary.only_null); \
+ DCHECK(!boundary.contains_null); \
+ lo_open[i] = cvr->is_low_value_minimum(); \
+ hi_open[i] = cvr->is_high_value_maximum(); \
+ if (!lo_open[i]) { \
+ lo_result_row[i] = lo_row_count++; \
+ lo_inner->insert_value(cvr->get_range_min_value()); \
+ lo_nulls->get_data().push_back(0); \
+ } \
+ if (!hi_open[i]) { \
+ hi_result_row[i] = hi_row_count++; \
+ hi_inner->insert_value(cvr->get_range_max_value()); \
+ hi_nulls->get_data().push_back(0); \
+ } \
+ } \
+ add_dummy_columns(lo_block, lo_row_count); \
+ add_dummy_columns(hi_block, hi_row_count); \
+ if (input_is_nullable) { \
+ auto lo_col = ColumnNullable::create(std::move(lo_inner_base), std::move(lo_nulls)); \
+ auto hi_col = ColumnNullable::create(std::move(hi_inner_base), std::move(hi_nulls)); \
+ lo_block.insert({std::move(lo_col), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_col), input_type, "leaf_slot"}); \
+ } else { \
+ lo_block.insert({std::move(lo_inner_base), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_inner_base), input_type, "leaf_slot"}); \
+ } \
+ input_built = true; \
+ break; \
+ }
+
+ switch (input_ptype) {
+ BUILD_INPUT_COLUMNS(TINYINT)
+ BUILD_INPUT_COLUMNS(SMALLINT)
+ BUILD_INPUT_COLUMNS(INT)
+ BUILD_INPUT_COLUMNS(BIGINT)
+ BUILD_INPUT_COLUMNS(LARGEINT)
+ BUILD_INPUT_COLUMNS(FLOAT)
+ BUILD_INPUT_COLUMNS(DOUBLE)
+ BUILD_INPUT_COLUMNS(CHAR)
+ BUILD_INPUT_COLUMNS(DATE)
+ BUILD_INPUT_COLUMNS(DATETIME)
+ BUILD_INPUT_COLUMNS(DATEV2)
+ BUILD_INPUT_COLUMNS(DATETIMEV2)
+ BUILD_INPUT_COLUMNS(TIMESTAMPTZ)
+ BUILD_INPUT_COLUMNS(VARCHAR)
+ BUILD_INPUT_COLUMNS(STRING)
+ BUILD_INPUT_COLUMNS(DECIMAL32)
+ BUILD_INPUT_COLUMNS(DECIMAL64)
+ BUILD_INPUT_COLUMNS(DECIMAL128I)
+ BUILD_INPUT_COLUMNS(DECIMAL256)
+ BUILD_INPUT_COLUMNS(DECIMALV2)
+ BUILD_INPUT_COLUMNS(BOOLEAN)
+ BUILD_INPUT_COLUMNS(IPV4)
+ BUILD_INPUT_COLUMNS(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_INPUT_COLUMNS
+
+ if (!input_built) {
+ return Status::InternalError(
+ "Runtime filter partition pruning expression target has unsupported input type, "
+ "filter_id={}, input_type={}, target_expr={}",
+ filter_id, input_type->get_name(), target_expr->expr_name());
+ }
+
+ int lo_result_id = -1;
+ int hi_result_id = -1;
+ ColumnPtr lo_result_col;
+ ColumnPtr hi_result_col;
+ if (lo_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &lo_block, &lo_result_id));
+ if (lo_result_id < 0) {
+ return Status::InternalError(
+ "Runtime filter partition pruning failed to project lower boundary, "
+ "filter_id={}, target_expr={}",
+ filter_id, target_expr->expr_name());
+ }
+ lo_result_col = lo_block.get_by_position(lo_result_id).column;
+ }
+ if (hi_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &hi_block, &hi_result_id));
+ if (hi_result_id < 0) {
+ return Status::InternalError(
+ "Runtime filter partition pruning failed to project upper boundary, "
+ "filter_id={}, target_expr={}",
+ filter_id, target_expr->expr_name());
+ }
+ hi_result_col = hi_block.get_by_position(hi_result_id).column;
+ }
+ int out_precision = cast_set<int>(target_expr->data_type()->get_precision());
+ int out_scale = cast_set<int>(target_expr->data_type()->get_scale());
+ bool out_nullable = target_expr->data_type()->is_nullable();
+ PrimitiveType output_ptype = target_expr->data_type()->get_primitive_type();
+
+ projected.reserve(N);
+
+ bool output_built = false;
+#define BUILD_PROJECTED_CVR(OUTPUT_PT) \
+ case TYPE_##OUTPUT_PT: { \
+ using OutputCppType = typename PrimitiveTypeTraits<TYPE_##OUTPUT_PT>::CppType; \
+ auto projected_value = [](const ColumnPtr& column, size_t row) -> OutputCppType { \
+ Field field = (*column)[row]; \
+ return field.get<TYPE_##OUTPUT_PT>(); \
+ }; \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& orig_boundary = orig_boundaries[selected_boundaries[i].first]; \
+ TTargetExprMonotonicity::type direction = selected_boundaries[i].second; \
+ if ((!lo_open[i] && lo_result_col->is_null_at(lo_result_row[i])) || \
+ (!hi_open[i] && hi_result_col->is_null_at(hi_result_row[i]))) { \
+ continue; \
+ } \
+ ColumnValueRange<TYPE_##OUTPUT_PT> cvr("", out_nullable, out_precision, out_scale); \
+ if (direction == TTargetExprMonotonicity::MONOTONIC_DECREASING) { \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, hi_projected)); \
+ } \
+ if (!lo_open[i]) { \
Review Comment:
│ 假设分区列是 dt，原始 RANGE 分区是： │
│ │
│ P1: dt in [2024-01-01 00:00:00, 2024-01-02 00:00:00) │
│ │
│ 上界 2024-01-02 00:00:00 是开区间，这一点没错。 │
│ │
│ 但 RF target 不是 dt 本身，而是一个非严格单调表达式，例如按天向上取整： │
│ │
│ target_expr = date_ceil(dt, 'day') │
│ │
│ 这个函数是单调递增的，但不是严格递增。分区里的真实行可能有： │
│ │
│ dt = 2024-01-01 12:00:00 │
│ │
│ 它满足原始分区范围，因为它小于开上界 2024-01-02 00:00:00。但投影后： │
│ │
│ date_ceil(2024-01-01 12:00:00, 'day') = 2024-01-02 00:00:00 │
│ │
│ 也就是说，虽然原始上界是开的，投影结果里仍然可能出现 target_expr(range_end) 这个值。 │
│ │
│ 如果 BE 把投影后的范围做成： │
│ │
│ [date_ceil(start), date_ceil(end)) -- 上界仍然用 < │
│ │
│ 那就会变成不包含 2024-01-02 00:00:00。假如 RF build side 正好只有： │
│ │
│ RF = {2024-01-02 00:00:00} │
│ │
│ BE 会认为投影分区范围和 RF 没交集，从而裁掉 P1；但实际上 P1 里 2024-01-01 12:00:00 这类行投影后能命中 RF，所以这是误裁剪。 │
│ │
│ 所以当前用 <= target_expr(end) 是一个安全的 over-approx：可能多保留一点分区，但不会把可能命中的分区裁掉。只有当我们能证明 target expr │
│ 是严格单调且开边界不会映射出端点值时，才可以安全地保留 <。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4279540844
# FE Regression Coverage Report
Increment line coverage ` 57.89% (77/133)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_bcfae3c60326b6ef46e5c8dfc68aaf68a3be4fdd_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_bcfae3c60326b6ef46e5c8dfc68aaf68a3be4fdd_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4288331395
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/6f985006ec32cea727f90e22b3cd688c1a0626e1_6f985006ec32cea727f90e22b3cd688c1a0626e1_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/6f985006ec32cea727f90e22b3cd688c1a0626e1_6f985006ec32cea727f90e22b3cd688c1a0626e1_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1843/2361) |
| Line Coverage | 64.76% (32993/50947) |
| Region Coverage | 65.27% (16374/25085) |
| Branch Coverage | 55.85% (8741/15652) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3117280644
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,750 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vruntimefilter_wrapper.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+void ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ if (!tb.__isset.partition_id || !tb.__isset.slot_id) {
+ continue;
+ }
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ if (slot_it == slot_descs.end()) {
+ continue;
+ }
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ if (!is_list && !is_range) break; \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ if (parsed_ok) {
+ _partition_column_slot_ids.insert(slot_id);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_partition_column_slot_ids.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ if (!expr) {
+ return nullptr;
+ }
+ if (expr->is_slot_ref()) {
+ return assert_cast<const VSlotRef*>(expr);
+ }
+ const VSlotRef* found = nullptr;
+ for (const auto& child : expr->children()) {
+ const VSlotRef* c = find_unique_slot_ref(child.get());
+ if (c) {
+ if (found) {
+ return nullptr; // multiple slot refs, can't handle
+ }
+ found = c;
+ }
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+const std::vector<ParsedBoundary>* ParsedPartitionBoundaries::get_or_compute_projection(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ TTargetExprMonotonicity::type direction, VExprContext* ctx) const {
+ {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ auto it = _projection_cache.find(filter_id);
+ if (it != _projection_cache.end()) {
+ return &it->second;
+ }
+ }
+
+ // Build projection
+ std::vector<ParsedBoundary> projected;
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ if (slot_type_it == _slot_data_types.end()) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+
+ // LIST partitions: TODO(rf-partition-prune) -- projecting through a
+ // monotonic function preserves set membership but requires re-grouping
+ // the projected values back per-partition. Skip for now (caller treats
+ // empty cache as "no projection available").
+ {
+ bool any_list = false;
+ std::visit([&](const auto& cvr) { any_list = cvr.is_fixed_value_range(); },
+ orig_boundaries[0].boundary_cvr);
+ if (any_list) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+ }
+
+ const DataTypePtr& input_type = slot_type_it->second;
+ bool input_is_nullable = input_type->is_nullable();
+ DataTypePtr inner_type = input_is_nullable ? remove_nullable(input_type) : input_type;
+ PrimitiveType input_ptype = input_type->get_primitive_type();
+
+ size_t N = orig_boundaries.size();
+
+ // Track open bounds (low_value == TYPE_MIN or high_value == TYPE_MAX)
+ // externally so we can propagate null_in_projection regardless of whether
+ // the input column is nullable.
+ std::vector<bool> ext_null_lo(N, false);
+ std::vector<bool> ext_null_hi(N, false);
+
+ // Build input blocks for lo and hi
+ Block lo_block;
+ Block hi_block;
+
+ // Pad columns 0..leaf_column_id-1 with placeholder columns so the leaf
+ // VSlotRef (whose column_id is leaf_column_id) reads our typed column.
+ for (int col_idx = 0; col_idx < leaf_column_id; ++col_idx) {
+ auto add_dummy = [&](Block& blk) {
+ auto col = ColumnUInt8::create(N, 0);
+ blk.insert({std::move(col), std::make_shared<DataTypeUInt8>(),
+ fmt::format("dummy_{}", col_idx)});
+ };
+ add_dummy(lo_block);
+ add_dummy(hi_block);
+ }
+
+ // Macro-dispatch on input PrimitiveType to build the typed value column
+ // for each side. Inserts NULL (when input column is nullable) or default
+ // (when not) for boundaries that are only_null / open / type-mismatched;
+ // ext_null_lo/hi tracks these so we can mark them null_in_projection
+ // after executing the projection.
+ bool input_built = false;
+#define BUILD_INPUT_COLUMNS(INPUT_PT) \
+ case TYPE_##INPUT_PT: { \
+ using InCol = typename PrimitiveTypeTraits<TYPE_##INPUT_PT>::ColumnType; \
+ MutableColumnPtr lo_inner_base = inner_type->create_column(); \
+ MutableColumnPtr hi_inner_base = inner_type->create_column(); \
+ auto* lo_inner = assert_cast<InCol*>(lo_inner_base.get()); \
+ auto* hi_inner = assert_cast<InCol*>(hi_inner_base.get()); \
+ lo_inner->reserve(N); \
+ hi_inner->reserve(N); \
+ auto lo_nulls = ColumnUInt8::create(); \
+ auto hi_nulls = ColumnUInt8::create(); \
+ lo_nulls->reserve(N); \
+ hi_nulls->reserve(N); \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& boundary = orig_boundaries[i]; \
+ bool null_lo = boundary.only_null; \
+ bool null_hi = boundary.only_null; \
+ const auto* cvr = \
+ std::get_if<ColumnValueRange<TYPE_##INPUT_PT>>(&boundary.boundary_cvr); \
+ if (cvr == nullptr) { \
+ null_lo = null_hi = true; \
+ } else { \
+ if (cvr->is_low_value_minimum()) null_lo = true; \
+ if (cvr->is_high_value_maximum()) null_hi = true; \
+ } \
+ ext_null_lo[i] = null_lo; \
+ ext_null_hi[i] = null_hi; \
+ if (null_lo) { \
+ lo_inner->insert_default(); \
+ lo_nulls->get_data().push_back(1); \
+ } else { \
+ lo_inner->insert_value(cvr->get_range_min_value()); \
+ lo_nulls->get_data().push_back(0); \
+ } \
+ if (null_hi) { \
+ hi_inner->insert_default(); \
+ hi_nulls->get_data().push_back(1); \
+ } else { \
+ hi_inner->insert_value(cvr->get_range_max_value()); \
+ hi_nulls->get_data().push_back(0); \
+ } \
+ } \
+ if (input_is_nullable) { \
+ auto lo_col = ColumnNullable::create(std::move(lo_inner_base), std::move(lo_nulls)); \
+ auto hi_col = ColumnNullable::create(std::move(hi_inner_base), std::move(hi_nulls)); \
+ lo_block.insert({std::move(lo_col), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_col), input_type, "leaf_slot"}); \
+ } else { \
+ lo_block.insert({std::move(lo_inner_base), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_inner_base), input_type, "leaf_slot"}); \
+ } \
+ input_built = true; \
+ break; \
+ }
+
+ switch (input_ptype) {
+ BUILD_INPUT_COLUMNS(TINYINT)
+ BUILD_INPUT_COLUMNS(SMALLINT)
+ BUILD_INPUT_COLUMNS(INT)
+ BUILD_INPUT_COLUMNS(BIGINT)
+ BUILD_INPUT_COLUMNS(LARGEINT)
+ BUILD_INPUT_COLUMNS(FLOAT)
+ BUILD_INPUT_COLUMNS(DOUBLE)
+ BUILD_INPUT_COLUMNS(CHAR)
+ BUILD_INPUT_COLUMNS(DATE)
+ BUILD_INPUT_COLUMNS(DATETIME)
+ BUILD_INPUT_COLUMNS(DATEV2)
+ BUILD_INPUT_COLUMNS(DATETIMEV2)
+ BUILD_INPUT_COLUMNS(TIMESTAMPTZ)
+ BUILD_INPUT_COLUMNS(VARCHAR)
+ BUILD_INPUT_COLUMNS(STRING)
+ BUILD_INPUT_COLUMNS(DECIMAL32)
+ BUILD_INPUT_COLUMNS(DECIMAL64)
+ BUILD_INPUT_COLUMNS(DECIMAL128I)
+ BUILD_INPUT_COLUMNS(DECIMAL256)
+ BUILD_INPUT_COLUMNS(DECIMALV2)
+ BUILD_INPUT_COLUMNS(BOOLEAN)
+ BUILD_INPUT_COLUMNS(IPV4)
+ BUILD_INPUT_COLUMNS(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_INPUT_COLUMNS
+
+ if (!input_built) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+
+ int lo_result_id = -1;
+ int hi_result_id = -1;
+ auto status_lo = target_expr->execute(ctx, &lo_block, &lo_result_id);
+ auto status_hi = target_expr->execute(ctx, &hi_block, &hi_result_id);
+ if (!status_lo.ok() || !status_hi.ok() || lo_result_id < 0 || hi_result_id < 0) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+
+ const auto& lo_result_col = lo_block.get_by_position(lo_result_id).column;
+ const auto& hi_result_col = hi_block.get_by_position(hi_result_id).column;
+
+ int out_precision = cast_set<int>(target_expr->data_type()->get_precision());
+ int out_scale = cast_set<int>(target_expr->data_type()->get_scale());
+ bool out_nullable = target_expr->data_type()->is_nullable();
+ PrimitiveType output_ptype = target_expr->data_type()->get_primitive_type();
+
+ projected.resize(N);
+
+#define BUILD_PROJECTED_CVR(OUTPUT_PT) \
+ case TYPE_##OUTPUT_PT: { \
+ using OutputCppType = typename PrimitiveTypeTraits<TYPE_##OUTPUT_PT>::CppType; \
+ for (size_t i = 0; i < N; ++i) { \
+ projected[i].partition_id = orig_boundaries[i].partition_id; \
+ projected[i].slot_id = leaf_slot_id; \
+ projected[i].is_nullable = out_nullable; \
+ projected[i].only_null = orig_boundaries[i].only_null; \
+ projected[i].contains_null = orig_boundaries[i].contains_null; \
+ bool lo_is_null = ext_null_lo[i] || lo_result_col->is_null_at(i); \
+ bool hi_is_null = ext_null_hi[i] || hi_result_col->is_null_at(i); \
+ if (lo_is_null || hi_is_null) { \
+ projected[i].null_in_projection = true; \
+ ColumnValueRange<TYPE_##OUTPUT_PT> full_range("", out_nullable, out_precision, \
+ out_scale); \
+ projected[i].boundary_cvr = std::move(full_range); \
+ } else { \
+ Field lo_field = (*lo_result_col)[i]; \
+ Field hi_field = (*hi_result_col)[i]; \
+ OutputCppType lo_projected = lo_field.get<TYPE_##OUTPUT_PT>(); \
+ OutputCppType hi_projected = hi_field.get<TYPE_##OUTPUT_PT>(); \
+ ColumnValueRange<TYPE_##OUTPUT_PT> cvr("", out_nullable, out_precision, \
+ out_scale); \
+ if (direction == TTargetExprMonotonicity::MONOTONIC_DECREASING) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, hi_projected)); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, lo_projected)); \
+ } else { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, lo_projected)); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, hi_projected)); \
+ } \
+ projected[i].boundary_cvr = std::move(cvr); \
+ } \
+ } \
+ break; \
+ }
+
+ switch (output_ptype) {
+ BUILD_PROJECTED_CVR(TINYINT)
+ BUILD_PROJECTED_CVR(SMALLINT)
+ BUILD_PROJECTED_CVR(INT)
+ BUILD_PROJECTED_CVR(BIGINT)
+ BUILD_PROJECTED_CVR(LARGEINT)
+ BUILD_PROJECTED_CVR(FLOAT)
+ BUILD_PROJECTED_CVR(DOUBLE)
+ BUILD_PROJECTED_CVR(CHAR)
+ BUILD_PROJECTED_CVR(DATE)
+ BUILD_PROJECTED_CVR(DATETIME)
+ BUILD_PROJECTED_CVR(DATEV2)
+ BUILD_PROJECTED_CVR(DATETIMEV2)
+ BUILD_PROJECTED_CVR(TIMESTAMPTZ)
+ BUILD_PROJECTED_CVR(VARCHAR)
+ BUILD_PROJECTED_CVR(STRING)
+ BUILD_PROJECTED_CVR(DECIMAL32)
+ BUILD_PROJECTED_CVR(DECIMAL64)
+ BUILD_PROJECTED_CVR(DECIMAL128I)
+ BUILD_PROJECTED_CVR(DECIMAL256)
+ BUILD_PROJECTED_CVR(DECIMALV2)
+ BUILD_PROJECTED_CVR(BOOLEAN)
+ BUILD_PROJECTED_CVR(IPV4)
+ BUILD_PROJECTED_CVR(IPV6)
+ default:
+ break;
+ }
+
+#undef BUILD_PROJECTED_CVR
+
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static SQLFilterOp convert_opcode_to_filter_op(TExprOpcode::type op) {
+ switch (op) {
+ case TExprOpcode::LE:
+ return FILTER_LESS_OR_EQUAL;
+ case TExprOpcode::LT:
+ return FILTER_LESS;
+ case TExprOpcode::GE:
+ return FILTER_LARGER_OR_EQUAL;
+ case TExprOpcode::GT:
+ return FILTER_LARGER;
+ default:
+ return FILTER_IN; // sentinel: caller should skip
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+void RuntimeFilterPartitionPruner::_try_prune_by_single_rf(
+ const std::vector<ParsedBoundary>& boundaries, const VExprSPtr& impl,
+ phmap::flat_hash_set<int64_t>& newly_pruned) {
+ // Pre-compute whether the RF "matches NULL" -- i.e. whether the RF would
+ // accept a row whose probe value is NULL. The signal differs by RF impl:
+ // * IN filter → HybridSet::contain_null()
+ // * Bloom filter → BloomFilterFuncBase::contain_null()
+ // * MinMax filter → encoded via the BinaryPredicate node_type:
+ // NULL_AWARE_BINARY_PRED means the build side had a
+ // NULL key and a null-safe equal join asked NULL to
+ // match NULL; plain BINARY_PRED never matches NULL.
+ // FilterBase::contain_null() already folds in `_null_aware`, so we only
+ // get a true result when the build side is actually null-aware AND
+ // produced a NULL value.
+ bool rf_contains_null = false;
+ if (auto hybrid_set = impl->get_set_func()) {
+ rf_contains_null = hybrid_set->contain_null();
+ } else if (impl->node_type() == TExprNodeType::BLOOM_PRED) {
+ auto bloom = impl->get_bloom_filter_func();
+ rf_contains_null = bloom && bloom->contain_null();
+ } else if (impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) {
+ // Min/Max RF built on a null-safe equal join. The literal child holds
+ // the min or max bound; the NULL semantic is conveyed by the node
+ // type itself (see create_vbin_predicate in runtime_filter/utils.cpp).
+ rf_contains_null = true;
+ }
+
+ for (const auto& pb : boundaries) {
+ if (_pruned_partition_ids.contains(pb.partition_id) ||
+ newly_pruned.contains(pb.partition_id)) {
+ continue;
+ }
+
+ // Skip partitions with NULL projection conservatively
+ if (pb.null_in_projection) {
+ continue;
+ }
+
+ // NULL handling:
+ // A partition row whose key is NULL matches the RF iff `rf_contains_null`.
+ // - only_null partition (rows are exclusively NULL): prunable iff !rf_contains_null.
+ // - mixed (NULL + concrete values): if rf_contains_null, NULL rows alone
+ // prevent pruning. Otherwise NULL rows can never match, so we ignore
+ // the NULL portion and let the regular non-NULL intersection decide.
+ if (pb.only_null) {
+ if (!rf_contains_null) {
+ newly_pruned.insert(pb.partition_id);
+ }
+ continue;
+ }
+ if (pb.contains_null && rf_contains_null) {
+ continue;
+ }
+
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+ using CppType = typename CvrType::CppType;
+
+ auto hybrid_set = impl->get_set_func();
Review Comment:
When an `IN_OR_BLOOM_FILTER` switches to its real `BLOOM_FILTER` shape (`RuntimeFilterWrapper::get_real_type()`), the consumer builds a `VBloomPredicate` (`RuntimeFilterConsumer::_get_push_exprs`, `BLOOM_FILTER` case). In this function we only use `BLOOM_PRED` to derive `rf_contains_null`, but the actual pruning logic below only handles `impl->get_set_func()` and binary min/max predicates. So any real bloom RF never intersects with the partition boundary at all and internal-table partition pruning silently becomes a no-op for larger build sides or remote-merged RFs. Please either add a concrete bloom-vs-boundary pruning path here or explicitly gate this feature to the IN/min-max shapes; otherwise queries that fall over `runtime_filter_max_in_num` will lose the pruning promised by this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3116279964
##########
be/src/exec/operator/olap_scan_operator.cpp:
##########
@@ -499,6 +508,36 @@ Status OlapScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {
_cond_ranges.emplace_back(new doris::OlapScanRange());
}
+ // Filter out tablets whose partitions have been pruned by runtime filters.
+ if (_rf_partition_pruner.pruned_partition_count() > 0) {
Review Comment:
By the time this compaction runs, `OlapScanLocalState::prepare()` has already called `get_tablet()` / `capture_read_source()` for every scan range, and in cloud mode `init()` / `prepare()` has already kicked off and waited for `sync_rowsets()` on every tablet. Since `ScanLocalState::open()` only acquires RFs afterwards, ready-at-start RFs still pay the full metadata/read-source setup cost for partitions that are immediately dropped here. On wide-partition tables that startup work can dominate, so the new feature currently saves scanner construction / scan IO, but not the expensive setup work that partition pruning was supposed to avoid.
##########
be/src/exec/operator/scan_operator.cpp:
##########
@@ -82,6 +82,7 @@ Status ScanLocalStateBase::update_late_arrival_runtime_filter(RuntimeState* stat
return a->execute_cost() < b->execute_cost();
});
};
+ _on_runtime_filter_update();
Review Comment:
`update_late_arrival_runtime_filter()` does not know whether `try_append_late_arrival_runtime_filter()` actually appended a new RF. The only "new arrival" check happens later in `Scanner::try_append_late_arrival_runtime_filter()` by comparing `arrived_rf_num` with `_applied_rf_num`, so this new hook now reruns `_do_partition_pruning_by_rf()` on every scanner reschedule while there are still unapplied RFs. In the current scanner code `_applied_rf_num` is never advanced, which means that after the first late RF arrives we will rescan all partition boundaries on every scheduler pass forever. Can we gate this callback on "new exprs were appended" before re-entering the pruner?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4287670951
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3116865659
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,736 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vruntimefilter_wrapper.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+void ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ if (!tb.__isset.partition_id || !tb.__isset.slot_id) {
+ continue;
+ }
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ if (slot_it == slot_descs.end()) {
+ continue;
+ }
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ if (!is_list && !is_range) break; \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ if (parsed_ok) {
+ _partition_column_slot_ids.insert(slot_id);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_partition_column_slot_ids.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ if (!expr) {
+ return nullptr;
+ }
+ if (expr->is_slot_ref()) {
+ return assert_cast<const VSlotRef*>(expr);
+ }
+ const VSlotRef* found = nullptr;
+ for (const auto& child : expr->children()) {
+ const VSlotRef* c = find_unique_slot_ref(child.get());
+ if (c) {
+ if (found) {
+ return nullptr; // multiple slot refs, can't handle
+ }
+ found = c;
+ }
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+const std::vector<ParsedBoundary>* ParsedPartitionBoundaries::get_or_compute_projection(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ TTargetExprMonotonicity::type direction, VExprContext* ctx) const {
+ {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ auto it = _projection_cache.find(filter_id);
+ if (it != _projection_cache.end()) {
+ return &it->second;
+ }
+ }
+
+ // Build projection
+ std::vector<ParsedBoundary> projected;
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ if (slot_type_it == _slot_data_types.end()) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+
+ // LIST partitions: TODO(rf-partition-prune) -- projecting through a
+ // monotonic function preserves set membership but requires re-grouping
+ // the projected values back per-partition. Skip for now (caller treats
+ // empty cache as "no projection available").
+ {
+ bool any_list = false;
+ std::visit([&](const auto& cvr) { any_list = cvr.is_fixed_value_range(); },
+ orig_boundaries[0].boundary_cvr);
+ if (any_list) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+ }
+
+ const DataTypePtr& input_type = slot_type_it->second;
+ bool input_is_nullable = input_type->is_nullable();
+ DataTypePtr inner_type = input_is_nullable ? remove_nullable(input_type) : input_type;
+ PrimitiveType input_ptype = input_type->get_primitive_type();
+
+ size_t N = orig_boundaries.size();
+
+ // Track open bounds (low_value == TYPE_MIN or high_value == TYPE_MAX)
+ // externally so we can propagate null_in_projection regardless of whether
+ // the input column is nullable.
+ std::vector<bool> ext_null_lo(N, false);
+ std::vector<bool> ext_null_hi(N, false);
+
+ // Build input blocks for lo and hi
+ Block lo_block;
+ Block hi_block;
+
+ // Pad columns 0..leaf_column_id-1 with placeholder columns so the leaf
+ // VSlotRef (whose column_id is leaf_column_id) reads our typed column.
+ for (int col_idx = 0; col_idx < leaf_column_id; ++col_idx) {
+ auto add_dummy = [&](Block& blk) {
+ auto col = ColumnUInt8::create(N, 0);
+ blk.insert({std::move(col), std::make_shared<DataTypeUInt8>(),
+ fmt::format("dummy_{}", col_idx)});
+ };
+ add_dummy(lo_block);
+ add_dummy(hi_block);
+ }
+
+ // Macro-dispatch on input PrimitiveType to build the typed value column
+ // for each side. Inserts NULL (when input column is nullable) or default
+ // (when not) for boundaries that are only_null / open / type-mismatched;
+ // ext_null_lo/hi tracks these so we can mark them null_in_projection
+ // after executing the projection.
+ bool input_built = false;
+#define BUILD_INPUT_COLUMNS(INPUT_PT) \
+ case TYPE_##INPUT_PT: { \
+ using InCol = typename PrimitiveTypeTraits<TYPE_##INPUT_PT>::ColumnType; \
+ MutableColumnPtr lo_inner_base = inner_type->create_column(); \
+ MutableColumnPtr hi_inner_base = inner_type->create_column(); \
+ auto* lo_inner = assert_cast<InCol*>(lo_inner_base.get()); \
+ auto* hi_inner = assert_cast<InCol*>(hi_inner_base.get()); \
+ lo_inner->reserve(N); \
+ hi_inner->reserve(N); \
+ auto lo_nulls = ColumnUInt8::create(); \
+ auto hi_nulls = ColumnUInt8::create(); \
+ lo_nulls->reserve(N); \
+ hi_nulls->reserve(N); \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& boundary = orig_boundaries[i]; \
+ bool null_lo = boundary.only_null; \
+ bool null_hi = boundary.only_null; \
+ const auto* cvr = \
+ std::get_if<ColumnValueRange<TYPE_##INPUT_PT>>(&boundary.boundary_cvr); \
+ if (cvr == nullptr) { \
+ null_lo = null_hi = true; \
+ } else { \
+ if (cvr->is_low_value_minimum()) null_lo = true; \
+ if (cvr->is_high_value_maximum()) null_hi = true; \
+ } \
+ ext_null_lo[i] = null_lo; \
+ ext_null_hi[i] = null_hi; \
+ if (null_lo) { \
+ lo_inner->insert_default(); \
+ lo_nulls->get_data().push_back(1); \
+ } else { \
+ lo_inner->insert_value(cvr->get_range_min_value()); \
+ lo_nulls->get_data().push_back(0); \
+ } \
+ if (null_hi) { \
+ hi_inner->insert_default(); \
+ hi_nulls->get_data().push_back(1); \
+ } else { \
+ hi_inner->insert_value(cvr->get_range_max_value()); \
+ hi_nulls->get_data().push_back(0); \
+ } \
+ } \
+ if (input_is_nullable) { \
+ auto lo_col = ColumnNullable::create(std::move(lo_inner_base), std::move(lo_nulls)); \
+ auto hi_col = ColumnNullable::create(std::move(hi_inner_base), std::move(hi_nulls)); \
+ lo_block.insert({std::move(lo_col), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_col), input_type, "leaf_slot"}); \
+ } else { \
+ lo_block.insert({std::move(lo_inner_base), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_inner_base), input_type, "leaf_slot"}); \
+ } \
+ input_built = true; \
+ break; \
+ }
+
+ switch (input_ptype) {
+ BUILD_INPUT_COLUMNS(TINYINT)
+ BUILD_INPUT_COLUMNS(SMALLINT)
+ BUILD_INPUT_COLUMNS(INT)
+ BUILD_INPUT_COLUMNS(BIGINT)
+ BUILD_INPUT_COLUMNS(LARGEINT)
+ BUILD_INPUT_COLUMNS(FLOAT)
+ BUILD_INPUT_COLUMNS(DOUBLE)
+ BUILD_INPUT_COLUMNS(CHAR)
+ BUILD_INPUT_COLUMNS(DATE)
+ BUILD_INPUT_COLUMNS(DATETIME)
+ BUILD_INPUT_COLUMNS(DATEV2)
+ BUILD_INPUT_COLUMNS(DATETIMEV2)
+ BUILD_INPUT_COLUMNS(TIMESTAMPTZ)
+ BUILD_INPUT_COLUMNS(VARCHAR)
+ BUILD_INPUT_COLUMNS(STRING)
+ BUILD_INPUT_COLUMNS(DECIMAL32)
+ BUILD_INPUT_COLUMNS(DECIMAL64)
+ BUILD_INPUT_COLUMNS(DECIMAL128I)
+ BUILD_INPUT_COLUMNS(DECIMAL256)
+ BUILD_INPUT_COLUMNS(DECIMALV2)
+ BUILD_INPUT_COLUMNS(BOOLEAN)
+ BUILD_INPUT_COLUMNS(IPV4)
+ BUILD_INPUT_COLUMNS(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_INPUT_COLUMNS
+
+ if (!input_built) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+
+ int lo_result_id = -1;
+ int hi_result_id = -1;
+ auto status_lo = target_expr->execute(ctx, &lo_block, &lo_result_id);
+ auto status_hi = target_expr->execute(ctx, &hi_block, &hi_result_id);
+ if (!status_lo.ok() || !status_hi.ok() || lo_result_id < 0 || hi_result_id < 0) {
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+ }
+
+ const auto& lo_result_col = lo_block.get_by_position(lo_result_id).column;
+ const auto& hi_result_col = hi_block.get_by_position(hi_result_id).column;
+
+ int out_precision = cast_set<int>(target_expr->data_type()->get_precision());
+ int out_scale = cast_set<int>(target_expr->data_type()->get_scale());
+ bool out_nullable = target_expr->data_type()->is_nullable();
+ PrimitiveType output_ptype = target_expr->data_type()->get_primitive_type();
+
+ projected.resize(N);
+
+#define BUILD_PROJECTED_CVR(OUTPUT_PT) \
+ case TYPE_##OUTPUT_PT: { \
+ using OutputCppType = typename PrimitiveTypeTraits<TYPE_##OUTPUT_PT>::CppType; \
+ for (size_t i = 0; i < N; ++i) { \
+ projected[i].partition_id = orig_boundaries[i].partition_id; \
+ projected[i].slot_id = leaf_slot_id; \
+ projected[i].is_nullable = out_nullable; \
+ projected[i].only_null = orig_boundaries[i].only_null; \
+ projected[i].contains_null = orig_boundaries[i].contains_null; \
+ bool lo_is_null = ext_null_lo[i] || lo_result_col->is_null_at(i); \
+ bool hi_is_null = ext_null_hi[i] || hi_result_col->is_null_at(i); \
+ if (lo_is_null || hi_is_null) { \
+ projected[i].null_in_projection = true; \
+ ColumnValueRange<TYPE_##OUTPUT_PT> full_range("", out_nullable, out_precision, \
+ out_scale); \
+ projected[i].boundary_cvr = std::move(full_range); \
+ } else { \
+ Field lo_field = (*lo_result_col)[i]; \
+ Field hi_field = (*hi_result_col)[i]; \
+ OutputCppType lo_projected = lo_field.get<TYPE_##OUTPUT_PT>(); \
+ OutputCppType hi_projected = hi_field.get<TYPE_##OUTPUT_PT>(); \
+ ColumnValueRange<TYPE_##OUTPUT_PT> cvr("", out_nullable, out_precision, \
+ out_scale); \
+ if (direction == TTargetExprMonotonicity::MONOTONIC_DECREASING) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, hi_projected)); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, lo_projected)); \
+ } else { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, lo_projected)); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, hi_projected)); \
+ } \
+ projected[i].boundary_cvr = std::move(cvr); \
+ } \
+ } \
+ break; \
+ }
+
+ switch (output_ptype) {
+ BUILD_PROJECTED_CVR(TINYINT)
+ BUILD_PROJECTED_CVR(SMALLINT)
+ BUILD_PROJECTED_CVR(INT)
+ BUILD_PROJECTED_CVR(BIGINT)
+ BUILD_PROJECTED_CVR(LARGEINT)
+ BUILD_PROJECTED_CVR(FLOAT)
+ BUILD_PROJECTED_CVR(DOUBLE)
+ BUILD_PROJECTED_CVR(CHAR)
+ BUILD_PROJECTED_CVR(DATE)
+ BUILD_PROJECTED_CVR(DATETIME)
+ BUILD_PROJECTED_CVR(DATEV2)
+ BUILD_PROJECTED_CVR(DATETIMEV2)
+ BUILD_PROJECTED_CVR(TIMESTAMPTZ)
+ BUILD_PROJECTED_CVR(VARCHAR)
+ BUILD_PROJECTED_CVR(STRING)
+ BUILD_PROJECTED_CVR(DECIMAL32)
+ BUILD_PROJECTED_CVR(DECIMAL64)
+ BUILD_PROJECTED_CVR(DECIMAL128I)
+ BUILD_PROJECTED_CVR(DECIMAL256)
+ BUILD_PROJECTED_CVR(DECIMALV2)
+ BUILD_PROJECTED_CVR(BOOLEAN)
+ BUILD_PROJECTED_CVR(IPV4)
+ BUILD_PROJECTED_CVR(IPV6)
+ default:
+ break;
+ }
+
+#undef BUILD_PROJECTED_CVR
+
+ std::lock_guard<std::mutex> lock(_projection_cache_mutex);
+ _projection_cache[filter_id] = std::move(projected);
+ return &_projection_cache[filter_id];
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static SQLFilterOp convert_opcode_to_filter_op(TExprOpcode::type op) {
+ switch (op) {
+ case TExprOpcode::LE:
+ return FILTER_LESS_OR_EQUAL;
+ case TExprOpcode::LT:
+ return FILTER_LESS;
+ case TExprOpcode::GE:
+ return FILTER_LARGER_OR_EQUAL;
+ case TExprOpcode::GT:
+ return FILTER_LARGER;
+ default:
+ return FILTER_IN; // sentinel: caller should skip
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+void RuntimeFilterPartitionPruner::_try_prune_by_single_rf(
+ const std::vector<ParsedBoundary>& boundaries, const VExprSPtr& impl,
+ phmap::flat_hash_set<int64_t>& newly_pruned) {
+ // Pre-compute whether the RF "matches NULL" -- i.e. whether the RF would
+ // accept a row whose probe value is NULL. The signal differs by RF impl:
+ // * IN filter → HybridSet::contain_null()
+ // * Bloom filter → BloomFilterFuncBase::contain_null()
+ // * MinMax filter → encoded via the BinaryPredicate node_type:
+ // NULL_AWARE_BINARY_PRED means the build side had a
+ // NULL key and a null-safe equal join asked NULL to
+ // match NULL; plain BINARY_PRED never matches NULL.
+ // FilterBase::contain_null() already folds in `_null_aware`, so we only
+ // get a true result when the build side is actually null-aware AND
+ // produced a NULL value.
+ bool rf_contains_null = false;
+ if (auto hybrid_set = impl->get_set_func()) {
+ rf_contains_null = hybrid_set->contain_null();
+ } else if (impl->node_type() == TExprNodeType::BLOOM_PRED) {
+ auto bloom = impl->get_bloom_filter_func();
+ rf_contains_null = bloom && bloom->contain_null();
+ } else if (impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) {
+ // Min/Max RF built on a null-safe equal join. The literal child holds
+ // the min or max bound; the NULL semantic is conveyed by the node
+ // type itself (see create_vbin_predicate in runtime_filter/utils.cpp).
+ rf_contains_null = true;
+ }
+
+ for (const auto& pb : boundaries) {
+ if (_pruned_partition_ids.contains(pb.partition_id) ||
+ newly_pruned.contains(pb.partition_id)) {
+ continue;
+ }
+
+ // Skip partitions with NULL projection conservatively
+ if (pb.null_in_projection) {
+ continue;
+ }
+
+ // NULL handling:
+ // A partition row whose key is NULL matches the RF iff `rf_contains_null`.
+ // - only_null partition (rows are exclusively NULL): prunable iff !rf_contains_null.
+ // - mixed (NULL + concrete values): if rf_contains_null, NULL rows alone
+ // prevent pruning. Otherwise NULL rows can never match, so we ignore
+ // the NULL portion and let the regular non-NULL intersection decide.
+ if (pb.only_null) {
+ if (!rf_contains_null) {
+ newly_pruned.insert(pb.partition_id);
+ }
+ continue;
+ }
+ if (pb.contains_null && rf_contains_null) {
+ continue;
+ }
+
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+ using CppType = typename CvrType::CppType;
+
+ auto hybrid_set = impl->get_set_func();
+ if (hybrid_set) {
+ // IN filter: build a fixed-value CVR from the HybridSet
+ auto rf_cvr = CvrType::create_empty_column_value_range(
+ pb.is_nullable, boundary_cvr.precision(), boundary_cvr.scale());
+ auto* iter = hybrid_set->begin();
+ while (iter->has_next()) {
+ const void* value = iter->get_value();
+ if (value) {
+ if constexpr (std::is_same_v<CppType, StringRef>) {
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else {
+ static_cast<void>(rf_cvr.add_fixed_value(
+ *reinterpret_cast<const CppType*>(value)));
+ }
+ }
+ iter->next();
+ }
+ auto boundary_copy = boundary_cvr;
+ boundary_copy.intersection(rf_cvr);
+ if (boundary_copy.is_empty_value_range()) {
+ newly_pruned.insert(pb.partition_id);
+ }
+ } else if ((impl->node_type() == TExprNodeType::BINARY_PRED ||
+ impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) &&
+ impl->children().size() == 2 && impl->children()[1]->is_literal()) {
+ // MinMax filter: binary pred with literal bound
+ auto* literal = assert_cast<VLiteral*>(impl->children()[1].get());
+ auto col_ptr = literal->get_column_ptr();
+ auto data = col_ptr->get_data_at(0);
+ CppType val {};
+ if constexpr (std::is_same_v<CppType, StringRef>) {
+ val = CppType(data.data, data.size);
+ } else {
+ val = *reinterpret_cast<const CppType*>(data.data);
Review Comment:
is , not (). In this MIN/MAX branch the literal comes from , which builds a (, , ), so here returns a pointing at the character bytes. Reinterpreting as on the branch is therefore invalid for CHAR/VARCHAR/STRING and can read garbage or crash.
This path is reachable for string join keys because builds / literals for filters on those types. Please construct string-like values from the returned (for example ) instead of dereferencing as , and add a string partition-pruning regression case so this stays covered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4266847960
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4441755835
# BE Regression && UT Coverage Report
Increment line coverage `48.72% (342/702)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_ca30c183fbab4ffab1f11b3d91b0561736c7c199_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_ca30c183fbab4ffab1f11b3d91b0561736c7c199_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.71% (27799/37715) |
| Line Coverage | 57.62% (301509/523286) |
| Region Coverage | 54.83% (252433/460402) |
| Branch Coverage | 56.01% (109112/194803) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4430190107
# FE Regression Coverage Report
Increment line coverage ` 59.03% (170/288)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_2391ea7f407dcc42f8f4ddc3bcd8e0838059d15c_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_2391ea7f407dcc42f8f4ddc3bcd8e0838059d15c_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4423649533
# FE UT Coverage Report
Increment line coverage ` 25.93% (70/270)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_b29520eb8c2fd176695ee6c263c89bab10446d4a/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_b29520eb8c2fd176695ee6c263c89bab10446d4a/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] Dev runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4428347988
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4430137536
# BE Regression && UT Coverage Report
Increment line coverage `46.72% (328/702)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_2391ea7f407dcc42f8f4ddc3bcd8e0838059d15c_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_2391ea7f407dcc42f8f4ddc3bcd8e0838059d15c_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.69% (27820/37751) |
| Line Coverage | 57.59% (301458/523496) |
| Region Coverage | 54.66% (251697/460447) |
| Branch Coverage | 55.86% (108810/194796) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4474166410
# FE UT Coverage Report
Increment line coverage ` 31.19% (92/295)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_ef2529ca3c35ebe1f89cb9fa4212f239fce43a97/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_ef2529ca3c35ebe1f89cb9fa4212f239fce43a97/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4493934116
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4494201625
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1854/2375) |
| Line Coverage | 64.53% (33339/51662) |
| Region Coverage | 65.21% (16524/25341) |
| Branch Coverage | 55.76% (8839/15852) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4505933053
# BE Regression && UT Coverage Report
Increment line coverage `75.28% (603/801)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_6be29ffaf962c85e6c243c68a1c26cf413cdb0b5_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.64% (27896/37883) |
| Line Coverage | 57.55% (303189/526849) |
| Region Coverage | 54.45% (253995/466433) |
| Branch Coverage | 55.95% (110089/196776) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4570705201
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/fb1122e0a26c0dc4fc1defe6bfb9b6b5da020008_fb1122e0a26c0dc4fc1defe6bfb9b6b5da020008_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/fb1122e0a26c0dc4fc1defe6bfb9b6b5da020008_fb1122e0a26c0dc4fc1defe6bfb9b6b5da020008_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 78.06% (1854/2375) |
| Line Coverage | 64.52% (33334/51663) |
| Region Coverage | 65.22% (16528/25343) |
| Branch Coverage | 55.72% (8834/15854) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4571781585
<details>
<summary>TPC-DS: <b>Total hot run time: 170116 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 32e14d6220221b6aa1e1cab5adf12ffaae3e9f5d, data reload: false
query5 4317 621 491 491
query6 438 204 181 181
query7 4833 563 300 300
query8 378 232 211 211
query9 8793 3989 4070 3989
query10 460 325 268 268
query11 5929 2537 2234 2234
query12 163 113 105 105
query13 1329 602 453 453
query14 6412 5429 5159 5159
query14_1 4476 4485 4476 4476
query15 212 207 187 187
query16 1058 473 480 473
query17 1142 719 616 616
query18 2734 497 361 361
query19 224 191 149 149
query20 113 111 104 104
query21 223 143 119 119
query22 13681 13745 13459 13459
query23 17339 16490 16288 16288
query23_1 16339 16326 16378 16326
query24 7655 1785 1359 1359
query24_1 1325 1316 1363 1316
query25 576 466 421 421
query26 1327 337 168 168
query27 2596 571 347 347
query28 4407 2031 2014 2014
query29 1104 638 502 502
query30 309 242 205 205
query31 1187 1081 961 961
query32 113 64 62 62
query33 540 337 271 271
query34 1256 1113 654 654
query35 768 804 711 711
query36 1399 1381 1229 1229
query37 160 104 100 100
query38 3227 3171 3084 3084
query39 944 910 887 887
query39_1 892 907 875 875
query40 218 123 101 101
query41 66 65 63 63
query42 95 97 95 95
query43 331 321 283 283
query44
query45 194 185 181 181
query46 1126 1227 742 742
query47 2370 2419 2206 2206
query48 413 427 266 266
query49 651 469 351 351
query50 1007 347 262 262
query51 4294 4418 4268 4268
query52 90 89 77 77
query53 248 267 200 200
query54 276 225 205 205
query55 77 83 73 73
query56 249 239 225 225
query57 1432 1406 1315 1315
query58 246 217 219 217
query59 1603 1665 1475 1475
query60 281 263 235 235
query61 164 166 163 163
query62 695 650 587 587
query63 224 181 190 181
query64 2541 828 628 628
query65
query66 1750 490 367 367
query67 29828 29087 29595 29087
query68
query69 441 306 275 275
query70 985 982 952 952
query71 301 229 216 216
query72 3035 2742 2481 2481
query73 878 745 437 437
query74 5138 5004 4797 4797
query75 2675 2587 2243 2243
query76 2329 1154 827 827
query77 347 388 290 290
query78 12563 12493 11901 11901
query79 1420 1111 799 799
query80 1303 469 392 392
query81 520 287 244 244
query82 639 161 128 128
query83 375 275 257 257
query84 261 143 116 116
query85 931 583 436 436
query86 420 307 303 303
query87 3432 3372 3244 3244
query88 3694 2780 2745 2745
query89 443 375 331 331
query90 1925 184 183 183
query91 187 175 142 142
query92 67 63 57 57
query93 1698 1466 860 860
query94 720 358 313 313
query95 692 390 426 390
query96 1037 777 354 354
query97 2691 2707 2552 2552
query98 211 206 203 203
query99 1190 1175 1038 1038
Total cold run time: 253763 ms
Total hot run time: 170116 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4572824526
# FE Regression Coverage Report
Increment line coverage ` 72.96% (224/307)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_fb1122e0a26c0dc4fc1defe6bfb9b6b5da020008_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_fb1122e0a26c0dc4fc1defe6bfb9b6b5da020008_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4543853212
# FE Regression Coverage Report
Increment line coverage ` 48.25% (220/456)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_3d38dc5ab813ed88625762426d63d5324bfff4fe_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_3d38dc5ab813ed88625762426d63d5324bfff4fe_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4539721621
# FE UT Coverage Report
Increment line coverage ` 32.00% (96/300)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_f779760a4db0a4e1f4605c63a8a9bc21f51af126/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_f779760a4db0a4e1f4605c63a8a9bc21f51af126/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4449905242
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #62589:
URL: https://github.com/apache/doris/pull/62589#discussion_r3331440040
##########
be/src/exec/runtime_filter/runtime_filter_partition_pruner.cpp:
##########
@@ -0,0 +1,866 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/runtime_filter/runtime_filter_partition_pruner.h"
+
+#include <gen_cpp/PlanNodes_types.h>
+
+#include <optional>
+#include <unordered_set>
+#include <utility>
+
+#include "core/block/block.h"
+#include "core/column/column.h"
+#include "core/column/column_nullable.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/field.h"
+#include "exprs/bloom_filter_func.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/runtime_filter_expr.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/descriptors.h"
+
+namespace doris {
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+// Complexity is inflated by macro expansion for each PrimitiveType case.
+Status ParsedPartitionBoundaries::parse(
+ const std::vector<TPartitionBoundary>& boundaries,
+ const phmap::flat_hash_map<int, SlotDescriptor*>& slot_descs) {
+ for (const auto& tb : boundaries) {
+ DORIS_CHECK(tb.__isset.partition_id);
+ DORIS_CHECK(tb.__isset.slot_id);
+ SlotId slot_id = tb.slot_id;
+
+ auto slot_it = slot_descs.find(slot_id);
+ DORIS_CHECK(slot_it != slot_descs.end());
+ SlotDescriptor* slot = slot_it->second;
+ // Reuse the slot's pre-built DataType: walking through VLiteral here
+ // would cost a `DataTypeFactory::create_data_type(node)` heap allocation
+ // and a one-row `ColumnConst` allocation per boundary endpoint. With
+ // thousands of partitions that dominates BuildTasksTime.
+ const DataTypePtr& slot_type = slot->type();
+ PrimitiveType ptype = slot_type->get_primitive_type();
+ int precision = cast_set<int>(slot_type->get_precision());
+ int scale = cast_set<int>(slot_type->get_scale());
+ bool is_nullable = slot->is_nullable();
+
+ // Store slot data type for potential projection use
+ _slot_data_types[slot_id] = slot_type;
+
+ ParsedBoundary boundary;
+ boundary.partition_id = tb.partition_id;
+ boundary.slot_id = slot_id;
+ boundary.is_nullable = is_nullable;
+
+ bool parsed_ok = false;
+
+#define BUILD_BOUNDARY_CVR(NAME) \
+ case TYPE_##NAME: { \
+ using CppType = typename PrimitiveTypeTraits<TYPE_##NAME>::CppType; \
+ bool is_list = tb.__isset.list_values && !tb.list_values.empty(); \
+ bool is_range = tb.__isset.range_start || tb.__isset.range_end; \
+ DORIS_CHECK(is_list || is_range); \
+ ColumnValueRange<TYPE_##NAME> cvr(slot->col_name(), is_nullable, precision, scale); \
+ /* Returns nullopt if `node` is a NULL literal; the caller then sets contain_null */ \
+ /* on the CVR instead of trying to extract a typed value (which would dereference */ \
+ /* a null data pointer for the non-string branch). */ \
+ auto parse_texpr_node = [&](const TExprNode& node) -> std::optional<CppType> { \
+ if (node.node_type == TExprNodeType::NULL_LITERAL) { \
+ return std::nullopt; \
+ } \
+ /* `Field` value is copied into the CVR by `add_fixed_value` / */ \
+ /* `add_range` (both take CppType by const-ref / by value), so the */ \
+ /* temporary `Field`'s lifetime ending at this expression's full-statement */ \
+ /* boundary is safe -- including for `String` payloads. */ \
+ Field field = slot_type->get_field(node); \
+ return std::make_optional<CppType>(field.get<TYPE_##NAME>()); \
+ }; \
+ if (is_list) { \
+ auto empty_cvr = ColumnValueRange<TYPE_##NAME>::create_empty_column_value_range( \
+ is_nullable, precision, scale); \
+ bool list_has_null = false; \
+ bool list_has_value = false; \
+ for (const auto& node : tb.list_values) { \
+ auto parsed = parse_texpr_node(node); \
+ if (!parsed) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ static_cast<void>(empty_cvr.add_fixed_value(*parsed)); \
+ list_has_value = true; \
+ } \
+ if (list_has_value) { \
+ cvr.intersection(empty_cvr); \
+ } \
+ if (list_has_null && is_nullable) { \
+ /* Track NULL membership on ParsedBoundary; calling */ \
+ /* cvr.set_contain_null(true) here would invoke */ \
+ /* set_empty_value_range() and discard the concrete fixed */ \
+ /* values we just inserted, turning {NULL, v} into a */ \
+ /* NULL-only boundary. */ \
+ boundary.contains_null = true; \
+ if (!list_has_value) { \
+ boundary.only_null = true; \
+ } \
+ } \
+ } else { \
+ bool range_has_null = false; \
+ if (is_nullable && !tb.__isset.range_start) { \
+ range_has_null = true; \
+ } \
+ if (tb.__isset.range_start) { \
+ auto parsed = parse_texpr_node(tb.range_start); \
+ if (parsed) { \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, *parsed)); \
+ } else if (is_nullable) { \
+ range_has_null = true; \
+ } \
+ } \
+ if (tb.__isset.range_end) { \
+ auto parsed = parse_texpr_node(tb.range_end); \
+ if (parsed) { \
+ /* Multi-column RANGE projection emits a CLOSED upper bound (see */ \
+ /* TPartitionBoundary.range_end_inclusive comment); single-column RANGE */ \
+ /* keeps the natural OPEN upper bound matching Doris semantics. */ \
+ SQLFilterOp upper_op = \
+ (tb.__isset.range_end_inclusive && tb.range_end_inclusive) \
+ ? FILTER_LESS_OR_EQUAL \
+ : FILTER_LESS; \
+ static_cast<void>(cvr.add_range(upper_op, *parsed)); \
+ } \
+ } \
+ if (range_has_null) { \
+ boundary.contains_null = true; \
+ } \
+ } \
+ boundary.boundary_cvr = std::move(cvr); \
+ parsed_ok = true; \
+ break; \
+ }
+
+ switch (ptype) {
+ BUILD_BOUNDARY_CVR(TINYINT)
+ BUILD_BOUNDARY_CVR(SMALLINT)
+ BUILD_BOUNDARY_CVR(INT)
+ BUILD_BOUNDARY_CVR(BIGINT)
+ BUILD_BOUNDARY_CVR(LARGEINT)
+ BUILD_BOUNDARY_CVR(FLOAT)
+ BUILD_BOUNDARY_CVR(DOUBLE)
+ BUILD_BOUNDARY_CVR(CHAR)
+ BUILD_BOUNDARY_CVR(DATE)
+ BUILD_BOUNDARY_CVR(DATETIME)
+ BUILD_BOUNDARY_CVR(DATEV2)
+ BUILD_BOUNDARY_CVR(DATETIMEV2)
+ BUILD_BOUNDARY_CVR(TIMESTAMPTZ)
+ BUILD_BOUNDARY_CVR(VARCHAR)
+ BUILD_BOUNDARY_CVR(STRING)
+ BUILD_BOUNDARY_CVR(DECIMAL32)
+ BUILD_BOUNDARY_CVR(DECIMAL64)
+ BUILD_BOUNDARY_CVR(DECIMAL128I)
+ BUILD_BOUNDARY_CVR(DECIMAL256)
+ BUILD_BOUNDARY_CVR(DECIMALV2)
+ BUILD_BOUNDARY_CVR(BOOLEAN)
+ BUILD_BOUNDARY_CVR(IPV4)
+ BUILD_BOUNDARY_CVR(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_BOUNDARY_CVR
+
+ DORIS_CHECK(parsed_ok);
+ _slot_to_boundaries[slot_id].push_back(std::move(boundary));
+ }
+
+ // Count distinct partition IDs across all boundaries.
+ if (!_slot_to_boundaries.empty()) {
+ phmap::flat_hash_set<int64_t> all_partition_ids;
+ for (const auto& [_, slot_boundaries] : _slot_to_boundaries) {
+ for (const auto& pb : slot_boundaries) {
+ all_partition_ids.insert(pb.partition_id);
+ }
+ }
+ _total_partition_count = static_cast<int64_t>(all_partition_ids.size());
+ }
+ return Status::OK();
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static bool find_unique_slot_ref_impl(const VExpr* expr, const VSlotRef** found) {
+ if (!expr) {
+ return true;
+ }
+ if (expr->is_slot_ref()) {
+ const auto* slot = assert_cast<const VSlotRef*>(expr);
+ if (*found == nullptr) {
+ *found = slot;
+ return true;
+ }
+ if ((*found)->slot_id() != slot->slot_id()) {
+ return false;
+ }
+ DORIS_CHECK((*found)->column_id() == slot->column_id());
+ return true;
+ }
+ for (const auto& child : expr->children()) {
+ if (!find_unique_slot_ref_impl(child.get(), found)) {
+ return false;
+ }
+ }
+ return true;
+}
+
+static const VSlotRef* find_unique_slot_ref(const VExpr* expr) {
+ const VSlotRef* found = nullptr;
+ if (!find_unique_slot_ref_impl(expr, &found)) {
+ return nullptr;
+ }
+ return found;
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+Status ParsedPartitionBoundaries::get_or_compute_projected_boundaries(
+ int filter_id, const VExprSPtr& target_expr, SlotId leaf_slot_id, int leaf_column_id,
+ const std::unordered_map<int64_t, TTargetExprMonotonicity::type>& partition_directions,
+ VExprContext* ctx, std::shared_ptr<const std::vector<ParsedBoundary>>* output) const {
+ {
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto it = _projected_boundaries_by_filter_id.find(filter_id);
+ if (it != _projected_boundaries_by_filter_id.end()) {
+ *output = it->second;
+ return Status::OK();
+ }
+ }
+
+ std::vector<ParsedBoundary> projected;
+ auto store_projected_boundaries = [&](std::vector<ParsedBoundary>&& boundaries) {
+ auto cached = std::make_shared<const std::vector<ParsedBoundary>>(std::move(boundaries));
+ std::lock_guard<std::mutex> lock(_projected_boundaries_mutex);
+ auto [it, inserted] = _projected_boundaries_by_filter_id.emplace(filter_id, cached);
+ *output = inserted ? cached : it->second;
+ return Status::OK();
+ };
+
+ auto slot_boundaries_it = _slot_to_boundaries.find(leaf_slot_id);
+ if (slot_boundaries_it == _slot_to_boundaries.end()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ const auto& orig_boundaries = slot_boundaries_it->second;
+ if (orig_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::unordered_set<int64_t> boundary_partition_ids;
+ boundary_partition_ids.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ boundary_partition_ids.insert(boundary.partition_id);
+ }
+ for (const auto& [partition_id, direction] : partition_directions) {
+ DORIS_CHECK(direction != TTargetExprMonotonicity::NON_MONOTONIC);
+ DORIS_CHECK(boundary_partition_ids.contains(partition_id));
+ }
+
+ if (target_expr->is_slot_ref()) {
+ auto* slot_ref = assert_cast<VSlotRef*>(target_expr.get());
+ DORIS_CHECK(slot_ref->slot_id() == leaf_slot_id);
+ std::vector<ParsedBoundary> slot_boundaries;
+ slot_boundaries.reserve(orig_boundaries.size());
+ for (const auto& boundary : orig_boundaries) {
+ auto direction_it = partition_directions.find(boundary.partition_id);
+ if (direction_it == partition_directions.end()) {
+ continue;
+ }
+ DORIS_CHECK(direction_it->second == TTargetExprMonotonicity::MONOTONIC_INCREASING);
+ slot_boundaries.emplace_back(boundary);
+ }
+ return store_projected_boundaries(std::move(slot_boundaries));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> selected_boundaries;
+ selected_boundaries.reserve(orig_boundaries.size());
+ for (size_t i = 0; i < orig_boundaries.size(); ++i) {
+ auto it = partition_directions.find(orig_boundaries[i].partition_id);
+ if (it != partition_directions.end() &&
+ it->second != TTargetExprMonotonicity::NON_MONOTONIC) {
+ selected_boundaries.emplace_back(i, it->second);
+ }
+ }
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ std::vector<std::pair<size_t, TTargetExprMonotonicity::type>> projectable_boundaries;
+ projectable_boundaries.reserve(selected_boundaries.size());
+ for (const auto& selected_boundary : selected_boundaries) {
+ const auto& boundary = orig_boundaries[selected_boundary.first];
+ if (!boundary.only_null && !boundary.contains_null) {
+ projectable_boundaries.emplace_back(selected_boundary);
+ }
+ }
+ selected_boundaries.swap(projectable_boundaries);
+ if (selected_boundaries.empty()) {
+ return store_projected_boundaries(std::move(projected));
+ }
+
+ auto slot_type_it = _slot_data_types.find(leaf_slot_id);
+ DORIS_CHECK(slot_type_it != _slot_data_types.end());
+
+ const DataTypePtr& input_type = slot_type_it->second;
+ bool input_is_nullable = input_type->is_nullable();
+ DataTypePtr inner_type = input_is_nullable ? remove_nullable(input_type) : input_type;
+ PrimitiveType input_ptype = input_type->get_primitive_type();
+
+ size_t N = selected_boundaries.size();
+ std::vector<bool> lo_open(N, false);
+ std::vector<bool> hi_open(N, false);
+ std::vector<bool> boundary_is_list(N, false);
+ std::vector<size_t> lo_result_row(N, 0);
+ std::vector<size_t> hi_result_row(N, 0);
+ std::vector<size_t> list_result_begin(N, 0);
+ std::vector<size_t> list_result_end(N, 0);
+ size_t lo_row_count = 0;
+ size_t hi_row_count = 0;
+ size_t list_row_count = 0;
+
+ // Build input blocks for RANGE endpoints and LIST values.
+ Block lo_block;
+ Block hi_block;
+ Block list_block;
+
+ // Pad columns 0..leaf_column_id-1 with placeholder columns so the leaf
+ // VSlotRef (whose column_id is leaf_column_id) reads our typed column.
+ auto add_dummy_columns = [&](Block& block, size_t row_count) {
+ for (int col_idx = 0; col_idx < leaf_column_id; ++col_idx) {
+ auto col = ColumnUInt8::create(row_count, 0);
+ block.insert({std::move(col), std::make_shared<DataTypeUInt8>(),
+ fmt::format("dummy_{}", col_idx)});
+ }
+ };
+
+ // Macro-dispatch on input PrimitiveType to build finite RANGE endpoint
+ // columns and LIST value columns. Unbounded RANGE endpoints are not executed
+ // through target_expr; they stay unbounded after projection and swap sides
+ // for monotonic decreasing targets.
+ bool input_built = false;
+#define BUILD_INPUT_COLUMNS(INPUT_PT) \
+ case TYPE_##INPUT_PT: { \
+ using InCol = typename PrimitiveTypeTraits<TYPE_##INPUT_PT>::ColumnType; \
+ MutableColumnPtr lo_inner_base = inner_type->create_column(); \
+ MutableColumnPtr hi_inner_base = inner_type->create_column(); \
+ MutableColumnPtr list_inner_base = inner_type->create_column(); \
+ auto* lo_inner = assert_cast<InCol*>(lo_inner_base.get()); \
+ auto* hi_inner = assert_cast<InCol*>(hi_inner_base.get()); \
+ auto* list_inner = assert_cast<InCol*>(list_inner_base.get()); \
+ lo_inner->reserve(N); \
+ hi_inner->reserve(N); \
+ list_inner->reserve(N); \
+ auto lo_nulls = ColumnUInt8::create(); \
+ auto hi_nulls = ColumnUInt8::create(); \
+ auto list_nulls = ColumnUInt8::create(); \
+ lo_nulls->reserve(N); \
+ hi_nulls->reserve(N); \
+ list_nulls->reserve(N); \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& boundary = orig_boundaries[selected_boundaries[i].first]; \
+ const auto* cvr = \
+ std::get_if<ColumnValueRange<TYPE_##INPUT_PT>>(&boundary.boundary_cvr); \
+ DCHECK(cvr != nullptr); \
+ DCHECK(!boundary.only_null); \
+ DCHECK(!boundary.contains_null); \
+ boundary_is_list[i] = cvr->is_fixed_value_range(); \
+ if (boundary_is_list[i]) { \
+ list_result_begin[i] = list_row_count; \
+ for (const auto& value : cvr->get_fixed_value_set()) { \
+ list_inner->insert_value(value); \
+ list_nulls->get_data().push_back(0); \
+ ++list_row_count; \
+ } \
+ list_result_end[i] = list_row_count; \
+ continue; \
+ } \
+ lo_open[i] = cvr->is_low_value_minimum(); \
+ hi_open[i] = cvr->is_high_value_maximum(); \
+ if (!lo_open[i]) { \
+ lo_result_row[i] = lo_row_count++; \
+ lo_inner->insert_value(cvr->get_range_min_value()); \
+ lo_nulls->get_data().push_back(0); \
+ } \
+ if (!hi_open[i]) { \
+ hi_result_row[i] = hi_row_count++; \
+ hi_inner->insert_value(cvr->get_range_max_value()); \
+ hi_nulls->get_data().push_back(0); \
+ } \
+ } \
+ add_dummy_columns(lo_block, lo_row_count); \
+ add_dummy_columns(hi_block, hi_row_count); \
+ add_dummy_columns(list_block, list_row_count); \
+ if (input_is_nullable) { \
+ auto lo_col = ColumnNullable::create(std::move(lo_inner_base), std::move(lo_nulls)); \
+ auto hi_col = ColumnNullable::create(std::move(hi_inner_base), std::move(hi_nulls)); \
+ auto list_col = \
+ ColumnNullable::create(std::move(list_inner_base), std::move(list_nulls)); \
+ lo_block.insert({std::move(lo_col), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_col), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_col), input_type, "leaf_slot"}); \
+ } else { \
+ lo_block.insert({std::move(lo_inner_base), input_type, "leaf_slot"}); \
+ hi_block.insert({std::move(hi_inner_base), input_type, "leaf_slot"}); \
+ list_block.insert({std::move(list_inner_base), input_type, "leaf_slot"}); \
+ } \
+ input_built = true; \
+ break; \
+ }
+
+ switch (input_ptype) {
+ BUILD_INPUT_COLUMNS(TINYINT)
+ BUILD_INPUT_COLUMNS(SMALLINT)
+ BUILD_INPUT_COLUMNS(INT)
+ BUILD_INPUT_COLUMNS(BIGINT)
+ BUILD_INPUT_COLUMNS(LARGEINT)
+ BUILD_INPUT_COLUMNS(FLOAT)
+ BUILD_INPUT_COLUMNS(DOUBLE)
+ BUILD_INPUT_COLUMNS(CHAR)
+ BUILD_INPUT_COLUMNS(DATE)
+ BUILD_INPUT_COLUMNS(DATETIME)
+ BUILD_INPUT_COLUMNS(DATEV2)
+ BUILD_INPUT_COLUMNS(DATETIMEV2)
+ BUILD_INPUT_COLUMNS(TIMESTAMPTZ)
+ BUILD_INPUT_COLUMNS(VARCHAR)
+ BUILD_INPUT_COLUMNS(STRING)
+ BUILD_INPUT_COLUMNS(DECIMAL32)
+ BUILD_INPUT_COLUMNS(DECIMAL64)
+ BUILD_INPUT_COLUMNS(DECIMAL128I)
+ BUILD_INPUT_COLUMNS(DECIMAL256)
+ BUILD_INPUT_COLUMNS(DECIMALV2)
+ BUILD_INPUT_COLUMNS(BOOLEAN)
+ BUILD_INPUT_COLUMNS(IPV4)
+ BUILD_INPUT_COLUMNS(IPV6)
+ default:
+ break;
+ }
+#undef BUILD_INPUT_COLUMNS
+
+ DORIS_CHECK(input_built);
+
+ int lo_result_id = -1;
+ int hi_result_id = -1;
+ int list_result_id = -1;
+ ColumnPtr lo_result_col;
+ ColumnPtr hi_result_col;
+ ColumnPtr list_result_col;
+ if (lo_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &lo_block, &lo_result_id));
+ DORIS_CHECK(lo_result_id >= 0);
+ lo_result_col = lo_block.get_by_position(lo_result_id).column;
+ }
+ if (hi_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &hi_block, &hi_result_id));
+ DORIS_CHECK(hi_result_id >= 0);
+ hi_result_col = hi_block.get_by_position(hi_result_id).column;
+ }
+ if (list_row_count > 0) {
+ RETURN_IF_ERROR(target_expr->execute(ctx, &list_block, &list_result_id));
+ DORIS_CHECK(list_result_id >= 0);
+ list_result_col = list_block.get_by_position(list_result_id).column;
+ }
+ int out_precision = cast_set<int>(target_expr->data_type()->get_precision());
+ int out_scale = cast_set<int>(target_expr->data_type()->get_scale());
+ bool out_nullable = target_expr->data_type()->is_nullable();
+ PrimitiveType output_ptype = target_expr->data_type()->get_primitive_type();
+
+ projected.reserve(N);
+
+ bool output_built = false;
+#define BUILD_PROJECTED_CVR(OUTPUT_PT) \
+ case TYPE_##OUTPUT_PT: { \
+ using OutputCppType = typename PrimitiveTypeTraits<TYPE_##OUTPUT_PT>::CppType; \
+ auto projected_value = [](const ColumnPtr& column, size_t row) -> OutputCppType { \
+ Field field = (*column)[row]; \
+ return field.get<TYPE_##OUTPUT_PT>(); \
+ }; \
+ for (size_t i = 0; i < N; ++i) { \
+ const auto& orig_boundary = orig_boundaries[selected_boundaries[i].first]; \
+ TTargetExprMonotonicity::type direction = selected_boundaries[i].second; \
+ if (boundary_is_list[i]) { \
+ auto cvr = ColumnValueRange<TYPE_##OUTPUT_PT>::create_empty_column_value_range( \
+ out_nullable, out_precision, out_scale); \
+ bool list_has_value = false; \
+ bool list_has_null = false; \
+ for (size_t row = list_result_begin[i]; row < list_result_end[i]; ++row) { \
+ if (list_result_col->is_null_at(row)) { \
+ list_has_null = true; \
+ continue; \
+ } \
+ auto projected_list_value = projected_value(list_result_col, row); \
+ static_cast<void>(cvr.add_fixed_value(projected_list_value)); \
+ list_has_value = true; \
+ } \
+ if (!list_has_value && !list_has_null) { \
+ continue; \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.contains_null = list_has_null; \
+ projected_boundary.only_null = list_has_null && !list_has_value; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ continue; \
+ } \
+ if ((!lo_open[i] && lo_result_col->is_null_at(lo_result_row[i])) || \
+ (!hi_open[i] && hi_result_col->is_null_at(hi_result_row[i]))) { \
+ continue; \
+ } \
+ ColumnValueRange<TYPE_##OUTPUT_PT> cvr("", out_nullable, out_precision, out_scale); \
+ /* Use closed projected finite bounds as a conservative envelope for */ \
+ /* non-strict monotonic expressions such as date ceil/floor/cast. */ \
+ if (direction == TTargetExprMonotonicity::MONOTONIC_DECREASING) { \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, hi_projected)); \
+ } \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, lo_projected)); \
+ } \
+ } else { \
+ if (!lo_open[i]) { \
+ auto lo_projected = projected_value(lo_result_col, lo_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LARGER_OR_EQUAL, lo_projected)); \
+ } \
+ if (!hi_open[i]) { \
+ auto hi_projected = projected_value(hi_result_col, hi_result_row[i]); \
+ static_cast<void>(cvr.add_range(FILTER_LESS_OR_EQUAL, hi_projected)); \
+ } \
+ } \
+ ParsedBoundary projected_boundary; \
+ projected_boundary.partition_id = orig_boundary.partition_id; \
+ projected_boundary.slot_id = leaf_slot_id; \
+ projected_boundary.is_nullable = out_nullable; \
+ projected_boundary.only_null = orig_boundary.only_null; \
+ projected_boundary.contains_null = orig_boundary.contains_null; \
+ projected_boundary.boundary_cvr = std::move(cvr); \
+ projected.emplace_back(std::move(projected_boundary)); \
+ } \
+ output_built = true; \
+ break; \
+ }
+
+ switch (output_ptype) {
+ BUILD_PROJECTED_CVR(TINYINT)
+ BUILD_PROJECTED_CVR(SMALLINT)
+ BUILD_PROJECTED_CVR(INT)
+ BUILD_PROJECTED_CVR(BIGINT)
+ BUILD_PROJECTED_CVR(LARGEINT)
+ BUILD_PROJECTED_CVR(FLOAT)
+ BUILD_PROJECTED_CVR(DOUBLE)
+ BUILD_PROJECTED_CVR(CHAR)
+ BUILD_PROJECTED_CVR(DATE)
+ BUILD_PROJECTED_CVR(DATETIME)
+ BUILD_PROJECTED_CVR(DATEV2)
+ BUILD_PROJECTED_CVR(DATETIMEV2)
+ BUILD_PROJECTED_CVR(TIMESTAMPTZ)
+ BUILD_PROJECTED_CVR(VARCHAR)
+ BUILD_PROJECTED_CVR(STRING)
+ BUILD_PROJECTED_CVR(DECIMAL32)
+ BUILD_PROJECTED_CVR(DECIMAL64)
+ BUILD_PROJECTED_CVR(DECIMAL128I)
+ BUILD_PROJECTED_CVR(DECIMAL256)
+ BUILD_PROJECTED_CVR(DECIMALV2)
+ BUILD_PROJECTED_CVR(BOOLEAN)
+ BUILD_PROJECTED_CVR(IPV4)
+ BUILD_PROJECTED_CVR(IPV6)
+ default:
+ break;
+ }
+
+#undef BUILD_PROJECTED_CVR
+
+ DORIS_CHECK(output_built);
+
+ return store_projected_boundaries(std::move(projected));
+}
+// NOLINTEND(readability-function-cognitive-complexity,readability-function-size)
+
+static SQLFilterOp convert_opcode_to_filter_op(TExprOpcode::type op) {
+ switch (op) {
+ case TExprOpcode::LE:
+ return FILTER_LESS_OR_EQUAL;
+ case TExprOpcode::LT:
+ return FILTER_LESS;
+ case TExprOpcode::GE:
+ return FILTER_LARGER_OR_EQUAL;
+ case TExprOpcode::GT:
+ return FILTER_LARGER;
+ default:
+ return FILTER_IN; // sentinel: caller should skip
+ }
+}
+
+// NOLINTBEGIN(readability-function-cognitive-complexity,readability-function-size)
+void RuntimeFilterPartitionPruner::_try_prune_by_single_rf(
+ const std::vector<ParsedBoundary>& boundaries, const VExprSPtr& impl,
+ phmap::flat_hash_set<int64_t>& newly_pruned) {
+ // Pre-compute whether the RF "matches NULL" -- i.e. whether the RF would
+ // accept a row whose probe value is NULL. The signal differs by RF impl:
+ // * IN filter → HybridSet::contain_null()
+ // * Bloom filter → BloomFilterFuncBase::contain_null()
+ // * MinMax filter → encoded via the BinaryPredicate node_type:
+ // NULL_AWARE_BINARY_PRED means the build side had a
+ // NULL key and a null-safe equal join asked NULL to
+ // match NULL; plain BINARY_PRED never matches NULL.
+ // FilterBase::contain_null() already folds in `_null_aware`, so we only
+ // get a true result when the build side is actually null-aware AND
+ // produced a NULL value.
+ bool rf_contains_null = false;
+ if (auto hybrid_set = impl->get_set_func()) {
+ rf_contains_null = hybrid_set->contain_null();
+ } else if (impl->node_type() == TExprNodeType::BLOOM_PRED) {
+ auto bloom = impl->get_bloom_filter_func();
+ rf_contains_null = bloom && bloom->contain_null();
+ } else if (impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) {
+ // Min/Max RF built on a null-safe equal join. The literal child holds
+ // the min or max bound; the NULL semantic is conveyed by the node
+ // type itself (see create_vbin_predicate in runtime_filter/utils.cpp).
+ rf_contains_null = true;
+ }
+
+ std::optional<ColumnValueRangeType> rf_cvr;
+ if (!boundaries.empty()) {
+ std::visit(
+ [&](const auto& boundary_cvr) {
+ using CvrType = std::decay_t<decltype(boundary_cvr)>;
+ using CppType = typename CvrType::CppType;
+
+ auto hybrid_set = impl->get_set_func();
+ if (hybrid_set) {
+ auto typed_rf_cvr = CvrType::create_empty_column_value_range(
+ boundaries.front().is_nullable, boundary_cvr.precision(),
+ boundary_cvr.scale());
+ auto* iter = hybrid_set->begin();
+ while (iter->has_next()) {
+ const void* value = iter->get_value();
+ if (value) {
+ if constexpr (std::is_same_v<CppType, String>) {
+ // String HybridSets store StringRef*, but
+ // ColumnValueRange<String>::CppType is
+ // std::string. Construct from the bytes.
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else if constexpr (std::is_same_v<CppType, StringRef>) {
+ const auto* str_val = reinterpret_cast<const StringRef*>(value);
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ CppType(str_val->data, str_val->size)));
+ } else {
+ static_cast<void>(typed_rf_cvr.add_fixed_value(
+ *reinterpret_cast<const CppType*>(value)));
+ }
+ }
+ iter->next();
+ }
+ rf_cvr.emplace(std::move(typed_rf_cvr));
+ } else if ((impl->node_type() == TExprNodeType::BINARY_PRED ||
+ impl->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) &&
+ impl->children().size() == 2 && impl->children()[1]->is_literal()) {
+ // MinMax filter: binary pred with literal bound
+ auto* literal = assert_cast<VLiteral*>(impl->children()[1].get());
+ auto col_ptr = literal->get_column_ptr();
+ auto data = col_ptr->get_data_at(0);
+ CppType val {};
+ if constexpr (std::is_same_v<CppType, String>) {
+ // get_data_at returns a StringRef pointing at the
+ // raw character bytes; ColumnValueRange<String>'s
+ // CppType is std::string, so construct one from
+ // the bytes rather than dereferencing as String.
+ val = CppType(data.data, data.size);
+ } else if constexpr (std::is_same_v<CppType, StringRef>) {
+ val = CppType(data.data, data.size);
+ } else {
+ val = *reinterpret_cast<const CppType*>(data.data);
+ }
+
+ SQLFilterOp op = convert_opcode_to_filter_op(impl->op());
Review Comment:
改成用optional表示非法opcode
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4450398757
<details>
<summary>TPC-H: <b>Total hot run time: 29635 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c090ab6440346db9f2c052585ec7b6dc593d0df3, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17707 3885 3958 3885
q2 q3 10706 878 610 610
q4 4671 456 348 348
q5 7439 1325 1141 1141
q6 185 167 136 136
q7 912 962 745 745
q8 9505 1384 1283 1283
q9 6148 5442 5372 5372
q10 6327 2074 1823 1823
q11 476 271 259 259
q12 683 411 291 291
q13 18209 3343 2744 2744
q14 303 284 265 265
q15 q16 908 871 788 788
q17 1203 1061 767 767
q18 6817 5685 5611 5611
q19 1661 1160 1114 1114
q20 540 409 267 267
q21 4499 2294 1885 1885
q22 424 355 301 301
Total cold run time: 99323 ms
Total hot run time: 29635 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4277 4283 4210 4210
q2 q3 4640 4769 4207 4207
q4 2095 2182 1402 1402
q5 4976 4902 5126 4902
q6 210 163 131 131
q7 2051 2207 1653 1653
q8 3449 3143 3125 3125
q9 8599 8460 8485 8460
q10 4538 4495 4233 4233
q11 652 439 411 411
q12 684 742 531 531
q13 3234 3514 3002 3002
q14 309 312 287 287
q15 q16 782 785 709 709
q17 1334 1311 1314 1311
q18 8281 7100 7157 7100
q19 1166 1170 1168 1168
q20 2291 2234 1945 1945
q21 6304 5364 4926 4926
q22 556 509 420 420
Total cold run time: 60428 ms
Total hot run time: 54133 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4450460751
<details>
<summary>TPC-DS: <b>Total hot run time: 170469 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c090ab6440346db9f2c052585ec7b6dc593d0df3, data reload: false
query5 4316 638 478 478
query6 433 192 181 181
query7 4801 556 305 305
query8 366 213 202 202
query9 8795 4065 4075 4065
query10 450 317 257 257
query11 6130 2366 2222 2222
query12 163 111 99 99
query13 1281 633 435 435
query14 7272 5325 4967 4967
query14_1 4306 4277 4275 4275
query15 198 196 172 172
query16 1025 455 437 437
query17 1228 713 594 594
query18 2719 477 347 347
query19 201 181 139 139
query20 110 109 103 103
query21 214 136 117 117
query22 13582 13977 14451 13977
query23 17298 16428 16249 16249
query23_1 16346 16282 16368 16282
query24 8256 1811 1367 1367
query24_1 1392 1378 1340 1340
query25 598 494 431 431
query26 1287 321 163 163
query27 2674 618 328 328
query28 4367 1990 1959 1959
query29 1087 622 511 511
query30 299 243 195 195
query31 1113 1072 937 937
query32 107 60 57 57
query33 535 357 265 265
query34 1134 1124 607 607
query35 736 807 658 658
query36 1348 1343 1158 1158
query37 151 104 91 91
query38 3190 3106 3102 3102
query39 944 933 884 884
query39_1 873 888 894 888
query40 232 135 113 113
query41 70 67 67 67
query42 92 94 95 94
query43 317 315 274 274
query44
query45 205 178 185 178
query46 1070 1187 736 736
query47 2354 2279 2228 2228
query48 402 395 301 301
query49 659 485 384 384
query50 698 282 227 227
query51 4425 4292 4195 4195
query52 86 86 76 76
query53 249 273 187 187
query54 260 234 198 198
query55 77 74 69 69
query56 229 215 214 214
query57 1391 1377 1284 1284
query58 243 215 202 202
query59 1593 1613 1468 1468
query60 291 258 246 246
query61 160 156 169 156
query62 660 613 555 555
query63 229 184 189 184
query64 2511 824 669 669
query65
query66 1779 487 360 360
query67 30151 29934 29830 29830
query68
query69 430 307 275 275
query70 966 941 936 936
query71 298 221 214 214
query72 2850 2790 2429 2429
query73 833 755 428 428
query74 5032 4912 4734 4734
query75 2753 2615 2279 2279
query76 2323 1126 757 757
query77 353 389 288 288
query78 12950 12907 12379 12379
query79 1425 1017 716 716
query80 1318 497 407 407
query81 521 282 234 234
query82 625 157 126 126
query83 313 273 244 244
query84 261 139 110 110
query85 904 508 451 451
query86 434 305 287 287
query87 3395 3335 3150 3150
query88 3585 2718 2682 2682
query89 428 368 323 323
query90 2023 187 179 179
query91 184 174 137 137
query92 66 60 56 56
query93 1111 986 558 558
query94 718 345 298 298
query95 665 479 340 340
query96 1021 829 335 335
query97 2665 2667 2528 2528
query98 207 207 208 207
query99 1120 1104 980 980
Total cold run time: 253632 ms
Total hot run time: 170469 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4450448196
<details>
<summary>TPC-DS: <b>Total hot run time: 169794 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c090ab6440346db9f2c052585ec7b6dc593d0df3, data reload: false
query5 4338 619 472 472
query6 453 208 188 188
query7 4847 565 298 298
query8 379 223 211 211
query9 8817 4165 4130 4130
query10 444 315 277 277
query11 6060 2349 2196 2196
query12 167 103 100 100
query13 1272 597 446 446
query14 7375 5312 5005 5005
query14_1 4324 4319 4310 4310
query15 203 191 175 175
query16 1046 427 458 427
query17 1302 738 623 623
query18 2745 500 366 366
query19 330 187 150 150
query20 119 109 106 106
query21 239 154 137 137
query22 13776 13883 14441 13883
query23 17461 16625 16170 16170
query23_1 16309 16255 16170 16170
query24 7710 1751 1416 1416
query24_1 1315 1343 1372 1343
query25 603 486 395 395
query26 1355 302 157 157
query27 2563 613 333 333
query28 4274 1966 1952 1952
query29 1103 607 505 505
query30 291 228 193 193
query31 1103 1064 937 937
query32 111 62 57 57
query33 508 307 247 247
query34 1166 1167 643 643
query35 747 753 675 675
query36 1292 1325 1208 1208
query37 154 98 92 92
query38 3196 3140 3036 3036
query39 936 913 892 892
query39_1 867 874 898 874
query40 237 123 112 112
query41 62 63 59 59
query42 93 90 90 90
query43 312 312 270 270
query44
query45 190 186 177 177
query46 1075 1210 729 729
query47 2318 2312 2199 2199
query48 408 419 286 286
query49 629 482 390 390
query50 702 296 225 225
query51 4294 4391 4155 4155
query52 89 88 75 75
query53 245 264 186 186
query54 277 223 196 196
query55 77 74 71 71
query56 238 236 218 218
query57 1421 1388 1298 1298
query58 257 207 209 207
query59 1567 1636 1459 1459
query60 293 257 246 246
query61 158 157 149 149
query62 660 607 559 559
query63 229 183 185 183
query64 2504 773 659 659
query65
query66 1761 492 354 354
query67 30095 29922 29819 29819
query68
query69 427 297 271 271
query70 944 945 947 945
query71 309 221 208 208
query72 2934 2789 2486 2486
query73 803 765 422 422
query74 5125 4916 4730 4730
query75 2747 2652 2298 2298
query76 2324 1149 786 786
query77 360 388 298 298
query78 12906 12918 12318 12318
query79 1497 954 725 725
query80 1364 511 410 410
query81 535 274 241 241
query82 679 160 127 127
query83 341 273 246 246
query84 280 139 112 112
query85 920 496 426 426
query86 438 307 283 283
query87 3393 3331 3213 3213
query88 3532 2704 2694 2694
query89 418 370 326 326
query90 1927 192 183 183
query91 180 175 143 143
query92 63 63 56 56
query93 1124 960 558 558
query94 728 363 290 290
query95 661 376 439 376
query96 1073 748 363 363
query97 2693 2636 2531 2531
query98 220 207 209 207
query99 1104 1140 986 986
Total cold run time: 253751 ms
Total hot run time: 169794 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4560498825
# FE UT Coverage Report
Increment line coverage ` 31.37% (96/306)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_9af6867af36b4da0dec52406116ba2d158e2e2b9/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_9af6867af36b4da0dec52406116ba2d158e2e2b9/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](scan) support runtime partition prune [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62589:
URL: https://github.com/apache/doris/pull/62589#issuecomment-4563882796
# FE UT Coverage Report
Increment line coverage ` 31.37% (96/306)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62589_2ede6c8cafb5ff16b7121d7edac1018fc2e592e0/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62589_2ede6c8cafb5ff16b7121d7edac1018fc2e592e0/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org