You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mrhhsg (via GitHub)" <gi...@apache.org> on 2026/05/19 06:19:42 UTC
[PR] [feature](be) Support expression zonemap pruning [doris]
mrhhsg opened a new pull request, #63389:
URL: https://github.com/apache/doris/pull/63389
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Support expression-based ZoneMap pruning for internal and Parquet scan paths. Supported expressions include comparisons, IN/NOT IN, IS NULL/IS NOT NULL, and starts_with. The implementation adds segment/page/row-group pruning, late runtime-filter ZoneMap refresh for internal scans, conservative fallback semantics, and profile counters, with BE unit and SQL regression coverage.
### Release note
Support expression-based ZoneMap pruning for scan predicates to skip data using ZoneMap statistics.
### Check List (For Author)
- Test: Unit Test / Regression test / Format check
- DORIS_HOME=/mnt/disk7/hushenggang/doris-zonemap ninja -C be/ut_build_ASAN Exprs Storage Exec Format doris_be_test
- DORIS_HOME=/mnt/disk7/hushenggang/doris-zonemap ninja -C be/ut_build_ASAN Exprs doris_be_test
- ./run-be-ut.sh --run --filter=SegmentFilterHelpersTest.*
- ./run-be-ut.sh --run --filter=SegmentFilterHelpersTest.*:RuntimeFilterConsumerHelperTest.zonemap_runtime_filter_prepare_impl_before_publish:RuntimeFilterConsumerHelperTest.zonemap_runtime_filter_timeout_can_still_late_publish
- env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY -u http_proxy -u https_proxy -u all_proxy bash -lc '~/.codex/skills/doris-local-regression/scripts/doris-local-regression.sh start && ~/.codex/skills/doris-local-regression/scripts/doris-local-regression.sh run -d query_p0/expr_zonemap -s test_expr_zonemap_pruning'
- build-support/check-format.sh
- git diff --check HEAD~4..HEAD
- Behavior changed: Yes. Supported scan predicates can now prune data with expression ZoneMap evaluation.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3296624986
##########
be/src/storage/segment/segment_iterator.cpp:
##########
@@ -3435,6 +3470,99 @@ Status SegmentIterator::_construct_compound_expr_context() {
return Status::OK();
}
+Status SegmentIterator::_apply_expr_zonemap_to_row_ranges(const VExprContextSPtrs& conjuncts,
+ rowid_t min_rowid,
+ RowRanges* row_ranges) {
+ DORIS_CHECK(row_ranges != nullptr);
+ if (!config::enable_expr_zonemap_filter || conjuncts.empty() || row_ranges->is_empty()) {
+ return Status::OK();
+ }
+
+ std::unordered_map<int, VExprContextSPtrs> ctxs_by_slot;
+ for (const auto& conjunct : conjuncts) {
+ auto slot_index = single_slot_zonemap_index(conjunct);
+ if (slot_index.has_value()) {
+ ctxs_by_slot[*slot_index].emplace_back(conjunct);
+ }
+ }
+ // Page zone maps are stored per column. Multi-slot expressions need page alignment across
+ // multiple column readers and are therefore left to segment-level pruning for now.
+ if (ctxs_by_slot.empty()) {
+ return Status::OK();
+ }
+
+ ColumnIteratorOptions iter_opts {
+ .use_page_cache = _opts.use_page_cache,
+ .file_reader = _file_reader.get(),
+ .stats = _opts.stats,
+ .io_ctx = _opts.io_ctx,
+ };
+ for (const auto& [slot_index, slot_conjuncts] : ctxs_by_slot) {
+ if (cast_set<size_t>(slot_index) >= _schema->num_column_ids()) {
+ continue;
+ }
+ const auto cid = _schema->column_id(cast_set<size_t>(slot_index));
+ if (!_segment->can_apply_predicate_safely(cid, *_schema,
+ _opts.target_cast_type_for_variants, _opts)) {
+ continue;
+ }
+ const auto* field = _schema->column(cid);
+ if (field == nullptr) {
+ continue;
+ }
+ std::shared_ptr<ColumnReader> reader;
+ Status st = _segment->get_column_reader(*field, &reader, _opts.stats);
+ if (st.is<ErrorCode::NOT_FOUND>()) {
+ continue;
+ }
+ RETURN_IF_ERROR(st);
+ if (reader == nullptr || !reader->has_zone_map()) {
+ continue;
+ }
+ const std::vector<ZoneMapPB>* page_zone_maps = nullptr;
+ RETURN_IF_ERROR(reader->get_page_zone_maps(iter_opts, &page_zone_maps));
Review Comment:
This now loads page zonemaps for every single-slot common expression even when the expression cannot use zonemap evaluation. `single_slot_zonemap_index()` only checks slot count, while unsupported roots return `kUnsupported` and `VExprContext::evaluate_zonemap_filter()` turns that into `kMayMatch`, so a predicate like `abs(k) = 1` will read all page zonemaps and page row ranges and then keep every page. Because `enable_expr_zonemap_filter` defaults to true and common expr pushdown accepts arbitrary slot-based expressions, this is a storage hot-path regression. Please filter the conjunct list to expressions that have a supported zonemap evaluator before calling `get_page_zone_maps()`, or otherwise cache that capability so unsupported expressions skip this page path entirely.
##########
be/src/exprs/expr_zonemap_filter.cpp:
##########
@@ -0,0 +1,425 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/expr_zonemap_filter.h"
+
+#include <algorithm>
+#include <compare>
+#include <limits>
+#include <string>
+
+#include "core/column/column.h"
+#include "core/data_type/data_type_nullable.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vdirect_in_predicate.h"
+#include "exprs/vexpr.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+
+namespace doris::expr_zonemap {
+namespace {
+
+constexpr size_t kInZoneMapPointCheckThreshold = 64;
+
+VExprSPtr unwrap_identity_cast(const VExprSPtr& expr) {
+ if (expr->node_type() != TExprNodeType::CAST_EXPR || expr->get_num_children() != 1) {
+ return expr;
+ }
+ auto child = expr->get_child(0);
+ if (!remove_nullable(expr->data_type())->equals(*remove_nullable(child->data_type()))) {
+ return expr;
+ }
+ return unwrap_identity_cast(child);
+}
+
+std::optional<Field> field_from_literal_expr(const VExprSPtr& expr, DataTypePtr* literal_type) {
+ auto unwrapped = unwrap_identity_cast(expr);
+ auto literal = std::dynamic_pointer_cast<VLiteral>(unwrapped);
+ if (literal == nullptr) {
+ return std::nullopt;
+ }
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ *literal_type = literal->get_data_type();
+ return field;
+}
+
+std::optional<int> slot_index_from_expr(const VExprSPtr& expr) {
+ auto unwrapped = unwrap_identity_cast(expr);
+ auto slot = std::dynamic_pointer_cast<VSlotRef>(unwrapped);
+ if (slot == nullptr) {
+ return std::nullopt;
+ }
+ return slot->column_id();
+}
+
+bool field_types_compatible(PrimitiveType lhs, PrimitiveType rhs) {
+ return lhs == rhs || (is_string_type(lhs) && is_string_type(rhs));
+}
+
+bool field_less(const Field& lhs, const Field& rhs) {
+ return (lhs <=> rhs) == std::strong_ordering::less;
+}
+
+bool field_less_equal(const Field& lhs, const Field& rhs) {
+ const auto cmp = lhs <=> rhs;
+ return cmp == std::strong_ordering::less || cmp == std::strong_ordering::equal;
+}
+
+bool field_greater(const Field& lhs, const Field& rhs) {
+ return (lhs <=> rhs) == std::strong_ordering::greater;
+}
+
+bool field_greater_equal(const Field& lhs, const Field& rhs) {
+ const auto cmp = lhs <=> rhs;
+ return cmp == std::strong_ordering::greater || cmp == std::strong_ordering::equal;
+}
+
+ZoneMapFilterResult unsupported(ZoneMapEvalContext const& ctx) {
+ ++ctx.stats.unsupported_expr_count;
+ return ZoneMapFilterResult::kUnsupported;
+}
+
+ComparisonOp symmetric_op(ComparisonOp op) {
+ switch (op) {
+ case ComparisonOp::EQ:
+ case ComparisonOp::NE:
+ return op;
+ case ComparisonOp::LT:
+ return ComparisonOp::GT;
+ case ComparisonOp::LE:
+ return ComparisonOp::GE;
+ case ComparisonOp::GT:
+ return ComparisonOp::LT;
+ case ComparisonOp::GE:
+ return ComparisonOp::LE;
+ }
+ __builtin_unreachable();
+}
+
+bool value_in_range(const Field& value, const Field& min_value, const Field& max_value) {
+ return field_greater_equal(value, min_value) && field_less_equal(value, max_value);
+}
+
+std::optional<std::string> next_prefix(std::string_view prefix) {
+ // ZoneMap string bounds are compared by bytewise Field ordering. For starts_with(s, p),
+ // the safe upper bound is the next byte string after p: p <= s < next_prefix(p).
+ std::string upper(prefix);
+ for (auto i = static_cast<int64_t>(upper.size()) - 1; i >= 0; --i) {
+ auto byte = static_cast<unsigned char>(upper[i]);
+ if (byte != std::numeric_limits<unsigned char>::max()) {
+ upper[i] = static_cast<char>(byte + 1);
+ upper.resize(i + 1);
+ return upper;
+ }
+ }
+ return std::nullopt;
+}
+
+Field string_field(std::string_view value) {
+ return Field::create_field<TYPE_STRING>(std::string(value));
+}
+
+} // namespace
+
+std::optional<SlotLiteral> extract_slot_and_literal(const VExprSPtrs& args) {
+ if (args.size() != 2) {
+ return std::nullopt;
+ }
+
+ if (auto slot_index = slot_index_from_expr(args[0]); slot_index.has_value()) {
+ DataTypePtr literal_type;
+ auto literal = field_from_literal_expr(args[1], &literal_type);
+ if (!literal.has_value()) {
+ return std::nullopt;
+ }
+ return SlotLiteral {.slot_index = *slot_index,
+ .literal = std::move(*literal),
+ .literal_type = std::move(literal_type),
+ .literal_on_left = false};
+ }
+
+ if (auto slot_index = slot_index_from_expr(args[1]); slot_index.has_value()) {
+ DataTypePtr literal_type;
+ auto literal = field_from_literal_expr(args[0], &literal_type);
+ if (!literal.has_value()) {
+ return std::nullopt;
+ }
+ return SlotLiteral {.slot_index = *slot_index,
+ .literal = std::move(*literal),
+ .literal_type = std::move(literal_type),
+ .literal_on_left = true};
+ }
+
+ return std::nullopt;
+}
+
+bool check_type_compatible(const ZoneMapEvalContext& ctx, int slot_index,
+ const DataTypePtr& literal_type) {
+ const auto* slot_type = ctx.data_type(slot_index);
+ if (slot_type == nullptr || *slot_type == nullptr || literal_type == nullptr) {
+ return false;
+ }
+ if (!remove_nullable(*slot_type)->equals(*remove_nullable(literal_type))) {
+ ++ctx.stats.type_mismatch_count;
+ return false;
+ }
+ return true;
+}
+
+const segment_v2::ZoneMap* fetch_zone_map(const ZoneMapEvalContext& ctx, int slot_index) {
+ auto zone_map = ctx.zone_map(slot_index);
+ if (zone_map == nullptr) {
+ ++ctx.stats.unsupported_expr_count;
+ }
+ return zone_map;
+}
+
+bool range_stats_usable_for_zonemap(const segment_v2::ZoneMap& zone_map,
+ const DataTypePtr& data_type) {
+ if (zone_map.pass_all || zone_map.has_nan || zone_map.has_positive_inf ||
+ zone_map.has_negative_inf) {
+ return false;
+ }
+ if (data_type == nullptr) {
+ return false;
+ }
+ auto primitive_type = remove_nullable(data_type)->get_primitive_type();
+ if (primitive_type == TYPE_CHAR) {
+ return false;
+ }
+ return field_types_compatible(zone_map.min_value.get_type(), primitive_type) &&
+ field_types_compatible(zone_map.max_value.get_type(), primitive_type);
+}
+
+ZoneMapFilterResult eval_comparison_zonemap(const ZoneMapEvalContext& ctx,
+ const VExprSPtrs& arguments, ComparisonOp op) {
+ auto slot_literal = extract_slot_and_literal(arguments);
+ if (!slot_literal.has_value()) {
+ return unsupported(ctx);
+ }
+ if (slot_literal->literal.is_null()) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ if (!check_type_compatible(ctx, slot_literal->slot_index, slot_literal->literal_type)) {
+ return unsupported(ctx);
+ }
+ auto zone_map_ref = fetch_zone_map(ctx, slot_literal->slot_index);
+ if (zone_map_ref == nullptr) {
+ return ZoneMapFilterResult::kUnsupported;
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (!zone_map.has_not_null) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ const auto* slot_type = ctx.data_type(slot_literal->slot_index);
+ if (!range_stats_usable_for_zonemap(zone_map, *slot_type)) {
+ return unsupported(ctx);
+ }
+
+ const auto effective_op = slot_literal->literal_on_left ? symmetric_op(op) : op;
+ const auto& literal = slot_literal->literal;
+ switch (effective_op) {
+ case ComparisonOp::EQ:
+ return field_less(literal, zone_map.min_value) || field_greater(literal, zone_map.max_value)
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case ComparisonOp::NE:
+ return zone_map.min_value == literal && zone_map.max_value == literal
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case ComparisonOp::LT:
+ return field_greater_equal(zone_map.min_value, literal) ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case ComparisonOp::LE:
+ return field_greater(zone_map.min_value, literal) ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case ComparisonOp::GT:
+ return field_less_equal(zone_map.max_value, literal) ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case ComparisonOp::GE:
+ return field_less(zone_map.max_value, literal) ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ }
+ __builtin_unreachable();
+}
+
+ZoneMapFilterResult eval_null_zonemap(const ZoneMapEvalContext& ctx, const VExprSPtrs& arguments,
+ bool is_null) {
+ if (arguments.size() != 1) {
+ return unsupported(ctx);
+ }
+ auto slot_index = slot_index_from_expr(arguments[0]);
+ if (!slot_index.has_value()) {
+ return unsupported(ctx);
+ }
+ auto zone_map_ref = fetch_zone_map(ctx, *slot_index);
+ if (zone_map_ref == nullptr) {
+ return ZoneMapFilterResult::kUnsupported;
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (is_null) {
+ return zone_map.has_null ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+ }
+ return zone_map.has_not_null ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+}
+
+ZoneMapFilterResult eval_starts_with_zonemap(const ZoneMapEvalContext& ctx,
+ const VExprSPtrs& arguments) {
+ auto slot_literal = extract_slot_and_literal(arguments);
+ if (!slot_literal.has_value() || slot_literal->literal_on_left) {
+ return unsupported(ctx);
+ }
+ if (slot_literal->literal.is_null()) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ const auto* slot_type = ctx.data_type(slot_literal->slot_index);
+ if (slot_type == nullptr || *slot_type == nullptr || slot_literal->literal_type == nullptr) {
+ return unsupported(ctx);
+ }
+ if (!is_string_type(remove_nullable(*slot_type)->get_primitive_type()) ||
+ !is_string_type(remove_nullable(slot_literal->literal_type)->get_primitive_type())) {
+ ++ctx.stats.type_mismatch_count;
+ return unsupported(ctx);
+ }
+ auto zone_map_ref = fetch_zone_map(ctx, slot_literal->slot_index);
+ if (zone_map_ref == nullptr) {
+ return ZoneMapFilterResult::kUnsupported;
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (!zone_map.has_not_null) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ if (!range_stats_usable_for_zonemap(zone_map, *slot_type)) {
+ return unsupported(ctx);
+ }
+
+ const auto prefix = slot_literal->literal.as_string_view();
+ if (prefix.empty()) {
+ return ZoneMapFilterResult::kMayMatch;
+ }
+ auto lower = string_field(prefix);
+ if (field_less(zone_map.max_value, lower)) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ auto upper_prefix = next_prefix(prefix);
+ if (upper_prefix.has_value() && !field_less(zone_map.min_value, string_field(*upper_prefix))) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ return ZoneMapFilterResult::kMayMatch;
+}
+
+Status materialize_in_filter_for_zonemap(HybridSetBase& filter, const DataTypePtr& data_type,
+ std::vector<Field>& values, Field& min_value,
+ Field& max_value) {
+ values.clear();
+ auto* iterator = filter.begin();
+ while (iterator->has_next()) {
+ const void* value = iterator->get_value();
+ if (value != nullptr) {
+ TExprNode literal_node = create_texpr_node_from_hybrid_set_value(
+ value, remove_nullable(data_type)->get_primitive_type(),
+ remove_nullable(data_type)->get_precision(),
+ remove_nullable(data_type)->get_scale());
+ auto literal = VLiteral::create_shared(literal_node);
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ values.emplace_back(std::move(field));
+ }
+ iterator->next();
+ }
+ if (values.empty()) {
+ return Status::OK();
+ }
+ auto minmax = std::minmax_element(
+ values.begin(), values.end(),
+ [](const Field& lhs, const Field& rhs) { return field_less(lhs, rhs); });
+ min_value = *minmax.first;
+ max_value = *minmax.second;
+ return Status::OK();
+}
+
+ZoneMapFilterResult eval_in_zonemap(const ZoneMapEvalContext& ctx, const VExprSPtr& slot_expr,
+ bool is_not_in, const std::vector<Field>& values,
+ const Field& min_value, const Field& max_value) {
+ if (values.empty()) {
+ return is_not_in ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+ }
+ auto slot_index = slot_index_from_expr(slot_expr);
+ if (!slot_index.has_value()) {
+ return unsupported(ctx);
+ }
+ auto* slot_type = ctx.data_type(*slot_index);
+ if (slot_type == nullptr || *slot_type == nullptr) {
+ return unsupported(ctx);
+ }
+ auto zone_map_ref = fetch_zone_map(ctx, *slot_index);
+ if (zone_map_ref == nullptr) {
+ return ZoneMapFilterResult::kUnsupported;
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (!zone_map.has_not_null) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ if (!range_stats_usable_for_zonemap(zone_map, *slot_type)) {
+ return unsupported(ctx);
+ }
+
+ if (is_not_in) {
+ if (zone_map.min_value == zone_map.max_value) {
+ const bool only_value_is_filtered =
+ std::any_of(values.begin(), values.end(),
+ [&](const Field& value) { return value == zone_map.min_value; });
+ return only_value_is_filtered ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ }
+ return ZoneMapFilterResult::kMayMatch;
+ }
+
+ if (field_less(zone_map.max_value, min_value) || field_greater(zone_map.min_value, max_value)) {
+ return ZoneMapFilterResult::kNoMatch;
Review Comment:
`eval_in_zonemap()` reaches `Field` comparisons without validating that the materialized IN-list values have a type compatible with the slot/zonemap type. The comparison path calls `check_type_compatible()` before any `Field` ordering, but IN only checks that the zonemap range matches the slot type. If a BE expression has an INT slot with STRING literal fields, `field_less(zone_map.max_value, min_value)` throws from `Field::operator<=>` instead of conservatively returning `kUnsupported`. Please add the same compatibility guard for the IN min/max/value fields before these comparisons and cover this fallback in the unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4785969310
# BE Regression && UT Coverage Report
Increment line coverage `92.82% (931/1003)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_501cde75be748e273233042cab95400570d9a634_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_501cde75be748e273233042cab95400570d9a634_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 72.48% (27835/38404) |
| Line Coverage | 55.79% (298470/534997) |
| Region Coverage | 52.42% (248579/474217) |
| Branch Coverage | 53.64% (107961/201263) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4692658098
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3401607144
##########
be/src/exprs/function/functions_comparison.h:
##########
@@ -20,30 +20,143 @@
#pragma once
+#include <compare>
#include <limits>
+#include <optional>
+#include <string_view>
#include <type_traits>
+#include "common/check.h"
#include "common/logging.h"
#include "core/accurate_comparison.h"
#include "core/assert_cast.h"
#include "core/column/column_const.h"
#include "core/column/column_decimal.h"
#include "core/column/column_nullable.h"
#include "core/column/column_string.h"
+#include "core/data_type/data_type_nullable.h"
#include "core/data_type/data_type_number.h"
#include "core/data_type/data_type_string.h"
#include "core/data_type/define_primitive_type.h"
#include "core/decimal_comparison.h"
#include "core/field.h"
#include "core/memcmp_small.h"
#include "core/value/vdatetime_value.h"
+#include "exprs/expr_zonemap_filter.h"
#include "exprs/function/function.h"
#include "exprs/function/function_helpers.h"
#include "exprs/function/functions_logical.h"
+#include "exprs/vexpr.h"
#include "storage/index/index_reader_helper.h"
namespace doris {
+namespace comparison_zonemap_detail {
+
+enum class Op {
+ EQ,
Review Comment:
be/src/storage/predicate/column_predicate.h 这个定义在这里面也有，能不能统一一下，不要再新起一个定义了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4681948110
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4734238568
# FE Regression Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4734173714
# BE Regression && UT Coverage Report
Increment line coverage `80.78% (853/1056)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 64.16% (24602/38343) |
| Line Coverage | 47.96% (255964/533683) |
| Region Coverage | 44.68% (211323/473012) |
| Branch Coverage | 45.79% (91824/200529) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4738159212
# FE Regression Coverage Report
Increment line coverage ` 5.56% (2/36)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3450705935
##########
be/src/storage/segment/segment_iterator.h:
##########
@@ -73,6 +73,12 @@ class InvertedIndexIterator;
class RowRanges;
class IndexIterator;
+Status rebind_storage_exprs_to_reader_schema(const StorageReadOptions& opts, const Schema& schema,
Review Comment:
应该不需要了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4649165694
<details>
<summary>TPC-H: <b>Total hot run time: 29767 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d08314d7b632e04fd42e69ee6cfc095115445526, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17649 4096 4134 4096
q2 q3 10938 1529 838 838
q4 4811 493 345 345
q5 8473 900 602 602
q6 325 176 138 138
q7 913 888 641 641
q8 10823 1535 1595 1535
q9 6901 4549 4550 4549
q10 6826 1834 1586 1586
q11 513 340 321 321
q12 731 588 435 435
q13 18124 3387 2826 2826
q14 267 259 242 242
q15 q16 823 780 714 714
q17 1064 1036 1000 1000
q18 6830 5785 5593 5593
q19 1160 1417 1104 1104
q20 815 653 548 548
q21 5596 2699 2350 2350
q22 427 358 304 304
Total cold run time: 104009 ms
Total hot run time: 29767 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4395 4286 4264 4264
q2 q3 4545 4964 4359 4359
q4 2076 2181 1407 1407
q5 4445 4293 4754 4293
q6 264 202 148 148
q7 1963 1889 1650 1650
q8 2511 2127 2106 2106
q9 8016 7909 8012 7909
q10 4858 4990 4288 4288
q11 577 418 381 381
q12 752 767 583 583
q13 3236 3740 3066 3066
q14 295 326 283 283
q15 q16 725 776 648 648
q17 1386 1362 1320 1320
q18 7820 7156 6706 6706
q19 1116 1110 1121 1110
q20 2236 2229 1948 1948
q21 5282 4543 4516 4516
q22 514 453 416 416
Total cold run time: 57012 ms
Total hot run time: 51401 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4532241240
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4535253739
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3329965272
##########
be/src/exec/operator/olap_scan_operator.cpp:
##########
@@ -222,6 +222,20 @@ Status OlapScanLocalState::_init_profile() {
_stats_filtered_counter = ADD_COUNTER(_segment_profile, "RowsStatsFiltered", TUnit::UNIT);
_stats_rp_filtered_counter =
ADD_COUNTER(_segment_profile, "RowsZoneMapRuntimePredicateFiltered", TUnit::UNIT);
+ _expr_zonemap_filtered_segment_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapFilteredSegments", TUnit::UNIT);
+ _expr_zonemap_filtered_page_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapFilteredPages", TUnit::UNIT);
+ _expr_zonemap_unsupported_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapUnsupportedExprs", TUnit::UNIT);
+ _expr_zonemap_type_mismatch_counter =
Review Comment:
什么情况下这个会出现类型不匹配呢？举个栗子？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4681032579
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4697099584
# FE Regression Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_8a4c4d46c63a8083b97c25e9177806c0d0e9773d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_8a4c4d46c63a8083b97c25e9177806c0d0e9773d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3401682546
##########
be/src/exprs/function/functions_comparison.h:
##########
@@ -20,30 +20,143 @@
#pragma once
+#include <compare>
#include <limits>
+#include <optional>
+#include <string_view>
#include <type_traits>
+#include "common/check.h"
#include "common/logging.h"
#include "core/accurate_comparison.h"
#include "core/assert_cast.h"
#include "core/column/column_const.h"
#include "core/column/column_decimal.h"
#include "core/column/column_nullable.h"
#include "core/column/column_string.h"
+#include "core/data_type/data_type_nullable.h"
#include "core/data_type/data_type_number.h"
#include "core/data_type/data_type_string.h"
#include "core/data_type/define_primitive_type.h"
#include "core/decimal_comparison.h"
#include "core/field.h"
#include "core/memcmp_small.h"
#include "core/value/vdatetime_value.h"
+#include "exprs/expr_zonemap_filter.h"
#include "exprs/function/function.h"
#include "exprs/function/function_helpers.h"
#include "exprs/function/functions_logical.h"
+#include "exprs/vexpr.h"
#include "storage/index/index_reader_helper.h"
namespace doris {
+namespace comparison_zonemap_detail {
+
+enum class Op {
+ EQ,
+ NE,
+ LT,
+ LE,
+ GT,
+ GE,
+};
+
+inline Op symmetric_op(Op op) {
+ switch (op) {
+ case Op::EQ:
+ case Op::NE:
+ return op;
+ case Op::LT:
+ return Op::GT;
+ case Op::LE:
+ return Op::GE;
+ case Op::GT:
+ return Op::LT;
+ case Op::GE:
+ return Op::LE;
+ }
+ __builtin_unreachable();
+}
+
+inline ZoneMapFilterResult evaluate(const ZoneMapEvalContext& ctx, const VExprSPtrs& arguments,
+ Op op) {
+ auto slot_literal = expr_zonemap::extract_slot_and_literal(arguments);
+
+ auto slot_type = expr_zonemap::fetch_compatible_slot_type(ctx, slot_literal->slot_index,
+ slot_literal->slot_type);
+ if (slot_type == nullptr) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ const auto* zone_map_ref = ctx.zone_map(slot_literal->slot_index);
+ if (zone_map_ref == nullptr) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (!zone_map.has_not_null) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ if (!expr_zonemap::range_stats_usable_for_zonemap(zone_map, slot_type)) {
+ return unsupported_zonemap_filter(ctx);
+ }
+
+ const auto effective_op = slot_literal->literal_on_left ? symmetric_op(op) : op;
+ const auto& literal = slot_literal->literal;
+ switch (effective_op) {
+ case Op::EQ:
+ return expr_zonemap::field_less(literal, zone_map.min_value) ||
+ expr_zonemap::field_greater(literal, zone_map.max_value)
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case Op::NE:
Review Comment:
这里可能会有潜在问题，字符串在写入 ZoneMap 时会被裁剪到 512 字节（MAX_ZONE_MAP_INDEX_SIZE。这个不等于判断会不会有问题呢？你可能也得看一下现在的Column predicate怎么处理这个问题的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3402016529
##########
be/src/exprs/vin_predicate.cpp:
##########
@@ -112,6 +114,55 @@ Status VInPredicate::evaluate_inverted_index(VExprContext* context, uint32_t seg
return _evaluate_inverted_index(context, _function, segment_num_rows);
}
+Status VInPredicate::_materialize_for_zonemap_filter() {
Review Comment:
这里应该和column predicate的in做一个统一的抽象，从set里面提取最大最小值，这样三个地方的代码都能统一
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4732349317
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3456564296
##########
be/src/storage/index/zone_map/zonemap_eval_context.cpp:
##########
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "storage/index/zone_map/zonemap_eval_context.h"
+
+#include <algorithm>
+#include <string>
+
+#include "runtime/runtime_profile.h"
+
+namespace doris {
+
+const segment_v2::ZoneMap* ZoneMapEvalContext::zone_map(int slot_index) const {
Review Comment:
因为可能会有大量的 page zonemap， 如果放在 shared_ptr 里面会有大量的堆内存申请&释放，对性能应该还是有一定影响的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3456575398
##########
be/src/exprs/expr_zonemap_filter.h:
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <compare>
+#include <optional>
+#include <vector>
+
+#include "common/check.h"
+#include "common/status.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/define_primitive_type.h"
+#include "core/field.h"
+#include "exprs/vexpr_fwd.h"
+#include "storage/index/zone_map/zonemap_eval_context.h"
+#include "storage/index/zone_map/zonemap_filter_result.h"
+
+namespace doris {
+class HybridSetBase;
+class RuntimeState;
+class TExprNode;
+} // namespace doris
+
+namespace doris::expr_zonemap {
+
+struct InZonemapMaterializedSet {
+ bool contains_null = false;
+ std::vector<Field> values;
+ Field min_value;
+ Field max_value;
+};
+
+struct SlotLiteral {
+ int slot_index;
+ DataTypePtr slot_type;
+ Field literal;
+ DataTypePtr literal_type;
+ bool literal_on_left;
+};
+
+std::optional<SlotLiteral> extract_slot_and_literal(const VExprSPtrs& args);
+
+TExprNode create_texpr_node_from_hybrid_set_value(const void* data, const PrimitiveType& type,
+ int precision, int scale);
+
+Status materialize_hybrid_set_for_zonemap_filter(HybridSetBase& set, const DataTypePtr& data_type,
+ InZonemapMaterializedSet* result);
+
+inline bool field_types_compatible(PrimitiveType lhs, PrimitiveType rhs) {
+ return lhs == rhs || (is_string_type(lhs) && is_string_type(rhs));
+}
+
+inline bool data_types_compatible(const DataTypePtr& lhs, const DataTypePtr& rhs) {
+ if (lhs == nullptr || rhs == nullptr) {
+ return false;
+ }
+ const auto lhs_type = remove_nullable(lhs);
+ const auto rhs_type = remove_nullable(rhs);
+ const auto lhs_primitive_type = lhs_type->get_primitive_type();
+ const auto rhs_primitive_type = rhs_type->get_primitive_type();
+ if (is_string_type(lhs_primitive_type) && is_string_type(rhs_primitive_type)) {
+ return true;
+ }
+ return lhs_type->equals(*rhs_type);
+}
+
+inline DataTypePtr fetch_compatible_slot_type(const ZoneMapEvalContext& ctx, int slot_index,
+ const DataTypePtr& expr_slot_type) {
+ const auto* slot_type = ctx.data_type(slot_index);
+ if (slot_type == nullptr || *slot_type == nullptr) {
+ return nullptr;
+ }
+ DORIS_CHECK(data_types_compatible(*slot_type, expr_slot_type));
+ return *slot_type;
+}
+
+inline bool field_less(const Field& lhs, const Field& rhs) {
+ return (lhs <=> rhs) == std::strong_ordering::less;
Review Comment:
我倾向于：不放到 field.h，先继续留在 expr_zonemap_filter.h 更合适。
理由：
1. 当前使用范围很窄
field_less / field_greater / field_less_equal / field_greater_equal 目前只被 expr-zonemap 相关路径使用：
- expr_zonemap_filter.cpp
- functions_comparison.h
- function_string.cpp
2. field.h 已经有通用能力
Field 本身已经提供：
```cpp
std::strong_ordering operator<=>(const Field& rhs) const;
```
这几个 helper 只是对 <=> 的 zonemap 使用场景包装。放进 field.h 会扩大 core API surface，但没有明显复用收益。
3. field.h 是很基础的 core header
它被大量模块包含，最好避免加入只服务 expr-zonemap 的便捷函数，尤其是名字比较泛的 field_less，放在 doris 全局 namespace 下会显得比较突兀。
4. 语义上这些 helper 属于 zonemap pruning
它们服务的是 min/max 区间判断、literal 与 zone map 边界比较，和 range_stats_usable_for_zonemap()、fetch_compatible_slot_type() 属于同一组 evaluator helper。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3450823224
##########
be/src/format/parquet/vparquet_reader.cpp:
##########
@@ -1065,6 +1077,43 @@ void ParquetReader::set_condition_cache_context(std::shared_ptr<ConditionCacheCo
}
}
+bool ParquetReader::_expr_zonemap_page_slot_index(const VExprContextSPtr& conjunct,
Review Comment:
你这个PR 是把外表的zonemap 一起做了？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4778631129
# FE Regression Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_2b07745d310f37a8f54e20cefac75e3bf482f18f_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_2b07745d310f37a8f54e20cefac75e3bf482f18f_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4786022572
# FE Regression Coverage Report
Increment line coverage ` 10.00% (2/20)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_501cde75be748e273233042cab95400570d9a634_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_501cde75be748e273233042cab95400570d9a634_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3457994035
##########
be/src/exprs/vdirect_in_predicate.h:
##########
@@ -30,8 +37,12 @@ class VDirectInPredicate final : public VExpr {
ENABLE_FACTORY_CREATOR(VDirectInPredicate);
public:
- VDirectInPredicate(const TExprNode& node, const std::shared_ptr<HybridSetBase>& filter)
- : VExpr(node), _filter(filter), _expr_name("direct_in_predicate") {}
+ VDirectInPredicate(const TExprNode& node, const std::shared_ptr<HybridSetBase>& filter,
+ bool hybrid_set_values_match_child_type = true)
+ : VExpr(node),
Review Comment:
别加各种默认值了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4488194859
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3329966858
##########
be/src/exec/operator/olap_scan_operator.cpp:
##########
@@ -222,6 +222,20 @@ Status OlapScanLocalState::_init_profile() {
_stats_filtered_counter = ADD_COUNTER(_segment_profile, "RowsStatsFiltered", TUnit::UNIT);
_stats_rp_filtered_counter =
ADD_COUNTER(_segment_profile, "RowsZoneMapRuntimePredicateFiltered", TUnit::UNIT);
+ _expr_zonemap_filtered_segment_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapFilteredSegments", TUnit::UNIT);
+ _expr_zonemap_filtered_page_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapFilteredPages", TUnit::UNIT);
+ _expr_zonemap_unsupported_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapUnsupportedExprs", TUnit::UNIT);
+ _expr_zonemap_type_mismatch_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapTypeMismatch", TUnit::UNIT);
+ _in_zonemap_point_check_counter =
+ ADD_COUNTER(_segment_profile, "InZoneMapPointCheckCount", TUnit::UNIT);
+ _in_zonemap_range_only_counter =
+ ADD_COUNTER(_segment_profile, "InZoneMapRangeOnlyCount", TUnit::UNIT);
+ _in_zonemap_point_check_skipped_counter =
+ ADD_COUNTER(_segment_profile, "InZoneMapPointCheckSkippedDueToThreshold", TUnit::UNIT);
Review Comment:
这个count统计是冗余的，InZoneMapRangeOnlyCount和这个统计总是一致的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4530792897
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4535112094
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4535929862
<details>
<summary>TPC-DS: <b>Total hot run time: 174161 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3704e2d2892b9d0a7da311a4236a2371c84b889b, data reload: false
query5 4309 681 537 537
query6 351 245 228 228
query7 4249 607 347 347
query8 331 243 234 234
query9 8820 4269 4237 4237
query10 485 383 338 338
query11 5808 2623 2262 2262
query12 186 135 126 126
query13 1309 616 452 452
query14 6154 5575 5237 5237
query14_1 4589 4589 4516 4516
query15 221 210 187 187
query16 1049 471 459 459
query17 1179 790 617 617
query18 2655 507 380 380
query19 226 220 175 175
query20 136 141 133 133
query21 233 160 136 136
query22 13645 13615 13340 13340
query23 17558 16633 16314 16314
query23_1 16523 16388 16388 16388
query24 7401 1818 1358 1358
query24_1 1348 1374 1349 1349
query25 584 520 445 445
query26 1339 383 222 222
query27 2602 613 408 408
query28 4440 2057 2030 2030
query29 1010 679 543 543
query30 341 267 223 223
query31 1131 1118 998 998
query32 90 82 77 77
query33 549 371 316 316
query34 1189 1161 677 677
query35 785 843 696 696
query36 1414 1421 1286 1286
query37 157 106 87 87
query38 3253 3193 3083 3083
query39 927 925 908 908
query39_1 890 902 905 902
query40 258 192 179 179
query41 65 63 62 62
query42 121 124 123 123
query43 338 351 296 296
query44
query45 213 204 198 198
query46 1124 1258 735 735
query47 2384 2419 2247 2247
query48 379 419 300 300
query49 649 495 381 381
query50 1023 424 329 329
query51 4336 4367 4301 4301
query52 116 118 102 102
query53 259 308 221 221
query54 328 280 265 265
query55 94 97 90 90
query56 371 370 373 370
query57 1458 1434 1353 1353
query58 335 312 312 312
query59 1657 1772 1502 1502
query60 348 348 334 334
query61 164 156 156 156
query62 692 649 592 592
query63 260 218 234 218
query64 2400 820 639 639
query65
query66 1739 519 399 399
query67 29804 29710 29511 29511
query68
query69 504 341 318 318
query70 1045 1035 1031 1031
query71 385 380 370 370
query72 2989 2742 2443 2443
query73 878 793 463 463
query74 5125 4965 4794 4794
query75 2759 2670 2369 2369
query76 2312 1163 807 807
query77 401 417 351 351
query78 12512 12661 11885 11885
query79 1524 1084 788 788
query80 1316 650 566 566
query81 561 326 283 283
query82 990 170 123 123
query83 380 340 300 300
query84 278 163 133 133
query85 989 630 569 569
query86 458 354 350 350
query87 3468 3411 3266 3266
query88 3745 2804 2773 2773
query89 486 411 367 367
query90 1936 207 211 207
query91 222 210 180 180
query92 89 94 76 76
query93 1663 1532 937 937
query94 737 336 333 333
query95 794 589 483 483
query96 1088 832 351 351
query97 2729 2726 2579 2579
query98 233 227 236 227
query99 1192 1154 1023 1023
Total cold run time: 257068 ms
Total hot run time: 174161 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4717928553
# FE Regression Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4738176923
# BE Regression && UT Coverage Report
Increment line coverage `80.97% (855/1056)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 64.14% (24626/38397) |
| Line Coverage | 47.91% (256132/534630) |
| Region Coverage | 44.56% (211111/473727) |
| Branch Coverage | 45.68% (91818/201002) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4742352824
# FE Regression Coverage Report
Increment line coverage ` 3.12% (2/64)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3371127414
##########
be/src/exprs/function/function_string.cpp:
##########
@@ -1341,8 +1347,118 @@ using FunctionCrc32 = FunctionUnaryToType<Crc32Impl, NameCrc32>;
using FunctionStringUTF8Length = FunctionUnaryToType<StringUtf8LengthImpl, NameStringUtf8Length>;
using FunctionStringSpace = FunctionUnaryToType<StringSpace, NameStringSpace>;
using FunctionIsValidUTF8 = FunctionUnaryToType<IsValidUTF8Impl, NameIsValidUTF8>;
-using FunctionStringStartsWith =
- FunctionBinaryToType<DataTypeString, DataTypeString, StringStartsWithImpl, NameStartsWith>;
+
+namespace {
+
+ZoneMapFilterResult unsupported_starts_with_zonemap(const ZoneMapEvalContext& ctx) {
+ ++ctx.stats.unsupported_expr_count;
+ return ZoneMapFilterResult::kUnsupported;
+}
+
+bool field_less_for_starts_with_zonemap(const Field& lhs, const Field& rhs) {
+ return (lhs <=> rhs) == std::strong_ordering::less;
+}
+
+Field string_field_for_starts_with_zonemap(std::string_view value) {
+ return Field::create_field<TYPE_STRING>(std::string(value));
+}
+
+std::optional<std::string> next_prefix_for_starts_with_zonemap(std::string_view prefix) {
+ // ZoneMap string bounds are compared by bytewise Field ordering. For starts_with(s, p),
+ // the safe upper bound is the next byte string after p: p <= s < next_prefix(p).
+ std::string upper(prefix);
+ for (auto i = static_cast<int64_t>(upper.size()) - 1; i >= 0; --i) {
+ auto byte = static_cast<unsigned char>(upper[i]);
+ if (byte != std::numeric_limits<unsigned char>::max()) {
+ upper[i] = static_cast<char>(byte + 1);
+ upper.resize(i + 1);
+ return upper;
+ }
+ }
+ return std::nullopt;
+}
+
+bool supports_starts_with_zonemap_slot_type(const DataTypePtr& data_type) {
+ if (data_type == nullptr) {
+ return false;
+ }
+ auto primitive_type = remove_nullable(data_type)->get_primitive_type();
+ return primitive_type != TYPE_CHAR && is_string_type(primitive_type);
+}
+
+ZoneMapFilterResult evaluate_starts_with_zonemap(const ZoneMapEvalContext& ctx,
+ const VExprSPtrs& arguments) {
+ auto slot_literal = expr_zonemap::extract_slot_and_literal(arguments);
+ if (!slot_literal.has_value() || slot_literal->literal_on_left) {
+ return unsupported_starts_with_zonemap(ctx);
+ }
+ if (slot_literal->literal.is_null()) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ const auto* slot_type = ctx.data_type(slot_literal->slot_index);
+ if (slot_type == nullptr || *slot_type == nullptr || slot_literal->literal_type == nullptr) {
+ return unsupported_starts_with_zonemap(ctx);
+ }
+ if (!supports_starts_with_zonemap_slot_type(*slot_type) ||
+ !is_string_type(remove_nullable(slot_literal->literal_type)->get_primitive_type())) {
+ ++ctx.stats.type_mismatch_count;
+ return unsupported_starts_with_zonemap(ctx);
+ }
+ auto zone_map_ref = expr_zonemap::fetch_zone_map(ctx, slot_literal->slot_index);
+ if (zone_map_ref == nullptr) {
+ return ZoneMapFilterResult::kUnsupported;
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (!zone_map.has_not_null) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ if (!expr_zonemap::range_stats_usable_for_zonemap(zone_map, *slot_type)) {
+ return unsupported_starts_with_zonemap(ctx);
+ }
+
+ const auto prefix = slot_literal->literal.as_string_view();
+ if (prefix.empty()) {
+ return ZoneMapFilterResult::kMayMatch;
+ }
+ auto lower = string_field_for_starts_with_zonemap(prefix);
+ if (field_less_for_starts_with_zonemap(zone_map.max_value, lower)) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ auto upper_prefix = next_prefix_for_starts_with_zonemap(prefix);
+ if (upper_prefix.has_value() &&
+ !field_less_for_starts_with_zonemap(zone_map.min_value,
+ string_field_for_starts_with_zonemap(*upper_prefix))) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ return ZoneMapFilterResult::kMayMatch;
+}
+
+bool can_evaluate_starts_with_zonemap(const VExprSPtrs& arguments) {
+ auto slot_literal = expr_zonemap::extract_slot_and_literal(arguments);
+ if (!slot_literal.has_value() || slot_literal->literal_on_left) {
+ return false;
+ }
+ return supports_starts_with_zonemap_slot_type(slot_literal->slot_type) &&
+ slot_literal->literal_type != nullptr &&
+ is_string_type(remove_nullable(slot_literal->literal_type)->get_primitive_type());
+}
+
+} // namespace
+
+class FunctionStringStartsWith : public FunctionBinaryToType<DataTypeString, DataTypeString,
+ StringStartsWithImpl, NameStartsWith> {
+public:
+ static FunctionPtr create() { return std::make_shared<FunctionStringStartsWith>(); }
+ ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx,
+ const VExprSPtrs& arguments) const override {
+ return evaluate_starts_with_zonemap(ctx, arguments);
+ }
+
+ bool can_evaluate_zonemap_filter(const VExprSPtrs& arguments) const override {
+ return can_evaluate_starts_with_zonemap(arguments);
Review Comment:
为什么不就在这里实现，要冗余的调用一次函数呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4829917712
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3456575398
##########
be/src/exprs/expr_zonemap_filter.h:
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <compare>
+#include <optional>
+#include <vector>
+
+#include "common/check.h"
+#include "common/status.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/define_primitive_type.h"
+#include "core/field.h"
+#include "exprs/vexpr_fwd.h"
+#include "storage/index/zone_map/zonemap_eval_context.h"
+#include "storage/index/zone_map/zonemap_filter_result.h"
+
+namespace doris {
+class HybridSetBase;
+class RuntimeState;
+class TExprNode;
+} // namespace doris
+
+namespace doris::expr_zonemap {
+
+struct InZonemapMaterializedSet {
+ bool contains_null = false;
+ std::vector<Field> values;
+ Field min_value;
+ Field max_value;
+};
+
+struct SlotLiteral {
+ int slot_index;
+ DataTypePtr slot_type;
+ Field literal;
+ DataTypePtr literal_type;
+ bool literal_on_left;
+};
+
+std::optional<SlotLiteral> extract_slot_and_literal(const VExprSPtrs& args);
+
+TExprNode create_texpr_node_from_hybrid_set_value(const void* data, const PrimitiveType& type,
+ int precision, int scale);
+
+Status materialize_hybrid_set_for_zonemap_filter(HybridSetBase& set, const DataTypePtr& data_type,
+ InZonemapMaterializedSet* result);
+
+inline bool field_types_compatible(PrimitiveType lhs, PrimitiveType rhs) {
+ return lhs == rhs || (is_string_type(lhs) && is_string_type(rhs));
+}
+
+inline bool data_types_compatible(const DataTypePtr& lhs, const DataTypePtr& rhs) {
+ if (lhs == nullptr || rhs == nullptr) {
+ return false;
+ }
+ const auto lhs_type = remove_nullable(lhs);
+ const auto rhs_type = remove_nullable(rhs);
+ const auto lhs_primitive_type = lhs_type->get_primitive_type();
+ const auto rhs_primitive_type = rhs_type->get_primitive_type();
+ if (is_string_type(lhs_primitive_type) && is_string_type(rhs_primitive_type)) {
+ return true;
+ }
+ return lhs_type->equals(*rhs_type);
+}
+
+inline DataTypePtr fetch_compatible_slot_type(const ZoneMapEvalContext& ctx, int slot_index,
+ const DataTypePtr& expr_slot_type) {
+ const auto* slot_type = ctx.data_type(slot_index);
+ if (slot_type == nullptr || *slot_type == nullptr) {
+ return nullptr;
+ }
+ DORIS_CHECK(data_types_compatible(*slot_type, expr_slot_type));
+ return *slot_type;
+}
+
+inline bool field_less(const Field& lhs, const Field& rhs) {
+ return (lhs <=> rhs) == std::strong_ordering::less;
Review Comment:
我倾向于：不放到 field.h，先继续留在 expr_zonemap_filter.h 更合适。
理由：
1. 当前使用范围很窄
field_less / field_greater / field_less_equal / field_greater_equal 目前只被 expr-zonemap 相关路径使用：
- expr_zonemap_filter.cpp
- functions_comparison.h
- function_string.cpp
2. field.h 已经有通用能力
Field 本身已经提供：
```cpp
std::strong_ordering operator<=>(const Field& rhs) const;
```
这几个 helper 只是对 <=> 的 zonemap 使用场景包装。放进 field.h 会扩大 core API surface，但没有明显复用收益。
3. field.h 是很基础的 core header
它被大量模块包含，最好避免加入只服务 expr-zonemap 的便捷函数，尤其是名字比较泛的 field_less，放在 doris 全局 namespace 下会显得比较突兀。
4. 语义上这些 helper 属于 zonemap pruning
它们服务的是 min/max 区间判断、literal 与 zone map 边界比较，和 range_stats_usable_for_zonemap()、fetch_compatible_slot_type() 属于同一组 evaluator helper。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4781078735
# BE Regression && UT Coverage Report
Increment line coverage `92.01% (956/1039)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_f1cd5cca24e94354de8c70b10e47cc4c9ecf4d80_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_f1cd5cca24e94354de8c70b10e47cc4c9ecf4d80_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 74.35% (28558/38411) |
| Line Coverage | 58.20% (311397/535039) |
| Region Coverage | 54.96% (260643/474234) |
| Branch Coverage | 56.33% (113386/201275) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4658282993
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3402078439
##########
be/src/exprs/vin_predicate.cpp:
##########
@@ -112,6 +114,55 @@ Status VInPredicate::evaluate_inverted_index(VExprContext* context, uint32_t seg
return _evaluate_inverted_index(context, _function, segment_num_rows);
}
+Status VInPredicate::_materialize_for_zonemap_filter() {
Review Comment:
可能不太好抽象：
`InListPredicateBase` 是和类型绑定的，它的 min/max 存的是实际类型的值，比如 int/string 等。
而 `VInPredicate` 的 min/max 存的是 `Field` 类型
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3329957917
##########
be/src/common/config.cpp:
##########
@@ -1059,6 +1059,7 @@ DEFINE_mInt64(small_column_size_buffer, "100");
// Perform the always_true check at intervals determined by runtime_filter_sampling_frequency
DEFINE_mInt32(runtime_filter_sampling_frequency, "32");
+DEFINE_mBool(enable_expr_zonemap_filter, "true");
Review Comment:
用fe的session 来控制把，不用be的conf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3330511558
##########
be/src/common/config.cpp:
##########
@@ -1059,6 +1059,7 @@ DEFINE_mInt64(small_column_size_buffer, "100");
// Perform the always_true check at intervals determined by runtime_filter_sampling_frequency
DEFINE_mInt32(runtime_filter_sampling_frequency, "32");
+DEFINE_mBool(enable_expr_zonemap_filter, "true");
Review Comment:
已在 cc67a753274 中处理：删除 BE mutable config，新增 FE session variable `enable_expr_zonemap_filter`，通过 `TQueryOptions` 下发到 BE。OLAP segment/page 和 Parquet row-group/page 路径现在都读取 query option；默认保持 true，测试里也改为通过 `RuntimeState` 控制。
##########
be/src/exec/operator/olap_scan_operator.cpp:
##########
@@ -222,6 +222,20 @@ Status OlapScanLocalState::_init_profile() {
_stats_filtered_counter = ADD_COUNTER(_segment_profile, "RowsStatsFiltered", TUnit::UNIT);
_stats_rp_filtered_counter =
ADD_COUNTER(_segment_profile, "RowsZoneMapRuntimePredicateFiltered", TUnit::UNIT);
+ _expr_zonemap_filtered_segment_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapFilteredSegments", TUnit::UNIT);
+ _expr_zonemap_filtered_page_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapFilteredPages", TUnit::UNIT);
+ _expr_zonemap_unsupported_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapUnsupportedExprs", TUnit::UNIT);
+ _expr_zonemap_type_mismatch_counter =
Review Comment:
这里主要是 BE 侧表达式物化值类型和 slot/zonemap 类型不一致时的保守 fallback 统计。例如 slot 是数值列但比较 literal/IN-list 物化后不能和该列类型安全比较，或者字典 rewrite 这类场景让表达式值类型不再等价于原始 slot 类型时，都不能继续用 ZoneMap 做精确裁剪，只能返回 unsupported/may-match。这个 counter 用来观测这类 fallback，不表示正常 SQL 会报错。
##########
be/src/exec/operator/olap_scan_operator.cpp:
##########
@@ -222,6 +222,20 @@ Status OlapScanLocalState::_init_profile() {
_stats_filtered_counter = ADD_COUNTER(_segment_profile, "RowsStatsFiltered", TUnit::UNIT);
_stats_rp_filtered_counter =
ADD_COUNTER(_segment_profile, "RowsZoneMapRuntimePredicateFiltered", TUnit::UNIT);
+ _expr_zonemap_filtered_segment_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapFilteredSegments", TUnit::UNIT);
+ _expr_zonemap_filtered_page_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapFilteredPages", TUnit::UNIT);
+ _expr_zonemap_unsupported_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapUnsupportedExprs", TUnit::UNIT);
+ _expr_zonemap_type_mismatch_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapTypeMismatch", TUnit::UNIT);
+ _in_zonemap_point_check_counter =
+ ADD_COUNTER(_segment_profile, "InZoneMapPointCheckCount", TUnit::UNIT);
+ _in_zonemap_range_only_counter =
+ ADD_COUNTER(_segment_profile, "InZoneMapRangeOnlyCount", TUnit::UNIT);
+ _in_zonemap_point_check_skipped_counter =
+ ADD_COUNTER(_segment_profile, "InZoneMapPointCheckSkippedDueToThreshold", TUnit::UNIT);
Review Comment:
已在 cc67a753274 中处理：删除 `InZoneMapPointCheckSkippedDueToThreshold` 以及对应 stats/profile 字段和 UT 断言。现在大 IN-list 跳过 point check 只统计 `InZoneMapRangeOnlyCount`。
##########
be/src/exec/runtime_filter/runtime_filter_consumer_helper.cpp:
##########
@@ -17,10 +17,14 @@
#include "exec/runtime_filter/runtime_filter_consumer_helper.h"
+#include "common/logging.h"
Review Comment:
已在 cc67a753274 中处理：这部分只是之前补 include 时留下的无关 diff，当前已从 PR 中移除，`runtime_filter_consumer_helper.cpp` 不再有无代码修改的头文件变更。
##########
be/src/exec/runtime_filter/runtime_filter_consumer_helper.h:
##########
@@ -39,6 +39,7 @@ class RuntimeFilterConsumerHelper {
// Called by Operator.
Status acquire_runtime_filter(RuntimeState* state, VExprContextSPtrs& conjuncts,
const RowDescriptor& row_descriptor);
+
// The un-arrival filters will be checked every time the scanner is scheduled.
Review Comment:
已在 cc67a753274 中处理：同上，已移除 `runtime_filter_consumer_helper.h` 中无关空行/格式 diff，避免把无代码逻辑变化带进这个 PR。
##########
be/src/exprs/function/function.h:
##########
@@ -226,6 +229,17 @@ class IFunctionBase {
virtual bool can_push_down_to_index() const { return false; }
virtual bool is_blockable() const { return false; }
+
+ virtual bool is_deterministic() const { return false; }
Review Comment:
已在 cc67a753274 中处理：去掉了 `IFunctionBase::is_deterministic()` 这层判断，不再用“确定性”描述 ZoneMap 能力。现在只通过 `can_evaluate_zonemap_filter()` / `evaluate_zonemap_filter()` 表达具体函数是否支持 ZoneMap 推导，避免把该能力和 determinism 混在一起。
##########
be/src/exprs/function/function_string.cpp:
##########
@@ -1341,8 +1342,21 @@ using FunctionCrc32 = FunctionUnaryToType<Crc32Impl, NameCrc32>;
using FunctionStringUTF8Length = FunctionUnaryToType<StringUtf8LengthImpl, NameStringUtf8Length>;
using FunctionStringSpace = FunctionUnaryToType<StringSpace, NameStringSpace>;
using FunctionIsValidUTF8 = FunctionUnaryToType<IsValidUTF8Impl, NameIsValidUTF8>;
-using FunctionStringStartsWith =
- FunctionBinaryToType<DataTypeString, DataTypeString, StringStartsWithImpl, NameStartsWith>;
+class FunctionStringStartsWith : public FunctionBinaryToType<DataTypeString, DataTypeString,
+ StringStartsWithImpl, NameStartsWith> {
+public:
+ static FunctionPtr create() { return std::make_shared<FunctionStringStartsWith>(); }
+ bool is_deterministic() const override { return true; }
+ ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx,
+ const VExprSPtrs& arguments) const override {
+ return expr_zonemap::eval_starts_with_zonemap(ctx, arguments);
Review Comment:
这里先保留集中实现，原因是 `starts_with` 与 comparison/IN/null 共用同一套 Field 类型兼容、CHAR range 禁用、`ZoneMapEvalContext` 统计和 conservative fallback 逻辑；函数侧只暴露 can/evaluate hook，具体 ZoneMap 语义放在 expr_zonemap 里，避免在各个 function 文件里重复维护这些边界。后续如果支持的字符串函数变多，可以再按函数族拆分文件降低单文件复杂度。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4536622338
# BE Regression && UT Coverage Report
Increment line coverage `87.07% (1003/1152)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_3704e2d2892b9d0a7da311a4236a2371c84b889b_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_3704e2d2892b9d0a7da311a4236a2371c84b889b_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 72.07% (27451/38092) |
| Line Coverage | 55.40% (292916/528752) |
| Region Coverage | 52.35% (243613/465374) |
| Branch Coverage | 53.67% (105154/195914) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3295673162
##########
be/src/storage/segment/segment.cpp:
##########
@@ -290,6 +344,21 @@ Status Segment::new_iterator(SchemaSPtr schema, const StorageReadOptions& read_o
}
}
Review Comment:
This whole-segment expr-zonemap pruning can use the wrong column on AGG/UNIQUE-MOR readers because `read_options.common_expr_ctxs_push_down` still contains scan-tuple slot ordinals here, but `schema` is the rowset reader schema. The PR already handles this later in `SegmentIterator::_init_common_expr_ctxs()` with `rebind_storage_exprs_to_reader_schema()` because these layouts can diverge. For example, on an AGG_KEYS table `(k1, k2, v)`, a query that only returns/pushes an expression on `k2` can have the slot ref at scan ordinal 0 while the reader schema still has `k1` at position 0 for merge/aggregation. This helper then reads `schema.column_id(0)` and evaluates the `k2` predicate against `k1`'s segment zonemap; if `k1`'s min/max is disjoint, the entire segment can be incorrectly replaced with `EmptySegmentIterator` even though matching `k2` rows exist. Please either rebind cloned expressions before this segment-level evaluation (same mapping as `SegmentIterator`) or skip this who
le-segment expr-zonemap path when the storage expression slot layout is not guaranteed to match the reader schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3371116012
##########
be/src/exec/operator/olap_scan_operator.cpp:
##########
@@ -222,6 +222,20 @@ Status OlapScanLocalState::_init_profile() {
_stats_filtered_counter = ADD_COUNTER(_segment_profile, "RowsStatsFiltered", TUnit::UNIT);
_stats_rp_filtered_counter =
ADD_COUNTER(_segment_profile, "RowsZoneMapRuntimePredicateFiltered", TUnit::UNIT);
+ _expr_zonemap_filtered_segment_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapFilteredSegments", TUnit::UNIT);
+ _expr_zonemap_filtered_page_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapFilteredPages", TUnit::UNIT);
+ _expr_zonemap_unsupported_counter =
+ ADD_COUNTER(_segment_profile, "ExprZoneMapUnsupportedExprs", TUnit::UNIT);
+ _expr_zonemap_type_mismatch_counter =
Review Comment:
剔除这个没有用的比较
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4796185228
<details>
<summary>TPC-DS: <b>Total hot run time: 173890 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 501cde75be748e273233042cab95400570d9a634, data reload: false
query5 4303 633 491 491
query6 473 228 194 194
query7 4829 582 370 370
query8 380 222 224 222
query9 8752 4025 4029 4025
query10 485 349 304 304
query11 5861 2383 2141 2141
query12 158 105 102 102
query13 1282 623 419 419
query14 6420 5441 5123 5123
query14_1 4385 4395 4415 4395
query15 207 204 179 179
query16 1045 479 503 479
query17 1116 720 592 592
query18 2725 495 354 354
query19 208 192 152 152
query20 124 118 105 105
query21 233 161 134 134
query22 13575 13747 13393 13393
query23 17468 16526 16078 16078
query23_1 16307 16296 16367 16296
query24 7512 1799 1337 1337
query24_1 1324 1298 1283 1283
query25 581 461 403 403
query26 1362 374 208 208
query27 2517 574 379 379
query28 4452 2076 2064 2064
query29 1100 635 513 513
query30 339 260 238 238
query31 1131 1106 998 998
query32 105 65 64 64
query33 534 333 302 302
query34 1152 1119 665 665
query35 754 791 671 671
query36 1415 1381 1234 1234
query37 151 107 94 94
query38 1882 1709 1659 1659
query39 928 924 895 895
query39_1 894 885 872 872
query40 240 159 136 136
query41 64 62 59 59
query42 91 90 91 90
query43 318 325 278 278
query44 1419 769 772 769
query45 195 184 177 177
query46 1085 1186 708 708
query47 2379 2393 2255 2255
query48 413 401 292 292
query49 634 464 356 356
query50 1062 426 342 342
query51 4401 4395 4331 4331
query52 87 87 74 74
query53 271 277 208 208
query54 286 229 214 214
query55 73 76 67 67
query56 283 287 300 287
query57 1443 1408 1302 1302
query58 290 253 266 253
query59 1526 1617 1449 1449
query60 307 266 253 253
query61 152 146 142 142
query62 686 646 530 530
query63 242 204 217 204
query64 2450 763 623 623
query65 4854 4812 4799 4799
query66 1744 504 370 370
query67 29787 29707 29516 29516
query68 3208 1597 926 926
query69 407 299 271 271
query70 1079 944 985 944
query71 342 334 303 303
query72 2974 2608 2273 2273
query73 832 742 446 446
query74 5142 4963 4760 4760
query75 2699 2673 2290 2290
query76 2309 1193 779 779
query77 336 380 279 279
query78 12331 12465 11881 11881
query79 1428 1158 752 752
query80 1315 542 471 471
query81 529 315 284 284
query82 601 155 123 123
query83 381 316 294 294
query84 284 162 131 131
query85 972 606 491 491
query86 439 311 286 286
query87 1831 1838 1761 1761
query88 3701 2855 2799 2799
query89 453 397 352 352
query90 1931 196 189 189
query91 200 187 161 161
query92 65 60 58 58
query93 1648 1582 1032 1032
query94 730 347 312 312
query95 791 503 471 471
query96 1062 773 371 371
query97 2718 2704 2556 2556
query98 218 205 203 203
query99 1184 1151 1027 1027
Total cold run time: 259818 ms
Total hot run time: 173890 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4715727509
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3457921972
##########
be/src/exprs/function/functions_comparison.h:
##########
@@ -20,30 +20,147 @@
#pragma once
+#include <compare>
#include <limits>
+#include <optional>
+#include <string_view>
#include <type_traits>
+#include "common/check.h"
#include "common/logging.h"
#include "core/accurate_comparison.h"
#include "core/assert_cast.h"
#include "core/column/column_const.h"
#include "core/column/column_decimal.h"
#include "core/column/column_nullable.h"
#include "core/column/column_string.h"
+#include "core/data_type/data_type_nullable.h"
#include "core/data_type/data_type_number.h"
#include "core/data_type/data_type_string.h"
#include "core/data_type/define_primitive_type.h"
#include "core/decimal_comparison.h"
#include "core/field.h"
#include "core/memcmp_small.h"
#include "core/value/vdatetime_value.h"
+#include "exprs/expr_zonemap_filter.h"
#include "exprs/function/function.h"
#include "exprs/function/function_helpers.h"
#include "exprs/function/functions_logical.h"
+#include "exprs/vexpr.h"
#include "storage/index/index_reader_helper.h"
namespace doris {
+namespace comparison_zonemap_detail {
Review Comment:
zonemap 这个namespace 太多了，你看看收敛到一个吧
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4733949269
# BE UT Coverage Report
Increment line coverage `67.71% (717/1059)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/b264ceb03f62a92109e263f696cc7aa99d35167b_b264ceb03f62a92109e263f696cc7aa99d35167b/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/b264ceb03f62a92109e263f696cc7aa99d35167b_b264ceb03f62a92109e263f696cc7aa99d35167b/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 54.45% (21379/39267) |
| Line Coverage | 38.12% (204659/536835) |
| Region Coverage | 34.09% (160538/470867) |
| Branch Coverage | 35.12% (70332/200274) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4739542623
# FE Regression Coverage Report
Increment line coverage ` 3.12% (2/64)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4785082608
<details>
<summary>ClickBench: <b>Total hot run time: 26.14 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f640be29bedf8dc904f15104f2d1a290b0569798, data reload: false
query1 0.01 0.01 0.01
query2 0.15 0.08 0.08
query3 0.41 0.25 0.24
query4 1.63 0.25 0.25
query5 0.33 0.31 0.31
query6 1.15 0.68 0.67
query7 0.04 0.01 0.01
query8 0.09 0.08 0.07
query9 0.50 0.38 0.38
query10 0.60 0.59 0.59
query11 0.31 0.20 0.18
query12 0.33 0.19 0.19
query13 0.54 0.55 0.55
query14 0.94 0.93 0.93
query15 0.68 0.60 0.60
query16 0.39 0.39 0.40
query17 1.02 1.02 1.11
query18 0.31 0.31 0.30
query19 1.99 1.90 1.84
query20 0.02 0.01 0.01
query21 15.36 0.39 0.31
query22 4.85 0.14 0.13
query23 15.81 0.48 0.30
query24 2.51 0.62 0.42
query25 0.16 0.11 0.11
query26 0.75 0.29 0.22
query27 0.10 0.09 0.09
query28 3.37 0.94 0.56
query29 12.50 4.58 3.54
query30 0.39 0.26 0.26
query31 2.76 0.63 0.34
query32 3.23 0.60 0.50
query33 3.06 2.96 3.03
query34 15.70 4.10 3.37
query35 3.30 3.30 3.30
query36 0.65 0.54 0.53
query37 0.12 0.09 0.10
query38 0.08 0.06 0.06
query39 0.07 0.06 0.07
query40 0.21 0.18 0.17
query41 0.14 0.09 0.08
query42 0.10 0.07 0.06
query43 0.08 0.07 0.06
Total cold run time: 96.74 s
Total hot run time: 26.14 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3451625717
##########
be/src/exprs/expr_zonemap_filter.cpp:
##########
@@ -0,0 +1,269 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/expr_zonemap_filter.h"
+
+#include <algorithm>
+#include <set>
+
+#include "common/check.h"
+#include "common/logging.h"
+#include "core/column/column.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/string_ref.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/runtime_state.h"
+
+namespace doris::expr_zonemap {
+namespace {
+
+constexpr size_t kInZoneMapPointCheckThreshold = 64;
+
+std::optional<Field> field_from_literal_expr(const VExprSPtr& expr, DataTypePtr* literal_type) {
Review Comment:
literal 的 data type和 slot 的datatype 分别是什么意思
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4694848008
# BE Regression && UT Coverage Report
Increment line coverage `69.31% (734/1059)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_8a4c4d46c63a8083b97c25e9177806c0d0e9773d_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_8a4c4d46c63a8083b97c25e9177806c0d0e9773d_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 58.18% (22316/38357) |
| Line Coverage | 41.67% (222315/533564) |
| Region Coverage | 37.95% (179401/472791) |
| Branch Coverage | 38.64% (77442/200440) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3401937858
##########
be/src/exprs/expr_zonemap_filter.cpp:
##########
@@ -0,0 +1,249 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/expr_zonemap_filter.h"
+
+#include <algorithm>
+#include <set>
+
+#include "common/check.h"
+#include "common/logging.h"
+#include "core/column/column.h"
+#include "core/data_type/data_type_nullable.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vdirect_in_predicate.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/runtime_state.h"
+
+namespace doris::expr_zonemap {
+namespace {
+
+constexpr size_t kInZoneMapPointCheckThreshold = 64;
+
+VExprSPtr unwrap_identity_cast(const VExprSPtr& expr) {
+ if (expr->node_type() != TExprNodeType::CAST_EXPR || expr->get_num_children() != 1) {
+ return expr;
+ }
+ auto child = expr->get_child(0);
+ if (!remove_nullable(expr->data_type())->equals(*remove_nullable(child->data_type()))) {
+ return expr;
+ }
+ return unwrap_identity_cast(child);
+}
+
+std::optional<Field> field_from_literal_expr(const VExprSPtr& expr, DataTypePtr* literal_type) {
+ auto unwrapped = unwrap_identity_cast(expr);
+ auto literal = std::dynamic_pointer_cast<VLiteral>(unwrapped);
+ if (literal == nullptr) {
+ return std::nullopt;
+ }
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ *literal_type = literal->get_data_type();
+ return field;
+}
+
+struct SlotExpr {
+ int index;
+ DataTypePtr data_type;
+};
+
+std::optional<SlotExpr> slot_from_expr(const VExprSPtr& expr) {
+ auto unwrapped = unwrap_identity_cast(expr);
+ auto slot = std::dynamic_pointer_cast<VSlotRef>(unwrapped);
+ if (slot == nullptr) {
+ return std::nullopt;
+ }
+ return SlotExpr {.index = slot->column_id(), .data_type = slot->data_type()};
+}
+
+bool value_in_range(const Field& value, const Field& min_value, const Field& max_value) {
+ return field_greater_equal(value, min_value) && field_less_equal(value, max_value);
+}
+
+} // namespace
+
+std::optional<SlotLiteral> extract_slot_and_literal(const VExprSPtrs& args) {
+ if (args.size() != 2) {
+ return std::nullopt;
+ }
+
+ if (auto slot = slot_from_expr(args[0]); slot.has_value()) {
+ DataTypePtr literal_type;
+ auto literal = field_from_literal_expr(args[1], &literal_type);
+ if (!literal.has_value()) {
+ return std::nullopt;
+ }
+ return SlotLiteral {.slot_index = slot->index,
+ .slot_type = std::move(slot->data_type),
+ .literal = std::move(*literal),
+ .literal_type = std::move(literal_type),
+ .literal_on_left = false};
+ }
+
+ if (auto slot = slot_from_expr(args[1]); slot.has_value()) {
+ DataTypePtr literal_type;
+ auto literal = field_from_literal_expr(args[0], &literal_type);
+ if (!literal.has_value()) {
+ return std::nullopt;
+ }
+ return SlotLiteral {.slot_index = slot->index,
+ .slot_type = std::move(slot->data_type),
+ .literal = std::move(*literal),
+ .literal_type = std::move(literal_type),
+ .literal_on_left = true};
+ }
+
+ return std::nullopt;
+}
+
+bool range_stats_usable_for_zonemap(const segment_v2::ZoneMap& zone_map,
+ const DataTypePtr& data_type) {
+ if (zone_map.pass_all || zone_map.has_nan || zone_map.has_positive_inf ||
+ zone_map.has_negative_inf) {
+ return false;
+ }
+ DORIS_CHECK(data_type != nullptr);
+ auto primitive_type = remove_nullable(data_type)->get_primitive_type();
+ DORIS_CHECK(field_types_compatible(zone_map.min_value.get_type(), primitive_type));
+ DORIS_CHECK(field_types_compatible(zone_map.max_value.get_type(), primitive_type));
+ return true;
+}
+
+ZoneMapFilterResult eval_null_zonemap(const ZoneMapEvalContext& ctx, const VExprSPtrs& arguments,
+ bool is_null) {
+ DORIS_CHECK(arguments.size() == 1);
+ auto slot = slot_from_expr(arguments[0]);
+ DORIS_CHECK(slot.has_value());
+ const auto* zone_map_ref = ctx.zone_map(slot->index);
+ if (zone_map_ref == nullptr) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (is_null) {
+ return zone_map.has_null ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+ }
+ return zone_map.has_not_null ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+}
+
+bool can_eval_null_zonemap(const VExprSPtrs& arguments) {
+ if (arguments.size() != 1) {
+ return false;
+ }
+ return slot_from_expr(arguments[0]).has_value();
Review Comment:
这个什么情况下会没有slot呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4682871053
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4489625445
<details>
<summary>TPC-H: <b>Total hot run time: 32343 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fdb02e11fe807a3a965e4543dac6f6f3fed0539c, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17858 4064 4005 4005
q2 q3 10769 1418 819 819
q4 4701 473 344 344
q5 7694 2268 2176 2176
q6 383 179 147 147
q7 974 789 651 651
q8 9382 1669 1735 1669
q9 7014 4966 4932 4932
q10 6436 2204 1810 1810
q11 507 353 319 319
q12 716 567 435 435
q13 18103 3380 2813 2813
q14 269 260 239 239
q15 q16 823 773 719 719
q17 1015 952 1016 952
q18 6980 5912 5720 5720
q19 1217 1399 1161 1161
q20 840 674 526 526
q21 6082 2786 2572 2572
q22 469 389 334 334
Total cold run time: 102232 ms
Total hot run time: 32343 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4694 4852 4584 4584
q2 q3 4887 5246 4609 4609
q4 2139 2244 1471 1471
q5 4829 4702 4666 4666
q6 226 179 128 128
q7 1863 1814 1471 1471
q8 2210 1934 2011 1934
q9 7278 7248 7293 7248
q10 4479 4394 3969 3969
q11 541 386 365 365
q12 721 717 530 530
q13 3070 3367 2774 2774
q14 276 282 252 252
q15 q16 691 693 614 614
q17 1293 1249 1249 1249
q18 7579 7014 6781 6781
q19 1108 1100 1091 1091
q20 2247 2235 1961 1961
q21 5362 4649 4554 4554
q22 529 468 431 431
Total cold run time: 56022 ms
Total hot run time: 50682 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4489717888
<details>
<summary>TPC-DS: <b>Total hot run time: 171489 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fdb02e11fe807a3a965e4543dac6f6f3fed0539c, data reload: false
query5 4348 679 536 536
query6 347 251 229 229
query7 4285 615 331 331
query8 334 262 251 251
query9 8829 4102 4100 4100
query10 469 365 314 314
query11 5797 2427 2254 2254
query12 184 131 127 127
query13 1306 639 448 448
query14 6014 5365 5063 5063
query14_1 4359 4429 4312 4312
query15 211 208 185 185
query16 1017 448 426 426
query17 1098 723 569 569
query18 2430 479 345 345
query19 230 202 165 165
query20 143 134 133 133
query21 229 157 136 136
query22 13726 13564 13638 13564
query23 17209 16311 16005 16005
query23_1 16231 16064 16092 16064
query24 7543 1761 1302 1302
query24_1 1299 1312 1296 1296
query25 561 465 420 420
query26 1355 362 218 218
query27 2619 584 366 366
query28 4461 1972 1979 1972
query29 1002 668 500 500
query30 329 269 235 235
query31 1142 1092 973 973
query32 87 77 73 73
query33 539 364 292 292
query34 1150 1170 662 662
query35 780 782 671 671
query36 1340 1350 1183 1183
query37 154 104 92 92
query38 3233 3124 3042 3042
query39 937 956 905 905
query39_1 888 876 876 876
query40 271 191 170 170
query41 65 64 62 62
query42 116 115 123 115
query43 321 331 298 298
query44
query45 209 201 199 199
query46 1062 1199 722 722
query47 2375 2365 2237 2237
query48 404 425 298 298
query49 623 495 403 403
query50 1041 412 331 331
query51 4319 4307 4192 4192
query52 114 114 102 102
query53 273 303 223 223
query54 334 297 272 272
query55 95 92 88 88
query56 424 384 367 367
query57 1416 1388 1324 1324
query58 342 314 310 310
query59 1611 1648 1401 1401
query60 345 351 336 336
query61 157 179 180 179
query62 675 630 567 567
query63 255 220 230 220
query64 2503 826 651 651
query65
query66 1756 524 411 411
query67 30097 30063 29843 29843
query68
query69 471 342 303 303
query70 1070 988 953 953
query71 398 392 362 362
query72 2990 2712 2402 2402
query73 858 751 442 442
query74 5152 4916 4782 4782
query75 2785 2669 2334 2334
query76 2303 1174 823 823
query77 412 439 353 353
query78 12136 12261 11633 11633
query79 1297 1041 760 760
query80 699 636 566 566
query81 456 340 298 298
query82 246 166 125 125
query83 327 329 305 305
query84 294 166 134 134
query85 1004 710 662 662
query86 373 357 310 310
query87 3449 3388 3243 3243
query88 3609 2721 2719 2719
query89 470 423 378 378
query90 2176 211 212 211
query91 230 266 176 176
query92 82 80 76 76
query93 1493 1516 957 957
query94 557 339 329 329
query95 771 587 468 468
query96 1024 808 336 336
query97 2709 2672 2605 2605
query98 246 229 237 229
query99 1100 1121 967 967
Total cold run time: 253610 ms
Total hot run time: 171489 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4510504770
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4645928753
# BE Regression && UT Coverage Report
Increment line coverage `88.86% (1045/1176)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_d08314d7b632e04fd42e69ee6cfc095115445526_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_d08314d7b632e04fd42e69ee6cfc095115445526_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 72.09% (27680/38397) |
| Line Coverage | 55.59% (296216/532899) |
| Region Coverage | 52.29% (246958/472252) |
| Branch Coverage | 53.56% (107137/200048) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3330015641
##########
be/src/storage/segment/segment.cpp:
##########
@@ -85,6 +89,87 @@ namespace doris::segment_v2 {
class InvertedIndexIterator;
+namespace {
+
+Status build_segment_zonemap_context(Segment* segment, const Schema& schema,
+ const StorageReadOptions& read_options,
+ const VExprContextSPtrs& conjuncts, ZoneMapEvalContext* ctx) {
+ DORIS_CHECK(segment != nullptr);
+ DORIS_CHECK(ctx != nullptr);
+ std::set<int> slot_indexes;
+ for (const auto& conjunct : conjuncts) {
+ if (conjunct->root() != nullptr) {
+ conjunct->root()->collect_slot_column_ids(slot_indexes);
+ }
+ }
+ for (const int slot_index : slot_indexes) {
Review Comment:
这里是不是只需要处理表达式里只有一个slot依赖的列呢？和下面判断
```
if (slot_indexes.size() != 1) {
return std::nullopt;
}``` 一样呢
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3329969226
##########
be/src/exec/runtime_filter/runtime_filter_consumer_helper.h:
##########
@@ -39,6 +39,7 @@ class RuntimeFilterConsumerHelper {
// Called by Operator.
Status acquire_runtime_filter(RuntimeState* state, VExprContextSPtrs& conjuncts,
const RowDescriptor& row_descriptor);
+
// The un-arrival filters will be checked every time the scanner is scheduled.
Review Comment:
同上
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4796209131
<details>
<summary>ClickBench: <b>Total hot run time: 25.39 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 501cde75be748e273233042cab95400570d9a634, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.26 0.14 0.14
query4 1.62 0.14 0.14
query5 0.25 0.22 0.22
query6 1.27 1.05 1.06
query7 0.03 0.01 0.00
query8 0.06 0.04 0.04
query9 0.38 0.31 0.31
query10 0.54 0.56 0.57
query11 0.19 0.16 0.14
query12 0.18 0.14 0.14
query13 0.46 0.47 0.48
query14 1.03 1.02 1.00
query15 0.61 0.59 0.60
query16 0.33 0.32 0.33
query17 1.11 1.12 1.14
query18 0.23 0.21 0.21
query19 2.04 1.95 1.94
query20 0.02 0.01 0.01
query21 15.42 0.22 0.15
query22 4.91 0.05 0.05
query23 16.12 0.31 0.13
query24 2.92 0.43 0.33
query25 0.10 0.04 0.05
query26 0.76 0.22 0.16
query27 0.04 0.04 0.04
query28 3.46 0.93 0.54
query29 12.58 4.30 3.48
query30 0.27 0.15 0.16
query31 2.78 0.61 0.31
query32 3.22 0.59 0.49
query33 3.18 3.21 3.24
query34 15.72 4.22 3.55
query35 3.49 3.54 3.51
query36 0.56 0.43 0.42
query37 0.09 0.07 0.07
query38 0.05 0.04 0.03
query39 0.04 0.03 0.03
query40 0.19 0.16 0.15
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.05 0.03 0.04
Total cold run time: 96.8 s
Total hot run time: 25.39 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4733740128
<details>
<summary>TPC-DS: <b>Total hot run time: 176448 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b264ceb03f62a92109e263f696cc7aa99d35167b, data reload: false
query5 4415 618 482 482
query6 456 234 199 199
query7 4845 563 334 334
query8 377 223 229 223
query9 8772 4026 4003 4003
query10 464 372 286 286
query11 5832 2335 2137 2137
query12 154 124 97 97
query13 1256 577 429 429
query14 6367 5346 5015 5015
query14_1 4359 4702 4384 4384
query15 204 195 170 170
query16 1007 460 437 437
query17 1121 672 558 558
query18 2422 478 334 334
query19 202 184 145 145
query20 108 106 101 101
query21 234 156 132 132
query22 13584 13564 13360 13360
query23 17291 16396 16167 16167
query23_1 16246 16223 16238 16223
query24 7407 1728 1304 1304
query24_1 1329 1293 1286 1286
query25 544 440 379 379
query26 1364 349 204 204
query27 2603 610 390 390
query28 4494 2062 2047 2047
query29 1045 610 496 496
query30 335 265 220 220
query31 1126 1097 998 998
query32 106 58 57 57
query33 510 314 239 239
query34 1199 1132 671 671
query35 763 782 685 685
query36 1424 1422 1253 1253
query37 172 104 92 92
query38 3226 3169 3042 3042
query39 934 916 901 901
query39_1 884 882 888 882
query40 246 168 142 142
query41 74 83 67 67
query42 105 103 104 103
query43 315 329 278 278
query44 1447 784 780 780
query45 197 185 183 183
query46 1074 1189 759 759
query47 2411 2343 2196 2196
query48 410 431 299 299
query49 647 488 369 369
query50 1111 486 343 343
query51 4312 4358 4215 4215
query52 98 96 89 89
query53 269 280 216 216
query54 296 239 229 229
query55 86 82 73 73
query56 309 308 320 308
query57 1435 1414 1315 1315
query58 306 275 252 252
query59 1584 1641 1436 1436
query60 320 272 263 263
query61 178 178 177 177
query62 693 646 584 584
query63 252 212 212 212
query64 2607 825 661 661
query65 4893 4778 4808 4778
query66 1837 510 393 393
query67 29793 29548 29475 29475
query68 3147 1632 1034 1034
query69 427 307 281 281
query70 1105 1017 963 963
query71 355 332 323 323
query72 2974 2615 2349 2349
query73 837 756 460 460
query74 5084 4980 4734 4734
query75 2665 2673 2293 2293
query76 2316 1173 780 780
query77 373 391 282 282
query78 12506 12509 11758 11758
query79 1454 1161 770 770
query80 1308 542 481 481
query81 525 324 279 279
query82 669 159 119 119
query83 359 322 291 291
query84 280 166 125 125
query85 978 587 515 515
query86 434 311 284 284
query87 3367 3371 3163 3163
query88 3725 2805 2797 2797
query89 457 404 360 360
query90 1947 209 187 187
query91 205 206 159 159
query92 59 61 58 58
query93 1749 1532 985 985
query94 743 373 315 315
query95 772 507 537 507
query96 1077 859 342 342
query97 2728 2672 2560 2560
query98 219 205 215 205
query99 1183 1178 1017 1017
Total cold run time: 262966 ms
Total hot run time: 176448 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4737667958
# FE UT Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4734257431
# BE Regression && UT Coverage Report
Increment line coverage `69.51% (734/1056)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 58.45% (22413/38343) |
| Line Coverage | 41.86% (223408/533683) |
| Region Coverage | 38.10% (180205/473012) |
| Branch Coverage | 38.80% (77804/200529) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4785054738
<details>
<summary>TPC-DS: <b>Total hot run time: 174840 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f640be29bedf8dc904f15104f2d1a290b0569798, data reload: false
query5 4314 634 492 492
query6 460 212 199 199
query7 4846 592 332 332
query8 372 228 203 203
query9 8723 4049 4036 4036
query10 496 370 298 298
query11 5924 2363 2141 2141
query12 157 104 105 104
query13 1258 613 443 443
query14 6805 5435 5101 5101
query14_1 4385 4400 4392 4392
query15 201 194 184 184
query16 1005 489 429 429
query17 1096 689 553 553
query18 2762 489 345 345
query19 215 179 149 149
query20 113 109 111 109
query21 229 158 133 133
query22 13673 13690 13371 13371
query23 17339 16474 16278 16278
query23_1 16253 16347 16356 16347
query24 7469 1785 1360 1360
query24_1 1302 1349 1332 1332
query25 522 417 377 377
query26 1309 360 207 207
query27 2586 585 395 395
query28 4411 2042 2032 2032
query29 1096 652 474 474
query30 337 259 230 230
query31 1111 1087 1000 1000
query32 106 60 61 60
query33 512 320 252 252
query34 1208 1147 660 660
query35 748 812 686 686
query36 1424 1395 1244 1244
query37 149 105 89 89
query38 1905 1735 1669 1669
query39 916 927 904 904
query39_1 871 886 891 886
query40 240 162 145 145
query41 67 63 61 61
query42 95 96 90 90
query43 317 324 281 281
query44 1430 774 779 774
query45 199 188 180 180
query46 1040 1196 745 745
query47 2418 2361 2278 2278
query48 403 421 290 290
query49 629 467 357 357
query50 1032 420 319 319
query51 4461 4485 4334 4334
query52 86 86 75 75
query53 265 283 202 202
query54 275 228 207 207
query55 75 70 66 66
query56 293 282 301 282
query57 1443 1420 1364 1364
query58 293 261 250 250
query59 1614 1648 1458 1458
query60 286 263 263 263
query61 158 159 142 142
query62 688 658 593 593
query63 251 214 209 209
query64 2542 828 693 693
query65 4872 4846 4643 4643
query66 1776 550 421 421
query67 30179 29834 29594 29594
query68 3323 1507 929 929
query69 422 302 278 278
query70 1088 950 955 950
query71 345 334 378 334
query72 2950 2573 2368 2368
query73 870 738 459 459
query74 5160 4987 4795 4795
query75 2666 2663 2283 2283
query76 2354 1212 777 777
query77 355 388 291 291
query78 12451 12493 12042 12042
query79 1436 1124 753 753
query80 916 561 474 474
query81 501 317 287 287
query82 578 158 121 121
query83 418 320 292 292
query84 276 158 137 137
query85 988 608 507 507
query86 410 288 291 288
query87 1837 1848 1782 1782
query88 3721 2829 2789 2789
query89 452 396 359 359
query90 1835 203 195 195
query91 200 190 161 161
query92 63 63 58 58
query93 1485 1605 949 949
query94 635 364 317 317
query95 803 592 488 488
query96 1062 760 382 382
query97 2686 2683 2579 2579
query98 223 205 202 202
query99 1178 1183 1040 1040
Total cold run time: 260177 ms
Total hot run time: 174840 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3451574594
##########
be/src/storage/segment/segment_iterator.cpp:
##########
@@ -3400,6 +3419,107 @@ Status SegmentIterator::_construct_compound_expr_context() {
return Status::OK();
}
+Status SegmentIterator::_apply_expr_zonemap_to_row_ranges(const VExprContextSPtrs& conjuncts,
+ rowid_t min_rowid,
+ RowRanges* row_ranges) {
+ DORIS_CHECK(row_ranges != nullptr);
+ if (!expr_zonemap::is_expr_zonemap_filter_enabled(_opts.runtime_state) || conjuncts.empty() ||
+ row_ranges->is_empty()) {
+ return Status::OK();
+ }
+ // Page zone maps are addressed by reader-schema column ordinals. When storage expr slots need
+ // rebinding to the reader schema, VExprContext::clone() does not deep-copy the expression tree
+ // yet, so rebinding may mutate shared slot refs. Keep page-level expr-zonemap on layouts whose
+ // scan tuple slot ordinals already match the reader schema.
+ if (!storage_expr_slots_match_reader_schema(_opts)) {
+ return Status::OK();
+ }
+
+ std::unordered_map<int, VExprContextSPtrs> ctxs_by_slot;
+ for (const auto& conjunct : conjuncts) {
+ auto slot_index = expr_zonemap::single_slot_zonemap_index(conjunct);
+ if (slot_index.has_value()) {
+ ctxs_by_slot[*slot_index].emplace_back(conjunct);
+ }
+ }
+ // Page zone maps are stored per column. Multi-slot expressions need page alignment across
+ // multiple column readers and are therefore left to segment-level pruning for now.
+ if (ctxs_by_slot.empty()) {
+ return Status::OK();
+ }
+
+ ColumnIteratorOptions iter_opts {
+ .use_page_cache = _opts.use_page_cache,
+ .file_reader = _file_reader.get(),
+ .stats = _opts.stats,
+ .io_ctx = _opts.io_ctx,
+ };
+ for (const auto& [slot_index, slot_conjuncts] : ctxs_by_slot) {
+ if (cast_set<size_t>(slot_index) >= _schema->num_column_ids()) {
+ continue;
+ }
+ const auto cid = _schema->column_id(cast_set<size_t>(slot_index));
+ if (!_segment->can_apply_predicate_safely(cid, *_schema,
+ _opts.target_cast_type_for_variants, _opts)) {
+ continue;
+ }
+ const auto* field = _schema->column(cid);
Review Comment:
这应该是tablet column 不是field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3401727000
##########
be/src/exprs/expr_zonemap_filter.cpp:
##########
@@ -0,0 +1,249 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/expr_zonemap_filter.h"
+
+#include <algorithm>
+#include <set>
+
+#include "common/check.h"
+#include "common/logging.h"
+#include "core/column/column.h"
+#include "core/data_type/data_type_nullable.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vdirect_in_predicate.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/runtime_state.h"
+
+namespace doris::expr_zonemap {
+namespace {
+
+constexpr size_t kInZoneMapPointCheckThreshold = 64;
+
+VExprSPtr unwrap_identity_cast(const VExprSPtr& expr) {
+ if (expr->node_type() != TExprNodeType::CAST_EXPR || expr->get_num_children() != 1) {
+ return expr;
+ }
+ auto child = expr->get_child(0);
+ if (!remove_nullable(expr->data_type())->equals(*remove_nullable(child->data_type()))) {
+ return expr;
+ }
+ return unwrap_identity_cast(child);
+}
+
+std::optional<Field> field_from_literal_expr(const VExprSPtr& expr, DataTypePtr* literal_type) {
+ auto unwrapped = unwrap_identity_cast(expr);
+ auto literal = std::dynamic_pointer_cast<VLiteral>(unwrapped);
+ if (literal == nullptr) {
+ return std::nullopt;
+ }
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ *literal_type = literal->get_data_type();
+ return field;
+}
+
+struct SlotExpr {
+ int index;
+ DataTypePtr data_type;
+};
+
+std::optional<SlotExpr> slot_from_expr(const VExprSPtr& expr) {
+ auto unwrapped = unwrap_identity_cast(expr);
+ auto slot = std::dynamic_pointer_cast<VSlotRef>(unwrapped);
+ if (slot == nullptr) {
+ return std::nullopt;
+ }
+ return SlotExpr {.index = slot->column_id(), .data_type = slot->data_type()};
+}
+
+bool value_in_range(const Field& value, const Field& min_value, const Field& max_value) {
+ return field_greater_equal(value, min_value) && field_less_equal(value, max_value);
+}
+
+} // namespace
+
+std::optional<SlotLiteral> extract_slot_and_literal(const VExprSPtrs& args) {
+ if (args.size() != 2) {
+ return std::nullopt;
+ }
+
+ if (auto slot = slot_from_expr(args[0]); slot.has_value()) {
+ DataTypePtr literal_type;
+ auto literal = field_from_literal_expr(args[1], &literal_type);
+ if (!literal.has_value()) {
+ return std::nullopt;
+ }
+ return SlotLiteral {.slot_index = slot->index,
+ .slot_type = std::move(slot->data_type),
+ .literal = std::move(*literal),
+ .literal_type = std::move(literal_type),
+ .literal_on_left = false};
+ }
+
+ if (auto slot = slot_from_expr(args[1]); slot.has_value()) {
+ DataTypePtr literal_type;
+ auto literal = field_from_literal_expr(args[0], &literal_type);
+ if (!literal.has_value()) {
+ return std::nullopt;
+ }
+ return SlotLiteral {.slot_index = slot->index,
+ .slot_type = std::move(slot->data_type),
+ .literal = std::move(*literal),
+ .literal_type = std::move(literal_type),
+ .literal_on_left = true};
+ }
+
+ return std::nullopt;
+}
+
+bool range_stats_usable_for_zonemap(const segment_v2::ZoneMap& zone_map,
+ const DataTypePtr& data_type) {
+ if (zone_map.pass_all || zone_map.has_nan || zone_map.has_positive_inf ||
+ zone_map.has_negative_inf) {
+ return false;
+ }
+ DORIS_CHECK(data_type != nullptr);
+ auto primitive_type = remove_nullable(data_type)->get_primitive_type();
+ DORIS_CHECK(field_types_compatible(zone_map.min_value.get_type(), primitive_type));
+ DORIS_CHECK(field_types_compatible(zone_map.max_value.get_type(), primitive_type));
+ return true;
+}
+
+ZoneMapFilterResult eval_null_zonemap(const ZoneMapEvalContext& ctx, const VExprSPtrs& arguments,
+ bool is_null) {
+ DORIS_CHECK(arguments.size() == 1);
+ auto slot = slot_from_expr(arguments[0]);
+ DORIS_CHECK(slot.has_value());
+ const auto* zone_map_ref = ctx.zone_map(slot->index);
+ if (zone_map_ref == nullptr) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (is_null) {
+ return zone_map.has_null ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+ }
+ return zone_map.has_not_null ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+}
+
+bool can_eval_null_zonemap(const VExprSPtrs& arguments) {
+ if (arguments.size() != 1) {
Review Comment:
这里为什么会出现arg != 1的情况？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3450818224
##########
be/src/exprs/expr_zonemap_filter.cpp:
##########
@@ -0,0 +1,269 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/expr_zonemap_filter.h"
+
+#include <algorithm>
+#include <set>
+
+#include "common/check.h"
+#include "common/logging.h"
+#include "core/column/column.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/string_ref.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/runtime_state.h"
+
+namespace doris::expr_zonemap {
+namespace {
+
+constexpr size_t kInZoneMapPointCheckThreshold = 64;
+
+std::optional<Field> field_from_literal_expr(const VExprSPtr& expr, DataTypePtr* literal_type) {
+ auto literal = std::dynamic_pointer_cast<VLiteral>(expr);
+ if (literal == nullptr) {
+ return std::nullopt;
+ }
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ *literal_type = literal->get_data_type();
+ return field;
+}
+
+struct SlotExpr {
+ int index;
+ DataTypePtr data_type;
+};
+
+std::optional<SlotExpr> slot_from_expr(const VExprSPtr& expr) {
+ auto slot = std::dynamic_pointer_cast<VSlotRef>(expr);
+ if (slot == nullptr) {
+ return std::nullopt;
+ }
+ return SlotExpr {.index = slot->column_id(), .data_type = slot->data_type()};
+}
+
+bool value_in_range(const Field& value, const Field& min_value, const Field& max_value) {
+ return field_greater_equal(value, min_value) && field_less_equal(value, max_value);
+}
+
+} // namespace
+
+TExprNode create_texpr_node_from_hybrid_set_value(const void* data, const PrimitiveType& type,
+ int precision, int scale) {
+ if (is_string_type(type)) {
+ const auto* value = reinterpret_cast<const StringRef*>(data);
+ auto field = Field::create_field<TYPE_STRING>(String(value->data, value->size));
+ return create_texpr_node_from(field, type, precision, scale);
+ }
+ return create_texpr_node_from(data, type, precision, scale);
+}
+
+Status materialize_hybrid_set_for_zonemap_filter(HybridSetBase& set, const DataTypePtr& data_type,
+ InZonemapMaterializedSet* result) {
+ DORIS_CHECK(result != nullptr);
+ DORIS_CHECK(data_type != nullptr);
+ const auto value_type = remove_nullable(data_type);
+ DORIS_CHECK(value_type != nullptr);
+
+ result->contains_null = set.contain_null();
+ result->values.clear();
+ result->min_value = Field();
+ result->max_value = Field();
+
+ auto* iterator = set.begin();
+ while (iterator->has_next()) {
+ const void* value = iterator->get_value();
+ if (value != nullptr) {
+ TExprNode literal_node = create_texpr_node_from_hybrid_set_value(
+ value, value_type->get_primitive_type(), value_type->get_precision(),
+ value_type->get_scale());
+ auto literal = VLiteral::create_shared(literal_node);
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ result->values.emplace_back(std::move(field));
+ }
+ iterator->next();
+ }
+
+ if (!result->values.empty()) {
+ auto minmax = std::ranges::minmax_element(result->values, expr_zonemap::field_less);
+ result->min_value = *minmax.min;
+ result->max_value = *minmax.max;
+ }
+ return Status::OK();
+}
+
+std::optional<SlotLiteral> extract_slot_and_literal(const VExprSPtrs& args) {
+ if (args.size() != 2) {
+ return std::nullopt;
+ }
+
+ if (auto slot = slot_from_expr(args[0]); slot.has_value()) {
+ DataTypePtr literal_type;
+ auto literal = field_from_literal_expr(args[1], &literal_type);
+ if (!literal.has_value()) {
+ return std::nullopt;
+ }
+ return SlotLiteral {.slot_index = slot->index,
+ .slot_type = std::move(slot->data_type),
+ .literal = std::move(*literal),
+ .literal_type = std::move(literal_type),
+ .literal_on_left = false};
+ }
+
+ if (auto slot = slot_from_expr(args[1]); slot.has_value()) {
+ DataTypePtr literal_type;
+ auto literal = field_from_literal_expr(args[0], &literal_type);
+ if (!literal.has_value()) {
+ return std::nullopt;
+ }
+ return SlotLiteral {.slot_index = slot->index,
+ .slot_type = std::move(slot->data_type),
+ .literal = std::move(*literal),
+ .literal_type = std::move(literal_type),
+ .literal_on_left = true};
+ }
+
+ return std::nullopt;
+}
+
+bool range_stats_usable_for_zonemap(const segment_v2::ZoneMap& zone_map,
+ const DataTypePtr& data_type) {
+ if (zone_map.pass_all || zone_map.has_nan || zone_map.has_positive_inf ||
+ zone_map.has_negative_inf) {
+ return false;
+ }
+ DORIS_CHECK(data_type != nullptr);
+ auto primitive_type = remove_nullable(data_type)->get_primitive_type();
+ DORIS_CHECK(field_types_compatible(zone_map.min_value.get_type(), primitive_type));
+ DORIS_CHECK(field_types_compatible(zone_map.max_value.get_type(), primitive_type));
+ return true;
+}
+
+ZoneMapFilterResult eval_null_zonemap(const ZoneMapEvalContext& ctx, const VExprSPtrs& arguments,
+ bool is_null) {
+ DORIS_CHECK(arguments.size() == 1);
+ auto slot = slot_from_expr(arguments[0]);
+ DORIS_CHECK(slot.has_value());
+ const auto* zone_map_ref = ctx.zone_map(slot->index);
+ if (zone_map_ref == nullptr) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (is_null) {
+ return zone_map.has_null ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+ }
+ return zone_map.has_not_null ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+}
+
+ZoneMapFilterResult eval_in_zonemap(const ZoneMapEvalContext& ctx, const VExprSPtr& slot_expr,
+ bool is_not_in, const std::vector<Field>& values,
+ const Field& min_value, const Field& max_value) {
+ auto slot = slot_from_expr(slot_expr);
+ DORIS_CHECK(slot.has_value());
+ if (values.empty()) {
+ return is_not_in ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+ }
+
+ DORIS_CHECK(!min_value.is_null());
+ DORIS_CHECK(!max_value.is_null());
+ auto data_type = remove_nullable(slot->data_type);
+ DORIS_CHECK(data_type != nullptr);
+ DORIS_CHECK(field_types_compatible(min_value.get_type(), data_type->get_primitive_type()));
+ DORIS_CHECK(field_types_compatible(max_value.get_type(), data_type->get_primitive_type()));
+
+ auto slot_type = fetch_compatible_slot_type(ctx, slot->index, slot->data_type);
+ if (slot_type == nullptr) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ const auto* zone_map_ref = ctx.zone_map(slot->index);
+ if (zone_map_ref == nullptr) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (!zone_map.has_not_null) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+
+ if (!range_stats_usable_for_zonemap(zone_map, slot_type)) {
+ return unsupported_zonemap_filter(ctx);
+ }
+
+ if (is_not_in) {
+ if (zone_map.min_value == zone_map.max_value) {
+ const bool only_value_is_filtered = std::ranges::any_of(
+ values, [&](const Field& value) { return value == zone_map.min_value; });
+ return only_value_is_filtered ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ }
+ return ZoneMapFilterResult::kMayMatch;
+ }
+
+ if (field_less(zone_map.max_value, min_value) || field_greater(zone_map.min_value, max_value)) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+
+ if (values.size() > kInZoneMapPointCheckThreshold) {
+ ++ctx.stats.in_zonemap_range_only_count;
+ return ZoneMapFilterResult::kMayMatch;
+ }
+
+ ++ctx.stats.in_zonemap_point_check_count;
+ for (const auto& value : values) {
+ if (value_in_range(value, zone_map.min_value, zone_map.max_value)) {
+ return ZoneMapFilterResult::kMayMatch;
+ }
+ }
+ return ZoneMapFilterResult::kNoMatch;
+}
+
+std::optional<int> single_slot_zonemap_index(const VExprContextSPtr& ctx) {
Review Comment:
这个里面的方法都一行注释都没有啊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3403941726
##########
be/src/exprs/runtime_filter_expr.cpp:
##########
@@ -190,4 +193,17 @@ Status RuntimeFilterExpr::execute_filter(VExprContext* context, const Block* blo
return Status::OK();
}
+ZoneMapFilterResult RuntimeFilterExpr::evaluate_zonemap_filter(
+ const ZoneMapEvalContext& ctx) const {
+ return _impl->evaluate_zonemap_filter(ctx);
+}
+
+bool RuntimeFilterExpr::can_evaluate_zonemap_filter() const {
+ return !_null_aware && _impl->can_evaluate_zonemap_filter();
Review Comment:
`_null_aware` 这个值就是来自 runtime filter 的内部数据：
```cpp
bool RuntimeFilterWrapper::contain_null() const {
if (get_real_type() == RuntimeFilterType::BLOOM_FILTER) {
return _bloom_filter_func->contain_null();
}
if (_hybrid_set) {
return _hybrid_set->contain_null();
}
if (_minmax_func) {
return _minmax_func->contain_null();
}
return false;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4694916753
# FE Regression Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_8a4c4d46c63a8083b97c25e9177806c0d0e9773d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_8a4c4d46c63a8083b97c25e9177806c0d0e9773d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3272297851
##########
be/src/format/parquet/vparquet_reader.cpp:
##########
@@ -1443,6 +1577,76 @@ Status ParquetReader::_process_column_stat_filter(
return Status::OK();
}
+Status ParquetReader::_process_expr_zonemap_filter(const tparquet::RowGroup& row_group,
+ bool* filter_group) {
+ DORIS_CHECK(filter_group != nullptr);
+ if (!config::enable_expr_zonemap_filter || _zonemap_filter_conjuncts == nullptr ||
+ _zonemap_filter_conjuncts->empty() || !_enable_filter_by_min_max) {
+ return Status::OK();
+ }
+
+ std::set<int> column_ids;
+ for (const auto& conjunct : *_zonemap_filter_conjuncts) {
+ if (conjunct->root() != nullptr) {
+ conjunct->root()->collect_slot_column_ids(column_ids);
+ }
+ }
+ if (column_ids.empty()) {
+ return Status::OK();
+ }
+
+ ZoneMapEvalContext ctx;
+ for (const int cid : column_ids) {
+ if (cid < 0 || cid >= _tuple_descriptor->slots().size()) {
+ continue;
+ }
+ auto* slot = _tuple_descriptor->slots()[cid];
+ ZoneMapEvalContext::SlotZoneMap slot_zone_map;
+ slot_zone_map.data_type = slot->type();
+ if (!_table_info_node_ptr->children_column_exists(slot->col_name())) {
+ ctx.slots.emplace(cid, std::move(slot_zone_map));
+ continue;
+ }
+ const auto& file_col_name =
+ _table_info_node_ptr->children_file_column_name(slot->col_name());
+ const FieldSchema* col_schema = _file_metadata->schema().get_column(file_col_name);
+ int parquet_col_id = col_schema->physical_column_index;
+ const auto& meta_data = row_group.columns[parquet_col_id].meta_data;
+
Review Comment:
已处理：row-group expr zonemap 路径现在会在访问 `row_group.columns[parquet_col_id]` 前检查 `parquet_col_id < 0`，对复杂类型父列按无 zonemap 统计保守 fallback，不再访问负下标。
同时补了 `ParquetExprTest.test_expr_zonemap_row_group_filter_skips_complex_parent_column` 覆盖该场景，并已运行：
- `./run-be-ut.sh --run --filter=ParquetExprTest.test_expr_zonemap_*`
- `build-support/clang-format.sh`
- `build-support/check-format.sh`
- `git diff --check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4645977629
# FE Regression Coverage Report
Increment line coverage ` 1.67% (2/120)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_d08314d7b632e04fd42e69ee6cfc095115445526_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_d08314d7b632e04fd42e69ee6cfc095115445526_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3451546798
##########
be/src/exprs/expr_zonemap_filter.cpp:
##########
@@ -0,0 +1,269 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/expr_zonemap_filter.h"
+
+#include <algorithm>
+#include <set>
+
+#include "common/check.h"
+#include "common/logging.h"
+#include "core/column/column.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/string_ref.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/runtime_state.h"
+
+namespace doris::expr_zonemap {
+namespace {
+
+constexpr size_t kInZoneMapPointCheckThreshold = 64;
+
+std::optional<Field> field_from_literal_expr(const VExprSPtr& expr, DataTypePtr* literal_type) {
+ auto literal = std::dynamic_pointer_cast<VLiteral>(expr);
+ if (literal == nullptr) {
+ return std::nullopt;
+ }
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ *literal_type = literal->get_data_type();
+ return field;
+}
+
+struct SlotExpr {
Review Comment:
我们一旦多了一个数据结构，后面就可能会扩散的用，各种概念就会看着很乱
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3451553354
##########
be/src/exprs/expr_zonemap_filter.cpp:
##########
@@ -0,0 +1,269 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/expr_zonemap_filter.h"
+
+#include <algorithm>
+#include <set>
+
+#include "common/check.h"
+#include "common/logging.h"
+#include "core/column/column.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/string_ref.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/runtime_state.h"
+
+namespace doris::expr_zonemap {
+namespace {
+
+constexpr size_t kInZoneMapPointCheckThreshold = 64;
+
+std::optional<Field> field_from_literal_expr(const VExprSPtr& expr, DataTypePtr* literal_type) {
+ auto literal = std::dynamic_pointer_cast<VLiteral>(expr);
+ if (literal == nullptr) {
+ return std::nullopt;
+ }
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ *literal_type = literal->get_data_type();
+ return field;
+}
+
+struct SlotExpr {
+ int index;
+ DataTypePtr data_type;
+};
+
+std::optional<SlotExpr> slot_from_expr(const VExprSPtr& expr) {
+ auto slot = std::dynamic_pointer_cast<VSlotRef>(expr);
+ if (slot == nullptr) {
+ return std::nullopt;
+ }
+ return SlotExpr {.index = slot->column_id(), .data_type = slot->data_type()};
+}
+
+bool value_in_range(const Field& value, const Field& min_value, const Field& max_value) {
+ return field_greater_equal(value, min_value) && field_less_equal(value, max_value);
+}
+
+} // namespace
+
+TExprNode create_texpr_node_from_hybrid_set_value(const void* data, const PrimitiveType& type,
+ int precision, int scale) {
+ if (is_string_type(type)) {
+ const auto* value = reinterpret_cast<const StringRef*>(data);
+ auto field = Field::create_field<TYPE_STRING>(String(value->data, value->size));
+ return create_texpr_node_from(field, type, precision, scale);
+ }
+ return create_texpr_node_from(data, type, precision, scale);
+}
+
+Status materialize_hybrid_set_for_zonemap_filter(HybridSetBase& set, const DataTypePtr& data_type,
+ InZonemapMaterializedSet* result) {
+ DORIS_CHECK(result != nullptr);
+ DORIS_CHECK(data_type != nullptr);
+ const auto value_type = remove_nullable(data_type);
+ DORIS_CHECK(value_type != nullptr);
+
+ result->contains_null = set.contain_null();
+ result->values.clear();
+ result->min_value = Field();
+ result->max_value = Field();
+
+ auto* iterator = set.begin();
+ while (iterator->has_next()) {
+ const void* value = iterator->get_value();
+ if (value != nullptr) {
+ TExprNode literal_node = create_texpr_node_from_hybrid_set_value(
+ value, value_type->get_primitive_type(), value_type->get_precision(),
+ value_type->get_scale());
+ auto literal = VLiteral::create_shared(literal_node);
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ result->values.emplace_back(std::move(field));
+ }
+ iterator->next();
+ }
+
+ if (!result->values.empty()) {
+ auto minmax = std::ranges::minmax_element(result->values, expr_zonemap::field_less);
+ result->min_value = *minmax.min;
+ result->max_value = *minmax.max;
+ }
+ return Status::OK();
+}
+
+std::optional<SlotLiteral> extract_slot_and_literal(const VExprSPtrs& args) {
+ if (args.size() != 2) {
+ return std::nullopt;
+ }
+
+ if (auto slot = slot_from_expr(args[0]); slot.has_value()) {
+ DataTypePtr literal_type;
+ auto literal = field_from_literal_expr(args[1], &literal_type);
+ if (!literal.has_value()) {
+ return std::nullopt;
+ }
+ return SlotLiteral {.slot_index = slot->index,
+ .slot_type = std::move(slot->data_type),
+ .literal = std::move(*literal),
+ .literal_type = std::move(literal_type),
+ .literal_on_left = false};
Review Comment:
literal_on_left 这种字段都得有一个注释，解释啥含义，并且举例子
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3451690471
##########
be/src/storage/index/zone_map/zonemap_eval_context.cpp:
##########
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "storage/index/zone_map/zonemap_eval_context.h"
+
+#include <algorithm>
+#include <string>
+
+#include "runtime/runtime_profile.h"
+
+namespace doris {
+
+const segment_v2::ZoneMap* ZoneMapEvalContext::zone_map(int slot_index) const {
Review Comment:
这里返回指针应该不会有多大的性能提升，因为zonemap？
如果需要的话，为啥不直接返回shared ptr？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3453511905
##########
be/src/format/parquet/vparquet_reader.cpp:
##########
@@ -1065,6 +1077,43 @@ void ParquetReader::set_condition_cache_context(std::shared_ptr<ConditionCacheCo
}
}
+bool ParquetReader::_expr_zonemap_page_slot_index(const VExprContextSPtr& conjunct,
+ int* cid) const {
+ DORIS_CHECK(cid != nullptr);
+ if (conjunct == nullptr || conjunct->root() == nullptr ||
+ !conjunct->root()->can_evaluate_zonemap_filter()) {
+ return false;
+ }
+
+ std::set<int> column_ids;
+ conjunct->root()->collect_slot_column_ids(column_ids);
+ if (column_ids.size() != 1) {
+ return false;
+ }
+
+ const int slot_index = *column_ids.begin();
+ if (slot_index < 0 || static_cast<size_t>(slot_index) >= _tuple_descriptor->slots().size()) {
+ return false;
+ }
+ auto* slot = _tuple_descriptor->slots()[slot_index];
+ if (!_exists_in_file(slot->col_name()) || !_type_matches(slot_index)) {
Review Comment:
这种就不走 zonemap 过滤
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3395968401
##########
regression-test/suites/query_p0/expr_zonemap/test_expr_zonemap_pruning.groovy:
##########
@@ -0,0 +1,227 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import java.util.regex.Matcher
+import java.util.regex.Pattern
+import org.apache.doris.regression.action.ProfileAction
+
+suite("test_expr_zonemap_pruning") {
+ sql """ set enable_common_expr_pushdown = true """
+ sql """ set enable_profile = true """
+ sql """ set profile_level = 2 """
+
+ sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning """
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning (
+ id INT,
+ v VARCHAR(32)
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+
+ sql """
+ INSERT INTO test_expr_zonemap_pruning
+ SELECT CAST(number AS INT), CONCAT('a', CAST(number AS STRING))
+ FROM numbers("number" = "4096")
+ """
+ sql """ sync """
+
+ def profileAction = new ProfileAction(context)
+
+ def getProfileByToken = { String token ->
+ for (int retry = 0; retry < 10; ++retry) {
+ List profileData = profileAction.getProfileList()
+ for (final def profileItem in profileData) {
+ if (profileItem["Sql Statement"].toString().contains(token)) {
+ return profileAction.getProfile(profileItem["Profile ID"].toString())
+ }
+ }
+ Thread.sleep(500)
+ }
+ throw new IllegalStateException("Missing profile for token: " + token)
+ }
+
+ def counterSum = { String profile, String counterName ->
+ Pattern pattern = Pattern.compile(Pattern.quote(counterName) + ":\\s*([0-9,]+)")
+ Matcher matcher = pattern.matcher(profile)
+ long sum = 0
+ while (matcher.find()) {
+ sum += Long.parseLong(matcher.group(1).replace(",", ""))
+ }
+ return sum
+ }
+
+ def assertExprZonemapPruned = { String token ->
+ String profile = ""
+ long filteredSegments = 0
+ long filteredPages = 0
+ for (int retry = 0; retry < 20; ++retry) {
+ profile = getProfileByToken(token).toString()
+ filteredSegments = counterSum(profile, "ExprZoneMapFilteredSegments")
+ filteredPages = counterSum(profile, "ExprZoneMapFilteredPages")
+ if (filteredSegments + filteredPages > 0) {
+ logger.info("${token} Profile Data: ${profile}")
+ return
+ }
+ Thread.sleep(500)
+ }
+ logger.info("${token} Profile Data: ${profile}")
+ assertTrue(filteredSegments + filteredPages > 0)
+ }
+
+ def assertProfileCounterPositive = { String token, String counterName ->
+ String profile = ""
+ long counterValue = 0
+ for (int retry = 0; retry < 20; ++retry) {
+ profile = getProfileByToken(token).toString()
+ counterValue = counterSum(profile, counterName)
+ if (counterValue > 0) {
+ logger.info("${token} Profile Data: ${profile}")
+ return
+ }
+ Thread.sleep(500)
+ }
+ logger.info("${token} Profile Data: ${profile}")
+ assertTrue(counterValue > 0)
+ }
+
+ def matchedRows = sql """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning WHERE starts_with(v, 'a')
+ """
+ assertEquals(4096L, matchedRows[0][0] as long)
Review Comment:
The deterministic query results in this new regression suite are checked with Groovy `assertEquals`, and the PR does not add a generated `.out` file under `regression-test/data/...`. Doris regression tests should record deterministic SQL results with `qt_sql`/`order_qt` (or explicit `ORDER BY`) and commit the generated output; direct assertions should be reserved for checks that cannot be represented in golden output, such as profile counters. Please convert these result checks to `qt_...` cases and add the generated `.out` file.
##########
regression-test/suites/query_p1/expr_zonemap/test_expr_zonemap_pruning_p1.groovy:
##########
@@ -0,0 +1,157 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import java.util.regex.Matcher
+import java.util.regex.Pattern
+import org.apache.doris.regression.action.ProfileAction
+
+suite("test_expr_zonemap_pruning_p1") {
+ sql """ set enable_common_expr_pushdown = true """
+ sql """ set enable_profile = true """
+ sql """ set profile_level = 2 """
+
+ sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_p1 """
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_p1 (
+ id INT,
+ k BIGINT,
+ v VARCHAR(32),
+ nullable_v VARCHAR(32) NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_p1
+ SELECT CAST(number AS INT), CAST(number AS BIGINT),
+ CONCAT('left_', CAST(number AS STRING)), NULL
+ FROM numbers("number" = "2048")
+ """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_p1
+ SELECT CAST(number + 2048 AS INT), CAST(number + 2048 AS BIGINT),
+ CONCAT('right_', CAST(number + 2048 AS STRING)),
+ CONCAT('not_null_', CAST(number + 2048 AS STRING))
+ FROM numbers("number" = "2048")
+ """
+ sql """ sync """
+
+ def profileAction = new ProfileAction(context)
+
+ def getProfileByToken = { String token ->
+ for (int retry = 0; retry < 10; ++retry) {
+ List profileData = profileAction.getProfileList()
+ for (final def profileItem in profileData) {
+ if (profileItem["Sql Statement"].toString().contains(token)) {
+ return profileAction.getProfile(profileItem["Profile ID"].toString())
+ }
+ }
+ Thread.sleep(500)
+ }
+ throw new IllegalStateException("Missing profile for token: " + token)
+ }
+
+ def counterSum = { String profile, String counterName ->
+ Pattern pattern = Pattern.compile(Pattern.quote(counterName) + ":\\s*([0-9,]+)")
+ Matcher matcher = pattern.matcher(profile)
+ long sum = 0
+ while (matcher.find()) {
+ sum += Long.parseLong(matcher.group(1).replace(",", ""))
+ }
+ return sum
+ }
+
+ def assertProfileCounterPositive = { String token, String counterName ->
+ String profile = ""
+ long counterValue = 0
+ for (int retry = 0; retry < 20; ++retry) {
+ profile = getProfileByToken(token).toString()
+ counterValue = counterSum(profile, counterName)
+ if (counterValue > 0) {
+ logger.info("${token} Profile Data: ${profile}")
+ return
+ }
+ Thread.sleep(500)
+ }
+ logger.info("${token} Profile Data: ${profile}")
+ assertTrue(counterValue > 0)
+ }
+
+ def assertExprZonemapPruned = { String token ->
+ String profile = ""
+ long filteredSegments = 0
+ long filteredPages = 0
+ for (int retry = 0; retry < 20; ++retry) {
+ profile = getProfileByToken(token).toString()
+ filteredSegments = counterSum(profile, "ExprZoneMapFilteredSegments")
+ filteredPages = counterSum(profile, "ExprZoneMapFilteredPages")
+ if (filteredSegments + filteredPages > 0) {
+ logger.info("${token} Profile Data: ${profile}")
+ return
+ }
+ Thread.sleep(500)
+ }
+ logger.info("${token} Profile Data: ${profile}")
+ assertTrue(filteredSegments + filteredPages > 0)
+ }
+
+ def notNullToken = "expr_zonemap_pruning_p1_is_not_null_" + UUID.randomUUID().toString()
+ def notNullRows = sql """
+ SELECT '${notNullToken}', COUNT(*) FROM test_expr_zonemap_pruning_p1
+ WHERE nullable_v IS NOT NULL OR starts_with(v, 'never_')
+ """
+ assertEquals(2048L, notNullRows[0][1] as long)
Review Comment:
Same issue in this P1 suite: deterministic SQL result checks use Groovy `assertEquals`, and the PR does not add a generated `.out` file. Please use `qt_sql`/`order_qt` for the expected result queries and commit the generated output, leaving only the profile-counter checks as Groovy assertions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4680757604
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4687015724
# BE Regression && UT Coverage Report
Increment line coverage `88.84% (931/1048)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_19c67233fc78266e5f15df46446848ee09d82292_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_19c67233fc78266e5f15df46446848ee09d82292_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 69.43% (26616/38333) |
| Line Coverage | 52.63% (280583/533089) |
| Region Coverage | 48.99% (231402/472309) |
| Branch Coverage | 50.18% (100406/200102) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3394257778
##########
be/src/exprs/function/function_string.cpp:
##########
@@ -1341,8 +1349,93 @@ using FunctionCrc32 = FunctionUnaryToType<Crc32Impl, NameCrc32>;
using FunctionStringUTF8Length = FunctionUnaryToType<StringUtf8LengthImpl, NameStringUtf8Length>;
using FunctionStringSpace = FunctionUnaryToType<StringSpace, NameStringSpace>;
using FunctionIsValidUTF8 = FunctionUnaryToType<IsValidUTF8Impl, NameIsValidUTF8>;
-using FunctionStringStartsWith =
- FunctionBinaryToType<DataTypeString, DataTypeString, StringStartsWithImpl, NameStartsWith>;
+
+class FunctionStringStartsWith : public FunctionBinaryToType<DataTypeString, DataTypeString,
+ StringStartsWithImpl, NameStartsWith> {
+public:
+ static FunctionPtr create() { return std::make_shared<FunctionStringStartsWith>(); }
+
+ ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx,
+ const VExprSPtrs& arguments) const override {
+ auto slot_literal = expr_zonemap::extract_slot_and_literal(arguments);
+ DORIS_CHECK(slot_literal.has_value());
+ DORIS_CHECK(!slot_literal->literal_on_left);
+ DORIS_CHECK(!slot_literal->literal.is_null());
+ DORIS_CHECK(slot_literal->slot_type != nullptr);
+ DORIS_CHECK(slot_literal->literal_type != nullptr);
+ DORIS_CHECK(is_string_type(remove_nullable(slot_literal->slot_type)->get_primitive_type()));
+ DORIS_CHECK(
+ is_string_type(remove_nullable(slot_literal->literal_type)->get_primitive_type()));
+
+ auto slot_type = expr_zonemap::fetch_compatible_slot_type(ctx, slot_literal->slot_index,
+ slot_literal->slot_type);
+ if (slot_type == nullptr) {
+ return ZoneMapFilterResult::kUnsupported;
+ }
+ const auto* zone_map_ref = expr_zonemap::fetch_zone_map(ctx, slot_literal->slot_index);
+ if (zone_map_ref == nullptr) {
+ return ZoneMapFilterResult::kUnsupported;
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (!zone_map.has_not_null) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ if (!expr_zonemap::range_stats_usable_for_zonemap(zone_map, slot_type)) {
+ return unsupported_zonemap_filter(ctx);
+ }
+
+ const auto prefix = slot_literal->literal.as_string_view();
Review Comment:
这个是不是可以放到这个can_evaluate_zonemap_filter里面直接判断了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4687094109
# FE Regression Coverage Report
Increment line coverage ` 6.67% (2/30)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_19c67233fc78266e5f15df46446848ee09d82292_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_19c67233fc78266e5f15df46446848ee09d82292_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4697276571
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4775427104
Latest update pushed to `mrhhsg:zonemap` (`da4ac2deea94a27599449dd28373f96b94d39ad4`).
Validation:
- BE UT: `./run-be-ut.sh --run -j 16 --filter='ExprZonemapFilterTest.*:ParquetExprTest.test_expr_zonemap_page_filter_prunes_pages_and_intersects_ranges:ParquetExprTest.test_page_index_filter_applies_expr_zonemap_without_pushdown_predicate:ParquetExprTest.test_page_index_filter_skips_column_index_for_unsupported_expr_zonemap:ParquetExprTest.test_expr_zonemap_page_filter_handles_all_null_pages:ParquetExprTest.test_expr_zonemap_page_filter_intersects_multiple_columns:ParquetExprTest.test_expr_zonemap_page_filter_skips_when_disabled_or_no_candidate:ParquetExprTest.test_expr_zonemap_page_filter_skips_unsupported_columns_and_metadata:ParquetExprTest.test_expr_zonemap_page_filter_keeps_unsupported_results_and_counts_stats:ParquetExprTest.test_expr_zonemap_row_group_filter_skips_complex_parent_column:ParquetExprTest.test_expr_zonemap_row_group_filter_skips_type_mismatch_column'` — passed, 30 tests.
- Regression: `doris-local-regression --network 10.26.20.3/24 all -d query_p0/expr_zonemap -s test_expr_zonemap_pruning` — passed, 1 suite.
- Format: `build-support/clang-format.sh` and `git diff --check` — passed.
- Static analysis: `build-support/run-clang-tidy.sh --build-dir be/build_Release` was run, but the local run is blocked by toolchain/baseline issues (`stddef.h` not found and existing clang-tidy warnings such as unmatched `NOLINTEND` / old-file checks), so it did not complete cleanly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4508601117
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3329971245
##########
be/src/exprs/function/function.h:
##########
@@ -226,6 +229,17 @@ class IFunctionBase {
virtual bool can_push_down_to_index() const { return false; }
virtual bool is_blockable() const { return false; }
+
+ virtual bool is_deterministic() const { return false; }
Review Comment:
应该改成线性变化，而不是确定性
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4530979842
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4697707937
# FE Regression Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_76d48a1883c19a3e0f6b8c8d16755da1afb8b8f6_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_76d48a1883c19a3e0f6b8c8d16755da1afb8b8f6_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4788897954
# BE Regression && UT Coverage Report
Increment line coverage `92.82% (931/1003)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_501cde75be748e273233042cab95400570d9a634_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_501cde75be748e273233042cab95400570d9a634_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 72.52% (27851/38404) |
| Line Coverage | 55.81% (298586/534997) |
| Region Coverage | 52.44% (248672/474217) |
| Branch Coverage | 53.66% (108001/201263) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4715941184
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/abd5ad27998383128193300ac0e21b531f4e41a8_abd5ad27998383128193300ac0e21b531f4e41a8_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/abd5ad27998383128193300ac0e21b531f4e41a8_abd5ad27998383128193300ac0e21b531f4e41a8_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.32% (1889/2443) |
| Line Coverage | 64.39% (33950/52725) |
| Region Coverage | 64.75% (17450/26948) |
| Branch Coverage | 53.93% (9338/17316) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4733431763
# FE UT Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4784992897
<details>
<summary>TPC-H: <b>Total hot run time: 29766 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f640be29bedf8dc904f15104f2d1a290b0569798, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17763 4136 4391 4136
q2 2033 321 200 200
q3 10584 1463 864 864
q4 4805 481 343 343
q5 8240 896 585 585
q6 295 173 136 136
q7 826 845 633 633
q8 10350 1592 1618 1592
q9 5714 4552 4540 4540
q10 6850 1807 1514 1514
q11 513 347 316 316
q12 732 567 436 436
q13 18086 3397 2783 2783
q14 268 273 242 242
q15 q16 793 778 710 710
q17 1035 902 1011 902
q18 7187 5768 5519 5519
q19 1181 1354 1017 1017
q20 759 673 529 529
q21 5620 2655 2472 2472
q22 446 357 297 297
Total cold run time: 104080 ms
Total hot run time: 29766 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4373 4309 4275 4275
q2 304 345 240 240
q3 4566 4990 4420 4420
q4 2098 2195 1410 1410
q5 4487 4390 4563 4390
q6 315 229 146 146
q7 2106 1753 1621 1621
q8 2516 2204 2200 2200
q9 8300 8131 8227 8131
q10 4896 4997 4339 4339
q11 569 410 367 367
q12 750 757 558 558
q13 3196 3602 2905 2905
q14 294 302 299 299
q15 q16 721 744 639 639
q17 1409 1368 1354 1354
q18 7903 7343 6922 6922
q19 1131 1130 1105 1105
q20 2235 2273 1975 1975
q21 5347 4601 4552 4552
q22 539 467 400 400
Total cold run time: 58055 ms
Total hot run time: 52248 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4785299876
# FE Regression Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_f640be29bedf8dc904f15104f2d1a290b0569798_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_f640be29bedf8dc904f15104f2d1a290b0569798_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4484956635
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3450806176
##########
be/src/storage/segment/segment_iterator.h:
##########
@@ -73,6 +73,12 @@ class InvertedIndexIterator;
class RowRanges;
class IndexIterator;
+Status rebind_storage_exprs_to_reader_schema(const StorageReadOptions& opts, const Schema& schema,
+ const VExprContextSPtrs& common_exprs,
+ std::map<ColumnId, VExprContextSPtr>& virtual_exprs);
+
+bool storage_expr_slots_match_reader_schema(const StorageReadOptions& read_options);
Review Comment:
这个函数是做啥用的？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3451642232
##########
be/src/exprs/vdirect_in_predicate.h:
##########
@@ -148,8 +173,34 @@ class VDirectInPredicate final : public VExpr {
return Status::OK();
}
+ Status _materialize_for_zonemap_filter() {
Review Comment:
什么含义？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4742762460
# BE Regression && UT Coverage Report
Increment line coverage `80.97% (855/1056)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 64.08% (24603/38397) |
| Line Coverage | 47.88% (255991/534630) |
| Region Coverage | 44.57% (211123/473727) |
| Branch Coverage | 45.64% (91744/201002) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4733632248
<details>
<summary>TPC-H: <b>Total hot run time: 29960 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b264ceb03f62a92109e263f696cc7aa99d35167b, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17586 4075 4076 4075
q2 2014 311 192 192
q3 10906 1448 818 818
q4 4821 478 345 345
q5 8499 840 576 576
q6 347 176 138 138
q7 855 812 610 610
q8 10557 1633 1635 1633
q9 6084 4550 4505 4505
q10 6834 1801 1506 1506
q11 523 341 310 310
q12 718 552 437 437
q13 18228 3715 2813 2813
q14 267 263 242 242
q15 q16 801 771 711 711
q17 1024 1008 1059 1008
q18 6800 5775 5610 5610
q19 1208 1226 1092 1092
q20 754 671 561 561
q21 5527 2657 2487 2487
q22 430 352 291 291
Total cold run time: 104783 ms
Total hot run time: 29960 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4618 4332 4252 4252
q2 324 363 218 218
q3 4670 5009 4385 4385
q4 2075 2172 1366 1366
q5 4456 4317 4760 4317
q6 254 212 153 153
q7 2013 1811 1613 1613
q8 2521 2171 2149 2149
q9 8013 8012 8039 8012
q10 5008 4761 4297 4297
q11 570 438 376 376
q12 751 784 547 547
q13 3307 3693 2967 2967
q14 305 303 287 287
q15 q16 731 722 647 647
q17 1357 1348 1340 1340
q18 7915 7276 6753 6753
q19 1164 1103 1105 1103
q20 2222 2214 1941 1941
q21 5286 4682 4481 4481
q22 556 462 426 426
Total cold run time: 58116 ms
Total hot run time: 51630 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4734056519
# BE Regression && UT Coverage Report
Increment line coverage `80.78% (853/1056)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 64.16% (24602/38343) |
| Line Coverage | 47.96% (255964/533683) |
| Region Coverage | 44.68% (211323/473012) |
| Branch Coverage | 45.79% (91824/200529) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4785248708
# BE Regression && UT Coverage Report
Increment line coverage `92.06% (939/1020)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_f640be29bedf8dc904f15104f2d1a290b0569798_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_f640be29bedf8dc904f15104f2d1a290b0569798_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 74.23% (28511/38408) |
| Line Coverage | 58.11% (310879/535021) |
| Region Coverage | 55.04% (261030/474227) |
| Branch Coverage | 56.33% (113376/201267) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4787774099
# FE Regression Coverage Report
Increment line coverage ` 2.41% (2/83)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_501cde75be748e273233042cab95400570d9a634_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_501cde75be748e273233042cab95400570d9a634_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4785438746
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/501cde75be748e273233042cab95400570d9a634_501cde75be748e273233042cab95400570d9a634_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/501cde75be748e273233042cab95400570d9a634_501cde75be748e273233042cab95400570d9a634_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.36% (1893/2447) |
| Line Coverage | 64.42% (33977/52739) |
| Region Coverage | 64.82% (17477/26962) |
| Branch Coverage | 53.97% (9352/17328) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3457970359
##########
be/src/exprs/expr_zonemap_filter.cpp:
##########
@@ -0,0 +1,267 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/expr_zonemap_filter.h"
+
+#include <algorithm>
+#include <set>
+
+#include "common/check.h"
+#include "common/logging.h"
+#include "core/column/column.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/string_ref.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/runtime_state.h"
+
+namespace doris::expr_zonemap {
+namespace {
+
+constexpr size_t kInZoneMapPointCheckThreshold = 64;
+
+std::optional<Field> field_from_literal_expr(const VExprSPtr& expr, DataTypePtr* literal_type) {
Review Comment:
这种的，为啥不直接返回pair 啊？ 一个用return value，一个用指针感觉怪怪的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4506618252
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4536254719
# BE UT Coverage Report
Increment line coverage `57.84% (668/1155)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/3704e2d2892b9d0a7da311a4236a2371c84b889b_3704e2d2892b9d0a7da311a4236a2371c84b889b/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/3704e2d2892b9d0a7da311a4236a2371c84b889b_3704e2d2892b9d0a7da311a4236a2371c84b889b/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.84% (20937/38891) |
| Line Coverage | 37.46% (198575/530108) |
| Region Coverage | 33.77% (155649/460950) |
| Branch Coverage | 34.74% (67815/195188) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3330511846
##########
be/src/storage/segment/segment.cpp:
##########
@@ -85,6 +89,87 @@ namespace doris::segment_v2 {
class InvertedIndexIterator;
+namespace {
+
+Status build_segment_zonemap_context(Segment* segment, const Schema& schema,
+ const StorageReadOptions& read_options,
+ const VExprContextSPtrs& conjuncts, ZoneMapEvalContext* ctx) {
+ DORIS_CHECK(segment != nullptr);
+ DORIS_CHECK(ctx != nullptr);
+ std::set<int> slot_indexes;
+ for (const auto& conjunct : conjuncts) {
+ if (conjunct->root() != nullptr) {
Review Comment:
已在 cc67a753274 中处理：这里没有合理的空 root 场景，已把防御式 if 改为 `DORIS_CHECK(conjunct != nullptr)` / `DORIS_CHECK(root != nullptr)`，只断言内部不变量。
##########
be/src/storage/segment/segment.cpp:
##########
@@ -85,6 +89,87 @@ namespace doris::segment_v2 {
class InvertedIndexIterator;
+namespace {
+
+Status build_segment_zonemap_context(Segment* segment, const Schema& schema,
+ const StorageReadOptions& read_options,
+ const VExprContextSPtrs& conjuncts, ZoneMapEvalContext* ctx) {
+ DORIS_CHECK(segment != nullptr);
+ DORIS_CHECK(ctx != nullptr);
+ std::set<int> slot_indexes;
+ for (const auto& conjunct : conjuncts) {
+ if (conjunct->root() != nullptr) {
+ conjunct->root()->collect_slot_column_ids(slot_indexes);
+ }
+ }
+ for (const int slot_index : slot_indexes) {
+ if (slot_index < 0 || cast_set<size_t>(slot_index) >= schema.num_column_ids()) {
+ continue;
+ }
+ const auto column_id = schema.column_id(cast_set<size_t>(slot_index));
+ const auto* field = schema.column(column_id);
+ if (field == nullptr) {
Review Comment:
已在 cc67a753274 中处理：slot index 已按 `schema.num_column_ids()` 做边界检查，`schema.column(column_id)` 正常不应为空；这里已改成 `DORIS_CHECK(field != nullptr)`，不再做重复的防御式 continue。
##########
be/src/storage/segment/segment.cpp:
##########
@@ -85,6 +89,87 @@ namespace doris::segment_v2 {
class InvertedIndexIterator;
+namespace {
+
+Status build_segment_zonemap_context(Segment* segment, const Schema& schema,
+ const StorageReadOptions& read_options,
+ const VExprContextSPtrs& conjuncts, ZoneMapEvalContext* ctx) {
+ DORIS_CHECK(segment != nullptr);
+ DORIS_CHECK(ctx != nullptr);
+ std::set<int> slot_indexes;
+ for (const auto& conjunct : conjuncts) {
+ if (conjunct->root() != nullptr) {
+ conjunct->root()->collect_slot_column_ids(slot_indexes);
+ }
+ }
+ for (const int slot_index : slot_indexes) {
Review Comment:
已在 cc67a753274 中处理：segment-level context 构建现在先检查 `root->can_evaluate_zonemap_filter()`，并且只收集 slot 依赖数为 1 的表达式；多 slot/unsupported expression 不再触发无意义的 segment zonemap 读取。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4650786456
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3371189702
##########
be/src/exprs/function/function_string.cpp:
##########
@@ -1341,8 +1347,118 @@ using FunctionCrc32 = FunctionUnaryToType<Crc32Impl, NameCrc32>;
using FunctionStringUTF8Length = FunctionUnaryToType<StringUtf8LengthImpl, NameStringUtf8Length>;
using FunctionStringSpace = FunctionUnaryToType<StringSpace, NameStringSpace>;
using FunctionIsValidUTF8 = FunctionUnaryToType<IsValidUTF8Impl, NameIsValidUTF8>;
-using FunctionStringStartsWith =
- FunctionBinaryToType<DataTypeString, DataTypeString, StringStartsWithImpl, NameStartsWith>;
+
+namespace {
+
+ZoneMapFilterResult unsupported_starts_with_zonemap(const ZoneMapEvalContext& ctx) {
+ ++ctx.stats.unsupported_expr_count;
+ return ZoneMapFilterResult::kUnsupported;
+}
+
+bool field_less_for_starts_with_zonemap(const Field& lhs, const Field& rhs) {
+ return (lhs <=> rhs) == std::strong_ordering::less;
+}
+
+Field string_field_for_starts_with_zonemap(std::string_view value) {
+ return Field::create_field<TYPE_STRING>(std::string(value));
Review Comment:
这块是不是可以用stringview呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4697892808
# FE Regression Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_76d48a1883c19a3e0f6b8c8d16755da1afb8b8f6_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_76d48a1883c19a3e0f6b8c8d16755da1afb8b8f6_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4697058764
# BE Regression && UT Coverage Report
Increment line coverage `69.31% (734/1059)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_8a4c4d46c63a8083b97c25e9177806c0d0e9773d_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_8a4c4d46c63a8083b97c25e9177806c0d0e9773d_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 58.18% (22316/38357) |
| Line Coverage | 41.67% (222315/533564) |
| Region Coverage | 37.95% (179401/472791) |
| Branch Coverage | 38.64% (77442/200440) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4700745494
# FE Regression Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_76d48a1883c19a3e0f6b8c8d16755da1afb8b8f6_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_76d48a1883c19a3e0f6b8c8d16755da1afb8b8f6_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3330015641
##########
be/src/storage/segment/segment.cpp:
##########
@@ -85,6 +89,87 @@ namespace doris::segment_v2 {
class InvertedIndexIterator;
+namespace {
+
+Status build_segment_zonemap_context(Segment* segment, const Schema& schema,
+ const StorageReadOptions& read_options,
+ const VExprContextSPtrs& conjuncts, ZoneMapEvalContext* ctx) {
+ DORIS_CHECK(segment != nullptr);
+ DORIS_CHECK(ctx != nullptr);
+ std::set<int> slot_indexes;
+ for (const auto& conjunct : conjuncts) {
+ if (conjunct->root() != nullptr) {
+ conjunct->root()->collect_slot_column_ids(slot_indexes);
+ }
+ }
+ for (const int slot_index : slot_indexes) {
Review Comment:
这里是不是只需要处理表达式里只有一个slot依赖的列呢？和下面判断
```
if (slot_indexes.size() != 1) {
return std::nullopt;
}```
一样呢，这会导致一些zonemap被无端读取
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3329995497
##########
be/src/storage/segment/segment.cpp:
##########
@@ -85,6 +89,87 @@ namespace doris::segment_v2 {
class InvertedIndexIterator;
+namespace {
+
+Status build_segment_zonemap_context(Segment* segment, const Schema& schema,
+ const StorageReadOptions& read_options,
+ const VExprContextSPtrs& conjuncts, ZoneMapEvalContext* ctx) {
+ DORIS_CHECK(segment != nullptr);
+ DORIS_CHECK(ctx != nullptr);
+ std::set<int> slot_indexes;
+ for (const auto& conjunct : conjuncts) {
+ if (conjunct->root() != nullptr) {
Review Comment:
什么情况下这个expr会是空指针
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4610975494
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3451631273
##########
be/src/exprs/expr_zonemap_filter.h:
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <compare>
+#include <optional>
+#include <vector>
+
+#include "common/check.h"
+#include "common/status.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/define_primitive_type.h"
+#include "core/field.h"
+#include "exprs/vexpr_fwd.h"
+#include "storage/index/zone_map/zonemap_eval_context.h"
+#include "storage/index/zone_map/zonemap_filter_result.h"
+
+namespace doris {
+class HybridSetBase;
+class RuntimeState;
+class TExprNode;
+} // namespace doris
+
+namespace doris::expr_zonemap {
+
+struct InZonemapMaterializedSet {
+ bool contains_null = false;
+ std::vector<Field> values;
+ Field min_value;
+ Field max_value;
+};
+
+struct SlotLiteral {
+ int slot_index;
+ DataTypePtr slot_type;
+ Field literal;
+ DataTypePtr literal_type;
+ bool literal_on_left;
+};
+
+std::optional<SlotLiteral> extract_slot_and_literal(const VExprSPtrs& args);
+
+TExprNode create_texpr_node_from_hybrid_set_value(const void* data, const PrimitiveType& type,
+ int precision, int scale);
+
+Status materialize_hybrid_set_for_zonemap_filter(HybridSetBase& set, const DataTypePtr& data_type,
+ InZonemapMaterializedSet* result);
+
+inline bool field_types_compatible(PrimitiveType lhs, PrimitiveType rhs) {
+ return lhs == rhs || (is_string_type(lhs) && is_string_type(rhs));
+}
+
+inline bool data_types_compatible(const DataTypePtr& lhs, const DataTypePtr& rhs) {
+ if (lhs == nullptr || rhs == nullptr) {
+ return false;
+ }
+ const auto lhs_type = remove_nullable(lhs);
+ const auto rhs_type = remove_nullable(rhs);
+ const auto lhs_primitive_type = lhs_type->get_primitive_type();
+ const auto rhs_primitive_type = rhs_type->get_primitive_type();
+ if (is_string_type(lhs_primitive_type) && is_string_type(rhs_primitive_type)) {
+ return true;
+ }
+ return lhs_type->equals(*rhs_type);
+}
+
+inline DataTypePtr fetch_compatible_slot_type(const ZoneMapEvalContext& ctx, int slot_index,
+ const DataTypePtr& expr_slot_type) {
+ const auto* slot_type = ctx.data_type(slot_index);
+ if (slot_type == nullptr || *slot_type == nullptr) {
+ return nullptr;
+ }
+ DORIS_CHECK(data_types_compatible(*slot_type, expr_slot_type));
+ return *slot_type;
+}
+
+inline bool field_less(const Field& lhs, const Field& rhs) {
+ return (lhs <=> rhs) == std::strong_ordering::less;
Review Comment:
这种定义为什么不是在field 类的定义中？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3451113345
##########
be/src/exprs/vdirect_in_predicate.h:
##########
@@ -30,8 +37,12 @@ class VDirectInPredicate final : public VExpr {
ENABLE_FACTORY_CREATOR(VDirectInPredicate);
public:
- VDirectInPredicate(const TExprNode& node, const std::shared_ptr<HybridSetBase>& filter)
- : VExpr(node), _filter(filter), _expr_name("direct_in_predicate") {}
+ VDirectInPredicate(const TExprNode& node, const std::shared_ptr<HybridSetBase>& filter,
Review Comment:
这个是干啥用的？如果这个predicate 不是那么重要，就先别实现了，感觉这个好乱
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4765462857
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3451556377
##########
be/src/exprs/expr_zonemap_filter.cpp:
##########
@@ -0,0 +1,269 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/expr_zonemap_filter.h"
+
+#include <algorithm>
+#include <set>
+
+#include "common/check.h"
+#include "common/logging.h"
+#include "core/column/column.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/string_ref.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/runtime_state.h"
+
+namespace doris::expr_zonemap {
+namespace {
+
+constexpr size_t kInZoneMapPointCheckThreshold = 64;
+
+std::optional<Field> field_from_literal_expr(const VExprSPtr& expr, DataTypePtr* literal_type) {
+ auto literal = std::dynamic_pointer_cast<VLiteral>(expr);
+ if (literal == nullptr) {
+ return std::nullopt;
+ }
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ *literal_type = literal->get_data_type();
+ return field;
+}
+
+struct SlotExpr {
+ int index;
+ DataTypePtr data_type;
+};
+
+std::optional<SlotExpr> slot_from_expr(const VExprSPtr& expr) {
+ auto slot = std::dynamic_pointer_cast<VSlotRef>(expr);
+ if (slot == nullptr) {
+ return std::nullopt;
+ }
+ return SlotExpr {.index = slot->column_id(), .data_type = slot->data_type()};
+}
+
+bool value_in_range(const Field& value, const Field& min_value, const Field& max_value) {
+ return field_greater_equal(value, min_value) && field_less_equal(value, max_value);
+}
+
+} // namespace
+
+TExprNode create_texpr_node_from_hybrid_set_value(const void* data, const PrimitiveType& type,
+ int precision, int scale) {
+ if (is_string_type(type)) {
+ const auto* value = reinterpret_cast<const StringRef*>(data);
+ auto field = Field::create_field<TYPE_STRING>(String(value->data, value->size));
+ return create_texpr_node_from(field, type, precision, scale);
+ }
+ return create_texpr_node_from(data, type, precision, scale);
+}
+
+Status materialize_hybrid_set_for_zonemap_filter(HybridSetBase& set, const DataTypePtr& data_type,
+ InZonemapMaterializedSet* result) {
+ DORIS_CHECK(result != nullptr);
+ DORIS_CHECK(data_type != nullptr);
+ const auto value_type = remove_nullable(data_type);
+ DORIS_CHECK(value_type != nullptr);
+
+ result->contains_null = set.contain_null();
+ result->values.clear();
+ result->min_value = Field();
+ result->max_value = Field();
+
+ auto* iterator = set.begin();
+ while (iterator->has_next()) {
+ const void* value = iterator->get_value();
+ if (value != nullptr) {
+ TExprNode literal_node = create_texpr_node_from_hybrid_set_value(
+ value, value_type->get_primitive_type(), value_type->get_precision(),
+ value_type->get_scale());
+ auto literal = VLiteral::create_shared(literal_node);
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ result->values.emplace_back(std::move(field));
+ }
+ iterator->next();
+ }
+
+ if (!result->values.empty()) {
+ auto minmax = std::ranges::minmax_element(result->values, expr_zonemap::field_less);
+ result->min_value = *minmax.min;
+ result->max_value = *minmax.max;
+ }
+ return Status::OK();
+}
+
+std::optional<SlotLiteral> extract_slot_and_literal(const VExprSPtrs& args) {
+ if (args.size() != 2) {
+ return std::nullopt;
+ }
+
+ if (auto slot = slot_from_expr(args[0]); slot.has_value()) {
+ DataTypePtr literal_type;
+ auto literal = field_from_literal_expr(args[1], &literal_type);
+ if (!literal.has_value()) {
+ return std::nullopt;
+ }
+ return SlotLiteral {.slot_index = slot->index,
Review Comment:
原来如果是column id 就继续column id 用下去，别再换一个名字叫index了，各种概念来回变换挺复杂的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3457994035
##########
be/src/exprs/vdirect_in_predicate.h:
##########
@@ -30,8 +37,12 @@ class VDirectInPredicate final : public VExpr {
ENABLE_FACTORY_CREATOR(VDirectInPredicate);
public:
- VDirectInPredicate(const TExprNode& node, const std::shared_ptr<HybridSetBase>& filter)
- : VExpr(node), _filter(filter), _expr_name("direct_in_predicate") {}
+ VDirectInPredicate(const TExprNode& node, const std::shared_ptr<HybridSetBase>& filter,
+ bool hybrid_set_values_match_child_type = true)
+ : VExpr(node),
Review Comment:
别加各种默认值了
再就是莫名奇妙加了一个参数也不知道是做啥用的，注释都没有
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3458132999
##########
be/src/storage/index/zone_map/zonemap_eval_context.h:
##########
@@ -0,0 +1,74 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <parallel_hashmap/phmap.h>
+
+#include <optional>
+
+#include "core/data_type/data_type.h"
+#include "storage/index/zone_map/zone_map_index.h"
+#include "storage/index/zone_map/zonemap_filter_result.h"
+
+namespace doris {
+
+class RuntimeProfile;
+
+class ZoneMapEvalStats {
+public:
+ void merge_page_eval_stats(const ZoneMapEvalStats& src);
+ void accumulate_into_profile(RuntimeProfile* profile) const;
+
+ template <typename ReaderStatistics>
+ void accumulate_to(ReaderStatistics* stats) const {
+ if (stats == nullptr) {
+ return;
+ }
+ stats->expr_zonemap_unusable_evals += unusable_zonemap_eval_count;
+ stats->in_zonemap_point_check_count += in_zonemap_point_check_count;
+ stats->in_zonemap_range_only_count += in_zonemap_range_only_count;
+ }
+
+ // Evaluations that reached expr-zonemap but could not use the current zone map/context.
+ int64_t unusable_zonemap_eval_count = 0;
+ int64_t in_zonemap_point_check_count = 0;
+ int64_t in_zonemap_range_only_count = 0;
+};
+
+class ZoneMapEvalContext {
+public:
+ struct SlotZoneMap {
+ DataTypePtr data_type;
+ std::optional<segment_v2::ZoneMap> zone_map;
+ };
+
+ const segment_v2::ZoneMap* zone_map(int slot_index) const;
Review Comment:
这个返回shared ptr吧
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3458116102
##########
be/src/storage/segment/segment_iterator.cpp:
##########
@@ -3400,6 +3419,107 @@ Status SegmentIterator::_construct_compound_expr_context() {
return Status::OK();
}
+Status SegmentIterator::_apply_expr_zonemap_to_row_ranges(const VExprContextSPtrs& conjuncts,
+ rowid_t min_rowid,
+ RowRanges* row_ranges) {
+ DORIS_CHECK(row_ranges != nullptr);
+ if (!expr_zonemap::is_expr_zonemap_filter_enabled(_opts.runtime_state) || conjuncts.empty() ||
+ row_ranges->is_empty()) {
+ return Status::OK();
+ }
+ // Page zone maps are addressed by reader-schema column ordinals. When storage expr slots need
+ // rebinding to the reader schema, VExprContext::clone() does not deep-copy the expression tree
+ // yet, so rebinding may mutate shared slot refs. Keep page-level expr-zonemap on layouts whose
+ // scan tuple slot ordinals already match the reader schema.
+ if (!storage_expr_slots_match_reader_schema(_opts)) {
Review Comment:
这个逻辑是没用的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4734460033
# FE Regression Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4486422823
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4490679383
# BE Regression && UT Coverage Report
Increment line coverage `77.76% (790/1016)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_fdb02e11fe807a3a965e4543dac6f6f3fed0539c_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_fdb02e11fe807a3a965e4543dac6f6f3fed0539c_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.90% (27312/37984) |
| Line Coverage | 55.28% (291191/526769) |
| Region Coverage | 52.38% (242822/463573) |
| Branch Coverage | 53.63% (104569/194980) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4496517105
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4486552395
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4507122300
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4648062316
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4535863921
<details>
<summary>TPC-H: <b>Total hot run time: 32196 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3704e2d2892b9d0a7da311a4236a2371c84b889b, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17663 4035 4031 4031
q2 q3 10762 1397 994 994
q4 4717 472 353 353
q5 8044 2323 2127 2127
q6 388 183 142 142
q7 999 781 641 641
q8 9782 1768 1537 1537
q9 7146 4998 4958 4958
q10 6456 2269 1874 1874
q11 496 357 336 336
q12 774 568 446 446
q13 18132 3504 2764 2764
q14 268 261 239 239
q15 q16 811 772 711 711
q17 1018 1048 1027 1027
q18 7360 5898 5530 5530
q19 1201 1262 1116 1116
q20 803 701 566 566
q21 5836 2734 2492 2492
q22 444 373 312 312
Total cold run time: 103100 ms
Total hot run time: 32196 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4417 4337 4319 4319
q2 q3 4569 4990 4346 4346
q4 2149 2243 1428 1428
q5 4519 4823 5136 4823
q6 254 186 142 142
q7 1986 1850 1616 1616
q8 2543 2229 2170 2170
q9 8152 8142 8241 8142
q10 4813 4770 4339 4339
q11 611 453 399 399
q12 768 821 591 591
q13 3302 3665 3002 3002
q14 304 309 274 274
q15 q16 732 739 646 646
q17 1440 1389 1378 1378
q18 8005 7296 6809 6809
q19 1148 1128 1124 1124
q20 2245 2248 1984 1984
q21 5456 4742 4611 4611
q22 535 471 425 425
Total cold run time: 57948 ms
Total hot run time: 52568 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3401763985
##########
be/src/exprs/expr_zonemap_filter.cpp:
##########
@@ -0,0 +1,249 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/expr_zonemap_filter.h"
+
+#include <algorithm>
+#include <set>
+
+#include "common/check.h"
+#include "common/logging.h"
+#include "core/column/column.h"
+#include "core/data_type/data_type_nullable.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vdirect_in_predicate.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/runtime_state.h"
+
+namespace doris::expr_zonemap {
+namespace {
+
+constexpr size_t kInZoneMapPointCheckThreshold = 64;
+
+VExprSPtr unwrap_identity_cast(const VExprSPtr& expr) {
Review Comment:
什么情况会生成这种unwarp的cast呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3402020245
##########
be/src/exprs/expr_zonemap_filter.cpp:
##########
@@ -0,0 +1,249 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/expr_zonemap_filter.h"
+
+#include <algorithm>
+#include <set>
+
+#include "common/check.h"
+#include "common/logging.h"
+#include "core/column/column.h"
+#include "core/data_type/data_type_nullable.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vdirect_in_predicate.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/runtime_state.h"
+
+namespace doris::expr_zonemap {
+namespace {
+
+constexpr size_t kInZoneMapPointCheckThreshold = 64;
+
+VExprSPtr unwrap_identity_cast(const VExprSPtr& expr) {
+ if (expr->node_type() != TExprNodeType::CAST_EXPR || expr->get_num_children() != 1) {
+ return expr;
+ }
+ auto child = expr->get_child(0);
+ if (!remove_nullable(expr->data_type())->equals(*remove_nullable(child->data_type()))) {
+ return expr;
+ }
+ return unwrap_identity_cast(child);
+}
+
+std::optional<Field> field_from_literal_expr(const VExprSPtr& expr, DataTypePtr* literal_type) {
+ auto unwrapped = unwrap_identity_cast(expr);
+ auto literal = std::dynamic_pointer_cast<VLiteral>(unwrapped);
+ if (literal == nullptr) {
+ return std::nullopt;
+ }
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ *literal_type = literal->get_data_type();
+ return field;
+}
+
+struct SlotExpr {
+ int index;
+ DataTypePtr data_type;
+};
+
+std::optional<SlotExpr> slot_from_expr(const VExprSPtr& expr) {
+ auto unwrapped = unwrap_identity_cast(expr);
+ auto slot = std::dynamic_pointer_cast<VSlotRef>(unwrapped);
+ if (slot == nullptr) {
+ return std::nullopt;
+ }
+ return SlotExpr {.index = slot->column_id(), .data_type = slot->data_type()};
+}
+
+bool value_in_range(const Field& value, const Field& min_value, const Field& max_value) {
+ return field_greater_equal(value, min_value) && field_less_equal(value, max_value);
+}
+
+} // namespace
+
+std::optional<SlotLiteral> extract_slot_and_literal(const VExprSPtrs& args) {
+ if (args.size() != 2) {
+ return std::nullopt;
+ }
+
+ if (auto slot = slot_from_expr(args[0]); slot.has_value()) {
+ DataTypePtr literal_type;
+ auto literal = field_from_literal_expr(args[1], &literal_type);
+ if (!literal.has_value()) {
+ return std::nullopt;
+ }
+ return SlotLiteral {.slot_index = slot->index,
+ .slot_type = std::move(slot->data_type),
+ .literal = std::move(*literal),
+ .literal_type = std::move(literal_type),
+ .literal_on_left = false};
+ }
+
+ if (auto slot = slot_from_expr(args[1]); slot.has_value()) {
+ DataTypePtr literal_type;
+ auto literal = field_from_literal_expr(args[0], &literal_type);
+ if (!literal.has_value()) {
+ return std::nullopt;
+ }
+ return SlotLiteral {.slot_index = slot->index,
+ .slot_type = std::move(slot->data_type),
+ .literal = std::move(*literal),
+ .literal_type = std::move(literal_type),
+ .literal_on_left = true};
+ }
+
+ return std::nullopt;
+}
+
+bool range_stats_usable_for_zonemap(const segment_v2::ZoneMap& zone_map,
+ const DataTypePtr& data_type) {
+ if (zone_map.pass_all || zone_map.has_nan || zone_map.has_positive_inf ||
+ zone_map.has_negative_inf) {
+ return false;
+ }
+ DORIS_CHECK(data_type != nullptr);
+ auto primitive_type = remove_nullable(data_type)->get_primitive_type();
+ DORIS_CHECK(field_types_compatible(zone_map.min_value.get_type(), primitive_type));
+ DORIS_CHECK(field_types_compatible(zone_map.max_value.get_type(), primitive_type));
+ return true;
+}
+
+ZoneMapFilterResult eval_null_zonemap(const ZoneMapEvalContext& ctx, const VExprSPtrs& arguments,
+ bool is_null) {
+ DORIS_CHECK(arguments.size() == 1);
+ auto slot = slot_from_expr(arguments[0]);
+ DORIS_CHECK(slot.has_value());
+ const auto* zone_map_ref = ctx.zone_map(slot->index);
+ if (zone_map_ref == nullptr) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (is_null) {
+ return zone_map.has_null ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+ }
+ return zone_map.has_not_null ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+}
+
+bool can_eval_null_zonemap(const VExprSPtrs& arguments) {
+ if (arguments.size() != 1) {
+ return false;
+ }
+ return slot_from_expr(arguments[0]).has_value();
Review Comment:
比如 `starts_with(v, 'a') IS NULL`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3402124326
##########
be/src/exprs/runtime_filter_expr.cpp:
##########
@@ -190,4 +193,17 @@ Status RuntimeFilterExpr::execute_filter(VExprContext* context, const Block* blo
return Status::OK();
}
+ZoneMapFilterResult RuntimeFilterExpr::evaluate_zonemap_filter(
+ const ZoneMapEvalContext& ctx) const {
+ return _impl->evaluate_zonemap_filter(ctx);
+}
+
+bool RuntimeFilterExpr::can_evaluate_zonemap_filter() const {
+ return !_null_aware && _impl->can_evaluate_zonemap_filter();
Review Comment:
这里null_aware应该要支持的，发现有null的话直接就不过滤就行了，不用执行child的过滤了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3375052812
##########
be/src/storage/segment/segment_iterator.cpp:
##########
@@ -3411,6 +3447,100 @@ Status SegmentIterator::_construct_compound_expr_context() {
return Status::OK();
}
+Status SegmentIterator::_apply_expr_zonemap_to_row_ranges(const VExprContextSPtrs& conjuncts,
+ rowid_t min_rowid,
+ RowRanges* row_ranges) {
+ DORIS_CHECK(row_ranges != nullptr);
+ if (!expr_zonemap::is_expr_zonemap_filter_enabled(_opts.runtime_state) || conjuncts.empty() ||
+ row_ranges->is_empty()) {
+ return Status::OK();
+ }
+
+ std::unordered_map<int, VExprContextSPtrs> ctxs_by_slot;
+ for (const auto& conjunct : conjuncts) {
+ auto slot_index = single_slot_zonemap_index(conjunct);
+ if (slot_index.has_value() && conjunct->root()->can_evaluate_zonemap_filter()) {
Review Comment:
This page-pruning path depends on the `VSlotRef::column_id` values in `_common_expr_ctxs_push_down`, but those contexts were produced by `VExprContext::clone()` in `_construct_compound_expr_context()`, which still shares the same `_root` pointer. `rebind_storage_exprs_to_reader_schema()` then mutates that shared root via `set_column_id()`. On AGG_KEYS or non-MOW UNIQUE_KEYS scans with schema-evolution/merge readers, two segment iterators can rebind the same pushed-down expression tree to different reader schemas; while this new code is grouping by `single_slot_zonemap_index()` and later evaluating page zonemaps, another iterator can change the slot id underneath it. That can make the page-level expr zonemap read a different column's page zonemap and intersect away rows that actually match.
This is distinct from the existing whole-segment thread: the segment-level path is now gated by `storage_expr_slots_match_reader_schema()`, but the iterator-level path added here still runs after shallow-clone rebinding. Please deep-clone the expression tree before rebinding, or skip this page-level expr-zonemap path whenever rebinding is required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4709546273
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4739450435
# BE Regression && UT Coverage Report
Increment line coverage `80.97% (855/1056)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 64.07% (24601/38397) |
| Line Coverage | 47.88% (255963/534630) |
| Region Coverage | 44.56% (211078/473727) |
| Branch Coverage | 45.64% (91736/201002) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4734325636
# FE Regression Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4649252268
<details>
<summary>TPC-DS: <b>Total hot run time: 171504 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d08314d7b632e04fd42e69ee6cfc095115445526, data reload: false
query5 4354 652 492 492
query6 459 227 216 216
query7 4949 579 348 348
query8 371 228 215 215
query9 8769 4009 4002 4002
query10 469 354 322 322
query11 5922 2349 2208 2208
query12 164 112 101 101
query13 1267 602 444 444
query14 6605 5476 5096 5096
query14_1 4460 4384 4387 4384
query15 203 199 179 179
query16 1015 468 449 449
query17 1107 695 558 558
query18 2719 482 339 339
query19 201 186 143 143
query20 114 107 108 107
query21 227 164 134 134
query22 13681 13589 13336 13336
query23 17244 16451 16152 16152
query23_1 16344 16281 16257 16257
query24 7817 1778 1294 1294
query24_1 1319 1317 1320 1317
query25 537 467 387 387
query26 1334 324 205 205
query27 2562 591 378 378
query28 4439 2062 2040 2040
query29 1101 658 511 511
query30 345 269 233 233
query31 1143 1102 1004 1004
query32 113 66 62 62
query33 534 325 274 274
query34 1181 1121 643 643
query35 764 787 697 697
query36 1400 1420 1233 1233
query37 162 115 97 97
query38 3244 3148 3119 3119
query39 945 917 902 902
query39_1 881 869 882 869
query40 267 165 151 151
query41 71 72 68 68
query42 106 109 104 104
query43 318 322 283 283
query44
query45 201 194 181 181
query46 1117 1184 721 721
query47 2432 2306 2252 2252
query48 404 409 308 308
query49 656 492 375 375
query50 1093 446 356 356
query51 4335 4329 4208 4208
query52 102 100 88 88
query53 259 290 209 209
query54 298 261 227 227
query55 84 80 73 73
query56 325 309 311 309
query57 1431 1402 1312 1312
query58 322 284 268 268
query59 1594 1662 1399 1399
query60 331 293 265 265
query61 194 177 190 177
query62 716 641 593 593
query63 241 210 210 210
query64 2650 804 627 627
query65
query66 1761 518 389 389
query67 29175 29644 29460 29460
query68
query69 428 307 270 270
query70 910 975 893 893
query71 368 323 305 305
query72 2917 2691 2428 2428
query73 869 758 441 441
query74 5106 4954 4787 4787
query75 2726 2651 2276 2276
query76 2325 1197 778 778
query77 350 376 295 295
query78 12423 12529 11893 11893
query79 1419 1025 759 759
query80 687 554 483 483
query81 478 337 288 288
query82 615 158 119 119
query83 381 329 295 295
query84
query85 940 650 535 535
query86 369 307 302 302
query87 3408 3321 3160 3160
query88 3668 2768 2747 2747
query89 455 422 370 370
query90 2022 203 197 197
query91 212 195 164 164
query92 68 64 58 58
query93 1495 1518 988 988
query94 573 352 310 310
query95 789 516 465 465
query96 1004 783 359 359
query97 2684 2706 2580 2580
query98 213 204 199 199
query99 1148 1169 1033 1033
Total cold run time: 252467 ms
Total hot run time: 171504 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4645181978
# FE UT Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_d08314d7b632e04fd42e69ee6cfc095115445526/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_d08314d7b632e04fd42e69ee6cfc095115445526/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4696814582
# BE Regression && UT Coverage Report
Increment line coverage `69.31% (734/1059)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_8a4c4d46c63a8083b97c25e9177806c0d0e9773d_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_8a4c4d46c63a8083b97c25e9177806c0d0e9773d_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 58.18% (22316/38357) |
| Line Coverage | 41.67% (222315/533564) |
| Region Coverage | 37.95% (179401/472791) |
| Branch Coverage | 38.64% (77442/200440) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4781186506
# FE Regression Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_f1cd5cca24e94354de8c70b10e47cc4c9ecf4d80_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_f1cd5cca24e94354de8c70b10e47cc4c9ecf4d80_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4784736267
# BE UT Coverage Report
Increment line coverage `76.34% (781/1023)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/f640be29bedf8dc904f15104f2d1a290b0569798_f640be29bedf8dc904f15104f2d1a290b0569798/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/f640be29bedf8dc904f15104f2d1a290b0569798_f640be29bedf8dc904f15104f2d1a290b0569798/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 54.68% (21513/39341) |
| Line Coverage | 38.19% (205512/538186) |
| Region Coverage | 34.19% (161393/472099) |
| Branch Coverage | 35.18% (70712/201016) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4787688364
# BE Regression && UT Coverage Report
Increment line coverage `92.82% (931/1003)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_501cde75be748e273233042cab95400570d9a634_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_501cde75be748e273233042cab95400570d9a634_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 72.47% (27833/38404) |
| Line Coverage | 55.79% (298459/534997) |
| Region Coverage | 52.42% (248598/474217) |
| Branch Coverage | 53.64% (107965/201263) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3451615136
##########
be/src/exprs/function/function_string.cpp:
##########
@@ -1341,8 +1349,88 @@ using FunctionCrc32 = FunctionUnaryToType<Crc32Impl, NameCrc32>;
using FunctionStringUTF8Length = FunctionUnaryToType<StringUtf8LengthImpl, NameStringUtf8Length>;
using FunctionStringSpace = FunctionUnaryToType<StringSpace, NameStringSpace>;
using FunctionIsValidUTF8 = FunctionUnaryToType<IsValidUTF8Impl, NameIsValidUTF8>;
-using FunctionStringStartsWith =
- FunctionBinaryToType<DataTypeString, DataTypeString, StringStartsWithImpl, NameStartsWith>;
+
+class FunctionStringStartsWith : public FunctionBinaryToType<DataTypeString, DataTypeString,
+ StringStartsWithImpl, NameStartsWith> {
+public:
+ static FunctionPtr create() { return std::make_shared<FunctionStringStartsWith>(); }
+
+ ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx,
Review Comment:
所有的，要使用expr 做zonemap filter的，我们都得有beut，这种代码不能指望regression test。
包括后面的comparison 这些
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4778591971
# BE Regression && UT Coverage Report
Increment line coverage `79.41% (829/1044)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_2b07745d310f37a8f54e20cefac75e3bf482f18f_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_2b07745d310f37a8f54e20cefac75e3bf482f18f_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 64.37% (24716/38398) |
| Line Coverage | 47.99% (256705/534924) |
| Region Coverage | 44.59% (211438/474166) |
| Branch Coverage | 45.74% (92041/201241) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3458134463
##########
be/src/format/parquet/vparquet_reader.cpp:
##########
@@ -1266,6 +1316,85 @@ Status ParquetReader::_process_page_index_filter(
}
RowRanges::ranges_intersection(*candidate_row_ranges, tmp_row_range, candidate_row_ranges);
}
+ bool filtered_row_group_by_expr_zonemap = false;
+ RETURN_IF_ERROR(_process_expr_zonemap_page_filter(&cached_page_index, candidate_row_ranges,
+ &filtered_row_group_by_expr_zonemap));
+ if (filtered_row_group_by_expr_zonemap) {
+ ++_reader_statistics.filtered_row_groups_by_expr_zonemap;
+ }
+ return Status::OK();
+}
+
+Status ParquetReader::_process_expr_zonemap_page_filter(
Review Comment:
这些代码都得有单测
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4789946411
# BE Regression && UT Coverage Report
Increment line coverage `93.62% (939/1003)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_501cde75be748e273233042cab95400570d9a634_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_501cde75be748e273233042cab95400570d9a634_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 74.36% (28559/38404) |
| Line Coverage | 58.22% (311467/534997) |
| Region Coverage | 54.93% (260490/474217) |
| Branch Coverage | 56.37% (113459/201263) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4829927929
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee merged PR #63389:
URL: https://github.com/apache/doris/pull/63389
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3402015668
##########
be/src/exprs/function/functions_comparison.h:
##########
@@ -20,30 +20,143 @@
#pragma once
+#include <compare>
#include <limits>
+#include <optional>
+#include <string_view>
#include <type_traits>
+#include "common/check.h"
#include "common/logging.h"
#include "core/accurate_comparison.h"
#include "core/assert_cast.h"
#include "core/column/column_const.h"
#include "core/column/column_decimal.h"
#include "core/column/column_nullable.h"
#include "core/column/column_string.h"
+#include "core/data_type/data_type_nullable.h"
#include "core/data_type/data_type_number.h"
#include "core/data_type/data_type_string.h"
#include "core/data_type/define_primitive_type.h"
#include "core/decimal_comparison.h"
#include "core/field.h"
#include "core/memcmp_small.h"
#include "core/value/vdatetime_value.h"
+#include "exprs/expr_zonemap_filter.h"
#include "exprs/function/function.h"
#include "exprs/function/function_helpers.h"
#include "exprs/function/functions_logical.h"
+#include "exprs/vexpr.h"
#include "storage/index/index_reader_helper.h"
namespace doris {
+namespace comparison_zonemap_detail {
+
+enum class Op {
+ EQ,
+ NE,
+ LT,
+ LE,
+ GT,
+ GE,
+};
+
+inline Op symmetric_op(Op op) {
+ switch (op) {
+ case Op::EQ:
+ case Op::NE:
+ return op;
+ case Op::LT:
+ return Op::GT;
+ case Op::LE:
+ return Op::GE;
+ case Op::GT:
+ return Op::LT;
+ case Op::GE:
+ return Op::LE;
+ }
+ __builtin_unreachable();
+}
+
+inline ZoneMapFilterResult evaluate(const ZoneMapEvalContext& ctx, const VExprSPtrs& arguments,
+ Op op) {
+ auto slot_literal = expr_zonemap::extract_slot_and_literal(arguments);
+
+ auto slot_type = expr_zonemap::fetch_compatible_slot_type(ctx, slot_literal->slot_index,
+ slot_literal->slot_type);
+ if (slot_type == nullptr) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ const auto* zone_map_ref = ctx.zone_map(slot_literal->slot_index);
+ if (zone_map_ref == nullptr) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (!zone_map.has_not_null) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ if (!expr_zonemap::range_stats_usable_for_zonemap(zone_map, slot_type)) {
+ return unsupported_zonemap_filter(ctx);
+ }
+
+ const auto effective_op = slot_literal->literal_on_left ? symmetric_op(op) : op;
+ const auto& literal = slot_literal->literal;
+ switch (effective_op) {
+ case Op::EQ:
+ return expr_zonemap::field_less(literal, zone_map.min_value) ||
+ expr_zonemap::field_greater(literal, zone_map.max_value)
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case Op::NE:
Review Comment:
实际上截断的时候会在最后一个字节 +1:
```cpp
template <PrimitiveType Type>
void TypedZoneMapIndexWriter<Type>::modify_index_before_flush(
struct doris::segment_v2::ZoneMap& zone_map) {
// Only varchar/string filed need modify zone map index when zone map max_value
// For varchar/string type, the zone map buffer is truncated at MAX_ZONE_MAP_INDEX_SIZE (512 bytes).
// When a string value is longer than 512 bytes, only the first 512 bytes are stored.
//
// Truncating the max value creates a correctness problem: the truncated max is now smaller than the actual max.
// This means the zone map could incorrectly skip pages that actually contain matching rows.
//
// So here we add one for the last byte if the max value is truncated, which makes the truncated max value
// slightly larger than any real string that shares the same 512-byte prefix, ensuring no false negatives —
// the zone map will never incorrectly skip a page that contains matching data.
//
// In UTF8 encoding, here do not appear 0xff in last byte
if constexpr (Type == TYPE_CHAR || Type == TYPE_VARCHAR || Type == TYPE_STRING) {
auto& str = zone_map.max_value.get<Type>();
if (str.size() == MAX_ZONE_MAP_INDEX_SIZE) {
str[str.size() - 1] += 1;
}
}
}
```
> be/src/storage/index/zone_map/zone_map_index.cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4686680435
# BE UT Coverage Report
Increment line coverage `65.18% (685/1051)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/19c67233fc78266e5f15df46446848ee09d82292_19c67233fc78266e5f15df46446848ee09d82292/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/19c67233fc78266e5f15df46446848ee09d82292_19c67233fc78266e5f15df46446848ee09d82292/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.96% (21177/39245) |
| Line Coverage | 37.72% (202257/536209) |
| Region Coverage | 33.71% (158479/470141) |
| Branch Coverage | 34.79% (69520/199839) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4680583123
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4486551633
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4495808709
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4505665990
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4497698878
<details>
<summary>TPC-H: <b>Total hot run time: 32367 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b1a1b0efd8d26a90634283b0e201f5cfc2d7ae5c, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17643 4038 4072 4038
q2 q3 10837 1485 834 834
q4 4698 495 352 352
q5 7795 2328 2147 2147
q6 390 187 148 148
q7 1002 813 653 653
q8 9414 1787 1755 1755
q9 7160 4937 4939 4937
q10 6434 2145 1830 1830
q11 496 347 338 338
q12 740 567 444 444
q13 18133 3570 2822 2822
q14 269 257 238 238
q15 q16 835 785 717 717
q17 958 974 961 961
q18 7074 5968 5688 5688
q19 1165 1408 1138 1138
q20 843 677 595 595
q21 5833 2681 2419 2419
q22 444 368 313 313
Total cold run time: 102163 ms
Total hot run time: 32367 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4319 4214 4156 4156
q2 q3 4625 4999 4410 4410
q4 2271 2394 1539 1539
q5 4506 4703 5060 4703
q6 279 206 148 148
q7 2058 1870 1804 1804
q8 2515 2307 2223 2223
q9 7982 7786 7980 7786
q10 4712 4596 4065 4065
q11 633 446 426 426
q12 757 741 546 546
q13 3429 3789 3078 3078
q14 318 299 283 283
q15 q16 738 772 682 682
q17 1392 1379 1349 1349
q18 8339 7405 6992 6992
q19 1151 1100 1134 1100
q20 2288 2250 1975 1975
q21 5526 4938 4792 4792
q22 541 498 438 438
Total cold run time: 58379 ms
Total hot run time: 52495 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4497023252
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4533199306
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3295767924
##########
be/src/format/parquet/vparquet_reader.cpp:
##########
@@ -1256,6 +1272,85 @@ Status ParquetReader::_process_page_index_filter(
}
RowRanges::ranges_intersection(*candidate_row_ranges, tmp_row_range, candidate_row_ranges);
}
+ RETURN_IF_ERROR(_process_expr_zonemap_page_filter(&cached_page_index, candidate_row_ranges));
+ return Status::OK();
Review Comment:
This page-level expr-zonemap pruning is unreachable for expression-only predicates because `_process_page_index_filter()` returns earlier when `push_down_pred.empty()` (lines 1159-1162). A query such as `WHERE starts_with(col, 'z')` can populate `_lazy_read_ctx.conjuncts` for expr-zonemap evaluation while producing no ordinary Parquet `push_down_pred`; in that case the function calls `read_whole_row_group()` and returns before this new call. Please move the expr-zonemap page filtering so it can run whenever page indexes/min-max filtering are enabled and `_lazy_read_ctx.conjuncts` is non-empty, not only after ordinary pushdown predicates have already run.
##########
be/src/storage/segment/segment_iterator.cpp:
##########
@@ -1248,6 +1271,17 @@ Status SegmentIterator::_get_row_ranges_from_conditions(RowRanges* condition_row
_opts.stats->rows_stats_filtered += (pre_size - condition_row_ranges->count());
}
+ {
+ SCOPED_RAW_TIMER(&_opts.stats->generate_row_ranges_by_zonemap_ns);
+ if (!_common_expr_ctxs_push_down.empty()) {
+ const auto pre_expr_zonemap_size = condition_row_ranges->count();
+ RETURN_IF_ERROR(_apply_expr_zonemap_to_row_ranges(_common_expr_ctxs_push_down, 0,
+ condition_row_ranges));
Review Comment:
This page-level expr-zonemap call is currently gated by the caller's condition at lines 1000-1002, which only invokes `_get_row_ranges_from_conditions()` when there are topn filters, column predicates, or delete predicates. For the main new case, e.g. `WHERE starts_with(v, 'm')`, `_common_expr_ctxs_push_down` is non-empty but those other predicate collections can all be empty; if the segment-level zonemap does not eliminate the whole segment, page-level expr pruning is skipped entirely. Please include `_common_expr_ctxs_push_down` in the caller's condition or otherwise invoke this page-pruning block for expression-only predicates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3300226961
##########
be/src/storage/segment/segment.cpp:
##########
@@ -290,6 +344,21 @@ Status Segment::new_iterator(SchemaSPtr schema, const StorageReadOptions& read_o
}
}
Review Comment:
已处理：whole-segment expr-zonemap 现在会通过 `prepare_segment_zonemap_exprs()` 在 storage expr slot layout 与 reader schema 不一致时 clone 并调用 `rebind_storage_exprs_to_reader_schema()`，避免 AGG/UNIQUE-MOR 场景下用 scan-tuple ordinal 去读 rowset reader schema 的错误列。已补充 `ExprZonemapFilterTest.RebindStorageExprsMapsAggScanSlotToReaderSchema` 覆盖该映射。
##########
be/src/format/parquet/vparquet_reader.cpp:
##########
@@ -1256,6 +1272,85 @@ Status ParquetReader::_process_page_index_filter(
}
RowRanges::ranges_intersection(*candidate_row_ranges, tmp_row_range, candidate_row_ranges);
}
+ RETURN_IF_ERROR(_process_expr_zonemap_page_filter(&cached_page_index, candidate_row_ranges));
+ return Status::OK();
Review Comment:
已处理：Parquet `_process_page_index_filter()` 现在会单独计算 `has_expr_zonemap_filter`，当 `push_down_pred` 为空但 `_lazy_read_ctx.conjuncts` 存在时不再提前 `read_whole_row_group()` 返回，page-level expr-zonemap 仍会执行。已补充 `ParquetExprTest.test_page_index_filter_applies_expr_zonemap_without_pushdown_predicate` 覆盖 expression-only predicate。
##########
be/src/storage/segment/segment_iterator.cpp:
##########
@@ -1248,6 +1271,17 @@ Status SegmentIterator::_get_row_ranges_from_conditions(RowRanges* condition_row
_opts.stats->rows_stats_filtered += (pre_size - condition_row_ranges->count());
}
+ {
+ SCOPED_RAW_TIMER(&_opts.stats->generate_row_ranges_by_zonemap_ns);
+ if (!_common_expr_ctxs_push_down.empty()) {
+ const auto pre_expr_zonemap_size = condition_row_ranges->count();
+ RETURN_IF_ERROR(_apply_expr_zonemap_to_row_ranges(_common_expr_ctxs_push_down, 0,
+ condition_row_ranges));
Review Comment:
已处理：OLAP `SegmentIterator` 进入 `_get_row_ranges_from_conditions()` 的条件已包含 `!_common_expr_ctxs_push_down.empty()`，因此 `starts_with(v, ...)` 这类 expression-only predicate 在 segment-level 未整体裁剪时仍会进入 page-level expr-zonemap pruning。回归用例 `test_expr_zonemap_pruning` 中增加了 `ExprZoneMapFilteredPages` 断言覆盖。
##########
be/src/exprs/expr_zonemap_filter.cpp:
##########
@@ -0,0 +1,425 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/expr_zonemap_filter.h"
+
+#include <algorithm>
+#include <compare>
+#include <limits>
+#include <string>
+
+#include "core/column/column.h"
+#include "core/data_type/data_type_nullable.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vdirect_in_predicate.h"
+#include "exprs/vexpr.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+
+namespace doris::expr_zonemap {
+namespace {
+
+constexpr size_t kInZoneMapPointCheckThreshold = 64;
+
+VExprSPtr unwrap_identity_cast(const VExprSPtr& expr) {
+ if (expr->node_type() != TExprNodeType::CAST_EXPR || expr->get_num_children() != 1) {
+ return expr;
+ }
+ auto child = expr->get_child(0);
+ if (!remove_nullable(expr->data_type())->equals(*remove_nullable(child->data_type()))) {
+ return expr;
+ }
+ return unwrap_identity_cast(child);
+}
+
+std::optional<Field> field_from_literal_expr(const VExprSPtr& expr, DataTypePtr* literal_type) {
+ auto unwrapped = unwrap_identity_cast(expr);
+ auto literal = std::dynamic_pointer_cast<VLiteral>(unwrapped);
+ if (literal == nullptr) {
+ return std::nullopt;
+ }
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ *literal_type = literal->get_data_type();
+ return field;
+}
+
+std::optional<int> slot_index_from_expr(const VExprSPtr& expr) {
+ auto unwrapped = unwrap_identity_cast(expr);
+ auto slot = std::dynamic_pointer_cast<VSlotRef>(unwrapped);
+ if (slot == nullptr) {
+ return std::nullopt;
+ }
+ return slot->column_id();
+}
+
+bool field_types_compatible(PrimitiveType lhs, PrimitiveType rhs) {
+ return lhs == rhs || (is_string_type(lhs) && is_string_type(rhs));
+}
+
+bool field_less(const Field& lhs, const Field& rhs) {
+ return (lhs <=> rhs) == std::strong_ordering::less;
+}
+
+bool field_less_equal(const Field& lhs, const Field& rhs) {
+ const auto cmp = lhs <=> rhs;
+ return cmp == std::strong_ordering::less || cmp == std::strong_ordering::equal;
+}
+
+bool field_greater(const Field& lhs, const Field& rhs) {
+ return (lhs <=> rhs) == std::strong_ordering::greater;
+}
+
+bool field_greater_equal(const Field& lhs, const Field& rhs) {
+ const auto cmp = lhs <=> rhs;
+ return cmp == std::strong_ordering::greater || cmp == std::strong_ordering::equal;
+}
+
+ZoneMapFilterResult unsupported(ZoneMapEvalContext const& ctx) {
+ ++ctx.stats.unsupported_expr_count;
+ return ZoneMapFilterResult::kUnsupported;
+}
+
+ComparisonOp symmetric_op(ComparisonOp op) {
+ switch (op) {
+ case ComparisonOp::EQ:
+ case ComparisonOp::NE:
+ return op;
+ case ComparisonOp::LT:
+ return ComparisonOp::GT;
+ case ComparisonOp::LE:
+ return ComparisonOp::GE;
+ case ComparisonOp::GT:
+ return ComparisonOp::LT;
+ case ComparisonOp::GE:
+ return ComparisonOp::LE;
+ }
+ __builtin_unreachable();
+}
+
+bool value_in_range(const Field& value, const Field& min_value, const Field& max_value) {
+ return field_greater_equal(value, min_value) && field_less_equal(value, max_value);
+}
+
+std::optional<std::string> next_prefix(std::string_view prefix) {
+ // ZoneMap string bounds are compared by bytewise Field ordering. For starts_with(s, p),
+ // the safe upper bound is the next byte string after p: p <= s < next_prefix(p).
+ std::string upper(prefix);
+ for (auto i = static_cast<int64_t>(upper.size()) - 1; i >= 0; --i) {
+ auto byte = static_cast<unsigned char>(upper[i]);
+ if (byte != std::numeric_limits<unsigned char>::max()) {
+ upper[i] = static_cast<char>(byte + 1);
+ upper.resize(i + 1);
+ return upper;
+ }
+ }
+ return std::nullopt;
+}
+
+Field string_field(std::string_view value) {
+ return Field::create_field<TYPE_STRING>(std::string(value));
+}
+
+} // namespace
+
+std::optional<SlotLiteral> extract_slot_and_literal(const VExprSPtrs& args) {
+ if (args.size() != 2) {
+ return std::nullopt;
+ }
+
+ if (auto slot_index = slot_index_from_expr(args[0]); slot_index.has_value()) {
+ DataTypePtr literal_type;
+ auto literal = field_from_literal_expr(args[1], &literal_type);
+ if (!literal.has_value()) {
+ return std::nullopt;
+ }
+ return SlotLiteral {.slot_index = *slot_index,
+ .literal = std::move(*literal),
+ .literal_type = std::move(literal_type),
+ .literal_on_left = false};
+ }
+
+ if (auto slot_index = slot_index_from_expr(args[1]); slot_index.has_value()) {
+ DataTypePtr literal_type;
+ auto literal = field_from_literal_expr(args[0], &literal_type);
+ if (!literal.has_value()) {
+ return std::nullopt;
+ }
+ return SlotLiteral {.slot_index = *slot_index,
+ .literal = std::move(*literal),
+ .literal_type = std::move(literal_type),
+ .literal_on_left = true};
+ }
+
+ return std::nullopt;
+}
+
+bool check_type_compatible(const ZoneMapEvalContext& ctx, int slot_index,
+ const DataTypePtr& literal_type) {
+ const auto* slot_type = ctx.data_type(slot_index);
+ if (slot_type == nullptr || *slot_type == nullptr || literal_type == nullptr) {
+ return false;
+ }
+ if (!remove_nullable(*slot_type)->equals(*remove_nullable(literal_type))) {
+ ++ctx.stats.type_mismatch_count;
+ return false;
+ }
+ return true;
+}
+
+const segment_v2::ZoneMap* fetch_zone_map(const ZoneMapEvalContext& ctx, int slot_index) {
+ auto zone_map = ctx.zone_map(slot_index);
+ if (zone_map == nullptr) {
+ ++ctx.stats.unsupported_expr_count;
+ }
+ return zone_map;
+}
+
+bool range_stats_usable_for_zonemap(const segment_v2::ZoneMap& zone_map,
+ const DataTypePtr& data_type) {
+ if (zone_map.pass_all || zone_map.has_nan || zone_map.has_positive_inf ||
+ zone_map.has_negative_inf) {
+ return false;
+ }
+ if (data_type == nullptr) {
+ return false;
+ }
+ auto primitive_type = remove_nullable(data_type)->get_primitive_type();
+ if (primitive_type == TYPE_CHAR) {
+ return false;
+ }
+ return field_types_compatible(zone_map.min_value.get_type(), primitive_type) &&
+ field_types_compatible(zone_map.max_value.get_type(), primitive_type);
+}
+
+ZoneMapFilterResult eval_comparison_zonemap(const ZoneMapEvalContext& ctx,
+ const VExprSPtrs& arguments, ComparisonOp op) {
+ auto slot_literal = extract_slot_and_literal(arguments);
+ if (!slot_literal.has_value()) {
+ return unsupported(ctx);
+ }
+ if (slot_literal->literal.is_null()) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ if (!check_type_compatible(ctx, slot_literal->slot_index, slot_literal->literal_type)) {
+ return unsupported(ctx);
+ }
+ auto zone_map_ref = fetch_zone_map(ctx, slot_literal->slot_index);
+ if (zone_map_ref == nullptr) {
+ return ZoneMapFilterResult::kUnsupported;
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (!zone_map.has_not_null) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ const auto* slot_type = ctx.data_type(slot_literal->slot_index);
+ if (!range_stats_usable_for_zonemap(zone_map, *slot_type)) {
+ return unsupported(ctx);
+ }
+
+ const auto effective_op = slot_literal->literal_on_left ? symmetric_op(op) : op;
+ const auto& literal = slot_literal->literal;
+ switch (effective_op) {
+ case ComparisonOp::EQ:
+ return field_less(literal, zone_map.min_value) || field_greater(literal, zone_map.max_value)
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case ComparisonOp::NE:
+ return zone_map.min_value == literal && zone_map.max_value == literal
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case ComparisonOp::LT:
+ return field_greater_equal(zone_map.min_value, literal) ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case ComparisonOp::LE:
+ return field_greater(zone_map.min_value, literal) ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case ComparisonOp::GT:
+ return field_less_equal(zone_map.max_value, literal) ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case ComparisonOp::GE:
+ return field_less(zone_map.max_value, literal) ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ }
+ __builtin_unreachable();
+}
+
+ZoneMapFilterResult eval_null_zonemap(const ZoneMapEvalContext& ctx, const VExprSPtrs& arguments,
+ bool is_null) {
+ if (arguments.size() != 1) {
+ return unsupported(ctx);
+ }
+ auto slot_index = slot_index_from_expr(arguments[0]);
+ if (!slot_index.has_value()) {
+ return unsupported(ctx);
+ }
+ auto zone_map_ref = fetch_zone_map(ctx, *slot_index);
+ if (zone_map_ref == nullptr) {
+ return ZoneMapFilterResult::kUnsupported;
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (is_null) {
+ return zone_map.has_null ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+ }
+ return zone_map.has_not_null ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+}
+
+ZoneMapFilterResult eval_starts_with_zonemap(const ZoneMapEvalContext& ctx,
+ const VExprSPtrs& arguments) {
+ auto slot_literal = extract_slot_and_literal(arguments);
+ if (!slot_literal.has_value() || slot_literal->literal_on_left) {
+ return unsupported(ctx);
+ }
+ if (slot_literal->literal.is_null()) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ const auto* slot_type = ctx.data_type(slot_literal->slot_index);
+ if (slot_type == nullptr || *slot_type == nullptr || slot_literal->literal_type == nullptr) {
+ return unsupported(ctx);
+ }
+ if (!is_string_type(remove_nullable(*slot_type)->get_primitive_type()) ||
+ !is_string_type(remove_nullable(slot_literal->literal_type)->get_primitive_type())) {
+ ++ctx.stats.type_mismatch_count;
+ return unsupported(ctx);
+ }
+ auto zone_map_ref = fetch_zone_map(ctx, slot_literal->slot_index);
+ if (zone_map_ref == nullptr) {
+ return ZoneMapFilterResult::kUnsupported;
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (!zone_map.has_not_null) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ if (!range_stats_usable_for_zonemap(zone_map, *slot_type)) {
+ return unsupported(ctx);
+ }
+
+ const auto prefix = slot_literal->literal.as_string_view();
+ if (prefix.empty()) {
+ return ZoneMapFilterResult::kMayMatch;
+ }
+ auto lower = string_field(prefix);
+ if (field_less(zone_map.max_value, lower)) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ auto upper_prefix = next_prefix(prefix);
+ if (upper_prefix.has_value() && !field_less(zone_map.min_value, string_field(*upper_prefix))) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ return ZoneMapFilterResult::kMayMatch;
+}
+
+Status materialize_in_filter_for_zonemap(HybridSetBase& filter, const DataTypePtr& data_type,
+ std::vector<Field>& values, Field& min_value,
+ Field& max_value) {
+ values.clear();
+ auto* iterator = filter.begin();
+ while (iterator->has_next()) {
+ const void* value = iterator->get_value();
+ if (value != nullptr) {
+ TExprNode literal_node = create_texpr_node_from_hybrid_set_value(
+ value, remove_nullable(data_type)->get_primitive_type(),
+ remove_nullable(data_type)->get_precision(),
+ remove_nullable(data_type)->get_scale());
+ auto literal = VLiteral::create_shared(literal_node);
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ values.emplace_back(std::move(field));
+ }
+ iterator->next();
+ }
+ if (values.empty()) {
+ return Status::OK();
+ }
+ auto minmax = std::minmax_element(
+ values.begin(), values.end(),
+ [](const Field& lhs, const Field& rhs) { return field_less(lhs, rhs); });
+ min_value = *minmax.first;
+ max_value = *minmax.second;
+ return Status::OK();
+}
+
+ZoneMapFilterResult eval_in_zonemap(const ZoneMapEvalContext& ctx, const VExprSPtr& slot_expr,
+ bool is_not_in, const std::vector<Field>& values,
+ const Field& min_value, const Field& max_value) {
+ if (values.empty()) {
+ return is_not_in ? ZoneMapFilterResult::kMayMatch : ZoneMapFilterResult::kNoMatch;
+ }
+ auto slot_index = slot_index_from_expr(slot_expr);
+ if (!slot_index.has_value()) {
+ return unsupported(ctx);
+ }
+ auto* slot_type = ctx.data_type(*slot_index);
+ if (slot_type == nullptr || *slot_type == nullptr) {
+ return unsupported(ctx);
+ }
+ auto zone_map_ref = fetch_zone_map(ctx, *slot_index);
+ if (zone_map_ref == nullptr) {
+ return ZoneMapFilterResult::kUnsupported;
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (!zone_map.has_not_null) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ if (!range_stats_usable_for_zonemap(zone_map, *slot_type)) {
+ return unsupported(ctx);
+ }
+
+ if (is_not_in) {
+ if (zone_map.min_value == zone_map.max_value) {
+ const bool only_value_is_filtered =
+ std::any_of(values.begin(), values.end(),
+ [&](const Field& value) { return value == zone_map.min_value; });
+ return only_value_is_filtered ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ }
+ return ZoneMapFilterResult::kMayMatch;
+ }
+
+ if (field_less(zone_map.max_value, min_value) || field_greater(zone_map.min_value, max_value)) {
+ return ZoneMapFilterResult::kNoMatch;
Review Comment:
已处理：`eval_in_zonemap()` 在任何 `Field` range/order 比较前增加了 `in_fields_compatible_with_slot_type()` 兼容性检查；IN-list 值、min、max 与 slot/zonemap 类型不兼容时会统计 type mismatch 并保守返回 unsupported/may-match，不再触发 `Field` 异常。对应 fallback 已由 `ExprZonemapFilterTest` 覆盖。
##########
be/src/storage/segment/segment_iterator.cpp:
##########
@@ -3435,6 +3470,99 @@ Status SegmentIterator::_construct_compound_expr_context() {
return Status::OK();
}
+Status SegmentIterator::_apply_expr_zonemap_to_row_ranges(const VExprContextSPtrs& conjuncts,
+ rowid_t min_rowid,
+ RowRanges* row_ranges) {
+ DORIS_CHECK(row_ranges != nullptr);
+ if (!config::enable_expr_zonemap_filter || conjuncts.empty() || row_ranges->is_empty()) {
+ return Status::OK();
+ }
+
+ std::unordered_map<int, VExprContextSPtrs> ctxs_by_slot;
+ for (const auto& conjunct : conjuncts) {
+ auto slot_index = single_slot_zonemap_index(conjunct);
+ if (slot_index.has_value()) {
+ ctxs_by_slot[*slot_index].emplace_back(conjunct);
+ }
+ }
+ // Page zone maps are stored per column. Multi-slot expressions need page alignment across
+ // multiple column readers and are therefore left to segment-level pruning for now.
+ if (ctxs_by_slot.empty()) {
+ return Status::OK();
+ }
+
+ ColumnIteratorOptions iter_opts {
+ .use_page_cache = _opts.use_page_cache,
+ .file_reader = _file_reader.get(),
+ .stats = _opts.stats,
+ .io_ctx = _opts.io_ctx,
+ };
+ for (const auto& [slot_index, slot_conjuncts] : ctxs_by_slot) {
+ if (cast_set<size_t>(slot_index) >= _schema->num_column_ids()) {
+ continue;
+ }
+ const auto cid = _schema->column_id(cast_set<size_t>(slot_index));
+ if (!_segment->can_apply_predicate_safely(cid, *_schema,
+ _opts.target_cast_type_for_variants, _opts)) {
+ continue;
+ }
+ const auto* field = _schema->column(cid);
+ if (field == nullptr) {
+ continue;
+ }
+ std::shared_ptr<ColumnReader> reader;
+ Status st = _segment->get_column_reader(*field, &reader, _opts.stats);
+ if (st.is<ErrorCode::NOT_FOUND>()) {
+ continue;
+ }
+ RETURN_IF_ERROR(st);
+ if (reader == nullptr || !reader->has_zone_map()) {
+ continue;
+ }
+ const std::vector<ZoneMapPB>* page_zone_maps = nullptr;
+ RETURN_IF_ERROR(reader->get_page_zone_maps(iter_opts, &page_zone_maps));
Review Comment:
已处理：page-level OLAP expr-zonemap 现在在按 slot 分组前先检查 `conjunct->root()->can_evaluate_zonemap_filter()`；unsupported single-slot expressions 会直接跳过，不再调用 `get_page_zone_maps()` 读取 page zonemap，从而避免热路径无效开销。`ExprZonemapFilterTest` 已覆盖 unsupported expression capability 判断。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3339430294
##########
be/src/storage/segment/segment.cpp:
##########
@@ -85,6 +90,93 @@ namespace doris::segment_v2 {
class InvertedIndexIterator;
+namespace {
+
+Status build_segment_zonemap_context(Segment* segment, const Schema& schema,
+ const StorageReadOptions& read_options,
+ const VExprContextSPtrs& conjuncts, ZoneMapEvalContext* ctx) {
+ DORIS_CHECK(segment != nullptr);
+ DORIS_CHECK(ctx != nullptr);
+ std::set<int> slot_indexes;
+ for (const auto& conjunct : conjuncts) {
+ DORIS_CHECK(conjunct != nullptr);
+ auto root = conjunct->root();
+ DORIS_CHECK(root != nullptr);
+ if (!root->can_evaluate_zonemap_filter()) {
+ continue;
+ }
+ std::set<int> expr_slot_indexes;
+ root->collect_slot_column_ids(expr_slot_indexes);
+ if (expr_slot_indexes.size() == 1) {
+ slot_indexes.insert(*expr_slot_indexes.begin());
+ }
+ }
+ for (const int slot_index : slot_indexes) {
+ if (slot_index < 0 || cast_set<size_t>(slot_index) >= schema.num_column_ids()) {
+ continue;
+ }
+ const auto column_id = schema.column_id(cast_set<size_t>(slot_index));
+ const auto* field = schema.column(column_id);
+ DORIS_CHECK(field != nullptr);
+ if (!segment->can_apply_predicate_safely(
+ column_id, schema, read_options.target_cast_type_for_variants, read_options)) {
+ continue;
+ }
+ auto data_type = segment->get_data_type_of(*field, read_options);
+ if (data_type == nullptr) {
+ continue;
+ }
+ ZoneMapEvalContext::SlotZoneMap slot_zone_map;
+ slot_zone_map.data_type = data_type;
+ std::shared_ptr<ColumnReader> reader;
+ Status st = segment->get_column_reader(*field, &reader, read_options.stats);
+ if (st.is<ErrorCode::NOT_FOUND>()) {
+ ctx->slots.emplace(slot_index, std::move(slot_zone_map));
+ continue;
+ }
+ RETURN_IF_ERROR(st);
+ if (reader != nullptr && reader->has_zone_map()) {
+ ZoneMap zone_map;
+ RETURN_IF_ERROR(reader->get_segment_zone_map(&zone_map));
+ slot_zone_map.zone_map = std::move(zone_map);
+ }
+ ctx->slots.emplace(slot_index, std::move(slot_zone_map));
+ }
+ return Status::OK();
+}
+
+bool storage_expr_slots_match_reader_schema(const StorageReadOptions& read_options) {
+ DORIS_CHECK(read_options.tablet_schema != nullptr);
+ const auto keys_type = read_options.tablet_schema->keys_type();
+ return keys_type == KeysType::DUP_KEYS ||
+ (keys_type == KeysType::UNIQUE_KEYS && read_options.enable_unique_key_merge_on_write);
+}
+
+Status prepare_segment_zonemap_exprs(const StorageReadOptions& read_options, const Schema& schema,
+ const VExprContextSPtrs** expr_ctxs,
+ VExprContextSPtrs* rebound_expr_ctxs) {
+ DORIS_CHECK(expr_ctxs != nullptr);
+ DORIS_CHECK(rebound_expr_ctxs != nullptr);
+ *expr_ctxs = &read_options.common_expr_ctxs_push_down;
+ if (storage_expr_slots_match_reader_schema(read_options)) {
+ return Status::OK();
+ }
+
+ rebound_expr_ctxs->reserve(read_options.common_expr_ctxs_push_down.size());
+ for (const auto& expr_ctx : read_options.common_expr_ctxs_push_down) {
+ VExprContextSPtr cloned_ctx;
+ RETURN_IF_ERROR(expr_ctx->clone(read_options.runtime_state, cloned_ctx));
+ rebound_expr_ctxs->emplace_back(std::move(cloned_ctx));
Review Comment:
This does not create an isolated expression tree before the rebinding below. `VExprContext::clone()` constructs the new context with the same `_root` pointer (`vexpr_context.cpp:153`), so `rebind_storage_exprs_to_reader_schema()` mutates the original `read_options.common_expr_ctxs_push_down` tree shared by other segments/scanners. On AGG_KEYS or non-MOW UNIQUE_KEYS scans, concurrent segment iterators can rebind the same `VSlotRef` column ids while this segment-level zonemap evaluation is running, or leave the shared tree rebound for a different reader schema before later evaluation. This is distinct from the earlier slot-rebinding thread: rebinding is now attempted, but the cloned context is shallow. Please deep-clone the expression tree before rebinding, or skip this segment-level path whenever rebinding would be required.
##########
be/src/format/parquet/vparquet_reader.cpp:
##########
@@ -1139,8 +1153,10 @@ Status ParquetReader::_process_page_index_filter(
// from https://github.com/apache/doris/pull/55795
RETURN_IF_ERROR(parse_offset_index());
- // Check if page index is needed for min-max filter.
- if (!_enable_filter_by_min_max || push_down_pred.empty()) {
+ // Check if page index is needed for min-max or expr-zonemap filter.
+ const bool has_expr_zonemap_filter = expr_zonemap::is_expr_zonemap_filter_enabled(_state) &&
+ !_lazy_read_ctx.conjuncts.empty();
Review Comment:
This gate is true for any conjunct, but `_process_expr_zonemap_page_filter()` later only keeps single-slot conjuncts whose root `can_evaluate_zonemap_filter()`. For an expression-only Parquet scan with an unsupported predicate such as `abs(k) = 1` and no ordinary `push_down_pred`, this still reads and parses the column-index buffer for every row group, then `ctxs_by_column` is empty and all pages are kept. Since `enable_expr_zonemap_filter` defaults to true, this is a hot-path regression. Please compute this flag from the same supported single-slot capability check used by the page filter before deciding to read the page index.
##########
be/src/format/parquet/vparquet_reader.cpp:
##########
@@ -1256,6 +1272,85 @@ Status ParquetReader::_process_page_index_filter(
}
RowRanges::ranges_intersection(*candidate_row_ranges, tmp_row_range, candidate_row_ranges);
}
+ RETURN_IF_ERROR(_process_expr_zonemap_page_filter(&cached_page_index, candidate_row_ranges));
+ return Status::OK();
+}
+
+Status ParquetReader::_process_expr_zonemap_page_filter(
+ ParquetPredicate::CachedPageIndexStat* cached_page_index, RowRanges* candidate_row_ranges) {
+ DORIS_CHECK(cached_page_index != nullptr);
+ DORIS_CHECK(candidate_row_ranges != nullptr);
+ const auto& all_conjuncts = _lazy_read_ctx.conjuncts;
+ if (!expr_zonemap::is_expr_zonemap_filter_enabled(_state) || all_conjuncts.empty() ||
+ candidate_row_ranges->is_empty()) {
+ return Status::OK();
+ }
+
+ std::unordered_map<int, VExprContextSPtrs> ctxs_by_column;
+ for (const auto& conjunct : all_conjuncts) {
+ if (conjunct == nullptr || conjunct->root() == nullptr) {
+ continue;
+ }
+ std::set<int> column_ids;
+ conjunct->root()->collect_slot_column_ids(column_ids);
+ if (column_ids.size() == 1 && conjunct->root()->can_evaluate_zonemap_filter()) {
+ ctxs_by_column[*column_ids.begin()].emplace_back(conjunct);
+ }
+ }
+
+ for (const auto& [cid, conjuncts] : ctxs_by_column) {
+ if (cid < 0 || cid >= _tuple_descriptor->slots().size()) {
+ continue;
+ }
+ auto* slot = _tuple_descriptor->slots()[cid];
+ if (!_exists_in_file(slot->col_name()) || !_type_matches(cid)) {
+ continue;
+ }
+ ParquetPredicate::PageIndexStat* stat = nullptr;
+ if (!cached_page_index->get_stat_func(&stat, cid) || stat == nullptr || !stat->available) {
+ continue;
+ }
+ RowRanges expr_ranges;
+ ZoneMapEvalStats page_stats;
+ for (int64_t page_id = 0; page_id < stat->num_of_pages; ++page_id) {
+ DORIS_CHECK(page_id < stat->ranges.size());
+ DORIS_CHECK(page_id < stat->has_null.size());
+ DORIS_CHECK(page_id < stat->is_all_null.size());
+ const auto& page_range = stat->ranges[page_id];
+ DORIS_CHECK(page_range.is_valid());
+
+ ZoneMapEvalContext ctx;
+ ZoneMapEvalContext::SlotZoneMap slot_zone_map;
+ slot_zone_map.data_type = slot->type();
+ segment_v2::ZoneMap zone_map;
+ zone_map.has_null = stat->has_null[page_id];
+ zone_map.has_not_null = !stat->is_all_null[page_id];
+ if (!stat->is_all_null[page_id]) {
+ DORIS_CHECK(page_id < stat->encoded_min_value.size());
+ DORIS_CHECK(page_id < stat->encoded_max_value.size());
+ Status status = ParquetPredicate::parse_min_max_value(
+ stat->col_schema, stat->encoded_min_value[page_id],
+ stat->encoded_max_value[page_id], *cached_page_index->ctz,
+ &zone_map.min_value, &zone_map.max_value);
+ if (status.ok()) {
+ slot_zone_map.zone_map = std::move(zone_map);
+ }
+ } else {
+ slot_zone_map.zone_map = std::move(zone_map);
+ }
+ ctx.slots.emplace(cid, std::move(slot_zone_map));
+ const auto result = VExprContext::evaluate_zonemap_filter(conjuncts, ctx);
+ page_stats.merge_page_eval_stats(ctx.stats);
+ if (result != ZoneMapFilterResult::kNoMatch) {
+ expr_ranges.add(page_range);
+ }
+ }
+ page_stats.accumulate_to(&_reader_statistics);
+ RowRanges::ranges_intersection(*candidate_row_ranges, expr_ranges, candidate_row_ranges);
+ if (candidate_row_ranges->is_empty()) {
+ return Status::OK();
Review Comment:
When expression page pruning makes `candidate_row_ranges` empty here, `_next_row_group_reader()` counts the row group as filtered, but `filtered_row_groups_by_expr_zonemap` is only incremented in the row-group-level path. A row group whose row-group stats are too wide but whose pages are all eliminated by `starts_with`/`IN` will therefore show `ParquetExprZoneMapFilteredRowGroups = 0`, underreporting this feature's effect. Please attribute the empty-range case to expr-zonemap as well, or add a separate page-level expr-zonemap row-group counter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4533129547
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3329985552
##########
be/src/exprs/function/function_string.cpp:
##########
@@ -1341,8 +1342,21 @@ using FunctionCrc32 = FunctionUnaryToType<Crc32Impl, NameCrc32>;
using FunctionStringUTF8Length = FunctionUnaryToType<StringUtf8LengthImpl, NameStringUtf8Length>;
using FunctionStringSpace = FunctionUnaryToType<StringSpace, NameStringSpace>;
using FunctionIsValidUTF8 = FunctionUnaryToType<IsValidUTF8Impl, NameIsValidUTF8>;
-using FunctionStringStartsWith =
- FunctionBinaryToType<DataTypeString, DataTypeString, StringStartsWithImpl, NameStartsWith>;
+class FunctionStringStartsWith : public FunctionBinaryToType<DataTypeString, DataTypeString,
+ StringStartsWithImpl, NameStartsWith> {
+public:
+ static FunctionPtr create() { return std::make_shared<FunctionStringStartsWith>(); }
+ bool is_deterministic() const override { return true; }
+ ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx,
+ const VExprSPtrs& arguments) const override {
+ return expr_zonemap::eval_starts_with_zonemap(ctx, arguments);
Review Comment:
你这里为什么要跳转到zone map的命名空间上统一处理呢？这样我觉得增加了代码维护的複雜度
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3329967785
##########
be/src/exec/runtime_filter/runtime_filter_consumer_helper.cpp:
##########
@@ -17,10 +17,14 @@
#include "exec/runtime_filter/runtime_filter_consumer_helper.h"
+#include "common/logging.h"
Review Comment:
这个cpp为什么加了一堆头文件没有实际修改代码？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3329998260
##########
be/src/storage/segment/segment.cpp:
##########
@@ -85,6 +89,87 @@ namespace doris::segment_v2 {
class InvertedIndexIterator;
+namespace {
+
+Status build_segment_zonemap_context(Segment* segment, const Schema& schema,
+ const StorageReadOptions& read_options,
+ const VExprContextSPtrs& conjuncts, ZoneMapEvalContext* ctx) {
+ DORIS_CHECK(segment != nullptr);
+ DORIS_CHECK(ctx != nullptr);
+ std::set<int> slot_indexes;
+ for (const auto& conjunct : conjuncts) {
+ if (conjunct->root() != nullptr) {
+ conjunct->root()->collect_slot_column_ids(slot_indexes);
+ }
+ }
+ for (const int slot_index : slot_indexes) {
+ if (slot_index < 0 || cast_set<size_t>(slot_index) >= schema.num_column_ids()) {
+ continue;
+ }
+ const auto column_id = schema.column_id(cast_set<size_t>(slot_index));
+ const auto* field = schema.column(column_id);
+ if (field == nullptr) {
Review Comment:
106行做了判断了，为什么这里还要判断空指针
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3394238296
##########
be/src/storage/index/zone_map/zonemap_eval_context.h:
##########
@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <parallel_hashmap/phmap.h>
+
+#include <optional>
+
+#include "core/data_type/data_type.h"
+#include "storage/index/zone_map/zone_map_index.h"
+#include "storage/index/zone_map/zonemap_filter_result.h"
+
+namespace doris {
+
+class RuntimeProfile;
+
+class ZoneMapEvalStats {
+public:
+ void merge_page_eval_stats(const ZoneMapEvalStats& src);
+ void accumulate_into_profile(RuntimeProfile* profile) const;
+
+ template <typename ReaderStatistics>
+ void accumulate_to(ReaderStatistics* stats) const {
+ if (stats == nullptr) {
+ return;
+ }
+ stats->expr_zonemap_unsupported_exprs += unsupported_expr_count;
+ stats->in_zonemap_point_check_count += in_zonemap_point_check_count;
+ stats->in_zonemap_range_only_count += in_zonemap_range_only_count;
+ }
+
+ int64_t unsupported_expr_count = 0;
+ int64_t in_zonemap_point_check_count = 0;
+ int64_t in_zonemap_range_only_count = 0;
+};
+
+class ZoneMapEvalContext {
+public:
+ struct SlotZoneMap {
+ DataTypePtr data_type;
+ std::optional<segment_v2::ZoneMap> zone_map;
+ };
+
+ const segment_v2::ZoneMap* zone_map(int slot_index) const;
+ const DataTypePtr* data_type(int slot_index) const;
+ void accumulate_into_profile(RuntimeProfile* profile) const;
+
+ phmap::flat_hash_map<int, SlotZoneMap> slots;
+
+ mutable ZoneMapEvalStats stats;
+};
+
+inline void record_unsupported_zonemap_filter(const ZoneMapEvalContext& ctx) {
+ ++ctx.stats.unsupported_expr_count;
Review Comment:
unsupported_expr_count 这个指导底统计的是什么东西啊？你这不应该是失效的这个zonemap数目吗?为什么搞一个这个名字还要额外搞个函数，拢共就一个加法
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4696841935
# FE Regression Coverage Report
Increment line coverage ` 40.00% (2/5)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_8a4c4d46c63a8083b97c25e9177806c0d0e9773d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_8a4c4d46c63a8083b97c25e9177806c0d0e9773d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4738096357
# BE Regression && UT Coverage Report
Increment line coverage `80.97% (855/1056)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 64.14% (24626/38397) |
| Line Coverage | 47.91% (256132/534630) |
| Region Coverage | 44.56% (211111/473727) |
| Branch Coverage | 45.68% (91818/201002) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4733792197
<details>
<summary>ClickBench: <b>Total hot run time: 25.97 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b264ceb03f62a92109e263f696cc7aa99d35167b, data reload: false
query1 0.01 0.01 0.01
query2 0.15 0.09 0.08
query3 0.40 0.25 0.24
query4 1.61 0.24 0.24
query5 0.37 0.32 0.32
query6 1.16 0.68 0.66
query7 0.03 0.01 0.00
query8 0.10 0.08 0.08
query9 0.52 0.38 0.37
query10 0.58 0.59 0.59
query11 0.32 0.18 0.18
query12 0.32 0.19 0.19
query13 0.52 0.52 0.55
query14 0.94 0.93 0.93
query15 0.67 0.60 0.60
query16 0.40 0.40 0.39
query17 1.02 1.04 1.03
query18 0.31 0.30 0.29
query19 1.89 1.81 1.88
query20 0.02 0.02 0.01
query21 15.46 0.38 0.31
query22 4.86 0.14 0.14
query23 15.88 0.50 0.31
query24 2.39 0.61 0.45
query25 0.16 0.12 0.10
query26 0.75 0.28 0.23
query27 0.10 0.10 0.09
query28 3.42 0.91 0.53
query29 12.51 4.53 3.54
query30 0.37 0.26 0.26
query31 2.77 0.67 0.34
query32 3.26 0.61 0.47
query33 3.02 3.01 2.97
query34 15.68 3.99 3.34
query35 3.25 3.27 3.28
query36 0.66 0.53 0.51
query37 0.12 0.09 0.09
query38 0.08 0.07 0.06
query39 0.08 0.06 0.07
query40 0.20 0.18 0.16
query41 0.12 0.08 0.08
query42 0.09 0.06 0.06
query43 0.08 0.07 0.06
Total cold run time: 96.65 s
Total hot run time: 25.97 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4734356456
# BE Regression && UT Coverage Report
Increment line coverage `69.70% (736/1056)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_b264ceb03f62a92109e263f696cc7aa99d35167b_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 58.48% (22422/38343) |
| Line Coverage | 41.89% (223550/533683) |
| Region Coverage | 38.13% (180376/473012) |
| Branch Coverage | 38.84% (77879/200529) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3451109822
##########
be/src/exprs/vdirect_in_predicate.h:
##########
@@ -49,6 +60,7 @@ class VDirectInPredicate final : public VExpr {
FunctionContext::FunctionStateScope scope) override {
DCHECK(_prepare_finished);
RETURN_IF_ERROR(VExpr::open(state, context, scope));
+ RETURN_IF_ERROR(_materialize_for_zonemap_filter());
Review Comment:
这里是干啥的？感觉materialized 好奇怪
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3453402428
##########
be/src/exprs/expr_zonemap_filter.cpp:
##########
@@ -0,0 +1,269 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/expr_zonemap_filter.h"
+
+#include <algorithm>
+#include <set>
+
+#include "common/check.h"
+#include "common/logging.h"
+#include "core/column/column.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/string_ref.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/runtime_state.h"
+
+namespace doris::expr_zonemap {
+namespace {
+
+constexpr size_t kInZoneMapPointCheckThreshold = 64;
+
+std::optional<Field> field_from_literal_expr(const VExprSPtr& expr, DataTypePtr* literal_type) {
+ auto literal = std::dynamic_pointer_cast<VLiteral>(expr);
+ if (literal == nullptr) {
+ return std::nullopt;
+ }
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ *literal_type = literal->get_data_type();
+ return field;
+}
+
+struct SlotExpr {
+ int index;
+ DataTypePtr data_type;
+};
+
+std::optional<SlotExpr> slot_from_expr(const VExprSPtr& expr) {
+ auto slot = std::dynamic_pointer_cast<VSlotRef>(expr);
+ if (slot == nullptr) {
+ return std::nullopt;
+ }
+ return SlotExpr {.index = slot->column_id(), .data_type = slot->data_type()};
+}
+
+bool value_in_range(const Field& value, const Field& min_value, const Field& max_value) {
+ return field_greater_equal(value, min_value) && field_less_equal(value, max_value);
+}
+
+} // namespace
+
+TExprNode create_texpr_node_from_hybrid_set_value(const void* data, const PrimitiveType& type,
+ int precision, int scale) {
+ if (is_string_type(type)) {
+ const auto* value = reinterpret_cast<const StringRef*>(data);
+ auto field = Field::create_field<TYPE_STRING>(String(value->data, value->size));
+ return create_texpr_node_from(field, type, precision, scale);
+ }
+ return create_texpr_node_from(data, type, precision, scale);
+}
+
+Status materialize_hybrid_set_for_zonemap_filter(HybridSetBase& set, const DataTypePtr& data_type,
+ InZonemapMaterializedSet* result) {
+ DORIS_CHECK(result != nullptr);
+ DORIS_CHECK(data_type != nullptr);
+ const auto value_type = remove_nullable(data_type);
+ DORIS_CHECK(value_type != nullptr);
+
+ result->contains_null = set.contain_null();
+ result->values.clear();
+ result->min_value = Field();
+ result->max_value = Field();
+
+ auto* iterator = set.begin();
+ while (iterator->has_next()) {
+ const void* value = iterator->get_value();
+ if (value != nullptr) {
+ TExprNode literal_node = create_texpr_node_from_hybrid_set_value(
+ value, value_type->get_primitive_type(), value_type->get_precision(),
+ value_type->get_scale());
+ auto literal = VLiteral::create_shared(literal_node);
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ result->values.emplace_back(std::move(field));
+ }
+ iterator->next();
+ }
+
+ if (!result->values.empty()) {
+ auto minmax = std::ranges::minmax_element(result->values, expr_zonemap::field_less);
+ result->min_value = *minmax.min;
+ result->max_value = *minmax.max;
+ }
+ return Status::OK();
+}
+
+std::optional<SlotLiteral> extract_slot_and_literal(const VExprSPtrs& args) {
+ if (args.size() != 2) {
+ return std::nullopt;
+ }
+
+ if (auto slot = slot_from_expr(args[0]); slot.has_value()) {
+ DataTypePtr literal_type;
+ auto literal = field_from_literal_expr(args[1], &literal_type);
+ if (!literal.has_value()) {
+ return std::nullopt;
+ }
+ return SlotLiteral {.slot_index = slot->index,
Review Comment:
这里的 slot index 对应的是 `VSlotRef::_column_id`, 这个 `_column_id` 它不是真正的column id，实际是在 block 中的 index。
拿到这个 slot index 的后续操作有调用 `ColumnId Schema::column_id(size_t index) const` 来获取实际的 column id
所以是 `VSlotRef::_column_id` 这个字段的名字不太合理。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3451540483
##########
be/src/exprs/expr_zonemap_filter.cpp:
##########
@@ -0,0 +1,269 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exprs/expr_zonemap_filter.h"
+
+#include <algorithm>
+#include <set>
+
+#include "common/check.h"
+#include "common/logging.h"
+#include "core/column/column.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/string_ref.h"
+#include "exprs/hybrid_set.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vliteral.h"
+#include "exprs/vslot_ref.h"
+#include "runtime/runtime_state.h"
+
+namespace doris::expr_zonemap {
+namespace {
+
+constexpr size_t kInZoneMapPointCheckThreshold = 64;
+
+std::optional<Field> field_from_literal_expr(const VExprSPtr& expr, DataTypePtr* literal_type) {
+ auto literal = std::dynamic_pointer_cast<VLiteral>(expr);
+ if (literal == nullptr) {
+ return std::nullopt;
+ }
+ Field field;
+ literal->get_column_ptr()->get(0, field);
+ *literal_type = literal->get_data_type();
+ return field;
+}
+
+struct SlotExpr {
Review Comment:
这个结构别要了，这个只是吧原来的VSlotRef 里部分字段拿出来了，现在多了这个结构，容易跟slot ref 混淆。
这里不如都直接用slot ref了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4766250907
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3451683289
##########
be/src/format/parquet/vparquet_reader.cpp:
##########
@@ -1065,6 +1077,43 @@ void ParquetReader::set_condition_cache_context(std::shared_ptr<ConditionCacheCo
}
}
+bool ParquetReader::_expr_zonemap_page_slot_index(const VExprContextSPtr& conjunct,
+ int* cid) const {
+ DORIS_CHECK(cid != nullptr);
+ if (conjunct == nullptr || conjunct->root() == nullptr ||
+ !conjunct->root()->can_evaluate_zonemap_filter()) {
+ return false;
+ }
+
+ std::set<int> column_ids;
+ conjunct->root()->collect_slot_column_ids(column_ids);
+ if (column_ids.size() != 1) {
+ return false;
+ }
+
+ const int slot_index = *column_ids.begin();
+ if (slot_index < 0 || static_cast<size_t>(slot_index) >= _tuple_descriptor->slots().size()) {
+ return false;
+ }
+ auto* slot = _tuple_descriptor->slots()[slot_index];
+ if (!_exists_in_file(slot->col_name()) || !_type_matches(slot_index)) {
Review Comment:
如果做过rename 操作，这个slot name 应该是新名字，那么老的parquet 文件里，还是老名字怎么办？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3456622799
##########
be/src/storage/segment/segment_iterator.h:
##########
@@ -73,6 +73,12 @@ class InvertedIndexIterator;
class RowRanges;
class IndexIterator;
+Status rebind_storage_exprs_to_reader_schema(const StorageReadOptions& opts, const Schema& schema,
+ const VExprContextSPtrs& common_exprs,
+ std::map<ColumnId, VExprContextSPtr>& virtual_exprs);
+
+bool storage_expr_slots_match_reader_schema(const StorageReadOptions& read_options);
Review Comment:
就是可能存在表达式里面的 slot id 和实际读取的 block 没有对齐的情况，参考：
https://github.com/apache/doris/pull/64413
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3456575398
##########
be/src/exprs/expr_zonemap_filter.h:
##########
@@ -0,0 +1,125 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <compare>
+#include <optional>
+#include <vector>
+
+#include "common/check.h"
+#include "common/status.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/define_primitive_type.h"
+#include "core/field.h"
+#include "exprs/vexpr_fwd.h"
+#include "storage/index/zone_map/zonemap_eval_context.h"
+#include "storage/index/zone_map/zonemap_filter_result.h"
+
+namespace doris {
+class HybridSetBase;
+class RuntimeState;
+class TExprNode;
+} // namespace doris
+
+namespace doris::expr_zonemap {
+
+struct InZonemapMaterializedSet {
+ bool contains_null = false;
+ std::vector<Field> values;
+ Field min_value;
+ Field max_value;
+};
+
+struct SlotLiteral {
+ int slot_index;
+ DataTypePtr slot_type;
+ Field literal;
+ DataTypePtr literal_type;
+ bool literal_on_left;
+};
+
+std::optional<SlotLiteral> extract_slot_and_literal(const VExprSPtrs& args);
+
+TExprNode create_texpr_node_from_hybrid_set_value(const void* data, const PrimitiveType& type,
+ int precision, int scale);
+
+Status materialize_hybrid_set_for_zonemap_filter(HybridSetBase& set, const DataTypePtr& data_type,
+ InZonemapMaterializedSet* result);
+
+inline bool field_types_compatible(PrimitiveType lhs, PrimitiveType rhs) {
+ return lhs == rhs || (is_string_type(lhs) && is_string_type(rhs));
+}
+
+inline bool data_types_compatible(const DataTypePtr& lhs, const DataTypePtr& rhs) {
+ if (lhs == nullptr || rhs == nullptr) {
+ return false;
+ }
+ const auto lhs_type = remove_nullable(lhs);
+ const auto rhs_type = remove_nullable(rhs);
+ const auto lhs_primitive_type = lhs_type->get_primitive_type();
+ const auto rhs_primitive_type = rhs_type->get_primitive_type();
+ if (is_string_type(lhs_primitive_type) && is_string_type(rhs_primitive_type)) {
+ return true;
+ }
+ return lhs_type->equals(*rhs_type);
+}
+
+inline DataTypePtr fetch_compatible_slot_type(const ZoneMapEvalContext& ctx, int slot_index,
+ const DataTypePtr& expr_slot_type) {
+ const auto* slot_type = ctx.data_type(slot_index);
+ if (slot_type == nullptr || *slot_type == nullptr) {
+ return nullptr;
+ }
+ DORIS_CHECK(data_types_compatible(*slot_type, expr_slot_type));
+ return *slot_type;
+}
+
+inline bool field_less(const Field& lhs, const Field& rhs) {
+ return (lhs <=> rhs) == std::strong_ordering::less;
Review Comment:
我倾向于：不放到 field.h，先继续留在 expr_zonemap_filter.h 更合适。
理由：
1. 当前使用范围很窄
field_less / field_greater / field_less_equal / field_greater_equal 目前只被 expr-zonemap 相关路径使用：
- expr_zonemap_filter.cpp
- functions_comparison.h
- function_string.cpp
2. field.h 已经有通用能力
Field 本身已经提供：
```cpp
std::strong_ordering operator<=>(const Field& rhs) const;
```
这几个 helper 只是对 <=> 的 zonemap 使用场景包装。放进 field.h 会扩大 core API surface，但没有明显复用收益。
3. field.h 是很基础的 core header
它被大量模块包含，最好避免加入只服务 expr-zonemap 的便捷函数，尤其是名字比较泛的 field_less，放在 doris 全局 namespace 下会显得比较突兀。
4. 语义上这些 helper 属于 zonemap pruning
它们服务的是 min/max 区间判断、literal 与 zone map 边界比较，和 range_stats_usable_for_zonemap()、fetch_compatible_slot_type() 属于同一组 evaluator helper。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3379953222
##########
be/src/exprs/function/functions_comparison.h:
##########
@@ -20,30 +20,141 @@
#pragma once
+#include <compare>
#include <limits>
+#include <optional>
+#include <string_view>
#include <type_traits>
+#include "common/check.h"
#include "common/logging.h"
#include "core/accurate_comparison.h"
#include "core/assert_cast.h"
#include "core/column/column_const.h"
#include "core/column/column_decimal.h"
#include "core/column/column_nullable.h"
#include "core/column/column_string.h"
+#include "core/data_type/data_type_nullable.h"
#include "core/data_type/data_type_number.h"
#include "core/data_type/data_type_string.h"
#include "core/data_type/define_primitive_type.h"
#include "core/decimal_comparison.h"
#include "core/field.h"
#include "core/memcmp_small.h"
#include "core/value/vdatetime_value.h"
+#include "exprs/expr_zonemap_filter.h"
#include "exprs/function/function.h"
#include "exprs/function/function_helpers.h"
#include "exprs/function/functions_logical.h"
+#include "exprs/vexpr.h"
#include "storage/index/index_reader_helper.h"
namespace doris {
+namespace comparison_zonemap_detail {
+
+enum class Op {
+ EQ,
+ NE,
+ LT,
+ LE,
+ GT,
+ GE,
+};
+
+inline Op symmetric_op(Op op) {
+ switch (op) {
+ case Op::EQ:
+ case Op::NE:
+ return op;
+ case Op::LT:
+ return Op::GT;
+ case Op::LE:
+ return Op::GE;
+ case Op::GT:
+ return Op::LT;
+ case Op::GE:
+ return Op::LE;
+ }
+ __builtin_unreachable();
+}
+
+inline ZoneMapFilterResult evaluate(const ZoneMapEvalContext& ctx, const VExprSPtrs& arguments,
+ Op op) {
+ auto slot_literal = expr_zonemap::extract_slot_and_literal(arguments);
+ DORIS_CHECK(slot_literal.has_value());
+ DORIS_CHECK(slot_literal->slot_type != nullptr);
+ DORIS_CHECK(slot_literal->literal_type != nullptr);
+ DORIS_CHECK(!slot_literal->literal.is_null());
+ DORIS_CHECK_EQ(expr_zonemap::data_types_compatible(slot_literal->slot_type,
+ slot_literal->literal_type),
+ true)
+ << "slot type: " << slot_literal->slot_type->get_name()
+ << ", literal type: " << slot_literal->literal_type->get_name();
+
+ auto slot_type = expr_zonemap::fetch_compatible_slot_type(ctx, slot_literal->slot_index,
+ slot_literal->slot_type);
+ if (slot_type == nullptr) {
+ return ZoneMapFilterResult::kUnsupported;
+ }
+ const auto* zone_map_ref = expr_zonemap::fetch_zone_map(ctx, slot_literal->slot_index);
+ if (zone_map_ref == nullptr) {
+ return ZoneMapFilterResult::kUnsupported;
+ }
+ const auto& zone_map = *zone_map_ref;
+ if (!zone_map.has_not_null) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ if (!expr_zonemap::range_stats_usable_for_zonemap(zone_map, slot_type)) {
+ return unsupported_zonemap_filter(ctx);
+ }
+
+ const auto effective_op = slot_literal->literal_on_left ? symmetric_op(op) : op;
+ const auto& literal = slot_literal->literal;
+ switch (effective_op) {
+ case Op::EQ:
+ return expr_zonemap::field_less(literal, zone_map.min_value) ||
+ expr_zonemap::field_greater(literal, zone_map.max_value)
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case Op::NE:
+ return zone_map.min_value == literal && zone_map.max_value == literal
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case Op::LT:
+ return expr_zonemap::field_greater_equal(zone_map.min_value, literal)
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case Op::LE:
+ return expr_zonemap::field_greater(zone_map.min_value, literal)
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case Op::GT:
+ return expr_zonemap::field_less_equal(zone_map.max_value, literal)
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ case Op::GE:
+ return expr_zonemap::field_less(zone_map.max_value, literal)
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+ }
+ __builtin_unreachable();
+}
+
+inline bool can_evaluate(const VExprSPtrs& arguments) {
+ auto slot_literal = expr_zonemap::extract_slot_and_literal(arguments);
+ if (!slot_literal.has_value()) {
+ return false;
+ }
+
+ // A NULL literal makes the comparison evaluate to NULL instead of a byte range
+ // predicate on the slot. This zonemap evaluator only derives bounds from non-NULL
+ // literals, and evaluate_zonemap_filter asserts that precondition with DORIS_CHECK.
+ return !slot_literal->literal.is_null();
+}
Review Comment:
`can_evaluate()` currently advertises zonemap support for any slot/literal comparison as long as the literal is non-NULL, but `evaluate()` immediately `DORIS_CHECK`s `data_types_compatible()` before it can fall back. If a pushed-down BE comparison has a materialized literal type that is not compatible with the slot type, this path is selected for expr-zonemap pruning and the scanner aborts instead of conservatively returning unsupported/may-match. This is distinct from the existing IN-list compatibility thread because this is the comparison-function capability gate. Please include the same compatibility check in `can_evaluate()` or make `evaluate()` return `kUnsupported` instead of asserting for this fallback case, and add a unit test for the mismatch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4497790206
<details>
<summary>TPC-DS: <b>Total hot run time: 170946 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b1a1b0efd8d26a90634283b0e201f5cfc2d7ae5c, data reload: false
query5 4342 657 525 525
query6 348 255 233 233
query7 4325 600 331 331
query8 343 237 223 223
query9 8850 4019 4042 4019
query10 470 377 337 337
query11 5815 2430 2192 2192
query12 185 132 152 132
query13 1301 613 384 384
query14 5969 5350 5058 5058
query14_1 4341 4316 4325 4316
query15 214 207 180 180
query16 1052 450 440 440
query17 1128 726 579 579
query18 2524 476 341 341
query19 224 217 194 194
query20 147 134 135 134
query21 233 162 146 146
query22 13663 13546 13410 13410
query23 17244 16368 16132 16132
query23_1 16163 16324 16160 16160
query24 7494 1779 1329 1329
query24_1 1320 1334 1321 1321
query25 591 506 450 450
query26 1332 376 219 219
query27 2585 604 385 385
query28 4455 2005 2004 2004
query29 1024 649 531 531
query30 328 271 228 228
query31 1124 1101 966 966
query32 90 79 75 75
query33 561 374 314 314
query34 1175 1126 646 646
query35 796 807 673 673
query36 1334 1351 1158 1158
query37 161 105 91 91
query38 3205 3149 3062 3062
query39 937 925 900 900
query39_1 890 900 879 879
query40 269 193 176 176
query41 71 70 70 70
query42 127 120 117 117
query43 335 326 289 289
query44
query45 222 213 199 199
query46 1075 1188 705 705
query47 2352 2387 2206 2206
query48 419 422 301 301
query49 649 499 406 406
query50 1033 422 331 331
query51 4379 4371 4192 4192
query52 114 117 104 104
query53 277 297 222 222
query54 331 325 287 287
query55 96 95 91 91
query56 371 387 380 380
query57 1408 1409 1348 1348
query58 344 321 308 308
query59 1568 1634 1417 1417
query60 392 354 336 336
query61 156 154 155 154
query62 671 615 562 562
query63 253 225 222 222
query64 2424 793 628 628
query65
query66 1744 515 398 398
query67 29985 29281 29884 29281
query68
query69 475 346 317 317
query70 1068 968 984 968
query71 416 350 356 350
query72 3023 2691 2469 2469
query73 847 725 425 425
query74 5088 4893 4718 4718
query75 2722 2639 2288 2288
query76 2300 1177 797 797
query77 408 410 346 346
query78 12167 12051 11662 11662
query79 1519 1052 738 738
query80 728 612 532 532
query81 473 328 290 290
query82 1439 158 134 134
query83 399 333 299 299
query84 290 150 129 129
query85 944 655 573 573
query86 389 325 317 317
query87 3504 3357 3221 3221
query88 3566 2686 2679 2679
query89 477 416 362 362
query90 1994 198 196 196
query91 210 199 167 167
query92 82 80 74 74
query93 1629 1495 954 954
query94 538 359 329 329
query95 799 598 473 473
query96 1050 764 373 373
query97 2694 2710 2560 2560
query98 243 238 237 237
query99 1162 1099 990 990
Total cold run time: 254754 ms
Total hot run time: 170946 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4496131244
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3272205384
##########
be/src/format/parquet/vparquet_reader.cpp:
##########
@@ -1443,6 +1577,76 @@ Status ParquetReader::_process_column_stat_filter(
return Status::OK();
}
+Status ParquetReader::_process_expr_zonemap_filter(const tparquet::RowGroup& row_group,
+ bool* filter_group) {
+ DORIS_CHECK(filter_group != nullptr);
+ if (!config::enable_expr_zonemap_filter || _zonemap_filter_conjuncts == nullptr ||
+ _zonemap_filter_conjuncts->empty() || !_enable_filter_by_min_max) {
+ return Status::OK();
+ }
+
+ std::set<int> column_ids;
+ for (const auto& conjunct : *_zonemap_filter_conjuncts) {
+ if (conjunct->root() != nullptr) {
+ conjunct->root()->collect_slot_column_ids(column_ids);
+ }
+ }
+ if (column_ids.empty()) {
+ return Status::OK();
+ }
+
+ ZoneMapEvalContext ctx;
+ for (const int cid : column_ids) {
+ if (cid < 0 || cid >= _tuple_descriptor->slots().size()) {
+ continue;
+ }
+ auto* slot = _tuple_descriptor->slots()[cid];
+ ZoneMapEvalContext::SlotZoneMap slot_zone_map;
+ slot_zone_map.data_type = slot->type();
+ if (!_table_info_node_ptr->children_column_exists(slot->col_name())) {
+ ctx.slots.emplace(cid, std::move(slot_zone_map));
+ continue;
+ }
+ const auto& file_col_name =
+ _table_info_node_ptr->children_file_column_name(slot->col_name());
+ const FieldSchema* col_schema = _file_metadata->schema().get_column(file_col_name);
+ int parquet_col_id = col_schema->physical_column_index;
+ const auto& meta_data = row_group.columns[parquet_col_id].meta_data;
+
Review Comment:
This row-group expr-zonemap path needs the same complex-column guard used by the page-index path above. For an expression such as `complex_col IS NULL`, `collect_slot_column_ids()` can include a complex slot, `children_column_exists()` can be true, but `FieldSchema::physical_column_index` is `-1` for complex parent fields. This then indexes `row_group.columns[-1]` and can crash instead of conservatively skipping zonemap pruning. Please check `parquet_col_id < 0` (and fall back by emplacing the slot without a zonemap or continuing) before accessing `row_group.columns[parquet_col_id]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4514037456
<details>
<summary>TPC-DS: <b>Total hot run time: 170316 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a255954e4bc2d5c1832aa8bf2208f6edc02d0bb9, data reload: false
query5 4348 668 534 534
query6 346 236 229 229
query7 4255 588 326 326
query8 331 231 234 231
query9 8842 4034 4061 4034
query10 515 370 338 338
query11 5780 2416 2236 2236
query12 190 130 128 128
query13 1296 598 384 384
query14 6076 5312 4986 4986
query14_1 4302 4291 4327 4291
query15 216 202 181 181
query16 1002 472 430 430
query17 1117 713 580 580
query18 2580 482 348 348
query19 221 198 163 163
query20 139 131 127 127
query21 232 154 132 132
query22 13814 13629 13359 13359
query23 17309 16305 16036 16036
query23_1 16151 16110 16196 16110
query24 7783 1794 1291 1291
query24_1 1295 1284 1312 1284
query25 580 507 445 445
query26 1390 363 222 222
query27 2630 575 385 385
query28 4474 1961 1969 1961
query29 1032 645 536 536
query30 336 270 230 230
query31 1135 1089 981 981
query32 92 79 76 76
query33 580 368 311 311
query34 1173 1096 631 631
query35 770 801 686 686
query36 1382 1328 1258 1258
query37 168 110 93 93
query38 3166 3127 3035 3035
query39 923 928 920 920
query39_1 875 881 865 865
query40 262 200 173 173
query41 71 71 72 71
query42 125 118 117 117
query43 325 326 290 290
query44
query45 213 203 197 197
query46 1080 1197 740 740
query47 2269 2313 2153 2153
query48 404 449 302 302
query49 657 509 408 408
query50 1063 439 329 329
query51 4310 4244 4196 4196
query52 120 117 102 102
query53 268 301 232 232
query54 359 301 300 300
query55 100 95 89 89
query56 386 397 396 396
query57 1407 1391 1299 1299
query58 366 325 322 322
query59 1550 1603 1418 1418
query60 366 368 345 345
query61 221 154 159 154
query62 671 623 536 536
query63 258 208 208 208
query64 2476 810 628 628
query65
query66 1737 510 399 399
query67 30016 29918 29852 29852
query68
query69 465 351 316 316
query70 998 949 933 933
query71 398 361 340 340
query72 3045 2754 2612 2612
query73 869 797 439 439
query74 5117 4906 4762 4762
query75 2743 2641 2311 2311
query76 2315 1147 758 758
query77 414 408 335 335
query78 11905 12088 11452 11452
query79 1496 1003 764 764
query80 1360 624 532 532
query81 553 322 274 274
query82 1176 156 126 126
query83 396 324 294 294
query84 280 193 132 132
query85 999 623 534 534
query86 456 321 322 321
query87 3472 3357 3200 3200
query88 3525 2706 2645 2645
query89 472 409 365 365
query90 1936 220 195 195
query91 211 194 167 167
query92 84 74 75 74
query93 1531 1493 870 870
query94 799 365 303 303
query95 784 604 471 471
query96 1043 804 353 353
query97 2669 2671 2586 2586
query98 239 224 222 222
query99 1130 1108 970 970
Total cold run time: 255790 ms
Total hot run time: 170316 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4509290917
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4513992786
<details>
<summary>TPC-H: <b>Total hot run time: 31268 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a255954e4bc2d5c1832aa8bf2208f6edc02d0bb9, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17822 3923 3905 3905
q2 q3 10834 1471 820 820
q4 4718 475 352 352
q5 8448 2229 2104 2104
q6 387 177 142 142
q7 942 799 645 645
q8 9676 1704 1469 1469
q9 6956 4944 4856 4856
q10 6457 2102 1831 1831
q11 500 343 319 319
q12 703 552 446 446
q13 18125 3428 2814 2814
q14 263 256 237 237
q15 q16 818 763 700 700
q17 992 878 953 878
q18 6868 5626 5435 5435
q19 1168 1271 1040 1040
q20 772 646 544 544
q21 5600 2671 2426 2426
q22 427 356 305 305
Total cold run time: 102476 ms
Total hot run time: 31268 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4228 4108 4223 4108
q2 q3 4546 4900 4334 4334
q4 2086 2199 1389 1389
q5 4373 4306 4898 4306
q6 244 193 145 145
q7 1926 1859 1548 1548
q8 2473 2074 2109 2074
q9 7784 7712 7696 7696
q10 4593 4467 4199 4199
q11 625 413 409 409
q12 732 735 523 523
q13 3262 3669 2972 2972
q14 315 312 282 282
q15 q16 728 728 639 639
q17 1345 1327 1312 1312
q18 7944 7349 6813 6813
q19 1145 1095 1090 1090
q20 2212 2218 1963 1963
q21 5281 4623 4463 4463
q22 532 475 414 414
Total cold run time: 56374 ms
Total hot run time: 50679 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3497346957
##########
be/src/storage/segment/segment.cpp:
##########
@@ -85,6 +89,63 @@ namespace doris::segment_v2 {
class InvertedIndexIterator;
+namespace {
+
+Status build_segment_zonemap_context(Segment* segment, const Schema& schema,
+ const StorageReadOptions& read_options,
+ const VExprContextSPtrs& conjuncts, ZoneMapEvalContext* ctx) {
+ DORIS_CHECK(segment != nullptr);
+ DORIS_CHECK(ctx != nullptr);
+ std::set<int> slot_indexes;
+ for (const auto& conjunct : conjuncts) {
+ DORIS_CHECK(conjunct != nullptr);
+ const auto& root = conjunct->root();
+ DORIS_CHECK(root != nullptr);
+ if (!root->can_evaluate_zonemap_filter()) {
+ continue;
+ }
+ // Segment zone maps have one min/max/null summary per column for the whole segment, so a
+ // segment-level context can safely hold every slot referenced by a compound expression.
+ // Page zone maps are page-aligned per column and still use single-slot filtering in
+ // SegmentIterator.
+ root->collect_slot_column_ids(slot_indexes);
+ }
+ for (const int slot_index : slot_indexes) {
+ if (slot_index < 0 || cast_set<size_t>(slot_index) >= schema.num_column_ids()) {
+ continue;
+ }
+ const auto column_id = schema.column_id(cast_set<size_t>(slot_index));
Review Comment:
This is a known issue, and there is already a PR fixing it: https://github.com/apache/doris/pull/64413
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4789028438
# FE Regression Coverage Report
Increment line coverage ` 1.39% (2/144)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63389_501cde75be748e273233042cab95400570d9a634_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63389_501cde75be748e273233042cab95400570d9a634_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4599698747
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4644889314
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63389:
URL: https://github.com/apache/doris/pull/63389#issuecomment-4796126228
<details>
<summary>TPC-H: <b>Total hot run time: 29730 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 501cde75be748e273233042cab95400570d9a634, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17712 4003 4018 4003
q2 2014 310 211 211
q3 10403 1447 841 841
q4 4743 470 343 343
q5 8016 877 574 574
q6 308 171 136 136
q7 841 856 639 639
q8 10719 1729 1585 1585
q9 5919 4511 4478 4478
q10 6807 1818 1531 1531
q11 512 349 320 320
q12 744 565 428 428
q13 18183 3447 2740 2740
q14 265 259 239 239
q15 q16 791 778 714 714
q17 1064 929 1169 929
q18 6919 5840 5550 5550
q19 1180 1242 1053 1053
q20 769 642 521 521
q21 5576 2694 2584 2584
q22 433 360 311 311
Total cold run time: 103918 ms
Total hot run time: 29730 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4425 4315 4338 4315
q2 314 340 214 214
q3 4635 5058 4359 4359
q4 2082 2173 1353 1353
q5 4420 4316 4351 4316
q6 232 233 239 233
q7 2275 1834 1670 1670
q8 2510 2200 2215 2200
q9 8662 8081 8170 8081
q10 4801 4770 4335 4335
q11 579 434 408 408
q12 752 750 552 552
q13 3341 3528 2956 2956
q14 288 321 281 281
q15 q16 703 710 661 661
q17 1480 1349 1355 1349
q18 7856 7299 7062 7062
q19 1130 1121 1141 1121
q20 2255 2221 1947 1947
q21 5242 4726 4498 4498
q22 537 453 411 411
Total cold run time: 58519 ms
Total hot run time: 52322 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](be) Support expression zonemap pruning [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on code in PR #63389:
URL: https://github.com/apache/doris/pull/63389#discussion_r3497252105
##########
be/src/storage/segment/segment.cpp:
##########
@@ -85,6 +89,63 @@ namespace doris::segment_v2 {
class InvertedIndexIterator;
+namespace {
+
+Status build_segment_zonemap_context(Segment* segment, const Schema& schema,
+ const StorageReadOptions& read_options,
+ const VExprContextSPtrs& conjuncts, ZoneMapEvalContext* ctx) {
+ DORIS_CHECK(segment != nullptr);
+ DORIS_CHECK(ctx != nullptr);
+ std::set<int> slot_indexes;
+ for (const auto& conjunct : conjuncts) {
+ DORIS_CHECK(conjunct != nullptr);
+ const auto& root = conjunct->root();
+ DORIS_CHECK(root != nullptr);
+ if (!root->can_evaluate_zonemap_filter()) {
+ continue;
+ }
+ // Segment zone maps have one min/max/null summary per column for the whole segment, so a
+ // segment-level context can safely hold every slot referenced by a compound expression.
+ // Page zone maps are page-aligned per column and still use single-slot filtering in
+ // SegmentIterator.
+ root->collect_slot_column_ids(slot_indexes);
+ }
+ for (const int slot_index : slot_indexes) {
+ if (slot_index < 0 || cast_set<size_t>(slot_index) >= schema.num_column_ids()) {
+ continue;
+ }
+ const auto column_id = schema.column_id(cast_set<size_t>(slot_index));
Review Comment:
At this point, segment-level expression zonemap pruning runs before the pushed-down expression is rebound to the reader schema. For non-direct AGG / normal UNIQUE-MOR scans, the reader schema may be expanded to all key columns plus requested value columns, so the expression slot ordinal may not match the ordinal in this schema. In that case, schema.column_id(slot_index) could read the zonemap of a different column, and a kNoMatch result might incorrectly skip the whole segment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org