You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "yujun777 (via GitHub)" <gi...@apache.org> on 2026/10/22 08:56:19 UTC
[PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
yujun777 opened a new pull request, #69456:
URL: https://github.com/apache/doris/pull/69456
### What problem does this PR solve?
Issue Number: N/A (tracked in https://github.com/apache/doris/issues/65418)
Related PR: #69286
Problem Summary:
This is PR 1 of the incremental MIN/MAX top-K work. It adds the four functions that the
IVM delta/apply plan needs; the FE IVM integration is a separate PR.
- `topk_min_count(x, k)` / `topk_max_count(x, k)`: an aggregate returning
`STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T NULL>` — the k smallest
(largest) distinct non-NULL values with their exact row counts, ordered by value.
`bound IS NOT NULL` is the truncation flag: it holds the k-th value when more than k
distinct values were seen, and is NULL otherwise. With `k = 0` no list is built and
`bound` is the extreme value of this side, so an append-only delta can be applied from
the insert extreme alone. `k` is a constant `BIGINT` with `0 <= k <= 65536`.
- `topk_merge_min` / `topk_merge_max`: the scalar apply step. Given the pre-image state
`(S, b)`, the insert/delete top-K lists `(I, b_i)` and `(D, b_d)`, `n`, `dn` and `k`, it
computes `bE = best(b, b_i, b_d)`, drops every value outside `bE` (the range whose counts
both sides know exactly), computes `S + I - D`, removes non-positive counts, keeps the
first k values and returns `(values, bound, n, exhausted)`. A negative count means the
state invariant is broken and returns an internal error; `exhausted` asks the caller to
fall back to a complete refresh.
The aggregates keep a sorted `(value, count)` state with checked int64 counting, so partial
aggregation and merge stay exact (`topK(A ∪ B) ⊆ topK(A) ∪ topK(B)`). FLOAT/DOUBLE order NaN
as greater than every number so the distinct list keeps a strict order.
The scalar disables the default NULL handling: a NULL list is an empty list, a NULL bound
means "not truncated", and the result struct itself is never NULL.
### Release note
Add the `topk_min_count`/`topk_max_count` aggregate functions and the
`topk_merge_min`/`topk_merge_max` scalar functions.
### Check List (For Author)
- Test: Regression test
- `regression-test/suites/query_p0/aggregate/topk_min_max_count.groovy`: k = 0/1/2/8/65536/65537, empty input, all-NULL groups, NOT NULL input types, string/decimal/datetime, -0.0/NaN, agreement with MIN/MAX, two-phase partial+final merge, every `topk_merge_*` branch (pre-image/insert/delete bounds, cancellation, truncation, exhausted, all-NULL) and the error paths.
- FE unit test `TopKFunctionSignatureTest`.
- BE unit test `agg_topk_count_test.cpp` (k = 0, truncation, MIN/MAX merge, serialize/deserialize round-trip, count overflow).
- Behavior changed: No (new functions only)
- Does this need documentation: Yes (function docs will come with the IVM user documentation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6494708538
# BE UT Coverage Report
Increment line coverage `88.37% (547/619)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/d9d17e663629a2bc41b45deb64a2e384b1f41df2_d9d17e663629a2bc41b45deb64a2e384b1f41df2/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/d9d17e663629a2bc41b45deb64a2e384b1f41df2_d9d17e663629a2bc41b45deb64a2e384b1f41df2/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 67.24% (32587/48467) |
| Line Coverage | 52.09% (345092/662502) |
| Region Coverage | 47.51% (280293/589976) |
| Branch Coverage | 49.18% (132338/269081) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6491415828
Codex automated review failed and did not complete.
Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/40835283918
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6490636281
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6498454297
# BE Regression && UT Coverage Report
Increment line coverage `96.94% (602/621)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_29fb3c47891613984533a1602cf3c9e6d15e2bb2_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_29fb3c47891613984533a1602cf3c9e6d15e2bb2_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.43% (36383/46989) |
| Line Coverage | 62.72% (411305/655822) |
| Region Coverage | 59.12% (347801/588265) |
| Branch Coverage | 60.06% (161381/268701) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6359626097
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4357993727
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArraySortedMerge.java:
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.scalar;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.ArrayItemReference;
+import org.apache.doris.nereids.trees.expressions.Cast;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.ImplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.BooleanType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.LambdaType;
+import org.apache.doris.nereids.types.coercion.FollowToAnyDataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc): merge two arrays that are already sorted.
+ *
+ * <pre>
+ * array_sorted_merge((a, b) -> ..., arr1, arr2, true)
+ * </pre>
+ *
+ * <p>The inputs must already be sorted by the elements' natural order in the direction given by
+ * {@code is_asc}; the function only merges them. The merge lambda receives the two current
+ * candidates as {@code (a, b)} and returns the combined element; returning NULL means the two
+ * elements must not be combined, so both are kept. Each key occurs at most once per input, so a
+ * key appears at most twice across the two arrays and the lambda is called at most once per key.
+ *
+ * <p>A NULL input array makes the result NULL, like the other higher-order array functions. The
+ * ascending flag must be a non-null constant; it deliberately does not take part in nullable
+ * propagation, which would fold a NULL flag into a NULL result instead of failing the analysis.
+ */
+public class ArraySortedMerge extends ScalarFunction
+ implements ImplicitlyCastableSignature {
+
+ // The bound form: the lambda plus the ascending flag; the arrays live in the lambda argument
+ // references, like the other lambda functions.
+ public static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
+ FunctionSignature.ret(new FollowToAnyDataType(0))
+ .args(LambdaType.INSTANCE, BooleanType.INSTANCE));
+
+ /**
+ * The extra arguments are variadic so that the function builder resolved with only the lambda
+ * (the higher-order lookup) can also build the bound form that keeps the ascending flag.
+ */
+ public ArraySortedMerge(Expression lambda, Expression... extra) {
+ super("array_sorted_merge",
+ ImmutableList.<Expression>builder().add(lambda).add(extra).build());
+ }
+
+ /** constructor for withChildren and reuse signature */
+ private ArraySortedMerge(ScalarFunctionParams functionParams) {
+ super(functionParams);
+ }
+
+ @Override
+ public List<FunctionSignature> getSignatures() {
+ return SIGNATURES;
+ }
+
+ @Override
+ public ArraySortedMerge withChildren(List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 2);
+ return new ArraySortedMerge(getFunctionParams(children));
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ if (!(child(0) instanceof Lambda)) {
+ throw new AnalysisException(getName() + " requires a merge lambda as the first argument");
+ }
+ Lambda lambda = (Lambda) child(0);
+ if (lambda.getLambdaArgumentNames().size() != 2) {
+ throw new AnalysisException(getName() + " requires a two-argument merge lambda, but got "
+ + lambda.getLambdaArgumentNames().size());
+ }
+ ArrayType left = requireArray(lambda, 0);
+ ArrayType right = requireArray(lambda, 1);
+ if (!left.getItemType().equals(right.getItemType())) {
+ throw new AnalysisException(getName()
+ + " requires both arrays to have the same element type, but got "
+ + left.getItemType().toSql() + " and " + right.getItemType().toSql());
+ }
+ if (!lambda.getRetType().equals(left.getItemType())) {
Review Comment:
[P2] Reconcile nested nullability when consuming TopK output
The new aggregates produce required `value`/`count` struct fields, but the ordinary count-combining lambda reconstructs nullable fields: `ArrayItemReference` is nullable, `ElementAt` is always nullable, and `named_struct` records that nullability. For example:
```text
Project(array_sorted_merge(merge_counts, s, s, true))
Aggregate(min_topk(i, 2) AS s)
Scan(i INT)
```
with `merge_counts = (a,b) -> IF(element_at(a,'value') = element_at(b,'value'), named_struct('value',element_at(a,'value'),'count',element_at(a,'count') + element_at(b,'count')), NULL)`. A single input `i=1` should yield `[{"value":1,"count":2}]`, but this equality rejects the query because `StructField.equals` includes field nullability, before any coercion can reconcile it. MAX has the same problem. The struct regression casts its input arrays to nullable-field structs and misses direct composition with the new aggregates. Normalize compatible nested nullability and the corresponding BE column layout, then add direct TopK-output merge coverage; simply dropping the equality would leave mismatched column wrappers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4352283215
##########
be/src/exprs/lambda_function/varray_sorted_merge_function.cpp:
##########
@@ -0,0 +1,320 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+#include <map>
+#include <memory>
+#include <set>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "exprs/lambda_function/lambda_execution_context.h"
+#include "exprs/lambda_function/lambda_function.h"
+#include "exprs/lambda_function/lambda_function_factory.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vlambda_function_expr.h"
+
+namespace doris {
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc)
+ *
+ * Merges two arrays that are already sorted by the elements' natural order. The merge lambda
+ * receives the two current candidates and returns the combined element; returning NULL means the
+ * two elements must not be combined, so both are kept. Each input key occurs at most once per
+ * array, so a key appears at most twice across the two arrays and the lambda is called at most
+ * once per key.
+ */
+class ArraySortedMergeFunction : public LambdaFunction {
+ ENABLE_FACTORY_CREATOR(ArraySortedMergeFunction);
+
+public:
+ ~ArraySortedMergeFunction() override = default;
+
+ static constexpr auto name = "array_sorted_merge";
+
+ static LambdaFunctionPtr create() { return std::make_shared<ArraySortedMergeFunction>(); }
+
+ std::string get_name() const override { return name; }
+
+ /// Rejects a nested lambda that captures one of this lambda's arguments: the position-based
+ /// frame below does not publish them into nested lambdas, so such a capture would resolve
+ /// against the wrong column. Nested lambdas may still use their own arguments (shadowing).
+ static Status validate_nested_captures(const VExprSPtr& expr,
+ const std::vector<std::string>& argument_names,
+ bool inside_nested_lambda,
+ std::vector<std::string> shadowed_names) {
+ if (expr->is_column_ref()) {
+ const std::string& column_name = expr->expr_name();
+ if (std::ranges::find(shadowed_names, column_name) != shadowed_names.end()) {
+ return Status::OK();
+ }
+ if (inside_nested_lambda &&
+ std::ranges::find(argument_names, column_name) != argument_names.end()) {
+ return Status::NotSupported(
+ "array_sorted_merge merge lambda argument '{}' cannot be captured by a "
+ "nested lambda",
+ column_name);
+ }
+ return Status::OK();
+ }
+ if (expr->node_type() == TExprNodeType::LAMBDA_FUNCTION_EXPR) {
+ const auto* lambda = assert_cast<const VLambdaFunctionExpr*>(expr.get());
+ if (lambda->has_argument_names()) {
+ for (const auto& argument_name : lambda->argument_names()) {
+ shadowed_names.emplace_back(argument_name);
+ }
+ }
+ return validate_nested_captures(lambda->get_child(0), argument_names, true,
+ std::move(shadowed_names));
+ }
+ for (const auto& child : expr->children()) {
+ RETURN_IF_ERROR(validate_nested_captures(child, argument_names, inside_nested_lambda,
+ shadowed_names));
+ }
+ return Status::OK();
+ }
+
+ Status prepare(RuntimeState* state, const VExprSPtrs& children) override {
+ RETURN_IF_ERROR(LambdaFunction::prepare(state, children));
+ DCHECK_EQ(children.size(), 4);
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ if (!lambda_expr->has_argument_names()) {
+ return Status::OK();
+ }
+ // A nested lambda that captures one of this lambda's arguments would need a binding
+ // that does not exist in the position-based frame below; reject it during prepare so
+ // the query fails instead of resolving the argument against the wrong column.
+ return validate_nested_captures(children[0]->get_child(0), lambda_expr->argument_names(),
+ false, {});
+ }
+
+ Status execute(VExprContext* context, const Block* block, const Selector* expr_selector,
+ size_t count, ColumnPtr& result_column, const DataTypePtr& result_type,
+ const VExprSPtrs& children) const override {
+ DCHECK_EQ(children.size(), 4);
+
+ const ColumnArray* arrays[2] = {nullptr, nullptr};
+ const IColumn* data[2] = {nullptr, nullptr};
+ DataTypePtr nested_types[2];
+ ColumnPtr holders[2];
+ const ColumnUInt8::Container* nullmaps[2] = {nullptr, nullptr};
+ for (int i = 0; i < 2; ++i) {
+ RETURN_IF_ERROR(children[i + 1]->execute_column(context, block, expr_selector, count,
+ holders[i]));
+ holders[i] = holders[i]->convert_to_full_column_if_const();
+ DataTypePtr type = children[i + 1]->execute_type(block);
+ const IColumn* array_column = holders[i].get();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(array_column)) {
+ nullmaps[i] = &nullable->get_null_map_column().get_data();
+ array_column = &nullable->get_nested_column();
+ type = assert_cast<const DataTypeNullable&>(*type).get_nested_type();
+ }
+ arrays[i] = assert_cast<const ColumnArray*>(array_column);
+ data[i] = arrays[i]->get_data_ptr().get();
+ nested_types[i] = assert_cast<const DataTypeArray&>(*type).get_nested_type();
+ }
+
+ ColumnPtr flag_holder;
+ RETURN_IF_ERROR(
+ children[3]->execute_column(context, block, expr_selector, count, flag_holder));
+ flag_holder = flag_holder->convert_to_full_column_if_const();
+ const IColumn* flag_column = flag_holder.get();
+ const ColumnUInt8::Container* flag_nullmap = nullptr;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(flag_column)) {
+ flag_nullmap = &nullable->get_null_map_column().get_data();
+ flag_column = &nullable->get_nested_column();
+ }
+ const auto& flags = assert_cast<const ColumnUInt8&>(*flag_column).get_data();
+
+ const size_t rows = arrays[0]->size();
+ auto result_nested = nested_types[0]->create_column();
+ auto result_offsets = ColumnArray::ColumnOffsets::create();
+ auto& offsets_data = result_offsets->get_data();
+ offsets_data.reserve(rows);
+ auto result_nullmap = ColumnUInt8::create(rows, 0);
+ auto& result_nullmap_data = result_nullmap->get_data();
+
+ // The lambda may capture enclosing columns (for example a table column compared against
+ // the candidates). Materialize every captured input at its original position in the
+ // one-row lambda block, so slot refs read the current row; the two merge candidates
+ // take the positions after the last captured column and are bound by name.
+ std::set<int> captured_ids;
+ children[0]->collect_slot_column_ids(captured_ids);
+ if (!captured_ids.empty() && block == nullptr) {
+ return Status::InternalError(
+ "array_sorted_merge lambda references outer columns but has no input block");
+ }
+ std::map<int, ColumnPtr> captured_columns;
+ for (int id : captured_ids) {
+ if (id < 0 || static_cast<size_t>(id) >= block->columns()) {
+ return Status::InternalError(
+ "array_sorted_merge lambda input column id {} is outside input block", id);
+ }
+ captured_columns[id] =
+ block->get_by_position(id).column->convert_to_full_column_if_const();
+ }
+ const int lambda_argument_base = captured_ids.empty() ? 0 : *captured_ids.rbegin() + 1;
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ const std::vector<std::string>* argument_names =
+ lambda_expr->has_argument_names() ? &lambda_expr->argument_names() : nullptr;
+ Block lambda_block;
+ for (int i = 0; i < lambda_argument_base; ++i) {
+ auto captured = captured_columns.find(i);
+ if (captured != captured_columns.end()) {
+ lambda_block.insert(ColumnWithTypeAndName(captured->second->clone_empty(),
+ block->get_by_position(i).type,
+ block->get_by_position(i).name));
+ } else {
+ lambda_block.insert(ColumnWithTypeAndName(
+ ColumnUInt8::create(1, 0), std::make_shared<DataTypeUInt8>(), "temp"));
+ }
+ }
+ for (int i = 0; i < 2; ++i) {
+ const std::string argument_name =
+ argument_names != nullptr && i < argument_names->size()
+ ? (*argument_names)[i]
+ : "merge_arg_" + std::to_string(i);
+ lambda_block.insert(ColumnWithTypeAndName(nested_types[i]->create_column(),
+ nested_types[i], argument_name));
+ }
+ // The lambda arguments are bound by name (the column refs carry the argument names),
+ // so the body reads these two one-row columns.
+ IColumn* argument_columns[2] = {
+ lambda_block.get_by_position(lambda_argument_base).column->assert_mutable().get(),
+ lambda_block.get_by_position(lambda_argument_base + 1)
+ .column->assert_mutable()
+ .get()};
+
+ LambdaExecutionContext::Frame lambda_frame;
+ lambda_frame.bind_by_name = false;
+ lambda_frame.parent_bindings_visible = false;
+ LambdaExecutionContext::FrameGuard lambda_frame_guard(context->lambda_execution_context(),
+ std::move(lambda_frame));
+
+ const auto& offsets0 = arrays[0]->get_offsets();
+ const auto& offsets1 = arrays[1]->get_offsets();
+ const int lambda_result_base = static_cast<int>(lambda_block.columns());
+
+ // The merged element may come from a non-nullable lambda body (for example COALESCE),
+ // while array elements are always nullable; normalize before inserting.
+ auto append_element = [&](const IColumn& source, size_t source_row) {
+ if (auto* nullable_target = check_and_get_column<ColumnNullable>(result_nested.get())) {
+ if (const auto* nullable_source = check_and_get_column<ColumnNullable>(&source)) {
+ nullable_target->insert_from(*nullable_source, source_row);
+ } else {
+ nullable_target->get_nested_column().insert_from(source, source_row);
+ nullable_target->get_null_map_data().push_back(0);
+ }
+ } else {
+ result_nested->insert_from(source, source_row);
+ }
+ };
+
+ for (size_t row = 0; row < rows; ++row) {
+ const bool row_is_null = (nullmaps[0] != nullptr && (*nullmaps[0])[row] != 0) ||
+ (nullmaps[1] != nullptr && (*nullmaps[1])[row] != 0);
+ if (row_is_null) {
+ result_nullmap_data[row] = 1;
+ offsets_data.push_back(result_nested->size());
+ continue;
+ }
+ if (flag_nullmap != nullptr && (*flag_nullmap)[row] != 0) {
+ return Status::InvalidArgument(
+ "array_sorted_merge requires a non-null ascending flag");
+ }
+ const bool ascending = flags[row] != 0;
+ for (const auto& [id, column] : captured_columns) {
+ lambda_block.get_by_position(id).column = column->cut(row, 1);
+ }
+
+ size_t left = row == 0 ? 0 : offsets0[row - 1];
+ const size_t left_end = offsets0[row];
+ size_t right = row == 0 ? 0 : offsets1[row - 1];
+ const size_t right_end = offsets1[row];
+
+ while (left < left_end || right < right_end) {
+ if (left == left_end) {
+ append_element(*data[1], right++);
+ continue;
+ }
+ if (right == right_end) {
+ append_element(*data[0], left++);
+ continue;
+ }
+
+ argument_columns[0]->clear();
+ argument_columns[0]->insert_from(*data[0], left);
+ argument_columns[1]->clear();
+ argument_columns[1]->insert_from(*data[1], right);
+
+ int lambda_res_id = lambda_result_base;
+ RETURN_IF_ERROR(children[0]->execute(context, &lambda_block, &lambda_res_id));
+ ColumnPtr merged = lambda_block.get_by_position(lambda_res_id)
+ .column->convert_to_full_column_if_const();
+ const IColumn* merged_column = merged.get();
+ bool merged_is_null = false;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(merged_column)) {
+ merged_is_null = nullable->is_null_at(0);
+ }
+ if (!merged_is_null) {
+ append_element(*merged_column, 0);
+ ++left;
+ ++right;
+ } else {
+ const int cmp = data[0]->compare_at(left, right, *data[1], 1);
Review Comment:
Fixed in 1e321f5e25c: the merge now compares candidates with `nan_direction_hint = -1`, the same hint `FunctionArraySort::Less` uses, so NULL and NaN sort least. `array_sorted_merge((a,b)->IF(a=b,a,NULL), array_sort(CAST([NULL,1] AS ARRAY<INT>)), CAST([2] AS ARRAY<INT>), true)` returns `[NULL,1,2]`, and the descending direction returns `[2,1,NULL]`; both are pinned by `order_qt_null_ordering` in `test_array_sorted_merge`.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArraySortedMerge.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.scalar;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.ArrayItemReference;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.ImplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.BooleanType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.LambdaType;
+import org.apache.doris.nereids.types.coercion.FollowToAnyDataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc): merge two arrays that are already sorted.
+ *
+ * <pre>
+ * array_sorted_merge((a, b) -> ..., arr1, arr2, true)
+ * </pre>
+ *
+ * <p>The inputs must already be sorted by the elements' natural order in the direction given by
+ * {@code is_asc}; the function only merges them. The merge lambda receives the two current
+ * candidates as {@code (a, b)} and returns the combined element; returning NULL means the two
+ * elements must not be combined, so both are kept. Each key occurs at most once per input, so a
+ * key appears at most twice across the two arrays and the lambda is called at most once per key.
+ *
+ * <p>A NULL input array makes the result NULL, like the other higher-order array functions.
+ */
+public class ArraySortedMerge extends ScalarFunction
+ implements ImplicitlyCastableSignature, PropagateNullable {
Review Comment:
Fixed in 1e321f5e25c: `CAST(NULL AS BOOLEAN)` no longer folds the call to NULL. The function no longer implements `PropagateNullable`, and `checkLegalityAfterRewrite` requires the flag to be a non-null constant, so the query fails analysis with "requires a non-null constant ascending flag" instead of returning NULL; a NULL input array still returns NULL. Both are covered in `test_array_sorted_merge`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6337431379
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4347517673
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/TopKMergeFunction.java:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.scalar;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable;
+import org.apache.doris.nereids.trees.expressions.functions.CustomSignature;
+import org.apache.doris.nereids.trees.expressions.functions.agg.TopKCountFunction;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
+
+import java.util.List;
+
+/**
+ * Shared implementation of {@code topk_merge_min} and {@code topk_merge_max}, the apply-side
+ * merge of a MIN/MAX top-K state with the insert/delete top-K lists of one delta batch:
+ *
+ * <pre>
+ * topk_merge_min(S, b, I, b_i, D, b_d, n, dn, k)
+ * -> STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T>
+ * </pre>
+ *
+ * <p>The result is the state itself and has exactly the type the aggregate returns, so apply can
+ * write it back to the hidden top-K state columns without rebuilding a struct. The caller derives
+ * {@code n' = n + dn} and {@code exhausted = (size(values) == 0 && n' > 0)} itself.
+ *
+ * <p>A NULL list is an empty list; a NULL {@code b}/{@code b_i}/{@code b_d} means that side is
+ * complete/untruncated. The result struct itself is never NULL, only its {@code bound} field can
+ * be: the merge only trusts values in {@code v <= min(b, b_i, b_d)}, so {@code bound} is the
+ * effective upper bound (for MIN; the lower bound for MAX) of the resulting state.
+ */
+public abstract class TopKMergeFunction extends ScalarFunction
+ implements CustomSignature, AlwaysNotNullable {
+
+ protected TopKMergeFunction(String name, List<Expression> children) {
+ super(name, children);
+ }
+
+ /** constructor for withChildren and reuse signature */
+ protected TopKMergeFunction(ScalarFunctionParams functionParams) {
+ super(functionParams);
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ if (arity() != 9) {
+ throw new AnalysisException(getName() + " requires 9 arguments, but got " + arity());
+ }
+ DataType valueType = inferValueType();
+ if (valueType == null) {
+ throw new AnalysisException(getName() + " can not infer the value type from arguments: "
+ + toSql());
+ }
+ if (!TopKCountFunction.isSupportedValueType(valueType)) {
+ throw new AnalysisException(getName() + " does not support type " + valueType.toSql());
+ }
+ for (int index : new int[] {0, 2, 4}) {
+ DataType type = getArgumentType(index);
+ if (type.isNullType()) {
+ continue;
+ }
+ if (!(type instanceof ArrayType)
+ || !(((ArrayType) type).getItemType() instanceof StructType)) {
+ throw new AnalysisException(getName() + " requires argument " + (index + 1)
+ + " to be ARRAY<STRUCT<value, count>>, but got " + type.toSql());
+ }
+ }
+ }
+
+ @Override
+ public FunctionSignature customSignature() {
+ DataType valueType = inferValueType();
+ if (valueType == null) {
+ throw new AnalysisException(getName() + " can not infer the value type from arguments: "
+ + toSql());
+ }
+ ArrayType listType = ArrayType.of(TopKCountFunction.pairType(valueType));
+ return FunctionSignature.ret(resultType(valueType)).args(
+ listType, valueType,
+ listType, valueType,
+ listType, valueType,
+ BigIntType.INSTANCE, BigIntType.INSTANCE, BigIntType.INSTANCE);
+ }
+
+ /**
+ * The value type is read from the first argument that carries it: the per-value type of S/I/D
+ * (argument positions 1, 3, 5) or, when all lists are untyped NULL literals, the bound type
+ * (positions 2, 4, 6).
+ */
+ private DataType inferValueType() {
+ for (int index : new int[] {0, 2, 4}) {
+ DataType type = getArgumentType(index);
+ if (!(type instanceof ArrayType)) {
+ continue;
+ }
+ DataType itemType = ((ArrayType) type).getItemType();
+ if (!(itemType instanceof StructType)) {
+ continue;
+ }
+ List<StructField> fields = ((StructType) itemType).getFields();
+ if (fields.size() != 2) {
+ continue;
+ }
+ return TopKCountFunction.normalizeValueType(fields.get(0).getDataType());
Review Comment:
[P2] Preserve distinct values when coercing the input lists
Taking T from the first typed list forces later lists through potentially lossy casts. With empty INT S/D, sorted DOUBLE I=[(1.1,1),(1.9,1)], NULL bounds, n=0, dn=2 and k=2, FE implicitly casts I to [(1,1),(1,1)]. `consume_equal` advances a side only once per head, so the scalar returns two entries for value 1 instead of one entry with the summed count. Each source list was ordered, distinct and non-NULL before this function coerced it; MAX has the symmetric case. Choose a lossless common value type across all lists/bounds, or reject incompatible input types before coercion, and add mixed-type merge coverage.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/TopKMergeFunction.java:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.scalar;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable;
+import org.apache.doris.nereids.trees.expressions.functions.CustomSignature;
+import org.apache.doris.nereids.trees.expressions.functions.agg.TopKCountFunction;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
+
+import java.util.List;
+
+/**
+ * Shared implementation of {@code topk_merge_min} and {@code topk_merge_max}, the apply-side
+ * merge of a MIN/MAX top-K state with the insert/delete top-K lists of one delta batch:
+ *
+ * <pre>
+ * topk_merge_min(S, b, I, b_i, D, b_d, n, dn, k)
+ * -> STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T>
+ * </pre>
+ *
+ * <p>The result is the state itself and has exactly the type the aggregate returns, so apply can
+ * write it back to the hidden top-K state columns without rebuilding a struct. The caller derives
+ * {@code n' = n + dn} and {@code exhausted = (size(values) == 0 && n' > 0)} itself.
+ *
+ * <p>A NULL list is an empty list; a NULL {@code b}/{@code b_i}/{@code b_d} means that side is
+ * complete/untruncated. The result struct itself is never NULL, only its {@code bound} field can
+ * be: the merge only trusts values in {@code v <= min(b, b_i, b_d)}, so {@code bound} is the
+ * effective upper bound (for MIN; the lower bound for MAX) of the resulting state.
+ */
+public abstract class TopKMergeFunction extends ScalarFunction
+ implements CustomSignature, AlwaysNotNullable {
+
+ protected TopKMergeFunction(String name, List<Expression> children) {
+ super(name, children);
+ }
+
+ /** constructor for withChildren and reuse signature */
+ protected TopKMergeFunction(ScalarFunctionParams functionParams) {
+ super(functionParams);
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ if (arity() != 9) {
+ throw new AnalysisException(getName() + " requires 9 arguments, but got " + arity());
+ }
+ DataType valueType = inferValueType();
+ if (valueType == null) {
+ throw new AnalysisException(getName() + " can not infer the value type from arguments: "
+ + toSql());
+ }
+ if (!TopKCountFunction.isSupportedValueType(valueType)) {
+ throw new AnalysisException(getName() + " does not support type " + valueType.toSql());
+ }
+ for (int index : new int[] {0, 2, 4}) {
+ DataType type = getArgumentType(index);
+ if (type.isNullType()) {
+ continue;
+ }
+ if (!(type instanceof ArrayType)
+ || !(((ArrayType) type).getItemType() instanceof StructType)) {
+ throw new AnalysisException(getName() + " requires argument " + (index + 1)
+ + " to be ARRAY<STRUCT<value, count>>, but got " + type.toSql());
+ }
+ }
+ }
+
+ @Override
+ public FunctionSignature customSignature() {
+ DataType valueType = inferValueType();
+ if (valueType == null) {
+ throw new AnalysisException(getName() + " can not infer the value type from arguments: "
+ + toSql());
+ }
+ ArrayType listType = ArrayType.of(TopKCountFunction.pairType(valueType));
+ return FunctionSignature.ret(resultType(valueType)).args(
Review Comment:
[P1] Preserve nested decimal precision when bounds are NULL
The inherited decimal precision pass changes the value type chosen here even for homogeneous input lists. With S=[(1.23,1)] and empty I/D all typed as ARRAY<STRUCT<value:DECIMAL(4,2),count:BIGINT>>, plain NULL bounds, n=1, dn=0 and k=2, `extractBySignature` skips the STRUCT entries, so only the NULL bounds contribute and select DECIMAL(1,0). `defaultDecimalV3PrecisionPromotion` then rewrites every nested decimal argument to that type. Coercion changes 1.23 to 1, and normal analysis recomputes the result type from the narrowed children, returning 1 instead of 1.23. Both MIN and MAX are affected. Override precision computation to include the nested value types and ignore untyped NULL bounds, so a complete decimal state retains its values; add a decimal merge regression with plain NULL bounds.
##########
be/src/exprs/function/function_topk_merge.cpp:
##########
@@ -0,0 +1,465 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <glog/logging.h>
+
+#include <cstddef>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_numbers.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/types.h"
+#include "exprs/function/function.h"
+#include "exprs/function/simple_function_factory.h"
+
+namespace doris {
+class FunctionContext;
+} // namespace doris
+
+namespace doris {
+
+namespace {
+
+/// Upper bound of k. Keep in sync with the aggregate's MAX_TOPK_K
+/// (aggregate_function_topk_count.h) and FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/// One row of an ARRAY<STRUCT<value, count>> argument. A NULL array is an empty view.
+struct TopKArrayView {
+ const IColumn* values = nullptr;
+ const ColumnInt64* counts = nullptr;
+ /// NULL bitmap of the array element (the pair struct).
+ const NullMap* null_map = nullptr;
+ /// NULL bitmap of the `value` field; NULL values are not part of the contract.
+ const NullMap* value_null_map = nullptr;
+ /// NULL bitmap of the `count` field; NULL counts are not part of the contract.
+ const NullMap* count_null_map = nullptr;
+ size_t begin = 0;
+ size_t end = 0;
+};
+
+TopKArrayView make_array_view(const IColumn& column, size_t row) {
+ TopKArrayView view;
+ const IColumn* array_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ array_column = &nullable->get_nested_column();
+ }
+ const auto& array = assert_cast<const ColumnArray&>(*array_column);
+ // offsets[i] is the cumulative size after row i, so row's range is [offsets[row-1], offsets[row]).
+ // offset_at(0) reads offsets[-1], which PaddedPODArray keeps as 0.
+ view.begin = array.offset_at(static_cast<ssize_t>(row));
+ view.end = array.offset_at(static_cast<ssize_t>(row) + 1);
+
+ const IColumn* pair_column = &array.get_data();
+ if (const auto* element_nullable = check_and_get_column<ColumnNullable>(pair_column)) {
+ view.null_map = &element_nullable->get_null_map_data();
+ pair_column = &element_nullable->get_nested_column();
+ }
+ const auto& pairs = assert_cast<const ColumnStruct&>(*pair_column);
+ view.values = &pairs.get_column(0);
+ // A cast may widen the `value` field to a nullable column; the entries themselves are never
+ // NULL, so only the nested column is used.
+ if (const auto* value_nullable = check_and_get_column<ColumnNullable>(view.values)) {
+ view.value_null_map = &value_nullable->get_null_map_data();
+ view.values = &value_nullable->get_nested_column();
+ }
+ // The same holds for `count`: a nested cast may have made the field nullable.
+ const auto* count_column = &pairs.get_column(1);
+ if (const auto* count_nullable = check_and_get_column<ColumnNullable>(count_column)) {
+ view.count_null_map = &count_nullable->get_null_map_data();
+ count_column = &count_nullable->get_nested_column();
+ }
+ view.counts = &assert_cast<const ColumnInt64&>(*count_column);
+ return view;
+}
+
+/// One scalar bound argument: NULL means the corresponding side was not truncated. `row` is the
+/// index of the current input row inside `value`: bounds are ordinary per-row columns, so a block
+/// can carry a different bound in every row.
+struct TopKBoundView {
+ const IColumn* value = nullptr;
+ size_t row = 0;
+ bool is_null = true;
+};
+
+TopKBoundView make_bound_view(const IColumn& column, size_t row) {
+ TopKBoundView view;
+ const IColumn* value_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ view.value = value_column;
+ view.row = row;
+ view.is_null = false;
+ return view;
+}
+
+int64_t read_int64(const IColumn& column, size_t row, const char* argument_name) {
+ const IColumn* nested = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ // The contract declares these arguments as non-NULL BIGINT.
+ DORIS_CHECK(!nullable->is_null_at(row)) << argument_name << " must not be NULL";
+ nested = &nullable->get_nested_column();
+ }
+ return assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(*nested).get_element(row);
+}
+
+int64_t checked_add(int64_t lhs, int64_t rhs, const char* function_name) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "{}: value count overflow", function_name);
+ }
+ return result;
+}
+
+int64_t checked_sub(int64_t lhs, int64_t rhs, const char* function_name) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_sub_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "{}: value count overflow", function_name);
+ }
+ return result;
+}
+
+/// FLOAT/DOUBLE use CompareFloat so NaN sorts greatest and is distinct from every number; other
+/// types use the column comparison, i.e. exactly the MIN/MAX comparison.
+int compare_values(PrimitiveType value_type, const IColumn& lhs, size_t lhs_row, const IColumn& rhs,
+ size_t rhs_row) {
+ if (value_type == TYPE_FLOAT) {
+ return CompareFloat(
+ assert_cast<const ColumnFloat32&, TypeCheckOnRelease::DISABLE>(lhs).get_element(
+ lhs_row),
+ assert_cast<const ColumnFloat32&, TypeCheckOnRelease::DISABLE>(rhs).get_element(
+ rhs_row));
+ }
+ if (value_type == TYPE_DOUBLE) {
+ return CompareFloat(
+ assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(lhs).get_element(
+ lhs_row),
+ assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(rhs).get_element(
+ rhs_row));
+ }
+ return lhs.compare_at(lhs_row, rhs_row, rhs, 1);
+}
+
+DataTypePtr find_value_type(const DataTypes& arguments) {
+ // Prefer the per-value type of S/I/D; it is also present when all bound arguments are NULL.
+ for (size_t index : {0, 2, 4}) {
+ const auto& type = remove_nullable(arguments[index]);
+ if (type->get_primitive_type() != TYPE_ARRAY) {
+ continue;
+ }
+ const auto& nested =
+ remove_nullable(assert_cast<const DataTypeArray&>(*type).get_nested_type());
+ if (nested->get_primitive_type() != TYPE_STRUCT) {
+ continue;
+ }
+ const auto& fields = assert_cast<const DataTypeStruct&>(*nested).get_elements();
+ if (fields.size() != 2) {
+ continue;
+ }
+ // The result struct always declares a non-nullable `value` field, while an argument cast
+ // may have produced a nullable one; drop it so both sides agree on the return type.
+ return remove_nullable(fields[0]);
+ }
+ for (size_t index : {1, 3, 5}) {
+ const auto& type = remove_nullable(arguments[index]);
+ if (!type->is_null_literal()) {
+ return type;
+ }
+ }
+ return nullptr;
+}
+
+/**
+ * topk_merge_min / topk_merge_max: one apply step of the IVM MIN/MAX top-K state.
+ *
+ * Arguments: S, b, I, b_i, D, b_d, n, dn, k. S is the pre-image state, I/D are the insert/delete
+ * top-K lists of this delta batch and b/i/d bounds are their truncation points (NULL = not
+ * truncated). Only values in [<= bE] are trusted, where bE = min of all non-NULL bounds (max for
+ * MAX): outside it at least one side may have dropped values. The post count of a value is
+ * S + I - D; non-positive counts are removed (net-zero cancellations), then the first k values are
+ * kept.
+ *
+ * The result is the state itself, i.e. the same type topk_min_count/topk_max_count return:
+ * STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T NULL>. n' = n + dn and
+ * exhausted = (size(values) == 0 && n' > 0) are derived by the caller, which writes n' to the
+ * COUNT state column and uses exhausted to trigger the fallback.
+ */
+template <bool IsMin>
+class FunctionTopKMerge : public IFunction {
+public:
+ static constexpr auto name = IsMin ? "topk_merge_min" : "topk_merge_max";
+
+ static FunctionPtr create() { return std::make_shared<FunctionTopKMerge>(); }
+
+ String get_name() const override { return name; }
+
+ size_t get_number_of_arguments() const override { return 9; }
+
+ /// A NULL list is an empty list and a NULL bound is a valid value: the default implementation
+ /// would turn the whole row into NULL and break the contract.
+ bool use_default_implementation_for_nulls() const override { return false; }
+
+ DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ DataTypePtr value_type = find_value_type(arguments);
+ if (value_type == nullptr) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} can not infer the value type from arguments", name);
+ }
+ // The result is the state type, identical to the aggregate's return type, so that the
+ // apply expression can write it straight back to the hidden MIN_TOPK/MAX_TOPK and BOUND
+ // columns.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {value_type, std::make_shared<DataTypeInt64>()},
+ Strings {"value", "count"});
+ auto values_type = std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ return std::make_shared<DataTypeStruct>(DataTypes {values_type, make_nullable(value_type)},
+ Strings {"values", "bound"});
+ }
+
+ Status execute_impl(FunctionContext* /*context*/, Block& block, const ColumnNumbers& arguments,
+ uint32_t result, size_t input_rows_count) const override {
+ std::vector<ColumnPtr> columns(arguments.size());
+ DataTypes argument_types(arguments.size());
+ for (size_t i = 0; i != arguments.size(); ++i) {
+ columns[i] =
+ block.get_by_position(arguments[i]).column->convert_to_full_column_if_const();
+ argument_types[i] = block.get_by_position(arguments[i]).type;
+ }
+ const DataTypePtr value_type = find_value_type(argument_types);
+ // An untyped call is rejected while planning get_return_type(); the plan always carries T.
+ DCHECK(value_type != nullptr);
+ const PrimitiveType primitive_type = value_type->get_primitive_type();
+
+ auto result_column = block.get_by_position(result).type->create_column();
+ auto& result_struct = assert_cast<ColumnStruct&>(*result_column);
+ auto& out_values = assert_cast<ColumnArray&>(result_struct.get_column(0));
+ auto& out_pair_nullable = assert_cast<ColumnNullable&>(out_values.get_data());
+ auto& out_pair = assert_cast<ColumnStruct&>(out_pair_nullable.get_nested_column());
+ auto& out_value_column = out_pair.get_column(0);
+ auto& out_count_column = assert_cast<ColumnInt64&>(out_pair.get_column(1));
+ auto& out_bound_column = assert_cast<ColumnNullable&>(result_struct.get_column(1));
+
+ for (size_t row = 0; row != input_rows_count; ++row) {
+ const int64_t k = read_int64(*columns[8], row, "k");
+ if (UNLIKELY(k < 1 || k > MAX_TOPK_K)) {
+ return Status::InvalidArgument("{} requires 1 <= k <= {}, but got {}", name,
+ MAX_TOPK_K, k);
+ }
+ const int64_t n = read_int64(*columns[6], row, "n");
+ const int64_t dn = read_int64(*columns[7], row, "dn");
+ const int64_t new_n = checked_add(n, dn, name);
+
+ const TopKArrayView s_view = make_array_view(*columns[0], row);
+ const TopKArrayView i_view = make_array_view(*columns[2], row);
+ const TopKArrayView d_view = make_array_view(*columns[4], row);
+ const TopKBoundView b_view = make_bound_view(*columns[1], row);
+ const TopKBoundView bi_view = make_bound_view(*columns[3], row);
+ const TopKBoundView bd_view = make_bound_view(*columns[5], row);
+ const TopKBoundView effective_bound =
+ effective_bound_of(b_view, bi_view, bd_view, primitive_type);
+
+ size_t positive = 0;
+ RETURN_IF_ERROR(append_trusted_values(s_view, i_view, d_view, effective_bound, k,
+ primitive_type, out_value_column,
+ out_count_column, out_pair_nullable, &positive));
+ out_values.get_offsets().push_back(out_pair.size());
+
+ // bound: NULL while the state is complete, the k-th kept value only when values were
+ // actually dropped (pos > k), or the effective bound otherwise.
+ // b IS NULL means the pre-image covers every distinct value; a NULL b_i/b_d means
+ // that side of the delta was not truncated.
+ // pos == k is NOT truncation: every known value fit, so keep bE instead of shrinking
+ // the trusted range to max(S').
+ const bool sides_complete = b_view.is_null && bi_view.is_null && bd_view.is_null;
+ if (new_n == 0 || (sides_complete && std::cmp_less_equal(positive, k))) {
+ out_bound_column.insert_default();
+ } else if (std::cmp_greater(positive, k)) {
+ out_bound_column.get_nested_column().insert_from(out_value_column,
+ out_pair.size() - 1);
+ out_bound_column.get_null_map_data().push_back(0);
+ } else {
+ DCHECK(!effective_bound.is_null);
+ out_bound_column.get_nested_column().insert_from(*effective_bound.value,
+ effective_bound.row);
+ out_bound_column.get_null_map_data().push_back(0);
+ }
+ }
+
+ block.replace_by_position(result, std::move(result_column));
+ return Status::OK();
+ }
+
+private:
+ /// bE: the best (smallest for MIN, largest for MAX) of the non-NULL bounds, or nullptr when
+ /// every side is complete/untruncated.
+ static TopKBoundView effective_bound_of(const TopKBoundView& b_view,
+ const TopKBoundView& bi_view,
+ const TopKBoundView& bd_view, PrimitiveType type) {
+ TopKBoundView bound;
+ for (const TopKBoundView* view : {&b_view, &bi_view, &bd_view}) {
+ if (view->is_null) {
+ continue;
+ }
+ if (bound.is_null) {
+ bound = *view;
+ continue;
+ }
+ int cmp = compare_values(type, *view->value, view->row, *bound.value, bound.row);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ bound = *view;
+ }
+ }
+ return bound;
+ }
+
+ /// Finds the best head (smallest for MIN, largest for MAX) among the three lists; `head` is
+ /// set to nullptr when all of them are exhausted.
+ static void best_head(const TopKArrayView& s_view, size_t s_index, const TopKArrayView& i_view,
+ size_t i_index, const TopKArrayView& d_view, size_t d_index,
+ PrimitiveType type, const IColumn** head, size_t* head_row) {
+ *head = nullptr;
+ const TopKArrayView* views[] = {&s_view, &i_view, &d_view};
+ const size_t indexes[] = {s_index, i_index, d_index};
+ for (size_t v = 0; v != 3; ++v) {
+ if (indexes[v] == views[v]->end) {
+ continue;
+ }
+ if (*head == nullptr) {
+ *head = views[v]->values;
+ *head_row = indexes[v];
+ continue;
+ }
+ int cmp = compare_values(type, *views[v]->values, indexes[v], **head, *head_row);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ *head = views[v]->values;
+ *head_row = indexes[v];
+ }
+ }
+ }
+
+ /// Consumes one entry equal to the current head value, returning its count.
+ static bool consume_equal(const TopKArrayView& view, size_t& index, const IColumn& head,
+ size_t head_row, PrimitiveType type, int64_t* count) {
+ if (index == view.end || compare_values(type, *view.values, index, head, head_row) != 0) {
+ return false;
+ }
+ // The top-K producers never emit a NULL entry, a NULL value or a NULL count.
+ DCHECK(view.null_map == nullptr || (*view.null_map)[index] == 0);
+ DCHECK(view.value_null_map == nullptr || (*view.value_null_map)[index] == 0);
Review Comment:
[P2] Reject NULL state fields before consuming their nested payload
These DCHECKs are the only checks for actual NULL entries/fields, but the public scalar signature does not guarantee their absence: FE complex coercion preserves source struct-field nullability. For example, S=`array(named_struct('value', CAST(NULL AS INT), 'count', CAST(1 AS BIGINT)))`, empty typed INT I/D, NULL bounds, n=1, dn=0 and k=1 is accepted. In release the value NULL map is ignored and its default INT payload is emitted as a non-NULL value 0 with count 1; checked builds hit the assertion. NULL counts/elements have the same unchecked path, and MAX shares it. Return a query error for actual NULL state entries/fields before comparing or consuming their payload, while continuing to allow NULL top-level lists and nullable wrappers containing non-NULL data.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/TopKCountFunction.java:
##########
@@ -0,0 +1,148 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.agg;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.CustomSignature;
+import org.apache.doris.nereids.trees.expressions.literal.ArrayLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.StructLiteral;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.DecimalV2Type;
+import org.apache.doris.nereids.types.DecimalV3Type;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Shared implementation of {@code topk_min_count(x, k)} and {@code topk_max_count(x, k)}.
+ *
+ * <p>Both aggregates return {@code STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T>}.
+ * {@code values} holds at most k distinct non-NULL values ordered by value (ascending for MIN,
+ * descending for MAX) with the number of rows equal to each value. {@code bound} is NULL when the
+ * group was not truncated; otherwise it is the k-th value and {@code bound IS NOT NULL} is the
+ * truncation flag. With {@code k = 0} no list is built and {@code bound} is the extreme value seen
+ * on this side (NULL when no non-NULL input was seen).
+ */
+public abstract class TopKCountFunction extends NotNullableAggregateFunction
+ implements CustomSignature, NullIgnoringAggregateFunction {
+
+ /** Upper bound of k, shared with the BE implementation. */
+ public static final long MAX_TOPK_K = 65536;
+
+ protected TopKCountFunction(String name, boolean distinct, List<Expression> children) {
+ super(name, distinct, false, children);
+ }
+
+ /** constructor for withChildren and reuse signature */
+ protected TopKCountFunction(AggregateFunctionParams functionParams) {
+ super(functionParams);
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ if (distinct) {
+ throw new AnalysisException(getName() + " does not support DISTINCT");
+ }
+ if (!getArgument(1).isConstant()) {
+ throw new AnalysisException(
+ getName() + " requires second parameter must be a constant: " + toSql());
+ }
+ if (!isSupportedValueType(getArgumentType(0))) {
+ throw new AnalysisException(getName() + " does not support type "
+ + getArgumentType(0).toSql());
+ }
+ }
+
+ @Override
+ public void checkLegalityAfterRewrite() {
+ checkLegalityBeforeTypeCoercion();
+ Expression count = getArgument(1);
+ // A constant FE cannot fold is left to BE, which reads the value from the count column.
+ if (!(count instanceof Literal)) {
Review Comment:
[P2] Validate evaluated k before nullable argument filtering
Deferring validation here does not ensure BE ever reads the count. With BE constant folding disabled (the default), `SELECT g, topk_min_count(v, bitmap_min(bitmap_from_string(''))) FROM (SELECT 1 AS g, 1 AS v UNION ALL SELECT 1 AS g, 2 AS v) t GROUP BY g` passes the constant check and remains nonliteral in FE, but its count evaluates to NULL. The variadic nullable aggregate wrapper then skips every input row before `AggregateFunctionTopKCount::check_k` is called, so this nonempty group silently returns an empty state instead of the required non-null-count error. MAX shares this path. Validate the evaluated constant k independently of value-row null filtering, and cover a backend-evaluated NULL count as well as a literal NULL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6432065857
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6434561488
<details>
<summary>ClickBench: <b>Total hot run time: 24.28 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5309c29db2b57c559711d38b6621f5c934c95d3a, data reload: false
query1 0.00 0.00 0.00
query2 0.09 0.05 0.05
query3 0.26 0.09 0.08
query4 1.62 0.10 0.12
query5 0.25 0.24 0.24
query6 1.17 0.67 0.68
query7 0.04 0.01 0.00
query8 0.05 0.04 0.03
query9 0.40 0.34 0.36
query10 0.58 0.60 0.58
query11 0.20 0.13 0.15
query12 0.19 0.15 0.16
query13 0.49 0.48 0.48
query14 1.01 1.00 1.00
query15 0.63 0.60 0.61
query16 0.35 0.34 0.34
query17 1.15 1.09 1.14
query18 0.19 0.19 0.18
query19 2.15 2.04 1.96
query20 0.02 0.02 0.01
query21 15.49 0.21 0.14
query22 4.90 0.05 0.06
query23 16.28 0.34 0.11
query24 3.11 0.47 0.39
query25 0.11 0.07 0.07
query26 1.34 0.22 0.17
query27 0.03 0.04 0.04
query28 3.47 0.77 0.35
query29 12.52 4.11 3.28
query30 0.28 0.13 0.09
query31 2.77 0.55 0.32
query32 3.22 0.58 0.51
query33 3.25 3.35 3.23
query34 15.30 4.01 3.37
query35 3.37 3.35 3.37
query36 0.55 0.44 0.44
query37 0.09 0.06 0.07
query38 0.04 0.03 0.04
query39 0.03 0.04 0.03
query40 0.17 0.15 0.14
query41 0.10 0.03 0.02
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 97.34 s
Total hot run time: 24.28 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4399856693
##########
be/src/exprs/lambda_function/varray_sorted_merge_function.cpp:
##########
@@ -0,0 +1,399 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+#include <map>
+#include <memory>
+#include <set>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "exprs/lambda_function/lambda_execution_context.h"
+#include "exprs/lambda_function/lambda_function.h"
+#include "exprs/lambda_function/lambda_function_factory.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/virtual_slot_ref.h"
+#include "exprs/vlambda_function_expr.h"
+
+namespace doris {
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc)
+ *
+ * Merges two arrays that are already sorted by the elements' natural order. The merge lambda
+ * receives the two current candidates and returns the combined element; returning NULL means the
+ * two elements must not be combined, so both are kept. Each input key occurs at most once per
+ * array, so a key appears at most twice across the two arrays and the lambda is called at most
+ * once per key.
+ */
+class ArraySortedMergeFunction : public LambdaFunction {
+ ENABLE_FACTORY_CREATOR(ArraySortedMergeFunction);
+
+public:
+ ~ArraySortedMergeFunction() override = default;
+
+ static constexpr auto name = "array_sorted_merge";
+
+ static LambdaFunctionPtr create() { return std::make_shared<ArraySortedMergeFunction>(); }
+
+ std::string get_name() const override { return name; }
+
+ /// Rejects a nested lambda that captures one of this lambda's arguments: a merge argument
+ /// stands for the candidate of one comparison step, which is not part of the supported
+ /// contract for a nested lambda. Nested lambdas may still use their own arguments
+ /// (shadowing) and may capture enclosing lambda arguments.
+ static Status validate_nested_captures(const VExprSPtr& expr,
+ const std::vector<std::string>& argument_names,
+ bool inside_nested_lambda,
+ std::vector<std::string> shadowed_names) {
+ if (expr->is_column_ref()) {
+ const std::string& column_name = expr->expr_name();
+ if (std::ranges::find(shadowed_names, column_name) != shadowed_names.end()) {
+ return Status::OK();
+ }
+ if (inside_nested_lambda &&
+ std::ranges::find(argument_names, column_name) != argument_names.end()) {
+ return Status::NotSupported(
+ "array_sorted_merge merge lambda argument '{}' cannot be captured by a "
+ "nested lambda",
+ column_name);
+ }
+ return Status::OK();
+ }
+ if (expr->node_type() == TExprNodeType::LAMBDA_FUNCTION_EXPR) {
+ const auto* lambda = assert_cast<const VLambdaFunctionExpr*>(expr.get());
+ if (lambda->has_argument_names()) {
+ for (const auto& argument_name : lambda->argument_names()) {
+ shadowed_names.emplace_back(argument_name);
+ }
+ }
+ return validate_nested_captures(lambda->get_child(0), argument_names, true,
+ std::move(shadowed_names));
+ }
+ for (const auto& child : expr->children()) {
+ RETURN_IF_ERROR(validate_nested_captures(child, argument_names, inside_nested_lambda,
+ shadowed_names));
+ }
+ return Status::OK();
+ }
+
+ /// A virtual column is materialized lazily on first read, from an expression over its own
+ /// input columns; its block entry starts as a ColumnNothing placeholder and the virtual slot
+ /// only records its own column id. Copying just that id into the private lambda block leaves
+ /// the inputs behind the padded "temp" columns, so the lazy evaluation reads a wrong-typed
+ /// column. Collect the virtual expressions' inputs as well, so they are copied with their
+ /// real types and the evaluation reads the current row.
+ static void collect_virtual_capture_inputs(const VExprSPtr& expr, std::set<int>& column_ids) {
+ if (expr->node_type() == TExprNodeType::VIRTUAL_SLOT_REF) {
+ const auto* virtual_slot = assert_cast<const VirtualSlotRef*>(expr.get());
+ const auto virtual_expr = virtual_slot->get_virtual_column_expr();
+ if (virtual_expr != nullptr) {
+ virtual_expr->collect_slot_column_ids(column_ids);
+ collect_virtual_capture_inputs(virtual_expr, column_ids);
+ }
+ }
+ for (const auto& child : expr->children()) {
+ collect_virtual_capture_inputs(child, column_ids);
+ }
+ }
+
+ Status prepare(RuntimeState* state, const VExprSPtrs& children) override {
+ RETURN_IF_ERROR(LambdaFunction::prepare(state, children));
+ DCHECK_EQ(children.size(), 4);
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ if (!lambda_expr->has_argument_names()) {
+ return Status::OK();
+ }
+ // A nested lambda that captures one of this lambda's arguments has no defined meaning
+ // (the argument is the candidate of a single comparison step); reject it during prepare
+ // instead of resolving it against some column at runtime.
+ return validate_nested_captures(children[0]->get_child(0), lambda_expr->argument_names(),
+ false, {});
+ }
+
+ Status execute(VExprContext* context, const Block* block, const Selector* expr_selector,
+ size_t count, ColumnPtr& result_column, const DataTypePtr& result_type,
+ const VExprSPtrs& children) const override {
+ DCHECK_EQ(children.size(), 4);
+
+ const ColumnArray* arrays[2] = {nullptr, nullptr};
+ const IColumn* data[2] = {nullptr, nullptr};
+ DataTypePtr nested_types[2];
+ ColumnPtr holders[2];
+ const ColumnUInt8::Container* nullmaps[2] = {nullptr, nullptr};
+ for (int i = 0; i < 2; ++i) {
+ RETURN_IF_ERROR(children[i + 1]->execute_column(context, block, expr_selector, count,
+ holders[i]));
+ holders[i] = holders[i]->convert_to_full_column_if_const();
+ DataTypePtr type = children[i + 1]->execute_type(block);
+ const IColumn* array_column = holders[i].get();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(array_column)) {
+ nullmaps[i] = &nullable->get_null_map_column().get_data();
+ array_column = &nullable->get_nested_column();
+ type = assert_cast<const DataTypeNullable&>(*type).get_nested_type();
+ }
+ arrays[i] = assert_cast<const ColumnArray*>(array_column);
+ data[i] = arrays[i]->get_data_ptr().get();
+ nested_types[i] = assert_cast<const DataTypeArray&>(*type).get_nested_type();
+ }
+
+ ColumnPtr flag_holder;
+ RETURN_IF_ERROR(
+ children[3]->execute_column(context, block, expr_selector, count, flag_holder));
+ flag_holder = flag_holder->convert_to_full_column_if_const();
+ const IColumn* flag_column = flag_holder.get();
+ const ColumnUInt8::Container* flag_nullmap = nullptr;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(flag_column)) {
+ flag_nullmap = &nullable->get_null_map_column().get_data();
+ flag_column = &nullable->get_nested_column();
+ }
+ const auto& flags = assert_cast<const ColumnUInt8&>(*flag_column).get_data();
+
+ const size_t rows = arrays[0]->size();
+ auto result_nested = nested_types[0]->create_column();
+ auto result_offsets = ColumnArray::ColumnOffsets::create();
+ auto& offsets_data = result_offsets->get_data();
+ offsets_data.reserve(rows);
+ auto result_nullmap = ColumnUInt8::create(rows, 0);
+ auto& result_nullmap_data = result_nullmap->get_data();
+
+ // The lambda may capture enclosing columns (for example a table column compared against
+ // the candidates). Materialize every captured input at its original position in the
+ // one-row lambda block, so slot refs read the current row; the two merge candidates
+ // take the positions after the last captured column and are bound by name.
+ std::set<int> captured_ids;
+ children[0]->collect_slot_column_ids(captured_ids);
+ collect_virtual_capture_inputs(children[0], captured_ids);
+ // An enclosing lambda's argument is a name reference rather than a slot, and this
+ // function executes its lambda against a block of its own that hides the parent frames.
+ // Publish the visible binding values under their names in that block, so the reference
+ // resolves by name like the merge arguments do.
+ std::vector<LambdaExecutionContext::Binding> enclosing_bindings;
+ context->lambda_execution_context().collect_visible_bindings(enclosing_bindings);
+ if ((!captured_ids.empty() || !enclosing_bindings.empty()) && block == nullptr) {
+ return Status::InternalError(
+ "array_sorted_merge lambda references outer columns but has no input block");
+ }
+ std::map<int, ColumnPtr> captured_columns;
+ for (int id : captured_ids) {
+ if (id < 0 || static_cast<size_t>(id) >= block->columns()) {
+ return Status::InternalError(
+ "array_sorted_merge lambda input column id {} is outside input block", id);
+ }
+ captured_columns[id] =
+ block->get_by_position(id).column->convert_to_full_column_if_const();
+ }
+ const int lambda_argument_base = captured_ids.empty() ? 0 : *captured_ids.rbegin() + 1;
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ const std::vector<std::string>* argument_names =
+ lambda_expr->has_argument_names() ? &lambda_expr->argument_names() : nullptr;
+ Block lambda_block;
+ for (int i = 0; i < lambda_argument_base; ++i) {
+ auto captured = captured_columns.find(i);
+ if (captured != captured_columns.end()) {
+ lambda_block.insert(ColumnWithTypeAndName(captured->second->clone_empty(),
+ block->get_by_position(i).type,
+ block->get_by_position(i).name));
+ } else {
+ lambda_block.insert(ColumnWithTypeAndName(
+ ColumnUInt8::create(1, 0), std::make_shared<DataTypeUInt8>(), "temp"));
+ }
+ }
+ // The enclosing bindings go before the merge arguments, and the name lookup scans the
+ // block backwards, so a merge argument with the same name shadows the outer one.
+ std::vector<ColumnPtr> enclosing_columns;
+ std::vector<int> enclosing_positions;
+ for (const auto& binding : enclosing_bindings) {
+ if (binding.column_position < 0 ||
+ static_cast<size_t>(binding.column_position) >= block->columns()) {
+ return Status::InternalError(
+ "array_sorted_merge enclosing lambda argument '{}' is outside the input "
+ "block (position {})",
+ binding.name, binding.column_position);
+ }
+ const auto& source = block->get_by_position(binding.column_position);
+ enclosing_columns.push_back(source.column->convert_to_full_column_if_const());
+ enclosing_positions.push_back(static_cast<int>(lambda_block.columns()));
+ lambda_block.insert(ColumnWithTypeAndName(enclosing_columns.back()->clone_empty(),
+ source.type, binding.name));
+ }
+ const int merge_argument_base = static_cast<int>(lambda_block.columns());
+ std::string merge_argument_names[2];
+ for (int i = 0; i < 2; ++i) {
+ merge_argument_names[i] = argument_names != nullptr && i < argument_names->size()
+ ? (*argument_names)[i]
+ : "merge_arg_" + std::to_string(i);
+ lambda_block.insert(ColumnWithTypeAndName(nested_types[i]->create_column(),
+ nested_types[i], merge_argument_names[i]));
+ }
+ // The lambda arguments are bound by name (the column refs carry the argument names),
+ // so the body reads these two one-row columns.
+ IColumn* argument_columns[2] = {
+ lambda_block.get_by_position(merge_argument_base).column->assert_mutable().get(),
+ lambda_block.get_by_position(merge_argument_base + 1)
+ .column->assert_mutable()
+ .get()};
+
+ LambdaExecutionContext::Frame lambda_frame;
+ if (argument_names != nullptr) {
+ // Rebase every binding this lambda can see onto the private block, so a nested lambda
+ // resolves an enclosing argument to the copy materialized here instead of walking into
+ // the parent frames, whose positions do not describe this block. The merge arguments
+ // are published last and therefore shadow an outer binding with the same name.
+ lambda_frame.bind_by_name = true;
+ lambda_frame.parent_bindings_visible = false;
+ for (size_t i = 0; i < enclosing_bindings.size(); ++i) {
+ lambda_frame.argument_bindings.push_back(
+ {enclosing_bindings[i].name, enclosing_positions[i]});
+ }
+ for (int i = 0; i < 2; ++i) {
+ lambda_frame.argument_bindings.push_back(
+ {merge_argument_names[i], merge_argument_base + i});
+ }
+ } else {
+ // Positional (legacy) references resolve through the block, so keep the old frame.
+ lambda_frame.bind_by_name = false;
+ lambda_frame.parent_bindings_visible = false;
+ }
+ LambdaExecutionContext::FrameGuard lambda_frame_guard(context->lambda_execution_context(),
+ std::move(lambda_frame));
+
+ const auto& offsets0 = arrays[0]->get_offsets();
+ const auto& offsets1 = arrays[1]->get_offsets();
+ const int lambda_result_base = static_cast<int>(lambda_block.columns());
+
+ // The merged element may come from a non-nullable lambda body (for example COALESCE),
+ // while array elements are always nullable; normalize before inserting.
+ auto append_element = [&](const IColumn& source, size_t element_row) {
+ if (auto* nullable_target = check_and_get_column<ColumnNullable>(result_nested.get())) {
+ if (const auto* nullable_source = check_and_get_column<ColumnNullable>(&source)) {
+ nullable_target->insert_from(*nullable_source, element_row);
+ } else {
+ nullable_target->get_nested_column().insert_from(source, element_row);
+ nullable_target->get_null_map_data().push_back(0);
+ }
+ } else {
+ result_nested->insert_from(source, element_row);
+ }
+ };
+
+ for (size_t row = 0; row < rows; ++row) {
+ // The ascending flag is an argument contract, not data: it is validated on every row,
+ // including a row whose input array is NULL, so a backend-evaluated NULL flag reports
+ // the same error regardless of the value column.
+ if (flag_nullmap != nullptr && (*flag_nullmap)[row] != 0) {
+ return Status::InvalidArgument(
+ "array_sorted_merge requires a non-null ascending flag");
+ }
+ const bool row_is_null = (nullmaps[0] != nullptr && (*nullmaps[0])[row] != 0) ||
+ (nullmaps[1] != nullptr && (*nullmaps[1])[row] != 0);
+ if (row_is_null) {
+ result_nullmap_data[row] = 1;
+ offsets_data.push_back(result_nested->size());
+ continue;
+ }
+ const bool ascending = flags[row] != 0;
+ // A short-circuit branch receives a selector and only the selected rows, so the row
+ // index is compacted: read the captured column at its original row, like the arrays.
+ const auto source_row = expr_selector == nullptr ? row : (*expr_selector)[row];
+ for (const auto& [id, column] : captured_columns) {
+ lambda_block.get_by_position(id).column = column->cut(source_row, 1);
+ }
+ for (size_t i = 0; i < enclosing_columns.size(); ++i) {
+ lambda_block.get_by_position(enclosing_positions[i]).column =
+ enclosing_columns[i]->cut(source_row, 1);
+ }
+
+ size_t left = row == 0 ? 0 : offsets0[row - 1];
+ const size_t left_end = offsets0[row];
+ size_t right = row == 0 ? 0 : offsets1[row - 1];
+ const size_t right_end = offsets1[row];
+
+ while (left < left_end || right < right_end) {
+ if (left == left_end) {
+ append_element(*data[1], right++);
+ continue;
+ }
+ if (right == right_end) {
+ append_element(*data[0], left++);
+ continue;
+ }
+
+ argument_columns[0]->clear();
+ argument_columns[0]->insert_from(*data[0], left);
+ argument_columns[1]->clear();
+ argument_columns[1]->insert_from(*data[1], right);
Review Comment:
Fixed in 69670f3e1a6: each one-row candidate column is now refreshed only when its element index advances, and both sides use the same cache, so the descending and reversed cases are covered as well. The indices are positions in the whole block's element column, so a cached candidate also stays valid across rows. `test_array_sorted_merge` passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6487925080
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6344680638
<details>
<summary>TPC-H: <b>Total hot run time: 28308 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a7420809fc69a4ea822cfd1f29ed9f8b03afbb3a, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17580 5121 5031 5031
q2 2143 332 245 245
q3 10204 1376 817 817
q4 4681 469 334 334
q5 7429 821 543 543
q6 182 182 151 151
q7 742 788 604 604
q8 9248 1330 1082 1082
q9 5526 4207 4168 4168
q10 6831 1291 1026 1026
q11 427 241 228 228
q12 639 396 277 277
q13 18094 3663 3014 3014
q14 278 265 244 244
q15 q16 740 726 649 649
q17 1182 747 563 563
q18 7767 6184 6110 6110
q19 1142 954 586 586
q20 404 335 223 223
q21 5556 2167 2257 2167
q22 365 306 246 246
Total cold run time: 101160 ms
Total hot run time: 28308 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5610 5585 5614 5585
q2 664 571 532 532
q3 5209 5369 4916 4916
q4 2000 2137 1544 1544
q5 5302 5175 5798 5175
q6 254 196 157 157
q7 2363 2158 1893 1893
q8 3077 2928 2722 2722
q9 8112 8205 8143 8143
q10 3685 3649 3226 3226
q11 606 428 399 399
q12 675 683 512 512
q13 3518 3899 3152 3152
q14 302 316 294 294
q15 q16 708 702 667 667
q17 8212 7521 7227 7227
q18 12921 12166 12905 12166
q19 883 760 766 760
q20 2154 2162 1926 1926
q21 5541 4563 4675 4563
q22 488 472 413 413
Total cold run time: 72284 ms
Total hot run time: 65972 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6341165179
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6362178464
# BE UT Coverage Report
Increment line coverage `81.50% (260/319)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/51bd4e1021eb209791aee5937061ac8805784483_51bd4e1021eb209791aee5937061ac8805784483/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/51bd4e1021eb209791aee5937061ac8805784483_51bd4e1021eb209791aee5937061ac8805784483/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 66.58% (32010/48080) |
| Line Coverage | 51.37% (337669/657352) |
| Region Coverage | 46.70% (272952/584486) |
| Branch Coverage | 48.45% (128736/265724) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6342765296
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6342766748
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6348747466
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6348748125
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6355797341
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6433028298
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6432420138
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6494658210
Memory of the final top-K state, for `min_topk` against `MIN`.
Setup: release build, single BE restarted before every query (cold page cache on both sides), `enable_sql_cache=false`, `exec_mem_limit` raised, RSS peak delta of one query. Two value shapes per type: 10M groups with 1 row each (10M rows) and 10M groups with 8 rows each (80M rows).
| value type | rows per group | total rows | `MIN(v)` | `min_topk(v, 8)` | state per group |
| --- | --- | --- | --- | --- | --- |
| BIGINT | 1 | 10M | 999 MB | 1687 MB | ~69 B |
| BIGINT | 8 | 80M | 1886 MB | 5038 MB | ~315 B |
| VARCHAR | 1 | 10M | 1501 MB | 2987 MB | ~149 B |
| VARCHAR | 8 | 80M | 2742 MB | 8214 MB | ~547 B |
"state per group" is the `min_topk` query minus the `MIN` query of the same shape, divided by the group count.
Query profile cross-check (10M groups, 1 row per group, per-operator `MemoryUsagePeak`): the aggregation sink peaks at 774 MB for `MIN(v)` and 1.25 GB for `min_topk(v, 8)` on BIGINT, and at 2.35 GB for `min_topk(v, 8)` on VARCHAR. The profile counts the state record but not the slot vector's payload, so it reads lower than RSS at 8 values per group.
What the state costs: 56 bytes of fixed part per group (k, the slot vector header, the cached worst slot, the live/dead counters and the priority seed) plus 32 bytes per retained value for BIGINT and 40 bytes for VARCHAR (count, both treap links, the priority and the value or its span). A `MIN` state is a single 8-byte value, so the comparison is dominated by that fixed part at one value per group and by the slots at eight.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6498248469
<details>
<summary>TPC-H: <b>Total hot run time: 28694 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 29fb3c47891613984533a1602cf3c9e6d15e2bb2, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17596 5109 5017 5017
q2 2174 319 261 261
q3 10221 1452 819 819
q4 4675 471 335 335
q5 7422 793 531 531
q6 175 179 149 149
q7 742 789 599 599
q8 9253 1373 1086 1086
q9 5540 4187 4203 4187
q10 6831 1275 1032 1032
q11 418 243 230 230
q12 638 400 280 280
q13 18116 3677 3011 3011
q14 275 274 247 247
q15 q16 710 719 647 647
q17 1214 773 573 573
q18 7034 6138 6093 6093
q19 1115 960 589 589
q20 385 330 220 220
q21 5400 2495 2509 2495
q22 383 334 293 293
Total cold run time: 100317 ms
Total hot run time: 28694 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5916 5904 6103 5904
q2 722 597 589 589
q3 5722 5801 5295 5295
q4 2002 2146 1563 1563
q5 5709 5793 5606 5606
q6 235 187 142 142
q7 2259 1996 1791 1791
q8 2970 2634 2662 2634
q9 7931 7416 7444 7416
q10 3604 3573 3178 3178
q11 568 405 381 381
q12 654 683 513 513
q13 3302 3643 3048 3048
q14 291 293 270 270
q15 q16 651 672 609 609
q17 7654 7016 6918 6918
q18 12929 12105 12974 12105
q19 921 786 800 786
q20 2136 2141 1913 1913
q21 5535 4780 4695 4695
q22 484 427 426 426
Total cold run time: 72195 ms
Total hot run time: 65782 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6490977271
Reference data: what the query profile and the audit log report for the same queries, next to the RSS numbers from the previous comment. These are not the numbers the change is justified with — the profile does not see the per-slot payload the change removes — but they are useful as an independent view.
Setup: release build, single BE, `enable_profile=true` / default audit logging, `enable_sql_cache=false`, 10,000,000 rows.
**Query profile, per-operator `MemoryUsagePeak` (after the change)**
| value type | rows per group | groups | operator | `MIN(v)` | `min_topk(v, 8)` |
| --- | --- | --- | --- | --- | --- |
| BIGINT | 1 | 10M | `AGGREGATION_SINK_OPERATOR` | 774.53 MB | 1.50 GB |
| BIGINT | 8 | 1.25M | `BUCKETED_AGGREGATION_SINK_OPERATOR` | 95.99 MB | 159.99 MB |
| VARCHAR | 8 | 1.25M | `BUCKETED_AGGREGATION_SINK_OPERATOR` | 159.99 MB | 287.99 MB |
**Audit log `PeakMemoryBytes`**: the 1.25M-group BIGINT `min_topk(v, 8)` query logs `32768128` bytes (32 MB), while its RSS peak delta is ~460 MB, so the audit figure does not cover the aggregation state.
**Why the profile number is only a partial view.** At 1.25M groups the same query reports exactly `159.99 MB` for both `min_topk(v, 1)` and `min_topk(v, 8)`, even though the second state holds eight slots per group instead of one. The operator counter therefore captures the arena-allocated state record but not the slot vector's payload, which is exactly what this change shrinks. In the 10M-group case, where the state record dominates, the profile and the RSS figures agree (774 MB vs 841 MB for `MIN`, 1.50 GB vs 1.59 GB for `min_topk`).
For comparison, the same queries under RSS sampling (from the previous comment, second of two rounds):
| value type | rows per group | `MIN(v)` | `min_topk(v, 8)` |
| --- | --- | --- | --- |
| BIGINT | 1 | 841 MB | 1594 MB |
| BIGINT | 8 | 89 MB | 459 MB |
| VARCHAR | 1 | 1426 MB | 2847 MB |
| VARCHAR | 8 | 153 MB | 833 MB |
RSS additionally responds to `k` where the profile does not: at 1.25M BIGINT groups, `min_topk(v, 1)` peaks at 298/302 MB and `min_topk(v, 8)` at 455/455 MB across two rounds, which is the expected cost of the extra slots.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6492852568
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4398865520
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,448 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/string_buffer.hpp"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/**
+ * State shared by min_topk and max_topk.
+ *
+ * The entries are held as two parallel structures: a column with the distinct values (sorted
+ * ascending for MIN / descending for MAX, at most k entries) and their non-negative int64 counts.
+ * Only values with count > 0 are kept. With k == 0 the list holds at most the single extreme
+ * value and its row count; the caller derives truncation from the list length, so the state
+ * carries no separate bound.
+ */
+struct AggregateTopKCountData {
+ const DataTypePtr value_type;
+ bool has_k = false;
+ int64_t k = 0;
+ MutableColumnPtr value_column;
+ // Tracked so that the per-group counts participate in query memory accounting: at the maximum
+ // k this buffer alone is 512 KiB per group.
+ DorisVector<int64_t> counts;
+
+ explicit AggregateTopKCountData(const DataTypes& argument_types_)
+ : value_type(remove_nullable(argument_types_[0])),
+ value_column(value_type->create_column()) {}
+
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * min_topk(x, k) / max_topk(x, k).
+ *
+ * Result: ARRAY<STRUCT<value T, count BIGINT>>. NULL inputs are ignored: add validates k for
+ * every row and then skips a NULL x, so the aggregate deliberately does not use the
+ * nullable-argument wrapper. The array is ordered by value (ascending for MIN), holds at most k
+ * distinct values with count > 0, and carries no truncation flag: the caller infers it from the
+ * length (a list of exactly k entries is treated as possibly truncated).
+ *
+ * With k = 0 the array holds at most the extreme value and its row count, which lets the
+ * append-only IVM path compute MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopK final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopK<T, IsMin>, true>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using Base = IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopK<T, IsMin>, true>;
+
+ explicit AggregateFunctionTopK(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "min_topk" : "max_topk"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(remove_nullable(this->argument_types[0]));
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is read for every row, including rows where x is NULL: an evaluated count that is
+ // NULL or out of range must fail the query instead of being skipped like a NULL value.
+ check_k(state, *columns[1], row_num);
+ const IColumn* value_column = columns[0];
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(value_column)) {
+ if (nullable->is_null_at(row_num)) {
+ return;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ add_value(state, *value_column, row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw any row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else if (UNLIKELY(state.k != other.k)) {
+ // Ordinary partial/final aggregation always shares the same constant k, but the
+ // public _state/_merge/_union combinators do not encode k in the AggState type, so a
+ // UNION ALL of states built with different literal k can reach this merge.
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} states have incompatible counts: {} vs {}", get_name(), state.k,
+ other.k);
+ }
+ if (state.k == 0) {
+ merge_extreme(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ auto size = static_cast<int64_t>(state.counts.size());
+ buf.write_binary(size);
+ write_column(buf, *state.value_column);
+ if (size != 0) {
+ // An empty list keeps an empty vector, whose data() is null; passing it to memcpy is
+ // undefined even for a zero length.
+ buf.write(reinterpret_cast<const char*>(state.counts.data()),
+ state.counts.size() * sizeof(int64_t));
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ int64_t size = 0;
+ buf.read_binary(size);
+ state.value_column = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(state.value_column->size()), size);
+ state.counts.resize(size);
+ if (size != 0) {
+ buf.read(reinterpret_cast<char*>(state.counts.data()), size * sizeof(int64_t));
+ }
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_array = assert_cast<ColumnArray&>(to);
+ // The array element and both fields are nullable in the planned type, but a produced entry
+ // never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(result_array.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = assert_cast<ColumnNullable&>(pair_column.get_column(0));
+ auto& count_column = assert_cast<ColumnNullable&>(pair_column.get_column(1));
+ for (size_t i = 0; i != state.counts.size(); ++i) {
+ if (const auto* nullable_value =
+ check_and_get_column<ColumnNullable>(state.value_column.get())) {
+ // A nullable value argument carries its own null map, which is never set here.
+ value_column.insert_from(*nullable_value, i);
+ } else {
+ value_column.get_nested_column().insert_from(*state.value_column, i);
+ value_column.get_null_map_data().push_back(0);
+ }
+ assert_cast<ColumnInt64&>(count_column.get_nested_column())
+ .get_data()
+ .push_back(state.counts[i]);
+ count_column.get_null_map_data().push_back(0);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ result_array.get_offsets().push_back(pair_column.size());
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ // Both fields are nullable in the planned type: a consumer reads a field through
+ // element_at, which is always nullable, and named_struct then records nullable fields.
+ // Produced entries never contain a NULL field.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(value_type),
+ make_nullable(std::make_shared<DataTypeInt64>())},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ return std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ }
+
+private:
+ static void check_k(AggregateTopKCountData& state, const IColumn& count_column,
+ size_t row_num) {
+ const IColumn* nested = &count_column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&count_column)) {
+ if (UNLIKELY(nullable->is_null_at(row_num))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-null "
+ "constant");
+ }
+ nested = &nullable->get_nested_column();
+ }
+ int64_t value =
+ assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(*nested).get_element(
+ row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-negative "
+ "constant, but got {}",
+ value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ /// Ordering of the values. FLOAT/DOUBLE use CompareFloat so that NaN sorts greatest and is
+ /// distinct from every number, which keeps the list mergeable and the top-K deterministic;
+ /// every other type uses the column's compare_at, i.e. exactly the MIN/MAX comparison.
+ static int compare_at(const IColumn& lhs, size_t lhs_row, const IColumn& rhs, size_t rhs_row) {
+ if constexpr (is_float_or_double(T)) {
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ const auto& left =
+ assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(lhs).get_data();
+ const auto& right =
+ assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(rhs).get_data();
+ return CompareFloat(left[lhs_row], right[rhs_row]);
+ } else {
+ return lhs.compare_at(lhs_row, rhs_row, rhs, 1);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "min_topk/max_topk value count overflow");
+ }
+ return result;
+ }
+
+ /// Inserts (value, count) at `pos`, keeping the value column sorted by value.
+ static void insert_entry(AggregateTopKCountData& state, size_t pos, const IColumn& column,
+ size_t row_num, int64_t count) {
+ if (pos == state.value_column->size()) {
+ // Appending needs no rebuild: the state column is mutable, so ascending MIN input
+ // (descending MAX input) fills the list without copying the retained prefix.
+ state.value_column->insert_from(column, row_num);
+ state.counts.push_back(count);
+ return;
+ }
+ auto new_values = state.value_column->clone_empty();
+ new_values->reserve(state.value_column->size() + 1);
+ for (size_t i = 0; i < pos; ++i) {
+ new_values->insert_from(*state.value_column, i);
+ }
+ new_values->insert_from(column, row_num);
+ for (size_t i = pos; i < state.value_column->size(); ++i) {
+ new_values->insert_from(*state.value_column, i);
+ }
+ state.value_column = std::move(new_values);
+ state.counts.insert(state.counts.begin() + pos, count);
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(AggregateTopKCountData& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (state.counts.empty()) {
+ state.value_column->insert_from(column, row_num);
+ state.counts.push_back(1);
+ return;
+ }
+ int cmp = compare_at(column, row_num, *state.value_column, 0);
+ if (cmp == 0) {
+ state.counts[0] = checked_add(state.counts[0], 1);
+ } else if (IsMin ? cmp < 0 : cmp > 0) {
+ state.value_column->clear();
+ state.value_column->insert_from(column, row_num);
+ state.counts[0] = 1;
+ }
+ return;
+ }
+
+ // The list is ascending for MIN and descending for MAX, so the insertion point is where
+ // the stored values stop being better than the new one.
+ size_t left = 0;
+ size_t right = state.value_column->size();
+ while (left < right) {
+ size_t mid = left + (right - left) / 2;
+ int cmp = compare_at(*state.value_column, mid, column, row_num);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ left = mid + 1;
+ } else {
+ right = mid;
+ }
+ }
+ if (left < state.value_column->size() &&
+ compare_at(*state.value_column, left, column, row_num) == 0) {
+ state.counts[left] = checked_add(state.counts[left], 1);
+ return;
+ }
+
+ // A full list already holds the k best values, so a value worse than the last entry would
+ // be inserted at the end and popped immediately: it can only flip the truncation flag.
+ // Rebating it early avoids cloning and copying the whole retained column for every
+ // rejected distinct value (ascending inputs at k = 65536 otherwise copy ~k values each).
+ if (left == state.value_column->size() &&
+ std::cmp_greater_equal(state.value_column->size(), state.k)) {
+ return;
+ }
+
+ insert_entry(state, left, column, row_num, 1);
+ if (static_cast<int64_t>(state.value_column->size()) > state.k) {
+ // More than k distinct values: drop the worst one.
+ state.value_column->pop_back(1);
+ state.counts.pop_back();
+ }
+ }
+
+ /// k == 0: the state holds at most the extreme value and its count.
+ static void merge_extreme(AggregateTopKCountData& state, const AggregateTopKCountData& other) {
+ if (other.counts.empty()) {
+ return;
+ }
+ if (state.counts.empty()) {
+ state.value_column->insert_from(*other.value_column, 0);
+ state.counts.push_back(other.counts[0]);
+ return;
+ }
+ int cmp = compare_at(*state.value_column, 0, *other.value_column, 0);
+ if (cmp == 0) {
+ state.counts[0] = checked_add(state.counts[0], other.counts[0]);
+ } else if (IsMin ? cmp > 0 : cmp < 0) {
+ state.value_column->clear();
+ state.value_column->insert_from(*other.value_column, 0);
+ state.counts[0] = other.counts[0];
+ }
+ }
+
+ /// Merges two sorted lists, summing the counts of equal values and truncating to k.
+ static void merge_values(AggregateTopKCountData& state, const AggregateTopKCountData& other) {
+ if (other.counts.empty()) {
+ // The other partial state saw no non-NULL row.
+ return;
+ }
+ auto merged = state.value_column->clone_empty();
+ DorisVector<int64_t> merged_counts;
+ merged_counts.reserve(state.counts.size() + other.counts.size());
+ size_t lhs_index = 0;
+ size_t rhs_index = 0;
+ while (lhs_index < state.counts.size() || rhs_index < other.counts.size()) {
Review Comment:
[P2] Stop merging once the retained top-K prefix is complete
Once k distinct entries have been emitted, the remaining values cannot affect the result, but this loop still calls `checked_add` on their counts before truncating the column. For `min_topk`, consider two partial states with `k = 2`:
```text
A = [(0, 1), (2, INT64_MAX)]
B = [(1, 1), (2, 1)]
```
The retained result should be `[(0, 1), (1, 1)]`; both output counts fit BIGINT. Instead, after producing that prefix, the loop processes the discarded value 2 and throws on `INT64_MAX + 1`. A single-pass state that already retains 0 and 1 would discard the same worse value without counting it. MAX has the symmetric case.
Please stop the merge after k output entries, while retaining checked addition for every value that actually enters the result. Each input state already contains distinct values, so an emitted entry has consumed all contributions to that value. This also avoids materializing and then removing the unused tail.
Please add a serialized-state unit test for this discarded-tail overflow, alongside the existing test that requires overflow on a retained value to fail, and cover both MIN/MAX.
Validation: I reproduced the exception using the verbatim extracted `merge_values` loop with minimal column/allocator substitutes. I have not run this case in Doris BE UT or SQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6472166641
<details>
<summary>TPC-DS: <b>Total hot run time: 150367 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5c8beb3b4bc041961f08e39e9b5aae6284a64cd6, data reload: false
query5 4304 593 470 470
query6 415 205 199 199
query7 4800 455 222 222
query8 323 166 164 164
query9 8742 3233 3173 3173
query10 486 316 260 260
query11 5992 3871 3560 3560
query12 151 91 85 85
query13 1261 445 337 337
query14 6149 3902 3634 3634
query14_1 3397 3374 3416 3374
query15 209 203 182 182
query16 944 426 426 426
query17 856 654 536 536
query18 2408 423 315 315
query19 196 165 137 137
query20 86 85 86 85
query21 215 133 106 106
query22 12947 12958 12813 12813
query23 13517 12893 12315 12315
query23_1 12443 12580 12425 12425
query24 7354 836 481 481
query24_1 484 496 480 480
query25 523 421 373 373
query26 1265 261 138 138
query27 2768 462 275 275
query28 4588 2031 2033 2031
query29 1547 608 450 450
query30 301 222 191 191
query31 1001 929 812 812
query32 141 111 92 92
query33 521 307 247 247
query34 891 861 499 499
query35 762 734 628 628
query36 809 824 757 757
query37 123 95 83 83
query38 1772 1718 1660 1660
query39 793 781 739 739
query39_1 721 728 735 728
query40 218 122 95 95
query41 65 63 63 63
query42 86 83 81 81
query43 338 360 307 307
query44 1326 693 711 693
query45 182 172 168 168
query46 842 953 556 556
query47 2955 3016 2914 2914
query48 294 313 231 231
query49 564 392 295 295
query50 685 275 226 226
query51 10253 10164 10442 10164
query52 79 82 70 70
query53 182 206 153 153
query54 240 189 188 188
query55 80 70 64 64
query56 230 209 213 209
query57 1625 1613 1556 1556
query58 291 262 256 256
query59 2177 2218 1994 1994
query60 278 239 211 211
query61 148 152 147 147
query62 397 330 301 301
query63 191 159 157 157
query64 2726 931 755 755
query65 3359 3269 3263 3263
query66 1753 426 291 291
query67 20066 19989 20106 19989
query68 3077 1008 612 612
query69 386 293 239 239
query70 835 791 778 778
query71 283 230 213 213
query72 2717 2630 2064 2064
query73 521 523 295 295
query74 4606 4506 4296 4296
query75 2323 2285 1893 1893
query76 2318 1014 588 588
query77 346 374 284 284
query78 9665 9669 9044 9044
query79 1022 882 540 540
query80 543 410 331 331
query81 529 313 276 276
query82 579 140 116 116
query83 297 195 193 193
query84 307 125 104 104
query85 838 412 359 359
query86 329 249 230 230
query87 1977 2000 1811 1811
query88 3763 2756 2749 2749
query89 330 278 241 241
query90 1844 166 160 160
query91 162 152 127 127
query92 98 87 83 83
query93 1029 995 567 567
query94 462 305 291 291
query95 568 388 304 304
query96 685 479 249 249
query97 2474 2427 2331 2331
query98 160 157 147 147
query99 774 762 648 648
Total cold run time: 233395 ms
Total hot run time: 150367 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4347997245
##########
be/src/exprs/function/function_topk_merge.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <glog/logging.h>
+
+#include <cstddef>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_numbers.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/types.h"
+#include "exprs/function/function.h"
+#include "exprs/function/simple_function_factory.h"
+
+namespace doris {
+class FunctionContext;
+} // namespace doris
+
+namespace doris {
+
+namespace {
+
+/// Upper bound of k. Keep in sync with the aggregate's MAX_TOPK_K
+/// (aggregate_function_topk_count.h) and FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/// One row of an ARRAY<STRUCT<value, count>> argument. A NULL array is an empty view.
+struct TopKArrayView {
+ const IColumn* values = nullptr;
+ const ColumnInt64* counts = nullptr;
+ /// NULL bitmap of the array element (the pair struct).
+ const NullMap* null_map = nullptr;
+ /// NULL bitmap of the `value` field; NULL values are not part of the contract.
+ const NullMap* value_null_map = nullptr;
+ /// NULL bitmap of the `count` field; NULL counts are not part of the contract.
+ const NullMap* count_null_map = nullptr;
+ size_t begin = 0;
+ size_t end = 0;
+};
+
+TopKArrayView make_array_view(const IColumn& column, size_t row) {
+ TopKArrayView view;
+ const IColumn* array_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ array_column = &nullable->get_nested_column();
+ }
+ const auto& array = assert_cast<const ColumnArray&>(*array_column);
+ // offsets[i] is the cumulative size after row i, so row's range is [offsets[row-1], offsets[row]).
+ // offset_at(0) reads offsets[-1], which PaddedPODArray keeps as 0.
+ view.begin = array.offset_at(static_cast<ssize_t>(row));
+ view.end = array.offset_at(static_cast<ssize_t>(row) + 1);
+
+ const IColumn* pair_column = &array.get_data();
+ if (const auto* element_nullable = check_and_get_column<ColumnNullable>(pair_column)) {
+ view.null_map = &element_nullable->get_null_map_data();
+ pair_column = &element_nullable->get_nested_column();
+ }
+ const auto& pairs = assert_cast<const ColumnStruct&>(*pair_column);
+ view.values = &pairs.get_column(0);
+ // A cast may widen the `value` field to a nullable column; the entries themselves are never
+ // NULL, so only the nested column is used.
+ if (const auto* value_nullable = check_and_get_column<ColumnNullable>(view.values)) {
+ view.value_null_map = &value_nullable->get_null_map_data();
+ view.values = &value_nullable->get_nested_column();
+ }
+ // The same holds for `count`: a nested cast may have made the field nullable.
+ const auto* count_column = &pairs.get_column(1);
+ if (const auto* count_nullable = check_and_get_column<ColumnNullable>(count_column)) {
+ view.count_null_map = &count_nullable->get_null_map_data();
+ count_column = &count_nullable->get_nested_column();
+ }
+ view.counts = &assert_cast<const ColumnInt64&>(*count_column);
+ return view;
+}
+
+/// One scalar bound argument: NULL means the corresponding side was not truncated.
+struct TopKBoundView {
+ const IColumn* value = nullptr;
+ bool is_null = true;
+};
+
+TopKBoundView make_bound_view(const IColumn& column, size_t row) {
+ TopKBoundView view;
+ const IColumn* value_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ view.value = value_column;
Review Comment:
Adding a concrete counterexample for the original implementation at `ddda17f849`: this can produce a wrong nonempty candidate and suppress a required fallback.
Consider two rows in the same execution block, with `k = 1`:
| Row | Original group data | Stored S / b | Inserts I / bi | Deletes D / bd |
| --- | --- | --- | --- | --- |
| 0 | `{10,20}` | `[(10,1)] / 10` | `[] / NULL` | `[] / NULL` |
| 1 | `{1,3}` | `[(1,1)] / 1` | `[(5,1)] / NULL` | `[(1,1)] / NULL` |
For row 1, `n = 2`, `dn = 0`, and the actual post-update data is `{3,5}`, so MIN is **3**. Its old state has discarded 3, and only values <= 1 remain within the trusted range.
**Expected:** use row 1's bound, `bE = 1`. The count of 1 becomes `1 + 0 - 1 = 0`; 5 is outside the trusted range. Return `(values=[], bound=1)`. Since `n' = 2`, the empty candidate list tells the caller to fall back to a complete refresh, which finds 3.
**Original behavior:** the lists are taken from row 1, but the bound is read from row 0, giving `bE = 10`. After removing 1, the merge incorrectly admits 5 and returns `(values=[(5,1)], bound=10)`. The candidate list is nonempty, so the documented exhaustion check does not trigger fallback; using the first candidate as MIN would return **5**, although 3 still exists.
I checked the latest head, `e4acad346d`: `TopKBoundView` now carries the row index, and effective-bound selection, range filtering, and bound output all use it. This fixes the identified indexing cause by source inspection. The added multirow regression uses empty insert/delete lists; please also cover this delete-plus-insert case and its MAX counterpart, with both rows in one execution block, to guard against suppressing a required fallback.
This example is derived from the code; I have not executed it against Doris.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4348580115
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/TopKCountFunction.java:
##########
@@ -0,0 +1,148 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.agg;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.CustomSignature;
+import org.apache.doris.nereids.trees.expressions.literal.ArrayLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.StructLiteral;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.DecimalV2Type;
+import org.apache.doris.nereids.types.DecimalV3Type;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Shared implementation of {@code topk_min_count(x, k)} and {@code topk_max_count(x, k)}.
+ *
+ * <p>Both aggregates return {@code STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T>}.
+ * {@code values} holds at most k distinct non-NULL values ordered by value (ascending for MIN,
+ * descending for MAX) with the number of rows equal to each value. {@code bound} is NULL when the
+ * group was not truncated; otherwise it is the k-th value and {@code bound IS NOT NULL} is the
+ * truncation flag. With {@code k = 0} no list is built and {@code bound} is the extreme value seen
+ * on this side (NULL when no non-NULL input was seen).
+ */
+public abstract class TopKCountFunction extends NotNullableAggregateFunction
+ implements CustomSignature, NullIgnoringAggregateFunction {
+
+ /** Upper bound of k, shared with the BE implementation. */
+ public static final long MAX_TOPK_K = 65536;
+
+ protected TopKCountFunction(String name, boolean distinct, List<Expression> children) {
+ super(name, distinct, false, children);
+ }
+
+ /** constructor for withChildren and reuse signature */
+ protected TopKCountFunction(AggregateFunctionParams functionParams) {
+ super(functionParams);
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ if (distinct) {
+ throw new AnalysisException(getName() + " does not support DISTINCT");
+ }
+ if (!getArgument(1).isConstant()) {
+ throw new AnalysisException(
+ getName() + " requires second parameter must be a constant: " + toSql());
+ }
+ if (!isSupportedValueType(getArgumentType(0))) {
+ throw new AnalysisException(getName() + " does not support type "
+ + getArgumentType(0).toSql());
+ }
+ }
+
+ @Override
+ public void checkLegalityAfterRewrite() {
+ checkLegalityBeforeTypeCoercion();
+ Expression count = getArgument(1);
+ // A constant FE cannot fold is left to BE, which reads the value from the count column.
+ if (!(count instanceof Literal)) {
Review Comment:
Fixed in e4acad346dd. The aggregate is now constructed directly instead of through the nullable-argument adapter, which dropped rows before `add()` ran, so `add()` calls `check_k` for every row, including rows whose `x` is NULL, and a NULL or out-of-range evaluated k fails the query. The `bitmap_min` regression covers the non-foldable NULL count.
##########
be/src/exprs/function/function_topk_merge.cpp:
##########
@@ -0,0 +1,482 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <glog/logging.h>
+
+#include <cstddef>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_numbers.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/types.h"
+#include "exprs/function/function.h"
+#include "exprs/function/simple_function_factory.h"
+
+namespace doris {
+class FunctionContext;
+} // namespace doris
+
+namespace doris {
+
+namespace {
+
+/// Upper bound of k. Keep in sync with the aggregate's MAX_TOPK_K
+/// (aggregate_function_topk_count.h) and FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/// One row of an ARRAY<STRUCT<value, count>> argument. A NULL array is an empty view.
+struct TopKArrayView {
+ const IColumn* values = nullptr;
+ const ColumnInt64* counts = nullptr;
+ /// NULL bitmap of the array element (the pair struct).
+ const NullMap* null_map = nullptr;
+ /// NULL bitmap of the `value` field; NULL values are not part of the contract.
+ const NullMap* value_null_map = nullptr;
+ /// NULL bitmap of the `count` field; NULL counts are not part of the contract.
+ const NullMap* count_null_map = nullptr;
+ size_t begin = 0;
+ size_t end = 0;
+};
+
+TopKArrayView make_array_view(const IColumn& column, size_t row) {
+ TopKArrayView view;
+ const IColumn* array_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ array_column = &nullable->get_nested_column();
+ }
+ const auto& array = assert_cast<const ColumnArray&>(*array_column);
+ // offsets[i] is the cumulative size after row i, so row's range is [offsets[row-1], offsets[row]).
+ // offset_at(0) reads offsets[-1], which PaddedPODArray keeps as 0.
+ view.begin = array.offset_at(static_cast<ssize_t>(row));
+ view.end = array.offset_at(static_cast<ssize_t>(row) + 1);
+
+ const IColumn* pair_column = &array.get_data();
+ if (const auto* element_nullable = check_and_get_column<ColumnNullable>(pair_column)) {
+ view.null_map = &element_nullable->get_null_map_data();
+ pair_column = &element_nullable->get_nested_column();
+ }
+ const auto& pairs = assert_cast<const ColumnStruct&>(*pair_column);
+ view.values = &pairs.get_column(0);
+ // A cast may widen the `value` field to a nullable column; the entries themselves are never
+ // NULL, so only the nested column is used.
+ if (const auto* value_nullable = check_and_get_column<ColumnNullable>(view.values)) {
+ view.value_null_map = &value_nullable->get_null_map_data();
+ view.values = &value_nullable->get_nested_column();
+ }
+ // The same holds for `count`: a nested cast may have made the field nullable.
+ const auto* count_column = &pairs.get_column(1);
+ if (const auto* count_nullable = check_and_get_column<ColumnNullable>(count_column)) {
+ view.count_null_map = &count_nullable->get_null_map_data();
+ count_column = &count_nullable->get_nested_column();
+ }
+ view.counts = &assert_cast<const ColumnInt64&>(*count_column);
+ return view;
+}
+
+/// One scalar bound argument: NULL means the corresponding side was not truncated. `row` is the
+/// index of the current input row inside `value`: bounds are ordinary per-row columns, so a block
+/// can carry a different bound in every row.
+struct TopKBoundView {
+ const IColumn* value = nullptr;
+ size_t row = 0;
+ bool is_null = true;
+};
+
+TopKBoundView make_bound_view(const IColumn& column, size_t row) {
+ TopKBoundView view;
+ const IColumn* value_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ view.value = value_column;
+ view.row = row;
+ view.is_null = false;
+ return view;
+}
+
+int64_t read_int64(const IColumn& column, size_t row, const char* argument_name) {
+ const IColumn* nested = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ // The contract declares these arguments as non-NULL BIGINT.
+ DORIS_CHECK(!nullable->is_null_at(row)) << argument_name << " must not be NULL";
+ nested = &nullable->get_nested_column();
+ }
+ return assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(*nested).get_element(row);
+}
+
+int64_t checked_add(int64_t lhs, int64_t rhs, const char* function_name) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "{}: value count overflow", function_name);
+ }
+ return result;
+}
+
+int64_t checked_sub(int64_t lhs, int64_t rhs, const char* function_name) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_sub_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "{}: value count overflow", function_name);
+ }
+ return result;
+}
+
+/// FLOAT/DOUBLE use CompareFloat so NaN sorts greatest and is distinct from every number; other
+/// types use the column comparison, i.e. exactly the MIN/MAX comparison.
+int compare_values(PrimitiveType value_type, const IColumn& lhs, size_t lhs_row, const IColumn& rhs,
+ size_t rhs_row) {
+ if (value_type == TYPE_FLOAT) {
+ return CompareFloat(
+ assert_cast<const ColumnFloat32&, TypeCheckOnRelease::DISABLE>(lhs).get_element(
+ lhs_row),
+ assert_cast<const ColumnFloat32&, TypeCheckOnRelease::DISABLE>(rhs).get_element(
+ rhs_row));
+ }
+ if (value_type == TYPE_DOUBLE) {
+ return CompareFloat(
+ assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(lhs).get_element(
+ lhs_row),
+ assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(rhs).get_element(
+ rhs_row));
+ }
+ return lhs.compare_at(lhs_row, rhs_row, rhs, 1);
+}
+
+DataTypePtr find_value_type(const DataTypes& arguments) {
+ // Prefer the per-value type of S/I/D; it is also present when all bound arguments are NULL.
+ for (size_t index : {0, 2, 4}) {
+ const auto& type = remove_nullable(arguments[index]);
+ if (type->get_primitive_type() != TYPE_ARRAY) {
+ continue;
+ }
+ const auto& nested =
+ remove_nullable(assert_cast<const DataTypeArray&>(*type).get_nested_type());
+ if (nested->get_primitive_type() != TYPE_STRUCT) {
+ continue;
+ }
+ const auto& fields = assert_cast<const DataTypeStruct&>(*nested).get_elements();
+ if (fields.size() != 2) {
+ continue;
+ }
+ // The result struct always declares a non-nullable `value` field, while an argument cast
+ // may have produced a nullable one; drop it so both sides agree on the return type.
+ return remove_nullable(fields[0]);
+ }
+ for (size_t index : {1, 3, 5}) {
+ const auto& type = remove_nullable(arguments[index]);
+ if (!type->is_null_literal()) {
+ return type;
+ }
+ }
+ return nullptr;
+}
+
+/**
+ * topk_merge_min / topk_merge_max: one apply step of the IVM MIN/MAX top-K state.
+ *
+ * Arguments: S, b, I, b_i, D, b_d, n, dn, k. S is the pre-image state, I/D are the insert/delete
+ * top-K lists of this delta batch and b/i/d bounds are their truncation points (NULL = not
+ * truncated). Only values in [<= bE] are trusted, where bE = min of all non-NULL bounds (max for
+ * MAX): outside it at least one side may have dropped values. The post count of a value is
+ * S + I - D; non-positive counts are removed (net-zero cancellations), then the first k values are
+ * kept.
+ *
+ * The result is the state itself, i.e. the same type topk_min_count/topk_max_count return:
+ * STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T NULL>. n' = n + dn and
+ * exhausted = (size(values) == 0 && n' > 0) are derived by the caller, which writes n' to the
+ * COUNT state column and uses exhausted to trigger the fallback.
+ */
+template <bool IsMin>
+class FunctionTopKMerge : public IFunction {
+public:
+ static constexpr auto name = IsMin ? "topk_merge_min" : "topk_merge_max";
+
+ static FunctionPtr create() { return std::make_shared<FunctionTopKMerge>(); }
+
+ String get_name() const override { return name; }
+
+ size_t get_number_of_arguments() const override { return 9; }
+
+ /// A NULL list is an empty list and a NULL bound is a valid value: the default implementation
+ /// would turn the whole row into NULL and break the contract.
+ bool use_default_implementation_for_nulls() const override { return false; }
+
+ DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ DataTypePtr value_type = find_value_type(arguments);
+ if (value_type == nullptr) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} can not infer the value type from arguments", name);
+ }
+ // The result is the state type, identical to the aggregate's return type, so that the
+ // apply expression can write it straight back to the hidden MIN_TOPK/MAX_TOPK and BOUND
+ // columns.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {value_type, std::make_shared<DataTypeInt64>()},
+ Strings {"value", "count"});
+ auto values_type = std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ return std::make_shared<DataTypeStruct>(DataTypes {values_type, make_nullable(value_type)},
+ Strings {"values", "bound"});
+ }
+
+ Status execute_impl(FunctionContext* /*context*/, Block& block, const ColumnNumbers& arguments,
+ uint32_t result, size_t input_rows_count) const override {
+ std::vector<ColumnPtr> columns(arguments.size());
+ DataTypes argument_types(arguments.size());
+ for (size_t i = 0; i != arguments.size(); ++i) {
+ columns[i] =
+ block.get_by_position(arguments[i]).column->convert_to_full_column_if_const();
+ argument_types[i] = block.get_by_position(arguments[i]).type;
+ }
+ const DataTypePtr value_type = find_value_type(argument_types);
+ // An untyped call is rejected while planning get_return_type(); the plan always carries T.
+ DCHECK(value_type != nullptr);
+ const PrimitiveType primitive_type = value_type->get_primitive_type();
+
+ auto result_column = block.get_by_position(result).type->create_column();
+ auto& result_struct = assert_cast<ColumnStruct&>(*result_column);
+ auto& out_values = assert_cast<ColumnArray&>(result_struct.get_column(0));
+ auto& out_pair_nullable = assert_cast<ColumnNullable&>(out_values.get_data());
+ auto& out_pair = assert_cast<ColumnStruct&>(out_pair_nullable.get_nested_column());
+ auto& out_value_column = out_pair.get_column(0);
+ auto& out_count_column = assert_cast<ColumnInt64&>(out_pair.get_column(1));
+ auto& out_bound_column = assert_cast<ColumnNullable&>(result_struct.get_column(1));
+
+ for (size_t row = 0; row != input_rows_count; ++row) {
+ const int64_t k = read_int64(*columns[8], row, "k");
+ if (UNLIKELY(k < 1 || k > MAX_TOPK_K)) {
+ return Status::InvalidArgument("{} requires 1 <= k <= {}, but got {}", name,
+ MAX_TOPK_K, k);
+ }
+ const int64_t n = read_int64(*columns[6], row, "n");
+ const int64_t dn = read_int64(*columns[7], row, "dn");
+ const int64_t new_n = checked_add(n, dn, name);
+
+ const TopKArrayView s_view = make_array_view(*columns[0], row);
+ const TopKArrayView i_view = make_array_view(*columns[2], row);
+ const TopKArrayView d_view = make_array_view(*columns[4], row);
+ const TopKBoundView b_view = make_bound_view(*columns[1], row);
+ const TopKBoundView bi_view = make_bound_view(*columns[3], row);
+ const TopKBoundView bd_view = make_bound_view(*columns[5], row);
+ const TopKBoundView effective_bound =
+ effective_bound_of(b_view, bi_view, bd_view, primitive_type);
+
+ size_t positive = 0;
+ RETURN_IF_ERROR(append_trusted_values(s_view, i_view, d_view, effective_bound, k,
+ primitive_type, out_value_column,
+ out_count_column, out_pair_nullable, &positive));
+ out_values.get_offsets().push_back(out_pair.size());
+
+ // bound: NULL while the state is complete, the k-th kept value only when values were
+ // actually dropped (pos > k), or the effective bound otherwise.
+ // b IS NULL means the pre-image covers every distinct value; a NULL b_i/b_d means
+ // that side of the delta was not truncated.
+ // pos == k is NOT truncation: every known value fit, so keep bE instead of shrinking
+ // the trusted range to max(S').
+ const bool sides_complete = b_view.is_null && bi_view.is_null && bd_view.is_null;
+ if (new_n == 0 || (sides_complete && std::cmp_less_equal(positive, k))) {
+ out_bound_column.insert_default();
+ } else if (std::cmp_greater(positive, k)) {
+ out_bound_column.get_nested_column().insert_from(out_value_column,
+ out_pair.size() - 1);
+ out_bound_column.get_null_map_data().push_back(0);
+ } else {
+ DCHECK(!effective_bound.is_null);
+ out_bound_column.get_nested_column().insert_from(*effective_bound.value,
+ effective_bound.row);
+ out_bound_column.get_null_map_data().push_back(0);
+ }
+ }
+
+ block.replace_by_position(result, std::move(result_column));
+ return Status::OK();
+ }
+
+private:
+ /// bE: the best (smallest for MIN, largest for MAX) of the non-NULL bounds, or nullptr when
+ /// every side is complete/untruncated.
+ static TopKBoundView effective_bound_of(const TopKBoundView& b_view,
+ const TopKBoundView& bi_view,
+ const TopKBoundView& bd_view, PrimitiveType type) {
+ TopKBoundView bound;
+ for (const TopKBoundView* view : {&b_view, &bi_view, &bd_view}) {
+ if (view->is_null) {
+ continue;
+ }
+ if (bound.is_null) {
+ bound = *view;
+ continue;
+ }
+ int cmp = compare_values(type, *view->value, view->row, *bound.value, bound.row);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ bound = *view;
+ }
+ }
+ return bound;
+ }
+
+ /// A list may have a nullable element type, but its entries, values and counts must not be
+ /// NULL: the lists are top-K states, where NULL is neither counted nor representable. Reading
+ /// a NULL payload would emit the column's default value as a real entry, so reject it.
+ static Status check_entry(const TopKArrayView& view, size_t index) {
+ if ((view.null_map != nullptr && (*view.null_map)[index] != 0) ||
+ (view.value_null_map != nullptr && (*view.value_null_map)[index] != 0) ||
+ (view.count_null_map != nullptr && (*view.count_null_map)[index] != 0)) {
+ return Status::InvalidArgument(
+ "{}: the state lists must not contain NULL entries, values or counts", name);
+ }
+ return Status::OK();
+ }
+
+ /// Finds the best head (smallest for MIN, largest for MAX) among the three lists; `head` is
+ /// set to nullptr when all of them are exhausted.
+ static Status best_head(const TopKArrayView& s_view, size_t s_index,
+ const TopKArrayView& i_view, size_t i_index,
+ const TopKArrayView& d_view, size_t d_index, PrimitiveType type,
+ const IColumn** head, size_t* head_row) {
+ *head = nullptr;
+ const TopKArrayView* views[] = {&s_view, &i_view, &d_view};
+ const size_t indexes[] = {s_index, i_index, d_index};
+ for (size_t v = 0; v != 3; ++v) {
+ if (indexes[v] == views[v]->end) {
+ continue;
+ }
+ RETURN_IF_ERROR(check_entry(*views[v], indexes[v]));
+ if (*head == nullptr) {
+ *head = views[v]->values;
+ *head_row = indexes[v];
+ continue;
+ }
+ int cmp = compare_values(type, *views[v]->values, indexes[v], **head, *head_row);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ *head = views[v]->values;
+ *head_row = indexes[v];
+ }
+ }
+ return Status::OK();
+ }
+
+ /// Consumes one entry equal to the current head value, returning its count.
+ static Status consume_equal(const TopKArrayView& view, size_t& index, const IColumn& head,
+ size_t head_row, PrimitiveType type, int64_t* count) {
+ if (index == view.end) {
+ return Status::OK();
+ }
+ RETURN_IF_ERROR(check_entry(view, index));
+ if (compare_values(type, *view.values, index, head, head_row) != 0) {
+ return Status::OK();
+ }
+ *count = view.counts->get_element(index);
+ ++index;
+ return Status::OK();
+ }
+
+ /// Merges the trusted part (v <= bE for MIN) of S, I and D into the output list, keeping the
+ /// first k values with a positive post count S + I - D. `positive` receives the number of
+ /// positive candidates, capped at k + 1: once more than k are found the scan stops, which is
+ /// enough to decide the bound.
+ Status append_trusted_values(const TopKArrayView& s_view, const TopKArrayView& i_view,
+ const TopKArrayView& d_view, const TopKBoundView& effective_bound,
+ int64_t k, PrimitiveType type, IColumn& out_value_column,
+ ColumnInt64& out_count_column, ColumnNullable& out_pair_nullable,
+ size_t* positive) const {
+ size_t s_index = s_view.begin;
+ size_t i_index = i_view.begin;
+ size_t d_index = d_view.begin;
+ size_t kept = 0;
+ while (true) {
+ const IColumn* head = nullptr;
+ size_t head_row = 0;
+ RETURN_IF_ERROR(best_head(s_view, s_index, i_view, i_index, d_view, d_index, type,
+ &head, &head_row));
+ if (head == nullptr) {
+ break;
+ }
+ if (!effective_bound.is_null) {
+ int cmp = compare_values(type, *head, head_row, *effective_bound.value,
+ effective_bound.row);
+ if (IsMin ? cmp > 0 : cmp < 0) {
+ // Everything left is outside the range whose counts both sides know exactly.
+ break;
+ }
+ }
+
+ int64_t count_s = 0;
+ int64_t count_i = 0;
+ int64_t count_d = 0;
+ RETURN_IF_ERROR(consume_equal(s_view, s_index, *head, head_row, type, &count_s));
+ RETURN_IF_ERROR(consume_equal(i_view, i_index, *head, head_row, type, &count_i));
+ RETURN_IF_ERROR(consume_equal(d_view, d_index, *head, head_row, type, &count_d));
+
+ int64_t post = checked_add(count_s, count_i, name);
Review Comment:
Fixed in e7a3dbfae6. `S + I - D` is evaluated in `__int128` and range-checked before narrowing to int64, so the INT64_MAX cancel pair returns the original state instead of overflowing. `order_qt_merge_min_cancel_at_int64_max` pins it.
##########
be/src/exprs/aggregate/aggregate_function_topk_count.h:
##########
@@ -0,0 +1,428 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/string_buffer.hpp"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// Upper bound of k, shared with FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/**
+ * State shared by topk_min_count and topk_max_count.
+ *
+ * The entries are held as two parallel structures: a column with the distinct values (sorted
+ * ascending for MIN / descending for MAX, at most k entries) and their non-negative int64 counts.
+ * Only values with count > 0 are kept. `bound` is a one-row column with the extreme value seen
+ * when k == 0, or the k-th value once the list was truncated when k >= 1; `has_bound` is the
+ * truncation flag, and for k >= 1 `bound IS NULL` means "the list holds every distinct value".
+ */
+struct AggregateTopKCountData {
+ const DataTypePtr value_type;
+ bool has_k = false;
+ int64_t k = 0;
+ bool has_bound = false;
+ MutableColumnPtr bound_column;
+ MutableColumnPtr value_column;
+ std::vector<int64_t> counts;
+
+ explicit AggregateTopKCountData(const DataTypes& argument_types_)
+ : value_type(argument_types_[0]),
+ bound_column(value_type->create_column()),
+ value_column(value_type->create_column()) {}
+
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * topk_min_count(x, k) / topk_max_count(x, k).
+ *
+ * Result: STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T NULL>. NULL inputs are
+ * ignored (the nullable-argument wrapper never calls add for them). `values` is ordered by value
+ * (ascending for MIN), holds at most k distinct values, and `bound` is the truncation point: the
+ * k-th value when more than k distinct values were seen, otherwise NULL.
+ *
+ * With k = 0 no list is built: `values` is empty and `bound` is the extreme value seen on this
+ * side (NULL when there was no non-NULL input), which lets the append-only IVM path compute
+ * MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopKCount final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopKCount<T, IsMin>, true>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using Base = IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopKCount<T, IsMin>, true>;
+
+ explicit AggregateFunctionTopKCount(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "topk_min_count" : "topk_max_count"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(this->argument_types[0]);
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is a constant expression, but only the first row carries it: the state remembers it so
+ // that merge can check both sides used the same capacity. A state without has_k is empty.
+ check_k(state, *columns[1], row_num);
+ add_value(state, *columns[0], row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw a non-NULL row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else {
+ // Both sides are partial states of the same constant k.
+ DCHECK_EQ(state.k, other.k);
+ }
+ if (state.k == 0) {
+ merge_bound(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ buf.write_binary(state.has_bound);
+ if (state.has_bound) {
+ write_column(buf, *state.bound_column);
+ }
+ auto size = static_cast<int64_t>(state.counts.size());
+ buf.write_binary(size);
+ write_column(buf, *state.value_column);
+ buf.write(reinterpret_cast<const char*>(state.counts.data()),
+ state.counts.size() * sizeof(int64_t));
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ buf.read_binary(state.has_bound);
+ if (state.has_bound) {
+ state.bound_column = read_column(buf);
+ } else {
+ state.bound_column = state.value_type->create_column();
+ }
+ int64_t size = 0;
+ buf.read_binary(size);
+ state.value_column = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(state.value_column->size()), size);
+ state.counts.resize(size);
+ buf.read(reinterpret_cast<char*>(state.counts.data()), size * sizeof(int64_t));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_struct = assert_cast<ColumnStruct&>(to);
+ auto& values_column = assert_cast<ColumnArray&>(result_struct.get_column(0));
+ // The array element is nullable in the planned type (ARRAY<STRUCT<...>> defaults to
+ // contains_null = true), but a produced entry never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(values_column.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = pair_column.get_column(0);
+ auto& count_column = assert_cast<ColumnInt64&>(pair_column.get_column(1));
+ for (size_t i = 0; i != state.counts.size(); ++i) {
+ value_column.insert_from(*state.value_column, i);
+ count_column.get_data().push_back(state.counts[i]);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ values_column.get_offsets().push_back(pair_column.size());
+
+ auto& bound_column = assert_cast<ColumnNullable&>(result_struct.get_column(1));
+ if (state.has_bound) {
+ bound_column.get_nested_column().insert_from(*state.bound_column, 0);
+ bound_column.get_null_map_data().push_back(0);
+ } else {
+ bound_column.get_nested_column().insert_default();
+ bound_column.get_null_map_data().push_back(1);
+ }
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {value_type, std::make_shared<DataTypeInt64>()},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ auto values_type = std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ return std::make_shared<DataTypeStruct>(DataTypes {values_type, make_nullable(value_type)},
+ Strings {"values", "bound"});
+ }
+
+private:
+ static void check_k(AggregateTopKCountData& state, const IColumn& count_column,
+ size_t row_num) {
+ DCHECK(!count_column.is_nullable());
+ int64_t value = assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(count_column)
+ .get_element(row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0 || value > MAX_TOPK_K)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of topk_min_count and topk_max_count must be a constant "
+ "between 0 and {}, but got {}",
+ MAX_TOPK_K, value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ /// Ordering of the values. FLOAT/DOUBLE use CompareFloat so that NaN sorts greatest and is
+ /// distinct from every number, which keeps the list mergeable and the top-K deterministic;
+ /// every other type uses the column's compare_at, i.e. exactly the MIN/MAX comparison.
+ static int compare_at(const IColumn& lhs, size_t lhs_row, const IColumn& rhs, size_t rhs_row) {
+ if constexpr (is_float_or_double(T)) {
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ const auto& left =
+ assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(lhs).get_data();
+ const auto& right =
+ assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(rhs).get_data();
+ return CompareFloat(left[lhs_row], right[rhs_row]);
+ } else {
+ return lhs.compare_at(lhs_row, rhs_row, rhs, 1);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "topk_min_count/topk_max_count value count overflow");
+ }
+ return result;
+ }
+
+ static void set_bound_from_value(AggregateTopKCountData& state, size_t index) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(*state.value_column, index);
+ state.has_bound = true;
+ }
+
+ /// Inserts (value, count) at `pos`, keeping the value column sorted by value.
+ static void insert_entry(AggregateTopKCountData& state, size_t pos, const IColumn& column,
+ size_t row_num, int64_t count) {
+ auto new_values = state.value_column->clone_empty();
+ new_values->reserve(state.value_column->size() + 1);
+ for (size_t i = 0; i < pos; ++i) {
+ new_values->insert_from(*state.value_column, i);
+ }
+ new_values->insert_from(column, row_num);
+ for (size_t i = pos; i < state.value_column->size(); ++i) {
+ new_values->insert_from(*state.value_column, i);
+ }
+ state.value_column = std::move(new_values);
+ state.counts.insert(state.counts.begin() + pos, count);
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(AggregateTopKCountData& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (!state.has_bound) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(column, row_num);
+ state.has_bound = true;
+ return;
+ }
+ int cmp = compare_at(column, row_num, *state.bound_column, 0);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(column, row_num);
+ }
+ return;
+ }
+
+ // The list is ascending for MIN and descending for MAX, so the insertion point is where
+ // the stored values stop being better than the new one.
+ size_t left = 0;
+ size_t right = state.value_column->size();
+ while (left < right) {
+ size_t mid = left + (right - left) / 2;
+ int cmp = compare_at(*state.value_column, mid, column, row_num);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ left = mid + 1;
+ } else {
+ right = mid;
+ }
+ }
+ if (left < state.value_column->size() &&
+ compare_at(*state.value_column, left, column, row_num) == 0) {
+ state.counts[left] = checked_add(state.counts[left], 1);
+ return;
+ }
+
+ insert_entry(state, left, column, row_num, 1);
Review Comment:
Fixed as suggested in e7a3dbfae6: `insert_entry()` appends in place when `pos == state.value_column->size()`, before the clone/copy path, and `state.counts.push_back(count)` keeps the parallel vector in sync. Together with the full-list rejection in `add_value()`, sorted input at large k only appends while the list fills. I did not add the benchmark; `TopKCountTest.*` and the regression suite still pass.
##########
be/src/exprs/function/function_topk_merge.cpp:
##########
@@ -0,0 +1,482 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <glog/logging.h>
+
+#include <cstddef>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_numbers.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/types.h"
+#include "exprs/function/function.h"
+#include "exprs/function/simple_function_factory.h"
+
+namespace doris {
+class FunctionContext;
+} // namespace doris
+
+namespace doris {
+
+namespace {
+
+/// Upper bound of k. Keep in sync with the aggregate's MAX_TOPK_K
+/// (aggregate_function_topk_count.h) and FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/// One row of an ARRAY<STRUCT<value, count>> argument. A NULL array is an empty view.
+struct TopKArrayView {
+ const IColumn* values = nullptr;
+ const ColumnInt64* counts = nullptr;
+ /// NULL bitmap of the array element (the pair struct).
+ const NullMap* null_map = nullptr;
+ /// NULL bitmap of the `value` field; NULL values are not part of the contract.
+ const NullMap* value_null_map = nullptr;
+ /// NULL bitmap of the `count` field; NULL counts are not part of the contract.
+ const NullMap* count_null_map = nullptr;
+ size_t begin = 0;
+ size_t end = 0;
+};
+
+TopKArrayView make_array_view(const IColumn& column, size_t row) {
+ TopKArrayView view;
+ const IColumn* array_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ array_column = &nullable->get_nested_column();
+ }
+ const auto& array = assert_cast<const ColumnArray&>(*array_column);
+ // offsets[i] is the cumulative size after row i, so row's range is [offsets[row-1], offsets[row]).
+ // offset_at(0) reads offsets[-1], which PaddedPODArray keeps as 0.
+ view.begin = array.offset_at(static_cast<ssize_t>(row));
+ view.end = array.offset_at(static_cast<ssize_t>(row) + 1);
+
+ const IColumn* pair_column = &array.get_data();
+ if (const auto* element_nullable = check_and_get_column<ColumnNullable>(pair_column)) {
+ view.null_map = &element_nullable->get_null_map_data();
+ pair_column = &element_nullable->get_nested_column();
+ }
+ const auto& pairs = assert_cast<const ColumnStruct&>(*pair_column);
+ view.values = &pairs.get_column(0);
+ // A cast may widen the `value` field to a nullable column; the entries themselves are never
+ // NULL, so only the nested column is used.
+ if (const auto* value_nullable = check_and_get_column<ColumnNullable>(view.values)) {
+ view.value_null_map = &value_nullable->get_null_map_data();
+ view.values = &value_nullable->get_nested_column();
+ }
+ // The same holds for `count`: a nested cast may have made the field nullable.
+ const auto* count_column = &pairs.get_column(1);
+ if (const auto* count_nullable = check_and_get_column<ColumnNullable>(count_column)) {
+ view.count_null_map = &count_nullable->get_null_map_data();
+ count_column = &count_nullable->get_nested_column();
+ }
+ view.counts = &assert_cast<const ColumnInt64&>(*count_column);
+ return view;
+}
+
+/// One scalar bound argument: NULL means the corresponding side was not truncated. `row` is the
+/// index of the current input row inside `value`: bounds are ordinary per-row columns, so a block
+/// can carry a different bound in every row.
+struct TopKBoundView {
+ const IColumn* value = nullptr;
+ size_t row = 0;
+ bool is_null = true;
+};
+
+TopKBoundView make_bound_view(const IColumn& column, size_t row) {
+ TopKBoundView view;
+ const IColumn* value_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ view.value = value_column;
+ view.row = row;
+ view.is_null = false;
+ return view;
+}
+
+int64_t read_int64(const IColumn& column, size_t row, const char* argument_name) {
+ const IColumn* nested = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ // The contract declares these arguments as non-NULL BIGINT.
+ DORIS_CHECK(!nullable->is_null_at(row)) << argument_name << " must not be NULL";
Review Comment:
Fixed in e7a3dbfae6: `read_int64` returns a `Status` and reports `INVALID_ARGUMENT` for a NULL `n`, `dn` or `k`. b0ae93fe5 adds the coverage requested here: NULL for all three counters on both `topk_merge_min` and `topk_merge_max`, plus a table whose nullable BIGINT columns feed the counters, so the nullable-column input path is exercised as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6352893419
<details>
<summary>TPC-DS: <b>Total hot run time: 150471 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d, data reload: false
query5 4307 588 445 445
query6 445 204 185 185
query7 4798 439 233 233
query8 322 163 154 154
query9 8798 3140 3144 3140
query10 474 317 251 251
query11 5984 3865 3581 3581
query12 148 92 86 86
query13 1239 432 318 318
query14 6159 3941 3639 3639
query14_1 3460 3418 3390 3390
query15 205 192 183 183
query16 948 430 410 410
query17 858 667 529 529
query18 2424 425 312 312
query19 201 169 133 133
query20 84 83 83 83
query21 208 127 108 108
query22 12950 12920 12782 12782
query23 13457 12843 12414 12414
query23_1 12551 12479 12496 12479
query24 7373 805 459 459
query24_1 461 480 463 463
query25 506 389 335 335
query26 1260 256 135 135
query27 2749 432 265 265
query28 4602 1957 1964 1957
query29 1541 586 440 440
query30 301 213 195 195
query31 995 941 830 830
query32 142 91 90 90
query33 514 286 235 235
query34 887 843 498 498
query35 729 754 659 659
query36 806 820 729 729
query37 123 96 89 89
query38 1782 1707 1658 1658
query39 789 802 740 740
query39_1 744 722 741 722
query40 215 111 98 98
query41 70 60 62 60
query42 97 85 84 84
query43 343 372 315 315
query44 1311 674 687 674
query45 177 171 164 164
query46 819 975 561 561
query47 2958 2717 2900 2717
query48 289 306 216 216
query49 576 402 299 299
query50 675 272 212 212
query51 10191 10534 10364 10364
query52 82 87 72 72
query53 190 205 150 150
query54 252 198 177 177
query55 78 70 62 62
query56 274 211 247 211
query57 1658 1613 1545 1545
query58 289 266 262 262
query59 2095 2176 1910 1910
query60 289 252 227 227
query61 140 140 144 140
query62 385 324 280 280
query63 191 153 158 153
query64 2544 942 755 755
query65 3315 3266 3255 3255
query66 1744 434 298 298
query67 20295 20116 20030 20030
query68 3008 1006 611 611
query69 388 285 239 239
query70 834 774 761 761
query71 282 244 223 223
query72 2743 2572 2254 2254
query73 504 522 292 292
query74 4609 4512 4329 4329
query75 2309 2284 1919 1919
query76 2309 1010 607 607
query77 349 370 271 271
query78 9827 9592 9125 9125
query79 877 851 531 531
query80 502 400 337 337
query81 529 329 282 282
query82 205 131 109 109
query83 189 191 180 180
query84 287 128 104 104
query85 763 422 387 387
query86 256 236 221 221
query87 1954 1970 1799 1799
query88 3594 2691 2685 2685
query89 313 283 242 242
query90 2087 167 164 164
query91 161 153 127 127
query92 98 89 87 87
query93 972 976 580 580
query94 439 300 290 290
query95 574 324 364 324
query96 663 479 234 234
query97 2443 2457 2355 2355
query98 159 151 146 146
query99 747 768 644 644
Total cold run time: 232679 ms
Total hot run time: 150471 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6491686445
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6465616248
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4400645331
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,786 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ // The payload is proportional to the slot count, which the caller already compares.
+ size_t live_bytes() const { return 0; }
+ size_t dead_bytes() const { return 0; }
+ void on_evict(size_t /*slot*/) {}
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ size_t offset = 0;
+ size_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ /// PaddedPODArray, not PODArray: comparisons go through
+ /// memcmp_small_allow_overflow15, which loads a full 16-byte vector even for a one-byte
+ /// string, the same tail allowance ColumnString's chars buffer provides.
+ PaddedPODArray<char, 32> bytes;
+ DorisVector<Span> spans;
+ // Sum of the live slots' payload and of the payload evicted since the last compaction. The
+ // slot counters alone cannot bound the memory: replacing 1024 1 MiB strings with short ones
+ // leaves dead == live by slot count while ~1 GiB of discarded bytes stay in the arena.
+ size_t live_payload_bytes = 0;
+ size_t dead_payload_bytes = 0;
+
+ size_t size() const { return spans.size(); }
+ size_t live_bytes() const { return live_payload_bytes; }
+ size_t dead_bytes() const { return dead_payload_bytes; }
+ void reserve(size_t n) { spans.reserve(n); }
+ void clear() {
+ bytes.clear();
+ spans.clear();
+ live_payload_bytes = 0;
+ dead_payload_bytes = 0;
+ }
+ StringRef at(size_t slot) const {
+ const Span& span = spans[slot];
+ return StringRef(bytes.data() + span.offset, span.length);
+ }
+ void append(StringRef value) {
+ spans.push_back({bytes.size(), value.size});
+ live_payload_bytes += value.size;
+ if (value.size != 0) {
+ bytes.insert(value.data, value.data + value.size);
+ }
+ }
+ /// Called when `slot` is evicted: its payload stays in the arena until the next compaction.
+ void on_evict(size_t slot) {
+ live_payload_bytes -= spans[slot].length;
+ dead_payload_bytes += spans[slot].length;
+ }
+ void swap(StringTopKValueStore& other) noexcept {
+ bytes.swap(other.bytes);
+ spans.swap(other.spans);
+ std::swap(live_payload_bytes, other.live_payload_bytes);
+ std::swap(dead_payload_bytes, other.dead_payload_bytes);
+ }
+};
+
+/**
+ * State shared by min_topk and max_topk, for one value type.
+ *
+ * The distinct values sit in slots (see the stores above), with one Slot record per value carrying
+ * the row count and the links of a treap ordered best-first (ascending values for MIN, descending
+ * for MAX). A row costs one O(log k) descent: it either finds its value and bumps the count or
+ * links a new slot at the leaf. A sorted vector would shift O(k) elements for the same insert.
+ *
+ * The treap stores explicit child links because, unlike a heap, its shape is not the complete tree
+ * that an array index can describe: rotations move entries, and the in-order sequence is the value
+ * order rather than the arrival order. A heap could evict and append in O(log k), but it cannot
+ * find an existing value without a hash index or a linear scan, and it cannot produce the sorted
+ * result without sorting. The ordered tree is the structure that keeps find, insert and eviction
+ * all at O(log k) with comparisons only.
+ *
+ * An entry that is evicted keeps its slot and value until the next compaction. The tree never
+ * points at a dead slot.
+ *
+ * With k == 0 the state holds at most the single extreme value and its row count; the caller
+ * derives truncation from the list length, so the state carries no separate bound.
+ */
+template <PrimitiveType T>
+struct AggregateTopKCountData {
+ using CppType = typename PrimitiveTypeTraits<T>::CppType;
+ static constexpr bool IS_STRING = is_string_type(T);
+ using ValueStore =
+ std::conditional_t<IS_STRING, StringTopKValueStore, FixedTopKValueStore<CppType>>;
+ using ValueRef = std::conditional_t<IS_STRING, StringRef, const CppType&>;
+
+ static constexpr int32_t NO_SLOT = -1;
+ // Row counts are never negative, so this cannot collide with a live entry.
+ static constexpr int64_t DEAD_COUNT = -1;
+
+ /// One slot's metadata; the value itself lives in `values` at the same index.
+ struct Slot {
+ int64_t count = 0;
+ int32_t left = NO_SLOT;
+ int32_t right = NO_SLOT;
+ uint32_t priority = 0;
+ };
+
+ bool has_k = false;
+ int64_t k = 0;
+ // Indexed by slot; an evicted slot stays as a dead entry until compaction.
+ ValueStore values;
+ DorisVector<Slot> slots;
+ int32_t root = NO_SLOT;
+ // The last live slot in the aggregate's order: the value an incoming row has to beat to enter
+ // a full list. Cached so that rejecting a worse value costs one comparison, not a descent.
+ int32_t worst = NO_SLOT;
+ int64_t live = 0;
+ // Slots evicted since the last compaction.
+ int64_t dead = 0;
+ uint64_t rng = 0x243F6A8885A308D3ULL;
+
+ AggregateTopKCountData() = default;
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * min_topk(x, k) / max_topk(x, k).
+ *
+ * Result: ARRAY<STRUCT<value T, count BIGINT>>. NULL inputs are ignored: add validates k for
+ * every row and then skips a NULL x, so the aggregate deliberately does not use the
+ * nullable-argument wrapper. The array is ordered by value (ascending for MIN), holds at most k
+ * distinct values with count > 0, and carries no truncation flag: the caller infers it from the
+ * length (a list of exactly k entries is treated as possibly truncated).
+ *
+ * With k = 0 the array holds at most the extreme value and its row count, which lets the
+ * append-only IVM path compute MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopK final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData<T>,
+ AggregateFunctionTopK<T, IsMin>, false>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using State = AggregateTopKCountData<T>;
+ using Base = IAggregateFunctionDataHelper<State, AggregateFunctionTopK<T, IsMin>, false>;
+ using Slot = typename State::Slot;
+ using ValueStore = typename State::ValueStore;
+ using ValueRef = typename State::ValueRef;
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ static constexpr bool IS_STRING = State::IS_STRING;
+ static constexpr int32_t NO_SLOT = State::NO_SLOT;
+
+ explicit AggregateFunctionTopK(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "min_topk" : "max_topk"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(remove_nullable(this->argument_types[0]));
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is read for every row, including rows where x is NULL: an evaluated count that is
+ // NULL or out of range must fail the query instead of being skipped like a NULL value.
+ check_k(state, *columns[1], row_num);
+ const IColumn* value_column = columns[0];
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(value_column)) {
+ if (nullable->is_null_at(row_num)) {
+ return;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ add_value(state, *value_column, row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw any row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else if (UNLIKELY(state.k != other.k)) {
+ // Ordinary partial/final aggregation always shares the same constant k, but the
+ // public _state/_merge/_union combinators do not encode k in the AggState type, so a
+ // UNION ALL of states built with different literal k can reach this merge.
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} states have incompatible counts: {} vs {}", get_name(), state.k,
+ other.k);
+ }
+ if (state.k == 0) {
+ merge_extreme(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ // The slots are in arrival order and may include dead ones, while the serialized form is
+ // the sorted (value, count) list the reader rebuilds the tree from.
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ auto sorted_values = remove_nullable(this->argument_types[0])->create_column();
+ sorted_values->reserve(order.size());
+ DorisVector<int64_t> sorted_counts;
+ sorted_counts.reserve(order.size());
+ for (int32_t slot : order) {
+ insert_value(*sorted_values, state.values.at(slot));
+ sorted_counts.push_back(state.slots[slot].count);
+ }
+ auto size = static_cast<int64_t>(sorted_counts.size());
+ buf.write_binary(size);
+ write_column(buf, *sorted_values);
+ if (size != 0) {
+ // An empty list keeps an empty vector, whose data() is null; passing it to memcpy is
+ // undefined even for a zero length.
+ buf.write(reinterpret_cast<const char*>(sorted_counts.data()),
+ sorted_counts.size() * sizeof(int64_t));
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ int64_t size = 0;
+ buf.read_binary(size);
+ auto values = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(values->size()), size);
+ DorisVector<int64_t> counts(static_cast<size_t>(size));
+ if (size != 0) {
+ buf.read(reinterpret_cast<char*>(counts.data()), size * sizeof(int64_t));
+ }
+ ValueStore value_store;
+ value_store.reserve(static_cast<size_t>(size));
+ for (int64_t row = 0; row != size; ++row) {
+ value_store.append(value_at(*values, static_cast<size_t>(row)));
+ }
+ assign_sorted(state, std::move(value_store), std::move(counts));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_array = assert_cast<ColumnArray&>(to);
+ // The array element and both fields are nullable in the planned type, but a produced entry
+ // never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(result_array.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = assert_cast<ColumnNullable&>(pair_column.get_column(0));
+ auto& count_column = assert_cast<ColumnNullable&>(pair_column.get_column(1));
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ for (int32_t slot : order) {
+ insert_value(value_column.get_nested_column(), state.values.at(slot));
+ value_column.get_null_map_data().push_back(0);
+ assert_cast<ColumnInt64&>(count_column.get_nested_column())
+ .get_data()
+ .push_back(state.slots[slot].count);
+ count_column.get_null_map_data().push_back(0);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ result_array.get_offsets().push_back(pair_column.size());
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ // Both fields are nullable in the planned type: a consumer reads a field through
+ // element_at, which is always nullable, and named_struct then records nullable fields.
+ // Produced entries never contain a NULL field.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(value_type),
+ make_nullable(std::make_shared<DataTypeInt64>())},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ return std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ }
+
+private:
+ /// The value of the row in `column`. The returned view is valid while the column is alive.
+ static ValueRef value_at(const IColumn& column, size_t row_num) {
+ if constexpr (IS_STRING) {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column).get_data_at(
+ row_num);
+ } else {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column)
+ .get_data()[row_num];
+ }
+ }
+
+ /// Exactly the comparison the value column would apply: CompareFloat for FLOAT/DOUBLE through
+ /// Compare::compare (the column uses the same helper), the same memcmp helper ColumnString
+ /// uses for strings, and the column's scale-independent ordering for every other type.
+ static int compare_values(ValueRef lhs, ValueRef rhs) {
+ if constexpr (IS_STRING) {
+ return memcmp_small_allow_overflow15(
+ reinterpret_cast<const uint8_t*>(lhs.data), lhs.size,
+ reinterpret_cast<const uint8_t*>(rhs.data), rhs.size);
+ } else {
+ return Compare::compare(lhs, rhs);
+ }
+ }
+
+ static void insert_value(IColumn& column, ValueRef value) {
+ if constexpr (IS_STRING) {
+ assert_cast<ColumnString&>(column).insert_data(value.data, value.size);
+ } else {
+ assert_cast<ColumnType&>(column).get_data().push_back(value);
+ }
+ }
+
+ /// True when `cmp` puts the left operand before the right one in the aggregate's order.
+ static bool comes_before(int cmp) { return IsMin ? cmp < 0 : cmp > 0; }
+
+ /// True when `cmp`, which compares a value with the worst retained one, puts it after it: the
+ /// value is worse and cannot enter the list.
+ static bool is_worse(int cmp) { return IsMin ? cmp > 0 : cmp < 0; }
+
+ /// splitmix64. The priorities only have to be independent enough to keep the treap balanced;
+ /// a per-state sequence keeps the shape reproducible for the same insertion order.
+ static uint32_t next_priority(State& state) {
+ uint64_t z = (state.rng += 0x9E3779B97F4A7C15ULL);
+ z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9ULL;
+ z = (z ^ (z >> 27)) * 0x94D049BB133111EBULL;
+ return static_cast<uint32_t>(z ^ (z >> 31));
+ }
+
+ static int compare_slots(const State& state, int32_t lhs, int32_t rhs) {
+ return compare_values(state.values.at(lhs), state.values.at(rhs));
+ }
+
+ /// Appends (value, count) as the next slot.
+ static int32_t append_slot(State& state, ValueRef value, int64_t count) {
+ auto slot = static_cast<int32_t>(state.slots.size());
Review Comment:
Fixed in 305a309b7fb: the slot index is guarded before narrowing. append_slot and the merged-state rebuild both reject a state above INT32_MAX + 1 slots with a query error (the int32 link range); k stays unlimited because the FE cannot see the state size. Reaching it needs ~2^31 retained distinct values, so it has no unit test.
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,786 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ // The payload is proportional to the slot count, which the caller already compares.
+ size_t live_bytes() const { return 0; }
+ size_t dead_bytes() const { return 0; }
+ void on_evict(size_t /*slot*/) {}
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ size_t offset = 0;
+ size_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ /// PaddedPODArray, not PODArray: comparisons go through
+ /// memcmp_small_allow_overflow15, which loads a full 16-byte vector even for a one-byte
+ /// string, the same tail allowance ColumnString's chars buffer provides.
+ PaddedPODArray<char, 32> bytes;
+ DorisVector<Span> spans;
+ // Sum of the live slots' payload and of the payload evicted since the last compaction. The
+ // slot counters alone cannot bound the memory: replacing 1024 1 MiB strings with short ones
+ // leaves dead == live by slot count while ~1 GiB of discarded bytes stay in the arena.
+ size_t live_payload_bytes = 0;
+ size_t dead_payload_bytes = 0;
+
+ size_t size() const { return spans.size(); }
+ size_t live_bytes() const { return live_payload_bytes; }
+ size_t dead_bytes() const { return dead_payload_bytes; }
+ void reserve(size_t n) { spans.reserve(n); }
+ void clear() {
+ bytes.clear();
+ spans.clear();
+ live_payload_bytes = 0;
+ dead_payload_bytes = 0;
+ }
+ StringRef at(size_t slot) const {
+ const Span& span = spans[slot];
+ return StringRef(bytes.data() + span.offset, span.length);
+ }
+ void append(StringRef value) {
+ spans.push_back({bytes.size(), value.size});
+ live_payload_bytes += value.size;
+ if (value.size != 0) {
+ bytes.insert(value.data, value.data + value.size);
+ }
+ }
+ /// Called when `slot` is evicted: its payload stays in the arena until the next compaction.
+ void on_evict(size_t slot) {
+ live_payload_bytes -= spans[slot].length;
+ dead_payload_bytes += spans[slot].length;
+ }
+ void swap(StringTopKValueStore& other) noexcept {
+ bytes.swap(other.bytes);
+ spans.swap(other.spans);
+ std::swap(live_payload_bytes, other.live_payload_bytes);
+ std::swap(dead_payload_bytes, other.dead_payload_bytes);
+ }
+};
+
+/**
+ * State shared by min_topk and max_topk, for one value type.
+ *
+ * The distinct values sit in slots (see the stores above), with one Slot record per value carrying
+ * the row count and the links of a treap ordered best-first (ascending values for MIN, descending
+ * for MAX). A row costs one O(log k) descent: it either finds its value and bumps the count or
+ * links a new slot at the leaf. A sorted vector would shift O(k) elements for the same insert.
+ *
+ * The treap stores explicit child links because, unlike a heap, its shape is not the complete tree
+ * that an array index can describe: rotations move entries, and the in-order sequence is the value
+ * order rather than the arrival order. A heap could evict and append in O(log k), but it cannot
+ * find an existing value without a hash index or a linear scan, and it cannot produce the sorted
+ * result without sorting. The ordered tree is the structure that keeps find, insert and eviction
+ * all at O(log k) with comparisons only.
+ *
+ * An entry that is evicted keeps its slot and value until the next compaction. The tree never
+ * points at a dead slot.
+ *
+ * With k == 0 the state holds at most the single extreme value and its row count; the caller
+ * derives truncation from the list length, so the state carries no separate bound.
+ */
+template <PrimitiveType T>
+struct AggregateTopKCountData {
+ using CppType = typename PrimitiveTypeTraits<T>::CppType;
+ static constexpr bool IS_STRING = is_string_type(T);
+ using ValueStore =
+ std::conditional_t<IS_STRING, StringTopKValueStore, FixedTopKValueStore<CppType>>;
+ using ValueRef = std::conditional_t<IS_STRING, StringRef, const CppType&>;
+
+ static constexpr int32_t NO_SLOT = -1;
+ // Row counts are never negative, so this cannot collide with a live entry.
+ static constexpr int64_t DEAD_COUNT = -1;
+
+ /// One slot's metadata; the value itself lives in `values` at the same index.
+ struct Slot {
+ int64_t count = 0;
+ int32_t left = NO_SLOT;
+ int32_t right = NO_SLOT;
+ uint32_t priority = 0;
+ };
+
+ bool has_k = false;
+ int64_t k = 0;
+ // Indexed by slot; an evicted slot stays as a dead entry until compaction.
+ ValueStore values;
+ DorisVector<Slot> slots;
+ int32_t root = NO_SLOT;
+ // The last live slot in the aggregate's order: the value an incoming row has to beat to enter
+ // a full list. Cached so that rejecting a worse value costs one comparison, not a descent.
+ int32_t worst = NO_SLOT;
+ int64_t live = 0;
+ // Slots evicted since the last compaction.
+ int64_t dead = 0;
+ uint64_t rng = 0x243F6A8885A308D3ULL;
+
+ AggregateTopKCountData() = default;
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * min_topk(x, k) / max_topk(x, k).
+ *
+ * Result: ARRAY<STRUCT<value T, count BIGINT>>. NULL inputs are ignored: add validates k for
+ * every row and then skips a NULL x, so the aggregate deliberately does not use the
+ * nullable-argument wrapper. The array is ordered by value (ascending for MIN), holds at most k
+ * distinct values with count > 0, and carries no truncation flag: the caller infers it from the
+ * length (a list of exactly k entries is treated as possibly truncated).
+ *
+ * With k = 0 the array holds at most the extreme value and its row count, which lets the
+ * append-only IVM path compute MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopK final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData<T>,
+ AggregateFunctionTopK<T, IsMin>, false>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using State = AggregateTopKCountData<T>;
+ using Base = IAggregateFunctionDataHelper<State, AggregateFunctionTopK<T, IsMin>, false>;
+ using Slot = typename State::Slot;
+ using ValueStore = typename State::ValueStore;
+ using ValueRef = typename State::ValueRef;
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ static constexpr bool IS_STRING = State::IS_STRING;
+ static constexpr int32_t NO_SLOT = State::NO_SLOT;
+
+ explicit AggregateFunctionTopK(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "min_topk" : "max_topk"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(remove_nullable(this->argument_types[0]));
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is read for every row, including rows where x is NULL: an evaluated count that is
+ // NULL or out of range must fail the query instead of being skipped like a NULL value.
+ check_k(state, *columns[1], row_num);
+ const IColumn* value_column = columns[0];
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(value_column)) {
+ if (nullable->is_null_at(row_num)) {
+ return;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ add_value(state, *value_column, row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw any row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else if (UNLIKELY(state.k != other.k)) {
+ // Ordinary partial/final aggregation always shares the same constant k, but the
+ // public _state/_merge/_union combinators do not encode k in the AggState type, so a
+ // UNION ALL of states built with different literal k can reach this merge.
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} states have incompatible counts: {} vs {}", get_name(), state.k,
+ other.k);
+ }
+ if (state.k == 0) {
+ merge_extreme(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ // The slots are in arrival order and may include dead ones, while the serialized form is
+ // the sorted (value, count) list the reader rebuilds the tree from.
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ auto sorted_values = remove_nullable(this->argument_types[0])->create_column();
+ sorted_values->reserve(order.size());
+ DorisVector<int64_t> sorted_counts;
+ sorted_counts.reserve(order.size());
+ for (int32_t slot : order) {
+ insert_value(*sorted_values, state.values.at(slot));
+ sorted_counts.push_back(state.slots[slot].count);
+ }
+ auto size = static_cast<int64_t>(sorted_counts.size());
+ buf.write_binary(size);
+ write_column(buf, *sorted_values);
+ if (size != 0) {
+ // An empty list keeps an empty vector, whose data() is null; passing it to memcpy is
+ // undefined even for a zero length.
+ buf.write(reinterpret_cast<const char*>(sorted_counts.data()),
+ sorted_counts.size() * sizeof(int64_t));
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ int64_t size = 0;
+ buf.read_binary(size);
+ auto values = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(values->size()), size);
+ DorisVector<int64_t> counts(static_cast<size_t>(size));
+ if (size != 0) {
+ buf.read(reinterpret_cast<char*>(counts.data()), size * sizeof(int64_t));
+ }
+ ValueStore value_store;
+ value_store.reserve(static_cast<size_t>(size));
+ for (int64_t row = 0; row != size; ++row) {
+ value_store.append(value_at(*values, static_cast<size_t>(row)));
+ }
+ assign_sorted(state, std::move(value_store), std::move(counts));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_array = assert_cast<ColumnArray&>(to);
+ // The array element and both fields are nullable in the planned type, but a produced entry
+ // never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(result_array.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = assert_cast<ColumnNullable&>(pair_column.get_column(0));
+ auto& count_column = assert_cast<ColumnNullable&>(pair_column.get_column(1));
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ for (int32_t slot : order) {
+ insert_value(value_column.get_nested_column(), state.values.at(slot));
+ value_column.get_null_map_data().push_back(0);
+ assert_cast<ColumnInt64&>(count_column.get_nested_column())
+ .get_data()
+ .push_back(state.slots[slot].count);
+ count_column.get_null_map_data().push_back(0);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ result_array.get_offsets().push_back(pair_column.size());
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ // Both fields are nullable in the planned type: a consumer reads a field through
+ // element_at, which is always nullable, and named_struct then records nullable fields.
+ // Produced entries never contain a NULL field.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(value_type),
+ make_nullable(std::make_shared<DataTypeInt64>())},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ return std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ }
+
+private:
+ /// The value of the row in `column`. The returned view is valid while the column is alive.
+ static ValueRef value_at(const IColumn& column, size_t row_num) {
+ if constexpr (IS_STRING) {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column).get_data_at(
+ row_num);
+ } else {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column)
+ .get_data()[row_num];
+ }
+ }
+
+ /// Exactly the comparison the value column would apply: CompareFloat for FLOAT/DOUBLE through
+ /// Compare::compare (the column uses the same helper), the same memcmp helper ColumnString
+ /// uses for strings, and the column's scale-independent ordering for every other type.
+ static int compare_values(ValueRef lhs, ValueRef rhs) {
+ if constexpr (IS_STRING) {
+ return memcmp_small_allow_overflow15(
+ reinterpret_cast<const uint8_t*>(lhs.data), lhs.size,
+ reinterpret_cast<const uint8_t*>(rhs.data), rhs.size);
+ } else {
+ return Compare::compare(lhs, rhs);
+ }
+ }
+
+ static void insert_value(IColumn& column, ValueRef value) {
+ if constexpr (IS_STRING) {
+ assert_cast<ColumnString&>(column).insert_data(value.data, value.size);
+ } else {
+ assert_cast<ColumnType&>(column).get_data().push_back(value);
+ }
+ }
+
+ /// True when `cmp` puts the left operand before the right one in the aggregate's order.
+ static bool comes_before(int cmp) { return IsMin ? cmp < 0 : cmp > 0; }
+
+ /// True when `cmp`, which compares a value with the worst retained one, puts it after it: the
+ /// value is worse and cannot enter the list.
+ static bool is_worse(int cmp) { return IsMin ? cmp > 0 : cmp < 0; }
+
+ /// splitmix64. The priorities only have to be independent enough to keep the treap balanced;
+ /// a per-state sequence keeps the shape reproducible for the same insertion order.
+ static uint32_t next_priority(State& state) {
Review Comment:
Fixed in 305a309b7fb: every state seeds its priority sequence from a process-random base (std::random_device read once) plus an atomic counter, then advances it with splitmix64. The sequence is no longer derivable from the input, so feeding the priorities as values cannot correlate the value order with the priority order. The randomized reference test still passes.
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,786 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ // The payload is proportional to the slot count, which the caller already compares.
+ size_t live_bytes() const { return 0; }
+ size_t dead_bytes() const { return 0; }
+ void on_evict(size_t /*slot*/) {}
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ size_t offset = 0;
+ size_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ /// PaddedPODArray, not PODArray: comparisons go through
+ /// memcmp_small_allow_overflow15, which loads a full 16-byte vector even for a one-byte
+ /// string, the same tail allowance ColumnString's chars buffer provides.
+ PaddedPODArray<char, 32> bytes;
+ DorisVector<Span> spans;
+ // Sum of the live slots' payload and of the payload evicted since the last compaction. The
+ // slot counters alone cannot bound the memory: replacing 1024 1 MiB strings with short ones
+ // leaves dead == live by slot count while ~1 GiB of discarded bytes stay in the arena.
+ size_t live_payload_bytes = 0;
+ size_t dead_payload_bytes = 0;
+
+ size_t size() const { return spans.size(); }
+ size_t live_bytes() const { return live_payload_bytes; }
+ size_t dead_bytes() const { return dead_payload_bytes; }
+ void reserve(size_t n) { spans.reserve(n); }
+ void clear() {
+ bytes.clear();
+ spans.clear();
+ live_payload_bytes = 0;
+ dead_payload_bytes = 0;
+ }
+ StringRef at(size_t slot) const {
+ const Span& span = spans[slot];
+ return StringRef(bytes.data() + span.offset, span.length);
+ }
+ void append(StringRef value) {
+ spans.push_back({bytes.size(), value.size});
+ live_payload_bytes += value.size;
+ if (value.size != 0) {
+ bytes.insert(value.data, value.data + value.size);
+ }
+ }
+ /// Called when `slot` is evicted: its payload stays in the arena until the next compaction.
+ void on_evict(size_t slot) {
+ live_payload_bytes -= spans[slot].length;
+ dead_payload_bytes += spans[slot].length;
+ }
+ void swap(StringTopKValueStore& other) noexcept {
+ bytes.swap(other.bytes);
+ spans.swap(other.spans);
+ std::swap(live_payload_bytes, other.live_payload_bytes);
+ std::swap(dead_payload_bytes, other.dead_payload_bytes);
+ }
+};
+
+/**
+ * State shared by min_topk and max_topk, for one value type.
+ *
+ * The distinct values sit in slots (see the stores above), with one Slot record per value carrying
+ * the row count and the links of a treap ordered best-first (ascending values for MIN, descending
+ * for MAX). A row costs one O(log k) descent: it either finds its value and bumps the count or
+ * links a new slot at the leaf. A sorted vector would shift O(k) elements for the same insert.
+ *
+ * The treap stores explicit child links because, unlike a heap, its shape is not the complete tree
+ * that an array index can describe: rotations move entries, and the in-order sequence is the value
+ * order rather than the arrival order. A heap could evict and append in O(log k), but it cannot
+ * find an existing value without a hash index or a linear scan, and it cannot produce the sorted
+ * result without sorting. The ordered tree is the structure that keeps find, insert and eviction
+ * all at O(log k) with comparisons only.
+ *
+ * An entry that is evicted keeps its slot and value until the next compaction. The tree never
+ * points at a dead slot.
+ *
+ * With k == 0 the state holds at most the single extreme value and its row count; the caller
+ * derives truncation from the list length, so the state carries no separate bound.
+ */
+template <PrimitiveType T>
+struct AggregateTopKCountData {
+ using CppType = typename PrimitiveTypeTraits<T>::CppType;
+ static constexpr bool IS_STRING = is_string_type(T);
+ using ValueStore =
+ std::conditional_t<IS_STRING, StringTopKValueStore, FixedTopKValueStore<CppType>>;
+ using ValueRef = std::conditional_t<IS_STRING, StringRef, const CppType&>;
+
+ static constexpr int32_t NO_SLOT = -1;
+ // Row counts are never negative, so this cannot collide with a live entry.
+ static constexpr int64_t DEAD_COUNT = -1;
+
+ /// One slot's metadata; the value itself lives in `values` at the same index.
+ struct Slot {
+ int64_t count = 0;
+ int32_t left = NO_SLOT;
+ int32_t right = NO_SLOT;
+ uint32_t priority = 0;
+ };
+
+ bool has_k = false;
+ int64_t k = 0;
+ // Indexed by slot; an evicted slot stays as a dead entry until compaction.
+ ValueStore values;
+ DorisVector<Slot> slots;
+ int32_t root = NO_SLOT;
+ // The last live slot in the aggregate's order: the value an incoming row has to beat to enter
+ // a full list. Cached so that rejecting a worse value costs one comparison, not a descent.
+ int32_t worst = NO_SLOT;
+ int64_t live = 0;
+ // Slots evicted since the last compaction.
+ int64_t dead = 0;
+ uint64_t rng = 0x243F6A8885A308D3ULL;
+
+ AggregateTopKCountData() = default;
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * min_topk(x, k) / max_topk(x, k).
+ *
+ * Result: ARRAY<STRUCT<value T, count BIGINT>>. NULL inputs are ignored: add validates k for
+ * every row and then skips a NULL x, so the aggregate deliberately does not use the
+ * nullable-argument wrapper. The array is ordered by value (ascending for MIN), holds at most k
+ * distinct values with count > 0, and carries no truncation flag: the caller infers it from the
+ * length (a list of exactly k entries is treated as possibly truncated).
+ *
+ * With k = 0 the array holds at most the extreme value and its row count, which lets the
+ * append-only IVM path compute MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopK final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData<T>,
+ AggregateFunctionTopK<T, IsMin>, false>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using State = AggregateTopKCountData<T>;
+ using Base = IAggregateFunctionDataHelper<State, AggregateFunctionTopK<T, IsMin>, false>;
+ using Slot = typename State::Slot;
+ using ValueStore = typename State::ValueStore;
+ using ValueRef = typename State::ValueRef;
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ static constexpr bool IS_STRING = State::IS_STRING;
+ static constexpr int32_t NO_SLOT = State::NO_SLOT;
+
+ explicit AggregateFunctionTopK(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "min_topk" : "max_topk"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(remove_nullable(this->argument_types[0]));
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is read for every row, including rows where x is NULL: an evaluated count that is
+ // NULL or out of range must fail the query instead of being skipped like a NULL value.
+ check_k(state, *columns[1], row_num);
+ const IColumn* value_column = columns[0];
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(value_column)) {
+ if (nullable->is_null_at(row_num)) {
+ return;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ add_value(state, *value_column, row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw any row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else if (UNLIKELY(state.k != other.k)) {
+ // Ordinary partial/final aggregation always shares the same constant k, but the
+ // public _state/_merge/_union combinators do not encode k in the AggState type, so a
+ // UNION ALL of states built with different literal k can reach this merge.
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} states have incompatible counts: {} vs {}", get_name(), state.k,
+ other.k);
+ }
+ if (state.k == 0) {
+ merge_extreme(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ // The slots are in arrival order and may include dead ones, while the serialized form is
+ // the sorted (value, count) list the reader rebuilds the tree from.
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ auto sorted_values = remove_nullable(this->argument_types[0])->create_column();
+ sorted_values->reserve(order.size());
+ DorisVector<int64_t> sorted_counts;
+ sorted_counts.reserve(order.size());
+ for (int32_t slot : order) {
+ insert_value(*sorted_values, state.values.at(slot));
+ sorted_counts.push_back(state.slots[slot].count);
+ }
+ auto size = static_cast<int64_t>(sorted_counts.size());
+ buf.write_binary(size);
+ write_column(buf, *sorted_values);
+ if (size != 0) {
+ // An empty list keeps an empty vector, whose data() is null; passing it to memcpy is
+ // undefined even for a zero length.
+ buf.write(reinterpret_cast<const char*>(sorted_counts.data()),
+ sorted_counts.size() * sizeof(int64_t));
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ int64_t size = 0;
+ buf.read_binary(size);
+ auto values = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(values->size()), size);
+ DorisVector<int64_t> counts(static_cast<size_t>(size));
+ if (size != 0) {
+ buf.read(reinterpret_cast<char*>(counts.data()), size * sizeof(int64_t));
+ }
+ ValueStore value_store;
+ value_store.reserve(static_cast<size_t>(size));
+ for (int64_t row = 0; row != size; ++row) {
+ value_store.append(value_at(*values, static_cast<size_t>(row)));
+ }
+ assign_sorted(state, std::move(value_store), std::move(counts));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_array = assert_cast<ColumnArray&>(to);
+ // The array element and both fields are nullable in the planned type, but a produced entry
+ // never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(result_array.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = assert_cast<ColumnNullable&>(pair_column.get_column(0));
+ auto& count_column = assert_cast<ColumnNullable&>(pair_column.get_column(1));
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ for (int32_t slot : order) {
+ insert_value(value_column.get_nested_column(), state.values.at(slot));
+ value_column.get_null_map_data().push_back(0);
+ assert_cast<ColumnInt64&>(count_column.get_nested_column())
+ .get_data()
+ .push_back(state.slots[slot].count);
+ count_column.get_null_map_data().push_back(0);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ result_array.get_offsets().push_back(pair_column.size());
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ // Both fields are nullable in the planned type: a consumer reads a field through
+ // element_at, which is always nullable, and named_struct then records nullable fields.
+ // Produced entries never contain a NULL field.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(value_type),
+ make_nullable(std::make_shared<DataTypeInt64>())},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ return std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ }
+
+private:
+ /// The value of the row in `column`. The returned view is valid while the column is alive.
+ static ValueRef value_at(const IColumn& column, size_t row_num) {
+ if constexpr (IS_STRING) {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column).get_data_at(
+ row_num);
+ } else {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column)
+ .get_data()[row_num];
+ }
+ }
+
+ /// Exactly the comparison the value column would apply: CompareFloat for FLOAT/DOUBLE through
+ /// Compare::compare (the column uses the same helper), the same memcmp helper ColumnString
+ /// uses for strings, and the column's scale-independent ordering for every other type.
+ static int compare_values(ValueRef lhs, ValueRef rhs) {
+ if constexpr (IS_STRING) {
+ return memcmp_small_allow_overflow15(
+ reinterpret_cast<const uint8_t*>(lhs.data), lhs.size,
+ reinterpret_cast<const uint8_t*>(rhs.data), rhs.size);
+ } else {
+ return Compare::compare(lhs, rhs);
+ }
+ }
+
+ static void insert_value(IColumn& column, ValueRef value) {
+ if constexpr (IS_STRING) {
+ assert_cast<ColumnString&>(column).insert_data(value.data, value.size);
+ } else {
+ assert_cast<ColumnType&>(column).get_data().push_back(value);
+ }
+ }
+
+ /// True when `cmp` puts the left operand before the right one in the aggregate's order.
+ static bool comes_before(int cmp) { return IsMin ? cmp < 0 : cmp > 0; }
+
+ /// True when `cmp`, which compares a value with the worst retained one, puts it after it: the
+ /// value is worse and cannot enter the list.
+ static bool is_worse(int cmp) { return IsMin ? cmp > 0 : cmp < 0; }
+
+ /// splitmix64. The priorities only have to be independent enough to keep the treap balanced;
+ /// a per-state sequence keeps the shape reproducible for the same insertion order.
+ static uint32_t next_priority(State& state) {
+ uint64_t z = (state.rng += 0x9E3779B97F4A7C15ULL);
+ z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9ULL;
+ z = (z ^ (z >> 27)) * 0x94D049BB133111EBULL;
+ return static_cast<uint32_t>(z ^ (z >> 31));
+ }
+
+ static int compare_slots(const State& state, int32_t lhs, int32_t rhs) {
+ return compare_values(state.values.at(lhs), state.values.at(rhs));
+ }
+
+ /// Appends (value, count) as the next slot.
+ static int32_t append_slot(State& state, ValueRef value, int64_t count) {
+ auto slot = static_cast<int32_t>(state.slots.size());
+ DCHECK_LT(slot, std::numeric_limits<int32_t>::max());
+ state.values.append(value);
+ state.slots.push_back(Slot {count, State::NO_SLOT, State::NO_SLOT, next_priority(state)});
+ ++state.live;
+ return slot;
+ }
+
+ static int32_t append_slot(State& state, const IColumn& column, size_t row_num, int64_t count) {
+ return append_slot(state, value_at(column, row_num), count);
+ }
+
+ /// Rotates the subtree rooted at `node` right: the left child becomes the root and `node` its
+ /// right child. The in-order sequence is unchanged, so the treap stays ordered.
+ static int32_t rotate_right(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].left;
+ state.slots[node].left = state.slots[pivot].right;
+ state.slots[pivot].right = node;
+ return pivot;
+ }
+
+ static int32_t rotate_left(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].right;
+ state.slots[node].right = state.slots[pivot].left;
+ state.slots[pivot].left = node;
+ return pivot;
+ }
+
+ /// One descent that performs the whole per-row update: an equal value bumps its count and
+ /// reports no insertion; a missing value is linked as a new slot at the leaf and rotated up
+ /// while the recursion unwinds. `inserted_slot` receives the new slot, NO_SLOT on a hit.
+ static int32_t add_or_increment(State& state, int32_t node, const IColumn& column,
+ size_t row_num, int32_t* inserted_slot) {
+ if (node == State::NO_SLOT) {
+ *inserted_slot = append_slot(state, column, row_num, 1);
+ return *inserted_slot;
+ }
+ int cmp = compare_values(state.values.at(node), value_at(column, row_num));
+ if (cmp == 0) {
+ state.slots[node].count = checked_add(state.slots[node].count, 1);
+ return node;
+ }
+ if (comes_before(cmp)) {
+ // The stored value is the better one, so the input belongs in the right subtree.
+ state.slots[node].right = add_or_increment(state, state.slots[node].right, column,
+ row_num, inserted_slot);
+ if (state.slots[state.slots[node].right].priority > state.slots[node].priority) {
+ return rotate_left(state, node);
+ }
+ } else {
+ state.slots[node].left =
+ add_or_increment(state, state.slots[node].left, column, row_num, inserted_slot);
+ if (state.slots[state.slots[node].left].priority > state.slots[node].priority) {
+ return rotate_right(state, node);
+ }
+ }
+ return node;
+ }
+
+ /// Removes the last entry in the aggregate's order (the largest value for MIN) and returns the
+ /// new subtree root. That entry has no right child, so its left subtree takes its place; the
+ /// child's priority is below the parent's, so no rotation is needed.
+ static int32_t erase_worst(State& state, int32_t node, int32_t* removed) {
+ if (state.slots[node].right == State::NO_SLOT) {
+ *removed = node;
+ return state.slots[node].left;
+ }
+ state.slots[node].right = erase_worst(state, state.slots[node].right, removed);
+ return node;
+ }
+
+ static void refresh_worst(State& state) {
+ int32_t node = state.root;
+ while (node != State::NO_SLOT && state.slots[node].right != State::NO_SLOT) {
+ node = state.slots[node].right;
+ }
+ state.worst = node;
+ }
+
+ /// The first live slot in the aggregate's order, i.e. the best value the state holds.
+ static int32_t best_slot(const State& state) {
+ int32_t node = state.root;
+ while (node != State::NO_SLOT && state.slots[node].left != State::NO_SLOT) {
+ node = state.slots[node].left;
+ }
+ return node;
+ }
+
+ /// Slots in the aggregate's order, i.e. the order of the result array.
+ static void collect_in_order(const State& state, DorisVector<int32_t>& slots) {
+ slots.reserve(static_cast<size_t>(state.live));
+ DorisVector<int32_t> stack;
+ stack.reserve(64);
+ int32_t node = state.root;
+ while (node != State::NO_SLOT || !stack.empty()) {
+ while (node != State::NO_SLOT) {
+ stack.push_back(node);
+ node = state.slots[node].left;
+ }
+ node = stack.back();
+ stack.pop_back();
+ // Only live slots are reachable from the root; an evicted slot has no row count.
+ DCHECK_GE(state.slots[node].count, 0);
+ slots.push_back(node);
+ node = state.slots[node].right;
+ }
+ DCHECK_EQ(static_cast<int64_t>(slots.size()), state.live);
+ }
+
+ /// Rebuilds the tree from `values`, already in the aggregate's order, and the matching counts.
+ /// The treap is built in O(size) as a Cartesian tree over the fresh priorities.
+ static void assign_sorted(State& state, ValueStore values, DorisVector<int64_t> counts) {
+ const size_t size = counts.size();
+ DCHECK_EQ(values.size(), size);
+ state.values.swap(values);
+ state.slots.assign(size, Slot {});
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].count = counts[slot];
+ }
+ state.live = static_cast<int64_t>(size);
+ state.dead = 0;
+ DorisVector<int32_t> stack;
+ stack.reserve(size);
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].priority = next_priority(state);
+ int32_t last = State::NO_SLOT;
+ while (!stack.empty() &&
+ state.slots[stack.back()].priority < state.slots[slot].priority) {
+ last = stack.back();
+ stack.pop_back();
+ }
+ state.slots[slot].left = last;
+ if (!stack.empty()) {
+ state.slots[stack.back()].right = static_cast<int32_t>(slot);
+ }
+ stack.push_back(static_cast<int32_t>(slot));
+ }
+ state.root = stack.empty() ? State::NO_SLOT : stack.front();
+ refresh_worst(state);
+ }
+
+ /// Reclaims the slots evicted since the last compaction. The O(k) rebuild is charged to the
+ /// k/2 evictions that produced the dead slots, so it stays O(1) amortized per row.
+ static void compact(State& state) {
+ DorisVector<int32_t> slots;
+ collect_in_order(state, slots);
+ ValueStore values;
+ values.reserve(slots.size());
+ DorisVector<int64_t> counts;
+ counts.reserve(slots.size());
+ for (int32_t slot : slots) {
+ values.append(state.values.at(slot));
+ counts.push_back(state.slots[slot].count);
+ }
+ assign_sorted(state, std::move(values), std::move(counts));
+ }
+
+ static void check_k(State& state, const IColumn& count_column, size_t row_num) {
+ const IColumn* nested = &count_column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&count_column)) {
+ if (UNLIKELY(nullable->is_null_at(row_num))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-null "
+ "constant");
+ }
+ nested = &nullable->get_nested_column();
+ }
+ int64_t value =
+ assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(*nested).get_element(
+ row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-negative "
+ "constant, but got {}",
+ value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "min_topk/max_topk value count overflow");
+ }
+ return result;
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(State& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (state.live == 0) {
+ state.root = append_slot(state, column, row_num, 1);
+ state.worst = state.root;
+ return;
+ }
+ int cmp = compare_values(value_at(column, row_num), state.values.at(0));
+ if (cmp == 0) {
+ state.slots[0].count = checked_add(state.slots[0].count, 1);
+ } else if (comes_before(cmp)) {
+ // The new value is better: replace the tracked extreme. The slot and the
+ // single-node tree stay in place, only the stored value changes.
+ state.values.clear();
+ state.values.append(value_at(column, row_num));
+ state.slots[0].count = 1;
+ }
+ return;
+ }
+
+ // Dead slots are reclaimed before they can outnumber the live ones, and evicted string
+ // payload before it dwarfs the live one; see compact().
+ if (state.dead > state.live || state.values.dead_bytes() > state.values.live_bytes()) {
+ compact(state);
+ }
+
+ if (state.live >= state.k) {
+ // A full list rejects or absorbs most rows: comparing with the eviction candidate
+ // costs one comparison, while the tree descent would cost log2(k).
+ int cmp = compare_values(value_at(column, row_num), state.values.at(state.worst));
+ if (cmp == 0) {
+ state.slots[state.worst].count = checked_add(state.slots[state.worst].count, 1);
+ return;
+ }
+ if (is_worse(cmp)) {
+ return;
+ }
+ }
+
+ int32_t inserted = State::NO_SLOT;
+ state.root = add_or_increment(state, state.root, column, row_num, &inserted);
+ if (inserted == State::NO_SLOT) {
+ return;
+ }
+ if (state.live <= state.k) {
+ // Not over capacity: the new entry competes with the cached worst for the eviction
+ // candidate role.
+ if (state.worst == State::NO_SLOT ||
+ is_worse(compare_slots(state, inserted, state.worst))) {
+ state.worst = inserted;
+ }
+ return;
+ }
+ // The list was full and the value is new. It beat the cached worst above, so it cannot be
+ // the worst itself; the old worst is still the rightmost entry and leaves the list.
+ int32_t removed = State::NO_SLOT;
+ state.root = erase_worst(state, state.root, &removed);
+ DCHECK_EQ(removed, state.worst);
+ state.values.on_evict(static_cast<size_t>(removed));
+ state.slots[removed].count = State::DEAD_COUNT;
+ --state.live;
+ ++state.dead;
+ refresh_worst(state);
+ }
+
+ /// k == 0: the state holds at most the extreme value and its count.
+ static void merge_extreme(State& state, const State& other) {
+ if (other.live == 0) {
+ return;
+ }
+ if (state.live == 0) {
+ state.root = append_slot(state, other.values.at(0), other.slots[0].count);
+ state.worst = state.root;
+ return;
+ }
+ int cmp = compare_values(state.values.at(0), other.values.at(0));
+ if (cmp == 0) {
+ state.slots[0].count = checked_add(state.slots[0].count, other.slots[0].count);
+ } else if (is_worse(cmp)) {
+ state.values.clear();
+ state.values.append(other.values.at(0));
+ state.slots[0].count = other.slots[0].count;
+ }
+ }
+
+ /// Merges two ordered lists, summing the counts of equal values and keeping the k best.
+ ///
+ /// The merge stops as soon as k entries are produced. Both inputs are best-first and hold
+ /// distinct values, and equal values have already been combined, so every remaining value is
+ /// worse than all retained ones: it can neither enter the result nor overflow it. Stopping
+ /// here also avoids materializing a tail that would be truncated away.
+ static void merge_values(State& state, const State& other) {
+ if (other.live == 0) {
+ // The other partial state saw no non-NULL row.
+ return;
+ }
+ if (state.live >= state.k) {
+ // A full state already holds the k best values. If every value of the other state is
+ // worse than the worst one retained here, the merge cannot change the result, so skip
+ // the collect/rebuild. An equal value would raise a count, hence the strict test.
+ // `_state`/`_merge`/`_union` merges one singleton per row, and this turns each one
+ // into a single comparison once the list is full.
+ int cmp =
+ compare_values(other.values.at(best_slot(other)), state.values.at(state.worst));
+ if (is_worse(cmp)) {
+ return;
+ }
+ }
+ DorisVector<int32_t> lhs_slots;
+ DorisVector<int32_t> rhs_slots;
+ collect_in_order(state, lhs_slots);
+ collect_in_order(other, rhs_slots);
+ const auto limit = static_cast<size_t>(state.k);
+ ValueStore merged_values;
+ merged_values.reserve(std::min(limit, lhs_slots.size() + rhs_slots.size()));
+ DorisVector<int64_t> merged_counts;
+ merged_counts.reserve(std::min(limit, lhs_slots.size() + rhs_slots.size()));
+ size_t lhs_index = 0;
+ size_t rhs_index = 0;
+ while (merged_counts.size() < limit &&
+ (lhs_index < lhs_slots.size() || rhs_index < rhs_slots.size())) {
+ if (rhs_index == rhs_slots.size()) {
+ merged_values.append(state.values.at(lhs_slots[lhs_index]));
+ merged_counts.push_back(state.slots[lhs_slots[lhs_index]].count);
+ ++lhs_index;
+ } else if (lhs_index == lhs_slots.size()) {
+ merged_values.append(other.values.at(rhs_slots[rhs_index]));
+ merged_counts.push_back(other.slots[rhs_slots[rhs_index]].count);
+ ++rhs_index;
+ } else {
+ ValueRef lhs_value = state.values.at(lhs_slots[lhs_index]);
+ ValueRef rhs_value = other.values.at(rhs_slots[rhs_index]);
+ int cmp = compare_values(lhs_value, rhs_value);
+ // Both lists are ordered best-first (ascending for MIN, descending for MAX).
+ const bool lhs_is_better = comes_before(cmp);
+ if (lhs_is_better) {
+ merged_values.append(lhs_value);
+ merged_counts.push_back(state.slots[lhs_slots[lhs_index]].count);
+ ++lhs_index;
+ } else if (cmp != 0) {
+ // Not better and not equal: the right-hand value is better.
+ merged_values.append(rhs_value);
+ merged_counts.push_back(other.slots[rhs_slots[rhs_index]].count);
+ ++rhs_index;
+ } else {
+ merged_values.append(lhs_value);
+ merged_counts.push_back(checked_add(state.slots[lhs_slots[lhs_index]].count,
+ other.slots[rhs_slots[rhs_index]].count));
+ ++lhs_index;
+ ++rhs_index;
+ }
+ }
+ }
+ assign_sorted(state, std::move(merged_values), std::move(merged_counts));
Review Comment:
Fixed in 305a309b7fb: when a full state merges a RHS whose best value equals its worst, merge_values adds the RHS count to the worst slot and returns; the RHS's remaining values are distinct and worse, so they cannot enter. MergeStringStates covers the count-only case and the ordinary merge when the RHS best is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6470645470
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6350980885
# FE UT Coverage Report
Increment line coverage ` 28.07% (48/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6345393425
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4350949277
##########
be/src/exprs/lambda_function/varray_sorted_merge_function.cpp:
##########
@@ -0,0 +1,320 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+#include <map>
+#include <memory>
+#include <set>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "exprs/lambda_function/lambda_execution_context.h"
+#include "exprs/lambda_function/lambda_function.h"
+#include "exprs/lambda_function/lambda_function_factory.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vlambda_function_expr.h"
+
+namespace doris {
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc)
+ *
+ * Merges two arrays that are already sorted by the elements' natural order. The merge lambda
+ * receives the two current candidates and returns the combined element; returning NULL means the
+ * two elements must not be combined, so both are kept. Each input key occurs at most once per
+ * array, so a key appears at most twice across the two arrays and the lambda is called at most
+ * once per key.
+ */
+class ArraySortedMergeFunction : public LambdaFunction {
+ ENABLE_FACTORY_CREATOR(ArraySortedMergeFunction);
+
+public:
+ ~ArraySortedMergeFunction() override = default;
+
+ static constexpr auto name = "array_sorted_merge";
+
+ static LambdaFunctionPtr create() { return std::make_shared<ArraySortedMergeFunction>(); }
+
+ std::string get_name() const override { return name; }
+
+ /// Rejects a nested lambda that captures one of this lambda's arguments: the position-based
+ /// frame below does not publish them into nested lambdas, so such a capture would resolve
+ /// against the wrong column. Nested lambdas may still use their own arguments (shadowing).
+ static Status validate_nested_captures(const VExprSPtr& expr,
+ const std::vector<std::string>& argument_names,
+ bool inside_nested_lambda,
+ std::vector<std::string> shadowed_names) {
+ if (expr->is_column_ref()) {
+ const std::string& column_name = expr->expr_name();
+ if (std::ranges::find(shadowed_names, column_name) != shadowed_names.end()) {
+ return Status::OK();
+ }
+ if (inside_nested_lambda &&
+ std::ranges::find(argument_names, column_name) != argument_names.end()) {
+ return Status::NotSupported(
+ "array_sorted_merge merge lambda argument '{}' cannot be captured by a "
+ "nested lambda",
+ column_name);
+ }
+ return Status::OK();
+ }
+ if (expr->node_type() == TExprNodeType::LAMBDA_FUNCTION_EXPR) {
+ const auto* lambda = assert_cast<const VLambdaFunctionExpr*>(expr.get());
+ if (lambda->has_argument_names()) {
+ for (const auto& argument_name : lambda->argument_names()) {
+ shadowed_names.emplace_back(argument_name);
+ }
+ }
+ return validate_nested_captures(lambda->get_child(0), argument_names, true,
+ std::move(shadowed_names));
+ }
+ for (const auto& child : expr->children()) {
+ RETURN_IF_ERROR(validate_nested_captures(child, argument_names, inside_nested_lambda,
+ shadowed_names));
+ }
+ return Status::OK();
+ }
+
+ Status prepare(RuntimeState* state, const VExprSPtrs& children) override {
+ RETURN_IF_ERROR(LambdaFunction::prepare(state, children));
+ DCHECK_EQ(children.size(), 4);
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ if (!lambda_expr->has_argument_names()) {
+ return Status::OK();
+ }
+ // A nested lambda that captures one of this lambda's arguments would need a binding
+ // that does not exist in the position-based frame below; reject it during prepare so
+ // the query fails instead of resolving the argument against the wrong column.
+ return validate_nested_captures(children[0]->get_child(0), lambda_expr->argument_names(),
+ false, {});
+ }
+
+ Status execute(VExprContext* context, const Block* block, const Selector* expr_selector,
+ size_t count, ColumnPtr& result_column, const DataTypePtr& result_type,
+ const VExprSPtrs& children) const override {
+ DCHECK_EQ(children.size(), 4);
+
+ const ColumnArray* arrays[2] = {nullptr, nullptr};
+ const IColumn* data[2] = {nullptr, nullptr};
+ DataTypePtr nested_types[2];
+ ColumnPtr holders[2];
+ const ColumnUInt8::Container* nullmaps[2] = {nullptr, nullptr};
+ for (int i = 0; i < 2; ++i) {
+ RETURN_IF_ERROR(children[i + 1]->execute_column(context, block, expr_selector, count,
+ holders[i]));
+ holders[i] = holders[i]->convert_to_full_column_if_const();
+ DataTypePtr type = children[i + 1]->execute_type(block);
+ const IColumn* array_column = holders[i].get();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(array_column)) {
+ nullmaps[i] = &nullable->get_null_map_column().get_data();
+ array_column = &nullable->get_nested_column();
+ type = assert_cast<const DataTypeNullable&>(*type).get_nested_type();
+ }
+ arrays[i] = assert_cast<const ColumnArray*>(array_column);
+ data[i] = arrays[i]->get_data_ptr().get();
+ nested_types[i] = assert_cast<const DataTypeArray&>(*type).get_nested_type();
+ }
+
+ ColumnPtr flag_holder;
+ RETURN_IF_ERROR(
+ children[3]->execute_column(context, block, expr_selector, count, flag_holder));
+ flag_holder = flag_holder->convert_to_full_column_if_const();
+ const IColumn* flag_column = flag_holder.get();
+ const ColumnUInt8::Container* flag_nullmap = nullptr;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(flag_column)) {
+ flag_nullmap = &nullable->get_null_map_column().get_data();
+ flag_column = &nullable->get_nested_column();
+ }
+ const auto& flags = assert_cast<const ColumnUInt8&>(*flag_column).get_data();
+
+ const size_t rows = arrays[0]->size();
+ auto result_nested = nested_types[0]->create_column();
+ auto result_offsets = ColumnArray::ColumnOffsets::create();
+ auto& offsets_data = result_offsets->get_data();
+ offsets_data.reserve(rows);
+ auto result_nullmap = ColumnUInt8::create(rows, 0);
+ auto& result_nullmap_data = result_nullmap->get_data();
+
+ // The lambda may capture enclosing columns (for example a table column compared against
+ // the candidates). Materialize every captured input at its original position in the
+ // one-row lambda block, so slot refs read the current row; the two merge candidates
+ // take the positions after the last captured column and are bound by name.
+ std::set<int> captured_ids;
+ children[0]->collect_slot_column_ids(captured_ids);
+ if (!captured_ids.empty() && block == nullptr) {
+ return Status::InternalError(
+ "array_sorted_merge lambda references outer columns but has no input block");
+ }
+ std::map<int, ColumnPtr> captured_columns;
+ for (int id : captured_ids) {
+ if (id < 0 || static_cast<size_t>(id) >= block->columns()) {
+ return Status::InternalError(
+ "array_sorted_merge lambda input column id {} is outside input block", id);
+ }
+ captured_columns[id] =
+ block->get_by_position(id).column->convert_to_full_column_if_const();
+ }
+ const int lambda_argument_base = captured_ids.empty() ? 0 : *captured_ids.rbegin() + 1;
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ const std::vector<std::string>* argument_names =
+ lambda_expr->has_argument_names() ? &lambda_expr->argument_names() : nullptr;
+ Block lambda_block;
+ for (int i = 0; i < lambda_argument_base; ++i) {
+ auto captured = captured_columns.find(i);
+ if (captured != captured_columns.end()) {
+ lambda_block.insert(ColumnWithTypeAndName(captured->second->clone_empty(),
+ block->get_by_position(i).type,
+ block->get_by_position(i).name));
+ } else {
+ lambda_block.insert(ColumnWithTypeAndName(
+ ColumnUInt8::create(1, 0), std::make_shared<DataTypeUInt8>(), "temp"));
+ }
+ }
+ for (int i = 0; i < 2; ++i) {
+ const std::string argument_name =
+ argument_names != nullptr && i < argument_names->size()
+ ? (*argument_names)[i]
+ : "merge_arg_" + std::to_string(i);
+ lambda_block.insert(ColumnWithTypeAndName(nested_types[i]->create_column(),
+ nested_types[i], argument_name));
+ }
+ // The lambda arguments are bound by name (the column refs carry the argument names),
+ // so the body reads these two one-row columns.
+ IColumn* argument_columns[2] = {
+ lambda_block.get_by_position(lambda_argument_base).column->assert_mutable().get(),
+ lambda_block.get_by_position(lambda_argument_base + 1)
+ .column->assert_mutable()
+ .get()};
+
+ LambdaExecutionContext::Frame lambda_frame;
+ lambda_frame.bind_by_name = false;
+ lambda_frame.parent_bindings_visible = false;
+ LambdaExecutionContext::FrameGuard lambda_frame_guard(context->lambda_execution_context(),
+ std::move(lambda_frame));
+
+ const auto& offsets0 = arrays[0]->get_offsets();
+ const auto& offsets1 = arrays[1]->get_offsets();
+ const int lambda_result_base = static_cast<int>(lambda_block.columns());
+
+ // The merged element may come from a non-nullable lambda body (for example COALESCE),
+ // while array elements are always nullable; normalize before inserting.
+ auto append_element = [&](const IColumn& source, size_t source_row) {
+ if (auto* nullable_target = check_and_get_column<ColumnNullable>(result_nested.get())) {
+ if (const auto* nullable_source = check_and_get_column<ColumnNullable>(&source)) {
+ nullable_target->insert_from(*nullable_source, source_row);
+ } else {
+ nullable_target->get_nested_column().insert_from(source, source_row);
+ nullable_target->get_null_map_data().push_back(0);
+ }
+ } else {
+ result_nested->insert_from(source, source_row);
+ }
+ };
+
+ for (size_t row = 0; row < rows; ++row) {
+ const bool row_is_null = (nullmaps[0] != nullptr && (*nullmaps[0])[row] != 0) ||
+ (nullmaps[1] != nullptr && (*nullmaps[1])[row] != 0);
+ if (row_is_null) {
+ result_nullmap_data[row] = 1;
+ offsets_data.push_back(result_nested->size());
+ continue;
+ }
+ if (flag_nullmap != nullptr && (*flag_nullmap)[row] != 0) {
+ return Status::InvalidArgument(
+ "array_sorted_merge requires a non-null ascending flag");
+ }
+ const bool ascending = flags[row] != 0;
+ for (const auto& [id, column] : captured_columns) {
+ lambda_block.get_by_position(id).column = column->cut(row, 1);
+ }
+
+ size_t left = row == 0 ? 0 : offsets0[row - 1];
+ const size_t left_end = offsets0[row];
+ size_t right = row == 0 ? 0 : offsets1[row - 1];
+ const size_t right_end = offsets1[row];
+
+ while (left < left_end || right < right_end) {
+ if (left == left_end) {
+ append_element(*data[1], right++);
+ continue;
+ }
+ if (right == right_end) {
+ append_element(*data[0], left++);
+ continue;
+ }
+
+ argument_columns[0]->clear();
+ argument_columns[0]->insert_from(*data[0], left);
+ argument_columns[1]->clear();
+ argument_columns[1]->insert_from(*data[1], right);
+
+ int lambda_res_id = lambda_result_base;
+ RETURN_IF_ERROR(children[0]->execute(context, &lambda_block, &lambda_res_id));
+ ColumnPtr merged = lambda_block.get_by_position(lambda_res_id)
+ .column->convert_to_full_column_if_const();
+ const IColumn* merged_column = merged.get();
+ bool merged_is_null = false;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(merged_column)) {
+ merged_is_null = nullable->is_null_at(0);
+ }
+ if (!merged_is_null) {
+ append_element(*merged_column, 0);
+ ++left;
+ ++right;
+ } else {
+ const int cmp = data[0]->compare_at(left, right, *data[1], 1);
Review Comment:
[P2] Match the NULL ordering used by the array sort functions
`array_sorted_merge((a,b)->IF(a=b,a,NULL), array_sort(CAST([NULL,1] AS ARRAY<INT>)), CAST([2] AS ARRAY<INT>), true)` receives naturally sorted inputs `[NULL,1]` and `[2]`, but returns `[2,NULL,1]`. `FunctionArraySort::Less` uses `compare_at(...,-1)`, whereas this call uses `+1`: `ColumnNullable::compare_at(NULL,2,+1)` chooses 2 before NULL, then the exhausted-right branch appends NULL and 1. Descending inputs from `array_reverse_sort` have the symmetric mismatch. Use the same comparison semantics as the array sort functions and cover NULL elements in both directions.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArraySortedMerge.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.scalar;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.ArrayItemReference;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.ImplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.BooleanType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.LambdaType;
+import org.apache.doris.nereids.types.coercion.FollowToAnyDataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc): merge two arrays that are already sorted.
+ *
+ * <pre>
+ * array_sorted_merge((a, b) -> ..., arr1, arr2, true)
+ * </pre>
+ *
+ * <p>The inputs must already be sorted by the elements' natural order in the direction given by
+ * {@code is_asc}; the function only merges them. The merge lambda receives the two current
+ * candidates as {@code (a, b)} and returns the combined element; returning NULL means the two
+ * elements must not be combined, so both are kept. Each key occurs at most once per input, so a
+ * key appears at most twice across the two arrays and the lambda is called at most once per key.
+ *
+ * <p>A NULL input array makes the result NULL, like the other higher-order array functions.
+ */
+public class ArraySortedMerge extends ScalarFunction
+ implements ImplicitlyCastableSignature, PropagateNullable {
Review Comment:
[P2] Avoid folding a NULL ascending flag into a NULL result
For `array_sorted_merge((a,b)->IF(a=b,a,NULL), [1], [2], CAST(NULL AS BOOLEAN))`, the flag passes BOOLEAN validation. However, `FoldConstantRuleOnFE.preProcess` sees this `PropagateNullable` marker and replaces the entire bound function with a typed NULL array as soon as the cast folds, even though the lambda is not a literal. With the same NULL flag supplied by a column, BE instead returns `INVALID_ARGUMENT` (requires a non-null ascending flag). This makes argument validation depend on constant folding. Use a nullable contract without generic NULL propagation, or validate the flag before this rewrite, and cover both literal and column NULL flags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4355468827
##########
be/src/exprs/lambda_function/varray_sorted_merge_function.cpp:
##########
@@ -0,0 +1,323 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+#include <map>
+#include <memory>
+#include <set>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "exprs/lambda_function/lambda_execution_context.h"
+#include "exprs/lambda_function/lambda_function.h"
+#include "exprs/lambda_function/lambda_function_factory.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vlambda_function_expr.h"
+
+namespace doris {
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc)
+ *
+ * Merges two arrays that are already sorted by the elements' natural order. The merge lambda
+ * receives the two current candidates and returns the combined element; returning NULL means the
+ * two elements must not be combined, so both are kept. Each input key occurs at most once per
+ * array, so a key appears at most twice across the two arrays and the lambda is called at most
+ * once per key.
+ */
+class ArraySortedMergeFunction : public LambdaFunction {
+ ENABLE_FACTORY_CREATOR(ArraySortedMergeFunction);
+
+public:
+ ~ArraySortedMergeFunction() override = default;
+
+ static constexpr auto name = "array_sorted_merge";
+
+ static LambdaFunctionPtr create() { return std::make_shared<ArraySortedMergeFunction>(); }
+
+ std::string get_name() const override { return name; }
+
+ /// Rejects a nested lambda that captures one of this lambda's arguments: the position-based
+ /// frame below does not publish them into nested lambdas, so such a capture would resolve
+ /// against the wrong column. Nested lambdas may still use their own arguments (shadowing).
+ static Status validate_nested_captures(const VExprSPtr& expr,
+ const std::vector<std::string>& argument_names,
+ bool inside_nested_lambda,
+ std::vector<std::string> shadowed_names) {
+ if (expr->is_column_ref()) {
+ const std::string& column_name = expr->expr_name();
+ if (std::ranges::find(shadowed_names, column_name) != shadowed_names.end()) {
+ return Status::OK();
+ }
+ if (inside_nested_lambda &&
+ std::ranges::find(argument_names, column_name) != argument_names.end()) {
+ return Status::NotSupported(
+ "array_sorted_merge merge lambda argument '{}' cannot be captured by a "
+ "nested lambda",
+ column_name);
+ }
+ return Status::OK();
+ }
+ if (expr->node_type() == TExprNodeType::LAMBDA_FUNCTION_EXPR) {
+ const auto* lambda = assert_cast<const VLambdaFunctionExpr*>(expr.get());
+ if (lambda->has_argument_names()) {
+ for (const auto& argument_name : lambda->argument_names()) {
+ shadowed_names.emplace_back(argument_name);
+ }
+ }
+ return validate_nested_captures(lambda->get_child(0), argument_names, true,
+ std::move(shadowed_names));
+ }
+ for (const auto& child : expr->children()) {
+ RETURN_IF_ERROR(validate_nested_captures(child, argument_names, inside_nested_lambda,
+ shadowed_names));
+ }
+ return Status::OK();
+ }
+
+ Status prepare(RuntimeState* state, const VExprSPtrs& children) override {
+ RETURN_IF_ERROR(LambdaFunction::prepare(state, children));
+ DCHECK_EQ(children.size(), 4);
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ if (!lambda_expr->has_argument_names()) {
+ return Status::OK();
+ }
+ // A nested lambda that captures one of this lambda's arguments would need a binding
+ // that does not exist in the position-based frame below; reject it during prepare so
+ // the query fails instead of resolving the argument against the wrong column.
+ return validate_nested_captures(children[0]->get_child(0), lambda_expr->argument_names(),
+ false, {});
+ }
+
+ Status execute(VExprContext* context, const Block* block, const Selector* expr_selector,
+ size_t count, ColumnPtr& result_column, const DataTypePtr& result_type,
+ const VExprSPtrs& children) const override {
+ DCHECK_EQ(children.size(), 4);
+
+ const ColumnArray* arrays[2] = {nullptr, nullptr};
+ const IColumn* data[2] = {nullptr, nullptr};
+ DataTypePtr nested_types[2];
+ ColumnPtr holders[2];
+ const ColumnUInt8::Container* nullmaps[2] = {nullptr, nullptr};
+ for (int i = 0; i < 2; ++i) {
+ RETURN_IF_ERROR(children[i + 1]->execute_column(context, block, expr_selector, count,
+ holders[i]));
+ holders[i] = holders[i]->convert_to_full_column_if_const();
+ DataTypePtr type = children[i + 1]->execute_type(block);
+ const IColumn* array_column = holders[i].get();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(array_column)) {
+ nullmaps[i] = &nullable->get_null_map_column().get_data();
+ array_column = &nullable->get_nested_column();
+ type = assert_cast<const DataTypeNullable&>(*type).get_nested_type();
+ }
+ arrays[i] = assert_cast<const ColumnArray*>(array_column);
+ data[i] = arrays[i]->get_data_ptr().get();
+ nested_types[i] = assert_cast<const DataTypeArray&>(*type).get_nested_type();
+ }
+
+ ColumnPtr flag_holder;
+ RETURN_IF_ERROR(
+ children[3]->execute_column(context, block, expr_selector, count, flag_holder));
+ flag_holder = flag_holder->convert_to_full_column_if_const();
+ const IColumn* flag_column = flag_holder.get();
+ const ColumnUInt8::Container* flag_nullmap = nullptr;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(flag_column)) {
+ flag_nullmap = &nullable->get_null_map_column().get_data();
+ flag_column = &nullable->get_nested_column();
+ }
+ const auto& flags = assert_cast<const ColumnUInt8&>(*flag_column).get_data();
+
+ const size_t rows = arrays[0]->size();
+ auto result_nested = nested_types[0]->create_column();
+ auto result_offsets = ColumnArray::ColumnOffsets::create();
+ auto& offsets_data = result_offsets->get_data();
+ offsets_data.reserve(rows);
+ auto result_nullmap = ColumnUInt8::create(rows, 0);
+ auto& result_nullmap_data = result_nullmap->get_data();
+
+ // The lambda may capture enclosing columns (for example a table column compared against
+ // the candidates). Materialize every captured input at its original position in the
+ // one-row lambda block, so slot refs read the current row; the two merge candidates
+ // take the positions after the last captured column and are bound by name.
+ std::set<int> captured_ids;
+ children[0]->collect_slot_column_ids(captured_ids);
+ if (!captured_ids.empty() && block == nullptr) {
+ return Status::InternalError(
+ "array_sorted_merge lambda references outer columns but has no input block");
+ }
+ std::map<int, ColumnPtr> captured_columns;
+ for (int id : captured_ids) {
+ if (id < 0 || static_cast<size_t>(id) >= block->columns()) {
+ return Status::InternalError(
+ "array_sorted_merge lambda input column id {} is outside input block", id);
+ }
+ captured_columns[id] =
+ block->get_by_position(id).column->convert_to_full_column_if_const();
+ }
+ const int lambda_argument_base = captured_ids.empty() ? 0 : *captured_ids.rbegin() + 1;
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ const std::vector<std::string>* argument_names =
+ lambda_expr->has_argument_names() ? &lambda_expr->argument_names() : nullptr;
+ Block lambda_block;
+ for (int i = 0; i < lambda_argument_base; ++i) {
+ auto captured = captured_columns.find(i);
+ if (captured != captured_columns.end()) {
+ lambda_block.insert(ColumnWithTypeAndName(captured->second->clone_empty(),
+ block->get_by_position(i).type,
+ block->get_by_position(i).name));
+ } else {
+ lambda_block.insert(ColumnWithTypeAndName(
+ ColumnUInt8::create(1, 0), std::make_shared<DataTypeUInt8>(), "temp"));
+ }
+ }
+ for (int i = 0; i < 2; ++i) {
+ const std::string argument_name =
+ argument_names != nullptr && i < argument_names->size()
+ ? (*argument_names)[i]
+ : "merge_arg_" + std::to_string(i);
+ lambda_block.insert(ColumnWithTypeAndName(nested_types[i]->create_column(),
+ nested_types[i], argument_name));
+ }
+ // The lambda arguments are bound by name (the column refs carry the argument names),
+ // so the body reads these two one-row columns.
+ IColumn* argument_columns[2] = {
+ lambda_block.get_by_position(lambda_argument_base).column->assert_mutable().get(),
+ lambda_block.get_by_position(lambda_argument_base + 1)
+ .column->assert_mutable()
+ .get()};
+
+ LambdaExecutionContext::Frame lambda_frame;
+ lambda_frame.bind_by_name = false;
+ lambda_frame.parent_bindings_visible = false;
+ LambdaExecutionContext::FrameGuard lambda_frame_guard(context->lambda_execution_context(),
+ std::move(lambda_frame));
+
+ const auto& offsets0 = arrays[0]->get_offsets();
+ const auto& offsets1 = arrays[1]->get_offsets();
+ const int lambda_result_base = static_cast<int>(lambda_block.columns());
+
+ // The merged element may come from a non-nullable lambda body (for example COALESCE),
+ // while array elements are always nullable; normalize before inserting.
+ auto append_element = [&](const IColumn& source, size_t source_row) {
+ if (auto* nullable_target = check_and_get_column<ColumnNullable>(result_nested.get())) {
+ if (const auto* nullable_source = check_and_get_column<ColumnNullable>(&source)) {
+ nullable_target->insert_from(*nullable_source, source_row);
+ } else {
+ nullable_target->get_nested_column().insert_from(source, source_row);
+ nullable_target->get_null_map_data().push_back(0);
+ }
+ } else {
+ result_nested->insert_from(source, source_row);
+ }
+ };
+
+ for (size_t row = 0; row < rows; ++row) {
+ const bool row_is_null = (nullmaps[0] != nullptr && (*nullmaps[0])[row] != 0) ||
+ (nullmaps[1] != nullptr && (*nullmaps[1])[row] != 0);
+ if (row_is_null) {
+ result_nullmap_data[row] = 1;
+ offsets_data.push_back(result_nested->size());
+ continue;
+ }
+ if (flag_nullmap != nullptr && (*flag_nullmap)[row] != 0) {
+ return Status::InvalidArgument(
+ "array_sorted_merge requires a non-null ascending flag");
+ }
+ const bool ascending = flags[row] != 0;
+ for (const auto& [id, column] : captured_columns) {
+ lambda_block.get_by_position(id).column = column->cut(row, 1);
Review Comment:
Fixed in 49eb8ebc9c7: captured columns are now cut at `expr_selector[row]`, the same original row the array children read, so a short-circuit branch that only sees the selected rows reads the right captured value. `test_array_sorted_merge` adds `order_qt_short_circuit_selector`: with `short_circuit_evaluation = true`, row id=1 (lim=2) returns `[1]`; reading the compacted row 0 (lim=0) used to return `[1,1]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6355797662
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6513017723
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6517579852
Codex automated review failed and did not complete.
Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/41017087584
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4356686316
##########
be/src/exprs/lambda_function/varray_sorted_merge_function.cpp:
##########
@@ -0,0 +1,215 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <string>
+#include <utility>
+
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "exprs/lambda_function/lambda_execution_context.h"
+#include "exprs/lambda_function/lambda_function.h"
+#include "exprs/lambda_function/lambda_function_factory.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vlambda_function_expr.h"
+
+namespace doris {
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc)
+ *
+ * Merges two arrays that are already sorted by the elements' natural order. The merge lambda
+ * receives the two current candidates and returns the combined element; returning NULL means the
+ * two elements must not be combined, so both are kept. Each input key occurs at most once per
+ * array, so a key appears at most twice across the two arrays and the lambda is called at most
+ * once per key.
+ */
+class ArraySortedMergeFunction : public LambdaFunction {
+ ENABLE_FACTORY_CREATOR(ArraySortedMergeFunction);
+
+public:
+ ~ArraySortedMergeFunction() override = default;
+
+ static constexpr auto name = "array_sorted_merge";
+
+ static LambdaFunctionPtr create() { return std::make_shared<ArraySortedMergeFunction>(); }
+
+ std::string get_name() const override { return name; }
+
+ Status execute(VExprContext* context, const Block* block, const Selector* expr_selector,
+ size_t count, ColumnPtr& result_column, const DataTypePtr& result_type,
+ const VExprSPtrs& children) const override {
+ DCHECK_EQ(children.size(), 4);
+
+ const ColumnArray* arrays[2] = {nullptr, nullptr};
+ const IColumn* data[2] = {nullptr, nullptr};
+ DataTypePtr nested_types[2];
+ ColumnPtr holders[2];
+ const ColumnUInt8::Container* nullmaps[2] = {nullptr, nullptr};
+ for (int i = 0; i < 2; ++i) {
+ RETURN_IF_ERROR(children[i + 1]->execute_column(context, block, expr_selector, count,
+ holders[i]));
+ holders[i] = holders[i]->convert_to_full_column_if_const();
+ DataTypePtr type = children[i + 1]->execute_type(block);
+ const IColumn* array_column = holders[i].get();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(array_column)) {
+ nullmaps[i] = &nullable->get_null_map_column().get_data();
+ array_column = &nullable->get_nested_column();
+ type = assert_cast<const DataTypeNullable&>(*type).get_nested_type();
+ }
+ arrays[i] = assert_cast<const ColumnArray*>(array_column);
+ data[i] = arrays[i]->get_data_ptr().get();
+ nested_types[i] = assert_cast<const DataTypeArray&>(*type).get_nested_type();
+ }
+
+ ColumnPtr flag_holder;
+ RETURN_IF_ERROR(
+ children[3]->execute_column(context, block, expr_selector, count, flag_holder));
+ flag_holder = flag_holder->convert_to_full_column_if_const();
+ const IColumn* flag_column = flag_holder.get();
+ const ColumnUInt8::Container* flag_nullmap = nullptr;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(flag_column)) {
+ flag_nullmap = &nullable->get_null_map_column().get_data();
+ flag_column = &nullable->get_nested_column();
+ }
+ const auto& flags = assert_cast<const ColumnUInt8&>(*flag_column).get_data();
+
+ const size_t rows = arrays[0]->size();
+ auto result_nested = nested_types[0]->create_column();
+ auto result_offsets = ColumnArray::ColumnOffsets::create();
+ auto& offsets_data = result_offsets->get_data();
+ offsets_data.reserve(rows);
+ auto result_nullmap = ColumnUInt8::create(rows, 0);
+ auto& result_nullmap_data = result_nullmap->get_data();
+
+ // Two one-row columns hold the lambda arguments of the current merge step. They carry
+ // the lambda argument names because the column refs resolve by name when no
+ // position-based gap is set.
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ const std::vector<std::string>* argument_names =
+ lambda_expr->has_argument_names() ? &lambda_expr->argument_names() : nullptr;
+ Block lambda_block;
+ for (int i = 0; i < 2; ++i) {
+ const std::string argument_name =
+ argument_names != nullptr && i < argument_names->size()
+ ? (*argument_names)[i]
+ : "merge_arg_" + std::to_string(i);
+ lambda_block.insert(ColumnWithTypeAndName(nested_types[i]->create_column(),
+ nested_types[i], argument_name));
+ }
+ // The lambda arguments are bound by position (column ids 0 and 1), like array_sort's
+ // comparator, so the body reads these two one-row columns.
+ IColumn* argument_columns[2] = {
+ lambda_block.get_by_position(0).column->assert_mutable().get(),
+ lambda_block.get_by_position(1).column->assert_mutable().get()};
+
+ LambdaExecutionContext::Frame lambda_frame;
+ lambda_frame.bind_by_name = false;
+ lambda_frame.parent_bindings_visible = false;
+ LambdaExecutionContext::FrameGuard lambda_frame_guard(context->lambda_execution_context(),
+ std::move(lambda_frame));
+
+ const auto& offsets0 = arrays[0]->get_offsets();
+ const auto& offsets1 = arrays[1]->get_offsets();
+ const int lambda_result_base = static_cast<int>(lambda_block.columns());
+
+ for (size_t row = 0; row < rows; ++row) {
+ const bool row_is_null = (nullmaps[0] != nullptr && (*nullmaps[0])[row] != 0) ||
+ (nullmaps[1] != nullptr && (*nullmaps[1])[row] != 0);
+ if (row_is_null) {
+ result_nullmap_data[row] = 1;
+ offsets_data.push_back(result_nested->size());
+ continue;
+ }
+ if (flag_nullmap != nullptr && (*flag_nullmap)[row] != 0) {
+ return Status::InvalidArgument(
+ "array_sorted_merge requires a non-null ascending flag");
+ }
+ const bool ascending = flags[row] != 0;
+
+ size_t left = row == 0 ? 0 : offsets0[row - 1];
+ const size_t left_end = offsets0[row];
+ size_t right = row == 0 ? 0 : offsets1[row - 1];
+ const size_t right_end = offsets1[row];
+
+ while (left < left_end || right < right_end) {
+ if (left == left_end) {
+ result_nested->insert_from(*data[1], right++);
+ continue;
+ }
+ if (right == right_end) {
+ result_nested->insert_from(*data[0], left++);
+ continue;
+ }
+
+ argument_columns[0]->clear();
+ argument_columns[0]->insert_from(*data[0], left);
+ argument_columns[1]->clear();
+ argument_columns[1]->insert_from(*data[1], right);
+
+ int lambda_res_id = lambda_result_base;
+ RETURN_IF_ERROR(children[0]->execute(context, &lambda_block, &lambda_res_id));
Review Comment:
Fixed in 6ac1f1aea13: an enclosing lambda's argument now resolves correctly. The function collects the named bindings visible at the current execution point (`LambdaExecutionContext::collect_visible_bindings`) and publishes their values under those names in its one-row merge block, before the two merge arguments so that a merge argument with the same name still shadows an outer one. The values are read at the same row as the arrays, so short-circuit evaluation and batched lambda execution keep reading the right value.
`order_qt_capture_enclosing_lambda` in `test_array_sorted_merge` covers `array_map(x -> array_sorted_merge((a, b) -> IF(a = b AND a < x, a, NULL), [1], [1], true), [0, 2])` and returns `[[1, 1], [1]]` (x=0 keeps both values, x=2 merges them). A nested lambda that captures one of the merge arguments is still rejected during prepare, because those arguments are not published into nested frames.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6354483008
# FE Regression Coverage Report
Increment line coverage ` 79.53% (136/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6361917752
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6366140353
<details>
<summary>ClickBench: <b>Total hot run time: 24.09 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e250347cd120427a408a4db7a0a55e83da7b098f, data reload: false
query1 0.00 0.00 0.01
query2 0.09 0.04 0.05
query3 0.26 0.08 0.09
query4 1.60 0.10 0.09
query5 0.23 0.24 0.26
query6 1.18 0.68 0.67
query7 0.05 0.01 0.00
query8 0.06 0.04 0.04
query9 0.41 0.35 0.33
query10 0.59 0.56 0.59
query11 0.20 0.15 0.14
query12 0.19 0.14 0.15
query13 0.49 0.50 0.49
query14 1.04 1.00 1.00
query15 0.62 0.59 0.62
query16 0.35 0.35 0.34
query17 1.08 1.15 1.13
query18 0.21 0.20 0.20
query19 2.12 2.00 1.96
query20 0.02 0.01 0.02
query21 15.47 0.19 0.15
query22 4.82 0.06 0.06
query23 16.25 0.33 0.12
query24 2.95 0.57 0.30
query25 0.10 0.05 0.04
query26 0.86 0.20 0.16
query27 0.06 0.05 0.04
query28 3.86 0.83 0.36
query29 12.51 4.19 3.30
query30 0.27 0.13 0.10
query31 2.77 0.56 0.32
query32 3.21 0.59 0.50
query33 3.29 3.32 3.16
query34 15.32 4.00 3.36
query35 3.36 3.33 3.29
query36 0.55 0.43 0.42
query37 0.09 0.07 0.07
query38 0.05 0.03 0.04
query39 0.04 0.03 0.03
query40 0.17 0.14 0.14
query41 0.09 0.03 0.02
query42 0.04 0.03 0.03
query43 0.04 0.04 0.04
Total cold run time: 96.96 s
Total hot run time: 24.09 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6366179092
# BE Regression && UT Coverage Report
Increment line coverage `81.76% (269/329)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 68.35% (31841/46585) |
| Line Coverage | 53.00% (344694/650355) |
| Region Coverage | 48.35% (281610/582490) |
| Branch Coverage | 49.84% (132108/265083) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6353906487
# FE Regression Coverage Report
Increment line coverage ` 79.53% (136/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6362352183
<details>
<summary>TPC-DS: <b>Total hot run time: 150706 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 51bd4e1021eb209791aee5937061ac8805784483, data reload: false
query5 4307 574 445 445
query6 424 203 197 197
query7 4803 444 230 230
query8 319 167 146 146
query9 8795 3031 3067 3031
query10 453 320 263 263
query11 6027 3873 3556 3556
query12 150 89 87 87
query13 1274 435 321 321
query14 6074 3944 3624 3624
query14_1 3452 3416 3398 3398
query15 222 193 187 187
query16 945 444 402 402
query17 863 666 556 556
query18 2417 425 307 307
query19 203 168 139 139
query20 85 82 84 82
query21 219 128 105 105
query22 12982 13026 12740 12740
query23 13566 12935 12567 12567
query23_1 12710 12664 12516 12516
query24 7372 827 458 458
query24_1 480 473 486 473
query25 543 414 356 356
query26 1261 260 140 140
query27 2774 460 275 275
query28 4588 1904 1900 1900
query29 1554 598 459 459
query30 312 225 189 189
query31 1034 926 809 809
query32 165 94 93 93
query33 540 308 248 248
query34 891 835 497 497
query35 753 756 644 644
query36 826 819 707 707
query37 127 95 80 80
query38 1784 1730 1676 1676
query39 796 785 747 747
query39_1 733 727 746 727
query40 237 116 98 98
query41 70 66 67 66
query42 87 88 86 86
query43 342 360 317 317
query44 1298 688 712 688
query45 176 176 168 168
query46 803 953 573 573
query47 2853 2979 2951 2951
query48 287 299 218 218
query49 567 385 316 316
query50 670 267 207 207
query51 10332 10532 10210 10210
query52 76 82 74 74
query53 185 204 152 152
query54 255 197 182 182
query55 72 62 65 62
query56 264 228 205 205
query57 1649 1629 1565 1565
query58 301 263 262 262
query59 2092 2160 1926 1926
query60 286 239 222 222
query61 145 145 142 142
query62 391 331 285 285
query63 191 162 158 158
query64 2715 946 801 801
query65 3308 3276 3257 3257
query66 1793 419 298 298
query67 20086 20161 19921 19921
query68 3025 995 598 598
query69 389 287 290 287
query70 843 768 787 768
query71 300 240 219 219
query72 2730 2574 2345 2345
query73 516 530 292 292
query74 4656 4516 4334 4334
query75 2323 2282 1905 1905
query76 2318 1001 585 585
query77 332 371 265 265
query78 9806 9854 9235 9235
query79 1013 845 521 521
query80 1235 442 347 347
query81 605 324 270 270
query82 858 132 114 114
query83 309 206 182 182
query84 312 123 98 98
query85 887 415 368 368
query86 458 236 224 224
query87 1987 1948 1785 1785
query88 3699 2689 2671 2671
query89 343 279 245 245
query90 1938 169 173 169
query91 169 159 130 130
query92 101 86 76 76
query93 1028 1009 578 578
query94 755 317 265 265
query95 587 326 365 326
query96 664 485 228 228
query97 2453 2454 2337 2337
query98 172 149 148 148
query99 773 748 648 648
Total cold run time: 235761 ms
Total hot run time: 150706 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6366041157
<details>
<summary>TPC-DS: <b>Total hot run time: 150524 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e250347cd120427a408a4db7a0a55e83da7b098f, data reload: false
query5 4308 578 444 444
query6 427 205 188 188
query7 4810 448 226 226
query8 328 162 151 151
query9 8773 3125 3128 3125
query10 476 318 262 262
query11 5952 3858 3557 3557
query12 159 87 84 84
query13 1242 440 314 314
query14 6054 3929 3656 3656
query14_1 3400 3406 3442 3406
query15 205 196 189 189
query16 921 360 413 360
query17 849 663 525 525
query18 2421 425 311 311
query19 195 163 144 144
query20 83 84 82 82
query21 209 129 110 110
query22 12846 12869 12794 12794
query23 13421 12926 12444 12444
query23_1 12614 12678 12758 12678
query24 7384 825 470 470
query24_1 481 480 481 480
query25 537 417 369 369
query26 1269 258 142 142
query27 2771 439 270 270
query28 4612 1935 1936 1935
query29 1571 583 441 441
query30 305 206 186 186
query31 1020 924 816 816
query32 151 89 94 89
query33 528 308 243 243
query34 884 869 519 519
query35 726 747 648 648
query36 814 837 778 778
query37 123 102 83 83
query38 1777 1762 1685 1685
query39 783 820 754 754
query39_1 748 728 742 728
query40 228 123 92 92
query41 72 69 68 68
query42 91 88 82 82
query43 354 363 315 315
query44 1302 688 693 688
query45 184 183 180 180
query46 825 963 570 570
query47 2923 2950 2829 2829
query48 285 291 217 217
query49 580 388 273 273
query50 670 272 200 200
query51 10411 10523 10201 10201
query52 76 86 72 72
query53 187 202 147 147
query54 247 197 192 192
query55 70 66 69 66
query56 243 226 217 217
query57 1652 1602 1579 1579
query58 280 266 257 257
query59 2165 2212 1937 1937
query60 288 248 224 224
query61 144 141 149 141
query62 385 325 295 295
query63 187 156 156 156
query64 2723 948 744 744
query65 3323 3278 3244 3244
query66 1775 407 310 310
query67 20057 19976 19938 19938
query68 2939 1018 610 610
query69 380 283 252 252
query70 854 766 745 745
query71 305 227 203 203
query72 2627 2513 2083 2083
query73 523 525 292 292
query74 4569 4510 4339 4339
query75 2316 2294 1900 1900
query76 2312 1003 591 591
query77 344 387 276 276
query78 9760 9778 9232 9232
query79 999 862 516 516
query80 912 408 357 357
query81 574 316 264 264
query82 595 136 114 114
query83 275 194 185 185
query84 313 120 100 100
query85 972 418 362 362
query86 347 237 235 235
query87 2031 1965 1813 1813
query88 3616 2701 2644 2644
query89 327 279 234 234
query90 1750 172 170 170
query91 159 153 129 129
query92 95 87 92 87
query93 992 988 578 578
query94 523 327 260 260
query95 558 343 361 343
query96 671 484 237 237
query97 2456 2435 2304 2304
query98 160 151 147 147
query99 761 767 645 645
Total cold run time: 233651 ms
Total hot run time: 150524 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4399006193
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,448 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/string_buffer.hpp"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/**
+ * State shared by min_topk and max_topk.
+ *
+ * The entries are held as two parallel structures: a column with the distinct values (sorted
+ * ascending for MIN / descending for MAX, at most k entries) and their non-negative int64 counts.
+ * Only values with count > 0 are kept. With k == 0 the list holds at most the single extreme
+ * value and its row count; the caller derives truncation from the list length, so the state
+ * carries no separate bound.
+ */
+struct AggregateTopKCountData {
+ const DataTypePtr value_type;
+ bool has_k = false;
+ int64_t k = 0;
+ MutableColumnPtr value_column;
+ // Tracked so that the per-group counts participate in query memory accounting: at the maximum
+ // k this buffer alone is 512 KiB per group.
+ DorisVector<int64_t> counts;
+
+ explicit AggregateTopKCountData(const DataTypes& argument_types_)
+ : value_type(remove_nullable(argument_types_[0])),
+ value_column(value_type->create_column()) {}
+
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * min_topk(x, k) / max_topk(x, k).
+ *
+ * Result: ARRAY<STRUCT<value T, count BIGINT>>. NULL inputs are ignored: add validates k for
+ * every row and then skips a NULL x, so the aggregate deliberately does not use the
+ * nullable-argument wrapper. The array is ordered by value (ascending for MIN), holds at most k
+ * distinct values with count > 0, and carries no truncation flag: the caller infers it from the
+ * length (a list of exactly k entries is treated as possibly truncated).
+ *
+ * With k = 0 the array holds at most the extreme value and its row count, which lets the
+ * append-only IVM path compute MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopK final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopK<T, IsMin>, true>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using Base = IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopK<T, IsMin>, true>;
+
+ explicit AggregateFunctionTopK(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "min_topk" : "max_topk"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(remove_nullable(this->argument_types[0]));
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is read for every row, including rows where x is NULL: an evaluated count that is
+ // NULL or out of range must fail the query instead of being skipped like a NULL value.
+ check_k(state, *columns[1], row_num);
+ const IColumn* value_column = columns[0];
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(value_column)) {
+ if (nullable->is_null_at(row_num)) {
+ return;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ add_value(state, *value_column, row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw any row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else if (UNLIKELY(state.k != other.k)) {
+ // Ordinary partial/final aggregation always shares the same constant k, but the
+ // public _state/_merge/_union combinators do not encode k in the AggState type, so a
+ // UNION ALL of states built with different literal k can reach this merge.
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} states have incompatible counts: {} vs {}", get_name(), state.k,
+ other.k);
+ }
+ if (state.k == 0) {
+ merge_extreme(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ auto size = static_cast<int64_t>(state.counts.size());
+ buf.write_binary(size);
+ write_column(buf, *state.value_column);
+ if (size != 0) {
+ // An empty list keeps an empty vector, whose data() is null; passing it to memcpy is
+ // undefined even for a zero length.
+ buf.write(reinterpret_cast<const char*>(state.counts.data()),
+ state.counts.size() * sizeof(int64_t));
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ int64_t size = 0;
+ buf.read_binary(size);
+ state.value_column = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(state.value_column->size()), size);
+ state.counts.resize(size);
+ if (size != 0) {
+ buf.read(reinterpret_cast<char*>(state.counts.data()), size * sizeof(int64_t));
+ }
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_array = assert_cast<ColumnArray&>(to);
+ // The array element and both fields are nullable in the planned type, but a produced entry
+ // never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(result_array.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = assert_cast<ColumnNullable&>(pair_column.get_column(0));
+ auto& count_column = assert_cast<ColumnNullable&>(pair_column.get_column(1));
+ for (size_t i = 0; i != state.counts.size(); ++i) {
+ if (const auto* nullable_value =
+ check_and_get_column<ColumnNullable>(state.value_column.get())) {
+ // A nullable value argument carries its own null map, which is never set here.
+ value_column.insert_from(*nullable_value, i);
+ } else {
+ value_column.get_nested_column().insert_from(*state.value_column, i);
+ value_column.get_null_map_data().push_back(0);
+ }
+ assert_cast<ColumnInt64&>(count_column.get_nested_column())
+ .get_data()
+ .push_back(state.counts[i]);
+ count_column.get_null_map_data().push_back(0);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ result_array.get_offsets().push_back(pair_column.size());
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ // Both fields are nullable in the planned type: a consumer reads a field through
+ // element_at, which is always nullable, and named_struct then records nullable fields.
+ // Produced entries never contain a NULL field.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(value_type),
+ make_nullable(std::make_shared<DataTypeInt64>())},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ return std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ }
+
+private:
+ static void check_k(AggregateTopKCountData& state, const IColumn& count_column,
+ size_t row_num) {
+ const IColumn* nested = &count_column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&count_column)) {
+ if (UNLIKELY(nullable->is_null_at(row_num))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-null "
+ "constant");
+ }
+ nested = &nullable->get_nested_column();
+ }
+ int64_t value =
+ assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(*nested).get_element(
+ row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-negative "
+ "constant, but got {}",
+ value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ /// Ordering of the values. FLOAT/DOUBLE use CompareFloat so that NaN sorts greatest and is
+ /// distinct from every number, which keeps the list mergeable and the top-K deterministic;
+ /// every other type uses the column's compare_at, i.e. exactly the MIN/MAX comparison.
+ static int compare_at(const IColumn& lhs, size_t lhs_row, const IColumn& rhs, size_t rhs_row) {
+ if constexpr (is_float_or_double(T)) {
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ const auto& left =
+ assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(lhs).get_data();
+ const auto& right =
+ assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(rhs).get_data();
+ return CompareFloat(left[lhs_row], right[rhs_row]);
+ } else {
+ return lhs.compare_at(lhs_row, rhs_row, rhs, 1);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "min_topk/max_topk value count overflow");
+ }
+ return result;
+ }
+
+ /// Inserts (value, count) at `pos`, keeping the value column sorted by value.
+ static void insert_entry(AggregateTopKCountData& state, size_t pos, const IColumn& column,
+ size_t row_num, int64_t count) {
+ if (pos == state.value_column->size()) {
+ // Appending needs no rebuild: the state column is mutable, so ascending MIN input
+ // (descending MAX input) fills the list without copying the retained prefix.
+ state.value_column->insert_from(column, row_num);
+ state.counts.push_back(count);
+ return;
+ }
+ auto new_values = state.value_column->clone_empty();
+ new_values->reserve(state.value_column->size() + 1);
+ for (size_t i = 0; i < pos; ++i) {
+ new_values->insert_from(*state.value_column, i);
+ }
+ new_values->insert_from(column, row_num);
+ for (size_t i = pos; i < state.value_column->size(); ++i) {
+ new_values->insert_from(*state.value_column, i);
+ }
+ state.value_column = std::move(new_values);
+ state.counts.insert(state.counts.begin() + pos, count);
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(AggregateTopKCountData& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (state.counts.empty()) {
+ state.value_column->insert_from(column, row_num);
+ state.counts.push_back(1);
+ return;
+ }
+ int cmp = compare_at(column, row_num, *state.value_column, 0);
+ if (cmp == 0) {
+ state.counts[0] = checked_add(state.counts[0], 1);
+ } else if (IsMin ? cmp < 0 : cmp > 0) {
+ state.value_column->clear();
+ state.value_column->insert_from(column, row_num);
+ state.counts[0] = 1;
+ }
+ return;
+ }
+
+ // The list is ascending for MIN and descending for MAX, so the insertion point is where
+ // the stored values stop being better than the new one.
+ size_t left = 0;
+ size_t right = state.value_column->size();
+ while (left < right) {
+ size_t mid = left + (right - left) / 2;
+ int cmp = compare_at(*state.value_column, mid, column, row_num);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ left = mid + 1;
+ } else {
+ right = mid;
+ }
+ }
+ if (left < state.value_column->size() &&
+ compare_at(*state.value_column, left, column, row_num) == 0) {
+ state.counts[left] = checked_add(state.counts[left], 1);
+ return;
+ }
+
+ // A full list already holds the k best values, so a value worse than the last entry would
+ // be inserted at the end and popped immediately: it can only flip the truncation flag.
+ // Rebating it early avoids cloning and copying the whole retained column for every
+ // rejected distinct value (ascending inputs at k = 65536 otherwise copy ~k values each).
+ if (left == state.value_column->size() &&
+ std::cmp_greater_equal(state.value_column->size(), state.k)) {
+ return;
+ }
+
+ insert_entry(state, left, column, row_num, 1);
+ if (static_cast<int64_t>(state.value_column->size()) > state.k) {
+ // More than k distinct values: drop the worst one.
+ state.value_column->pop_back(1);
+ state.counts.pop_back();
+ }
+ }
+
+ /// k == 0: the state holds at most the extreme value and its count.
+ static void merge_extreme(AggregateTopKCountData& state, const AggregateTopKCountData& other) {
+ if (other.counts.empty()) {
+ return;
+ }
+ if (state.counts.empty()) {
+ state.value_column->insert_from(*other.value_column, 0);
+ state.counts.push_back(other.counts[0]);
+ return;
+ }
+ int cmp = compare_at(*state.value_column, 0, *other.value_column, 0);
+ if (cmp == 0) {
+ state.counts[0] = checked_add(state.counts[0], other.counts[0]);
+ } else if (IsMin ? cmp > 0 : cmp < 0) {
+ state.value_column->clear();
+ state.value_column->insert_from(*other.value_column, 0);
+ state.counts[0] = other.counts[0];
+ }
+ }
+
+ /// Merges two sorted lists, summing the counts of equal values and truncating to k.
+ static void merge_values(AggregateTopKCountData& state, const AggregateTopKCountData& other) {
+ if (other.counts.empty()) {
+ // The other partial state saw no non-NULL row.
+ return;
+ }
+ auto merged = state.value_column->clone_empty();
+ DorisVector<int64_t> merged_counts;
+ merged_counts.reserve(state.counts.size() + other.counts.size());
+ size_t lhs_index = 0;
+ size_t rhs_index = 0;
+ while (lhs_index < state.counts.size() || rhs_index < other.counts.size()) {
Review Comment:
Fixed in 7cc5a73646c: merge_values now stops as soon as k entries are emitted. Both inputs are best-first and hold distinct values, and equal values are combined on emission, so every remaining value is worse than all retained ones; it can neither enter the result nor overflow it. checked_add still applies to every value that enters the result, and the unused tail is no longer materialized.
Added both MIN and MAX unit tests next to the retained-value one:
- `TopKTest.MergeDoesNotCountTheDiscardedTail`: k=2, A = [(0,1),(2,INT64_MAX)], B = [(1,1),(2,1)] merge to [(0,1),(1,1)] without the discarded 2 reaching checked_add.
- `TopKTest.MergeDoesNotCountTheDiscardedTailMax`: symmetric case, A = [(2,1),(0,INT64_MAX)], B = [(1,1),(0,1)] merge to [(2,1),(1,1)].
`TopKTest.MergeOverflowIsReported` still passes: an overflowing count on a retained value still fails the query. To your earlier point, counts are group row counts, so INT64_MAX is not reachable from real data, but the early stop is the right semantics and removes the needless work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6466773790
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6490635788
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6434329661
# FE UT Coverage Report
Increment line coverage ` 28.07% (48/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_5309c29db2b57c559711d38b6621f5c934c95d3a/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_5309c29db2b57c559711d38b6621f5c934c95d3a/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6465865465
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4400070091
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,741 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ uint32_t offset = 0;
+ uint32_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ PODArray<char, 32> bytes;
Review Comment:
[P1] Pad the string store before overflow-reading comparisons
For `min_topk(s, 1)` over two rows with `s='a'`, the first append allocates a one-byte buffer: `PODArray::insert` calls `reserve` directly, so `initial_bytes=32` provides no padding. The next add compares the stored `StringRef` through `memcmp_small_allow_overflow15`, whose SSE2 branch reads 16 bytes even for a one-byte string. That load extends 15 bytes beyond the allocation, violating the comparator's memory contract and ASAN bounds. `max_topk`, `k=0` and state merges share this path. Use storage with at least 15 readable bytes after the payload, or a bounds-safe comparison, and cover short strings in aggregate/state-merge tests. This is established from the allocation and load paths; no ASAN run was performed.
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,741 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ uint32_t offset = 0;
+ uint32_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ PODArray<char, 32> bytes;
+ DorisVector<Span> spans;
+
+ size_t size() const { return spans.size(); }
+ void reserve(size_t n) { spans.reserve(n); }
+ void clear() {
+ bytes.clear();
+ spans.clear();
+ }
+ StringRef at(size_t slot) const {
+ const Span& span = spans[slot];
+ return StringRef(bytes.data() + span.offset, span.length);
+ }
+ void append(StringRef value) {
+ DCHECK_LE(bytes.size() + value.size, std::numeric_limits<uint32_t>::max());
+ DCHECK_LE(value.size, std::numeric_limits<uint32_t>::max());
+ spans.push_back({static_cast<uint32_t>(bytes.size()), static_cast<uint32_t>(value.size)});
Review Comment:
[P2] Prevent cumulative string offsets from wrapping
For `k=2049`, one group receiving 4097 descending distinct 1 MiB strings across input blocks reaches this append with `bytes.size()==2^32`. Only 2047 slots have been evicted before this append, so `dead > live` has never triggered compaction. With enough query memory, release builds narrow the new offset to zero: the newly retained smallest value then reads the first, already-evicted high string. Checked builds instead fail the size DCHECK at the preceding append. The live result is only 2049 MiB, so the per-column 4 GiB limit does not exclude this schedule. Use lossless offsets or compact/check the byte limit before narrowing, returning a query error for unsupported sizes.
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,741 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ uint32_t offset = 0;
+ uint32_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ PODArray<char, 32> bytes;
+ DorisVector<Span> spans;
+
+ size_t size() const { return spans.size(); }
+ void reserve(size_t n) { spans.reserve(n); }
+ void clear() {
+ bytes.clear();
+ spans.clear();
+ }
+ StringRef at(size_t slot) const {
+ const Span& span = spans[slot];
+ return StringRef(bytes.data() + span.offset, span.length);
+ }
+ void append(StringRef value) {
+ DCHECK_LE(bytes.size() + value.size, std::numeric_limits<uint32_t>::max());
+ DCHECK_LE(value.size, std::numeric_limits<uint32_t>::max());
+ spans.push_back({static_cast<uint32_t>(bytes.size()), static_cast<uint32_t>(value.size)});
+ if (value.size != 0) {
+ bytes.insert(value.data, value.data + value.size);
+ }
+ }
+ void swap(StringTopKValueStore& other) noexcept {
+ bytes.swap(other.bytes);
+ spans.swap(other.spans);
+ }
+};
+
+/**
+ * State shared by min_topk and max_topk, for one value type.
+ *
+ * The distinct values sit in slots (see the stores above), with one Slot record per value carrying
+ * the row count and the links of a treap ordered best-first (ascending values for MIN, descending
+ * for MAX). A row costs one O(log k) descent: it either finds its value and bumps the count or
+ * links a new slot at the leaf. A sorted vector would shift O(k) elements for the same insert.
+ *
+ * The treap stores explicit child links because, unlike a heap, its shape is not the complete tree
+ * that an array index can describe: rotations move entries, and the in-order sequence is the value
+ * order rather than the arrival order. A heap could evict and append in O(log k), but it cannot
+ * find an existing value without a hash index or a linear scan, and it cannot produce the sorted
+ * result without sorting. The ordered tree is the structure that keeps find, insert and eviction
+ * all at O(log k) with comparisons only.
+ *
+ * An entry that is evicted keeps its slot and value until the next compaction. The tree never
+ * points at a dead slot.
+ *
+ * With k == 0 the state holds at most the single extreme value and its row count; the caller
+ * derives truncation from the list length, so the state carries no separate bound.
+ */
+template <PrimitiveType T>
+struct AggregateTopKCountData {
+ using CppType = typename PrimitiveTypeTraits<T>::CppType;
+ static constexpr bool IS_STRING = is_string_type(T);
+ using ValueStore =
+ std::conditional_t<IS_STRING, StringTopKValueStore, FixedTopKValueStore<CppType>>;
+ using ValueRef = std::conditional_t<IS_STRING, StringRef, const CppType&>;
+
+ static constexpr int32_t NO_SLOT = -1;
+ // Row counts are never negative, so this cannot collide with a live entry.
+ static constexpr int64_t DEAD_COUNT = -1;
+
+ /// One slot's metadata; the value itself lives in `values` at the same index.
+ struct Slot {
+ int64_t count = 0;
+ int32_t left = NO_SLOT;
+ int32_t right = NO_SLOT;
+ uint32_t priority = 0;
+ };
+
+ bool has_k = false;
+ int64_t k = 0;
+ // Indexed by slot; an evicted slot stays as a dead entry until compaction.
+ ValueStore values;
+ DorisVector<Slot> slots;
+ int32_t root = NO_SLOT;
+ // The last live slot in the aggregate's order: the value an incoming row has to beat to enter
+ // a full list. Cached so that rejecting a worse value costs one comparison, not a descent.
+ int32_t worst = NO_SLOT;
+ int64_t live = 0;
+ // Slots evicted since the last compaction.
+ int64_t dead = 0;
+ uint64_t rng = 0x243F6A8885A308D3ULL;
+
+ AggregateTopKCountData() = default;
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * min_topk(x, k) / max_topk(x, k).
+ *
+ * Result: ARRAY<STRUCT<value T, count BIGINT>>. NULL inputs are ignored: add validates k for
+ * every row and then skips a NULL x, so the aggregate deliberately does not use the
+ * nullable-argument wrapper. The array is ordered by value (ascending for MIN), holds at most k
+ * distinct values with count > 0, and carries no truncation flag: the caller infers it from the
+ * length (a list of exactly k entries is treated as possibly truncated).
+ *
+ * With k = 0 the array holds at most the extreme value and its row count, which lets the
+ * append-only IVM path compute MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopK final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData<T>,
+ AggregateFunctionTopK<T, IsMin>, false>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using State = AggregateTopKCountData<T>;
+ using Base = IAggregateFunctionDataHelper<State, AggregateFunctionTopK<T, IsMin>, false>;
+ using Slot = typename State::Slot;
+ using ValueStore = typename State::ValueStore;
+ using ValueRef = typename State::ValueRef;
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ static constexpr bool IS_STRING = State::IS_STRING;
+ static constexpr int32_t NO_SLOT = State::NO_SLOT;
+
+ explicit AggregateFunctionTopK(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "min_topk" : "max_topk"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(remove_nullable(this->argument_types[0]));
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is read for every row, including rows where x is NULL: an evaluated count that is
+ // NULL or out of range must fail the query instead of being skipped like a NULL value.
+ check_k(state, *columns[1], row_num);
+ const IColumn* value_column = columns[0];
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(value_column)) {
+ if (nullable->is_null_at(row_num)) {
+ return;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ add_value(state, *value_column, row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw any row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else if (UNLIKELY(state.k != other.k)) {
+ // Ordinary partial/final aggregation always shares the same constant k, but the
+ // public _state/_merge/_union combinators do not encode k in the AggState type, so a
+ // UNION ALL of states built with different literal k can reach this merge.
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} states have incompatible counts: {} vs {}", get_name(), state.k,
+ other.k);
+ }
+ if (state.k == 0) {
+ merge_extreme(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ // The slots are in arrival order and may include dead ones, while the serialized form is
+ // the sorted (value, count) list the reader rebuilds the tree from.
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ auto sorted_values = remove_nullable(this->argument_types[0])->create_column();
+ sorted_values->reserve(order.size());
+ DorisVector<int64_t> sorted_counts;
+ sorted_counts.reserve(order.size());
+ for (int32_t slot : order) {
+ insert_value(*sorted_values, state.values.at(slot));
+ sorted_counts.push_back(state.slots[slot].count);
+ }
+ auto size = static_cast<int64_t>(sorted_counts.size());
+ buf.write_binary(size);
+ write_column(buf, *sorted_values);
+ if (size != 0) {
+ // An empty list keeps an empty vector, whose data() is null; passing it to memcpy is
+ // undefined even for a zero length.
+ buf.write(reinterpret_cast<const char*>(sorted_counts.data()),
+ sorted_counts.size() * sizeof(int64_t));
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ int64_t size = 0;
+ buf.read_binary(size);
+ auto values = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(values->size()), size);
+ DorisVector<int64_t> counts(static_cast<size_t>(size));
+ if (size != 0) {
+ buf.read(reinterpret_cast<char*>(counts.data()), size * sizeof(int64_t));
+ }
+ ValueStore value_store;
+ value_store.reserve(static_cast<size_t>(size));
+ for (int64_t row = 0; row != size; ++row) {
+ value_store.append(value_at(*values, static_cast<size_t>(row)));
+ }
+ assign_sorted(state, std::move(value_store), std::move(counts));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_array = assert_cast<ColumnArray&>(to);
+ // The array element and both fields are nullable in the planned type, but a produced entry
+ // never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(result_array.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = assert_cast<ColumnNullable&>(pair_column.get_column(0));
+ auto& count_column = assert_cast<ColumnNullable&>(pair_column.get_column(1));
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ for (int32_t slot : order) {
+ insert_value(value_column.get_nested_column(), state.values.at(slot));
+ value_column.get_null_map_data().push_back(0);
+ assert_cast<ColumnInt64&>(count_column.get_nested_column())
+ .get_data()
+ .push_back(state.slots[slot].count);
+ count_column.get_null_map_data().push_back(0);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ result_array.get_offsets().push_back(pair_column.size());
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ // Both fields are nullable in the planned type: a consumer reads a field through
+ // element_at, which is always nullable, and named_struct then records nullable fields.
+ // Produced entries never contain a NULL field.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(value_type),
+ make_nullable(std::make_shared<DataTypeInt64>())},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ return std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ }
+
+private:
+ /// The value of the row in `column`. The returned view is valid while the column is alive.
+ static ValueRef value_at(const IColumn& column, size_t row_num) {
+ if constexpr (IS_STRING) {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column).get_data_at(
+ row_num);
+ } else {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column)
+ .get_data()[row_num];
+ }
+ }
+
+ /// Exactly the comparison the value column would apply: CompareFloat for FLOAT/DOUBLE through
+ /// Compare::compare (the column uses the same helper), the same memcmp helper ColumnString
+ /// uses for strings, and the column's scale-independent ordering for every other type.
+ static int compare_values(ValueRef lhs, ValueRef rhs) {
+ if constexpr (IS_STRING) {
+ return memcmp_small_allow_overflow15(
+ reinterpret_cast<const uint8_t*>(lhs.data), lhs.size,
+ reinterpret_cast<const uint8_t*>(rhs.data), rhs.size);
+ } else {
+ return Compare::compare(lhs, rhs);
+ }
+ }
+
+ static void insert_value(IColumn& column, ValueRef value) {
+ if constexpr (IS_STRING) {
+ assert_cast<ColumnString&>(column).insert_data(value.data, value.size);
+ } else {
+ assert_cast<ColumnType&>(column).get_data().push_back(value);
+ }
+ }
+
+ /// True when `cmp` puts the left operand before the right one in the aggregate's order.
+ static bool comes_before(int cmp) { return IsMin ? cmp < 0 : cmp > 0; }
+
+ /// True when `cmp`, which compares a value with the worst retained one, puts it after it: the
+ /// value is worse and cannot enter the list.
+ static bool is_worse(int cmp) { return IsMin ? cmp > 0 : cmp < 0; }
+
+ /// splitmix64. The priorities only have to be independent enough to keep the treap balanced;
+ /// a per-state sequence keeps the shape reproducible for the same insertion order.
+ static uint32_t next_priority(State& state) {
+ uint64_t z = (state.rng += 0x9E3779B97F4A7C15ULL);
+ z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9ULL;
+ z = (z ^ (z >> 27)) * 0x94D049BB133111EBULL;
+ return static_cast<uint32_t>(z ^ (z >> 31));
+ }
+
+ static int compare_slots(const State& state, int32_t lhs, int32_t rhs) {
+ return compare_values(state.values.at(lhs), state.values.at(rhs));
+ }
+
+ /// Appends (value, count) as the next slot.
+ static int32_t append_slot(State& state, ValueRef value, int64_t count) {
+ auto slot = static_cast<int32_t>(state.slots.size());
+ DCHECK_LT(slot, std::numeric_limits<int32_t>::max());
+ state.values.append(value);
+ state.slots.push_back(Slot {count, State::NO_SLOT, State::NO_SLOT, next_priority(state)});
+ ++state.live;
+ return slot;
+ }
+
+ static int32_t append_slot(State& state, const IColumn& column, size_t row_num, int64_t count) {
+ return append_slot(state, value_at(column, row_num), count);
+ }
+
+ /// Rotates the subtree rooted at `node` right: the left child becomes the root and `node` its
+ /// right child. The in-order sequence is unchanged, so the treap stays ordered.
+ static int32_t rotate_right(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].left;
+ state.slots[node].left = state.slots[pivot].right;
+ state.slots[pivot].right = node;
+ return pivot;
+ }
+
+ static int32_t rotate_left(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].right;
+ state.slots[node].right = state.slots[pivot].left;
+ state.slots[pivot].left = node;
+ return pivot;
+ }
+
+ /// One descent that performs the whole per-row update: an equal value bumps its count and
+ /// reports no insertion; a missing value is linked as a new slot at the leaf and rotated up
+ /// while the recursion unwinds. `inserted_slot` receives the new slot, NO_SLOT on a hit.
+ static int32_t add_or_increment(State& state, int32_t node, const IColumn& column,
+ size_t row_num, int32_t* inserted_slot) {
+ if (node == State::NO_SLOT) {
+ *inserted_slot = append_slot(state, column, row_num, 1);
+ return *inserted_slot;
+ }
+ int cmp = compare_values(state.values.at(node), value_at(column, row_num));
+ if (cmp == 0) {
+ state.slots[node].count = checked_add(state.slots[node].count, 1);
+ return node;
+ }
+ if (comes_before(cmp)) {
+ // The stored value is the better one, so the input belongs in the right subtree.
+ state.slots[node].right = add_or_increment(state, state.slots[node].right, column,
+ row_num, inserted_slot);
+ if (state.slots[state.slots[node].right].priority > state.slots[node].priority) {
+ return rotate_left(state, node);
+ }
+ } else {
+ state.slots[node].left =
+ add_or_increment(state, state.slots[node].left, column, row_num, inserted_slot);
+ if (state.slots[state.slots[node].left].priority > state.slots[node].priority) {
+ return rotate_right(state, node);
+ }
+ }
+ return node;
+ }
+
+ /// Removes the last entry in the aggregate's order (the largest value for MIN) and returns the
+ /// new subtree root. That entry has no right child, so its left subtree takes its place; the
+ /// child's priority is below the parent's, so no rotation is needed.
+ static int32_t erase_worst(State& state, int32_t node, int32_t* removed) {
+ if (state.slots[node].right == State::NO_SLOT) {
+ *removed = node;
+ return state.slots[node].left;
+ }
+ state.slots[node].right = erase_worst(state, state.slots[node].right, removed);
+ return node;
+ }
+
+ static void refresh_worst(State& state) {
+ int32_t node = state.root;
+ while (node != State::NO_SLOT && state.slots[node].right != State::NO_SLOT) {
+ node = state.slots[node].right;
+ }
+ state.worst = node;
+ }
+
+ /// Slots in the aggregate's order, i.e. the order of the result array.
+ static void collect_in_order(const State& state, DorisVector<int32_t>& slots) {
+ slots.reserve(static_cast<size_t>(state.live));
+ DorisVector<int32_t> stack;
+ stack.reserve(64);
+ int32_t node = state.root;
+ while (node != State::NO_SLOT || !stack.empty()) {
+ while (node != State::NO_SLOT) {
+ stack.push_back(node);
+ node = state.slots[node].left;
+ }
+ node = stack.back();
+ stack.pop_back();
+ // Only live slots are reachable from the root; an evicted slot has no row count.
+ DCHECK_GE(state.slots[node].count, 0);
+ slots.push_back(node);
+ node = state.slots[node].right;
+ }
+ DCHECK_EQ(static_cast<int64_t>(slots.size()), state.live);
+ }
+
+ /// Rebuilds the tree from `values`, already in the aggregate's order, and the matching counts.
+ /// The treap is built in O(size) as a Cartesian tree over the fresh priorities.
+ static void assign_sorted(State& state, ValueStore values, DorisVector<int64_t> counts) {
+ const size_t size = counts.size();
+ DCHECK_EQ(values.size(), size);
+ state.values.swap(values);
+ state.slots.assign(size, Slot {});
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].count = counts[slot];
+ }
+ state.live = static_cast<int64_t>(size);
+ state.dead = 0;
+ DorisVector<int32_t> stack;
+ stack.reserve(size);
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].priority = next_priority(state);
+ int32_t last = State::NO_SLOT;
+ while (!stack.empty() &&
+ state.slots[stack.back()].priority < state.slots[slot].priority) {
+ last = stack.back();
+ stack.pop_back();
+ }
+ state.slots[slot].left = last;
+ if (!stack.empty()) {
+ state.slots[stack.back()].right = static_cast<int32_t>(slot);
+ }
+ stack.push_back(static_cast<int32_t>(slot));
+ }
+ state.root = stack.empty() ? State::NO_SLOT : stack.front();
+ refresh_worst(state);
+ }
+
+ /// Reclaims the slots evicted since the last compaction. The O(k) rebuild is charged to the
+ /// k/2 evictions that produced the dead slots, so it stays O(1) amortized per row.
+ static void compact(State& state) {
+ DorisVector<int32_t> slots;
+ collect_in_order(state, slots);
+ ValueStore values;
+ values.reserve(slots.size());
+ DorisVector<int64_t> counts;
+ counts.reserve(slots.size());
+ for (int32_t slot : slots) {
+ values.append(state.values.at(slot));
+ counts.push_back(state.slots[slot].count);
+ }
+ assign_sorted(state, std::move(values), std::move(counts));
+ }
+
+ static void check_k(State& state, const IColumn& count_column, size_t row_num) {
+ const IColumn* nested = &count_column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&count_column)) {
+ if (UNLIKELY(nullable->is_null_at(row_num))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-null "
+ "constant");
+ }
+ nested = &nullable->get_nested_column();
+ }
+ int64_t value =
+ assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(*nested).get_element(
+ row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-negative "
+ "constant, but got {}",
+ value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "min_topk/max_topk value count overflow");
+ }
+ return result;
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(State& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (state.live == 0) {
+ state.root = append_slot(state, column, row_num, 1);
+ state.worst = state.root;
+ return;
+ }
+ int cmp = compare_values(value_at(column, row_num), state.values.at(0));
+ if (cmp == 0) {
+ state.slots[0].count = checked_add(state.slots[0].count, 1);
+ } else if (comes_before(cmp)) {
+ // The new value is better: replace the tracked extreme. The slot and the
+ // single-node tree stay in place, only the stored value changes.
+ state.values.clear();
+ state.values.append(value_at(column, row_num));
+ state.slots[0].count = 1;
+ }
+ return;
+ }
+
+ // Dead slots are reclaimed before they can outnumber the live ones; see compact().
+ if (state.dead > state.live) {
Review Comment:
[P2] Reclaim evicted strings according to their byte size
Fill a MIN state at `k=1024` with 1024 distinct 1 MiB `z`-prefixed strings, then replace them with 1024 distinct short `a`-prefixed strings. The live result is a few KiB, but `dead==live==1024`, so this condition never compacts, including on subsequent duplicate or rejected rows. At least 1 GiB of discarded payload remains owned until group destruction or a later eviction; processing many groups sequentially can exhaust query memory despite tiny final states. Track discarded/live bytes and trigger byte-aware string reclamation. This is separate from offset wrap: it occurs below 4 GiB and remains after changing offsets to a wider type.
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,741 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ uint32_t offset = 0;
+ uint32_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ PODArray<char, 32> bytes;
+ DorisVector<Span> spans;
+
+ size_t size() const { return spans.size(); }
+ void reserve(size_t n) { spans.reserve(n); }
+ void clear() {
+ bytes.clear();
+ spans.clear();
+ }
+ StringRef at(size_t slot) const {
+ const Span& span = spans[slot];
+ return StringRef(bytes.data() + span.offset, span.length);
+ }
+ void append(StringRef value) {
+ DCHECK_LE(bytes.size() + value.size, std::numeric_limits<uint32_t>::max());
+ DCHECK_LE(value.size, std::numeric_limits<uint32_t>::max());
+ spans.push_back({static_cast<uint32_t>(bytes.size()), static_cast<uint32_t>(value.size)});
+ if (value.size != 0) {
+ bytes.insert(value.data, value.data + value.size);
+ }
+ }
+ void swap(StringTopKValueStore& other) noexcept {
+ bytes.swap(other.bytes);
+ spans.swap(other.spans);
+ }
+};
+
+/**
+ * State shared by min_topk and max_topk, for one value type.
+ *
+ * The distinct values sit in slots (see the stores above), with one Slot record per value carrying
+ * the row count and the links of a treap ordered best-first (ascending values for MIN, descending
+ * for MAX). A row costs one O(log k) descent: it either finds its value and bumps the count or
+ * links a new slot at the leaf. A sorted vector would shift O(k) elements for the same insert.
+ *
+ * The treap stores explicit child links because, unlike a heap, its shape is not the complete tree
+ * that an array index can describe: rotations move entries, and the in-order sequence is the value
+ * order rather than the arrival order. A heap could evict and append in O(log k), but it cannot
+ * find an existing value without a hash index or a linear scan, and it cannot produce the sorted
+ * result without sorting. The ordered tree is the structure that keeps find, insert and eviction
+ * all at O(log k) with comparisons only.
+ *
+ * An entry that is evicted keeps its slot and value until the next compaction. The tree never
+ * points at a dead slot.
+ *
+ * With k == 0 the state holds at most the single extreme value and its row count; the caller
+ * derives truncation from the list length, so the state carries no separate bound.
+ */
+template <PrimitiveType T>
+struct AggregateTopKCountData {
+ using CppType = typename PrimitiveTypeTraits<T>::CppType;
+ static constexpr bool IS_STRING = is_string_type(T);
+ using ValueStore =
+ std::conditional_t<IS_STRING, StringTopKValueStore, FixedTopKValueStore<CppType>>;
+ using ValueRef = std::conditional_t<IS_STRING, StringRef, const CppType&>;
+
+ static constexpr int32_t NO_SLOT = -1;
+ // Row counts are never negative, so this cannot collide with a live entry.
+ static constexpr int64_t DEAD_COUNT = -1;
+
+ /// One slot's metadata; the value itself lives in `values` at the same index.
+ struct Slot {
+ int64_t count = 0;
+ int32_t left = NO_SLOT;
+ int32_t right = NO_SLOT;
+ uint32_t priority = 0;
+ };
+
+ bool has_k = false;
+ int64_t k = 0;
+ // Indexed by slot; an evicted slot stays as a dead entry until compaction.
+ ValueStore values;
+ DorisVector<Slot> slots;
+ int32_t root = NO_SLOT;
+ // The last live slot in the aggregate's order: the value an incoming row has to beat to enter
+ // a full list. Cached so that rejecting a worse value costs one comparison, not a descent.
+ int32_t worst = NO_SLOT;
+ int64_t live = 0;
+ // Slots evicted since the last compaction.
+ int64_t dead = 0;
+ uint64_t rng = 0x243F6A8885A308D3ULL;
+
+ AggregateTopKCountData() = default;
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * min_topk(x, k) / max_topk(x, k).
+ *
+ * Result: ARRAY<STRUCT<value T, count BIGINT>>. NULL inputs are ignored: add validates k for
+ * every row and then skips a NULL x, so the aggregate deliberately does not use the
+ * nullable-argument wrapper. The array is ordered by value (ascending for MIN), holds at most k
+ * distinct values with count > 0, and carries no truncation flag: the caller infers it from the
+ * length (a list of exactly k entries is treated as possibly truncated).
+ *
+ * With k = 0 the array holds at most the extreme value and its row count, which lets the
+ * append-only IVM path compute MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopK final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData<T>,
+ AggregateFunctionTopK<T, IsMin>, false>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using State = AggregateTopKCountData<T>;
+ using Base = IAggregateFunctionDataHelper<State, AggregateFunctionTopK<T, IsMin>, false>;
+ using Slot = typename State::Slot;
+ using ValueStore = typename State::ValueStore;
+ using ValueRef = typename State::ValueRef;
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ static constexpr bool IS_STRING = State::IS_STRING;
+ static constexpr int32_t NO_SLOT = State::NO_SLOT;
+
+ explicit AggregateFunctionTopK(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "min_topk" : "max_topk"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(remove_nullable(this->argument_types[0]));
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is read for every row, including rows where x is NULL: an evaluated count that is
+ // NULL or out of range must fail the query instead of being skipped like a NULL value.
+ check_k(state, *columns[1], row_num);
+ const IColumn* value_column = columns[0];
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(value_column)) {
+ if (nullable->is_null_at(row_num)) {
+ return;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ add_value(state, *value_column, row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw any row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else if (UNLIKELY(state.k != other.k)) {
+ // Ordinary partial/final aggregation always shares the same constant k, but the
+ // public _state/_merge/_union combinators do not encode k in the AggState type, so a
+ // UNION ALL of states built with different literal k can reach this merge.
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} states have incompatible counts: {} vs {}", get_name(), state.k,
+ other.k);
+ }
+ if (state.k == 0) {
+ merge_extreme(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ // The slots are in arrival order and may include dead ones, while the serialized form is
+ // the sorted (value, count) list the reader rebuilds the tree from.
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ auto sorted_values = remove_nullable(this->argument_types[0])->create_column();
+ sorted_values->reserve(order.size());
+ DorisVector<int64_t> sorted_counts;
+ sorted_counts.reserve(order.size());
+ for (int32_t slot : order) {
+ insert_value(*sorted_values, state.values.at(slot));
+ sorted_counts.push_back(state.slots[slot].count);
+ }
+ auto size = static_cast<int64_t>(sorted_counts.size());
+ buf.write_binary(size);
+ write_column(buf, *sorted_values);
+ if (size != 0) {
+ // An empty list keeps an empty vector, whose data() is null; passing it to memcpy is
+ // undefined even for a zero length.
+ buf.write(reinterpret_cast<const char*>(sorted_counts.data()),
+ sorted_counts.size() * sizeof(int64_t));
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ int64_t size = 0;
+ buf.read_binary(size);
+ auto values = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(values->size()), size);
+ DorisVector<int64_t> counts(static_cast<size_t>(size));
+ if (size != 0) {
+ buf.read(reinterpret_cast<char*>(counts.data()), size * sizeof(int64_t));
+ }
+ ValueStore value_store;
+ value_store.reserve(static_cast<size_t>(size));
+ for (int64_t row = 0; row != size; ++row) {
+ value_store.append(value_at(*values, static_cast<size_t>(row)));
+ }
+ assign_sorted(state, std::move(value_store), std::move(counts));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_array = assert_cast<ColumnArray&>(to);
+ // The array element and both fields are nullable in the planned type, but a produced entry
+ // never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(result_array.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = assert_cast<ColumnNullable&>(pair_column.get_column(0));
+ auto& count_column = assert_cast<ColumnNullable&>(pair_column.get_column(1));
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ for (int32_t slot : order) {
+ insert_value(value_column.get_nested_column(), state.values.at(slot));
+ value_column.get_null_map_data().push_back(0);
+ assert_cast<ColumnInt64&>(count_column.get_nested_column())
+ .get_data()
+ .push_back(state.slots[slot].count);
+ count_column.get_null_map_data().push_back(0);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ result_array.get_offsets().push_back(pair_column.size());
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ // Both fields are nullable in the planned type: a consumer reads a field through
+ // element_at, which is always nullable, and named_struct then records nullable fields.
+ // Produced entries never contain a NULL field.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(value_type),
+ make_nullable(std::make_shared<DataTypeInt64>())},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ return std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ }
+
+private:
+ /// The value of the row in `column`. The returned view is valid while the column is alive.
+ static ValueRef value_at(const IColumn& column, size_t row_num) {
+ if constexpr (IS_STRING) {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column).get_data_at(
+ row_num);
+ } else {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column)
+ .get_data()[row_num];
+ }
+ }
+
+ /// Exactly the comparison the value column would apply: CompareFloat for FLOAT/DOUBLE through
+ /// Compare::compare (the column uses the same helper), the same memcmp helper ColumnString
+ /// uses for strings, and the column's scale-independent ordering for every other type.
+ static int compare_values(ValueRef lhs, ValueRef rhs) {
+ if constexpr (IS_STRING) {
+ return memcmp_small_allow_overflow15(
+ reinterpret_cast<const uint8_t*>(lhs.data), lhs.size,
+ reinterpret_cast<const uint8_t*>(rhs.data), rhs.size);
+ } else {
+ return Compare::compare(lhs, rhs);
+ }
+ }
+
+ static void insert_value(IColumn& column, ValueRef value) {
+ if constexpr (IS_STRING) {
+ assert_cast<ColumnString&>(column).insert_data(value.data, value.size);
+ } else {
+ assert_cast<ColumnType&>(column).get_data().push_back(value);
+ }
+ }
+
+ /// True when `cmp` puts the left operand before the right one in the aggregate's order.
+ static bool comes_before(int cmp) { return IsMin ? cmp < 0 : cmp > 0; }
+
+ /// True when `cmp`, which compares a value with the worst retained one, puts it after it: the
+ /// value is worse and cannot enter the list.
+ static bool is_worse(int cmp) { return IsMin ? cmp > 0 : cmp < 0; }
+
+ /// splitmix64. The priorities only have to be independent enough to keep the treap balanced;
+ /// a per-state sequence keeps the shape reproducible for the same insertion order.
+ static uint32_t next_priority(State& state) {
+ uint64_t z = (state.rng += 0x9E3779B97F4A7C15ULL);
+ z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9ULL;
+ z = (z ^ (z >> 27)) * 0x94D049BB133111EBULL;
+ return static_cast<uint32_t>(z ^ (z >> 31));
+ }
+
+ static int compare_slots(const State& state, int32_t lhs, int32_t rhs) {
+ return compare_values(state.values.at(lhs), state.values.at(rhs));
+ }
+
+ /// Appends (value, count) as the next slot.
+ static int32_t append_slot(State& state, ValueRef value, int64_t count) {
+ auto slot = static_cast<int32_t>(state.slots.size());
+ DCHECK_LT(slot, std::numeric_limits<int32_t>::max());
+ state.values.append(value);
+ state.slots.push_back(Slot {count, State::NO_SLOT, State::NO_SLOT, next_priority(state)});
+ ++state.live;
+ return slot;
+ }
+
+ static int32_t append_slot(State& state, const IColumn& column, size_t row_num, int64_t count) {
+ return append_slot(state, value_at(column, row_num), count);
+ }
+
+ /// Rotates the subtree rooted at `node` right: the left child becomes the root and `node` its
+ /// right child. The in-order sequence is unchanged, so the treap stays ordered.
+ static int32_t rotate_right(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].left;
+ state.slots[node].left = state.slots[pivot].right;
+ state.slots[pivot].right = node;
+ return pivot;
+ }
+
+ static int32_t rotate_left(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].right;
+ state.slots[node].right = state.slots[pivot].left;
+ state.slots[pivot].left = node;
+ return pivot;
+ }
+
+ /// One descent that performs the whole per-row update: an equal value bumps its count and
+ /// reports no insertion; a missing value is linked as a new slot at the leaf and rotated up
+ /// while the recursion unwinds. `inserted_slot` receives the new slot, NO_SLOT on a hit.
+ static int32_t add_or_increment(State& state, int32_t node, const IColumn& column,
+ size_t row_num, int32_t* inserted_slot) {
+ if (node == State::NO_SLOT) {
+ *inserted_slot = append_slot(state, column, row_num, 1);
+ return *inserted_slot;
+ }
+ int cmp = compare_values(state.values.at(node), value_at(column, row_num));
+ if (cmp == 0) {
+ state.slots[node].count = checked_add(state.slots[node].count, 1);
+ return node;
+ }
+ if (comes_before(cmp)) {
+ // The stored value is the better one, so the input belongs in the right subtree.
+ state.slots[node].right = add_or_increment(state, state.slots[node].right, column,
+ row_num, inserted_slot);
+ if (state.slots[state.slots[node].right].priority > state.slots[node].priority) {
+ return rotate_left(state, node);
+ }
+ } else {
+ state.slots[node].left =
+ add_or_increment(state, state.slots[node].left, column, row_num, inserted_slot);
+ if (state.slots[state.slots[node].left].priority > state.slots[node].priority) {
+ return rotate_right(state, node);
+ }
+ }
+ return node;
+ }
+
+ /// Removes the last entry in the aggregate's order (the largest value for MIN) and returns the
+ /// new subtree root. That entry has no right child, so its left subtree takes its place; the
+ /// child's priority is below the parent's, so no rotation is needed.
+ static int32_t erase_worst(State& state, int32_t node, int32_t* removed) {
+ if (state.slots[node].right == State::NO_SLOT) {
+ *removed = node;
+ return state.slots[node].left;
+ }
+ state.slots[node].right = erase_worst(state, state.slots[node].right, removed);
+ return node;
+ }
+
+ static void refresh_worst(State& state) {
+ int32_t node = state.root;
+ while (node != State::NO_SLOT && state.slots[node].right != State::NO_SLOT) {
+ node = state.slots[node].right;
+ }
+ state.worst = node;
+ }
+
+ /// Slots in the aggregate's order, i.e. the order of the result array.
+ static void collect_in_order(const State& state, DorisVector<int32_t>& slots) {
+ slots.reserve(static_cast<size_t>(state.live));
+ DorisVector<int32_t> stack;
+ stack.reserve(64);
+ int32_t node = state.root;
+ while (node != State::NO_SLOT || !stack.empty()) {
+ while (node != State::NO_SLOT) {
+ stack.push_back(node);
+ node = state.slots[node].left;
+ }
+ node = stack.back();
+ stack.pop_back();
+ // Only live slots are reachable from the root; an evicted slot has no row count.
+ DCHECK_GE(state.slots[node].count, 0);
+ slots.push_back(node);
+ node = state.slots[node].right;
+ }
+ DCHECK_EQ(static_cast<int64_t>(slots.size()), state.live);
+ }
+
+ /// Rebuilds the tree from `values`, already in the aggregate's order, and the matching counts.
+ /// The treap is built in O(size) as a Cartesian tree over the fresh priorities.
+ static void assign_sorted(State& state, ValueStore values, DorisVector<int64_t> counts) {
+ const size_t size = counts.size();
+ DCHECK_EQ(values.size(), size);
+ state.values.swap(values);
+ state.slots.assign(size, Slot {});
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].count = counts[slot];
+ }
+ state.live = static_cast<int64_t>(size);
+ state.dead = 0;
+ DorisVector<int32_t> stack;
+ stack.reserve(size);
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].priority = next_priority(state);
+ int32_t last = State::NO_SLOT;
+ while (!stack.empty() &&
+ state.slots[stack.back()].priority < state.slots[slot].priority) {
+ last = stack.back();
+ stack.pop_back();
+ }
+ state.slots[slot].left = last;
+ if (!stack.empty()) {
+ state.slots[stack.back()].right = static_cast<int32_t>(slot);
+ }
+ stack.push_back(static_cast<int32_t>(slot));
+ }
+ state.root = stack.empty() ? State::NO_SLOT : stack.front();
+ refresh_worst(state);
+ }
+
+ /// Reclaims the slots evicted since the last compaction. The O(k) rebuild is charged to the
+ /// k/2 evictions that produced the dead slots, so it stays O(1) amortized per row.
+ static void compact(State& state) {
+ DorisVector<int32_t> slots;
+ collect_in_order(state, slots);
+ ValueStore values;
+ values.reserve(slots.size());
+ DorisVector<int64_t> counts;
+ counts.reserve(slots.size());
+ for (int32_t slot : slots) {
+ values.append(state.values.at(slot));
+ counts.push_back(state.slots[slot].count);
+ }
+ assign_sorted(state, std::move(values), std::move(counts));
+ }
+
+ static void check_k(State& state, const IColumn& count_column, size_t row_num) {
+ const IColumn* nested = &count_column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&count_column)) {
+ if (UNLIKELY(nullable->is_null_at(row_num))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-null "
+ "constant");
+ }
+ nested = &nullable->get_nested_column();
+ }
+ int64_t value =
+ assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(*nested).get_element(
+ row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-negative "
+ "constant, but got {}",
+ value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "min_topk/max_topk value count overflow");
+ }
+ return result;
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(State& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (state.live == 0) {
+ state.root = append_slot(state, column, row_num, 1);
+ state.worst = state.root;
+ return;
+ }
+ int cmp = compare_values(value_at(column, row_num), state.values.at(0));
+ if (cmp == 0) {
+ state.slots[0].count = checked_add(state.slots[0].count, 1);
+ } else if (comes_before(cmp)) {
+ // The new value is better: replace the tracked extreme. The slot and the
+ // single-node tree stay in place, only the stored value changes.
+ state.values.clear();
+ state.values.append(value_at(column, row_num));
+ state.slots[0].count = 1;
+ }
+ return;
+ }
+
+ // Dead slots are reclaimed before they can outnumber the live ones; see compact().
+ if (state.dead > state.live) {
+ compact(state);
+ }
+
+ if (state.live >= state.k) {
+ // A full list rejects or absorbs most rows: comparing with the eviction candidate
+ // costs one comparison, while the tree descent would cost log2(k).
+ int cmp = compare_values(value_at(column, row_num), state.values.at(state.worst));
+ if (cmp == 0) {
+ state.slots[state.worst].count = checked_add(state.slots[state.worst].count, 1);
+ return;
+ }
+ if (is_worse(cmp)) {
+ return;
+ }
+ }
+
+ int32_t inserted = State::NO_SLOT;
+ state.root = add_or_increment(state, state.root, column, row_num, &inserted);
+ if (inserted == State::NO_SLOT) {
+ return;
+ }
+ if (state.live <= state.k) {
+ // Not over capacity: the new entry competes with the cached worst for the eviction
+ // candidate role.
+ if (state.worst == State::NO_SLOT ||
+ is_worse(compare_slots(state, inserted, state.worst))) {
+ state.worst = inserted;
+ }
+ return;
+ }
+ // The list was full and the value is new. It beat the cached worst above, so it cannot be
+ // the worst itself; the old worst is still the rightmost entry and leaves the list.
+ int32_t removed = State::NO_SLOT;
+ state.root = erase_worst(state, state.root, &removed);
+ DCHECK_EQ(removed, state.worst);
+ state.slots[removed].count = State::DEAD_COUNT;
+ --state.live;
+ ++state.dead;
+ refresh_worst(state);
+ }
+
+ /// k == 0: the state holds at most the extreme value and its count.
+ static void merge_extreme(State& state, const State& other) {
+ if (other.live == 0) {
+ return;
+ }
+ if (state.live == 0) {
+ state.root = append_slot(state, other.values.at(0), other.slots[0].count);
+ state.worst = state.root;
+ return;
+ }
+ int cmp = compare_values(state.values.at(0), other.values.at(0));
+ if (cmp == 0) {
+ state.slots[0].count = checked_add(state.slots[0].count, other.slots[0].count);
+ } else if (is_worse(cmp)) {
+ state.values.clear();
+ state.values.append(other.values.at(0));
+ state.slots[0].count = other.slots[0].count;
+ }
+ }
+
+ /// Merges two ordered lists, summing the counts of equal values and keeping the k best.
+ ///
+ /// The merge stops as soon as k entries are produced. Both inputs are best-first and hold
+ /// distinct values, and equal values have already been combined, so every remaining value is
+ /// worse than all retained ones: it can neither enter the result nor overflow it. Stopping
+ /// here also avoids materializing a tail that would be truncated away.
+ static void merge_values(State& state, const State& other) {
+ if (other.live == 0) {
+ // The other partial state saw no non-NULL row.
+ return;
+ }
+ DorisVector<int32_t> lhs_slots;
+ DorisVector<int32_t> rhs_slots;
+ collect_in_order(state, lhs_slots);
Review Comment:
[P2] Skip partial states that cannot change a full result
`min_topk_merge(min_topk_state(x,65536))` serializes one singleton per row, and `AggregateStateUnion` invokes this merge for each singleton. After MIN retains `0..65535`, every higher singleton still traverses/copies all 65536 retained values and `assign_sorted` rebuilds their tree. One million ascending inputs therefore cause over 61 billion unchanged-value copies after the list fills (source-derived, not benchmarked); MAX/descending input is symmetric. This serialized merge bypasses the existing `add_value` rejection fast path, and stopping after k outputs still rebuilds the unchanged prefix. Reject an entirely worse RHS using the cached worst and RHS best before collecting/rebuilding the prefix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6339447837
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6339449138
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6341231832
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4348577909
##########
regression-test/data/query_p0/aggregate/topk_min_max_count.out:
##########
@@ -0,0 +1,143 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !min_k0 --
+a {"values":[], "bound":1}
+b {"values":[], "bound":5}
+c {"values":[], "bound":null}
+d {"values":[], "bound":10}
+e {"values":[], "bound":0}
+
+-- !max_k0 --
+a {"values":[], "bound":3}
+b {"values":[], "bound":5}
+c {"values":[], "bound":null}
+d {"values":[], "bound":17}
+e {"values":[], "bound":0}
+
+-- !min_k1 --
+a {"values":[{"value":1, "count":2}], "bound":1}
+b {"values":[{"value":5, "count":1}], "bound":null}
+c {"values":[], "bound":null}
+d {"values":[{"value":10, "count":1}], "bound":10}
+e {"values":[{"value":0, "count":5}], "bound":null}
+
+-- !max_k1 --
+a {"values":[{"value":3, "count":2}], "bound":3}
+b {"values":[{"value":5, "count":1}], "bound":null}
+c {"values":[], "bound":null}
+d {"values":[{"value":17, "count":1}], "bound":17}
+e {"values":[{"value":0, "count":5}], "bound":null}
+
+-- !min_k2 --
+a {"values":[{"value":1, "count":2}, {"value":2, "count":1}], "bound":2}
+b {"values":[{"value":5, "count":1}], "bound":null}
+c {"values":[], "bound":null}
+d {"values":[{"value":10, "count":1}, {"value":11, "count":1}], "bound":11}
+e {"values":[{"value":0, "count":5}], "bound":null}
+
+-- !max_k2 --
+a {"values":[{"value":3, "count":2}, {"value":2, "count":1}], "bound":2}
+b {"values":[{"value":5, "count":1}], "bound":null}
+c {"values":[], "bound":null}
+d {"values":[{"value":17, "count":1}, {"value":16, "count":1}], "bound":16}
+e {"values":[{"value":0, "count":5}], "bound":null}
+
+-- !min_k8 --
+a {"values":[{"value":1, "count":2}, {"value":2, "count":1}, {"value":3, "count":2}], "bound":null}
+b {"values":[{"value":5, "count":1}], "bound":null}
+c {"values":[], "bound":null}
+d {"values":[{"value":10, "count":1}, {"value":11, "count":1}, {"value":12, "count":1}, {"value":13, "count":1}, {"value":14, "count":1}, {"value":15, "count":1}, {"value":16, "count":1}, {"value":17, "count":1}], "bound":null}
+e {"values":[{"value":0, "count":5}], "bound":null}
+
+-- !max_k8 --
+a {"values":[{"value":3, "count":2}, {"value":2, "count":1}, {"value":1, "count":2}], "bound":null}
+b {"values":[{"value":5, "count":1}], "bound":null}
+c {"values":[], "bound":null}
+d {"values":[{"value":17, "count":1}, {"value":16, "count":1}, {"value":15, "count":1}, {"value":14, "count":1}, {"value":13, "count":1}, {"value":12, "count":1}, {"value":11, "count":1}, {"value":10, "count":1}], "bound":null}
+e {"values":[{"value":0, "count":5}], "bound":null}
+
+-- !min_empty_input --
+{"values":[], "bound":null}
+
+-- !min_all_null_group --
+c {"values":[], "bound":null}
+
+-- !min_not_null_type --
+a {"values":[{"value":1, "count":2}, {"value":2, "count":1}], "bound":2}
+b {"values":[{"value":5, "count":1}], "bound":null}
+c {"values":[{"value":0, "count":2}], "bound":null}
+d {"values":[{"value":10, "count":1}, {"value":11, "count":1}], "bound":11}
+e {"values":[{"value":0, "count":5}], "bound":null}
+
+-- !min_not_null_type_empty --
+{"values":[], "bound":null}
+
+-- !min_string --
+a {"values":[{"value":"a", "count":1}, {"value":"b", "count":1}], "bound":"b"} {"values":[{"value":"z", "count":1}, {"value":"c", "count":3}], "bound":"c"}
+b {"values":[{"value":"x", "count":1}], "bound":null} {"values":[{"value":"x", "count":1}], "bound":null}
+c {"values":[], "bound":null} {"values":[], "bound":null}
+d {"values":[{"value":"j", "count":1}, {"value":"k", "count":1}], "bound":"k"} {"values":[{"value":"q", "count":1}, {"value":"p", "count":1}], "bound":"p"}
+e {"values":[{"value":"s", "count":5}], "bound":null} {"values":[{"value":"s", "count":5}], "bound":null}
+
+-- !min_decimal --
+a {"values":[{"value":1.10, "count":2}, {"value":2.20, "count":1}], "bound":2.20} {"values":[{"value":3.30, "count":2}, {"value":2.20, "count":1}], "bound":2.20}
+b {"values":[{"value":5.50, "count":1}], "bound":null} {"values":[{"value":5.50, "count":1}], "bound":null}
+c {"values":[], "bound":null} {"values":[], "bound":null}
+d {"values":[{"value":10.10, "count":1}, {"value":11.10, "count":1}], "bound":11.10} {"values":[{"value":17.10, "count":1}, {"value":16.10, "count":1}], "bound":16.10}
+e {"values":[{"value":0.00, "count":5}], "bound":null} {"values":[{"value":0.00, "count":5}], "bound":null}
+
+-- !min_datetime --
+a {"values":[{"value":"2024-01-01 09:00:00", "count":1}, {"value":"2024-01-01 10:00:00", "count":1}], "bound":"2024-01-01 10:00:00"} {"values":[{"value":"2024-01-03 10:00:00", "count":2}, {"value":"2024-01-02 10:00:00", "count":1}], "bound":"2024-01-02 10:00:00"}
+b {"values":[{"value":"2024-02-01 10:00:00", "count":1}], "bound":null} {"values":[{"value":"2024-02-01 10:00:00", "count":1}], "bound":null}
+c {"values":[], "bound":null} {"values":[], "bound":null}
+d {"values":[{"value":"2024-03-01 10:00:00", "count":1}, {"value":"2024-03-02 10:00:00", "count":1}], "bound":"2024-03-02 10:00:00"} {"values":[{"value":"2024-03-08 10:00:00", "count":1}, {"value":"2024-03-07 10:00:00", "count":1}], "bound":"2024-03-07 10:00:00"}
+e {"values":[], "bound":null} {"values":[], "bound":null}
+
+-- !double_special_values --
+e {"values":[{"value":-0, "count":2}, {"value":1, "count":1}, {"value":NaN, "count":2}], "bound":null} {"values":[{"value":NaN, "count":2}, {"value":1, "count":1}, {"value":-0, "count":2}], "bound":null}
Review Comment:
Fixed in 56a622307d2. The case no longer depends on scan order: it is an explicit `topk_merge_min` with `-0.0` on the left and `0.0` inserted on the right, so the retained representative is deterministic, and the golden pins that single merge result.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/TopKMergeFunction.java:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.scalar;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable;
+import org.apache.doris.nereids.trees.expressions.functions.CustomSignature;
+import org.apache.doris.nereids.trees.expressions.functions.agg.TopKCountFunction;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
+
+import java.util.List;
+
+/**
+ * Shared implementation of {@code topk_merge_min} and {@code topk_merge_max}, the apply-side
+ * merge of a MIN/MAX top-K state with the insert/delete top-K lists of one delta batch:
+ *
+ * <pre>
+ * topk_merge_min(S, b, I, b_i, D, b_d, n, dn, k)
+ * -> STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T>
+ * </pre>
+ *
+ * <p>The result is the state itself and has exactly the type the aggregate returns, so apply can
+ * write it back to the hidden top-K state columns without rebuilding a struct. The caller derives
+ * {@code n' = n + dn} and {@code exhausted = (size(values) == 0 && n' > 0)} itself.
+ *
+ * <p>A NULL list is an empty list; a NULL {@code b}/{@code b_i}/{@code b_d} means that side is
+ * complete/untruncated. The result struct itself is never NULL, only its {@code bound} field can
+ * be: the merge only trusts values in {@code v <= min(b, b_i, b_d)}, so {@code bound} is the
+ * effective upper bound (for MIN; the lower bound for MAX) of the resulting state.
+ */
+public abstract class TopKMergeFunction extends ScalarFunction
+ implements CustomSignature, AlwaysNotNullable {
+
+ protected TopKMergeFunction(String name, List<Expression> children) {
+ super(name, children);
+ }
+
+ /** constructor for withChildren and reuse signature */
+ protected TopKMergeFunction(ScalarFunctionParams functionParams) {
+ super(functionParams);
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ if (arity() != 9) {
+ throw new AnalysisException(getName() + " requires 9 arguments, but got " + arity());
+ }
+ DataType valueType = inferValueType();
+ if (valueType == null) {
+ throw new AnalysisException(getName() + " can not infer the value type from arguments: "
+ + toSql());
+ }
+ if (!TopKCountFunction.isSupportedValueType(valueType)) {
+ throw new AnalysisException(getName() + " does not support type " + valueType.toSql());
+ }
+ for (int index : new int[] {0, 2, 4}) {
+ DataType type = getArgumentType(index);
+ if (type.isNullType()) {
+ continue;
+ }
+ if (!(type instanceof ArrayType)
+ || !(((ArrayType) type).getItemType() instanceof StructType)) {
+ throw new AnalysisException(getName() + " requires argument " + (index + 1)
+ + " to be ARRAY<STRUCT<value, count>>, but got " + type.toSql());
+ }
+ }
+ }
+
+ @Override
+ public FunctionSignature customSignature() {
+ DataType valueType = inferValueType();
+ if (valueType == null) {
+ throw new AnalysisException(getName() + " can not infer the value type from arguments: "
+ + toSql());
+ }
+ ArrayType listType = ArrayType.of(TopKCountFunction.pairType(valueType));
+ return FunctionSignature.ret(resultType(valueType)).args(
Review Comment:
Fixed in e4acad346dd. `TopKMergeFunction` now overrides `computePrecision` and returns the signature unchanged, so a NULL bound can no longer contribute DECIMAL(1,0) and narrow the nested lists. `TopKFunctionSignatureTest.testMergeKeepsDecimalPrecisionWithNullBounds` covers S=[(1.23,1)] with NULL bounds, and `order_qt_merge_decimal_null_bounds` covers it end-to-end.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/TopKMergeFunction.java:
##########
@@ -0,0 +1,139 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.scalar;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable;
+import org.apache.doris.nereids.trees.expressions.functions.CustomSignature;
+import org.apache.doris.nereids.trees.expressions.functions.agg.TopKCountFunction;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
+
+import java.util.List;
+
+/**
+ * Shared implementation of {@code topk_merge_min} and {@code topk_merge_max}, the apply-side
+ * merge of a MIN/MAX top-K state with the insert/delete top-K lists of one delta batch:
+ *
+ * <pre>
+ * topk_merge_min(S, b, I, b_i, D, b_d, n, dn, k)
+ * -> STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T>
+ * </pre>
+ *
+ * <p>The result is the state itself and has exactly the type the aggregate returns, so apply can
+ * write it back to the hidden top-K state columns without rebuilding a struct. The caller derives
+ * {@code n' = n + dn} and {@code exhausted = (size(values) == 0 && n' > 0)} itself.
+ *
+ * <p>A NULL list is an empty list; a NULL {@code b}/{@code b_i}/{@code b_d} means that side is
+ * complete/untruncated. The result struct itself is never NULL, only its {@code bound} field can
+ * be: the merge only trusts values in {@code v <= min(b, b_i, b_d)}, so {@code bound} is the
+ * effective upper bound (for MIN; the lower bound for MAX) of the resulting state.
+ */
+public abstract class TopKMergeFunction extends ScalarFunction
+ implements CustomSignature, AlwaysNotNullable {
+
+ protected TopKMergeFunction(String name, List<Expression> children) {
+ super(name, children);
+ }
+
+ /** constructor for withChildren and reuse signature */
+ protected TopKMergeFunction(ScalarFunctionParams functionParams) {
+ super(functionParams);
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ if (arity() != 9) {
+ throw new AnalysisException(getName() + " requires 9 arguments, but got " + arity());
+ }
+ DataType valueType = inferValueType();
+ if (valueType == null) {
+ throw new AnalysisException(getName() + " can not infer the value type from arguments: "
+ + toSql());
+ }
+ if (!TopKCountFunction.isSupportedValueType(valueType)) {
+ throw new AnalysisException(getName() + " does not support type " + valueType.toSql());
+ }
+ for (int index : new int[] {0, 2, 4}) {
+ DataType type = getArgumentType(index);
+ if (type.isNullType()) {
+ continue;
+ }
+ if (!(type instanceof ArrayType)
+ || !(((ArrayType) type).getItemType() instanceof StructType)) {
+ throw new AnalysisException(getName() + " requires argument " + (index + 1)
+ + " to be ARRAY<STRUCT<value, count>>, but got " + type.toSql());
+ }
+ }
+ }
+
+ @Override
+ public FunctionSignature customSignature() {
+ DataType valueType = inferValueType();
+ if (valueType == null) {
+ throw new AnalysisException(getName() + " can not infer the value type from arguments: "
+ + toSql());
+ }
+ ArrayType listType = ArrayType.of(TopKCountFunction.pairType(valueType));
+ return FunctionSignature.ret(resultType(valueType)).args(
+ listType, valueType,
+ listType, valueType,
+ listType, valueType,
+ BigIntType.INSTANCE, BigIntType.INSTANCE, BigIntType.INSTANCE);
+ }
+
+ /**
+ * The value type is read from the first argument that carries it: the per-value type of S/I/D
+ * (argument positions 1, 3, 5) or, when all lists are untyped NULL literals, the bound type
+ * (positions 2, 4, 6).
+ */
+ private DataType inferValueType() {
+ for (int index : new int[] {0, 2, 4}) {
+ DataType type = getArgumentType(index);
+ if (!(type instanceof ArrayType)) {
+ continue;
+ }
+ DataType itemType = ((ArrayType) type).getItemType();
+ if (!(itemType instanceof StructType)) {
+ continue;
+ }
+ List<StructField> fields = ((StructType) itemType).getFields();
+ if (fields.size() != 2) {
+ continue;
+ }
+ return TopKCountFunction.normalizeValueType(fields.get(0).getDataType());
Review Comment:
Fixed in e4acad346dd. `inferValueType` now requires the same value type in every typed list and bound and raises an analysis error for mixed types, so no list is coerced to a narrower type. `TopKFunctionSignatureTest.testMergeRejectsMixedValueTypes` covers empty INT lists with a DOUBLE list, and the regression asserts the same error.
##########
be/src/exprs/function/function_topk_merge.cpp:
##########
@@ -0,0 +1,465 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <glog/logging.h>
+
+#include <cstddef>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_numbers.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/types.h"
+#include "exprs/function/function.h"
+#include "exprs/function/simple_function_factory.h"
+
+namespace doris {
+class FunctionContext;
+} // namespace doris
+
+namespace doris {
+
+namespace {
+
+/// Upper bound of k. Keep in sync with the aggregate's MAX_TOPK_K
+/// (aggregate_function_topk_count.h) and FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/// One row of an ARRAY<STRUCT<value, count>> argument. A NULL array is an empty view.
+struct TopKArrayView {
+ const IColumn* values = nullptr;
+ const ColumnInt64* counts = nullptr;
+ /// NULL bitmap of the array element (the pair struct).
+ const NullMap* null_map = nullptr;
+ /// NULL bitmap of the `value` field; NULL values are not part of the contract.
+ const NullMap* value_null_map = nullptr;
+ /// NULL bitmap of the `count` field; NULL counts are not part of the contract.
+ const NullMap* count_null_map = nullptr;
+ size_t begin = 0;
+ size_t end = 0;
+};
+
+TopKArrayView make_array_view(const IColumn& column, size_t row) {
+ TopKArrayView view;
+ const IColumn* array_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ array_column = &nullable->get_nested_column();
+ }
+ const auto& array = assert_cast<const ColumnArray&>(*array_column);
+ // offsets[i] is the cumulative size after row i, so row's range is [offsets[row-1], offsets[row]).
+ // offset_at(0) reads offsets[-1], which PaddedPODArray keeps as 0.
+ view.begin = array.offset_at(static_cast<ssize_t>(row));
+ view.end = array.offset_at(static_cast<ssize_t>(row) + 1);
+
+ const IColumn* pair_column = &array.get_data();
+ if (const auto* element_nullable = check_and_get_column<ColumnNullable>(pair_column)) {
+ view.null_map = &element_nullable->get_null_map_data();
+ pair_column = &element_nullable->get_nested_column();
+ }
+ const auto& pairs = assert_cast<const ColumnStruct&>(*pair_column);
+ view.values = &pairs.get_column(0);
+ // A cast may widen the `value` field to a nullable column; the entries themselves are never
+ // NULL, so only the nested column is used.
+ if (const auto* value_nullable = check_and_get_column<ColumnNullable>(view.values)) {
+ view.value_null_map = &value_nullable->get_null_map_data();
+ view.values = &value_nullable->get_nested_column();
+ }
+ // The same holds for `count`: a nested cast may have made the field nullable.
+ const auto* count_column = &pairs.get_column(1);
+ if (const auto* count_nullable = check_and_get_column<ColumnNullable>(count_column)) {
+ view.count_null_map = &count_nullable->get_null_map_data();
+ count_column = &count_nullable->get_nested_column();
+ }
+ view.counts = &assert_cast<const ColumnInt64&>(*count_column);
+ return view;
+}
+
+/// One scalar bound argument: NULL means the corresponding side was not truncated. `row` is the
+/// index of the current input row inside `value`: bounds are ordinary per-row columns, so a block
+/// can carry a different bound in every row.
+struct TopKBoundView {
+ const IColumn* value = nullptr;
+ size_t row = 0;
+ bool is_null = true;
+};
+
+TopKBoundView make_bound_view(const IColumn& column, size_t row) {
+ TopKBoundView view;
+ const IColumn* value_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ view.value = value_column;
+ view.row = row;
+ view.is_null = false;
+ return view;
+}
+
+int64_t read_int64(const IColumn& column, size_t row, const char* argument_name) {
+ const IColumn* nested = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ // The contract declares these arguments as non-NULL BIGINT.
+ DORIS_CHECK(!nullable->is_null_at(row)) << argument_name << " must not be NULL";
+ nested = &nullable->get_nested_column();
+ }
+ return assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(*nested).get_element(row);
+}
+
+int64_t checked_add(int64_t lhs, int64_t rhs, const char* function_name) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "{}: value count overflow", function_name);
+ }
+ return result;
+}
+
+int64_t checked_sub(int64_t lhs, int64_t rhs, const char* function_name) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_sub_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "{}: value count overflow", function_name);
+ }
+ return result;
+}
+
+/// FLOAT/DOUBLE use CompareFloat so NaN sorts greatest and is distinct from every number; other
+/// types use the column comparison, i.e. exactly the MIN/MAX comparison.
+int compare_values(PrimitiveType value_type, const IColumn& lhs, size_t lhs_row, const IColumn& rhs,
+ size_t rhs_row) {
+ if (value_type == TYPE_FLOAT) {
+ return CompareFloat(
+ assert_cast<const ColumnFloat32&, TypeCheckOnRelease::DISABLE>(lhs).get_element(
+ lhs_row),
+ assert_cast<const ColumnFloat32&, TypeCheckOnRelease::DISABLE>(rhs).get_element(
+ rhs_row));
+ }
+ if (value_type == TYPE_DOUBLE) {
+ return CompareFloat(
+ assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(lhs).get_element(
+ lhs_row),
+ assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(rhs).get_element(
+ rhs_row));
+ }
+ return lhs.compare_at(lhs_row, rhs_row, rhs, 1);
+}
+
+DataTypePtr find_value_type(const DataTypes& arguments) {
+ // Prefer the per-value type of S/I/D; it is also present when all bound arguments are NULL.
+ for (size_t index : {0, 2, 4}) {
+ const auto& type = remove_nullable(arguments[index]);
+ if (type->get_primitive_type() != TYPE_ARRAY) {
+ continue;
+ }
+ const auto& nested =
+ remove_nullable(assert_cast<const DataTypeArray&>(*type).get_nested_type());
+ if (nested->get_primitive_type() != TYPE_STRUCT) {
+ continue;
+ }
+ const auto& fields = assert_cast<const DataTypeStruct&>(*nested).get_elements();
+ if (fields.size() != 2) {
+ continue;
+ }
+ // The result struct always declares a non-nullable `value` field, while an argument cast
+ // may have produced a nullable one; drop it so both sides agree on the return type.
+ return remove_nullable(fields[0]);
+ }
+ for (size_t index : {1, 3, 5}) {
+ const auto& type = remove_nullable(arguments[index]);
+ if (!type->is_null_literal()) {
+ return type;
+ }
+ }
+ return nullptr;
+}
+
+/**
+ * topk_merge_min / topk_merge_max: one apply step of the IVM MIN/MAX top-K state.
+ *
+ * Arguments: S, b, I, b_i, D, b_d, n, dn, k. S is the pre-image state, I/D are the insert/delete
+ * top-K lists of this delta batch and b/i/d bounds are their truncation points (NULL = not
+ * truncated). Only values in [<= bE] are trusted, where bE = min of all non-NULL bounds (max for
+ * MAX): outside it at least one side may have dropped values. The post count of a value is
+ * S + I - D; non-positive counts are removed (net-zero cancellations), then the first k values are
+ * kept.
+ *
+ * The result is the state itself, i.e. the same type topk_min_count/topk_max_count return:
+ * STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T NULL>. n' = n + dn and
+ * exhausted = (size(values) == 0 && n' > 0) are derived by the caller, which writes n' to the
+ * COUNT state column and uses exhausted to trigger the fallback.
+ */
+template <bool IsMin>
+class FunctionTopKMerge : public IFunction {
+public:
+ static constexpr auto name = IsMin ? "topk_merge_min" : "topk_merge_max";
+
+ static FunctionPtr create() { return std::make_shared<FunctionTopKMerge>(); }
+
+ String get_name() const override { return name; }
+
+ size_t get_number_of_arguments() const override { return 9; }
+
+ /// A NULL list is an empty list and a NULL bound is a valid value: the default implementation
+ /// would turn the whole row into NULL and break the contract.
+ bool use_default_implementation_for_nulls() const override { return false; }
+
+ DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ DataTypePtr value_type = find_value_type(arguments);
+ if (value_type == nullptr) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} can not infer the value type from arguments", name);
+ }
+ // The result is the state type, identical to the aggregate's return type, so that the
+ // apply expression can write it straight back to the hidden MIN_TOPK/MAX_TOPK and BOUND
+ // columns.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {value_type, std::make_shared<DataTypeInt64>()},
+ Strings {"value", "count"});
+ auto values_type = std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ return std::make_shared<DataTypeStruct>(DataTypes {values_type, make_nullable(value_type)},
+ Strings {"values", "bound"});
+ }
+
+ Status execute_impl(FunctionContext* /*context*/, Block& block, const ColumnNumbers& arguments,
+ uint32_t result, size_t input_rows_count) const override {
+ std::vector<ColumnPtr> columns(arguments.size());
+ DataTypes argument_types(arguments.size());
+ for (size_t i = 0; i != arguments.size(); ++i) {
+ columns[i] =
+ block.get_by_position(arguments[i]).column->convert_to_full_column_if_const();
+ argument_types[i] = block.get_by_position(arguments[i]).type;
+ }
+ const DataTypePtr value_type = find_value_type(argument_types);
+ // An untyped call is rejected while planning get_return_type(); the plan always carries T.
+ DCHECK(value_type != nullptr);
+ const PrimitiveType primitive_type = value_type->get_primitive_type();
+
+ auto result_column = block.get_by_position(result).type->create_column();
+ auto& result_struct = assert_cast<ColumnStruct&>(*result_column);
+ auto& out_values = assert_cast<ColumnArray&>(result_struct.get_column(0));
+ auto& out_pair_nullable = assert_cast<ColumnNullable&>(out_values.get_data());
+ auto& out_pair = assert_cast<ColumnStruct&>(out_pair_nullable.get_nested_column());
+ auto& out_value_column = out_pair.get_column(0);
+ auto& out_count_column = assert_cast<ColumnInt64&>(out_pair.get_column(1));
+ auto& out_bound_column = assert_cast<ColumnNullable&>(result_struct.get_column(1));
+
+ for (size_t row = 0; row != input_rows_count; ++row) {
+ const int64_t k = read_int64(*columns[8], row, "k");
+ if (UNLIKELY(k < 1 || k > MAX_TOPK_K)) {
+ return Status::InvalidArgument("{} requires 1 <= k <= {}, but got {}", name,
+ MAX_TOPK_K, k);
+ }
+ const int64_t n = read_int64(*columns[6], row, "n");
+ const int64_t dn = read_int64(*columns[7], row, "dn");
+ const int64_t new_n = checked_add(n, dn, name);
+
+ const TopKArrayView s_view = make_array_view(*columns[0], row);
+ const TopKArrayView i_view = make_array_view(*columns[2], row);
+ const TopKArrayView d_view = make_array_view(*columns[4], row);
+ const TopKBoundView b_view = make_bound_view(*columns[1], row);
+ const TopKBoundView bi_view = make_bound_view(*columns[3], row);
+ const TopKBoundView bd_view = make_bound_view(*columns[5], row);
+ const TopKBoundView effective_bound =
+ effective_bound_of(b_view, bi_view, bd_view, primitive_type);
+
+ size_t positive = 0;
+ RETURN_IF_ERROR(append_trusted_values(s_view, i_view, d_view, effective_bound, k,
+ primitive_type, out_value_column,
+ out_count_column, out_pair_nullable, &positive));
+ out_values.get_offsets().push_back(out_pair.size());
+
+ // bound: NULL while the state is complete, the k-th kept value only when values were
+ // actually dropped (pos > k), or the effective bound otherwise.
+ // b IS NULL means the pre-image covers every distinct value; a NULL b_i/b_d means
+ // that side of the delta was not truncated.
+ // pos == k is NOT truncation: every known value fit, so keep bE instead of shrinking
+ // the trusted range to max(S').
+ const bool sides_complete = b_view.is_null && bi_view.is_null && bd_view.is_null;
+ if (new_n == 0 || (sides_complete && std::cmp_less_equal(positive, k))) {
+ out_bound_column.insert_default();
+ } else if (std::cmp_greater(positive, k)) {
+ out_bound_column.get_nested_column().insert_from(out_value_column,
+ out_pair.size() - 1);
+ out_bound_column.get_null_map_data().push_back(0);
+ } else {
+ DCHECK(!effective_bound.is_null);
+ out_bound_column.get_nested_column().insert_from(*effective_bound.value,
+ effective_bound.row);
+ out_bound_column.get_null_map_data().push_back(0);
+ }
+ }
+
+ block.replace_by_position(result, std::move(result_column));
+ return Status::OK();
+ }
+
+private:
+ /// bE: the best (smallest for MIN, largest for MAX) of the non-NULL bounds, or nullptr when
+ /// every side is complete/untruncated.
+ static TopKBoundView effective_bound_of(const TopKBoundView& b_view,
+ const TopKBoundView& bi_view,
+ const TopKBoundView& bd_view, PrimitiveType type) {
+ TopKBoundView bound;
+ for (const TopKBoundView* view : {&b_view, &bi_view, &bd_view}) {
+ if (view->is_null) {
+ continue;
+ }
+ if (bound.is_null) {
+ bound = *view;
+ continue;
+ }
+ int cmp = compare_values(type, *view->value, view->row, *bound.value, bound.row);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ bound = *view;
+ }
+ }
+ return bound;
+ }
+
+ /// Finds the best head (smallest for MIN, largest for MAX) among the three lists; `head` is
+ /// set to nullptr when all of them are exhausted.
+ static void best_head(const TopKArrayView& s_view, size_t s_index, const TopKArrayView& i_view,
+ size_t i_index, const TopKArrayView& d_view, size_t d_index,
+ PrimitiveType type, const IColumn** head, size_t* head_row) {
+ *head = nullptr;
+ const TopKArrayView* views[] = {&s_view, &i_view, &d_view};
+ const size_t indexes[] = {s_index, i_index, d_index};
+ for (size_t v = 0; v != 3; ++v) {
+ if (indexes[v] == views[v]->end) {
+ continue;
+ }
+ if (*head == nullptr) {
+ *head = views[v]->values;
+ *head_row = indexes[v];
+ continue;
+ }
+ int cmp = compare_values(type, *views[v]->values, indexes[v], **head, *head_row);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ *head = views[v]->values;
+ *head_row = indexes[v];
+ }
+ }
+ }
+
+ /// Consumes one entry equal to the current head value, returning its count.
+ static bool consume_equal(const TopKArrayView& view, size_t& index, const IColumn& head,
+ size_t head_row, PrimitiveType type, int64_t* count) {
+ if (index == view.end || compare_values(type, *view.values, index, head, head_row) != 0) {
+ return false;
+ }
+ // The top-K producers never emit a NULL entry, a NULL value or a NULL count.
+ DCHECK(view.null_map == nullptr || (*view.null_map)[index] == 0);
+ DCHECK(view.value_null_map == nullptr || (*view.value_null_map)[index] == 0);
Review Comment:
Fixed in e4acad346dd. `check_entry()` runs before any comparison or consumption and returns `INVALID_ARGUMENT` when an entry, its `value` or its `count` is NULL. NULL top-level lists are still accepted as empty states (`order_qt_merge_all_null`), and the NULL-value case is in the regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4349530794
##########
be/src/exprs/lambda_function/varray_sorted_merge_function.cpp:
##########
@@ -0,0 +1,215 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <string>
+#include <utility>
+
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "exprs/lambda_function/lambda_execution_context.h"
+#include "exprs/lambda_function/lambda_function.h"
+#include "exprs/lambda_function/lambda_function_factory.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vlambda_function_expr.h"
+
+namespace doris {
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc)
+ *
+ * Merges two arrays that are already sorted by the elements' natural order. The merge lambda
+ * receives the two current candidates and returns the combined element; returning NULL means the
+ * two elements must not be combined, so both are kept. Each input key occurs at most once per
+ * array, so a key appears at most twice across the two arrays and the lambda is called at most
+ * once per key.
+ */
+class ArraySortedMergeFunction : public LambdaFunction {
+ ENABLE_FACTORY_CREATOR(ArraySortedMergeFunction);
+
+public:
+ ~ArraySortedMergeFunction() override = default;
+
+ static constexpr auto name = "array_sorted_merge";
+
+ static LambdaFunctionPtr create() { return std::make_shared<ArraySortedMergeFunction>(); }
+
+ std::string get_name() const override { return name; }
+
+ Status execute(VExprContext* context, const Block* block, const Selector* expr_selector,
+ size_t count, ColumnPtr& result_column, const DataTypePtr& result_type,
+ const VExprSPtrs& children) const override {
+ DCHECK_EQ(children.size(), 4);
+
+ const ColumnArray* arrays[2] = {nullptr, nullptr};
+ const IColumn* data[2] = {nullptr, nullptr};
+ DataTypePtr nested_types[2];
+ ColumnPtr holders[2];
+ const ColumnUInt8::Container* nullmaps[2] = {nullptr, nullptr};
+ for (int i = 0; i < 2; ++i) {
+ RETURN_IF_ERROR(children[i + 1]->execute_column(context, block, expr_selector, count,
+ holders[i]));
+ holders[i] = holders[i]->convert_to_full_column_if_const();
+ DataTypePtr type = children[i + 1]->execute_type(block);
+ const IColumn* array_column = holders[i].get();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(array_column)) {
+ nullmaps[i] = &nullable->get_null_map_column().get_data();
+ array_column = &nullable->get_nested_column();
+ type = assert_cast<const DataTypeNullable&>(*type).get_nested_type();
+ }
+ arrays[i] = assert_cast<const ColumnArray*>(array_column);
+ data[i] = arrays[i]->get_data_ptr().get();
+ nested_types[i] = assert_cast<const DataTypeArray&>(*type).get_nested_type();
+ }
+
+ ColumnPtr flag_holder;
+ RETURN_IF_ERROR(
+ children[3]->execute_column(context, block, expr_selector, count, flag_holder));
+ flag_holder = flag_holder->convert_to_full_column_if_const();
+ const IColumn* flag_column = flag_holder.get();
+ const ColumnUInt8::Container* flag_nullmap = nullptr;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(flag_column)) {
+ flag_nullmap = &nullable->get_null_map_column().get_data();
+ flag_column = &nullable->get_nested_column();
+ }
+ const auto& flags = assert_cast<const ColumnUInt8&>(*flag_column).get_data();
+
+ const size_t rows = arrays[0]->size();
+ auto result_nested = nested_types[0]->create_column();
+ auto result_offsets = ColumnArray::ColumnOffsets::create();
+ auto& offsets_data = result_offsets->get_data();
+ offsets_data.reserve(rows);
+ auto result_nullmap = ColumnUInt8::create(rows, 0);
+ auto& result_nullmap_data = result_nullmap->get_data();
+
+ // Two one-row columns hold the lambda arguments of the current merge step. They carry
+ // the lambda argument names because the column refs resolve by name when no
+ // position-based gap is set.
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ const std::vector<std::string>* argument_names =
+ lambda_expr->has_argument_names() ? &lambda_expr->argument_names() : nullptr;
+ Block lambda_block;
+ for (int i = 0; i < 2; ++i) {
+ const std::string argument_name =
+ argument_names != nullptr && i < argument_names->size()
+ ? (*argument_names)[i]
+ : "merge_arg_" + std::to_string(i);
+ lambda_block.insert(ColumnWithTypeAndName(nested_types[i]->create_column(),
+ nested_types[i], argument_name));
+ }
+ // The lambda arguments are bound by position (column ids 0 and 1), like array_sort's
+ // comparator, so the body reads these two one-row columns.
+ IColumn* argument_columns[2] = {
+ lambda_block.get_by_position(0).column->assert_mutable().get(),
+ lambda_block.get_by_position(1).column->assert_mutable().get()};
+
+ LambdaExecutionContext::Frame lambda_frame;
+ lambda_frame.bind_by_name = false;
+ lambda_frame.parent_bindings_visible = false;
+ LambdaExecutionContext::FrameGuard lambda_frame_guard(context->lambda_execution_context(),
+ std::move(lambda_frame));
+
+ const auto& offsets0 = arrays[0]->get_offsets();
+ const auto& offsets1 = arrays[1]->get_offsets();
+ const int lambda_result_base = static_cast<int>(lambda_block.columns());
+
+ for (size_t row = 0; row < rows; ++row) {
+ const bool row_is_null = (nullmaps[0] != nullptr && (*nullmaps[0])[row] != 0) ||
+ (nullmaps[1] != nullptr && (*nullmaps[1])[row] != 0);
+ if (row_is_null) {
+ result_nullmap_data[row] = 1;
+ offsets_data.push_back(result_nested->size());
+ continue;
+ }
+ if (flag_nullmap != nullptr && (*flag_nullmap)[row] != 0) {
+ return Status::InvalidArgument(
+ "array_sorted_merge requires a non-null ascending flag");
+ }
+ const bool ascending = flags[row] != 0;
+
+ size_t left = row == 0 ? 0 : offsets0[row - 1];
+ const size_t left_end = offsets0[row];
+ size_t right = row == 0 ? 0 : offsets1[row - 1];
+ const size_t right_end = offsets1[row];
+
+ while (left < left_end || right < right_end) {
+ if (left == left_end) {
+ result_nested->insert_from(*data[1], right++);
+ continue;
+ }
+ if (right == right_end) {
+ result_nested->insert_from(*data[0], left++);
+ continue;
+ }
+
+ argument_columns[0]->clear();
+ argument_columns[0]->insert_from(*data[0], left);
+ argument_columns[1]->clear();
+ argument_columns[1]->insert_from(*data[1], right);
+
+ int lambda_res_id = lambda_result_base;
+ RETURN_IF_ERROR(children[0]->execute(context, &lambda_block, &lambda_res_id));
Review Comment:
[P1] Bind captured inputs before executing the merge lambda
The FE accepts enclosing slots in the merge lambda, but this executes them against `lambda_block`, which contains only `a` and `b`. For a reduced input plan `Project(array_sorted_merge((a,b)->IF(a=b AND a<lim,a,NULL),l,r,true)) -> Scan(lim INT,l ARRAY<INT>,r ARRAY<INT>)`, the row `lim=2,l=[1],r=[1]` should return `[1]`. The captured nullable INT `lim` retains input column index 0, so `VSlotRef` reads candidate `a=1` instead; the predicate becomes `a=b AND a<a`, and the result is `[1,1]`. With `lim` at index 2 the same path instead reports a missing-column error. Materialize and rebind current-row captures and enclosing lambda bindings, or reject unsupported captures during prepare as `array_sort` does. Add table-column and outer-lambda capture regressions.
##########
be/src/exprs/lambda_function/varray_sorted_merge_function.cpp:
##########
@@ -0,0 +1,215 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <string>
+#include <utility>
+
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "exprs/lambda_function/lambda_execution_context.h"
+#include "exprs/lambda_function/lambda_function.h"
+#include "exprs/lambda_function/lambda_function_factory.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vlambda_function_expr.h"
+
+namespace doris {
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc)
+ *
+ * Merges two arrays that are already sorted by the elements' natural order. The merge lambda
+ * receives the two current candidates and returns the combined element; returning NULL means the
+ * two elements must not be combined, so both are kept. Each input key occurs at most once per
+ * array, so a key appears at most twice across the two arrays and the lambda is called at most
+ * once per key.
+ */
+class ArraySortedMergeFunction : public LambdaFunction {
+ ENABLE_FACTORY_CREATOR(ArraySortedMergeFunction);
+
+public:
+ ~ArraySortedMergeFunction() override = default;
+
+ static constexpr auto name = "array_sorted_merge";
+
+ static LambdaFunctionPtr create() { return std::make_shared<ArraySortedMergeFunction>(); }
+
+ std::string get_name() const override { return name; }
+
+ Status execute(VExprContext* context, const Block* block, const Selector* expr_selector,
+ size_t count, ColumnPtr& result_column, const DataTypePtr& result_type,
+ const VExprSPtrs& children) const override {
+ DCHECK_EQ(children.size(), 4);
+
+ const ColumnArray* arrays[2] = {nullptr, nullptr};
+ const IColumn* data[2] = {nullptr, nullptr};
+ DataTypePtr nested_types[2];
+ ColumnPtr holders[2];
+ const ColumnUInt8::Container* nullmaps[2] = {nullptr, nullptr};
+ for (int i = 0; i < 2; ++i) {
+ RETURN_IF_ERROR(children[i + 1]->execute_column(context, block, expr_selector, count,
+ holders[i]));
+ holders[i] = holders[i]->convert_to_full_column_if_const();
+ DataTypePtr type = children[i + 1]->execute_type(block);
+ const IColumn* array_column = holders[i].get();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(array_column)) {
+ nullmaps[i] = &nullable->get_null_map_column().get_data();
+ array_column = &nullable->get_nested_column();
+ type = assert_cast<const DataTypeNullable&>(*type).get_nested_type();
+ }
+ arrays[i] = assert_cast<const ColumnArray*>(array_column);
+ data[i] = arrays[i]->get_data_ptr().get();
+ nested_types[i] = assert_cast<const DataTypeArray&>(*type).get_nested_type();
+ }
+
+ ColumnPtr flag_holder;
+ RETURN_IF_ERROR(
+ children[3]->execute_column(context, block, expr_selector, count, flag_holder));
+ flag_holder = flag_holder->convert_to_full_column_if_const();
+ const IColumn* flag_column = flag_holder.get();
+ const ColumnUInt8::Container* flag_nullmap = nullptr;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(flag_column)) {
+ flag_nullmap = &nullable->get_null_map_column().get_data();
+ flag_column = &nullable->get_nested_column();
+ }
+ const auto& flags = assert_cast<const ColumnUInt8&>(*flag_column).get_data();
+
+ const size_t rows = arrays[0]->size();
+ auto result_nested = nested_types[0]->create_column();
+ auto result_offsets = ColumnArray::ColumnOffsets::create();
+ auto& offsets_data = result_offsets->get_data();
+ offsets_data.reserve(rows);
+ auto result_nullmap = ColumnUInt8::create(rows, 0);
+ auto& result_nullmap_data = result_nullmap->get_data();
+
+ // Two one-row columns hold the lambda arguments of the current merge step. They carry
+ // the lambda argument names because the column refs resolve by name when no
+ // position-based gap is set.
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ const std::vector<std::string>* argument_names =
+ lambda_expr->has_argument_names() ? &lambda_expr->argument_names() : nullptr;
+ Block lambda_block;
+ for (int i = 0; i < 2; ++i) {
+ const std::string argument_name =
+ argument_names != nullptr && i < argument_names->size()
+ ? (*argument_names)[i]
+ : "merge_arg_" + std::to_string(i);
+ lambda_block.insert(ColumnWithTypeAndName(nested_types[i]->create_column(),
+ nested_types[i], argument_name));
+ }
+ // The lambda arguments are bound by position (column ids 0 and 1), like array_sort's
+ // comparator, so the body reads these two one-row columns.
+ IColumn* argument_columns[2] = {
+ lambda_block.get_by_position(0).column->assert_mutable().get(),
+ lambda_block.get_by_position(1).column->assert_mutable().get()};
+
+ LambdaExecutionContext::Frame lambda_frame;
+ lambda_frame.bind_by_name = false;
+ lambda_frame.parent_bindings_visible = false;
+ LambdaExecutionContext::FrameGuard lambda_frame_guard(context->lambda_execution_context(),
+ std::move(lambda_frame));
+
+ const auto& offsets0 = arrays[0]->get_offsets();
+ const auto& offsets1 = arrays[1]->get_offsets();
+ const int lambda_result_base = static_cast<int>(lambda_block.columns());
+
+ for (size_t row = 0; row < rows; ++row) {
+ const bool row_is_null = (nullmaps[0] != nullptr && (*nullmaps[0])[row] != 0) ||
+ (nullmaps[1] != nullptr && (*nullmaps[1])[row] != 0);
+ if (row_is_null) {
+ result_nullmap_data[row] = 1;
+ offsets_data.push_back(result_nested->size());
+ continue;
+ }
+ if (flag_nullmap != nullptr && (*flag_nullmap)[row] != 0) {
+ return Status::InvalidArgument(
+ "array_sorted_merge requires a non-null ascending flag");
+ }
+ const bool ascending = flags[row] != 0;
+
+ size_t left = row == 0 ? 0 : offsets0[row - 1];
+ const size_t left_end = offsets0[row];
+ size_t right = row == 0 ? 0 : offsets1[row - 1];
+ const size_t right_end = offsets1[row];
+
+ while (left < left_end || right < right_end) {
+ if (left == left_end) {
+ result_nested->insert_from(*data[1], right++);
+ continue;
+ }
+ if (right == right_end) {
+ result_nested->insert_from(*data[0], left++);
+ continue;
+ }
+
+ argument_columns[0]->clear();
+ argument_columns[0]->insert_from(*data[0], left);
+ argument_columns[1]->clear();
+ argument_columns[1]->insert_from(*data[1], right);
+
+ int lambda_res_id = lambda_result_base;
+ RETURN_IF_ERROR(children[0]->execute(context, &lambda_block, &lambda_res_id));
+ ColumnPtr merged = lambda_block.get_by_position(lambda_res_id)
+ .column->convert_to_full_column_if_const();
+ const IColumn* merged_column = merged.get();
+ bool merged_is_null = false;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(merged_column)) {
+ merged_is_null = nullable->is_null_at(0);
+ }
+ if (!merged_is_null) {
+ result_nested->insert_from(*merged_column, 0);
Review Comment:
[P1] Wrap nonnullable lambda results before inserting array elements
`SELECT array_sorted_merge((a,b)->COALESCE(a,b,CAST(0 AS INT)), CAST([1,2] AS ARRAY<INT>), CAST([1,2] AS ARRAY<INT>), true)` passes FE type validation and should return `[1,2]`. The fallback makes the body nonnullable, so `VectorizedCoalesceExpr` returns a raw `ColumnInt32`; `VLambdaFunctionExpr` preserves that body type, and materializing constants does not add a nullable wrapper. However, `result_nested` is `ColumnNullable(Int32)` because `DataTypeArray` always makes its elements nullable. This call therefore reaches `ColumnNullable::insert_from` with a nonnullable source and fails `assert_cast` (`FATAL_ERROR`, or `LOG(FATAL)` without `NDEBUG`). Normalize the lambda result to the nullable item representation before inserting it, and cover nonnullable COALESCE/IFNULL or constant bodies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6345694823
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6349753031
<details>
<summary>TPC-DS: <b>Total hot run time: 150976 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d, data reload: false
query5 4304 574 428 428
query6 442 202 192 192
query7 4814 446 229 229
query8 330 165 153 153
query9 8737 3076 3072 3072
query10 474 312 249 249
query11 5855 3879 3613 3613
query12 159 90 89 89
query13 1253 457 328 328
query14 6147 4007 3675 3675
query14_1 3469 3455 3439 3439
query15 217 199 184 184
query16 978 440 430 430
query17 886 660 538 538
query18 2427 426 319 319
query19 199 169 134 134
query20 86 89 86 86
query21 210 131 109 109
query22 12911 13138 12789 12789
query23 13628 12761 12511 12511
query23_1 12540 12623 12626 12623
query24 7416 824 480 480
query24_1 479 462 501 462
query25 544 424 375 375
query26 1248 260 142 142
query27 2748 452 273 273
query28 4592 1961 1955 1955
query29 1567 609 459 459
query30 302 222 189 189
query31 998 929 823 823
query32 151 94 114 94
query33 539 317 250 250
query34 880 850 499 499
query35 742 762 658 658
query36 839 801 724 724
query37 123 104 86 86
query38 1808 1780 1658 1658
query39 789 783 739 739
query39_1 745 724 743 724
query40 215 119 101 101
query41 71 69 68 68
query42 92 89 86 86
query43 353 367 314 314
query44 1286 684 688 684
query45 182 176 186 176
query46 838 965 558 558
query47 2999 2858 2900 2858
query48 288 306 217 217
query49 578 396 295 295
query50 686 276 215 215
query51 10471 10488 10276 10276
query52 91 85 69 69
query53 183 205 148 148
query54 234 195 189 189
query55 75 68 68 68
query56 238 211 221 211
query57 1663 1638 1554 1554
query58 287 269 267 267
query59 2103 2129 1930 1930
query60 294 250 232 232
query61 153 150 164 150
query62 387 345 301 301
query63 181 158 158 158
query64 2746 952 786 786
query65 3327 3294 3295 3294
query66 1786 420 312 312
query67 20163 20044 19949 19949
query68 2964 1013 605 605
query69 372 287 255 255
query70 828 758 768 758
query71 302 250 208 208
query72 2752 2680 2234 2234
query73 515 524 293 293
query74 4618 4563 4339 4339
query75 2359 2286 1926 1926
query76 2314 1005 593 593
query77 355 441 269 269
query78 9806 9735 9136 9136
query79 997 842 515 515
query80 1257 439 323 323
query81 596 315 278 278
query82 594 131 114 114
query83 309 217 177 177
query84 317 121 104 104
query85 859 427 374 374
query86 400 241 232 232
query87 2000 1926 1812 1812
query88 3695 2664 2635 2635
query89 336 281 241 241
query90 1918 174 165 165
query91 155 153 130 130
query92 100 90 82 82
query93 1055 988 573 573
query94 649 318 268 268
query95 588 398 307 307
query96 676 505 239 239
query97 2448 2459 2345 2345
query98 161 153 153 153
query99 776 755 648 648
Total cold run time: 235410 ms
Total hot run time: 150976 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6345392163
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6359781700
<details>
<summary>TPC-DS: <b>Total hot run time: 150222 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 48370febac57e0391be119b675afc90cfdba098d, data reload: false
query5 4303 570 435 435
query6 430 206 196 196
query7 4804 453 222 222
query8 330 172 155 155
query9 8794 3088 3117 3088
query10 474 322 269 269
query11 6017 3861 3555 3555
query12 152 92 88 88
query13 1260 413 316 316
query14 6094 3903 3664 3664
query14_1 3428 3404 3391 3391
query15 235 196 174 174
query16 945 428 411 411
query17 863 646 522 522
query18 2422 410 307 307
query19 198 159 131 131
query20 82 81 80 80
query21 214 126 101 101
query22 13139 12946 12761 12761
query23 13534 12944 12683 12683
query23_1 12697 12676 12461 12461
query24 7380 832 458 458
query24_1 467 465 478 465
query25 515 400 340 340
query26 1261 250 135 135
query27 2773 437 266 266
query28 4646 1924 1912 1912
query29 1559 589 425 425
query30 303 218 187 187
query31 1007 937 815 815
query32 141 84 89 84
query33 520 329 234 234
query34 899 841 497 497
query35 739 749 640 640
query36 793 792 704 704
query37 117 96 83 83
query38 1782 1732 1653 1653
query39 776 798 747 747
query39_1 737 727 735 727
query40 253 111 93 93
query41 77 63 62 62
query42 88 83 80 80
query43 341 366 315 315
query44 1334 661 697 661
query45 178 185 163 163
query46 807 963 563 563
query47 2856 2970 2842 2842
query48 293 322 213 213
query49 579 400 299 299
query50 675 269 220 220
query51 10621 10353 10216 10216
query52 78 79 69 69
query53 184 204 149 149
query54 238 194 193 193
query55 70 65 64 64
query56 231 222 215 215
query57 1654 1625 1568 1568
query58 282 258 253 253
query59 2069 2098 1903 1903
query60 296 240 221 221
query61 149 143 143 143
query62 388 327 279 279
query63 183 158 154 154
query64 2775 1043 844 844
query65 3304 3268 3245 3245
query66 1782 408 299 299
query67 20081 20021 19816 19816
query68 2972 989 612 612
query69 433 290 260 260
query70 872 756 759 756
query71 294 237 204 204
query72 2717 2581 2280 2280
query73 511 525 293 293
query74 4644 4518 4353 4353
query75 2348 2274 1913 1913
query76 2298 1009 587 587
query77 350 383 266 266
query78 9873 9637 9091 9091
query79 886 871 505 505
query80 465 419 355 355
query81 518 333 267 267
query82 203 142 103 103
query83 203 198 181 181
query84 287 121 99 99
query85 770 410 365 365
query86 280 237 232 232
query87 1984 1972 1806 1806
query88 3564 2673 2673 2673
query89 319 282 244 244
query90 2110 175 176 175
query91 170 161 124 124
query92 97 88 81 81
query93 993 1003 578 578
query94 446 284 281 281
query95 588 412 304 304
query96 663 489 231 231
query97 2450 2424 2325 2325
query98 161 150 145 145
query99 754 768 652 652
Total cold run time: 233660 ms
Total hot run time: 150222 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6432149612
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6433028992
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6433029674
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6434371587
<details>
<summary>TPC-H: <b>Total hot run time: 28580 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5309c29db2b57c559711d38b6621f5c934c95d3a, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17610 5216 5102 5102
q2 2134 335 288 288
q3 10176 1386 816 816
q4 4676 466 337 337
q5 7463 790 546 546
q6 179 184 152 152
q7 755 785 596 596
q8 9261 1323 1151 1151
q9 5541 4211 4204 4204
q10 6839 1342 1011 1011
q11 447 246 236 236
q12 641 396 279 279
q13 18093 3606 3007 3007
q14 275 269 247 247
q15 q16 737 718 662 662
q17 1208 757 593 593
q18 7017 6185 6133 6133
q19 1121 955 615 615
q20 389 341 225 225
q21 5054 2129 2270 2129
q22 348 302 251 251
Total cold run time: 99964 ms
Total hot run time: 28580 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5598 5613 5620 5613
q2 677 581 555 555
q3 5213 5405 4994 4994
q4 1988 2157 1518 1518
q5 5283 5168 5697 5168
q6 260 206 156 156
q7 2370 2149 1936 1936
q8 3186 2933 2712 2712
q9 8083 8050 8135 8050
q10 3742 3632 3278 3278
q11 611 436 404 404
q12 692 724 539 539
q13 3679 3872 3243 3243
q14 307 323 301 301
q15 q16 727 728 665 665
q17 8322 7545 6956 6956
q18 13006 12154 12920 12154
q19 935 812 845 812
q20 2124 2140 1904 1904
q21 5648 4578 4680 4578
q22 517 459 440 440
Total cold run time: 72968 ms
Total hot run time: 65976 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6365848693
<details>
<summary>TPC-H: <b>Total hot run time: 28151 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e250347cd120427a408a4db7a0a55e83da7b098f, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17592 5141 5016 5016
q2 2131 331 238 238
q3 10221 1362 825 825
q4 4677 456 328 328
q5 7429 783 518 518
q6 179 191 155 155
q7 732 779 588 588
q8 9250 1331 993 993
q9 5512 4195 4173 4173
q10 6832 1277 1038 1038
q11 436 260 226 226
q12 638 405 272 272
q13 18096 3555 2986 2986
q14 273 265 247 247
q15 q16 722 705 660 660
q17 1179 750 593 593
q18 7029 6175 6115 6115
q19 1094 941 587 587
q20 371 326 224 224
q21 4877 2110 2275 2110
q22 358 294 259 259
Total cold run time: 99628 ms
Total hot run time: 28151 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5596 5571 5560 5560
q2 687 573 552 552
q3 5198 5364 4972 4972
q4 1977 2126 1542 1542
q5 5243 5139 5319 5139
q6 265 223 146 146
q7 2339 2034 1801 1801
q8 3022 2712 2686 2686
q9 7991 7901 7990 7901
q10 3697 3641 3309 3309
q11 617 469 491 469
q12 671 709 501 501
q13 3539 3864 3196 3196
q14 325 318 300 300
q15 q16 736 720 640 640
q17 8294 7488 6862 6862
q18 12986 12150 12897 12150
q19 902 762 779 762
q20 2110 2136 1910 1910
q21 5540 4532 4633 4532
q22 504 439 404 404
Total cold run time: 72239 ms
Total hot run time: 65334 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6379846994
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6416296889
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6432418606
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4384698012
##########
be/src/exprs/lambda_function/varray_sorted_merge_function.cpp:
##########
@@ -0,0 +1,377 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+#include <map>
+#include <memory>
+#include <set>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "exprs/lambda_function/lambda_execution_context.h"
+#include "exprs/lambda_function/lambda_function.h"
+#include "exprs/lambda_function/lambda_function_factory.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vlambda_function_expr.h"
+
+namespace doris {
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc)
+ *
+ * Merges two arrays that are already sorted by the elements' natural order. The merge lambda
+ * receives the two current candidates and returns the combined element; returning NULL means the
+ * two elements must not be combined, so both are kept. Each input key occurs at most once per
+ * array, so a key appears at most twice across the two arrays and the lambda is called at most
+ * once per key.
+ */
+class ArraySortedMergeFunction : public LambdaFunction {
+ ENABLE_FACTORY_CREATOR(ArraySortedMergeFunction);
+
+public:
+ ~ArraySortedMergeFunction() override = default;
+
+ static constexpr auto name = "array_sorted_merge";
+
+ static LambdaFunctionPtr create() { return std::make_shared<ArraySortedMergeFunction>(); }
+
+ std::string get_name() const override { return name; }
+
+ /// Rejects a nested lambda that captures one of this lambda's arguments: a merge argument
+ /// stands for the candidate of one comparison step, which is not part of the supported
+ /// contract for a nested lambda. Nested lambdas may still use their own arguments
+ /// (shadowing) and may capture enclosing lambda arguments.
+ static Status validate_nested_captures(const VExprSPtr& expr,
+ const std::vector<std::string>& argument_names,
+ bool inside_nested_lambda,
+ std::vector<std::string> shadowed_names) {
+ if (expr->is_column_ref()) {
+ const std::string& column_name = expr->expr_name();
+ if (std::ranges::find(shadowed_names, column_name) != shadowed_names.end()) {
+ return Status::OK();
+ }
+ if (inside_nested_lambda &&
+ std::ranges::find(argument_names, column_name) != argument_names.end()) {
+ return Status::NotSupported(
+ "array_sorted_merge merge lambda argument '{}' cannot be captured by a "
+ "nested lambda",
+ column_name);
+ }
+ return Status::OK();
+ }
+ if (expr->node_type() == TExprNodeType::LAMBDA_FUNCTION_EXPR) {
+ const auto* lambda = assert_cast<const VLambdaFunctionExpr*>(expr.get());
+ if (lambda->has_argument_names()) {
+ for (const auto& argument_name : lambda->argument_names()) {
+ shadowed_names.emplace_back(argument_name);
+ }
+ }
+ return validate_nested_captures(lambda->get_child(0), argument_names, true,
+ std::move(shadowed_names));
+ }
+ for (const auto& child : expr->children()) {
+ RETURN_IF_ERROR(validate_nested_captures(child, argument_names, inside_nested_lambda,
+ shadowed_names));
+ }
+ return Status::OK();
+ }
+
+ Status prepare(RuntimeState* state, const VExprSPtrs& children) override {
+ RETURN_IF_ERROR(LambdaFunction::prepare(state, children));
+ DCHECK_EQ(children.size(), 4);
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ if (!lambda_expr->has_argument_names()) {
+ return Status::OK();
+ }
+ // A nested lambda that captures one of this lambda's arguments has no defined meaning
+ // (the argument is the candidate of a single comparison step); reject it during prepare
+ // instead of resolving it against some column at runtime.
+ return validate_nested_captures(children[0]->get_child(0), lambda_expr->argument_names(),
+ false, {});
+ }
+
+ Status execute(VExprContext* context, const Block* block, const Selector* expr_selector,
+ size_t count, ColumnPtr& result_column, const DataTypePtr& result_type,
+ const VExprSPtrs& children) const override {
+ DCHECK_EQ(children.size(), 4);
+
+ const ColumnArray* arrays[2] = {nullptr, nullptr};
+ const IColumn* data[2] = {nullptr, nullptr};
+ DataTypePtr nested_types[2];
+ ColumnPtr holders[2];
+ const ColumnUInt8::Container* nullmaps[2] = {nullptr, nullptr};
+ for (int i = 0; i < 2; ++i) {
+ RETURN_IF_ERROR(children[i + 1]->execute_column(context, block, expr_selector, count,
+ holders[i]));
+ holders[i] = holders[i]->convert_to_full_column_if_const();
+ DataTypePtr type = children[i + 1]->execute_type(block);
+ const IColumn* array_column = holders[i].get();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(array_column)) {
+ nullmaps[i] = &nullable->get_null_map_column().get_data();
+ array_column = &nullable->get_nested_column();
+ type = assert_cast<const DataTypeNullable&>(*type).get_nested_type();
+ }
+ arrays[i] = assert_cast<const ColumnArray*>(array_column);
+ data[i] = arrays[i]->get_data_ptr().get();
+ nested_types[i] = assert_cast<const DataTypeArray&>(*type).get_nested_type();
+ }
+
+ ColumnPtr flag_holder;
+ RETURN_IF_ERROR(
+ children[3]->execute_column(context, block, expr_selector, count, flag_holder));
+ flag_holder = flag_holder->convert_to_full_column_if_const();
+ const IColumn* flag_column = flag_holder.get();
+ const ColumnUInt8::Container* flag_nullmap = nullptr;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(flag_column)) {
+ flag_nullmap = &nullable->get_null_map_column().get_data();
+ flag_column = &nullable->get_nested_column();
+ }
+ const auto& flags = assert_cast<const ColumnUInt8&>(*flag_column).get_data();
+
+ const size_t rows = arrays[0]->size();
+ auto result_nested = nested_types[0]->create_column();
+ auto result_offsets = ColumnArray::ColumnOffsets::create();
+ auto& offsets_data = result_offsets->get_data();
+ offsets_data.reserve(rows);
+ auto result_nullmap = ColumnUInt8::create(rows, 0);
+ auto& result_nullmap_data = result_nullmap->get_data();
+
+ // The lambda may capture enclosing columns (for example a table column compared against
+ // the candidates). Materialize every captured input at its original position in the
+ // one-row lambda block, so slot refs read the current row; the two merge candidates
+ // take the positions after the last captured column and are bound by name.
+ std::set<int> captured_ids;
+ children[0]->collect_slot_column_ids(captured_ids);
+ // An enclosing lambda's argument is a name reference rather than a slot, and this
+ // function executes its lambda against a block of its own that hides the parent frames.
+ // Publish the visible binding values under their names in that block, so the reference
+ // resolves by name like the merge arguments do.
+ std::vector<LambdaExecutionContext::Binding> enclosing_bindings;
+ context->lambda_execution_context().collect_visible_bindings(enclosing_bindings);
+ if ((!captured_ids.empty() || !enclosing_bindings.empty()) && block == nullptr) {
+ return Status::InternalError(
+ "array_sorted_merge lambda references outer columns but has no input block");
+ }
+ std::map<int, ColumnPtr> captured_columns;
+ for (int id : captured_ids) {
+ if (id < 0 || static_cast<size_t>(id) >= block->columns()) {
+ return Status::InternalError(
+ "array_sorted_merge lambda input column id {} is outside input block", id);
+ }
+ captured_columns[id] =
+ block->get_by_position(id).column->convert_to_full_column_if_const();
Review Comment:
[P2] Materialize virtual captures before copying the lambda inputs
With `experimental_enable_virtual_slot_for_cse=true`, a DUP table `t(lim INT NOT NULL, l ARRAY<INT>, r ARRAY<INT>)` containing `(2,[1],[1])` should return 2 for:
```sql
SELECT lim FROM t
WHERE abs(lim)>0 AND abs(lim)<10
AND cardinality(array_sorted_merge(
(a,b)->IF(a=b AND a<abs(lim),a,NULL),l,r,true))=1;
```
`PushDownVirtualColumnsIntoOlapScan` extracts the repeated expression from the ordinary conjuncts, then replaces it inside the lambda too. The reduced rewritten tree inferred from that rule is:
```text
Filter(v>0, v<10, cardinality(merge((a,b)->IF(a=b AND a<v,a,NULL),l,r))=1)
OlapScan(t; virtual v=abs(lim))
```
For a pushed scan filter, `SegmentIterator::_process_common_expr` runs before final virtual-column materialization, so v can still be `ColumnNothing`. `VirtualSlotRef::collect_slot_column_ids` records only v, excluding its hidden lim dependency. This copy therefore preserves the lazy placeholder, while the private block fills lim's position with `UInt8 temp`. When `VirtualSlotRef` lazily evaluates `abs(lim)`, it reads that wrong type and `FunctionUnaryArithmetic` returns an expected-data-type error instead of the matching row. Earlier virtual comparisons do not cache their result in the block. Materialize virtual captures against the original block before cutting them, or preserve their transitive dependencies and types, and add a pushed-filter CSE regression. This is distinct from the repaired ordinary-slot capture thread: the missing inputs belong to an out-of-tree virtual expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6465864272
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4399781699
##########
be/src/exprs/lambda_function/varray_sorted_merge_function.cpp:
##########
@@ -0,0 +1,399 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+#include <map>
+#include <memory>
+#include <set>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "exprs/lambda_function/lambda_execution_context.h"
+#include "exprs/lambda_function/lambda_function.h"
+#include "exprs/lambda_function/lambda_function_factory.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/virtual_slot_ref.h"
+#include "exprs/vlambda_function_expr.h"
+
+namespace doris {
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc)
+ *
+ * Merges two arrays that are already sorted by the elements' natural order. The merge lambda
+ * receives the two current candidates and returns the combined element; returning NULL means the
+ * two elements must not be combined, so both are kept. Each input key occurs at most once per
+ * array, so a key appears at most twice across the two arrays and the lambda is called at most
+ * once per key.
+ */
+class ArraySortedMergeFunction : public LambdaFunction {
+ ENABLE_FACTORY_CREATOR(ArraySortedMergeFunction);
+
+public:
+ ~ArraySortedMergeFunction() override = default;
+
+ static constexpr auto name = "array_sorted_merge";
+
+ static LambdaFunctionPtr create() { return std::make_shared<ArraySortedMergeFunction>(); }
+
+ std::string get_name() const override { return name; }
+
+ /// Rejects a nested lambda that captures one of this lambda's arguments: a merge argument
+ /// stands for the candidate of one comparison step, which is not part of the supported
+ /// contract for a nested lambda. Nested lambdas may still use their own arguments
+ /// (shadowing) and may capture enclosing lambda arguments.
+ static Status validate_nested_captures(const VExprSPtr& expr,
+ const std::vector<std::string>& argument_names,
+ bool inside_nested_lambda,
+ std::vector<std::string> shadowed_names) {
+ if (expr->is_column_ref()) {
+ const std::string& column_name = expr->expr_name();
+ if (std::ranges::find(shadowed_names, column_name) != shadowed_names.end()) {
+ return Status::OK();
+ }
+ if (inside_nested_lambda &&
+ std::ranges::find(argument_names, column_name) != argument_names.end()) {
+ return Status::NotSupported(
+ "array_sorted_merge merge lambda argument '{}' cannot be captured by a "
+ "nested lambda",
+ column_name);
+ }
+ return Status::OK();
+ }
+ if (expr->node_type() == TExprNodeType::LAMBDA_FUNCTION_EXPR) {
+ const auto* lambda = assert_cast<const VLambdaFunctionExpr*>(expr.get());
+ if (lambda->has_argument_names()) {
+ for (const auto& argument_name : lambda->argument_names()) {
+ shadowed_names.emplace_back(argument_name);
+ }
+ }
+ return validate_nested_captures(lambda->get_child(0), argument_names, true,
+ std::move(shadowed_names));
+ }
+ for (const auto& child : expr->children()) {
+ RETURN_IF_ERROR(validate_nested_captures(child, argument_names, inside_nested_lambda,
+ shadowed_names));
+ }
+ return Status::OK();
+ }
+
+ /// A virtual column is materialized lazily on first read, from an expression over its own
+ /// input columns; its block entry starts as a ColumnNothing placeholder and the virtual slot
+ /// only records its own column id. Copying just that id into the private lambda block leaves
+ /// the inputs behind the padded "temp" columns, so the lazy evaluation reads a wrong-typed
+ /// column. Collect the virtual expressions' inputs as well, so they are copied with their
+ /// real types and the evaluation reads the current row.
+ static void collect_virtual_capture_inputs(const VExprSPtr& expr, std::set<int>& column_ids) {
+ if (expr->node_type() == TExprNodeType::VIRTUAL_SLOT_REF) {
+ const auto* virtual_slot = assert_cast<const VirtualSlotRef*>(expr.get());
+ const auto virtual_expr = virtual_slot->get_virtual_column_expr();
+ if (virtual_expr != nullptr) {
+ virtual_expr->collect_slot_column_ids(column_ids);
+ collect_virtual_capture_inputs(virtual_expr, column_ids);
+ }
+ }
+ for (const auto& child : expr->children()) {
+ collect_virtual_capture_inputs(child, column_ids);
+ }
+ }
+
+ Status prepare(RuntimeState* state, const VExprSPtrs& children) override {
+ RETURN_IF_ERROR(LambdaFunction::prepare(state, children));
+ DCHECK_EQ(children.size(), 4);
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ if (!lambda_expr->has_argument_names()) {
+ return Status::OK();
+ }
+ // A nested lambda that captures one of this lambda's arguments has no defined meaning
+ // (the argument is the candidate of a single comparison step); reject it during prepare
+ // instead of resolving it against some column at runtime.
+ return validate_nested_captures(children[0]->get_child(0), lambda_expr->argument_names(),
+ false, {});
+ }
+
+ Status execute(VExprContext* context, const Block* block, const Selector* expr_selector,
+ size_t count, ColumnPtr& result_column, const DataTypePtr& result_type,
+ const VExprSPtrs& children) const override {
+ DCHECK_EQ(children.size(), 4);
+
+ const ColumnArray* arrays[2] = {nullptr, nullptr};
+ const IColumn* data[2] = {nullptr, nullptr};
+ DataTypePtr nested_types[2];
+ ColumnPtr holders[2];
+ const ColumnUInt8::Container* nullmaps[2] = {nullptr, nullptr};
+ for (int i = 0; i < 2; ++i) {
+ RETURN_IF_ERROR(children[i + 1]->execute_column(context, block, expr_selector, count,
+ holders[i]));
+ holders[i] = holders[i]->convert_to_full_column_if_const();
+ DataTypePtr type = children[i + 1]->execute_type(block);
+ const IColumn* array_column = holders[i].get();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(array_column)) {
+ nullmaps[i] = &nullable->get_null_map_column().get_data();
+ array_column = &nullable->get_nested_column();
+ type = assert_cast<const DataTypeNullable&>(*type).get_nested_type();
+ }
+ arrays[i] = assert_cast<const ColumnArray*>(array_column);
+ data[i] = arrays[i]->get_data_ptr().get();
+ nested_types[i] = assert_cast<const DataTypeArray&>(*type).get_nested_type();
+ }
+
+ ColumnPtr flag_holder;
+ RETURN_IF_ERROR(
+ children[3]->execute_column(context, block, expr_selector, count, flag_holder));
+ flag_holder = flag_holder->convert_to_full_column_if_const();
+ const IColumn* flag_column = flag_holder.get();
+ const ColumnUInt8::Container* flag_nullmap = nullptr;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(flag_column)) {
+ flag_nullmap = &nullable->get_null_map_column().get_data();
+ flag_column = &nullable->get_nested_column();
+ }
+ const auto& flags = assert_cast<const ColumnUInt8&>(*flag_column).get_data();
+
+ const size_t rows = arrays[0]->size();
+ auto result_nested = nested_types[0]->create_column();
+ auto result_offsets = ColumnArray::ColumnOffsets::create();
+ auto& offsets_data = result_offsets->get_data();
+ offsets_data.reserve(rows);
+ auto result_nullmap = ColumnUInt8::create(rows, 0);
+ auto& result_nullmap_data = result_nullmap->get_data();
+
+ // The lambda may capture enclosing columns (for example a table column compared against
+ // the candidates). Materialize every captured input at its original position in the
+ // one-row lambda block, so slot refs read the current row; the two merge candidates
+ // take the positions after the last captured column and are bound by name.
+ std::set<int> captured_ids;
+ children[0]->collect_slot_column_ids(captured_ids);
+ collect_virtual_capture_inputs(children[0], captured_ids);
+ // An enclosing lambda's argument is a name reference rather than a slot, and this
+ // function executes its lambda against a block of its own that hides the parent frames.
+ // Publish the visible binding values under their names in that block, so the reference
+ // resolves by name like the merge arguments do.
+ std::vector<LambdaExecutionContext::Binding> enclosing_bindings;
+ context->lambda_execution_context().collect_visible_bindings(enclosing_bindings);
+ if ((!captured_ids.empty() || !enclosing_bindings.empty()) && block == nullptr) {
+ return Status::InternalError(
+ "array_sorted_merge lambda references outer columns but has no input block");
+ }
+ std::map<int, ColumnPtr> captured_columns;
+ for (int id : captured_ids) {
+ if (id < 0 || static_cast<size_t>(id) >= block->columns()) {
+ return Status::InternalError(
+ "array_sorted_merge lambda input column id {} is outside input block", id);
+ }
+ captured_columns[id] =
+ block->get_by_position(id).column->convert_to_full_column_if_const();
+ }
+ const int lambda_argument_base = captured_ids.empty() ? 0 : *captured_ids.rbegin() + 1;
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ const std::vector<std::string>* argument_names =
+ lambda_expr->has_argument_names() ? &lambda_expr->argument_names() : nullptr;
+ Block lambda_block;
+ for (int i = 0; i < lambda_argument_base; ++i) {
+ auto captured = captured_columns.find(i);
+ if (captured != captured_columns.end()) {
+ lambda_block.insert(ColumnWithTypeAndName(captured->second->clone_empty(),
+ block->get_by_position(i).type,
+ block->get_by_position(i).name));
+ } else {
+ lambda_block.insert(ColumnWithTypeAndName(
+ ColumnUInt8::create(1, 0), std::make_shared<DataTypeUInt8>(), "temp"));
+ }
+ }
+ // The enclosing bindings go before the merge arguments, and the name lookup scans the
+ // block backwards, so a merge argument with the same name shadows the outer one.
+ std::vector<ColumnPtr> enclosing_columns;
+ std::vector<int> enclosing_positions;
+ for (const auto& binding : enclosing_bindings) {
+ if (binding.column_position < 0 ||
+ static_cast<size_t>(binding.column_position) >= block->columns()) {
+ return Status::InternalError(
+ "array_sorted_merge enclosing lambda argument '{}' is outside the input "
+ "block (position {})",
+ binding.name, binding.column_position);
+ }
+ const auto& source = block->get_by_position(binding.column_position);
+ enclosing_columns.push_back(source.column->convert_to_full_column_if_const());
+ enclosing_positions.push_back(static_cast<int>(lambda_block.columns()));
+ lambda_block.insert(ColumnWithTypeAndName(enclosing_columns.back()->clone_empty(),
+ source.type, binding.name));
+ }
+ const int merge_argument_base = static_cast<int>(lambda_block.columns());
+ std::string merge_argument_names[2];
+ for (int i = 0; i < 2; ++i) {
+ merge_argument_names[i] = argument_names != nullptr && i < argument_names->size()
+ ? (*argument_names)[i]
+ : "merge_arg_" + std::to_string(i);
+ lambda_block.insert(ColumnWithTypeAndName(nested_types[i]->create_column(),
+ nested_types[i], merge_argument_names[i]));
+ }
+ // The lambda arguments are bound by name (the column refs carry the argument names),
+ // so the body reads these two one-row columns.
+ IColumn* argument_columns[2] = {
+ lambda_block.get_by_position(merge_argument_base).column->assert_mutable().get(),
+ lambda_block.get_by_position(merge_argument_base + 1)
+ .column->assert_mutable()
+ .get()};
+
+ LambdaExecutionContext::Frame lambda_frame;
+ if (argument_names != nullptr) {
+ // Rebase every binding this lambda can see onto the private block, so a nested lambda
+ // resolves an enclosing argument to the copy materialized here instead of walking into
+ // the parent frames, whose positions do not describe this block. The merge arguments
+ // are published last and therefore shadow an outer binding with the same name.
+ lambda_frame.bind_by_name = true;
+ lambda_frame.parent_bindings_visible = false;
+ for (size_t i = 0; i < enclosing_bindings.size(); ++i) {
+ lambda_frame.argument_bindings.push_back(
+ {enclosing_bindings[i].name, enclosing_positions[i]});
+ }
+ for (int i = 0; i < 2; ++i) {
+ lambda_frame.argument_bindings.push_back(
+ {merge_argument_names[i], merge_argument_base + i});
+ }
+ } else {
+ // Positional (legacy) references resolve through the block, so keep the old frame.
+ lambda_frame.bind_by_name = false;
+ lambda_frame.parent_bindings_visible = false;
+ }
+ LambdaExecutionContext::FrameGuard lambda_frame_guard(context->lambda_execution_context(),
+ std::move(lambda_frame));
+
+ const auto& offsets0 = arrays[0]->get_offsets();
+ const auto& offsets1 = arrays[1]->get_offsets();
+ const int lambda_result_base = static_cast<int>(lambda_block.columns());
+
+ // The merged element may come from a non-nullable lambda body (for example COALESCE),
+ // while array elements are always nullable; normalize before inserting.
+ auto append_element = [&](const IColumn& source, size_t element_row) {
+ if (auto* nullable_target = check_and_get_column<ColumnNullable>(result_nested.get())) {
+ if (const auto* nullable_source = check_and_get_column<ColumnNullable>(&source)) {
+ nullable_target->insert_from(*nullable_source, element_row);
+ } else {
+ nullable_target->get_nested_column().insert_from(source, element_row);
+ nullable_target->get_null_map_data().push_back(0);
+ }
+ } else {
+ result_nested->insert_from(source, element_row);
+ }
+ };
+
+ for (size_t row = 0; row < rows; ++row) {
+ // The ascending flag is an argument contract, not data: it is validated on every row,
+ // including a row whose input array is NULL, so a backend-evaluated NULL flag reports
+ // the same error regardless of the value column.
+ if (flag_nullmap != nullptr && (*flag_nullmap)[row] != 0) {
+ return Status::InvalidArgument(
+ "array_sorted_merge requires a non-null ascending flag");
+ }
+ const bool row_is_null = (nullmaps[0] != nullptr && (*nullmaps[0])[row] != 0) ||
+ (nullmaps[1] != nullptr && (*nullmaps[1])[row] != 0);
+ if (row_is_null) {
+ result_nullmap_data[row] = 1;
+ offsets_data.push_back(result_nested->size());
+ continue;
+ }
+ const bool ascending = flags[row] != 0;
+ // A short-circuit branch receives a selector and only the selected rows, so the row
+ // index is compacted: read the captured column at its original row, like the arrays.
+ const auto source_row = expr_selector == nullptr ? row : (*expr_selector)[row];
+ for (const auto& [id, column] : captured_columns) {
+ lambda_block.get_by_position(id).column = column->cut(source_row, 1);
+ }
+ for (size_t i = 0; i < enclosing_columns.size(); ++i) {
+ lambda_block.get_by_position(enclosing_positions[i]).column =
+ enclosing_columns[i]->cut(source_row, 1);
+ }
+
+ size_t left = row == 0 ? 0 : offsets0[row - 1];
+ const size_t left_end = offsets0[row];
+ size_t right = row == 0 ? 0 : offsets1[row - 1];
+ const size_t right_end = offsets1[row];
+
+ while (left < left_end || right < right_end) {
+ if (left == left_end) {
+ append_element(*data[1], right++);
+ continue;
+ }
+ if (right == right_end) {
+ append_element(*data[0], left++);
+ continue;
+ }
+
+ argument_columns[0]->clear();
+ argument_columns[0]->insert_from(*data[0], left);
+ argument_columns[1]->clear();
+ argument_columns[1]->insert_from(*data[1], right);
Review Comment:
[P2] Reuse the candidate whose input index has not advanced
A NULL merge result advances only one input index, but the next comparison clears and copies both candidates again. For an ascending left array of 65,536 distinct short strings below `z`, a right array containing one 1 MiB `z` string, and `(a,b) -> IF(a=b,a,NULL)`, `right` stays fixed for all 65,536 comparisons. `ColumnNullable::insert_from` delegates to `ColumnStr::insert_from`, which copies the full string payload each time: 64 GiB of redundant copying for that single right-hand element (a source-derived count, not a benchmark). Cache each one-row candidate and refresh it only when its corresponding index advances; the descending and reversed-input cases need the same handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4400172143
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,741 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ uint32_t offset = 0;
+ uint32_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ PODArray<char, 32> bytes;
Review Comment:
Fixed in d079ee9f1ea: the arena is now PaddedPODArray<char, 32>, which reserves the same 16-byte tail allowance as ColumnString's chars buffer, so the comparator's 16-byte load stays inside the allocation even for a one-byte string. Added TopKTest.ShortStringValues (one-byte values, MIN/MAX, duplicate at k=1), TopKTest.MergeStringStates and a string state serialization round trip; all pass under ASAN.
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,741 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ uint32_t offset = 0;
+ uint32_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ PODArray<char, 32> bytes;
+ DorisVector<Span> spans;
+
+ size_t size() const { return spans.size(); }
+ void reserve(size_t n) { spans.reserve(n); }
+ void clear() {
+ bytes.clear();
+ spans.clear();
+ }
+ StringRef at(size_t slot) const {
+ const Span& span = spans[slot];
+ return StringRef(bytes.data() + span.offset, span.length);
+ }
+ void append(StringRef value) {
+ DCHECK_LE(bytes.size() + value.size, std::numeric_limits<uint32_t>::max());
+ DCHECK_LE(value.size, std::numeric_limits<uint32_t>::max());
+ spans.push_back({static_cast<uint32_t>(bytes.size()), static_cast<uint32_t>(value.size)});
Review Comment:
Fixed in d079ee9f1ea: the slot offsets and lengths are now size_t, so a cumulative payload above 4 GiB neither narrows nor wraps and the arena has no 4 GiB limit. A unit test cannot reach that volume, so this one is covered by the type change rather than by a test.
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,741 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ uint32_t offset = 0;
+ uint32_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ PODArray<char, 32> bytes;
+ DorisVector<Span> spans;
+
+ size_t size() const { return spans.size(); }
+ void reserve(size_t n) { spans.reserve(n); }
+ void clear() {
+ bytes.clear();
+ spans.clear();
+ }
+ StringRef at(size_t slot) const {
+ const Span& span = spans[slot];
+ return StringRef(bytes.data() + span.offset, span.length);
+ }
+ void append(StringRef value) {
+ DCHECK_LE(bytes.size() + value.size, std::numeric_limits<uint32_t>::max());
+ DCHECK_LE(value.size, std::numeric_limits<uint32_t>::max());
+ spans.push_back({static_cast<uint32_t>(bytes.size()), static_cast<uint32_t>(value.size)});
+ if (value.size != 0) {
+ bytes.insert(value.data, value.data + value.size);
+ }
+ }
+ void swap(StringTopKValueStore& other) noexcept {
+ bytes.swap(other.bytes);
+ spans.swap(other.spans);
+ }
+};
+
+/**
+ * State shared by min_topk and max_topk, for one value type.
+ *
+ * The distinct values sit in slots (see the stores above), with one Slot record per value carrying
+ * the row count and the links of a treap ordered best-first (ascending values for MIN, descending
+ * for MAX). A row costs one O(log k) descent: it either finds its value and bumps the count or
+ * links a new slot at the leaf. A sorted vector would shift O(k) elements for the same insert.
+ *
+ * The treap stores explicit child links because, unlike a heap, its shape is not the complete tree
+ * that an array index can describe: rotations move entries, and the in-order sequence is the value
+ * order rather than the arrival order. A heap could evict and append in O(log k), but it cannot
+ * find an existing value without a hash index or a linear scan, and it cannot produce the sorted
+ * result without sorting. The ordered tree is the structure that keeps find, insert and eviction
+ * all at O(log k) with comparisons only.
+ *
+ * An entry that is evicted keeps its slot and value until the next compaction. The tree never
+ * points at a dead slot.
+ *
+ * With k == 0 the state holds at most the single extreme value and its row count; the caller
+ * derives truncation from the list length, so the state carries no separate bound.
+ */
+template <PrimitiveType T>
+struct AggregateTopKCountData {
+ using CppType = typename PrimitiveTypeTraits<T>::CppType;
+ static constexpr bool IS_STRING = is_string_type(T);
+ using ValueStore =
+ std::conditional_t<IS_STRING, StringTopKValueStore, FixedTopKValueStore<CppType>>;
+ using ValueRef = std::conditional_t<IS_STRING, StringRef, const CppType&>;
+
+ static constexpr int32_t NO_SLOT = -1;
+ // Row counts are never negative, so this cannot collide with a live entry.
+ static constexpr int64_t DEAD_COUNT = -1;
+
+ /// One slot's metadata; the value itself lives in `values` at the same index.
+ struct Slot {
+ int64_t count = 0;
+ int32_t left = NO_SLOT;
+ int32_t right = NO_SLOT;
+ uint32_t priority = 0;
+ };
+
+ bool has_k = false;
+ int64_t k = 0;
+ // Indexed by slot; an evicted slot stays as a dead entry until compaction.
+ ValueStore values;
+ DorisVector<Slot> slots;
+ int32_t root = NO_SLOT;
+ // The last live slot in the aggregate's order: the value an incoming row has to beat to enter
+ // a full list. Cached so that rejecting a worse value costs one comparison, not a descent.
+ int32_t worst = NO_SLOT;
+ int64_t live = 0;
+ // Slots evicted since the last compaction.
+ int64_t dead = 0;
+ uint64_t rng = 0x243F6A8885A308D3ULL;
+
+ AggregateTopKCountData() = default;
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * min_topk(x, k) / max_topk(x, k).
+ *
+ * Result: ARRAY<STRUCT<value T, count BIGINT>>. NULL inputs are ignored: add validates k for
+ * every row and then skips a NULL x, so the aggregate deliberately does not use the
+ * nullable-argument wrapper. The array is ordered by value (ascending for MIN), holds at most k
+ * distinct values with count > 0, and carries no truncation flag: the caller infers it from the
+ * length (a list of exactly k entries is treated as possibly truncated).
+ *
+ * With k = 0 the array holds at most the extreme value and its row count, which lets the
+ * append-only IVM path compute MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopK final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData<T>,
+ AggregateFunctionTopK<T, IsMin>, false>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using State = AggregateTopKCountData<T>;
+ using Base = IAggregateFunctionDataHelper<State, AggregateFunctionTopK<T, IsMin>, false>;
+ using Slot = typename State::Slot;
+ using ValueStore = typename State::ValueStore;
+ using ValueRef = typename State::ValueRef;
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ static constexpr bool IS_STRING = State::IS_STRING;
+ static constexpr int32_t NO_SLOT = State::NO_SLOT;
+
+ explicit AggregateFunctionTopK(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "min_topk" : "max_topk"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(remove_nullable(this->argument_types[0]));
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is read for every row, including rows where x is NULL: an evaluated count that is
+ // NULL or out of range must fail the query instead of being skipped like a NULL value.
+ check_k(state, *columns[1], row_num);
+ const IColumn* value_column = columns[0];
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(value_column)) {
+ if (nullable->is_null_at(row_num)) {
+ return;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ add_value(state, *value_column, row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw any row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else if (UNLIKELY(state.k != other.k)) {
+ // Ordinary partial/final aggregation always shares the same constant k, but the
+ // public _state/_merge/_union combinators do not encode k in the AggState type, so a
+ // UNION ALL of states built with different literal k can reach this merge.
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} states have incompatible counts: {} vs {}", get_name(), state.k,
+ other.k);
+ }
+ if (state.k == 0) {
+ merge_extreme(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ // The slots are in arrival order and may include dead ones, while the serialized form is
+ // the sorted (value, count) list the reader rebuilds the tree from.
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ auto sorted_values = remove_nullable(this->argument_types[0])->create_column();
+ sorted_values->reserve(order.size());
+ DorisVector<int64_t> sorted_counts;
+ sorted_counts.reserve(order.size());
+ for (int32_t slot : order) {
+ insert_value(*sorted_values, state.values.at(slot));
+ sorted_counts.push_back(state.slots[slot].count);
+ }
+ auto size = static_cast<int64_t>(sorted_counts.size());
+ buf.write_binary(size);
+ write_column(buf, *sorted_values);
+ if (size != 0) {
+ // An empty list keeps an empty vector, whose data() is null; passing it to memcpy is
+ // undefined even for a zero length.
+ buf.write(reinterpret_cast<const char*>(sorted_counts.data()),
+ sorted_counts.size() * sizeof(int64_t));
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ int64_t size = 0;
+ buf.read_binary(size);
+ auto values = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(values->size()), size);
+ DorisVector<int64_t> counts(static_cast<size_t>(size));
+ if (size != 0) {
+ buf.read(reinterpret_cast<char*>(counts.data()), size * sizeof(int64_t));
+ }
+ ValueStore value_store;
+ value_store.reserve(static_cast<size_t>(size));
+ for (int64_t row = 0; row != size; ++row) {
+ value_store.append(value_at(*values, static_cast<size_t>(row)));
+ }
+ assign_sorted(state, std::move(value_store), std::move(counts));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_array = assert_cast<ColumnArray&>(to);
+ // The array element and both fields are nullable in the planned type, but a produced entry
+ // never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(result_array.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = assert_cast<ColumnNullable&>(pair_column.get_column(0));
+ auto& count_column = assert_cast<ColumnNullable&>(pair_column.get_column(1));
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ for (int32_t slot : order) {
+ insert_value(value_column.get_nested_column(), state.values.at(slot));
+ value_column.get_null_map_data().push_back(0);
+ assert_cast<ColumnInt64&>(count_column.get_nested_column())
+ .get_data()
+ .push_back(state.slots[slot].count);
+ count_column.get_null_map_data().push_back(0);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ result_array.get_offsets().push_back(pair_column.size());
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ // Both fields are nullable in the planned type: a consumer reads a field through
+ // element_at, which is always nullable, and named_struct then records nullable fields.
+ // Produced entries never contain a NULL field.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(value_type),
+ make_nullable(std::make_shared<DataTypeInt64>())},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ return std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ }
+
+private:
+ /// The value of the row in `column`. The returned view is valid while the column is alive.
+ static ValueRef value_at(const IColumn& column, size_t row_num) {
+ if constexpr (IS_STRING) {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column).get_data_at(
+ row_num);
+ } else {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column)
+ .get_data()[row_num];
+ }
+ }
+
+ /// Exactly the comparison the value column would apply: CompareFloat for FLOAT/DOUBLE through
+ /// Compare::compare (the column uses the same helper), the same memcmp helper ColumnString
+ /// uses for strings, and the column's scale-independent ordering for every other type.
+ static int compare_values(ValueRef lhs, ValueRef rhs) {
+ if constexpr (IS_STRING) {
+ return memcmp_small_allow_overflow15(
+ reinterpret_cast<const uint8_t*>(lhs.data), lhs.size,
+ reinterpret_cast<const uint8_t*>(rhs.data), rhs.size);
+ } else {
+ return Compare::compare(lhs, rhs);
+ }
+ }
+
+ static void insert_value(IColumn& column, ValueRef value) {
+ if constexpr (IS_STRING) {
+ assert_cast<ColumnString&>(column).insert_data(value.data, value.size);
+ } else {
+ assert_cast<ColumnType&>(column).get_data().push_back(value);
+ }
+ }
+
+ /// True when `cmp` puts the left operand before the right one in the aggregate's order.
+ static bool comes_before(int cmp) { return IsMin ? cmp < 0 : cmp > 0; }
+
+ /// True when `cmp`, which compares a value with the worst retained one, puts it after it: the
+ /// value is worse and cannot enter the list.
+ static bool is_worse(int cmp) { return IsMin ? cmp > 0 : cmp < 0; }
+
+ /// splitmix64. The priorities only have to be independent enough to keep the treap balanced;
+ /// a per-state sequence keeps the shape reproducible for the same insertion order.
+ static uint32_t next_priority(State& state) {
+ uint64_t z = (state.rng += 0x9E3779B97F4A7C15ULL);
+ z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9ULL;
+ z = (z ^ (z >> 27)) * 0x94D049BB133111EBULL;
+ return static_cast<uint32_t>(z ^ (z >> 31));
+ }
+
+ static int compare_slots(const State& state, int32_t lhs, int32_t rhs) {
+ return compare_values(state.values.at(lhs), state.values.at(rhs));
+ }
+
+ /// Appends (value, count) as the next slot.
+ static int32_t append_slot(State& state, ValueRef value, int64_t count) {
+ auto slot = static_cast<int32_t>(state.slots.size());
+ DCHECK_LT(slot, std::numeric_limits<int32_t>::max());
+ state.values.append(value);
+ state.slots.push_back(Slot {count, State::NO_SLOT, State::NO_SLOT, next_priority(state)});
+ ++state.live;
+ return slot;
+ }
+
+ static int32_t append_slot(State& state, const IColumn& column, size_t row_num, int64_t count) {
+ return append_slot(state, value_at(column, row_num), count);
+ }
+
+ /// Rotates the subtree rooted at `node` right: the left child becomes the root and `node` its
+ /// right child. The in-order sequence is unchanged, so the treap stays ordered.
+ static int32_t rotate_right(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].left;
+ state.slots[node].left = state.slots[pivot].right;
+ state.slots[pivot].right = node;
+ return pivot;
+ }
+
+ static int32_t rotate_left(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].right;
+ state.slots[node].right = state.slots[pivot].left;
+ state.slots[pivot].left = node;
+ return pivot;
+ }
+
+ /// One descent that performs the whole per-row update: an equal value bumps its count and
+ /// reports no insertion; a missing value is linked as a new slot at the leaf and rotated up
+ /// while the recursion unwinds. `inserted_slot` receives the new slot, NO_SLOT on a hit.
+ static int32_t add_or_increment(State& state, int32_t node, const IColumn& column,
+ size_t row_num, int32_t* inserted_slot) {
+ if (node == State::NO_SLOT) {
+ *inserted_slot = append_slot(state, column, row_num, 1);
+ return *inserted_slot;
+ }
+ int cmp = compare_values(state.values.at(node), value_at(column, row_num));
+ if (cmp == 0) {
+ state.slots[node].count = checked_add(state.slots[node].count, 1);
+ return node;
+ }
+ if (comes_before(cmp)) {
+ // The stored value is the better one, so the input belongs in the right subtree.
+ state.slots[node].right = add_or_increment(state, state.slots[node].right, column,
+ row_num, inserted_slot);
+ if (state.slots[state.slots[node].right].priority > state.slots[node].priority) {
+ return rotate_left(state, node);
+ }
+ } else {
+ state.slots[node].left =
+ add_or_increment(state, state.slots[node].left, column, row_num, inserted_slot);
+ if (state.slots[state.slots[node].left].priority > state.slots[node].priority) {
+ return rotate_right(state, node);
+ }
+ }
+ return node;
+ }
+
+ /// Removes the last entry in the aggregate's order (the largest value for MIN) and returns the
+ /// new subtree root. That entry has no right child, so its left subtree takes its place; the
+ /// child's priority is below the parent's, so no rotation is needed.
+ static int32_t erase_worst(State& state, int32_t node, int32_t* removed) {
+ if (state.slots[node].right == State::NO_SLOT) {
+ *removed = node;
+ return state.slots[node].left;
+ }
+ state.slots[node].right = erase_worst(state, state.slots[node].right, removed);
+ return node;
+ }
+
+ static void refresh_worst(State& state) {
+ int32_t node = state.root;
+ while (node != State::NO_SLOT && state.slots[node].right != State::NO_SLOT) {
+ node = state.slots[node].right;
+ }
+ state.worst = node;
+ }
+
+ /// Slots in the aggregate's order, i.e. the order of the result array.
+ static void collect_in_order(const State& state, DorisVector<int32_t>& slots) {
+ slots.reserve(static_cast<size_t>(state.live));
+ DorisVector<int32_t> stack;
+ stack.reserve(64);
+ int32_t node = state.root;
+ while (node != State::NO_SLOT || !stack.empty()) {
+ while (node != State::NO_SLOT) {
+ stack.push_back(node);
+ node = state.slots[node].left;
+ }
+ node = stack.back();
+ stack.pop_back();
+ // Only live slots are reachable from the root; an evicted slot has no row count.
+ DCHECK_GE(state.slots[node].count, 0);
+ slots.push_back(node);
+ node = state.slots[node].right;
+ }
+ DCHECK_EQ(static_cast<int64_t>(slots.size()), state.live);
+ }
+
+ /// Rebuilds the tree from `values`, already in the aggregate's order, and the matching counts.
+ /// The treap is built in O(size) as a Cartesian tree over the fresh priorities.
+ static void assign_sorted(State& state, ValueStore values, DorisVector<int64_t> counts) {
+ const size_t size = counts.size();
+ DCHECK_EQ(values.size(), size);
+ state.values.swap(values);
+ state.slots.assign(size, Slot {});
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].count = counts[slot];
+ }
+ state.live = static_cast<int64_t>(size);
+ state.dead = 0;
+ DorisVector<int32_t> stack;
+ stack.reserve(size);
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].priority = next_priority(state);
+ int32_t last = State::NO_SLOT;
+ while (!stack.empty() &&
+ state.slots[stack.back()].priority < state.slots[slot].priority) {
+ last = stack.back();
+ stack.pop_back();
+ }
+ state.slots[slot].left = last;
+ if (!stack.empty()) {
+ state.slots[stack.back()].right = static_cast<int32_t>(slot);
+ }
+ stack.push_back(static_cast<int32_t>(slot));
+ }
+ state.root = stack.empty() ? State::NO_SLOT : stack.front();
+ refresh_worst(state);
+ }
+
+ /// Reclaims the slots evicted since the last compaction. The O(k) rebuild is charged to the
+ /// k/2 evictions that produced the dead slots, so it stays O(1) amortized per row.
+ static void compact(State& state) {
+ DorisVector<int32_t> slots;
+ collect_in_order(state, slots);
+ ValueStore values;
+ values.reserve(slots.size());
+ DorisVector<int64_t> counts;
+ counts.reserve(slots.size());
+ for (int32_t slot : slots) {
+ values.append(state.values.at(slot));
+ counts.push_back(state.slots[slot].count);
+ }
+ assign_sorted(state, std::move(values), std::move(counts));
+ }
+
+ static void check_k(State& state, const IColumn& count_column, size_t row_num) {
+ const IColumn* nested = &count_column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&count_column)) {
+ if (UNLIKELY(nullable->is_null_at(row_num))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-null "
+ "constant");
+ }
+ nested = &nullable->get_nested_column();
+ }
+ int64_t value =
+ assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(*nested).get_element(
+ row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-negative "
+ "constant, but got {}",
+ value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "min_topk/max_topk value count overflow");
+ }
+ return result;
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(State& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (state.live == 0) {
+ state.root = append_slot(state, column, row_num, 1);
+ state.worst = state.root;
+ return;
+ }
+ int cmp = compare_values(value_at(column, row_num), state.values.at(0));
+ if (cmp == 0) {
+ state.slots[0].count = checked_add(state.slots[0].count, 1);
+ } else if (comes_before(cmp)) {
+ // The new value is better: replace the tracked extreme. The slot and the
+ // single-node tree stay in place, only the stored value changes.
+ state.values.clear();
+ state.values.append(value_at(column, row_num));
+ state.slots[0].count = 1;
+ }
+ return;
+ }
+
+ // Dead slots are reclaimed before they can outnumber the live ones; see compact().
+ if (state.dead > state.live) {
Review Comment:
Fixed in d079ee9f1ea: the string store tracks the live and the evicted payload bytes, and compaction now triggers when the evicted payload exceeds the live one as well as when dead slots outnumber live ones, so the discarded 1 MiB strings are reclaimed. TopKTest.StringStateSurvivesEvictionAndSerde replaces eight 1000-byte values with one-byte ones, then checks the result and a serialization round trip.
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,741 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ uint32_t offset = 0;
+ uint32_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ PODArray<char, 32> bytes;
+ DorisVector<Span> spans;
+
+ size_t size() const { return spans.size(); }
+ void reserve(size_t n) { spans.reserve(n); }
+ void clear() {
+ bytes.clear();
+ spans.clear();
+ }
+ StringRef at(size_t slot) const {
+ const Span& span = spans[slot];
+ return StringRef(bytes.data() + span.offset, span.length);
+ }
+ void append(StringRef value) {
+ DCHECK_LE(bytes.size() + value.size, std::numeric_limits<uint32_t>::max());
+ DCHECK_LE(value.size, std::numeric_limits<uint32_t>::max());
+ spans.push_back({static_cast<uint32_t>(bytes.size()), static_cast<uint32_t>(value.size)});
+ if (value.size != 0) {
+ bytes.insert(value.data, value.data + value.size);
+ }
+ }
+ void swap(StringTopKValueStore& other) noexcept {
+ bytes.swap(other.bytes);
+ spans.swap(other.spans);
+ }
+};
+
+/**
+ * State shared by min_topk and max_topk, for one value type.
+ *
+ * The distinct values sit in slots (see the stores above), with one Slot record per value carrying
+ * the row count and the links of a treap ordered best-first (ascending values for MIN, descending
+ * for MAX). A row costs one O(log k) descent: it either finds its value and bumps the count or
+ * links a new slot at the leaf. A sorted vector would shift O(k) elements for the same insert.
+ *
+ * The treap stores explicit child links because, unlike a heap, its shape is not the complete tree
+ * that an array index can describe: rotations move entries, and the in-order sequence is the value
+ * order rather than the arrival order. A heap could evict and append in O(log k), but it cannot
+ * find an existing value without a hash index or a linear scan, and it cannot produce the sorted
+ * result without sorting. The ordered tree is the structure that keeps find, insert and eviction
+ * all at O(log k) with comparisons only.
+ *
+ * An entry that is evicted keeps its slot and value until the next compaction. The tree never
+ * points at a dead slot.
+ *
+ * With k == 0 the state holds at most the single extreme value and its row count; the caller
+ * derives truncation from the list length, so the state carries no separate bound.
+ */
+template <PrimitiveType T>
+struct AggregateTopKCountData {
+ using CppType = typename PrimitiveTypeTraits<T>::CppType;
+ static constexpr bool IS_STRING = is_string_type(T);
+ using ValueStore =
+ std::conditional_t<IS_STRING, StringTopKValueStore, FixedTopKValueStore<CppType>>;
+ using ValueRef = std::conditional_t<IS_STRING, StringRef, const CppType&>;
+
+ static constexpr int32_t NO_SLOT = -1;
+ // Row counts are never negative, so this cannot collide with a live entry.
+ static constexpr int64_t DEAD_COUNT = -1;
+
+ /// One slot's metadata; the value itself lives in `values` at the same index.
+ struct Slot {
+ int64_t count = 0;
+ int32_t left = NO_SLOT;
+ int32_t right = NO_SLOT;
+ uint32_t priority = 0;
+ };
+
+ bool has_k = false;
+ int64_t k = 0;
+ // Indexed by slot; an evicted slot stays as a dead entry until compaction.
+ ValueStore values;
+ DorisVector<Slot> slots;
+ int32_t root = NO_SLOT;
+ // The last live slot in the aggregate's order: the value an incoming row has to beat to enter
+ // a full list. Cached so that rejecting a worse value costs one comparison, not a descent.
+ int32_t worst = NO_SLOT;
+ int64_t live = 0;
+ // Slots evicted since the last compaction.
+ int64_t dead = 0;
+ uint64_t rng = 0x243F6A8885A308D3ULL;
+
+ AggregateTopKCountData() = default;
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * min_topk(x, k) / max_topk(x, k).
+ *
+ * Result: ARRAY<STRUCT<value T, count BIGINT>>. NULL inputs are ignored: add validates k for
+ * every row and then skips a NULL x, so the aggregate deliberately does not use the
+ * nullable-argument wrapper. The array is ordered by value (ascending for MIN), holds at most k
+ * distinct values with count > 0, and carries no truncation flag: the caller infers it from the
+ * length (a list of exactly k entries is treated as possibly truncated).
+ *
+ * With k = 0 the array holds at most the extreme value and its row count, which lets the
+ * append-only IVM path compute MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopK final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData<T>,
+ AggregateFunctionTopK<T, IsMin>, false>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using State = AggregateTopKCountData<T>;
+ using Base = IAggregateFunctionDataHelper<State, AggregateFunctionTopK<T, IsMin>, false>;
+ using Slot = typename State::Slot;
+ using ValueStore = typename State::ValueStore;
+ using ValueRef = typename State::ValueRef;
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ static constexpr bool IS_STRING = State::IS_STRING;
+ static constexpr int32_t NO_SLOT = State::NO_SLOT;
+
+ explicit AggregateFunctionTopK(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "min_topk" : "max_topk"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(remove_nullable(this->argument_types[0]));
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is read for every row, including rows where x is NULL: an evaluated count that is
+ // NULL or out of range must fail the query instead of being skipped like a NULL value.
+ check_k(state, *columns[1], row_num);
+ const IColumn* value_column = columns[0];
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(value_column)) {
+ if (nullable->is_null_at(row_num)) {
+ return;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ add_value(state, *value_column, row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw any row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else if (UNLIKELY(state.k != other.k)) {
+ // Ordinary partial/final aggregation always shares the same constant k, but the
+ // public _state/_merge/_union combinators do not encode k in the AggState type, so a
+ // UNION ALL of states built with different literal k can reach this merge.
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} states have incompatible counts: {} vs {}", get_name(), state.k,
+ other.k);
+ }
+ if (state.k == 0) {
+ merge_extreme(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ // The slots are in arrival order and may include dead ones, while the serialized form is
+ // the sorted (value, count) list the reader rebuilds the tree from.
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ auto sorted_values = remove_nullable(this->argument_types[0])->create_column();
+ sorted_values->reserve(order.size());
+ DorisVector<int64_t> sorted_counts;
+ sorted_counts.reserve(order.size());
+ for (int32_t slot : order) {
+ insert_value(*sorted_values, state.values.at(slot));
+ sorted_counts.push_back(state.slots[slot].count);
+ }
+ auto size = static_cast<int64_t>(sorted_counts.size());
+ buf.write_binary(size);
+ write_column(buf, *sorted_values);
+ if (size != 0) {
+ // An empty list keeps an empty vector, whose data() is null; passing it to memcpy is
+ // undefined even for a zero length.
+ buf.write(reinterpret_cast<const char*>(sorted_counts.data()),
+ sorted_counts.size() * sizeof(int64_t));
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ int64_t size = 0;
+ buf.read_binary(size);
+ auto values = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(values->size()), size);
+ DorisVector<int64_t> counts(static_cast<size_t>(size));
+ if (size != 0) {
+ buf.read(reinterpret_cast<char*>(counts.data()), size * sizeof(int64_t));
+ }
+ ValueStore value_store;
+ value_store.reserve(static_cast<size_t>(size));
+ for (int64_t row = 0; row != size; ++row) {
+ value_store.append(value_at(*values, static_cast<size_t>(row)));
+ }
+ assign_sorted(state, std::move(value_store), std::move(counts));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_array = assert_cast<ColumnArray&>(to);
+ // The array element and both fields are nullable in the planned type, but a produced entry
+ // never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(result_array.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = assert_cast<ColumnNullable&>(pair_column.get_column(0));
+ auto& count_column = assert_cast<ColumnNullable&>(pair_column.get_column(1));
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ for (int32_t slot : order) {
+ insert_value(value_column.get_nested_column(), state.values.at(slot));
+ value_column.get_null_map_data().push_back(0);
+ assert_cast<ColumnInt64&>(count_column.get_nested_column())
+ .get_data()
+ .push_back(state.slots[slot].count);
+ count_column.get_null_map_data().push_back(0);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ result_array.get_offsets().push_back(pair_column.size());
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ // Both fields are nullable in the planned type: a consumer reads a field through
+ // element_at, which is always nullable, and named_struct then records nullable fields.
+ // Produced entries never contain a NULL field.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(value_type),
+ make_nullable(std::make_shared<DataTypeInt64>())},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ return std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ }
+
+private:
+ /// The value of the row in `column`. The returned view is valid while the column is alive.
+ static ValueRef value_at(const IColumn& column, size_t row_num) {
+ if constexpr (IS_STRING) {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column).get_data_at(
+ row_num);
+ } else {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column)
+ .get_data()[row_num];
+ }
+ }
+
+ /// Exactly the comparison the value column would apply: CompareFloat for FLOAT/DOUBLE through
+ /// Compare::compare (the column uses the same helper), the same memcmp helper ColumnString
+ /// uses for strings, and the column's scale-independent ordering for every other type.
+ static int compare_values(ValueRef lhs, ValueRef rhs) {
+ if constexpr (IS_STRING) {
+ return memcmp_small_allow_overflow15(
+ reinterpret_cast<const uint8_t*>(lhs.data), lhs.size,
+ reinterpret_cast<const uint8_t*>(rhs.data), rhs.size);
+ } else {
+ return Compare::compare(lhs, rhs);
+ }
+ }
+
+ static void insert_value(IColumn& column, ValueRef value) {
+ if constexpr (IS_STRING) {
+ assert_cast<ColumnString&>(column).insert_data(value.data, value.size);
+ } else {
+ assert_cast<ColumnType&>(column).get_data().push_back(value);
+ }
+ }
+
+ /// True when `cmp` puts the left operand before the right one in the aggregate's order.
+ static bool comes_before(int cmp) { return IsMin ? cmp < 0 : cmp > 0; }
+
+ /// True when `cmp`, which compares a value with the worst retained one, puts it after it: the
+ /// value is worse and cannot enter the list.
+ static bool is_worse(int cmp) { return IsMin ? cmp > 0 : cmp < 0; }
+
+ /// splitmix64. The priorities only have to be independent enough to keep the treap balanced;
+ /// a per-state sequence keeps the shape reproducible for the same insertion order.
+ static uint32_t next_priority(State& state) {
+ uint64_t z = (state.rng += 0x9E3779B97F4A7C15ULL);
+ z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9ULL;
+ z = (z ^ (z >> 27)) * 0x94D049BB133111EBULL;
+ return static_cast<uint32_t>(z ^ (z >> 31));
+ }
+
+ static int compare_slots(const State& state, int32_t lhs, int32_t rhs) {
+ return compare_values(state.values.at(lhs), state.values.at(rhs));
+ }
+
+ /// Appends (value, count) as the next slot.
+ static int32_t append_slot(State& state, ValueRef value, int64_t count) {
+ auto slot = static_cast<int32_t>(state.slots.size());
+ DCHECK_LT(slot, std::numeric_limits<int32_t>::max());
+ state.values.append(value);
+ state.slots.push_back(Slot {count, State::NO_SLOT, State::NO_SLOT, next_priority(state)});
+ ++state.live;
+ return slot;
+ }
+
+ static int32_t append_slot(State& state, const IColumn& column, size_t row_num, int64_t count) {
+ return append_slot(state, value_at(column, row_num), count);
+ }
+
+ /// Rotates the subtree rooted at `node` right: the left child becomes the root and `node` its
+ /// right child. The in-order sequence is unchanged, so the treap stays ordered.
+ static int32_t rotate_right(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].left;
+ state.slots[node].left = state.slots[pivot].right;
+ state.slots[pivot].right = node;
+ return pivot;
+ }
+
+ static int32_t rotate_left(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].right;
+ state.slots[node].right = state.slots[pivot].left;
+ state.slots[pivot].left = node;
+ return pivot;
+ }
+
+ /// One descent that performs the whole per-row update: an equal value bumps its count and
+ /// reports no insertion; a missing value is linked as a new slot at the leaf and rotated up
+ /// while the recursion unwinds. `inserted_slot` receives the new slot, NO_SLOT on a hit.
+ static int32_t add_or_increment(State& state, int32_t node, const IColumn& column,
+ size_t row_num, int32_t* inserted_slot) {
+ if (node == State::NO_SLOT) {
+ *inserted_slot = append_slot(state, column, row_num, 1);
+ return *inserted_slot;
+ }
+ int cmp = compare_values(state.values.at(node), value_at(column, row_num));
+ if (cmp == 0) {
+ state.slots[node].count = checked_add(state.slots[node].count, 1);
+ return node;
+ }
+ if (comes_before(cmp)) {
+ // The stored value is the better one, so the input belongs in the right subtree.
+ state.slots[node].right = add_or_increment(state, state.slots[node].right, column,
+ row_num, inserted_slot);
+ if (state.slots[state.slots[node].right].priority > state.slots[node].priority) {
+ return rotate_left(state, node);
+ }
+ } else {
+ state.slots[node].left =
+ add_or_increment(state, state.slots[node].left, column, row_num, inserted_slot);
+ if (state.slots[state.slots[node].left].priority > state.slots[node].priority) {
+ return rotate_right(state, node);
+ }
+ }
+ return node;
+ }
+
+ /// Removes the last entry in the aggregate's order (the largest value for MIN) and returns the
+ /// new subtree root. That entry has no right child, so its left subtree takes its place; the
+ /// child's priority is below the parent's, so no rotation is needed.
+ static int32_t erase_worst(State& state, int32_t node, int32_t* removed) {
+ if (state.slots[node].right == State::NO_SLOT) {
+ *removed = node;
+ return state.slots[node].left;
+ }
+ state.slots[node].right = erase_worst(state, state.slots[node].right, removed);
+ return node;
+ }
+
+ static void refresh_worst(State& state) {
+ int32_t node = state.root;
+ while (node != State::NO_SLOT && state.slots[node].right != State::NO_SLOT) {
+ node = state.slots[node].right;
+ }
+ state.worst = node;
+ }
+
+ /// Slots in the aggregate's order, i.e. the order of the result array.
+ static void collect_in_order(const State& state, DorisVector<int32_t>& slots) {
+ slots.reserve(static_cast<size_t>(state.live));
+ DorisVector<int32_t> stack;
+ stack.reserve(64);
+ int32_t node = state.root;
+ while (node != State::NO_SLOT || !stack.empty()) {
+ while (node != State::NO_SLOT) {
+ stack.push_back(node);
+ node = state.slots[node].left;
+ }
+ node = stack.back();
+ stack.pop_back();
+ // Only live slots are reachable from the root; an evicted slot has no row count.
+ DCHECK_GE(state.slots[node].count, 0);
+ slots.push_back(node);
+ node = state.slots[node].right;
+ }
+ DCHECK_EQ(static_cast<int64_t>(slots.size()), state.live);
+ }
+
+ /// Rebuilds the tree from `values`, already in the aggregate's order, and the matching counts.
+ /// The treap is built in O(size) as a Cartesian tree over the fresh priorities.
+ static void assign_sorted(State& state, ValueStore values, DorisVector<int64_t> counts) {
+ const size_t size = counts.size();
+ DCHECK_EQ(values.size(), size);
+ state.values.swap(values);
+ state.slots.assign(size, Slot {});
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].count = counts[slot];
+ }
+ state.live = static_cast<int64_t>(size);
+ state.dead = 0;
+ DorisVector<int32_t> stack;
+ stack.reserve(size);
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].priority = next_priority(state);
+ int32_t last = State::NO_SLOT;
+ while (!stack.empty() &&
+ state.slots[stack.back()].priority < state.slots[slot].priority) {
+ last = stack.back();
+ stack.pop_back();
+ }
+ state.slots[slot].left = last;
+ if (!stack.empty()) {
+ state.slots[stack.back()].right = static_cast<int32_t>(slot);
+ }
+ stack.push_back(static_cast<int32_t>(slot));
+ }
+ state.root = stack.empty() ? State::NO_SLOT : stack.front();
+ refresh_worst(state);
+ }
+
+ /// Reclaims the slots evicted since the last compaction. The O(k) rebuild is charged to the
+ /// k/2 evictions that produced the dead slots, so it stays O(1) amortized per row.
+ static void compact(State& state) {
+ DorisVector<int32_t> slots;
+ collect_in_order(state, slots);
+ ValueStore values;
+ values.reserve(slots.size());
+ DorisVector<int64_t> counts;
+ counts.reserve(slots.size());
+ for (int32_t slot : slots) {
+ values.append(state.values.at(slot));
+ counts.push_back(state.slots[slot].count);
+ }
+ assign_sorted(state, std::move(values), std::move(counts));
+ }
+
+ static void check_k(State& state, const IColumn& count_column, size_t row_num) {
+ const IColumn* nested = &count_column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&count_column)) {
+ if (UNLIKELY(nullable->is_null_at(row_num))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-null "
+ "constant");
+ }
+ nested = &nullable->get_nested_column();
+ }
+ int64_t value =
+ assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(*nested).get_element(
+ row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-negative "
+ "constant, but got {}",
+ value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "min_topk/max_topk value count overflow");
+ }
+ return result;
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(State& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (state.live == 0) {
+ state.root = append_slot(state, column, row_num, 1);
+ state.worst = state.root;
+ return;
+ }
+ int cmp = compare_values(value_at(column, row_num), state.values.at(0));
+ if (cmp == 0) {
+ state.slots[0].count = checked_add(state.slots[0].count, 1);
+ } else if (comes_before(cmp)) {
+ // The new value is better: replace the tracked extreme. The slot and the
+ // single-node tree stay in place, only the stored value changes.
+ state.values.clear();
+ state.values.append(value_at(column, row_num));
+ state.slots[0].count = 1;
+ }
+ return;
+ }
+
+ // Dead slots are reclaimed before they can outnumber the live ones; see compact().
+ if (state.dead > state.live) {
+ compact(state);
+ }
+
+ if (state.live >= state.k) {
+ // A full list rejects or absorbs most rows: comparing with the eviction candidate
+ // costs one comparison, while the tree descent would cost log2(k).
+ int cmp = compare_values(value_at(column, row_num), state.values.at(state.worst));
+ if (cmp == 0) {
+ state.slots[state.worst].count = checked_add(state.slots[state.worst].count, 1);
+ return;
+ }
+ if (is_worse(cmp)) {
+ return;
+ }
+ }
+
+ int32_t inserted = State::NO_SLOT;
+ state.root = add_or_increment(state, state.root, column, row_num, &inserted);
+ if (inserted == State::NO_SLOT) {
+ return;
+ }
+ if (state.live <= state.k) {
+ // Not over capacity: the new entry competes with the cached worst for the eviction
+ // candidate role.
+ if (state.worst == State::NO_SLOT ||
+ is_worse(compare_slots(state, inserted, state.worst))) {
+ state.worst = inserted;
+ }
+ return;
+ }
+ // The list was full and the value is new. It beat the cached worst above, so it cannot be
+ // the worst itself; the old worst is still the rightmost entry and leaves the list.
+ int32_t removed = State::NO_SLOT;
+ state.root = erase_worst(state, state.root, &removed);
+ DCHECK_EQ(removed, state.worst);
+ state.slots[removed].count = State::DEAD_COUNT;
+ --state.live;
+ ++state.dead;
+ refresh_worst(state);
+ }
+
+ /// k == 0: the state holds at most the extreme value and its count.
+ static void merge_extreme(State& state, const State& other) {
+ if (other.live == 0) {
+ return;
+ }
+ if (state.live == 0) {
+ state.root = append_slot(state, other.values.at(0), other.slots[0].count);
+ state.worst = state.root;
+ return;
+ }
+ int cmp = compare_values(state.values.at(0), other.values.at(0));
+ if (cmp == 0) {
+ state.slots[0].count = checked_add(state.slots[0].count, other.slots[0].count);
+ } else if (is_worse(cmp)) {
+ state.values.clear();
+ state.values.append(other.values.at(0));
+ state.slots[0].count = other.slots[0].count;
+ }
+ }
+
+ /// Merges two ordered lists, summing the counts of equal values and keeping the k best.
+ ///
+ /// The merge stops as soon as k entries are produced. Both inputs are best-first and hold
+ /// distinct values, and equal values have already been combined, so every remaining value is
+ /// worse than all retained ones: it can neither enter the result nor overflow it. Stopping
+ /// here also avoids materializing a tail that would be truncated away.
+ static void merge_values(State& state, const State& other) {
+ if (other.live == 0) {
+ // The other partial state saw no non-NULL row.
+ return;
+ }
+ DorisVector<int32_t> lhs_slots;
+ DorisVector<int32_t> rhs_slots;
+ collect_in_order(state, lhs_slots);
Review Comment:
Fixed in d079ee9f1ea: merge_values returns immediately when the state is full and the other state's best value (the first slot in its order) is strictly worse than the state's worst. An equal value still goes through the merge because it raises the count. TopKTest.MergeStringStates covers the rejected-RHS case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6468188195
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6495725682
Codex automated review failed and did not complete.
Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/40860648656
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6487922191
Updated the top-K state layout: a group's value now lives in its slot instead of in a second vector, so a state owns one slot array (72 bytes and one heap allocation per group for fixed-width types, was 96 bytes and two allocations). The string case loses its separate span vector the same way. The serialized state format is unchanged.
Memory of `min_topk` vs `MIN` on a release build: single BE, 10,000,000 rows, RSS peak delta of one query (`enable_sql_cache=false`), second of two consecutive rounds in the same session.
BIGINT (`v` BIGINT, 10M rows):
| rows per group | groups | query | before | after |
| --- | --- | --- | --- | --- |
| 1 | 10M | `MIN(v)` | 863 MB | 841 MB |
| 1 | 10M | `min_topk(v, 8)` | 1896 MB | 1594 MB |
| 8 | 1.25M | `MIN(v)` | 93 MB | 89 MB |
| 8 | 1.25M | `min_topk(v, 8)` | 487 MB | 459 MB |
VARCHAR (`v` VARCHAR, 10M rows):
| rows per group | groups | query | before | after |
| --- | --- | --- | --- | --- |
| 1 | 10M | `MIN(v)` | 1403 MB | 1426 MB |
| 1 | 10M | `min_topk(v, 8)` | 3069 MB | 2847 MB |
| 8 | 1.25M | `MIN(v)` | 153 MB | 153 MB |
| 8 | 1.25M | `min_topk(v, 8)` | 889 MB | 833 MB |
The state kept on top of `MIN` (query minus `MIN`, divided by the group count):
| value type | rows per group | before | after |
| --- | --- | --- | --- |
| BIGINT | 1 | ~103 B/group | ~75 B/group |
| BIGINT | 8 | ~315 B/group | ~296 B/group |
| VARCHAR | 1 | ~167 B/group | ~142 B/group |
| VARCHAR | 8 | ~589 B/group | ~544 B/group |
Reading these numbers: at one value per group the per-group overhead is what changes, so the state drops by a quarter; at eight values per group the slots themselves dominate and the saving is the removed allocation and the 24-byte smaller state, roughly 5-8%. VARCHAR pays more than BIGINT per group at the same value count because a slot carries a 16-byte span instead of an 8-byte value and the state carries the arena header.
Tests on the updated code: `be/test/exprs/aggregate/agg_topk_test.cpp` (18 cases) and `regression-test/suites/query_p0/aggregate/min_max_topk.groovy` pass, on both the debug/ASAN and the release build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6352747026
# FE Regression Coverage Report
Increment line coverage ` 79.53% (136/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6354105538
# FE Regression Coverage Report
Increment line coverage ` 79.53% (136/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6344889123
<details>
<summary>TPC-DS: <b>Total hot run time: 151028 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a7420809fc69a4ea822cfd1f29ed9f8b03afbb3a, data reload: false
query5 4297 580 446 446
query6 431 202 199 199
query7 4819 447 243 243
query8 322 164 155 155
query9 8789 3125 3134 3125
query10 470 313 253 253
query11 5965 3865 3576 3576
query12 149 90 86 86
query13 1248 431 313 313
query14 6168 4042 3709 3709
query14_1 3455 3440 3456 3440
query15 215 204 184 184
query16 950 474 438 438
query17 860 664 526 526
query18 2429 425 305 305
query19 196 166 141 141
query20 85 85 95 85
query21 214 133 112 112
query22 13110 12980 12872 12872
query23 13465 12970 12488 12488
query23_1 12501 12455 12505 12455
query24 7433 811 453 453
query24_1 473 473 462 462
query25 506 389 335 335
query26 1259 258 138 138
query27 2757 442 269 269
query28 4605 1928 1927 1927
query29 1526 585 434 434
query30 303 226 183 183
query31 1013 940 814 814
query32 137 90 88 88
query33 518 282 235 235
query34 872 835 490 490
query35 737 746 631 631
query36 824 789 699 699
query37 120 99 83 83
query38 1794 1723 1688 1688
query39 784 796 739 739
query39_1 730 730 738 730
query40 220 119 98 98
query41 68 66 64 64
query42 93 81 80 80
query43 342 370 314 314
query44 1297 682 686 682
query45 184 173 163 163
query46 837 971 559 559
query47 3015 2954 2785 2785
query48 288 312 216 216
query49 564 394 295 295
query50 686 272 209 209
query51 10341 10564 10469 10469
query52 78 82 69 69
query53 188 203 148 148
query54 232 195 191 191
query55 69 67 65 65
query56 231 238 223 223
query57 1672 1631 1547 1547
query58 289 276 257 257
query59 2195 2178 1972 1972
query60 308 254 245 245
query61 172 165 166 165
query62 403 341 294 294
query63 195 160 163 160
query64 2865 1032 740 740
query65 3323 3279 3278 3278
query66 1798 407 299 299
query67 20124 20257 19975 19975
query68 2989 1003 609 609
query69 392 297 254 254
query70 841 815 792 792
query71 310 240 217 217
query72 2728 2637 2331 2331
query73 516 524 305 305
query74 4669 4532 4350 4350
query75 2336 2287 1896 1896
query76 2325 1022 593 593
query77 333 409 281 281
query78 9748 9687 9136 9136
query79 1046 837 508 508
query80 545 433 367 367
query81 544 315 274 274
query82 581 130 105 105
query83 308 200 184 184
query84 310 127 104 104
query85 822 411 369 369
query86 328 238 239 238
query87 2011 1955 1812 1812
query88 3629 2677 2648 2648
query89 326 277 239 239
query90 1875 174 169 169
query91 156 145 127 127
query92 101 108 89 89
query93 988 1001 579 579
query94 440 310 270 270
query95 581 383 300 300
query96 667 480 239 239
query97 2449 2453 2303 2303
query98 162 155 145 145
query99 793 741 639 639
Total cold run time: 234183 ms
Total hot run time: 151028 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6359073066
# FE UT Coverage Report
Increment line coverage ` 28.07% (48/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_48370febac57e0391be119b675afc90cfdba098d/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_48370febac57e0391be119b675afc90cfdba098d/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6361916749
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6362437396
<details>
<summary>ClickBench: <b>Total hot run time: 24.21 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 51bd4e1021eb209791aee5937061ac8805784483, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.04
query3 0.27 0.09 0.08
query4 1.61 0.10 0.11
query5 0.24 0.24 0.25
query6 1.17 0.68 0.70
query7 0.04 0.01 0.00
query8 0.06 0.04 0.04
query9 0.40 0.36 0.35
query10 0.63 0.56 0.58
query11 0.20 0.15 0.15
query12 0.18 0.15 0.15
query13 0.49 0.49 0.49
query14 1.03 1.03 1.01
query15 0.63 0.61 0.62
query16 0.36 0.36 0.34
query17 1.12 1.12 1.14
query18 0.20 0.19 0.19
query19 2.10 1.89 1.97
query20 0.02 0.02 0.01
query21 15.53 0.19 0.13
query22 4.91 0.06 0.05
query23 16.23 0.31 0.13
query24 2.93 0.72 0.36
query25 0.14 0.07 0.04
query26 1.58 0.22 0.15
query27 0.04 0.05 0.05
query28 3.55 0.75 0.35
query29 12.50 4.17 3.29
query30 0.29 0.12 0.10
query31 2.78 0.55 0.32
query32 3.22 0.60 0.50
query33 3.16 3.17 3.16
query34 15.24 4.04 3.40
query35 3.35 3.32 3.33
query36 0.55 0.44 0.44
query37 0.11 0.07 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.14 0.15
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.05 0.04 0.03
Total cold run time: 97.41 s
Total hot run time: 24.21 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6358726105
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6359635075
<details>
<summary>TPC-H: <b>Total hot run time: 28505 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 48370febac57e0391be119b675afc90cfdba098d, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17593 5175 5057 5057
q2 2158 323 268 268
q3 10179 1443 839 839
q4 4673 458 349 349
q5 7443 794 535 535
q6 182 188 156 156
q7 749 800 602 602
q8 9276 1343 1061 1061
q9 5521 4197 4354 4197
q10 6826 1290 1036 1036
q11 441 249 232 232
q12 638 397 277 277
q13 18114 4095 3020 3020
q14 266 270 241 241
q15 q16 711 708 649 649
q17 1187 749 591 591
q18 7036 6158 6090 6090
q19 1108 957 591 591
q20 391 335 227 227
q21 5418 2231 2272 2231
q22 355 295 256 256
Total cold run time: 100265 ms
Total hot run time: 28505 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5636 5613 5618 5613
q2 665 594 539 539
q3 5190 5398 4868 4868
q4 2042 2146 1527 1527
q5 5243 5124 5746 5124
q6 244 202 151 151
q7 2287 2140 1874 1874
q8 3107 2934 2654 2654
q9 8039 8057 8087 8057
q10 3695 3618 3237 3237
q11 618 435 400 400
q12 681 692 516 516
q13 3558 3838 3226 3226
q14 312 303 283 283
q15 q16 687 707 647 647
q17 8273 7557 7346 7346
q18 12911 12141 12992 12141
q19 898 777 767 767
q20 2133 2132 1916 1916
q21 5786 4710 4782 4710
q22 484 458 416 416
Total cold run time: 72489 ms
Total hot run time: 66012 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6359859933
<details>
<summary>ClickBench: <b>Total hot run time: 24.33 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 48370febac57e0391be119b675afc90cfdba098d, data reload: false
query1 0.01 0.00 0.00
query2 0.10 0.05 0.05
query3 0.26 0.08 0.09
query4 1.60 0.10 0.10
query5 0.23 0.23 0.26
query6 1.17 0.69 0.69
query7 0.04 0.01 0.00
query8 0.06 0.03 0.04
query9 0.40 0.33 0.36
query10 0.57 0.57 0.58
query11 0.20 0.15 0.14
query12 0.18 0.14 0.14
query13 0.51 0.49 0.47
query14 1.03 1.02 1.00
query15 0.62 0.60 0.60
query16 0.34 0.35 0.34
query17 1.13 1.14 1.15
query18 0.21 0.20 0.18
query19 2.07 1.99 2.00
query20 0.02 0.01 0.02
query21 15.49 0.18 0.13
query22 4.93 0.05 0.06
query23 16.27 0.32 0.13
query24 3.08 0.59 0.44
query25 0.11 0.05 0.07
query26 0.88 0.22 0.15
query27 0.04 0.04 0.05
query28 2.78 0.73 0.36
query29 12.47 4.19 3.29
query30 0.28 0.13 0.12
query31 2.77 0.55 0.32
query32 3.22 0.58 0.50
query33 3.14 3.30 3.15
query34 15.41 3.97 3.40
query35 3.33 3.34 3.36
query36 0.55 0.43 0.42
query37 0.09 0.07 0.06
query38 0.05 0.04 0.03
query39 0.03 0.03 0.03
query40 0.18 0.15 0.15
query41 0.09 0.03 0.03
query42 0.04 0.02 0.02
query43 0.04 0.04 0.03
Total cold run time: 96.02 s
Total hot run time: 24.33 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6367545135
# BE Regression && UT Coverage Report
Increment line coverage `81.76% (269/329)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 68.36% (31845/46585) |
| Line Coverage | 53.03% (344893/650355) |
| Region Coverage | 48.38% (281818/582490) |
| Branch Coverage | 49.86% (132166/265083) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6467487239
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6469233428
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6471931451
<details>
<summary>TPC-H: <b>Total hot run time: 28753 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5c8beb3b4bc041961f08e39e9b5aae6284a64cd6, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17586 5219 5054 5054
q2 2101 312 297 297
q3 10210 1382 809 809
q4 4675 463 337 337
q5 7420 803 533 533
q6 177 185 149 149
q7 735 801 607 607
q8 9254 1332 1096 1096
q9 5467 4193 4188 4188
q10 6823 1282 1014 1014
q11 438 251 230 230
q12 637 391 271 271
q13 18122 3716 3029 3029
q14 269 273 250 250
q15 q16 733 717 661 661
q17 1207 766 579 579
q18 7009 6201 6115 6115
q19 1252 971 611 611
q20 399 341 227 227
q21 5838 2418 2451 2418
q22 399 342 278 278
Total cold run time: 100751 ms
Total hot run time: 28753 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5965 6034 6119 6034
q2 755 624 591 591
q3 5659 5738 5230 5230
q4 2025 2168 1582 1582
q5 5872 5870 5641 5641
q6 240 187 139 139
q7 2257 2090 1779 1779
q8 2975 2674 2716 2674
q9 7947 7457 7521 7457
q10 3625 3569 3116 3116
q11 592 423 377 377
q12 666 650 498 498
q13 3311 3546 3029 3029
q14 288 295 263 263
q15 q16 658 681 616 616
q17 7793 7050 6865 6865
q18 12958 12171 12852 12171
q19 907 790 778 778
q20 2134 2124 1890 1890
q21 5561 4593 4671 4593
q22 534 466 402 402
Total cold run time: 72722 ms
Total hot run time: 65725 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen merged PR #69456:
URL: https://github.com/apache/doris/pull/69456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6487924613
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6491977223
<details>
<summary>TPC-H: <b>Total hot run time: 28696 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d9d17e663629a2bc41b45deb64a2e384b1f41df2, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17589 5177 5058 5058
q2 2152 300 256 256
q3 10249 1400 835 835
q4 4682 470 340 340
q5 7441 821 528 528
q6 184 191 155 155
q7 743 794 606 606
q8 9264 1391 1046 1046
q9 5483 4233 4194 4194
q10 6833 1291 1051 1051
q11 426 244 229 229
q12 657 402 282 282
q13 18082 3684 3037 3037
q14 288 281 251 251
q15 q16 742 732 681 681
q17 1218 758 597 597
q18 7075 6218 6161 6161
q19 1179 974 621 621
q20 403 344 222 222
q21 5852 2287 2298 2287
q22 348 313 259 259
Total cold run time: 100890 ms
Total hot run time: 28696 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5611 5580 5599 5580
q2 676 579 519 519
q3 5175 5421 4912 4912
q4 2028 2173 1589 1589
q5 5271 5170 5840 5170
q6 251 195 157 157
q7 2362 2102 1928 1928
q8 3104 2808 2660 2660
q9 8082 8058 8133 8058
q10 3744 3689 3255 3255
q11 633 441 404 404
q12 681 683 517 517
q13 3669 3846 3191 3191
q14 328 320 307 307
q15 q16 688 736 657 657
q17 8385 7611 6982 6982
q18 12943 12209 12932 12209
q19 908 787 792 787
q20 2154 2176 1914 1914
q21 5634 4644 4732 4644
q22 499 436 402 402
Total cold run time: 72826 ms
Total hot run time: 65842 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6493077931
Codex automated review failed and did not complete.
Error: Review recovery stopped: Command '['gh', 'api', 'repos/apache/doris/pulls/69456']' returned non-zero exit status 1.
Workflow run: https://github.com/apache/doris/actions/runs/40841771541
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6419999534
# BE Regression && UT Coverage Report
Increment line coverage `81.76% (269/329)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 68.40% (31863/46585) |
| Line Coverage | 53.06% (345074/650355) |
| Region Coverage | 48.42% (282023/582490) |
| Branch Coverage | 49.90% (132269/265083) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6431550671
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6431549843
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6337085137
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6352970439
<details>
<summary>ClickBench: <b>Total hot run time: 24.34 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d, data reload: false
query1 0.01 0.01 0.01
query2 0.09 0.05 0.05
query3 0.26 0.09 0.08
query4 1.61 0.10 0.10
query5 0.24 0.24 0.24
query6 1.17 0.69 0.67
query7 0.04 0.00 0.00
query8 0.06 0.04 0.04
query9 0.39 0.36 0.36
query10 0.56 0.58 0.59
query11 0.20 0.14 0.14
query12 0.18 0.15 0.15
query13 0.50 0.50 0.49
query14 1.02 1.03 1.00
query15 0.64 0.61 0.61
query16 0.36 0.35 0.35
query17 1.11 1.10 1.17
query18 0.21 0.20 0.19
query19 2.13 1.97 1.96
query20 0.02 0.01 0.02
query21 15.43 0.18 0.14
query22 4.79 0.06 0.05
query23 16.27 0.32 0.12
query24 3.21 0.47 0.30
query25 0.10 0.08 0.06
query26 0.81 0.22 0.16
query27 0.05 0.04 0.03
query28 3.25 0.78 0.38
query29 12.47 4.22 3.36
query30 0.29 0.15 0.12
query31 2.78 0.56 0.33
query32 3.22 0.61 0.50
query33 3.22 3.24 3.19
query34 15.42 4.00 3.37
query35 3.35 3.35 3.34
query36 0.55 0.43 0.42
query37 0.09 0.07 0.06
query38 0.06 0.05 0.03
query39 0.04 0.03 0.03
query40 0.18 0.14 0.14
query41 0.09 0.03 0.03
query42 0.04 0.02 0.02
query43 0.04 0.04 0.03
Total cold run time: 96.55 s
Total hot run time: 24.34 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4346834936
##########
be/src/exprs/function/function_topk_merge.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <glog/logging.h>
+
+#include <cstddef>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_numbers.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/types.h"
+#include "exprs/function/function.h"
+#include "exprs/function/simple_function_factory.h"
+
+namespace doris {
+class FunctionContext;
+} // namespace doris
+
+namespace doris {
+
+namespace {
+
+/// Upper bound of k. Keep in sync with the aggregate's MAX_TOPK_K
+/// (aggregate_function_topk_count.h) and FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/// One row of an ARRAY<STRUCT<value, count>> argument. A NULL array is an empty view.
+struct TopKArrayView {
+ const IColumn* values = nullptr;
+ const ColumnInt64* counts = nullptr;
+ /// NULL bitmap of the array element (the pair struct).
+ const NullMap* null_map = nullptr;
+ /// NULL bitmap of the `value` field; NULL values are not part of the contract.
+ const NullMap* value_null_map = nullptr;
+ /// NULL bitmap of the `count` field; NULL counts are not part of the contract.
+ const NullMap* count_null_map = nullptr;
+ size_t begin = 0;
+ size_t end = 0;
+};
+
+TopKArrayView make_array_view(const IColumn& column, size_t row) {
+ TopKArrayView view;
+ const IColumn* array_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ array_column = &nullable->get_nested_column();
+ }
+ const auto& array = assert_cast<const ColumnArray&>(*array_column);
+ // offsets[i] is the cumulative size after row i, so row's range is [offsets[row-1], offsets[row]).
+ // offset_at(0) reads offsets[-1], which PaddedPODArray keeps as 0.
+ view.begin = array.offset_at(static_cast<ssize_t>(row));
+ view.end = array.offset_at(static_cast<ssize_t>(row) + 1);
+
+ const IColumn* pair_column = &array.get_data();
+ if (const auto* element_nullable = check_and_get_column<ColumnNullable>(pair_column)) {
+ view.null_map = &element_nullable->get_null_map_data();
+ pair_column = &element_nullable->get_nested_column();
+ }
+ const auto& pairs = assert_cast<const ColumnStruct&>(*pair_column);
+ view.values = &pairs.get_column(0);
+ // A cast may widen the `value` field to a nullable column; the entries themselves are never
+ // NULL, so only the nested column is used.
+ if (const auto* value_nullable = check_and_get_column<ColumnNullable>(view.values)) {
+ view.value_null_map = &value_nullable->get_null_map_data();
+ view.values = &value_nullable->get_nested_column();
+ }
+ // The same holds for `count`: a nested cast may have made the field nullable.
+ const auto* count_column = &pairs.get_column(1);
+ if (const auto* count_nullable = check_and_get_column<ColumnNullable>(count_column)) {
+ view.count_null_map = &count_nullable->get_null_map_data();
+ count_column = &count_nullable->get_nested_column();
+ }
+ view.counts = &assert_cast<const ColumnInt64&>(*count_column);
+ return view;
+}
+
+/// One scalar bound argument: NULL means the corresponding side was not truncated.
+struct TopKBoundView {
+ const IColumn* value = nullptr;
+ bool is_null = true;
+};
+
+TopKBoundView make_bound_view(const IColumn& column, size_t row) {
+ TopKBoundView view;
+ const IColumn* value_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ view.value = value_column;
Review Comment:
[P1] Preserve the row index in bound views
`make_bound_view` checks the current row's NULL bit but retains only the whole column; effective-bound selection, trusted-range filtering and bound output then all read index 0. For a two-row MIN block with k=2, n=3, dn=0, empty I/D and NULL delta bounds, let S/b be [(1,1),(2,1)]/2 and [(10,1),(20,1)]/20. The second state should stay unchanged, but it becomes an empty list with bound 2. Nonconstant table arguments remain vectorized, so this affects normal multi-group apply calls; MAX is symmetric. Carry and use the bound row index throughout, and cover differing bounds in one block.
##########
regression-test/data/query_p0/aggregate/topk_min_max_count.out:
##########
@@ -0,0 +1,143 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !min_k0 --
+a {"values":[], "bound":1}
+b {"values":[], "bound":5}
+c {"values":[], "bound":null}
+d {"values":[], "bound":10}
+e {"values":[], "bound":0}
+
+-- !max_k0 --
+a {"values":[], "bound":3}
+b {"values":[], "bound":5}
+c {"values":[], "bound":null}
+d {"values":[], "bound":17}
+e {"values":[], "bound":0}
+
+-- !min_k1 --
+a {"values":[{"value":1, "count":2}], "bound":1}
+b {"values":[{"value":5, "count":1}], "bound":null}
+c {"values":[], "bound":null}
+d {"values":[{"value":10, "count":1}], "bound":10}
+e {"values":[{"value":0, "count":5}], "bound":null}
+
+-- !max_k1 --
+a {"values":[{"value":3, "count":2}], "bound":3}
+b {"values":[{"value":5, "count":1}], "bound":null}
+c {"values":[], "bound":null}
+d {"values":[{"value":17, "count":1}], "bound":17}
+e {"values":[{"value":0, "count":5}], "bound":null}
+
+-- !min_k2 --
+a {"values":[{"value":1, "count":2}, {"value":2, "count":1}], "bound":2}
+b {"values":[{"value":5, "count":1}], "bound":null}
+c {"values":[], "bound":null}
+d {"values":[{"value":10, "count":1}, {"value":11, "count":1}], "bound":11}
+e {"values":[{"value":0, "count":5}], "bound":null}
+
+-- !max_k2 --
+a {"values":[{"value":3, "count":2}, {"value":2, "count":1}], "bound":2}
+b {"values":[{"value":5, "count":1}], "bound":null}
+c {"values":[], "bound":null}
+d {"values":[{"value":17, "count":1}, {"value":16, "count":1}], "bound":16}
+e {"values":[{"value":0, "count":5}], "bound":null}
+
+-- !min_k8 --
+a {"values":[{"value":1, "count":2}, {"value":2, "count":1}, {"value":3, "count":2}], "bound":null}
+b {"values":[{"value":5, "count":1}], "bound":null}
+c {"values":[], "bound":null}
+d {"values":[{"value":10, "count":1}, {"value":11, "count":1}, {"value":12, "count":1}, {"value":13, "count":1}, {"value":14, "count":1}, {"value":15, "count":1}, {"value":16, "count":1}, {"value":17, "count":1}], "bound":null}
+e {"values":[{"value":0, "count":5}], "bound":null}
+
+-- !max_k8 --
+a {"values":[{"value":3, "count":2}, {"value":2, "count":1}, {"value":1, "count":2}], "bound":null}
+b {"values":[{"value":5, "count":1}], "bound":null}
+c {"values":[], "bound":null}
+d {"values":[{"value":17, "count":1}, {"value":16, "count":1}, {"value":15, "count":1}, {"value":14, "count":1}, {"value":13, "count":1}, {"value":12, "count":1}, {"value":11, "count":1}, {"value":10, "count":1}], "bound":null}
+e {"values":[{"value":0, "count":5}], "bound":null}
+
+-- !min_empty_input --
+{"values":[], "bound":null}
+
+-- !min_all_null_group --
+c {"values":[], "bound":null}
+
+-- !min_not_null_type --
+a {"values":[{"value":1, "count":2}, {"value":2, "count":1}], "bound":2}
+b {"values":[{"value":5, "count":1}], "bound":null}
+c {"values":[{"value":0, "count":2}], "bound":null}
+d {"values":[{"value":10, "count":1}, {"value":11, "count":1}], "bound":11}
+e {"values":[{"value":0, "count":5}], "bound":null}
+
+-- !min_not_null_type_empty --
+{"values":[], "bound":null}
+
+-- !min_string --
+a {"values":[{"value":"a", "count":1}, {"value":"b", "count":1}], "bound":"b"} {"values":[{"value":"z", "count":1}, {"value":"c", "count":3}], "bound":"c"}
+b {"values":[{"value":"x", "count":1}], "bound":null} {"values":[{"value":"x", "count":1}], "bound":null}
+c {"values":[], "bound":null} {"values":[], "bound":null}
+d {"values":[{"value":"j", "count":1}, {"value":"k", "count":1}], "bound":"k"} {"values":[{"value":"q", "count":1}, {"value":"p", "count":1}], "bound":"p"}
+e {"values":[{"value":"s", "count":5}], "bound":null} {"values":[{"value":"s", "count":5}], "bound":null}
+
+-- !min_decimal --
+a {"values":[{"value":1.10, "count":2}, {"value":2.20, "count":1}], "bound":2.20} {"values":[{"value":3.30, "count":2}, {"value":2.20, "count":1}], "bound":2.20}
+b {"values":[{"value":5.50, "count":1}], "bound":null} {"values":[{"value":5.50, "count":1}], "bound":null}
+c {"values":[], "bound":null} {"values":[], "bound":null}
+d {"values":[{"value":10.10, "count":1}, {"value":11.10, "count":1}], "bound":11.10} {"values":[{"value":17.10, "count":1}, {"value":16.10, "count":1}], "bound":16.10}
+e {"values":[{"value":0.00, "count":5}], "bound":null} {"values":[{"value":0.00, "count":5}], "bound":null}
+
+-- !min_datetime --
+a {"values":[{"value":"2024-01-01 09:00:00", "count":1}, {"value":"2024-01-01 10:00:00", "count":1}], "bound":"2024-01-01 10:00:00"} {"values":[{"value":"2024-01-03 10:00:00", "count":2}, {"value":"2024-01-02 10:00:00", "count":1}], "bound":"2024-01-02 10:00:00"}
+b {"values":[{"value":"2024-02-01 10:00:00", "count":1}], "bound":null} {"values":[{"value":"2024-02-01 10:00:00", "count":1}], "bound":null}
+c {"values":[], "bound":null} {"values":[], "bound":null}
+d {"values":[{"value":"2024-03-01 10:00:00", "count":1}, {"value":"2024-03-02 10:00:00", "count":1}], "bound":"2024-03-02 10:00:00"} {"values":[{"value":"2024-03-08 10:00:00", "count":1}, {"value":"2024-03-07 10:00:00", "count":1}], "bound":"2024-03-07 10:00:00"}
+e {"values":[], "bound":null} {"values":[], "bound":null}
+
+-- !double_special_values --
+e {"values":[{"value":-0, "count":2}, {"value":1, "count":1}, {"value":NaN, "count":2}], "bound":null} {"values":[{"value":NaN, "count":2}, {"value":1, "count":1}, {"value":-0, "count":2}], "bound":null}
Review Comment:
[P2] Avoid requiring a particular signed-zero representative
This expected struct requires -0, but the input contains both +0 and -0 on an unordered two-bucket scan. `add_value` keeps the first representative when CompareFloat reports equality, and `merge_values` keeps the left representative, so a valid scan/merge order can return +0 with the same count. Number JSON formatting preserves the sign, while the regression framework compares STRUCT cells as exact strings; `order_qt` only orders result rows. Canonicalize zero representations or project numeric values/counts for comparison, then regenerate the output so this test does not depend on aggregate input order.
##########
be/src/exprs/aggregate/aggregate_function_topk_count.h:
##########
@@ -0,0 +1,428 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/string_buffer.hpp"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// Upper bound of k, shared with FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/**
+ * State shared by topk_min_count and topk_max_count.
+ *
+ * The entries are held as two parallel structures: a column with the distinct values (sorted
+ * ascending for MIN / descending for MAX, at most k entries) and their non-negative int64 counts.
+ * Only values with count > 0 are kept. `bound` is a one-row column with the extreme value seen
+ * when k == 0, or the k-th value once the list was truncated when k >= 1; `has_bound` is the
+ * truncation flag, and for k >= 1 `bound IS NULL` means "the list holds every distinct value".
+ */
+struct AggregateTopKCountData {
+ const DataTypePtr value_type;
+ bool has_k = false;
+ int64_t k = 0;
+ bool has_bound = false;
+ MutableColumnPtr bound_column;
+ MutableColumnPtr value_column;
+ std::vector<int64_t> counts;
+
+ explicit AggregateTopKCountData(const DataTypes& argument_types_)
+ : value_type(argument_types_[0]),
+ bound_column(value_type->create_column()),
+ value_column(value_type->create_column()) {}
+
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * topk_min_count(x, k) / topk_max_count(x, k).
+ *
+ * Result: STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T NULL>. NULL inputs are
+ * ignored (the nullable-argument wrapper never calls add for them). `values` is ordered by value
+ * (ascending for MIN), holds at most k distinct values, and `bound` is the truncation point: the
+ * k-th value when more than k distinct values were seen, otherwise NULL.
+ *
+ * With k = 0 no list is built: `values` is empty and `bound` is the extreme value seen on this
+ * side (NULL when there was no non-NULL input), which lets the append-only IVM path compute
+ * MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopKCount final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopKCount<T, IsMin>, true>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using Base = IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopKCount<T, IsMin>, true>;
+
+ explicit AggregateFunctionTopKCount(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "topk_min_count" : "topk_max_count"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(this->argument_types[0]);
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is a constant expression, but only the first row carries it: the state remembers it so
+ // that merge can check both sides used the same capacity. A state without has_k is empty.
+ check_k(state, *columns[1], row_num);
+ add_value(state, *columns[0], row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw a non-NULL row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else {
+ // Both sides are partial states of the same constant k.
+ DCHECK_EQ(state.k, other.k);
+ }
+ if (state.k == 0) {
+ merge_bound(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ buf.write_binary(state.has_bound);
+ if (state.has_bound) {
+ write_column(buf, *state.bound_column);
+ }
+ auto size = static_cast<int64_t>(state.counts.size());
+ buf.write_binary(size);
+ write_column(buf, *state.value_column);
+ buf.write(reinterpret_cast<const char*>(state.counts.data()),
+ state.counts.size() * sizeof(int64_t));
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ buf.read_binary(state.has_bound);
+ if (state.has_bound) {
+ state.bound_column = read_column(buf);
+ } else {
+ state.bound_column = state.value_type->create_column();
+ }
+ int64_t size = 0;
+ buf.read_binary(size);
+ state.value_column = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(state.value_column->size()), size);
+ state.counts.resize(size);
+ buf.read(reinterpret_cast<char*>(state.counts.data()), size * sizeof(int64_t));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_struct = assert_cast<ColumnStruct&>(to);
+ auto& values_column = assert_cast<ColumnArray&>(result_struct.get_column(0));
+ // The array element is nullable in the planned type (ARRAY<STRUCT<...>> defaults to
+ // contains_null = true), but a produced entry never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(values_column.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = pair_column.get_column(0);
+ auto& count_column = assert_cast<ColumnInt64&>(pair_column.get_column(1));
+ for (size_t i = 0; i != state.counts.size(); ++i) {
+ value_column.insert_from(*state.value_column, i);
+ count_column.get_data().push_back(state.counts[i]);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ values_column.get_offsets().push_back(pair_column.size());
+
+ auto& bound_column = assert_cast<ColumnNullable&>(result_struct.get_column(1));
+ if (state.has_bound) {
+ bound_column.get_nested_column().insert_from(*state.bound_column, 0);
+ bound_column.get_null_map_data().push_back(0);
+ } else {
+ bound_column.get_nested_column().insert_default();
+ bound_column.get_null_map_data().push_back(1);
+ }
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {value_type, std::make_shared<DataTypeInt64>()},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ auto values_type = std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ return std::make_shared<DataTypeStruct>(DataTypes {values_type, make_nullable(value_type)},
+ Strings {"values", "bound"});
+ }
+
+private:
+ static void check_k(AggregateTopKCountData& state, const IColumn& count_column,
+ size_t row_num) {
+ DCHECK(!count_column.is_nullable());
+ int64_t value = assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(count_column)
+ .get_element(row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0 || value > MAX_TOPK_K)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of topk_min_count and topk_max_count must be a constant "
+ "between 0 and {}, but got {}",
+ MAX_TOPK_K, value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ /// Ordering of the values. FLOAT/DOUBLE use CompareFloat so that NaN sorts greatest and is
+ /// distinct from every number, which keeps the list mergeable and the top-K deterministic;
+ /// every other type uses the column's compare_at, i.e. exactly the MIN/MAX comparison.
+ static int compare_at(const IColumn& lhs, size_t lhs_row, const IColumn& rhs, size_t rhs_row) {
+ if constexpr (is_float_or_double(T)) {
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ const auto& left =
+ assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(lhs).get_data();
+ const auto& right =
+ assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(rhs).get_data();
+ return CompareFloat(left[lhs_row], right[rhs_row]);
+ } else {
+ return lhs.compare_at(lhs_row, rhs_row, rhs, 1);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "topk_min_count/topk_max_count value count overflow");
+ }
+ return result;
+ }
+
+ static void set_bound_from_value(AggregateTopKCountData& state, size_t index) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(*state.value_column, index);
+ state.has_bound = true;
+ }
+
+ /// Inserts (value, count) at `pos`, keeping the value column sorted by value.
+ static void insert_entry(AggregateTopKCountData& state, size_t pos, const IColumn& column,
+ size_t row_num, int64_t count) {
+ auto new_values = state.value_column->clone_empty();
+ new_values->reserve(state.value_column->size() + 1);
+ for (size_t i = 0; i < pos; ++i) {
+ new_values->insert_from(*state.value_column, i);
+ }
+ new_values->insert_from(column, row_num);
+ for (size_t i = pos; i < state.value_column->size(); ++i) {
+ new_values->insert_from(*state.value_column, i);
+ }
+ state.value_column = std::move(new_values);
+ state.counts.insert(state.counts.begin() + pos, count);
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(AggregateTopKCountData& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (!state.has_bound) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(column, row_num);
+ state.has_bound = true;
+ return;
+ }
+ int cmp = compare_at(column, row_num, *state.bound_column, 0);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(column, row_num);
+ }
+ return;
+ }
+
+ // The list is ascending for MIN and descending for MAX, so the insertion point is where
+ // the stored values stop being better than the new one.
+ size_t left = 0;
+ size_t right = state.value_column->size();
+ while (left < right) {
+ size_t mid = left + (right - left) / 2;
+ int cmp = compare_at(*state.value_column, mid, column, row_num);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ left = mid + 1;
+ } else {
+ right = mid;
+ }
+ }
+ if (left < state.value_column->size() &&
+ compare_at(*state.value_column, left, column, row_num) == 0) {
+ state.counts[left] = checked_add(state.counts[left], 1);
+ return;
+ }
+
+ insert_entry(state, left, column, row_num, 1);
Review Comment:
[P2] Reject worse values before rebuilding the retained column
Once a MIN state holds k values, every new distinct value larger than its last entry is immediately discarded, yet this call clones the column and copies all k retained values before popping the new entry. MAX has the symmetric path. One million ascending distinct inputs at k=65536 cause over 61 billion retained-value copies after filling the list; string inputs also copy the retained bytes. Detect this full-list rejection before `insert_entry`, update the truncation flag/bound, and return without rebuilding unchanged state.
##########
be/src/exprs/aggregate/aggregate_function_topk_count.h:
##########
@@ -0,0 +1,428 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/string_buffer.hpp"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// Upper bound of k, shared with FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/**
+ * State shared by topk_min_count and topk_max_count.
+ *
+ * The entries are held as two parallel structures: a column with the distinct values (sorted
+ * ascending for MIN / descending for MAX, at most k entries) and their non-negative int64 counts.
+ * Only values with count > 0 are kept. `bound` is a one-row column with the extreme value seen
+ * when k == 0, or the k-th value once the list was truncated when k >= 1; `has_bound` is the
+ * truncation flag, and for k >= 1 `bound IS NULL` means "the list holds every distinct value".
+ */
+struct AggregateTopKCountData {
+ const DataTypePtr value_type;
+ bool has_k = false;
+ int64_t k = 0;
+ bool has_bound = false;
+ MutableColumnPtr bound_column;
+ MutableColumnPtr value_column;
+ std::vector<int64_t> counts;
+
+ explicit AggregateTopKCountData(const DataTypes& argument_types_)
+ : value_type(argument_types_[0]),
+ bound_column(value_type->create_column()),
+ value_column(value_type->create_column()) {}
+
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * topk_min_count(x, k) / topk_max_count(x, k).
+ *
+ * Result: STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T NULL>. NULL inputs are
+ * ignored (the nullable-argument wrapper never calls add for them). `values` is ordered by value
+ * (ascending for MIN), holds at most k distinct values, and `bound` is the truncation point: the
+ * k-th value when more than k distinct values were seen, otherwise NULL.
+ *
+ * With k = 0 no list is built: `values` is empty and `bound` is the extreme value seen on this
+ * side (NULL when there was no non-NULL input), which lets the append-only IVM path compute
+ * MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopKCount final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopKCount<T, IsMin>, true>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using Base = IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopKCount<T, IsMin>, true>;
+
+ explicit AggregateFunctionTopKCount(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "topk_min_count" : "topk_max_count"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(this->argument_types[0]);
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is a constant expression, but only the first row carries it: the state remembers it so
+ // that merge can check both sides used the same capacity. A state without has_k is empty.
+ check_k(state, *columns[1], row_num);
+ add_value(state, *columns[0], row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw a non-NULL row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else {
+ // Both sides are partial states of the same constant k.
+ DCHECK_EQ(state.k, other.k);
+ }
+ if (state.k == 0) {
+ merge_bound(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ buf.write_binary(state.has_bound);
+ if (state.has_bound) {
+ write_column(buf, *state.bound_column);
+ }
+ auto size = static_cast<int64_t>(state.counts.size());
+ buf.write_binary(size);
+ write_column(buf, *state.value_column);
+ buf.write(reinterpret_cast<const char*>(state.counts.data()),
+ state.counts.size() * sizeof(int64_t));
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ buf.read_binary(state.has_bound);
+ if (state.has_bound) {
+ state.bound_column = read_column(buf);
+ } else {
+ state.bound_column = state.value_type->create_column();
+ }
+ int64_t size = 0;
+ buf.read_binary(size);
+ state.value_column = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(state.value_column->size()), size);
+ state.counts.resize(size);
+ buf.read(reinterpret_cast<char*>(state.counts.data()), size * sizeof(int64_t));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_struct = assert_cast<ColumnStruct&>(to);
+ auto& values_column = assert_cast<ColumnArray&>(result_struct.get_column(0));
+ // The array element is nullable in the planned type (ARRAY<STRUCT<...>> defaults to
+ // contains_null = true), but a produced entry never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(values_column.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = pair_column.get_column(0);
+ auto& count_column = assert_cast<ColumnInt64&>(pair_column.get_column(1));
+ for (size_t i = 0; i != state.counts.size(); ++i) {
+ value_column.insert_from(*state.value_column, i);
+ count_column.get_data().push_back(state.counts[i]);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ values_column.get_offsets().push_back(pair_column.size());
+
+ auto& bound_column = assert_cast<ColumnNullable&>(result_struct.get_column(1));
+ if (state.has_bound) {
+ bound_column.get_nested_column().insert_from(*state.bound_column, 0);
+ bound_column.get_null_map_data().push_back(0);
+ } else {
+ bound_column.get_nested_column().insert_default();
+ bound_column.get_null_map_data().push_back(1);
+ }
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {value_type, std::make_shared<DataTypeInt64>()},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ auto values_type = std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ return std::make_shared<DataTypeStruct>(DataTypes {values_type, make_nullable(value_type)},
+ Strings {"values", "bound"});
+ }
+
+private:
+ static void check_k(AggregateTopKCountData& state, const IColumn& count_column,
+ size_t row_num) {
+ DCHECK(!count_column.is_nullable());
+ int64_t value = assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(count_column)
+ .get_element(row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0 || value > MAX_TOPK_K)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of topk_min_count and topk_max_count must be a constant "
+ "between 0 and {}, but got {}",
+ MAX_TOPK_K, value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ /// Ordering of the values. FLOAT/DOUBLE use CompareFloat so that NaN sorts greatest and is
+ /// distinct from every number, which keeps the list mergeable and the top-K deterministic;
+ /// every other type uses the column's compare_at, i.e. exactly the MIN/MAX comparison.
+ static int compare_at(const IColumn& lhs, size_t lhs_row, const IColumn& rhs, size_t rhs_row) {
+ if constexpr (is_float_or_double(T)) {
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ const auto& left =
+ assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(lhs).get_data();
+ const auto& right =
+ assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(rhs).get_data();
+ return CompareFloat(left[lhs_row], right[rhs_row]);
+ } else {
+ return lhs.compare_at(lhs_row, rhs_row, rhs, 1);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "topk_min_count/topk_max_count value count overflow");
+ }
+ return result;
+ }
+
+ static void set_bound_from_value(AggregateTopKCountData& state, size_t index) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(*state.value_column, index);
+ state.has_bound = true;
+ }
+
+ /// Inserts (value, count) at `pos`, keeping the value column sorted by value.
+ static void insert_entry(AggregateTopKCountData& state, size_t pos, const IColumn& column,
+ size_t row_num, int64_t count) {
+ auto new_values = state.value_column->clone_empty();
+ new_values->reserve(state.value_column->size() + 1);
+ for (size_t i = 0; i < pos; ++i) {
+ new_values->insert_from(*state.value_column, i);
+ }
+ new_values->insert_from(column, row_num);
+ for (size_t i = pos; i < state.value_column->size(); ++i) {
+ new_values->insert_from(*state.value_column, i);
+ }
+ state.value_column = std::move(new_values);
+ state.counts.insert(state.counts.begin() + pos, count);
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(AggregateTopKCountData& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (!state.has_bound) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(column, row_num);
+ state.has_bound = true;
+ return;
+ }
+ int cmp = compare_at(column, row_num, *state.bound_column, 0);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(column, row_num);
+ }
+ return;
+ }
+
+ // The list is ascending for MIN and descending for MAX, so the insertion point is where
+ // the stored values stop being better than the new one.
+ size_t left = 0;
+ size_t right = state.value_column->size();
+ while (left < right) {
+ size_t mid = left + (right - left) / 2;
+ int cmp = compare_at(*state.value_column, mid, column, row_num);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ left = mid + 1;
+ } else {
+ right = mid;
+ }
+ }
+ if (left < state.value_column->size() &&
+ compare_at(*state.value_column, left, column, row_num) == 0) {
+ state.counts[left] = checked_add(state.counts[left], 1);
+ return;
+ }
+
+ insert_entry(state, left, column, row_num, 1);
+ if (static_cast<int64_t>(state.value_column->size()) > state.k) {
+ // More than k distinct values: drop the worst one. The k-th (last kept) value is the
+ // new truncation point, even when the value just inserted was the dropped one.
+ state.value_column->pop_back(1);
+ state.counts.pop_back();
+ set_bound_from_value(state, state.value_column->size() - 1);
+ }
+ }
+
+ static void merge_bound(AggregateTopKCountData& state, const AggregateTopKCountData& other) {
+ if (!other.has_bound) {
+ return;
+ }
+ if (!state.has_bound) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(*other.bound_column, 0);
+ state.has_bound = true;
+ return;
+ }
+ int cmp = compare_at(*other.bound_column, 0, *state.bound_column, 0);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(*other.bound_column, 0);
+ }
+ }
+
+ /// Merges two sorted lists, summing the counts of equal values and truncating to k.
+ static void merge_values(AggregateTopKCountData& state, const AggregateTopKCountData& other) {
+ if (other.counts.empty()) {
+ // The other partial state saw no non-NULL row, so it has neither entries nor a bound.
+ DCHECK(!other.has_bound);
+ return;
+ }
+ auto merged = state.value_column->clone_empty();
+ std::vector<int64_t> merged_counts;
+ merged_counts.reserve(state.counts.size() + other.counts.size());
+ size_t lhs_index = 0;
+ size_t rhs_index = 0;
+ while (lhs_index < state.counts.size() || rhs_index < other.counts.size()) {
+ if (rhs_index == other.counts.size()) {
+ merged->insert_from(*state.value_column, lhs_index);
+ merged_counts.push_back(state.counts[lhs_index]);
+ ++lhs_index;
+ } else if (lhs_index == state.counts.size()) {
+ merged->insert_from(*other.value_column, rhs_index);
+ merged_counts.push_back(other.counts[rhs_index]);
+ ++rhs_index;
+ } else {
+ int cmp =
+ compare_at(*state.value_column, lhs_index, *other.value_column, rhs_index);
+ // Both lists are ordered best-first (ascending for MIN, descending for MAX).
+ const bool lhs_is_better = IsMin ? cmp < 0 : cmp > 0;
+ if (lhs_is_better) {
+ merged->insert_from(*state.value_column, lhs_index);
+ merged_counts.push_back(state.counts[lhs_index]);
+ ++lhs_index;
+ } else if (cmp != 0) {
+ // Not better and not equal: the right-hand value is better.
+ merged->insert_from(*other.value_column, rhs_index);
+ merged_counts.push_back(other.counts[rhs_index]);
+ ++rhs_index;
+ } else {
+ merged->insert_from(*state.value_column, lhs_index);
+ merged_counts.push_back(
+ checked_add(state.counts[lhs_index], other.counts[rhs_index]));
+ ++lhs_index;
+ ++rhs_index;
+ }
+ }
+ }
+
+ // Once a side is truncated, values outside its bound were dropped, so the union cannot be
+ // complete either: the k-th merged value becomes the new truncation point.
+ bool truncated = state.has_bound || other.has_bound;
+ auto merged_size = static_cast<int64_t>(merged->size());
+ if (merged_size > state.k) {
+ merged->pop_back(merged_size - state.k);
+ merged_counts.resize(state.k);
+ truncated = true;
+ }
+ state.value_column = std::move(merged);
+ state.counts = std::move(merged_counts);
+ state.has_bound = truncated;
+ if (truncated) {
+ set_bound_from_value(state, state.value_column->size() - 1);
+ } else {
+ state.bound_column->clear();
+ }
+ }
+
+ void write_column(BufferWritable& buf, const IColumn& column) const {
+ int64_t size_bytes = this->argument_types[0]->get_uncompressed_serialized_bytes(
+ column, IAggregateFunction::version);
+ buf.write_binary(size_bytes);
+ buf.resize(size_bytes);
+ auto* pos =
+ this->argument_types[0]->serialize(column, buf.data(), IAggregateFunction::version);
+ DCHECK_EQ(pos, buf.data() + size_bytes);
Review Comment:
[P2] Distinguish serialized capacity from the encoded length
`get_uncompressed_serialized_bytes` returns an upper bound when a column exceeds the 256-byte compression threshold, while `serialize` returns the actual compressed end pointer. A partial INT state containing 0..64 with k=65 therefore fails this equality in debug/ASAN builds; a compressible string longer than 256 bytes can fail even with k=0. `read_column` repeats the same invalid equality. The existing round-trip test only stores two INT values. Frame the actual encoded length, or explicitly support padded capacity without asserting equality, and cover compressed states.
##########
be/src/exprs/aggregate/aggregate_function_topk_count.h:
##########
@@ -0,0 +1,428 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/string_buffer.hpp"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// Upper bound of k, shared with FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/**
+ * State shared by topk_min_count and topk_max_count.
+ *
+ * The entries are held as two parallel structures: a column with the distinct values (sorted
+ * ascending for MIN / descending for MAX, at most k entries) and their non-negative int64 counts.
+ * Only values with count > 0 are kept. `bound` is a one-row column with the extreme value seen
+ * when k == 0, or the k-th value once the list was truncated when k >= 1; `has_bound` is the
+ * truncation flag, and for k >= 1 `bound IS NULL` means "the list holds every distinct value".
+ */
+struct AggregateTopKCountData {
+ const DataTypePtr value_type;
+ bool has_k = false;
+ int64_t k = 0;
+ bool has_bound = false;
+ MutableColumnPtr bound_column;
+ MutableColumnPtr value_column;
+ std::vector<int64_t> counts;
+
+ explicit AggregateTopKCountData(const DataTypes& argument_types_)
+ : value_type(argument_types_[0]),
+ bound_column(value_type->create_column()),
+ value_column(value_type->create_column()) {}
+
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * topk_min_count(x, k) / topk_max_count(x, k).
+ *
+ * Result: STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T NULL>. NULL inputs are
+ * ignored (the nullable-argument wrapper never calls add for them). `values` is ordered by value
+ * (ascending for MIN), holds at most k distinct values, and `bound` is the truncation point: the
+ * k-th value when more than k distinct values were seen, otherwise NULL.
+ *
+ * With k = 0 no list is built: `values` is empty and `bound` is the extreme value seen on this
+ * side (NULL when there was no non-NULL input), which lets the append-only IVM path compute
+ * MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopKCount final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopKCount<T, IsMin>, true>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using Base = IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopKCount<T, IsMin>, true>;
+
+ explicit AggregateFunctionTopKCount(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "topk_min_count" : "topk_max_count"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(this->argument_types[0]);
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is a constant expression, but only the first row carries it: the state remembers it so
+ // that merge can check both sides used the same capacity. A state without has_k is empty.
+ check_k(state, *columns[1], row_num);
+ add_value(state, *columns[0], row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw a non-NULL row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else {
+ // Both sides are partial states of the same constant k.
+ DCHECK_EQ(state.k, other.k);
Review Comment:
[P2] Reject incompatible k values across AggState merges
The same-k assumption holds for ordinary partial aggregation, but these functions also expose `_state`/`_merge`/`_union`. AggState type identity records argument types and nullability, not literal k, so a UNION ALL of `topk_min_count_state(2, CAST(1 AS BIGINT))` and `topk_min_count_state(1, CAST(0 AS BIGINT))` can feed `topk_min_count_merge(s)`. With the k=1 state merged first, release builds ignore the k=0 state's extreme and return value 2 as a complete state; reversing the order changes the result. Checked builds hit this DCHECK. Reject mismatched serialized capacities with a query error, or disable AggState support until this case is defined.
##########
be/src/exprs/aggregate/aggregate_function_topk_count.h:
##########
@@ -0,0 +1,428 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/string_buffer.hpp"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// Upper bound of k, shared with FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/**
+ * State shared by topk_min_count and topk_max_count.
+ *
+ * The entries are held as two parallel structures: a column with the distinct values (sorted
+ * ascending for MIN / descending for MAX, at most k entries) and their non-negative int64 counts.
+ * Only values with count > 0 are kept. `bound` is a one-row column with the extreme value seen
+ * when k == 0, or the k-th value once the list was truncated when k >= 1; `has_bound` is the
+ * truncation flag, and for k >= 1 `bound IS NULL` means "the list holds every distinct value".
+ */
+struct AggregateTopKCountData {
+ const DataTypePtr value_type;
+ bool has_k = false;
+ int64_t k = 0;
+ bool has_bound = false;
+ MutableColumnPtr bound_column;
+ MutableColumnPtr value_column;
+ std::vector<int64_t> counts;
Review Comment:
[P2] Track the owned count buffers with a Doris allocator
Both this persistent `counts` vector and `merged_counts` use the default allocator, bypassing Doris query memory accounting and allocation-limit checks. At k=65536 the count payload alone is at least 512 KiB per group, with additional merge scratch; many groups can retain substantial unreported memory. The aggregate helper reports only sizeof(Data), and the aggregation sink counts arenas/container/hash-table storage, so neither accounts for these separate heaps. Use `DorisVector<int64_t>` or another tracked container for both buffers so growth, moves and destruction participate in query memory tracking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6345695422
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6355005059
# FE Regression Coverage Report
Increment line coverage ` 79.53% (136/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6358725163
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6380652830
# BE Regression && UT Coverage Report
Increment line coverage `81.76% (269/329)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 68.40% (31863/46585) |
| Line Coverage | 53.06% (345074/650355) |
| Region Coverage | 48.42% (282023/582490) |
| Branch Coverage | 49.90% (132269/265083) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6359625087
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4357355513
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/TopKFunction.java:
##########
@@ -0,0 +1,148 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.agg;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.CustomSignature;
+import org.apache.doris.nereids.trees.expressions.literal.ArrayLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.StructLiteral;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.DecimalV2Type;
+import org.apache.doris.nereids.types.DecimalV3Type;
+import org.apache.doris.nereids.types.StructField;
+import org.apache.doris.nereids.types.StructType;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Shared implementation of {@code topk_min_count(x, k)} and {@code topk_max_count(x, k)}.
+ *
+ * <p>Both aggregates return {@code STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T>}.
+ * {@code values} holds at most k distinct non-NULL values ordered by value (ascending for MIN,
+ * descending for MAX) with the number of rows equal to each value. {@code bound} is NULL when the
+ * group was not truncated; otherwise it is the k-th value and {@code bound IS NOT NULL} is the
+ * truncation flag. With {@code k = 0} no list is built and {@code bound} is the extreme value seen
+ * on this side (NULL when no non-NULL input was seen).
+ */
+public abstract class TopKCountFunction extends NotNullableAggregateFunction
+ implements CustomSignature, NullIgnoringAggregateFunction {
+
+ /** Upper bound of k, shared with the BE implementation. */
+ public static final long MAX_TOPK_K = 65536;
+
+ protected TopKCountFunction(String name, boolean distinct, List<Expression> children) {
+ super(name, distinct, false, children);
+ }
+
+ /** constructor for withChildren and reuse signature */
+ protected TopKCountFunction(AggregateFunctionParams functionParams) {
+ super(functionParams);
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ if (distinct) {
+ throw new AnalysisException(getName() + " does not support DISTINCT");
+ }
+ if (!getArgument(1).isConstant()) {
+ throw new AnalysisException(
+ getName() + " requires second parameter must be a constant: " + toSql());
+ }
+ if (!isSupportedValueType(getArgumentType(0))) {
+ throw new AnalysisException(getName() + " does not support type "
+ + getArgumentType(0).toSql());
+ }
+ }
+
+ @Override
+ public void checkLegalityAfterRewrite() {
+ checkLegalityBeforeTypeCoercion();
+ Expression count = getArgument(1);
+ // A constant FE cannot fold is left to BE, which reads the value from the count column.
+ if (!(count instanceof Literal)) {
Review Comment:
Fixed in 1c73d49d491: `TopKFunction` no longer implements `NullIgnoringAggregateFunction`. The marker's contract (an input consisting only of rows whose argument is NULL behaves like an empty input) does not hold while a backend-evaluated count is validated per input row, so `InferAggNotNull` must not filter those rows away for these aggregates.
Without the marker the global aggregate keeps its rows, `add` validates the count before skipping a NULL value, and an all-NULL input with a non-literal NULL count now reports `the count of min_topk and max_topk must be a non-null constant` instead of returning an empty array. NULL values of x are still ignored by `add` itself, so the aggregate's result is unchanged for valid counts.
Covered in `query_p0/aggregate/min_max_topk`: the all-NULL global input with `bitmap_min(bitmap_from_string(''))` fails for both aggregates, and the same shape with the valid non-literal count `bitmap_min(bitmap_from_string('1'))` still returns the empty array.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6360255624
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4357870878
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArraySortedMerge.java:
##########
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.scalar;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.ArrayItemReference;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.ImplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.functions.PropagateNullable;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.BooleanType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.LambdaType;
+import org.apache.doris.nereids.types.coercion.FollowToAnyDataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc): merge two arrays that are already sorted.
+ *
+ * <pre>
+ * array_sorted_merge((a, b) -> ..., arr1, arr2, true)
+ * </pre>
+ *
+ * <p>The inputs must already be sorted by the elements' natural order in the direction given by
+ * {@code is_asc}; the function only merges them. The merge lambda receives the two current
+ * candidates as {@code (a, b)} and returns the combined element; returning NULL means the two
+ * elements must not be combined, so both are kept. Each key occurs at most once per input, so a
+ * key appears at most twice across the two arrays and the lambda is called at most once per key.
+ *
+ * <p>A NULL input array makes the result NULL, like the other higher-order array functions.
+ */
+public class ArraySortedMerge extends ScalarFunction
+ implements ImplicitlyCastableSignature, PropagateNullable {
Review Comment:
Fixed in 8281c36a07d: the per-row null check of the ascending flag now runs before the branch that returns NULL for a NULL input array. The flag is an argument contract rather than data, so a backend-evaluated NULL flag now reports `array_sorted_merge requires a non-null ascending flag` even when the input array row is NULL; the same flag with a non-NULL array already failed.
A NULL input array with a valid flag still returns NULL, and the literal-null-flag case is still rejected during analysis. `test_array_sorted_merge` adds `CAST(bitmap_min(bitmap_from_string('')) AS BOOLEAN)` with a NULL first array, alongside the existing non-NULL-array case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6361365771
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6367938637
# FE Regression Coverage Report
Increment line coverage ` 81.29% (139/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6368797020
# FE Regression Coverage Report
Increment line coverage ` 81.29% (139/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6434499419
<details>
<summary>TPC-DS: <b>Total hot run time: 150748 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5309c29db2b57c559711d38b6621f5c934c95d3a, data reload: false
query5 4313 584 440 440
query6 447 210 194 194
query7 4808 452 240 240
query8 322 163 150 150
query9 8766 3119 3126 3119
query10 485 314 259 259
query11 6037 3872 3534 3534
query12 147 92 85 85
query13 1262 441 328 328
query14 6151 3925 3653 3653
query14_1 3424 3408 3439 3408
query15 205 198 176 176
query16 964 432 406 406
query17 862 674 540 540
query18 2416 422 306 306
query19 197 171 163 163
query20 83 82 81 81
query21 216 123 103 103
query22 12838 12917 12741 12741
query23 13392 12868 12420 12420
query23_1 12610 12558 12589 12558
query24 7327 811 449 449
query24_1 463 464 459 459
query25 502 392 345 345
query26 1256 248 133 133
query27 2804 428 259 259
query28 4652 1940 1949 1940
query29 1570 566 427 427
query30 297 222 178 178
query31 994 924 812 812
query32 145 89 87 87
query33 524 298 233 233
query34 894 833 495 495
query35 746 760 638 638
query36 795 810 706 706
query37 121 94 75 75
query38 1767 1698 1633 1633
query39 796 799 744 744
query39_1 723 727 738 727
query40 216 117 92 92
query41 70 68 64 64
query42 88 85 82 82
query43 338 361 302 302
query44 1335 697 690 690
query45 174 178 159 159
query46 806 954 563 563
query47 2845 2893 2729 2729
query48 295 312 234 234
query49 568 413 294 294
query50 693 273 207 207
query51 10607 10683 10558 10558
query52 76 80 68 68
query53 181 210 151 151
query54 233 199 188 188
query55 71 76 69 69
query56 241 228 228 228
query57 1635 1611 1522 1522
query58 299 309 255 255
query59 2174 2228 1981 1981
query60 281 237 234 234
query61 147 146 144 144
query62 384 331 280 280
query63 191 156 155 155
query64 2760 938 758 758
query65 3296 3229 3273 3229
query66 1787 407 297 297
query67 20232 20182 20118 20118
query68 2904 1019 602 602
query69 381 289 247 247
query70 842 778 774 774
query71 280 230 215 215
query72 2770 2616 2302 2302
query73 520 521 296 296
query74 4610 4479 4321 4321
query75 2357 2296 1920 1920
query76 2314 998 598 598
query77 328 370 262 262
query78 9696 9637 9212 9212
query79 871 835 516 516
query80 470 461 345 345
query81 523 334 278 278
query82 198 131 115 115
query83 193 196 204 196
query84 289 123 108 108
query85 773 429 379 379
query86 256 239 228 228
query87 1982 1941 1800 1800
query88 3554 2680 2638 2638
query89 317 286 242 242
query90 2092 166 159 159
query91 153 165 130 130
query92 96 87 89 87
query93 974 996 573 573
query94 438 304 282 282
query95 581 380 312 312
query96 667 487 235 235
query97 2480 2432 2318 2318
query98 158 154 148 148
query99 763 751 655 655
Total cold run time: 233011 ms
Total hot run time: 150748 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6432148752
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6494662544
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6492167073
<details>
<summary>TPC-DS: <b>Total hot run time: 151868 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d9d17e663629a2bc41b45deb64a2e384b1f41df2, data reload: false
query5 4314 592 447 447
query6 432 206 190 190
query7 4808 447 231 231
query8 322 175 159 159
query9 8790 3199 3191 3191
query10 478 333 262 262
query11 5997 3904 3598 3598
query12 145 94 88 88
query13 1251 433 336 336
query14 6140 4012 3652 3652
query14_1 3415 3444 3494 3444
query15 211 205 191 191
query16 937 449 332 332
query17 859 692 559 559
query18 2429 424 317 317
query19 193 165 138 138
query20 87 86 88 86
query21 212 131 107 107
query22 13035 13023 12994 12994
query23 13629 12855 12437 12437
query23_1 12622 12626 12510 12510
query24 7378 826 479 479
query24_1 490 482 479 479
query25 546 424 365 365
query26 1265 261 139 139
query27 2780 455 279 279
query28 4608 2025 2061 2025
query29 1567 616 466 466
query30 299 227 191 191
query31 1026 951 836 836
query32 148 101 95 95
query33 539 312 262 262
query34 910 866 500 500
query35 745 748 688 688
query36 812 820 724 724
query37 123 96 81 81
query38 1792 1735 1678 1678
query39 794 795 759 759
query39_1 728 721 735 721
query40 216 114 92 92
query41 67 62 61 61
query42 85 80 81 80
query43 348 360 313 313
query44 1310 703 703 703
query45 175 174 168 168
query46 851 972 576 576
query47 2933 2900 2743 2743
query48 296 305 222 222
query49 578 392 298 298
query50 701 268 205 205
query51 10191 10357 10561 10357
query52 79 78 71 71
query53 186 214 153 153
query54 239 188 201 188
query55 73 66 66 66
query56 255 211 221 211
query57 1674 1616 1570 1570
query58 286 268 248 248
query59 2156 2182 1955 1955
query60 283 242 227 227
query61 142 150 147 147
query62 392 331 282 282
query63 189 157 160 157
query64 2738 974 759 759
query65 3356 3315 3286 3286
query66 1766 428 324 324
query67 20355 20314 20147 20147
query68 3014 1003 612 612
query69 381 289 291 289
query70 862 779 774 774
query71 298 239 209 209
query72 2717 2621 2368 2368
query73 524 531 301 301
query74 4682 4524 4345 4345
query75 2353 2297 1942 1942
query76 2308 1034 623 623
query77 339 393 291 291
query78 9785 9755 9161 9161
query79 1044 921 533 533
query80 565 434 356 356
query81 547 328 271 271
query82 592 134 104 104
query83 301 210 181 181
query84 325 121 102 102
query85 864 450 372 372
query86 332 234 234 234
query87 2044 1969 1821 1821
query88 3785 2780 2763 2763
query89 333 281 245 245
query90 1861 170 166 166
query91 170 156 130 130
query92 105 107 91 91
query93 1031 1008 579 579
query94 443 332 281 281
query95 582 390 315 315
query96 690 495 244 244
query97 2465 2442 2321 2321
query98 163 156 152 152
query99 785 759 657 657
Total cold run time: 234859 ms
Total hot run time: 151868 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6465615016
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6469234016
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4400477765
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,786 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ // The payload is proportional to the slot count, which the caller already compares.
+ size_t live_bytes() const { return 0; }
+ size_t dead_bytes() const { return 0; }
+ void on_evict(size_t /*slot*/) {}
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ size_t offset = 0;
+ size_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ /// PaddedPODArray, not PODArray: comparisons go through
+ /// memcmp_small_allow_overflow15, which loads a full 16-byte vector even for a one-byte
+ /// string, the same tail allowance ColumnString's chars buffer provides.
+ PaddedPODArray<char, 32> bytes;
+ DorisVector<Span> spans;
+ // Sum of the live slots' payload and of the payload evicted since the last compaction. The
+ // slot counters alone cannot bound the memory: replacing 1024 1 MiB strings with short ones
+ // leaves dead == live by slot count while ~1 GiB of discarded bytes stay in the arena.
+ size_t live_payload_bytes = 0;
+ size_t dead_payload_bytes = 0;
+
+ size_t size() const { return spans.size(); }
+ size_t live_bytes() const { return live_payload_bytes; }
+ size_t dead_bytes() const { return dead_payload_bytes; }
+ void reserve(size_t n) { spans.reserve(n); }
+ void clear() {
+ bytes.clear();
+ spans.clear();
+ live_payload_bytes = 0;
+ dead_payload_bytes = 0;
+ }
+ StringRef at(size_t slot) const {
+ const Span& span = spans[slot];
+ return StringRef(bytes.data() + span.offset, span.length);
+ }
+ void append(StringRef value) {
+ spans.push_back({bytes.size(), value.size});
+ live_payload_bytes += value.size;
+ if (value.size != 0) {
+ bytes.insert(value.data, value.data + value.size);
+ }
+ }
+ /// Called when `slot` is evicted: its payload stays in the arena until the next compaction.
+ void on_evict(size_t slot) {
+ live_payload_bytes -= spans[slot].length;
+ dead_payload_bytes += spans[slot].length;
+ }
+ void swap(StringTopKValueStore& other) noexcept {
+ bytes.swap(other.bytes);
+ spans.swap(other.spans);
+ std::swap(live_payload_bytes, other.live_payload_bytes);
+ std::swap(dead_payload_bytes, other.dead_payload_bytes);
+ }
+};
+
+/**
+ * State shared by min_topk and max_topk, for one value type.
+ *
+ * The distinct values sit in slots (see the stores above), with one Slot record per value carrying
+ * the row count and the links of a treap ordered best-first (ascending values for MIN, descending
+ * for MAX). A row costs one O(log k) descent: it either finds its value and bumps the count or
+ * links a new slot at the leaf. A sorted vector would shift O(k) elements for the same insert.
+ *
+ * The treap stores explicit child links because, unlike a heap, its shape is not the complete tree
+ * that an array index can describe: rotations move entries, and the in-order sequence is the value
+ * order rather than the arrival order. A heap could evict and append in O(log k), but it cannot
+ * find an existing value without a hash index or a linear scan, and it cannot produce the sorted
+ * result without sorting. The ordered tree is the structure that keeps find, insert and eviction
+ * all at O(log k) with comparisons only.
+ *
+ * An entry that is evicted keeps its slot and value until the next compaction. The tree never
+ * points at a dead slot.
+ *
+ * With k == 0 the state holds at most the single extreme value and its row count; the caller
+ * derives truncation from the list length, so the state carries no separate bound.
+ */
+template <PrimitiveType T>
+struct AggregateTopKCountData {
+ using CppType = typename PrimitiveTypeTraits<T>::CppType;
+ static constexpr bool IS_STRING = is_string_type(T);
+ using ValueStore =
+ std::conditional_t<IS_STRING, StringTopKValueStore, FixedTopKValueStore<CppType>>;
+ using ValueRef = std::conditional_t<IS_STRING, StringRef, const CppType&>;
+
+ static constexpr int32_t NO_SLOT = -1;
+ // Row counts are never negative, so this cannot collide with a live entry.
+ static constexpr int64_t DEAD_COUNT = -1;
+
+ /// One slot's metadata; the value itself lives in `values` at the same index.
+ struct Slot {
+ int64_t count = 0;
+ int32_t left = NO_SLOT;
+ int32_t right = NO_SLOT;
+ uint32_t priority = 0;
+ };
+
+ bool has_k = false;
+ int64_t k = 0;
+ // Indexed by slot; an evicted slot stays as a dead entry until compaction.
+ ValueStore values;
+ DorisVector<Slot> slots;
+ int32_t root = NO_SLOT;
+ // The last live slot in the aggregate's order: the value an incoming row has to beat to enter
+ // a full list. Cached so that rejecting a worse value costs one comparison, not a descent.
+ int32_t worst = NO_SLOT;
+ int64_t live = 0;
+ // Slots evicted since the last compaction.
+ int64_t dead = 0;
+ uint64_t rng = 0x243F6A8885A308D3ULL;
+
+ AggregateTopKCountData() = default;
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * min_topk(x, k) / max_topk(x, k).
+ *
+ * Result: ARRAY<STRUCT<value T, count BIGINT>>. NULL inputs are ignored: add validates k for
+ * every row and then skips a NULL x, so the aggregate deliberately does not use the
+ * nullable-argument wrapper. The array is ordered by value (ascending for MIN), holds at most k
+ * distinct values with count > 0, and carries no truncation flag: the caller infers it from the
+ * length (a list of exactly k entries is treated as possibly truncated).
+ *
+ * With k = 0 the array holds at most the extreme value and its row count, which lets the
+ * append-only IVM path compute MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopK final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData<T>,
+ AggregateFunctionTopK<T, IsMin>, false>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using State = AggregateTopKCountData<T>;
+ using Base = IAggregateFunctionDataHelper<State, AggregateFunctionTopK<T, IsMin>, false>;
+ using Slot = typename State::Slot;
+ using ValueStore = typename State::ValueStore;
+ using ValueRef = typename State::ValueRef;
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ static constexpr bool IS_STRING = State::IS_STRING;
+ static constexpr int32_t NO_SLOT = State::NO_SLOT;
+
+ explicit AggregateFunctionTopK(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "min_topk" : "max_topk"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(remove_nullable(this->argument_types[0]));
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is read for every row, including rows where x is NULL: an evaluated count that is
+ // NULL or out of range must fail the query instead of being skipped like a NULL value.
+ check_k(state, *columns[1], row_num);
+ const IColumn* value_column = columns[0];
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(value_column)) {
+ if (nullable->is_null_at(row_num)) {
+ return;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ add_value(state, *value_column, row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw any row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else if (UNLIKELY(state.k != other.k)) {
+ // Ordinary partial/final aggregation always shares the same constant k, but the
+ // public _state/_merge/_union combinators do not encode k in the AggState type, so a
+ // UNION ALL of states built with different literal k can reach this merge.
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} states have incompatible counts: {} vs {}", get_name(), state.k,
+ other.k);
+ }
+ if (state.k == 0) {
+ merge_extreme(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ // The slots are in arrival order and may include dead ones, while the serialized form is
+ // the sorted (value, count) list the reader rebuilds the tree from.
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ auto sorted_values = remove_nullable(this->argument_types[0])->create_column();
+ sorted_values->reserve(order.size());
+ DorisVector<int64_t> sorted_counts;
+ sorted_counts.reserve(order.size());
+ for (int32_t slot : order) {
+ insert_value(*sorted_values, state.values.at(slot));
+ sorted_counts.push_back(state.slots[slot].count);
+ }
+ auto size = static_cast<int64_t>(sorted_counts.size());
+ buf.write_binary(size);
+ write_column(buf, *sorted_values);
+ if (size != 0) {
+ // An empty list keeps an empty vector, whose data() is null; passing it to memcpy is
+ // undefined even for a zero length.
+ buf.write(reinterpret_cast<const char*>(sorted_counts.data()),
+ sorted_counts.size() * sizeof(int64_t));
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ int64_t size = 0;
+ buf.read_binary(size);
+ auto values = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(values->size()), size);
+ DorisVector<int64_t> counts(static_cast<size_t>(size));
+ if (size != 0) {
+ buf.read(reinterpret_cast<char*>(counts.data()), size * sizeof(int64_t));
+ }
+ ValueStore value_store;
+ value_store.reserve(static_cast<size_t>(size));
+ for (int64_t row = 0; row != size; ++row) {
+ value_store.append(value_at(*values, static_cast<size_t>(row)));
+ }
+ assign_sorted(state, std::move(value_store), std::move(counts));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_array = assert_cast<ColumnArray&>(to);
+ // The array element and both fields are nullable in the planned type, but a produced entry
+ // never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(result_array.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = assert_cast<ColumnNullable&>(pair_column.get_column(0));
+ auto& count_column = assert_cast<ColumnNullable&>(pair_column.get_column(1));
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ for (int32_t slot : order) {
+ insert_value(value_column.get_nested_column(), state.values.at(slot));
+ value_column.get_null_map_data().push_back(0);
+ assert_cast<ColumnInt64&>(count_column.get_nested_column())
+ .get_data()
+ .push_back(state.slots[slot].count);
+ count_column.get_null_map_data().push_back(0);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ result_array.get_offsets().push_back(pair_column.size());
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ // Both fields are nullable in the planned type: a consumer reads a field through
+ // element_at, which is always nullable, and named_struct then records nullable fields.
+ // Produced entries never contain a NULL field.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(value_type),
+ make_nullable(std::make_shared<DataTypeInt64>())},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ return std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ }
+
+private:
+ /// The value of the row in `column`. The returned view is valid while the column is alive.
+ static ValueRef value_at(const IColumn& column, size_t row_num) {
+ if constexpr (IS_STRING) {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column).get_data_at(
+ row_num);
+ } else {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column)
+ .get_data()[row_num];
+ }
+ }
+
+ /// Exactly the comparison the value column would apply: CompareFloat for FLOAT/DOUBLE through
+ /// Compare::compare (the column uses the same helper), the same memcmp helper ColumnString
+ /// uses for strings, and the column's scale-independent ordering for every other type.
+ static int compare_values(ValueRef lhs, ValueRef rhs) {
+ if constexpr (IS_STRING) {
+ return memcmp_small_allow_overflow15(
+ reinterpret_cast<const uint8_t*>(lhs.data), lhs.size,
+ reinterpret_cast<const uint8_t*>(rhs.data), rhs.size);
+ } else {
+ return Compare::compare(lhs, rhs);
+ }
+ }
+
+ static void insert_value(IColumn& column, ValueRef value) {
+ if constexpr (IS_STRING) {
+ assert_cast<ColumnString&>(column).insert_data(value.data, value.size);
+ } else {
+ assert_cast<ColumnType&>(column).get_data().push_back(value);
+ }
+ }
+
+ /// True when `cmp` puts the left operand before the right one in the aggregate's order.
+ static bool comes_before(int cmp) { return IsMin ? cmp < 0 : cmp > 0; }
+
+ /// True when `cmp`, which compares a value with the worst retained one, puts it after it: the
+ /// value is worse and cannot enter the list.
+ static bool is_worse(int cmp) { return IsMin ? cmp > 0 : cmp < 0; }
+
+ /// splitmix64. The priorities only have to be independent enough to keep the treap balanced;
+ /// a per-state sequence keeps the shape reproducible for the same insertion order.
+ static uint32_t next_priority(State& state) {
+ uint64_t z = (state.rng += 0x9E3779B97F4A7C15ULL);
+ z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9ULL;
+ z = (z ^ (z >> 27)) * 0x94D049BB133111EBULL;
+ return static_cast<uint32_t>(z ^ (z >> 31));
+ }
+
+ static int compare_slots(const State& state, int32_t lhs, int32_t rhs) {
+ return compare_values(state.values.at(lhs), state.values.at(rhs));
+ }
+
+ /// Appends (value, count) as the next slot.
+ static int32_t append_slot(State& state, ValueRef value, int64_t count) {
+ auto slot = static_cast<int32_t>(state.slots.size());
Review Comment:
[P2] Check slot capacity before narrowing the index
The FE accepts any nonnegative BIGINT k, but tree links use int32_t. With `k=2147483649`, a group receiving that many distinct BIGINT values and enough query memory to retain them reaches `slots.size()==2147483648`; this cast becomes a negative index, which `add_or_increment` stores as a child and immediately uses to access its priority. Checked builds fail at the preceding index's DCHECK instead. This requires a very large memory allowance, but neither the allocator nor a prior aggregate-state size check excludes it; output/serialization checks occur after the state is built. Check the actual slot count before narrowing and return a normal query error, or widen the indices consistently in insertion and rebuild.
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,786 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ // The payload is proportional to the slot count, which the caller already compares.
+ size_t live_bytes() const { return 0; }
+ size_t dead_bytes() const { return 0; }
+ void on_evict(size_t /*slot*/) {}
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ size_t offset = 0;
+ size_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ /// PaddedPODArray, not PODArray: comparisons go through
+ /// memcmp_small_allow_overflow15, which loads a full 16-byte vector even for a one-byte
+ /// string, the same tail allowance ColumnString's chars buffer provides.
+ PaddedPODArray<char, 32> bytes;
+ DorisVector<Span> spans;
+ // Sum of the live slots' payload and of the payload evicted since the last compaction. The
+ // slot counters alone cannot bound the memory: replacing 1024 1 MiB strings with short ones
+ // leaves dead == live by slot count while ~1 GiB of discarded bytes stay in the arena.
+ size_t live_payload_bytes = 0;
+ size_t dead_payload_bytes = 0;
+
+ size_t size() const { return spans.size(); }
+ size_t live_bytes() const { return live_payload_bytes; }
+ size_t dead_bytes() const { return dead_payload_bytes; }
+ void reserve(size_t n) { spans.reserve(n); }
+ void clear() {
+ bytes.clear();
+ spans.clear();
+ live_payload_bytes = 0;
+ dead_payload_bytes = 0;
+ }
+ StringRef at(size_t slot) const {
+ const Span& span = spans[slot];
+ return StringRef(bytes.data() + span.offset, span.length);
+ }
+ void append(StringRef value) {
+ spans.push_back({bytes.size(), value.size});
+ live_payload_bytes += value.size;
+ if (value.size != 0) {
+ bytes.insert(value.data, value.data + value.size);
+ }
+ }
+ /// Called when `slot` is evicted: its payload stays in the arena until the next compaction.
+ void on_evict(size_t slot) {
+ live_payload_bytes -= spans[slot].length;
+ dead_payload_bytes += spans[slot].length;
+ }
+ void swap(StringTopKValueStore& other) noexcept {
+ bytes.swap(other.bytes);
+ spans.swap(other.spans);
+ std::swap(live_payload_bytes, other.live_payload_bytes);
+ std::swap(dead_payload_bytes, other.dead_payload_bytes);
+ }
+};
+
+/**
+ * State shared by min_topk and max_topk, for one value type.
+ *
+ * The distinct values sit in slots (see the stores above), with one Slot record per value carrying
+ * the row count and the links of a treap ordered best-first (ascending values for MIN, descending
+ * for MAX). A row costs one O(log k) descent: it either finds its value and bumps the count or
+ * links a new slot at the leaf. A sorted vector would shift O(k) elements for the same insert.
+ *
+ * The treap stores explicit child links because, unlike a heap, its shape is not the complete tree
+ * that an array index can describe: rotations move entries, and the in-order sequence is the value
+ * order rather than the arrival order. A heap could evict and append in O(log k), but it cannot
+ * find an existing value without a hash index or a linear scan, and it cannot produce the sorted
+ * result without sorting. The ordered tree is the structure that keeps find, insert and eviction
+ * all at O(log k) with comparisons only.
+ *
+ * An entry that is evicted keeps its slot and value until the next compaction. The tree never
+ * points at a dead slot.
+ *
+ * With k == 0 the state holds at most the single extreme value and its row count; the caller
+ * derives truncation from the list length, so the state carries no separate bound.
+ */
+template <PrimitiveType T>
+struct AggregateTopKCountData {
+ using CppType = typename PrimitiveTypeTraits<T>::CppType;
+ static constexpr bool IS_STRING = is_string_type(T);
+ using ValueStore =
+ std::conditional_t<IS_STRING, StringTopKValueStore, FixedTopKValueStore<CppType>>;
+ using ValueRef = std::conditional_t<IS_STRING, StringRef, const CppType&>;
+
+ static constexpr int32_t NO_SLOT = -1;
+ // Row counts are never negative, so this cannot collide with a live entry.
+ static constexpr int64_t DEAD_COUNT = -1;
+
+ /// One slot's metadata; the value itself lives in `values` at the same index.
+ struct Slot {
+ int64_t count = 0;
+ int32_t left = NO_SLOT;
+ int32_t right = NO_SLOT;
+ uint32_t priority = 0;
+ };
+
+ bool has_k = false;
+ int64_t k = 0;
+ // Indexed by slot; an evicted slot stays as a dead entry until compaction.
+ ValueStore values;
+ DorisVector<Slot> slots;
+ int32_t root = NO_SLOT;
+ // The last live slot in the aggregate's order: the value an incoming row has to beat to enter
+ // a full list. Cached so that rejecting a worse value costs one comparison, not a descent.
+ int32_t worst = NO_SLOT;
+ int64_t live = 0;
+ // Slots evicted since the last compaction.
+ int64_t dead = 0;
+ uint64_t rng = 0x243F6A8885A308D3ULL;
+
+ AggregateTopKCountData() = default;
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * min_topk(x, k) / max_topk(x, k).
+ *
+ * Result: ARRAY<STRUCT<value T, count BIGINT>>. NULL inputs are ignored: add validates k for
+ * every row and then skips a NULL x, so the aggregate deliberately does not use the
+ * nullable-argument wrapper. The array is ordered by value (ascending for MIN), holds at most k
+ * distinct values with count > 0, and carries no truncation flag: the caller infers it from the
+ * length (a list of exactly k entries is treated as possibly truncated).
+ *
+ * With k = 0 the array holds at most the extreme value and its row count, which lets the
+ * append-only IVM path compute MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopK final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData<T>,
+ AggregateFunctionTopK<T, IsMin>, false>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using State = AggregateTopKCountData<T>;
+ using Base = IAggregateFunctionDataHelper<State, AggregateFunctionTopK<T, IsMin>, false>;
+ using Slot = typename State::Slot;
+ using ValueStore = typename State::ValueStore;
+ using ValueRef = typename State::ValueRef;
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ static constexpr bool IS_STRING = State::IS_STRING;
+ static constexpr int32_t NO_SLOT = State::NO_SLOT;
+
+ explicit AggregateFunctionTopK(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "min_topk" : "max_topk"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(remove_nullable(this->argument_types[0]));
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is read for every row, including rows where x is NULL: an evaluated count that is
+ // NULL or out of range must fail the query instead of being skipped like a NULL value.
+ check_k(state, *columns[1], row_num);
+ const IColumn* value_column = columns[0];
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(value_column)) {
+ if (nullable->is_null_at(row_num)) {
+ return;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ add_value(state, *value_column, row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw any row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else if (UNLIKELY(state.k != other.k)) {
+ // Ordinary partial/final aggregation always shares the same constant k, but the
+ // public _state/_merge/_union combinators do not encode k in the AggState type, so a
+ // UNION ALL of states built with different literal k can reach this merge.
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} states have incompatible counts: {} vs {}", get_name(), state.k,
+ other.k);
+ }
+ if (state.k == 0) {
+ merge_extreme(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ // The slots are in arrival order and may include dead ones, while the serialized form is
+ // the sorted (value, count) list the reader rebuilds the tree from.
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ auto sorted_values = remove_nullable(this->argument_types[0])->create_column();
+ sorted_values->reserve(order.size());
+ DorisVector<int64_t> sorted_counts;
+ sorted_counts.reserve(order.size());
+ for (int32_t slot : order) {
+ insert_value(*sorted_values, state.values.at(slot));
+ sorted_counts.push_back(state.slots[slot].count);
+ }
+ auto size = static_cast<int64_t>(sorted_counts.size());
+ buf.write_binary(size);
+ write_column(buf, *sorted_values);
+ if (size != 0) {
+ // An empty list keeps an empty vector, whose data() is null; passing it to memcpy is
+ // undefined even for a zero length.
+ buf.write(reinterpret_cast<const char*>(sorted_counts.data()),
+ sorted_counts.size() * sizeof(int64_t));
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ int64_t size = 0;
+ buf.read_binary(size);
+ auto values = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(values->size()), size);
+ DorisVector<int64_t> counts(static_cast<size_t>(size));
+ if (size != 0) {
+ buf.read(reinterpret_cast<char*>(counts.data()), size * sizeof(int64_t));
+ }
+ ValueStore value_store;
+ value_store.reserve(static_cast<size_t>(size));
+ for (int64_t row = 0; row != size; ++row) {
+ value_store.append(value_at(*values, static_cast<size_t>(row)));
+ }
+ assign_sorted(state, std::move(value_store), std::move(counts));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_array = assert_cast<ColumnArray&>(to);
+ // The array element and both fields are nullable in the planned type, but a produced entry
+ // never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(result_array.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = assert_cast<ColumnNullable&>(pair_column.get_column(0));
+ auto& count_column = assert_cast<ColumnNullable&>(pair_column.get_column(1));
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ for (int32_t slot : order) {
+ insert_value(value_column.get_nested_column(), state.values.at(slot));
+ value_column.get_null_map_data().push_back(0);
+ assert_cast<ColumnInt64&>(count_column.get_nested_column())
+ .get_data()
+ .push_back(state.slots[slot].count);
+ count_column.get_null_map_data().push_back(0);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ result_array.get_offsets().push_back(pair_column.size());
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ // Both fields are nullable in the planned type: a consumer reads a field through
+ // element_at, which is always nullable, and named_struct then records nullable fields.
+ // Produced entries never contain a NULL field.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(value_type),
+ make_nullable(std::make_shared<DataTypeInt64>())},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ return std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ }
+
+private:
+ /// The value of the row in `column`. The returned view is valid while the column is alive.
+ static ValueRef value_at(const IColumn& column, size_t row_num) {
+ if constexpr (IS_STRING) {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column).get_data_at(
+ row_num);
+ } else {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column)
+ .get_data()[row_num];
+ }
+ }
+
+ /// Exactly the comparison the value column would apply: CompareFloat for FLOAT/DOUBLE through
+ /// Compare::compare (the column uses the same helper), the same memcmp helper ColumnString
+ /// uses for strings, and the column's scale-independent ordering for every other type.
+ static int compare_values(ValueRef lhs, ValueRef rhs) {
+ if constexpr (IS_STRING) {
+ return memcmp_small_allow_overflow15(
+ reinterpret_cast<const uint8_t*>(lhs.data), lhs.size,
+ reinterpret_cast<const uint8_t*>(rhs.data), rhs.size);
+ } else {
+ return Compare::compare(lhs, rhs);
+ }
+ }
+
+ static void insert_value(IColumn& column, ValueRef value) {
+ if constexpr (IS_STRING) {
+ assert_cast<ColumnString&>(column).insert_data(value.data, value.size);
+ } else {
+ assert_cast<ColumnType&>(column).get_data().push_back(value);
+ }
+ }
+
+ /// True when `cmp` puts the left operand before the right one in the aggregate's order.
+ static bool comes_before(int cmp) { return IsMin ? cmp < 0 : cmp > 0; }
+
+ /// True when `cmp`, which compares a value with the worst retained one, puts it after it: the
+ /// value is worse and cannot enter the list.
+ static bool is_worse(int cmp) { return IsMin ? cmp > 0 : cmp < 0; }
+
+ /// splitmix64. The priorities only have to be independent enough to keep the treap balanced;
+ /// a per-state sequence keeps the shape reproducible for the same insertion order.
+ static uint32_t next_priority(State& state) {
+ uint64_t z = (state.rng += 0x9E3779B97F4A7C15ULL);
+ z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9ULL;
+ z = (z ^ (z >> 27)) * 0x94D049BB133111EBULL;
+ return static_cast<uint32_t>(z ^ (z >> 31));
+ }
+
+ static int compare_slots(const State& state, int32_t lhs, int32_t rhs) {
+ return compare_values(state.values.at(lhs), state.values.at(rhs));
+ }
+
+ /// Appends (value, count) as the next slot.
+ static int32_t append_slot(State& state, ValueRef value, int64_t count) {
+ auto slot = static_cast<int32_t>(state.slots.size());
+ DCHECK_LT(slot, std::numeric_limits<int32_t>::max());
+ state.values.append(value);
+ state.slots.push_back(Slot {count, State::NO_SLOT, State::NO_SLOT, next_priority(state)});
+ ++state.live;
+ return slot;
+ }
+
+ static int32_t append_slot(State& state, const IColumn& column, size_t row_num, int64_t count) {
+ return append_slot(state, value_at(column, row_num), count);
+ }
+
+ /// Rotates the subtree rooted at `node` right: the left child becomes the root and `node` its
+ /// right child. The in-order sequence is unchanged, so the treap stays ordered.
+ static int32_t rotate_right(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].left;
+ state.slots[node].left = state.slots[pivot].right;
+ state.slots[pivot].right = node;
+ return pivot;
+ }
+
+ static int32_t rotate_left(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].right;
+ state.slots[node].right = state.slots[pivot].left;
+ state.slots[pivot].left = node;
+ return pivot;
+ }
+
+ /// One descent that performs the whole per-row update: an equal value bumps its count and
+ /// reports no insertion; a missing value is linked as a new slot at the leaf and rotated up
+ /// while the recursion unwinds. `inserted_slot` receives the new slot, NO_SLOT on a hit.
+ static int32_t add_or_increment(State& state, int32_t node, const IColumn& column,
+ size_t row_num, int32_t* inserted_slot) {
+ if (node == State::NO_SLOT) {
+ *inserted_slot = append_slot(state, column, row_num, 1);
+ return *inserted_slot;
+ }
+ int cmp = compare_values(state.values.at(node), value_at(column, row_num));
+ if (cmp == 0) {
+ state.slots[node].count = checked_add(state.slots[node].count, 1);
+ return node;
+ }
+ if (comes_before(cmp)) {
+ // The stored value is the better one, so the input belongs in the right subtree.
+ state.slots[node].right = add_or_increment(state, state.slots[node].right, column,
+ row_num, inserted_slot);
+ if (state.slots[state.slots[node].right].priority > state.slots[node].priority) {
+ return rotate_left(state, node);
+ }
+ } else {
+ state.slots[node].left =
+ add_or_increment(state, state.slots[node].left, column, row_num, inserted_slot);
+ if (state.slots[state.slots[node].left].priority > state.slots[node].priority) {
+ return rotate_right(state, node);
+ }
+ }
+ return node;
+ }
+
+ /// Removes the last entry in the aggregate's order (the largest value for MIN) and returns the
+ /// new subtree root. That entry has no right child, so its left subtree takes its place; the
+ /// child's priority is below the parent's, so no rotation is needed.
+ static int32_t erase_worst(State& state, int32_t node, int32_t* removed) {
+ if (state.slots[node].right == State::NO_SLOT) {
+ *removed = node;
+ return state.slots[node].left;
+ }
+ state.slots[node].right = erase_worst(state, state.slots[node].right, removed);
+ return node;
+ }
+
+ static void refresh_worst(State& state) {
+ int32_t node = state.root;
+ while (node != State::NO_SLOT && state.slots[node].right != State::NO_SLOT) {
+ node = state.slots[node].right;
+ }
+ state.worst = node;
+ }
+
+ /// The first live slot in the aggregate's order, i.e. the best value the state holds.
+ static int32_t best_slot(const State& state) {
+ int32_t node = state.root;
+ while (node != State::NO_SLOT && state.slots[node].left != State::NO_SLOT) {
+ node = state.slots[node].left;
+ }
+ return node;
+ }
+
+ /// Slots in the aggregate's order, i.e. the order of the result array.
+ static void collect_in_order(const State& state, DorisVector<int32_t>& slots) {
+ slots.reserve(static_cast<size_t>(state.live));
+ DorisVector<int32_t> stack;
+ stack.reserve(64);
+ int32_t node = state.root;
+ while (node != State::NO_SLOT || !stack.empty()) {
+ while (node != State::NO_SLOT) {
+ stack.push_back(node);
+ node = state.slots[node].left;
+ }
+ node = stack.back();
+ stack.pop_back();
+ // Only live slots are reachable from the root; an evicted slot has no row count.
+ DCHECK_GE(state.slots[node].count, 0);
+ slots.push_back(node);
+ node = state.slots[node].right;
+ }
+ DCHECK_EQ(static_cast<int64_t>(slots.size()), state.live);
+ }
+
+ /// Rebuilds the tree from `values`, already in the aggregate's order, and the matching counts.
+ /// The treap is built in O(size) as a Cartesian tree over the fresh priorities.
+ static void assign_sorted(State& state, ValueStore values, DorisVector<int64_t> counts) {
+ const size_t size = counts.size();
+ DCHECK_EQ(values.size(), size);
+ state.values.swap(values);
+ state.slots.assign(size, Slot {});
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].count = counts[slot];
+ }
+ state.live = static_cast<int64_t>(size);
+ state.dead = 0;
+ DorisVector<int32_t> stack;
+ stack.reserve(size);
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].priority = next_priority(state);
+ int32_t last = State::NO_SLOT;
+ while (!stack.empty() &&
+ state.slots[stack.back()].priority < state.slots[slot].priority) {
+ last = stack.back();
+ stack.pop_back();
+ }
+ state.slots[slot].left = last;
+ if (!stack.empty()) {
+ state.slots[stack.back()].right = static_cast<int32_t>(slot);
+ }
+ stack.push_back(static_cast<int32_t>(slot));
+ }
+ state.root = stack.empty() ? State::NO_SLOT : stack.front();
+ refresh_worst(state);
+ }
+
+ /// Reclaims the slots evicted since the last compaction. The O(k) rebuild is charged to the
+ /// k/2 evictions that produced the dead slots, so it stays O(1) amortized per row.
+ static void compact(State& state) {
+ DorisVector<int32_t> slots;
+ collect_in_order(state, slots);
+ ValueStore values;
+ values.reserve(slots.size());
+ DorisVector<int64_t> counts;
+ counts.reserve(slots.size());
+ for (int32_t slot : slots) {
+ values.append(state.values.at(slot));
+ counts.push_back(state.slots[slot].count);
+ }
+ assign_sorted(state, std::move(values), std::move(counts));
+ }
+
+ static void check_k(State& state, const IColumn& count_column, size_t row_num) {
+ const IColumn* nested = &count_column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&count_column)) {
+ if (UNLIKELY(nullable->is_null_at(row_num))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-null "
+ "constant");
+ }
+ nested = &nullable->get_nested_column();
+ }
+ int64_t value =
+ assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(*nested).get_element(
+ row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-negative "
+ "constant, but got {}",
+ value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "min_topk/max_topk value count overflow");
+ }
+ return result;
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(State& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (state.live == 0) {
+ state.root = append_slot(state, column, row_num, 1);
+ state.worst = state.root;
+ return;
+ }
+ int cmp = compare_values(value_at(column, row_num), state.values.at(0));
+ if (cmp == 0) {
+ state.slots[0].count = checked_add(state.slots[0].count, 1);
+ } else if (comes_before(cmp)) {
+ // The new value is better: replace the tracked extreme. The slot and the
+ // single-node tree stay in place, only the stored value changes.
+ state.values.clear();
+ state.values.append(value_at(column, row_num));
+ state.slots[0].count = 1;
+ }
+ return;
+ }
+
+ // Dead slots are reclaimed before they can outnumber the live ones, and evicted string
+ // payload before it dwarfs the live one; see compact().
+ if (state.dead > state.live || state.values.dead_bytes() > state.values.live_bytes()) {
+ compact(state);
+ }
+
+ if (state.live >= state.k) {
+ // A full list rejects or absorbs most rows: comparing with the eviction candidate
+ // costs one comparison, while the tree descent would cost log2(k).
+ int cmp = compare_values(value_at(column, row_num), state.values.at(state.worst));
+ if (cmp == 0) {
+ state.slots[state.worst].count = checked_add(state.slots[state.worst].count, 1);
+ return;
+ }
+ if (is_worse(cmp)) {
+ return;
+ }
+ }
+
+ int32_t inserted = State::NO_SLOT;
+ state.root = add_or_increment(state, state.root, column, row_num, &inserted);
+ if (inserted == State::NO_SLOT) {
+ return;
+ }
+ if (state.live <= state.k) {
+ // Not over capacity: the new entry competes with the cached worst for the eviction
+ // candidate role.
+ if (state.worst == State::NO_SLOT ||
+ is_worse(compare_slots(state, inserted, state.worst))) {
+ state.worst = inserted;
+ }
+ return;
+ }
+ // The list was full and the value is new. It beat the cached worst above, so it cannot be
+ // the worst itself; the old worst is still the rightmost entry and leaves the list.
+ int32_t removed = State::NO_SLOT;
+ state.root = erase_worst(state, state.root, &removed);
+ DCHECK_EQ(removed, state.worst);
+ state.values.on_evict(static_cast<size_t>(removed));
+ state.slots[removed].count = State::DEAD_COUNT;
+ --state.live;
+ ++state.dead;
+ refresh_worst(state);
+ }
+
+ /// k == 0: the state holds at most the extreme value and its count.
+ static void merge_extreme(State& state, const State& other) {
+ if (other.live == 0) {
+ return;
+ }
+ if (state.live == 0) {
+ state.root = append_slot(state, other.values.at(0), other.slots[0].count);
+ state.worst = state.root;
+ return;
+ }
+ int cmp = compare_values(state.values.at(0), other.values.at(0));
+ if (cmp == 0) {
+ state.slots[0].count = checked_add(state.slots[0].count, other.slots[0].count);
+ } else if (is_worse(cmp)) {
+ state.values.clear();
+ state.values.append(other.values.at(0));
+ state.slots[0].count = other.slots[0].count;
+ }
+ }
+
+ /// Merges two ordered lists, summing the counts of equal values and keeping the k best.
+ ///
+ /// The merge stops as soon as k entries are produced. Both inputs are best-first and hold
+ /// distinct values, and equal values have already been combined, so every remaining value is
+ /// worse than all retained ones: it can neither enter the result nor overflow it. Stopping
+ /// here also avoids materializing a tail that would be truncated away.
+ static void merge_values(State& state, const State& other) {
+ if (other.live == 0) {
+ // The other partial state saw no non-NULL row.
+ return;
+ }
+ if (state.live >= state.k) {
+ // A full state already holds the k best values. If every value of the other state is
+ // worse than the worst one retained here, the merge cannot change the result, so skip
+ // the collect/rebuild. An equal value would raise a count, hence the strict test.
+ // `_state`/`_merge`/`_union` merges one singleton per row, and this turns each one
+ // into a single comparison once the list is full.
+ int cmp =
+ compare_values(other.values.at(best_slot(other)), state.values.at(state.worst));
+ if (is_worse(cmp)) {
+ return;
+ }
+ }
+ DorisVector<int32_t> lhs_slots;
+ DorisVector<int32_t> rhs_slots;
+ collect_in_order(state, lhs_slots);
+ collect_in_order(other, rhs_slots);
+ const auto limit = static_cast<size_t>(state.k);
+ ValueStore merged_values;
+ merged_values.reserve(std::min(limit, lhs_slots.size() + rhs_slots.size()));
+ DorisVector<int64_t> merged_counts;
+ merged_counts.reserve(std::min(limit, lhs_slots.size() + rhs_slots.size()));
+ size_t lhs_index = 0;
+ size_t rhs_index = 0;
+ while (merged_counts.size() < limit &&
+ (lhs_index < lhs_slots.size() || rhs_index < rhs_slots.size())) {
+ if (rhs_index == rhs_slots.size()) {
+ merged_values.append(state.values.at(lhs_slots[lhs_index]));
+ merged_counts.push_back(state.slots[lhs_slots[lhs_index]].count);
+ ++lhs_index;
+ } else if (lhs_index == lhs_slots.size()) {
+ merged_values.append(other.values.at(rhs_slots[rhs_index]));
+ merged_counts.push_back(other.slots[rhs_slots[rhs_index]].count);
+ ++rhs_index;
+ } else {
+ ValueRef lhs_value = state.values.at(lhs_slots[lhs_index]);
+ ValueRef rhs_value = other.values.at(rhs_slots[rhs_index]);
+ int cmp = compare_values(lhs_value, rhs_value);
+ // Both lists are ordered best-first (ascending for MIN, descending for MAX).
+ const bool lhs_is_better = comes_before(cmp);
+ if (lhs_is_better) {
+ merged_values.append(lhs_value);
+ merged_counts.push_back(state.slots[lhs_slots[lhs_index]].count);
+ ++lhs_index;
+ } else if (cmp != 0) {
+ // Not better and not equal: the right-hand value is better.
+ merged_values.append(rhs_value);
+ merged_counts.push_back(other.slots[rhs_slots[rhs_index]].count);
+ ++rhs_index;
+ } else {
+ merged_values.append(lhs_value);
+ merged_counts.push_back(checked_add(state.slots[lhs_slots[lhs_index]].count,
+ other.slots[rhs_slots[rhs_index]].count));
+ ++lhs_index;
+ ++rhs_index;
+ }
+ }
+ }
+ assign_sorted(state, std::move(merged_values), std::move(merged_counts));
Review Comment:
[P2] Update retained singleton counts without rebuilding the tree
`min_topk_merge(min_topk_state(x,65536))` supplies one serialized singleton per row. After a state holds `0..65535`, another `65535` changes only the cached worst's count, but the strict-worse guard cannot reject it: this path copies all 65536 retained values and `assign_sorted` rebuilds the tree. A tail of 934,464 such rows causes 61,241,032,704 retained-value copies (source-derived, not benchmarked); MAX is symmetric. This differs from the earlier entirely-worse RHS finding, because the result count must change. Route singleton updates through a checked count-update/tree-insertion helper, preserving overflow checks, so a retained value can be updated without collecting and rebuilding all k entries.
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,786 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ // The payload is proportional to the slot count, which the caller already compares.
+ size_t live_bytes() const { return 0; }
+ size_t dead_bytes() const { return 0; }
+ void on_evict(size_t /*slot*/) {}
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ size_t offset = 0;
+ size_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ /// PaddedPODArray, not PODArray: comparisons go through
+ /// memcmp_small_allow_overflow15, which loads a full 16-byte vector even for a one-byte
+ /// string, the same tail allowance ColumnString's chars buffer provides.
+ PaddedPODArray<char, 32> bytes;
+ DorisVector<Span> spans;
+ // Sum of the live slots' payload and of the payload evicted since the last compaction. The
+ // slot counters alone cannot bound the memory: replacing 1024 1 MiB strings with short ones
+ // leaves dead == live by slot count while ~1 GiB of discarded bytes stay in the arena.
+ size_t live_payload_bytes = 0;
+ size_t dead_payload_bytes = 0;
+
+ size_t size() const { return spans.size(); }
+ size_t live_bytes() const { return live_payload_bytes; }
+ size_t dead_bytes() const { return dead_payload_bytes; }
+ void reserve(size_t n) { spans.reserve(n); }
+ void clear() {
+ bytes.clear();
+ spans.clear();
+ live_payload_bytes = 0;
+ dead_payload_bytes = 0;
+ }
+ StringRef at(size_t slot) const {
+ const Span& span = spans[slot];
+ return StringRef(bytes.data() + span.offset, span.length);
+ }
+ void append(StringRef value) {
+ spans.push_back({bytes.size(), value.size});
+ live_payload_bytes += value.size;
+ if (value.size != 0) {
+ bytes.insert(value.data, value.data + value.size);
+ }
+ }
+ /// Called when `slot` is evicted: its payload stays in the arena until the next compaction.
+ void on_evict(size_t slot) {
+ live_payload_bytes -= spans[slot].length;
+ dead_payload_bytes += spans[slot].length;
+ }
+ void swap(StringTopKValueStore& other) noexcept {
+ bytes.swap(other.bytes);
+ spans.swap(other.spans);
+ std::swap(live_payload_bytes, other.live_payload_bytes);
+ std::swap(dead_payload_bytes, other.dead_payload_bytes);
+ }
+};
+
+/**
+ * State shared by min_topk and max_topk, for one value type.
+ *
+ * The distinct values sit in slots (see the stores above), with one Slot record per value carrying
+ * the row count and the links of a treap ordered best-first (ascending values for MIN, descending
+ * for MAX). A row costs one O(log k) descent: it either finds its value and bumps the count or
+ * links a new slot at the leaf. A sorted vector would shift O(k) elements for the same insert.
+ *
+ * The treap stores explicit child links because, unlike a heap, its shape is not the complete tree
+ * that an array index can describe: rotations move entries, and the in-order sequence is the value
+ * order rather than the arrival order. A heap could evict and append in O(log k), but it cannot
+ * find an existing value without a hash index or a linear scan, and it cannot produce the sorted
+ * result without sorting. The ordered tree is the structure that keeps find, insert and eviction
+ * all at O(log k) with comparisons only.
+ *
+ * An entry that is evicted keeps its slot and value until the next compaction. The tree never
+ * points at a dead slot.
+ *
+ * With k == 0 the state holds at most the single extreme value and its row count; the caller
+ * derives truncation from the list length, so the state carries no separate bound.
+ */
+template <PrimitiveType T>
+struct AggregateTopKCountData {
+ using CppType = typename PrimitiveTypeTraits<T>::CppType;
+ static constexpr bool IS_STRING = is_string_type(T);
+ using ValueStore =
+ std::conditional_t<IS_STRING, StringTopKValueStore, FixedTopKValueStore<CppType>>;
+ using ValueRef = std::conditional_t<IS_STRING, StringRef, const CppType&>;
+
+ static constexpr int32_t NO_SLOT = -1;
+ // Row counts are never negative, so this cannot collide with a live entry.
+ static constexpr int64_t DEAD_COUNT = -1;
+
+ /// One slot's metadata; the value itself lives in `values` at the same index.
+ struct Slot {
+ int64_t count = 0;
+ int32_t left = NO_SLOT;
+ int32_t right = NO_SLOT;
+ uint32_t priority = 0;
+ };
+
+ bool has_k = false;
+ int64_t k = 0;
+ // Indexed by slot; an evicted slot stays as a dead entry until compaction.
+ ValueStore values;
+ DorisVector<Slot> slots;
+ int32_t root = NO_SLOT;
+ // The last live slot in the aggregate's order: the value an incoming row has to beat to enter
+ // a full list. Cached so that rejecting a worse value costs one comparison, not a descent.
+ int32_t worst = NO_SLOT;
+ int64_t live = 0;
+ // Slots evicted since the last compaction.
+ int64_t dead = 0;
+ uint64_t rng = 0x243F6A8885A308D3ULL;
+
+ AggregateTopKCountData() = default;
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * min_topk(x, k) / max_topk(x, k).
+ *
+ * Result: ARRAY<STRUCT<value T, count BIGINT>>. NULL inputs are ignored: add validates k for
+ * every row and then skips a NULL x, so the aggregate deliberately does not use the
+ * nullable-argument wrapper. The array is ordered by value (ascending for MIN), holds at most k
+ * distinct values with count > 0, and carries no truncation flag: the caller infers it from the
+ * length (a list of exactly k entries is treated as possibly truncated).
+ *
+ * With k = 0 the array holds at most the extreme value and its row count, which lets the
+ * append-only IVM path compute MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopK final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData<T>,
+ AggregateFunctionTopK<T, IsMin>, false>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using State = AggregateTopKCountData<T>;
+ using Base = IAggregateFunctionDataHelper<State, AggregateFunctionTopK<T, IsMin>, false>;
+ using Slot = typename State::Slot;
+ using ValueStore = typename State::ValueStore;
+ using ValueRef = typename State::ValueRef;
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ static constexpr bool IS_STRING = State::IS_STRING;
+ static constexpr int32_t NO_SLOT = State::NO_SLOT;
+
+ explicit AggregateFunctionTopK(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "min_topk" : "max_topk"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(remove_nullable(this->argument_types[0]));
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is read for every row, including rows where x is NULL: an evaluated count that is
+ // NULL or out of range must fail the query instead of being skipped like a NULL value.
+ check_k(state, *columns[1], row_num);
+ const IColumn* value_column = columns[0];
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(value_column)) {
+ if (nullable->is_null_at(row_num)) {
+ return;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ add_value(state, *value_column, row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw any row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else if (UNLIKELY(state.k != other.k)) {
+ // Ordinary partial/final aggregation always shares the same constant k, but the
+ // public _state/_merge/_union combinators do not encode k in the AggState type, so a
+ // UNION ALL of states built with different literal k can reach this merge.
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} states have incompatible counts: {} vs {}", get_name(), state.k,
+ other.k);
+ }
+ if (state.k == 0) {
+ merge_extreme(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ // The slots are in arrival order and may include dead ones, while the serialized form is
+ // the sorted (value, count) list the reader rebuilds the tree from.
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ auto sorted_values = remove_nullable(this->argument_types[0])->create_column();
+ sorted_values->reserve(order.size());
+ DorisVector<int64_t> sorted_counts;
+ sorted_counts.reserve(order.size());
+ for (int32_t slot : order) {
+ insert_value(*sorted_values, state.values.at(slot));
+ sorted_counts.push_back(state.slots[slot].count);
+ }
+ auto size = static_cast<int64_t>(sorted_counts.size());
+ buf.write_binary(size);
+ write_column(buf, *sorted_values);
+ if (size != 0) {
+ // An empty list keeps an empty vector, whose data() is null; passing it to memcpy is
+ // undefined even for a zero length.
+ buf.write(reinterpret_cast<const char*>(sorted_counts.data()),
+ sorted_counts.size() * sizeof(int64_t));
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ int64_t size = 0;
+ buf.read_binary(size);
+ auto values = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(values->size()), size);
+ DorisVector<int64_t> counts(static_cast<size_t>(size));
+ if (size != 0) {
+ buf.read(reinterpret_cast<char*>(counts.data()), size * sizeof(int64_t));
+ }
+ ValueStore value_store;
+ value_store.reserve(static_cast<size_t>(size));
+ for (int64_t row = 0; row != size; ++row) {
+ value_store.append(value_at(*values, static_cast<size_t>(row)));
+ }
+ assign_sorted(state, std::move(value_store), std::move(counts));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_array = assert_cast<ColumnArray&>(to);
+ // The array element and both fields are nullable in the planned type, but a produced entry
+ // never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(result_array.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = assert_cast<ColumnNullable&>(pair_column.get_column(0));
+ auto& count_column = assert_cast<ColumnNullable&>(pair_column.get_column(1));
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ for (int32_t slot : order) {
+ insert_value(value_column.get_nested_column(), state.values.at(slot));
+ value_column.get_null_map_data().push_back(0);
+ assert_cast<ColumnInt64&>(count_column.get_nested_column())
+ .get_data()
+ .push_back(state.slots[slot].count);
+ count_column.get_null_map_data().push_back(0);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ result_array.get_offsets().push_back(pair_column.size());
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ // Both fields are nullable in the planned type: a consumer reads a field through
+ // element_at, which is always nullable, and named_struct then records nullable fields.
+ // Produced entries never contain a NULL field.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(value_type),
+ make_nullable(std::make_shared<DataTypeInt64>())},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ return std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ }
+
+private:
+ /// The value of the row in `column`. The returned view is valid while the column is alive.
+ static ValueRef value_at(const IColumn& column, size_t row_num) {
+ if constexpr (IS_STRING) {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column).get_data_at(
+ row_num);
+ } else {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column)
+ .get_data()[row_num];
+ }
+ }
+
+ /// Exactly the comparison the value column would apply: CompareFloat for FLOAT/DOUBLE through
+ /// Compare::compare (the column uses the same helper), the same memcmp helper ColumnString
+ /// uses for strings, and the column's scale-independent ordering for every other type.
+ static int compare_values(ValueRef lhs, ValueRef rhs) {
+ if constexpr (IS_STRING) {
+ return memcmp_small_allow_overflow15(
+ reinterpret_cast<const uint8_t*>(lhs.data), lhs.size,
+ reinterpret_cast<const uint8_t*>(rhs.data), rhs.size);
+ } else {
+ return Compare::compare(lhs, rhs);
+ }
+ }
+
+ static void insert_value(IColumn& column, ValueRef value) {
+ if constexpr (IS_STRING) {
+ assert_cast<ColumnString&>(column).insert_data(value.data, value.size);
+ } else {
+ assert_cast<ColumnType&>(column).get_data().push_back(value);
+ }
+ }
+
+ /// True when `cmp` puts the left operand before the right one in the aggregate's order.
+ static bool comes_before(int cmp) { return IsMin ? cmp < 0 : cmp > 0; }
+
+ /// True when `cmp`, which compares a value with the worst retained one, puts it after it: the
+ /// value is worse and cannot enter the list.
+ static bool is_worse(int cmp) { return IsMin ? cmp > 0 : cmp < 0; }
+
+ /// splitmix64. The priorities only have to be independent enough to keep the treap balanced;
+ /// a per-state sequence keeps the shape reproducible for the same insertion order.
+ static uint32_t next_priority(State& state) {
Review Comment:
[P2] Keep treap priorities independent of the input sequence
Every state starts with the same RNG seed. Feed each successive `next_priority()` output itself as the next BIGINT value to `min_topk(x,65536)`: the first 48,790 outputs are distinct, starting `1254007329,588548489,1620966915,3736152064,1816407222`. Each key then equals its heap priority, so the BST/max-heap invariants force a single left chain. These 48,790 rows require 595,442,478 node comparisons and produce a tree of depth 48,790 (source-derived counts, not a runtime benchmark); `add_or_increment` traverses that tree recursively. The state stays below k, so compaction never intervenes; MAX has the symmetric case. Use independently randomized per-state priorities or enforce a balance/depth bound, and cover priority-correlated inputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6469962264
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4348575882
##########
be/src/exprs/function/function_topk_merge.cpp:
##########
@@ -0,0 +1,459 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <glog/logging.h>
+
+#include <cstddef>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_numbers.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/types.h"
+#include "exprs/function/function.h"
+#include "exprs/function/simple_function_factory.h"
+
+namespace doris {
+class FunctionContext;
+} // namespace doris
+
+namespace doris {
+
+namespace {
+
+/// Upper bound of k. Keep in sync with the aggregate's MAX_TOPK_K
+/// (aggregate_function_topk_count.h) and FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/// One row of an ARRAY<STRUCT<value, count>> argument. A NULL array is an empty view.
+struct TopKArrayView {
+ const IColumn* values = nullptr;
+ const ColumnInt64* counts = nullptr;
+ /// NULL bitmap of the array element (the pair struct).
+ const NullMap* null_map = nullptr;
+ /// NULL bitmap of the `value` field; NULL values are not part of the contract.
+ const NullMap* value_null_map = nullptr;
+ /// NULL bitmap of the `count` field; NULL counts are not part of the contract.
+ const NullMap* count_null_map = nullptr;
+ size_t begin = 0;
+ size_t end = 0;
+};
+
+TopKArrayView make_array_view(const IColumn& column, size_t row) {
+ TopKArrayView view;
+ const IColumn* array_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ array_column = &nullable->get_nested_column();
+ }
+ const auto& array = assert_cast<const ColumnArray&>(*array_column);
+ // offsets[i] is the cumulative size after row i, so row's range is [offsets[row-1], offsets[row]).
+ // offset_at(0) reads offsets[-1], which PaddedPODArray keeps as 0.
+ view.begin = array.offset_at(static_cast<ssize_t>(row));
+ view.end = array.offset_at(static_cast<ssize_t>(row) + 1);
+
+ const IColumn* pair_column = &array.get_data();
+ if (const auto* element_nullable = check_and_get_column<ColumnNullable>(pair_column)) {
+ view.null_map = &element_nullable->get_null_map_data();
+ pair_column = &element_nullable->get_nested_column();
+ }
+ const auto& pairs = assert_cast<const ColumnStruct&>(*pair_column);
+ view.values = &pairs.get_column(0);
+ // A cast may widen the `value` field to a nullable column; the entries themselves are never
+ // NULL, so only the nested column is used.
+ if (const auto* value_nullable = check_and_get_column<ColumnNullable>(view.values)) {
+ view.value_null_map = &value_nullable->get_null_map_data();
+ view.values = &value_nullable->get_nested_column();
+ }
+ // The same holds for `count`: a nested cast may have made the field nullable.
+ const auto* count_column = &pairs.get_column(1);
+ if (const auto* count_nullable = check_and_get_column<ColumnNullable>(count_column)) {
+ view.count_null_map = &count_nullable->get_null_map_data();
+ count_column = &count_nullable->get_nested_column();
+ }
+ view.counts = &assert_cast<const ColumnInt64&>(*count_column);
+ return view;
+}
+
+/// One scalar bound argument: NULL means the corresponding side was not truncated.
+struct TopKBoundView {
+ const IColumn* value = nullptr;
+ bool is_null = true;
+};
+
+TopKBoundView make_bound_view(const IColumn& column, size_t row) {
+ TopKBoundView view;
+ const IColumn* value_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ view.value = value_column;
Review Comment:
Fixed in 56a622307d2. `TopKBoundView` now carries the row index, and the effective-bound selection, the trusted-range comparison and the bound output all read that row instead of index 0. `order_qt_merge_multi_row_bounds` covers two rows in the same block with different bounds (2 and 20).
##########
be/src/exprs/aggregate/aggregate_function_topk_count.h:
##########
@@ -0,0 +1,428 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/string_buffer.hpp"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// Upper bound of k, shared with FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/**
+ * State shared by topk_min_count and topk_max_count.
+ *
+ * The entries are held as two parallel structures: a column with the distinct values (sorted
+ * ascending for MIN / descending for MAX, at most k entries) and their non-negative int64 counts.
+ * Only values with count > 0 are kept. `bound` is a one-row column with the extreme value seen
+ * when k == 0, or the k-th value once the list was truncated when k >= 1; `has_bound` is the
+ * truncation flag, and for k >= 1 `bound IS NULL` means "the list holds every distinct value".
+ */
+struct AggregateTopKCountData {
+ const DataTypePtr value_type;
+ bool has_k = false;
+ int64_t k = 0;
+ bool has_bound = false;
+ MutableColumnPtr bound_column;
+ MutableColumnPtr value_column;
+ std::vector<int64_t> counts;
+
+ explicit AggregateTopKCountData(const DataTypes& argument_types_)
+ : value_type(argument_types_[0]),
+ bound_column(value_type->create_column()),
+ value_column(value_type->create_column()) {}
+
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * topk_min_count(x, k) / topk_max_count(x, k).
+ *
+ * Result: STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T NULL>. NULL inputs are
+ * ignored (the nullable-argument wrapper never calls add for them). `values` is ordered by value
+ * (ascending for MIN), holds at most k distinct values, and `bound` is the truncation point: the
+ * k-th value when more than k distinct values were seen, otherwise NULL.
+ *
+ * With k = 0 no list is built: `values` is empty and `bound` is the extreme value seen on this
+ * side (NULL when there was no non-NULL input), which lets the append-only IVM path compute
+ * MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopKCount final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopKCount<T, IsMin>, true>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using Base = IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopKCount<T, IsMin>, true>;
+
+ explicit AggregateFunctionTopKCount(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "topk_min_count" : "topk_max_count"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(this->argument_types[0]);
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is a constant expression, but only the first row carries it: the state remembers it so
+ // that merge can check both sides used the same capacity. A state without has_k is empty.
+ check_k(state, *columns[1], row_num);
+ add_value(state, *columns[0], row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw a non-NULL row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else {
+ // Both sides are partial states of the same constant k.
+ DCHECK_EQ(state.k, other.k);
Review Comment:
Fixed in 56a622307d2. `merge()` now rejects two partial states with different `k` with `INVALID_ARGUMENT` instead of assuming they match, and `TopKCountTest.MergeRejectsIncompatibleK` merges states built with k=2 and k=1.
##########
be/src/exprs/aggregate/aggregate_function_topk_count.h:
##########
@@ -0,0 +1,428 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/string_buffer.hpp"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// Upper bound of k, shared with FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/**
+ * State shared by topk_min_count and topk_max_count.
+ *
+ * The entries are held as two parallel structures: a column with the distinct values (sorted
+ * ascending for MIN / descending for MAX, at most k entries) and their non-negative int64 counts.
+ * Only values with count > 0 are kept. `bound` is a one-row column with the extreme value seen
+ * when k == 0, or the k-th value once the list was truncated when k >= 1; `has_bound` is the
+ * truncation flag, and for k >= 1 `bound IS NULL` means "the list holds every distinct value".
+ */
+struct AggregateTopKCountData {
+ const DataTypePtr value_type;
+ bool has_k = false;
+ int64_t k = 0;
+ bool has_bound = false;
+ MutableColumnPtr bound_column;
+ MutableColumnPtr value_column;
+ std::vector<int64_t> counts;
+
+ explicit AggregateTopKCountData(const DataTypes& argument_types_)
+ : value_type(argument_types_[0]),
+ bound_column(value_type->create_column()),
+ value_column(value_type->create_column()) {}
+
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * topk_min_count(x, k) / topk_max_count(x, k).
+ *
+ * Result: STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T NULL>. NULL inputs are
+ * ignored (the nullable-argument wrapper never calls add for them). `values` is ordered by value
+ * (ascending for MIN), holds at most k distinct values, and `bound` is the truncation point: the
+ * k-th value when more than k distinct values were seen, otherwise NULL.
+ *
+ * With k = 0 no list is built: `values` is empty and `bound` is the extreme value seen on this
+ * side (NULL when there was no non-NULL input), which lets the append-only IVM path compute
+ * MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopKCount final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopKCount<T, IsMin>, true>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using Base = IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopKCount<T, IsMin>, true>;
+
+ explicit AggregateFunctionTopKCount(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "topk_min_count" : "topk_max_count"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(this->argument_types[0]);
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is a constant expression, but only the first row carries it: the state remembers it so
+ // that merge can check both sides used the same capacity. A state without has_k is empty.
+ check_k(state, *columns[1], row_num);
+ add_value(state, *columns[0], row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw a non-NULL row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else {
+ // Both sides are partial states of the same constant k.
+ DCHECK_EQ(state.k, other.k);
+ }
+ if (state.k == 0) {
+ merge_bound(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ buf.write_binary(state.has_bound);
+ if (state.has_bound) {
+ write_column(buf, *state.bound_column);
+ }
+ auto size = static_cast<int64_t>(state.counts.size());
+ buf.write_binary(size);
+ write_column(buf, *state.value_column);
+ buf.write(reinterpret_cast<const char*>(state.counts.data()),
+ state.counts.size() * sizeof(int64_t));
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ buf.read_binary(state.has_bound);
+ if (state.has_bound) {
+ state.bound_column = read_column(buf);
+ } else {
+ state.bound_column = state.value_type->create_column();
+ }
+ int64_t size = 0;
+ buf.read_binary(size);
+ state.value_column = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(state.value_column->size()), size);
+ state.counts.resize(size);
+ buf.read(reinterpret_cast<char*>(state.counts.data()), size * sizeof(int64_t));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_struct = assert_cast<ColumnStruct&>(to);
+ auto& values_column = assert_cast<ColumnArray&>(result_struct.get_column(0));
+ // The array element is nullable in the planned type (ARRAY<STRUCT<...>> defaults to
+ // contains_null = true), but a produced entry never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(values_column.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = pair_column.get_column(0);
+ auto& count_column = assert_cast<ColumnInt64&>(pair_column.get_column(1));
+ for (size_t i = 0; i != state.counts.size(); ++i) {
+ value_column.insert_from(*state.value_column, i);
+ count_column.get_data().push_back(state.counts[i]);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ values_column.get_offsets().push_back(pair_column.size());
+
+ auto& bound_column = assert_cast<ColumnNullable&>(result_struct.get_column(1));
+ if (state.has_bound) {
+ bound_column.get_nested_column().insert_from(*state.bound_column, 0);
+ bound_column.get_null_map_data().push_back(0);
+ } else {
+ bound_column.get_nested_column().insert_default();
+ bound_column.get_null_map_data().push_back(1);
+ }
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {value_type, std::make_shared<DataTypeInt64>()},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ auto values_type = std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ return std::make_shared<DataTypeStruct>(DataTypes {values_type, make_nullable(value_type)},
+ Strings {"values", "bound"});
+ }
+
+private:
+ static void check_k(AggregateTopKCountData& state, const IColumn& count_column,
+ size_t row_num) {
+ DCHECK(!count_column.is_nullable());
+ int64_t value = assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(count_column)
+ .get_element(row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0 || value > MAX_TOPK_K)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of topk_min_count and topk_max_count must be a constant "
+ "between 0 and {}, but got {}",
+ MAX_TOPK_K, value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ /// Ordering of the values. FLOAT/DOUBLE use CompareFloat so that NaN sorts greatest and is
+ /// distinct from every number, which keeps the list mergeable and the top-K deterministic;
+ /// every other type uses the column's compare_at, i.e. exactly the MIN/MAX comparison.
+ static int compare_at(const IColumn& lhs, size_t lhs_row, const IColumn& rhs, size_t rhs_row) {
+ if constexpr (is_float_or_double(T)) {
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ const auto& left =
+ assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(lhs).get_data();
+ const auto& right =
+ assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(rhs).get_data();
+ return CompareFloat(left[lhs_row], right[rhs_row]);
+ } else {
+ return lhs.compare_at(lhs_row, rhs_row, rhs, 1);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "topk_min_count/topk_max_count value count overflow");
+ }
+ return result;
+ }
+
+ static void set_bound_from_value(AggregateTopKCountData& state, size_t index) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(*state.value_column, index);
+ state.has_bound = true;
+ }
+
+ /// Inserts (value, count) at `pos`, keeping the value column sorted by value.
+ static void insert_entry(AggregateTopKCountData& state, size_t pos, const IColumn& column,
+ size_t row_num, int64_t count) {
+ auto new_values = state.value_column->clone_empty();
+ new_values->reserve(state.value_column->size() + 1);
+ for (size_t i = 0; i < pos; ++i) {
+ new_values->insert_from(*state.value_column, i);
+ }
+ new_values->insert_from(column, row_num);
+ for (size_t i = pos; i < state.value_column->size(); ++i) {
+ new_values->insert_from(*state.value_column, i);
+ }
+ state.value_column = std::move(new_values);
+ state.counts.insert(state.counts.begin() + pos, count);
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(AggregateTopKCountData& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (!state.has_bound) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(column, row_num);
+ state.has_bound = true;
+ return;
+ }
+ int cmp = compare_at(column, row_num, *state.bound_column, 0);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(column, row_num);
+ }
+ return;
+ }
+
+ // The list is ascending for MIN and descending for MAX, so the insertion point is where
+ // the stored values stop being better than the new one.
+ size_t left = 0;
+ size_t right = state.value_column->size();
+ while (left < right) {
+ size_t mid = left + (right - left) / 2;
+ int cmp = compare_at(*state.value_column, mid, column, row_num);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ left = mid + 1;
+ } else {
+ right = mid;
+ }
+ }
+ if (left < state.value_column->size() &&
+ compare_at(*state.value_column, left, column, row_num) == 0) {
+ state.counts[left] = checked_add(state.counts[left], 1);
+ return;
+ }
+
+ insert_entry(state, left, column, row_num, 1);
+ if (static_cast<int64_t>(state.value_column->size()) > state.k) {
+ // More than k distinct values: drop the worst one. The k-th (last kept) value is the
+ // new truncation point, even when the value just inserted was the dropped one.
+ state.value_column->pop_back(1);
+ state.counts.pop_back();
+ set_bound_from_value(state, state.value_column->size() - 1);
+ }
+ }
+
+ static void merge_bound(AggregateTopKCountData& state, const AggregateTopKCountData& other) {
+ if (!other.has_bound) {
+ return;
+ }
+ if (!state.has_bound) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(*other.bound_column, 0);
+ state.has_bound = true;
+ return;
+ }
+ int cmp = compare_at(*other.bound_column, 0, *state.bound_column, 0);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(*other.bound_column, 0);
+ }
+ }
+
+ /// Merges two sorted lists, summing the counts of equal values and truncating to k.
+ static void merge_values(AggregateTopKCountData& state, const AggregateTopKCountData& other) {
+ if (other.counts.empty()) {
+ // The other partial state saw no non-NULL row, so it has neither entries nor a bound.
+ DCHECK(!other.has_bound);
+ return;
+ }
+ auto merged = state.value_column->clone_empty();
+ std::vector<int64_t> merged_counts;
+ merged_counts.reserve(state.counts.size() + other.counts.size());
+ size_t lhs_index = 0;
+ size_t rhs_index = 0;
+ while (lhs_index < state.counts.size() || rhs_index < other.counts.size()) {
+ if (rhs_index == other.counts.size()) {
+ merged->insert_from(*state.value_column, lhs_index);
+ merged_counts.push_back(state.counts[lhs_index]);
+ ++lhs_index;
+ } else if (lhs_index == state.counts.size()) {
+ merged->insert_from(*other.value_column, rhs_index);
+ merged_counts.push_back(other.counts[rhs_index]);
+ ++rhs_index;
+ } else {
+ int cmp =
+ compare_at(*state.value_column, lhs_index, *other.value_column, rhs_index);
+ // Both lists are ordered best-first (ascending for MIN, descending for MAX).
+ const bool lhs_is_better = IsMin ? cmp < 0 : cmp > 0;
+ if (lhs_is_better) {
+ merged->insert_from(*state.value_column, lhs_index);
+ merged_counts.push_back(state.counts[lhs_index]);
+ ++lhs_index;
+ } else if (cmp != 0) {
+ // Not better and not equal: the right-hand value is better.
+ merged->insert_from(*other.value_column, rhs_index);
+ merged_counts.push_back(other.counts[rhs_index]);
+ ++rhs_index;
+ } else {
+ merged->insert_from(*state.value_column, lhs_index);
+ merged_counts.push_back(
+ checked_add(state.counts[lhs_index], other.counts[rhs_index]));
+ ++lhs_index;
+ ++rhs_index;
+ }
+ }
+ }
+
+ // Once a side is truncated, values outside its bound were dropped, so the union cannot be
+ // complete either: the k-th merged value becomes the new truncation point.
+ bool truncated = state.has_bound || other.has_bound;
+ auto merged_size = static_cast<int64_t>(merged->size());
+ if (merged_size > state.k) {
+ merged->pop_back(merged_size - state.k);
+ merged_counts.resize(state.k);
+ truncated = true;
+ }
+ state.value_column = std::move(merged);
+ state.counts = std::move(merged_counts);
+ state.has_bound = truncated;
+ if (truncated) {
+ set_bound_from_value(state, state.value_column->size() - 1);
+ } else {
+ state.bound_column->clear();
+ }
+ }
+
+ void write_column(BufferWritable& buf, const IColumn& column) const {
+ int64_t size_bytes = this->argument_types[0]->get_uncompressed_serialized_bytes(
+ column, IAggregateFunction::version);
+ buf.write_binary(size_bytes);
+ buf.resize(size_bytes);
+ auto* pos =
+ this->argument_types[0]->serialize(column, buf.data(), IAggregateFunction::version);
+ DCHECK_EQ(pos, buf.data() + size_bytes);
Review Comment:
Fixed in 56a622307d2. `write_column` declares `get_uncompressed_serialized_bytes` as the capacity and pads the buffer when `serialize` returns an earlier pointer; `read_column` skips the same declared capacity, so a compressed payload is not misread. `TopKCountTest.SerializeDeserializeRoundTrip` now round-trips 65 distinct INTs (260 bytes, above the compression threshold) with k=65.
##########
be/src/exprs/aggregate/aggregate_function_topk_count.h:
##########
@@ -0,0 +1,428 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/string_buffer.hpp"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// Upper bound of k, shared with FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/**
+ * State shared by topk_min_count and topk_max_count.
+ *
+ * The entries are held as two parallel structures: a column with the distinct values (sorted
+ * ascending for MIN / descending for MAX, at most k entries) and their non-negative int64 counts.
+ * Only values with count > 0 are kept. `bound` is a one-row column with the extreme value seen
+ * when k == 0, or the k-th value once the list was truncated when k >= 1; `has_bound` is the
+ * truncation flag, and for k >= 1 `bound IS NULL` means "the list holds every distinct value".
+ */
+struct AggregateTopKCountData {
+ const DataTypePtr value_type;
+ bool has_k = false;
+ int64_t k = 0;
+ bool has_bound = false;
+ MutableColumnPtr bound_column;
+ MutableColumnPtr value_column;
+ std::vector<int64_t> counts;
Review Comment:
Fixed in 56a622307d2. Both the per-group `counts` and the `merged_counts` scratch buffer are now `DorisVector<int64_t>`, so the 512 KiB-per-group payload at k=65536 is charged to the query memory tracker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4352413314
##########
be/src/exprs/lambda_function/varray_sorted_merge_function.cpp:
##########
@@ -0,0 +1,323 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+#include <map>
+#include <memory>
+#include <set>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "exprs/lambda_function/lambda_execution_context.h"
+#include "exprs/lambda_function/lambda_function.h"
+#include "exprs/lambda_function/lambda_function_factory.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vlambda_function_expr.h"
+
+namespace doris {
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc)
+ *
+ * Merges two arrays that are already sorted by the elements' natural order. The merge lambda
+ * receives the two current candidates and returns the combined element; returning NULL means the
+ * two elements must not be combined, so both are kept. Each input key occurs at most once per
+ * array, so a key appears at most twice across the two arrays and the lambda is called at most
+ * once per key.
+ */
+class ArraySortedMergeFunction : public LambdaFunction {
+ ENABLE_FACTORY_CREATOR(ArraySortedMergeFunction);
+
+public:
+ ~ArraySortedMergeFunction() override = default;
+
+ static constexpr auto name = "array_sorted_merge";
+
+ static LambdaFunctionPtr create() { return std::make_shared<ArraySortedMergeFunction>(); }
+
+ std::string get_name() const override { return name; }
+
+ /// Rejects a nested lambda that captures one of this lambda's arguments: the position-based
+ /// frame below does not publish them into nested lambdas, so such a capture would resolve
+ /// against the wrong column. Nested lambdas may still use their own arguments (shadowing).
+ static Status validate_nested_captures(const VExprSPtr& expr,
+ const std::vector<std::string>& argument_names,
+ bool inside_nested_lambda,
+ std::vector<std::string> shadowed_names) {
+ if (expr->is_column_ref()) {
+ const std::string& column_name = expr->expr_name();
+ if (std::ranges::find(shadowed_names, column_name) != shadowed_names.end()) {
+ return Status::OK();
+ }
+ if (inside_nested_lambda &&
+ std::ranges::find(argument_names, column_name) != argument_names.end()) {
+ return Status::NotSupported(
+ "array_sorted_merge merge lambda argument '{}' cannot be captured by a "
+ "nested lambda",
+ column_name);
+ }
+ return Status::OK();
+ }
+ if (expr->node_type() == TExprNodeType::LAMBDA_FUNCTION_EXPR) {
+ const auto* lambda = assert_cast<const VLambdaFunctionExpr*>(expr.get());
+ if (lambda->has_argument_names()) {
+ for (const auto& argument_name : lambda->argument_names()) {
+ shadowed_names.emplace_back(argument_name);
+ }
+ }
+ return validate_nested_captures(lambda->get_child(0), argument_names, true,
+ std::move(shadowed_names));
+ }
+ for (const auto& child : expr->children()) {
+ RETURN_IF_ERROR(validate_nested_captures(child, argument_names, inside_nested_lambda,
+ shadowed_names));
+ }
+ return Status::OK();
+ }
+
+ Status prepare(RuntimeState* state, const VExprSPtrs& children) override {
+ RETURN_IF_ERROR(LambdaFunction::prepare(state, children));
+ DCHECK_EQ(children.size(), 4);
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ if (!lambda_expr->has_argument_names()) {
+ return Status::OK();
+ }
+ // A nested lambda that captures one of this lambda's arguments would need a binding
+ // that does not exist in the position-based frame below; reject it during prepare so
+ // the query fails instead of resolving the argument against the wrong column.
+ return validate_nested_captures(children[0]->get_child(0), lambda_expr->argument_names(),
+ false, {});
+ }
+
+ Status execute(VExprContext* context, const Block* block, const Selector* expr_selector,
+ size_t count, ColumnPtr& result_column, const DataTypePtr& result_type,
+ const VExprSPtrs& children) const override {
+ DCHECK_EQ(children.size(), 4);
+
+ const ColumnArray* arrays[2] = {nullptr, nullptr};
+ const IColumn* data[2] = {nullptr, nullptr};
+ DataTypePtr nested_types[2];
+ ColumnPtr holders[2];
+ const ColumnUInt8::Container* nullmaps[2] = {nullptr, nullptr};
+ for (int i = 0; i < 2; ++i) {
+ RETURN_IF_ERROR(children[i + 1]->execute_column(context, block, expr_selector, count,
+ holders[i]));
+ holders[i] = holders[i]->convert_to_full_column_if_const();
+ DataTypePtr type = children[i + 1]->execute_type(block);
+ const IColumn* array_column = holders[i].get();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(array_column)) {
+ nullmaps[i] = &nullable->get_null_map_column().get_data();
+ array_column = &nullable->get_nested_column();
+ type = assert_cast<const DataTypeNullable&>(*type).get_nested_type();
+ }
+ arrays[i] = assert_cast<const ColumnArray*>(array_column);
+ data[i] = arrays[i]->get_data_ptr().get();
+ nested_types[i] = assert_cast<const DataTypeArray&>(*type).get_nested_type();
+ }
+
+ ColumnPtr flag_holder;
+ RETURN_IF_ERROR(
+ children[3]->execute_column(context, block, expr_selector, count, flag_holder));
+ flag_holder = flag_holder->convert_to_full_column_if_const();
+ const IColumn* flag_column = flag_holder.get();
+ const ColumnUInt8::Container* flag_nullmap = nullptr;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(flag_column)) {
+ flag_nullmap = &nullable->get_null_map_column().get_data();
+ flag_column = &nullable->get_nested_column();
+ }
+ const auto& flags = assert_cast<const ColumnUInt8&>(*flag_column).get_data();
+
+ const size_t rows = arrays[0]->size();
+ auto result_nested = nested_types[0]->create_column();
+ auto result_offsets = ColumnArray::ColumnOffsets::create();
+ auto& offsets_data = result_offsets->get_data();
+ offsets_data.reserve(rows);
+ auto result_nullmap = ColumnUInt8::create(rows, 0);
+ auto& result_nullmap_data = result_nullmap->get_data();
+
+ // The lambda may capture enclosing columns (for example a table column compared against
+ // the candidates). Materialize every captured input at its original position in the
+ // one-row lambda block, so slot refs read the current row; the two merge candidates
+ // take the positions after the last captured column and are bound by name.
+ std::set<int> captured_ids;
+ children[0]->collect_slot_column_ids(captured_ids);
+ if (!captured_ids.empty() && block == nullptr) {
+ return Status::InternalError(
+ "array_sorted_merge lambda references outer columns but has no input block");
+ }
+ std::map<int, ColumnPtr> captured_columns;
+ for (int id : captured_ids) {
+ if (id < 0 || static_cast<size_t>(id) >= block->columns()) {
+ return Status::InternalError(
+ "array_sorted_merge lambda input column id {} is outside input block", id);
+ }
+ captured_columns[id] =
+ block->get_by_position(id).column->convert_to_full_column_if_const();
+ }
+ const int lambda_argument_base = captured_ids.empty() ? 0 : *captured_ids.rbegin() + 1;
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ const std::vector<std::string>* argument_names =
+ lambda_expr->has_argument_names() ? &lambda_expr->argument_names() : nullptr;
+ Block lambda_block;
+ for (int i = 0; i < lambda_argument_base; ++i) {
+ auto captured = captured_columns.find(i);
+ if (captured != captured_columns.end()) {
+ lambda_block.insert(ColumnWithTypeAndName(captured->second->clone_empty(),
+ block->get_by_position(i).type,
+ block->get_by_position(i).name));
+ } else {
+ lambda_block.insert(ColumnWithTypeAndName(
+ ColumnUInt8::create(1, 0), std::make_shared<DataTypeUInt8>(), "temp"));
+ }
+ }
+ for (int i = 0; i < 2; ++i) {
+ const std::string argument_name =
+ argument_names != nullptr && i < argument_names->size()
+ ? (*argument_names)[i]
+ : "merge_arg_" + std::to_string(i);
+ lambda_block.insert(ColumnWithTypeAndName(nested_types[i]->create_column(),
+ nested_types[i], argument_name));
+ }
+ // The lambda arguments are bound by name (the column refs carry the argument names),
+ // so the body reads these two one-row columns.
+ IColumn* argument_columns[2] = {
+ lambda_block.get_by_position(lambda_argument_base).column->assert_mutable().get(),
+ lambda_block.get_by_position(lambda_argument_base + 1)
+ .column->assert_mutable()
+ .get()};
+
+ LambdaExecutionContext::Frame lambda_frame;
+ lambda_frame.bind_by_name = false;
+ lambda_frame.parent_bindings_visible = false;
+ LambdaExecutionContext::FrameGuard lambda_frame_guard(context->lambda_execution_context(),
+ std::move(lambda_frame));
+
+ const auto& offsets0 = arrays[0]->get_offsets();
+ const auto& offsets1 = arrays[1]->get_offsets();
+ const int lambda_result_base = static_cast<int>(lambda_block.columns());
+
+ // The merged element may come from a non-nullable lambda body (for example COALESCE),
+ // while array elements are always nullable; normalize before inserting.
+ auto append_element = [&](const IColumn& source, size_t source_row) {
+ if (auto* nullable_target = check_and_get_column<ColumnNullable>(result_nested.get())) {
+ if (const auto* nullable_source = check_and_get_column<ColumnNullable>(&source)) {
+ nullable_target->insert_from(*nullable_source, source_row);
+ } else {
+ nullable_target->get_nested_column().insert_from(source, source_row);
+ nullable_target->get_null_map_data().push_back(0);
+ }
+ } else {
+ result_nested->insert_from(source, source_row);
+ }
+ };
+
+ for (size_t row = 0; row < rows; ++row) {
+ const bool row_is_null = (nullmaps[0] != nullptr && (*nullmaps[0])[row] != 0) ||
+ (nullmaps[1] != nullptr && (*nullmaps[1])[row] != 0);
+ if (row_is_null) {
+ result_nullmap_data[row] = 1;
+ offsets_data.push_back(result_nested->size());
+ continue;
+ }
+ if (flag_nullmap != nullptr && (*flag_nullmap)[row] != 0) {
+ return Status::InvalidArgument(
+ "array_sorted_merge requires a non-null ascending flag");
+ }
+ const bool ascending = flags[row] != 0;
+ for (const auto& [id, column] : captured_columns) {
+ lambda_block.get_by_position(id).column = column->cut(row, 1);
Review Comment:
[P2] Apply the execution selector when copying captured values
With `short_circuit_evaluation=true`, consider `Project(IF(id=1, array_sorted_merge((a,b)->IF(a=b AND a<lim,a,NULL), l,r,true), CAST([] AS ARRAY<INT>))) -> Scan(id,lim,l,r)` and one block containing `(0,0,[1],[1])` followed by `(1,2,[1],[1])`. `ShortCircuitIfExpr` passes the original block and selector `[1]` to this branch. The arrays correctly read selected row 1, but this cut uses compacted `row=0` on the original captured `lim` column, so row 1 returns `[1,1]` instead of `[1]`. Apply `expr_selector` to captures too, or cut the original selected row index. This is a distinct defect in the newly added row-copy code: the earlier capture thread concerned missing bindings/incorrect column positions, while ordinary table captures now work and only nonidentity row selection triggers this wrong result. Add a two-row conditional-evaluation regression with different captured limits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6349567542
<details>
<summary>TPC-H: <b>Total hot run time: 28686 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17615 5176 5061 5061
q2 2140 329 264 264
q3 10191 1382 831 831
q4 4682 457 341 341
q5 7428 795 556 556
q6 186 182 152 152
q7 772 792 598 598
q8 9258 1353 1058 1058
q9 5460 4207 4175 4175
q10 6828 1284 1013 1013
q11 443 250 223 223
q12 632 403 277 277
q13 18086 3631 3006 3006
q14 274 268 243 243
q15 q16 722 719 673 673
q17 1246 758 568 568
q18 6942 6186 6107 6107
q19 1110 958 585 585
q20 381 347 226 226
q21 5372 2484 2453 2453
q22 383 348 276 276
Total cold run time: 100151 ms
Total hot run time: 28686 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5965 5874 6096 5874
q2 722 624 594 594
q3 5614 5833 5247 5247
q4 2084 2173 1610 1610
q5 5701 5869 5568 5568
q6 252 190 139 139
q7 2240 2030 1832 1832
q8 3006 2617 2661 2617
q9 7918 7932 7650 7650
q10 3635 3584 3144 3144
q11 595 406 389 389
q12 663 658 495 495
q13 3280 3544 3031 3031
q14 300 283 266 266
q15 q16 660 685 645 645
q17 7770 7062 6869 6869
q18 12984 12193 12806 12193
q19 916 791 773 773
q20 2163 2154 1973 1973
q21 5576 4587 4703 4587
q22 498 456 396 396
Total cold run time: 72542 ms
Total hot run time: 65892 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6363992203
<details>
<summary>TPC-H: <b>Total hot run time: 28331 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e250347cd120427a408a4db7a0a55e83da7b098f, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17615 5112 5054 5054
q2 2118 353 261 261
q3 10195 1398 819 819
q4 4680 465 342 342
q5 7439 796 531 531
q6 182 183 151 151
q7 738 803 595 595
q8 9267 1317 1025 1025
q9 5559 4197 4167 4167
q10 6834 1294 1052 1052
q11 421 246 230 230
q12 638 403 273 273
q13 18117 3612 3038 3038
q14 265 271 248 248
q15 q16 735 720 637 637
q17 1212 749 575 575
q18 7062 6190 6126 6126
q19 1114 973 589 589
q20 388 336 230 230
q21 5032 2135 2260 2135
q22 350 295 253 253
Total cold run time: 99961 ms
Total hot run time: 28331 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5652 5621 5615 5615
q2 675 563 532 532
q3 5214 5380 5005 5005
q4 1980 2144 1525 1525
q5 5310 5192 5641 5192
q6 266 207 158 158
q7 2345 2138 1860 1860
q8 3084 2837 2758 2758
q9 8176 8128 8120 8120
q10 3706 3696 3266 3266
q11 615 446 403 403
q12 681 728 525 525
q13 3561 3899 3224 3224
q14 304 316 285 285
q15 q16 701 690 647 647
q17 8310 7524 7423 7423
q18 12975 12101 12945 12101
q19 899 765 781 765
q20 2156 2132 1937 1937
q21 5590 4625 4713 4625
q22 516 432 421 421
Total cold run time: 72716 ms
Total hot run time: 66387 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6367490198
# BE Regression && UT Coverage Report
Increment line coverage `81.76% (269/329)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 68.35% (31841/46585) |
| Line Coverage | 53.00% (344694/650355) |
| Region Coverage | 48.35% (281610/582490) |
| Branch Coverage | 49.84% (132108/265083) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6367717472
# BE Regression && UT Coverage Report
Increment line coverage `81.76% (269/329)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 68.40% (31863/46585) |
| Line Coverage | 53.06% (345066/650355) |
| Region Coverage | 48.44% (282148/582490) |
| Branch Coverage | 49.89% (132244/265083) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6380784816
# FE Regression Coverage Report
Increment line coverage ` 81.29% (139/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6420350385
# FE Regression Coverage Report
Increment line coverage ` 9.75% (139/1425)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4350669601
##########
be/src/exprs/lambda_function/varray_sorted_merge_function.cpp:
##########
@@ -0,0 +1,215 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <string>
+#include <utility>
+
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "exprs/lambda_function/lambda_execution_context.h"
+#include "exprs/lambda_function/lambda_function.h"
+#include "exprs/lambda_function/lambda_function_factory.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vlambda_function_expr.h"
+
+namespace doris {
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc)
+ *
+ * Merges two arrays that are already sorted by the elements' natural order. The merge lambda
+ * receives the two current candidates and returns the combined element; returning NULL means the
+ * two elements must not be combined, so both are kept. Each input key occurs at most once per
+ * array, so a key appears at most twice across the two arrays and the lambda is called at most
+ * once per key.
+ */
+class ArraySortedMergeFunction : public LambdaFunction {
+ ENABLE_FACTORY_CREATOR(ArraySortedMergeFunction);
+
+public:
+ ~ArraySortedMergeFunction() override = default;
+
+ static constexpr auto name = "array_sorted_merge";
+
+ static LambdaFunctionPtr create() { return std::make_shared<ArraySortedMergeFunction>(); }
+
+ std::string get_name() const override { return name; }
+
+ Status execute(VExprContext* context, const Block* block, const Selector* expr_selector,
+ size_t count, ColumnPtr& result_column, const DataTypePtr& result_type,
+ const VExprSPtrs& children) const override {
+ DCHECK_EQ(children.size(), 4);
+
+ const ColumnArray* arrays[2] = {nullptr, nullptr};
+ const IColumn* data[2] = {nullptr, nullptr};
+ DataTypePtr nested_types[2];
+ ColumnPtr holders[2];
+ const ColumnUInt8::Container* nullmaps[2] = {nullptr, nullptr};
+ for (int i = 0; i < 2; ++i) {
+ RETURN_IF_ERROR(children[i + 1]->execute_column(context, block, expr_selector, count,
+ holders[i]));
+ holders[i] = holders[i]->convert_to_full_column_if_const();
+ DataTypePtr type = children[i + 1]->execute_type(block);
+ const IColumn* array_column = holders[i].get();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(array_column)) {
+ nullmaps[i] = &nullable->get_null_map_column().get_data();
+ array_column = &nullable->get_nested_column();
+ type = assert_cast<const DataTypeNullable&>(*type).get_nested_type();
+ }
+ arrays[i] = assert_cast<const ColumnArray*>(array_column);
+ data[i] = arrays[i]->get_data_ptr().get();
+ nested_types[i] = assert_cast<const DataTypeArray&>(*type).get_nested_type();
+ }
+
+ ColumnPtr flag_holder;
+ RETURN_IF_ERROR(
+ children[3]->execute_column(context, block, expr_selector, count, flag_holder));
+ flag_holder = flag_holder->convert_to_full_column_if_const();
+ const IColumn* flag_column = flag_holder.get();
+ const ColumnUInt8::Container* flag_nullmap = nullptr;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(flag_column)) {
+ flag_nullmap = &nullable->get_null_map_column().get_data();
+ flag_column = &nullable->get_nested_column();
+ }
+ const auto& flags = assert_cast<const ColumnUInt8&>(*flag_column).get_data();
+
+ const size_t rows = arrays[0]->size();
+ auto result_nested = nested_types[0]->create_column();
+ auto result_offsets = ColumnArray::ColumnOffsets::create();
+ auto& offsets_data = result_offsets->get_data();
+ offsets_data.reserve(rows);
+ auto result_nullmap = ColumnUInt8::create(rows, 0);
+ auto& result_nullmap_data = result_nullmap->get_data();
+
+ // Two one-row columns hold the lambda arguments of the current merge step. They carry
+ // the lambda argument names because the column refs resolve by name when no
+ // position-based gap is set.
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ const std::vector<std::string>* argument_names =
+ lambda_expr->has_argument_names() ? &lambda_expr->argument_names() : nullptr;
+ Block lambda_block;
+ for (int i = 0; i < 2; ++i) {
+ const std::string argument_name =
+ argument_names != nullptr && i < argument_names->size()
+ ? (*argument_names)[i]
+ : "merge_arg_" + std::to_string(i);
+ lambda_block.insert(ColumnWithTypeAndName(nested_types[i]->create_column(),
+ nested_types[i], argument_name));
+ }
+ // The lambda arguments are bound by position (column ids 0 and 1), like array_sort's
+ // comparator, so the body reads these two one-row columns.
+ IColumn* argument_columns[2] = {
+ lambda_block.get_by_position(0).column->assert_mutable().get(),
+ lambda_block.get_by_position(1).column->assert_mutable().get()};
+
+ LambdaExecutionContext::Frame lambda_frame;
+ lambda_frame.bind_by_name = false;
+ lambda_frame.parent_bindings_visible = false;
+ LambdaExecutionContext::FrameGuard lambda_frame_guard(context->lambda_execution_context(),
+ std::move(lambda_frame));
+
+ const auto& offsets0 = arrays[0]->get_offsets();
+ const auto& offsets1 = arrays[1]->get_offsets();
+ const int lambda_result_base = static_cast<int>(lambda_block.columns());
+
+ for (size_t row = 0; row < rows; ++row) {
+ const bool row_is_null = (nullmaps[0] != nullptr && (*nullmaps[0])[row] != 0) ||
+ (nullmaps[1] != nullptr && (*nullmaps[1])[row] != 0);
+ if (row_is_null) {
+ result_nullmap_data[row] = 1;
+ offsets_data.push_back(result_nested->size());
+ continue;
+ }
+ if (flag_nullmap != nullptr && (*flag_nullmap)[row] != 0) {
+ return Status::InvalidArgument(
+ "array_sorted_merge requires a non-null ascending flag");
+ }
+ const bool ascending = flags[row] != 0;
+
+ size_t left = row == 0 ? 0 : offsets0[row - 1];
+ const size_t left_end = offsets0[row];
+ size_t right = row == 0 ? 0 : offsets1[row - 1];
+ const size_t right_end = offsets1[row];
+
+ while (left < left_end || right < right_end) {
+ if (left == left_end) {
+ result_nested->insert_from(*data[1], right++);
+ continue;
+ }
+ if (right == right_end) {
+ result_nested->insert_from(*data[0], left++);
+ continue;
+ }
+
+ argument_columns[0]->clear();
+ argument_columns[0]->insert_from(*data[0], left);
+ argument_columns[1]->clear();
+ argument_columns[1]->insert_from(*data[1], right);
+
+ int lambda_res_id = lambda_result_base;
+ RETURN_IF_ERROR(children[0]->execute(context, &lambda_block, &lambda_res_id));
Review Comment:
Fixed in 42efc96b45b: captured input columns are materialized at their original positions in the one-row lambda block for the current row, so an enclosing slot reads the correct value. A nested lambda that captures one of the merge arguments is rejected in prepare with NotSupported instead of resolving against the wrong column. Regression covers a table-column capture at index 0 (previously resolved to candidate a) and one beyond the two candidates (previously a missing column), plus the rejected nested capture.
##########
be/src/exprs/lambda_function/varray_sorted_merge_function.cpp:
##########
@@ -0,0 +1,215 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <string>
+#include <utility>
+
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "exprs/lambda_function/lambda_execution_context.h"
+#include "exprs/lambda_function/lambda_function.h"
+#include "exprs/lambda_function/lambda_function_factory.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vlambda_function_expr.h"
+
+namespace doris {
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc)
+ *
+ * Merges two arrays that are already sorted by the elements' natural order. The merge lambda
+ * receives the two current candidates and returns the combined element; returning NULL means the
+ * two elements must not be combined, so both are kept. Each input key occurs at most once per
+ * array, so a key appears at most twice across the two arrays and the lambda is called at most
+ * once per key.
+ */
+class ArraySortedMergeFunction : public LambdaFunction {
+ ENABLE_FACTORY_CREATOR(ArraySortedMergeFunction);
+
+public:
+ ~ArraySortedMergeFunction() override = default;
+
+ static constexpr auto name = "array_sorted_merge";
+
+ static LambdaFunctionPtr create() { return std::make_shared<ArraySortedMergeFunction>(); }
+
+ std::string get_name() const override { return name; }
+
+ Status execute(VExprContext* context, const Block* block, const Selector* expr_selector,
+ size_t count, ColumnPtr& result_column, const DataTypePtr& result_type,
+ const VExprSPtrs& children) const override {
+ DCHECK_EQ(children.size(), 4);
+
+ const ColumnArray* arrays[2] = {nullptr, nullptr};
+ const IColumn* data[2] = {nullptr, nullptr};
+ DataTypePtr nested_types[2];
+ ColumnPtr holders[2];
+ const ColumnUInt8::Container* nullmaps[2] = {nullptr, nullptr};
+ for (int i = 0; i < 2; ++i) {
+ RETURN_IF_ERROR(children[i + 1]->execute_column(context, block, expr_selector, count,
+ holders[i]));
+ holders[i] = holders[i]->convert_to_full_column_if_const();
+ DataTypePtr type = children[i + 1]->execute_type(block);
+ const IColumn* array_column = holders[i].get();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(array_column)) {
+ nullmaps[i] = &nullable->get_null_map_column().get_data();
+ array_column = &nullable->get_nested_column();
+ type = assert_cast<const DataTypeNullable&>(*type).get_nested_type();
+ }
+ arrays[i] = assert_cast<const ColumnArray*>(array_column);
+ data[i] = arrays[i]->get_data_ptr().get();
+ nested_types[i] = assert_cast<const DataTypeArray&>(*type).get_nested_type();
+ }
+
+ ColumnPtr flag_holder;
+ RETURN_IF_ERROR(
+ children[3]->execute_column(context, block, expr_selector, count, flag_holder));
+ flag_holder = flag_holder->convert_to_full_column_if_const();
+ const IColumn* flag_column = flag_holder.get();
+ const ColumnUInt8::Container* flag_nullmap = nullptr;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(flag_column)) {
+ flag_nullmap = &nullable->get_null_map_column().get_data();
+ flag_column = &nullable->get_nested_column();
+ }
+ const auto& flags = assert_cast<const ColumnUInt8&>(*flag_column).get_data();
+
+ const size_t rows = arrays[0]->size();
+ auto result_nested = nested_types[0]->create_column();
+ auto result_offsets = ColumnArray::ColumnOffsets::create();
+ auto& offsets_data = result_offsets->get_data();
+ offsets_data.reserve(rows);
+ auto result_nullmap = ColumnUInt8::create(rows, 0);
+ auto& result_nullmap_data = result_nullmap->get_data();
+
+ // Two one-row columns hold the lambda arguments of the current merge step. They carry
+ // the lambda argument names because the column refs resolve by name when no
+ // position-based gap is set.
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ const std::vector<std::string>* argument_names =
+ lambda_expr->has_argument_names() ? &lambda_expr->argument_names() : nullptr;
+ Block lambda_block;
+ for (int i = 0; i < 2; ++i) {
+ const std::string argument_name =
+ argument_names != nullptr && i < argument_names->size()
+ ? (*argument_names)[i]
+ : "merge_arg_" + std::to_string(i);
+ lambda_block.insert(ColumnWithTypeAndName(nested_types[i]->create_column(),
+ nested_types[i], argument_name));
+ }
+ // The lambda arguments are bound by position (column ids 0 and 1), like array_sort's
+ // comparator, so the body reads these two one-row columns.
+ IColumn* argument_columns[2] = {
+ lambda_block.get_by_position(0).column->assert_mutable().get(),
+ lambda_block.get_by_position(1).column->assert_mutable().get()};
+
+ LambdaExecutionContext::Frame lambda_frame;
+ lambda_frame.bind_by_name = false;
+ lambda_frame.parent_bindings_visible = false;
+ LambdaExecutionContext::FrameGuard lambda_frame_guard(context->lambda_execution_context(),
+ std::move(lambda_frame));
+
+ const auto& offsets0 = arrays[0]->get_offsets();
+ const auto& offsets1 = arrays[1]->get_offsets();
+ const int lambda_result_base = static_cast<int>(lambda_block.columns());
+
+ for (size_t row = 0; row < rows; ++row) {
+ const bool row_is_null = (nullmaps[0] != nullptr && (*nullmaps[0])[row] != 0) ||
+ (nullmaps[1] != nullptr && (*nullmaps[1])[row] != 0);
+ if (row_is_null) {
+ result_nullmap_data[row] = 1;
+ offsets_data.push_back(result_nested->size());
+ continue;
+ }
+ if (flag_nullmap != nullptr && (*flag_nullmap)[row] != 0) {
+ return Status::InvalidArgument(
+ "array_sorted_merge requires a non-null ascending flag");
+ }
+ const bool ascending = flags[row] != 0;
+
+ size_t left = row == 0 ? 0 : offsets0[row - 1];
+ const size_t left_end = offsets0[row];
+ size_t right = row == 0 ? 0 : offsets1[row - 1];
+ const size_t right_end = offsets1[row];
+
+ while (left < left_end || right < right_end) {
+ if (left == left_end) {
+ result_nested->insert_from(*data[1], right++);
+ continue;
+ }
+ if (right == right_end) {
+ result_nested->insert_from(*data[0], left++);
+ continue;
+ }
+
+ argument_columns[0]->clear();
+ argument_columns[0]->insert_from(*data[0], left);
+ argument_columns[1]->clear();
+ argument_columns[1]->insert_from(*data[1], right);
+
+ int lambda_res_id = lambda_result_base;
+ RETURN_IF_ERROR(children[0]->execute(context, &lambda_block, &lambda_res_id));
+ ColumnPtr merged = lambda_block.get_by_position(lambda_res_id)
+ .column->convert_to_full_column_if_const();
+ const IColumn* merged_column = merged.get();
+ bool merged_is_null = false;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(merged_column)) {
+ merged_is_null = nullable->is_null_at(0);
+ }
+ if (!merged_is_null) {
+ result_nested->insert_from(*merged_column, 0);
Review Comment:
Fixed in 42efc96b45b: the lambda result is normalized to the nullable array element representation before insertion -- a non-nullable body inserts its payload into the nested column and appends a 0 to the null map. A regression case with COALESCE(a, b, 0) now returns [1, 2].
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6337429782
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4347779333
##########
be/src/exprs/function/function_topk_merge.cpp:
##########
@@ -0,0 +1,482 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <glog/logging.h>
+
+#include <cstddef>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_numbers.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/types.h"
+#include "exprs/function/function.h"
+#include "exprs/function/simple_function_factory.h"
+
+namespace doris {
+class FunctionContext;
+} // namespace doris
+
+namespace doris {
+
+namespace {
+
+/// Upper bound of k. Keep in sync with the aggregate's MAX_TOPK_K
+/// (aggregate_function_topk_count.h) and FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/// One row of an ARRAY<STRUCT<value, count>> argument. A NULL array is an empty view.
+struct TopKArrayView {
+ const IColumn* values = nullptr;
+ const ColumnInt64* counts = nullptr;
+ /// NULL bitmap of the array element (the pair struct).
+ const NullMap* null_map = nullptr;
+ /// NULL bitmap of the `value` field; NULL values are not part of the contract.
+ const NullMap* value_null_map = nullptr;
+ /// NULL bitmap of the `count` field; NULL counts are not part of the contract.
+ const NullMap* count_null_map = nullptr;
+ size_t begin = 0;
+ size_t end = 0;
+};
+
+TopKArrayView make_array_view(const IColumn& column, size_t row) {
+ TopKArrayView view;
+ const IColumn* array_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ array_column = &nullable->get_nested_column();
+ }
+ const auto& array = assert_cast<const ColumnArray&>(*array_column);
+ // offsets[i] is the cumulative size after row i, so row's range is [offsets[row-1], offsets[row]).
+ // offset_at(0) reads offsets[-1], which PaddedPODArray keeps as 0.
+ view.begin = array.offset_at(static_cast<ssize_t>(row));
+ view.end = array.offset_at(static_cast<ssize_t>(row) + 1);
+
+ const IColumn* pair_column = &array.get_data();
+ if (const auto* element_nullable = check_and_get_column<ColumnNullable>(pair_column)) {
+ view.null_map = &element_nullable->get_null_map_data();
+ pair_column = &element_nullable->get_nested_column();
+ }
+ const auto& pairs = assert_cast<const ColumnStruct&>(*pair_column);
+ view.values = &pairs.get_column(0);
+ // A cast may widen the `value` field to a nullable column; the entries themselves are never
+ // NULL, so only the nested column is used.
+ if (const auto* value_nullable = check_and_get_column<ColumnNullable>(view.values)) {
+ view.value_null_map = &value_nullable->get_null_map_data();
+ view.values = &value_nullable->get_nested_column();
+ }
+ // The same holds for `count`: a nested cast may have made the field nullable.
+ const auto* count_column = &pairs.get_column(1);
+ if (const auto* count_nullable = check_and_get_column<ColumnNullable>(count_column)) {
+ view.count_null_map = &count_nullable->get_null_map_data();
+ count_column = &count_nullable->get_nested_column();
+ }
+ view.counts = &assert_cast<const ColumnInt64&>(*count_column);
+ return view;
+}
+
+/// One scalar bound argument: NULL means the corresponding side was not truncated. `row` is the
+/// index of the current input row inside `value`: bounds are ordinary per-row columns, so a block
+/// can carry a different bound in every row.
+struct TopKBoundView {
+ const IColumn* value = nullptr;
+ size_t row = 0;
+ bool is_null = true;
+};
+
+TopKBoundView make_bound_view(const IColumn& column, size_t row) {
+ TopKBoundView view;
+ const IColumn* value_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ view.value = value_column;
+ view.row = row;
+ view.is_null = false;
+ return view;
+}
+
+int64_t read_int64(const IColumn& column, size_t row, const char* argument_name) {
+ const IColumn* nested = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ // The contract declares these arguments as non-NULL BIGINT.
+ DORIS_CHECK(!nullable->is_null_at(row)) << argument_name << " must not be NULL";
+ nested = &nullable->get_nested_column();
+ }
+ return assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(*nested).get_element(row);
+}
+
+int64_t checked_add(int64_t lhs, int64_t rhs, const char* function_name) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "{}: value count overflow", function_name);
+ }
+ return result;
+}
+
+int64_t checked_sub(int64_t lhs, int64_t rhs, const char* function_name) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_sub_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "{}: value count overflow", function_name);
+ }
+ return result;
+}
+
+/// FLOAT/DOUBLE use CompareFloat so NaN sorts greatest and is distinct from every number; other
+/// types use the column comparison, i.e. exactly the MIN/MAX comparison.
+int compare_values(PrimitiveType value_type, const IColumn& lhs, size_t lhs_row, const IColumn& rhs,
+ size_t rhs_row) {
+ if (value_type == TYPE_FLOAT) {
+ return CompareFloat(
+ assert_cast<const ColumnFloat32&, TypeCheckOnRelease::DISABLE>(lhs).get_element(
+ lhs_row),
+ assert_cast<const ColumnFloat32&, TypeCheckOnRelease::DISABLE>(rhs).get_element(
+ rhs_row));
+ }
+ if (value_type == TYPE_DOUBLE) {
+ return CompareFloat(
+ assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(lhs).get_element(
+ lhs_row),
+ assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(rhs).get_element(
+ rhs_row));
+ }
+ return lhs.compare_at(lhs_row, rhs_row, rhs, 1);
+}
+
+DataTypePtr find_value_type(const DataTypes& arguments) {
+ // Prefer the per-value type of S/I/D; it is also present when all bound arguments are NULL.
+ for (size_t index : {0, 2, 4}) {
+ const auto& type = remove_nullable(arguments[index]);
+ if (type->get_primitive_type() != TYPE_ARRAY) {
+ continue;
+ }
+ const auto& nested =
+ remove_nullable(assert_cast<const DataTypeArray&>(*type).get_nested_type());
+ if (nested->get_primitive_type() != TYPE_STRUCT) {
+ continue;
+ }
+ const auto& fields = assert_cast<const DataTypeStruct&>(*nested).get_elements();
+ if (fields.size() != 2) {
+ continue;
+ }
+ // The result struct always declares a non-nullable `value` field, while an argument cast
+ // may have produced a nullable one; drop it so both sides agree on the return type.
+ return remove_nullable(fields[0]);
+ }
+ for (size_t index : {1, 3, 5}) {
+ const auto& type = remove_nullable(arguments[index]);
+ if (!type->is_null_literal()) {
+ return type;
+ }
+ }
+ return nullptr;
+}
+
+/**
+ * topk_merge_min / topk_merge_max: one apply step of the IVM MIN/MAX top-K state.
+ *
+ * Arguments: S, b, I, b_i, D, b_d, n, dn, k. S is the pre-image state, I/D are the insert/delete
+ * top-K lists of this delta batch and b/i/d bounds are their truncation points (NULL = not
+ * truncated). Only values in [<= bE] are trusted, where bE = min of all non-NULL bounds (max for
+ * MAX): outside it at least one side may have dropped values. The post count of a value is
+ * S + I - D; non-positive counts are removed (net-zero cancellations), then the first k values are
+ * kept.
+ *
+ * The result is the state itself, i.e. the same type topk_min_count/topk_max_count return:
+ * STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T NULL>. n' = n + dn and
+ * exhausted = (size(values) == 0 && n' > 0) are derived by the caller, which writes n' to the
+ * COUNT state column and uses exhausted to trigger the fallback.
+ */
+template <bool IsMin>
+class FunctionTopKMerge : public IFunction {
+public:
+ static constexpr auto name = IsMin ? "topk_merge_min" : "topk_merge_max";
+
+ static FunctionPtr create() { return std::make_shared<FunctionTopKMerge>(); }
+
+ String get_name() const override { return name; }
+
+ size_t get_number_of_arguments() const override { return 9; }
+
+ /// A NULL list is an empty list and a NULL bound is a valid value: the default implementation
+ /// would turn the whole row into NULL and break the contract.
+ bool use_default_implementation_for_nulls() const override { return false; }
+
+ DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ DataTypePtr value_type = find_value_type(arguments);
+ if (value_type == nullptr) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} can not infer the value type from arguments", name);
+ }
+ // The result is the state type, identical to the aggregate's return type, so that the
+ // apply expression can write it straight back to the hidden MIN_TOPK/MAX_TOPK and BOUND
+ // columns.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {value_type, std::make_shared<DataTypeInt64>()},
+ Strings {"value", "count"});
+ auto values_type = std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ return std::make_shared<DataTypeStruct>(DataTypes {values_type, make_nullable(value_type)},
+ Strings {"values", "bound"});
+ }
+
+ Status execute_impl(FunctionContext* /*context*/, Block& block, const ColumnNumbers& arguments,
+ uint32_t result, size_t input_rows_count) const override {
+ std::vector<ColumnPtr> columns(arguments.size());
+ DataTypes argument_types(arguments.size());
+ for (size_t i = 0; i != arguments.size(); ++i) {
+ columns[i] =
+ block.get_by_position(arguments[i]).column->convert_to_full_column_if_const();
+ argument_types[i] = block.get_by_position(arguments[i]).type;
+ }
+ const DataTypePtr value_type = find_value_type(argument_types);
+ // An untyped call is rejected while planning get_return_type(); the plan always carries T.
+ DCHECK(value_type != nullptr);
+ const PrimitiveType primitive_type = value_type->get_primitive_type();
+
+ auto result_column = block.get_by_position(result).type->create_column();
+ auto& result_struct = assert_cast<ColumnStruct&>(*result_column);
+ auto& out_values = assert_cast<ColumnArray&>(result_struct.get_column(0));
+ auto& out_pair_nullable = assert_cast<ColumnNullable&>(out_values.get_data());
+ auto& out_pair = assert_cast<ColumnStruct&>(out_pair_nullable.get_nested_column());
+ auto& out_value_column = out_pair.get_column(0);
+ auto& out_count_column = assert_cast<ColumnInt64&>(out_pair.get_column(1));
+ auto& out_bound_column = assert_cast<ColumnNullable&>(result_struct.get_column(1));
+
+ for (size_t row = 0; row != input_rows_count; ++row) {
+ const int64_t k = read_int64(*columns[8], row, "k");
+ if (UNLIKELY(k < 1 || k > MAX_TOPK_K)) {
+ return Status::InvalidArgument("{} requires 1 <= k <= {}, but got {}", name,
+ MAX_TOPK_K, k);
+ }
+ const int64_t n = read_int64(*columns[6], row, "n");
+ const int64_t dn = read_int64(*columns[7], row, "dn");
+ const int64_t new_n = checked_add(n, dn, name);
+
+ const TopKArrayView s_view = make_array_view(*columns[0], row);
+ const TopKArrayView i_view = make_array_view(*columns[2], row);
+ const TopKArrayView d_view = make_array_view(*columns[4], row);
+ const TopKBoundView b_view = make_bound_view(*columns[1], row);
+ const TopKBoundView bi_view = make_bound_view(*columns[3], row);
+ const TopKBoundView bd_view = make_bound_view(*columns[5], row);
+ const TopKBoundView effective_bound =
+ effective_bound_of(b_view, bi_view, bd_view, primitive_type);
+
+ size_t positive = 0;
+ RETURN_IF_ERROR(append_trusted_values(s_view, i_view, d_view, effective_bound, k,
+ primitive_type, out_value_column,
+ out_count_column, out_pair_nullable, &positive));
+ out_values.get_offsets().push_back(out_pair.size());
+
+ // bound: NULL while the state is complete, the k-th kept value only when values were
+ // actually dropped (pos > k), or the effective bound otherwise.
+ // b IS NULL means the pre-image covers every distinct value; a NULL b_i/b_d means
+ // that side of the delta was not truncated.
+ // pos == k is NOT truncation: every known value fit, so keep bE instead of shrinking
+ // the trusted range to max(S').
+ const bool sides_complete = b_view.is_null && bi_view.is_null && bd_view.is_null;
+ if (new_n == 0 || (sides_complete && std::cmp_less_equal(positive, k))) {
+ out_bound_column.insert_default();
+ } else if (std::cmp_greater(positive, k)) {
+ out_bound_column.get_nested_column().insert_from(out_value_column,
+ out_pair.size() - 1);
+ out_bound_column.get_null_map_data().push_back(0);
+ } else {
+ DCHECK(!effective_bound.is_null);
+ out_bound_column.get_nested_column().insert_from(*effective_bound.value,
+ effective_bound.row);
+ out_bound_column.get_null_map_data().push_back(0);
+ }
+ }
+
+ block.replace_by_position(result, std::move(result_column));
+ return Status::OK();
+ }
+
+private:
+ /// bE: the best (smallest for MIN, largest for MAX) of the non-NULL bounds, or nullptr when
+ /// every side is complete/untruncated.
+ static TopKBoundView effective_bound_of(const TopKBoundView& b_view,
+ const TopKBoundView& bi_view,
+ const TopKBoundView& bd_view, PrimitiveType type) {
+ TopKBoundView bound;
+ for (const TopKBoundView* view : {&b_view, &bi_view, &bd_view}) {
+ if (view->is_null) {
+ continue;
+ }
+ if (bound.is_null) {
+ bound = *view;
+ continue;
+ }
+ int cmp = compare_values(type, *view->value, view->row, *bound.value, bound.row);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ bound = *view;
+ }
+ }
+ return bound;
+ }
+
+ /// A list may have a nullable element type, but its entries, values and counts must not be
+ /// NULL: the lists are top-K states, where NULL is neither counted nor representable. Reading
+ /// a NULL payload would emit the column's default value as a real entry, so reject it.
+ static Status check_entry(const TopKArrayView& view, size_t index) {
+ if ((view.null_map != nullptr && (*view.null_map)[index] != 0) ||
+ (view.value_null_map != nullptr && (*view.value_null_map)[index] != 0) ||
+ (view.count_null_map != nullptr && (*view.count_null_map)[index] != 0)) {
+ return Status::InvalidArgument(
+ "{}: the state lists must not contain NULL entries, values or counts", name);
+ }
+ return Status::OK();
+ }
+
+ /// Finds the best head (smallest for MIN, largest for MAX) among the three lists; `head` is
+ /// set to nullptr when all of them are exhausted.
+ static Status best_head(const TopKArrayView& s_view, size_t s_index,
+ const TopKArrayView& i_view, size_t i_index,
+ const TopKArrayView& d_view, size_t d_index, PrimitiveType type,
+ const IColumn** head, size_t* head_row) {
+ *head = nullptr;
+ const TopKArrayView* views[] = {&s_view, &i_view, &d_view};
+ const size_t indexes[] = {s_index, i_index, d_index};
+ for (size_t v = 0; v != 3; ++v) {
+ if (indexes[v] == views[v]->end) {
+ continue;
+ }
+ RETURN_IF_ERROR(check_entry(*views[v], indexes[v]));
+ if (*head == nullptr) {
+ *head = views[v]->values;
+ *head_row = indexes[v];
+ continue;
+ }
+ int cmp = compare_values(type, *views[v]->values, indexes[v], **head, *head_row);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ *head = views[v]->values;
+ *head_row = indexes[v];
+ }
+ }
+ return Status::OK();
+ }
+
+ /// Consumes one entry equal to the current head value, returning its count.
+ static Status consume_equal(const TopKArrayView& view, size_t& index, const IColumn& head,
+ size_t head_row, PrimitiveType type, int64_t* count) {
+ if (index == view.end) {
+ return Status::OK();
+ }
+ RETURN_IF_ERROR(check_entry(view, index));
+ if (compare_values(type, *view.values, index, head, head_row) != 0) {
+ return Status::OK();
+ }
+ *count = view.counts->get_element(index);
+ ++index;
+ return Status::OK();
+ }
+
+ /// Merges the trusted part (v <= bE for MIN) of S, I and D into the output list, keeping the
+ /// first k values with a positive post count S + I - D. `positive` receives the number of
+ /// positive candidates, capped at k + 1: once more than k are found the scan stops, which is
+ /// enough to decide the bound.
+ Status append_trusted_values(const TopKArrayView& s_view, const TopKArrayView& i_view,
+ const TopKArrayView& d_view, const TopKBoundView& effective_bound,
+ int64_t k, PrimitiveType type, IColumn& out_value_column,
+ ColumnInt64& out_count_column, ColumnNullable& out_pair_nullable,
+ size_t* positive) const {
+ size_t s_index = s_view.begin;
+ size_t i_index = i_view.begin;
+ size_t d_index = d_view.begin;
+ size_t kept = 0;
+ while (true) {
+ const IColumn* head = nullptr;
+ size_t head_row = 0;
+ RETURN_IF_ERROR(best_head(s_view, s_index, i_view, i_index, d_view, d_index, type,
+ &head, &head_row));
+ if (head == nullptr) {
+ break;
+ }
+ if (!effective_bound.is_null) {
+ int cmp = compare_values(type, *head, head_row, *effective_bound.value,
+ effective_bound.row);
+ if (IsMin ? cmp > 0 : cmp < 0) {
+ // Everything left is outside the range whose counts both sides know exactly.
+ break;
+ }
+ }
+
+ int64_t count_s = 0;
+ int64_t count_i = 0;
+ int64_t count_d = 0;
+ RETURN_IF_ERROR(consume_equal(s_view, s_index, *head, head_row, type, &count_s));
+ RETURN_IF_ERROR(consume_equal(i_view, i_index, *head, head_row, type, &count_i));
+ RETURN_IF_ERROR(consume_equal(d_view, d_index, *head, head_row, type, &count_d));
+
+ int64_t post = checked_add(count_s, count_i, name);
Review Comment:
[P2] Check overflow after insert/delete cancellation
With complete `S=[(1,9223372036854775807)]`, `I=D=[(1,1)]`, NULL bounds, `n=9223372036854775807`, `dn=0` and `k=1`, a valid delta replaces one occurrence with the identical value and the result should remain S. This addition overflows before the deletion is subtracted, although every input and final count fits BIGINT. MAX shares the path. Evaluate `S+I-D` with a wider intermediate or cancellation-safe ordering, range-check the final count, and cover this boundary cancellation.
##########
be/src/exprs/function/function_topk_merge.cpp:
##########
@@ -0,0 +1,482 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <glog/logging.h>
+
+#include <cstddef>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_numbers.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/types.h"
+#include "exprs/function/function.h"
+#include "exprs/function/simple_function_factory.h"
+
+namespace doris {
+class FunctionContext;
+} // namespace doris
+
+namespace doris {
+
+namespace {
+
+/// Upper bound of k. Keep in sync with the aggregate's MAX_TOPK_K
+/// (aggregate_function_topk_count.h) and FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/// One row of an ARRAY<STRUCT<value, count>> argument. A NULL array is an empty view.
+struct TopKArrayView {
+ const IColumn* values = nullptr;
+ const ColumnInt64* counts = nullptr;
+ /// NULL bitmap of the array element (the pair struct).
+ const NullMap* null_map = nullptr;
+ /// NULL bitmap of the `value` field; NULL values are not part of the contract.
+ const NullMap* value_null_map = nullptr;
+ /// NULL bitmap of the `count` field; NULL counts are not part of the contract.
+ const NullMap* count_null_map = nullptr;
+ size_t begin = 0;
+ size_t end = 0;
+};
+
+TopKArrayView make_array_view(const IColumn& column, size_t row) {
+ TopKArrayView view;
+ const IColumn* array_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ array_column = &nullable->get_nested_column();
+ }
+ const auto& array = assert_cast<const ColumnArray&>(*array_column);
+ // offsets[i] is the cumulative size after row i, so row's range is [offsets[row-1], offsets[row]).
+ // offset_at(0) reads offsets[-1], which PaddedPODArray keeps as 0.
+ view.begin = array.offset_at(static_cast<ssize_t>(row));
+ view.end = array.offset_at(static_cast<ssize_t>(row) + 1);
+
+ const IColumn* pair_column = &array.get_data();
+ if (const auto* element_nullable = check_and_get_column<ColumnNullable>(pair_column)) {
+ view.null_map = &element_nullable->get_null_map_data();
+ pair_column = &element_nullable->get_nested_column();
+ }
+ const auto& pairs = assert_cast<const ColumnStruct&>(*pair_column);
+ view.values = &pairs.get_column(0);
+ // A cast may widen the `value` field to a nullable column; the entries themselves are never
+ // NULL, so only the nested column is used.
+ if (const auto* value_nullable = check_and_get_column<ColumnNullable>(view.values)) {
+ view.value_null_map = &value_nullable->get_null_map_data();
+ view.values = &value_nullable->get_nested_column();
+ }
+ // The same holds for `count`: a nested cast may have made the field nullable.
+ const auto* count_column = &pairs.get_column(1);
+ if (const auto* count_nullable = check_and_get_column<ColumnNullable>(count_column)) {
+ view.count_null_map = &count_nullable->get_null_map_data();
+ count_column = &count_nullable->get_nested_column();
+ }
+ view.counts = &assert_cast<const ColumnInt64&>(*count_column);
+ return view;
+}
+
+/// One scalar bound argument: NULL means the corresponding side was not truncated. `row` is the
+/// index of the current input row inside `value`: bounds are ordinary per-row columns, so a block
+/// can carry a different bound in every row.
+struct TopKBoundView {
+ const IColumn* value = nullptr;
+ size_t row = 0;
+ bool is_null = true;
+};
+
+TopKBoundView make_bound_view(const IColumn& column, size_t row) {
+ TopKBoundView view;
+ const IColumn* value_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ view.value = value_column;
+ view.row = row;
+ view.is_null = false;
+ return view;
+}
+
+int64_t read_int64(const IColumn& column, size_t row, const char* argument_name) {
+ const IColumn* nested = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ // The contract declares these arguments as non-NULL BIGINT.
+ DORIS_CHECK(!nullable->is_null_at(row)) << argument_name << " must not be NULL";
Review Comment:
[P2] Reject NULL scalar counters with a query error
The FE signature accepts nullable BIGINT arguments, so `SELECT topk_merge_min(CAST(NULL AS ARRAY<STRUCT<value:INT,count:BIGINT>>),NULL,NULL,NULL,NULL,NULL,0,0,CAST(NULL AS BIGINT))` reaches this check. `DORIS_CHECK` calls `Status::FatalError`, which executes `LOG(FATAL)` in builds without `NDEBUG`, terminating the BE for invalid SQL input; release builds instead return `FATAL_ERROR`. NULL `n`/`dn` and `topk_merge_max` follow the same path. Return `INVALID_ARGUMENT` for actual NULL counter values, including nullable nonliteral inputs, and cover all three counters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6432150367
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6435393286
# BE Regression && UT Coverage Report
Increment line coverage `93.39% (311/333)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_5309c29db2b57c559711d38b6621f5c934c95d3a_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_5309c29db2b57c559711d38b6621f5c934c95d3a_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.44% (36116/46639) |
| Line Coverage | 62.70% (408528/651542) |
| Region Coverage | 59.14% (345245/583733) |
| Branch Coverage | 60.08% (159836/266034) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6407753895
# BE Regression && UT Coverage Report
Increment line coverage `81.76% (269/329)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 68.40% (31863/46585) |
| Line Coverage | 53.06% (345074/650355) |
| Region Coverage | 48.42% (282023/582490) |
| Branch Coverage | 49.90% (132269/265083) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4400984563
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,786 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ // The payload is proportional to the slot count, which the caller already compares.
+ size_t live_bytes() const { return 0; }
+ size_t dead_bytes() const { return 0; }
+ void on_evict(size_t /*slot*/) {}
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ size_t offset = 0;
+ size_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ /// PaddedPODArray, not PODArray: comparisons go through
+ /// memcmp_small_allow_overflow15, which loads a full 16-byte vector even for a one-byte
+ /// string, the same tail allowance ColumnString's chars buffer provides.
+ PaddedPODArray<char, 32> bytes;
+ DorisVector<Span> spans;
+ // Sum of the live slots' payload and of the payload evicted since the last compaction. The
+ // slot counters alone cannot bound the memory: replacing 1024 1 MiB strings with short ones
+ // leaves dead == live by slot count while ~1 GiB of discarded bytes stay in the arena.
+ size_t live_payload_bytes = 0;
+ size_t dead_payload_bytes = 0;
+
+ size_t size() const { return spans.size(); }
+ size_t live_bytes() const { return live_payload_bytes; }
+ size_t dead_bytes() const { return dead_payload_bytes; }
+ void reserve(size_t n) { spans.reserve(n); }
+ void clear() {
+ bytes.clear();
+ spans.clear();
+ live_payload_bytes = 0;
+ dead_payload_bytes = 0;
+ }
+ StringRef at(size_t slot) const {
+ const Span& span = spans[slot];
+ return StringRef(bytes.data() + span.offset, span.length);
+ }
+ void append(StringRef value) {
+ spans.push_back({bytes.size(), value.size});
+ live_payload_bytes += value.size;
+ if (value.size != 0) {
+ bytes.insert(value.data, value.data + value.size);
+ }
+ }
+ /// Called when `slot` is evicted: its payload stays in the arena until the next compaction.
+ void on_evict(size_t slot) {
+ live_payload_bytes -= spans[slot].length;
+ dead_payload_bytes += spans[slot].length;
+ }
+ void swap(StringTopKValueStore& other) noexcept {
+ bytes.swap(other.bytes);
+ spans.swap(other.spans);
+ std::swap(live_payload_bytes, other.live_payload_bytes);
+ std::swap(dead_payload_bytes, other.dead_payload_bytes);
+ }
+};
+
+/**
+ * State shared by min_topk and max_topk, for one value type.
+ *
+ * The distinct values sit in slots (see the stores above), with one Slot record per value carrying
+ * the row count and the links of a treap ordered best-first (ascending values for MIN, descending
+ * for MAX). A row costs one O(log k) descent: it either finds its value and bumps the count or
+ * links a new slot at the leaf. A sorted vector would shift O(k) elements for the same insert.
+ *
+ * The treap stores explicit child links because, unlike a heap, its shape is not the complete tree
+ * that an array index can describe: rotations move entries, and the in-order sequence is the value
+ * order rather than the arrival order. A heap could evict and append in O(log k), but it cannot
+ * find an existing value without a hash index or a linear scan, and it cannot produce the sorted
+ * result without sorting. The ordered tree is the structure that keeps find, insert and eviction
+ * all at O(log k) with comparisons only.
+ *
+ * An entry that is evicted keeps its slot and value until the next compaction. The tree never
+ * points at a dead slot.
+ *
+ * With k == 0 the state holds at most the single extreme value and its row count; the caller
+ * derives truncation from the list length, so the state carries no separate bound.
+ */
+template <PrimitiveType T>
+struct AggregateTopKCountData {
+ using CppType = typename PrimitiveTypeTraits<T>::CppType;
+ static constexpr bool IS_STRING = is_string_type(T);
+ using ValueStore =
+ std::conditional_t<IS_STRING, StringTopKValueStore, FixedTopKValueStore<CppType>>;
+ using ValueRef = std::conditional_t<IS_STRING, StringRef, const CppType&>;
+
+ static constexpr int32_t NO_SLOT = -1;
+ // Row counts are never negative, so this cannot collide with a live entry.
+ static constexpr int64_t DEAD_COUNT = -1;
+
+ /// One slot's metadata; the value itself lives in `values` at the same index.
+ struct Slot {
+ int64_t count = 0;
+ int32_t left = NO_SLOT;
+ int32_t right = NO_SLOT;
+ uint32_t priority = 0;
+ };
+
+ bool has_k = false;
+ int64_t k = 0;
+ // Indexed by slot; an evicted slot stays as a dead entry until compaction.
+ ValueStore values;
+ DorisVector<Slot> slots;
+ int32_t root = NO_SLOT;
+ // The last live slot in the aggregate's order: the value an incoming row has to beat to enter
+ // a full list. Cached so that rejecting a worse value costs one comparison, not a descent.
+ int32_t worst = NO_SLOT;
+ int64_t live = 0;
+ // Slots evicted since the last compaction.
+ int64_t dead = 0;
+ uint64_t rng = 0x243F6A8885A308D3ULL;
+
+ AggregateTopKCountData() = default;
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * min_topk(x, k) / max_topk(x, k).
+ *
+ * Result: ARRAY<STRUCT<value T, count BIGINT>>. NULL inputs are ignored: add validates k for
+ * every row and then skips a NULL x, so the aggregate deliberately does not use the
+ * nullable-argument wrapper. The array is ordered by value (ascending for MIN), holds at most k
+ * distinct values with count > 0, and carries no truncation flag: the caller infers it from the
+ * length (a list of exactly k entries is treated as possibly truncated).
+ *
+ * With k = 0 the array holds at most the extreme value and its row count, which lets the
+ * append-only IVM path compute MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopK final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData<T>,
+ AggregateFunctionTopK<T, IsMin>, false>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using State = AggregateTopKCountData<T>;
+ using Base = IAggregateFunctionDataHelper<State, AggregateFunctionTopK<T, IsMin>, false>;
+ using Slot = typename State::Slot;
+ using ValueStore = typename State::ValueStore;
+ using ValueRef = typename State::ValueRef;
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ static constexpr bool IS_STRING = State::IS_STRING;
+ static constexpr int32_t NO_SLOT = State::NO_SLOT;
+
+ explicit AggregateFunctionTopK(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "min_topk" : "max_topk"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(remove_nullable(this->argument_types[0]));
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is read for every row, including rows where x is NULL: an evaluated count that is
+ // NULL or out of range must fail the query instead of being skipped like a NULL value.
+ check_k(state, *columns[1], row_num);
+ const IColumn* value_column = columns[0];
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(value_column)) {
+ if (nullable->is_null_at(row_num)) {
+ return;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ add_value(state, *value_column, row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw any row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else if (UNLIKELY(state.k != other.k)) {
+ // Ordinary partial/final aggregation always shares the same constant k, but the
+ // public _state/_merge/_union combinators do not encode k in the AggState type, so a
+ // UNION ALL of states built with different literal k can reach this merge.
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} states have incompatible counts: {} vs {}", get_name(), state.k,
+ other.k);
+ }
+ if (state.k == 0) {
+ merge_extreme(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ // The slots are in arrival order and may include dead ones, while the serialized form is
+ // the sorted (value, count) list the reader rebuilds the tree from.
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ auto sorted_values = remove_nullable(this->argument_types[0])->create_column();
+ sorted_values->reserve(order.size());
+ DorisVector<int64_t> sorted_counts;
+ sorted_counts.reserve(order.size());
+ for (int32_t slot : order) {
+ insert_value(*sorted_values, state.values.at(slot));
+ sorted_counts.push_back(state.slots[slot].count);
+ }
+ auto size = static_cast<int64_t>(sorted_counts.size());
+ buf.write_binary(size);
+ write_column(buf, *sorted_values);
+ if (size != 0) {
+ // An empty list keeps an empty vector, whose data() is null; passing it to memcpy is
+ // undefined even for a zero length.
+ buf.write(reinterpret_cast<const char*>(sorted_counts.data()),
+ sorted_counts.size() * sizeof(int64_t));
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ int64_t size = 0;
+ buf.read_binary(size);
+ auto values = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(values->size()), size);
+ DorisVector<int64_t> counts(static_cast<size_t>(size));
+ if (size != 0) {
+ buf.read(reinterpret_cast<char*>(counts.data()), size * sizeof(int64_t));
+ }
+ ValueStore value_store;
+ value_store.reserve(static_cast<size_t>(size));
+ for (int64_t row = 0; row != size; ++row) {
+ value_store.append(value_at(*values, static_cast<size_t>(row)));
+ }
+ assign_sorted(state, std::move(value_store), std::move(counts));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_array = assert_cast<ColumnArray&>(to);
+ // The array element and both fields are nullable in the planned type, but a produced entry
+ // never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(result_array.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = assert_cast<ColumnNullable&>(pair_column.get_column(0));
+ auto& count_column = assert_cast<ColumnNullable&>(pair_column.get_column(1));
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ for (int32_t slot : order) {
+ insert_value(value_column.get_nested_column(), state.values.at(slot));
+ value_column.get_null_map_data().push_back(0);
+ assert_cast<ColumnInt64&>(count_column.get_nested_column())
+ .get_data()
+ .push_back(state.slots[slot].count);
+ count_column.get_null_map_data().push_back(0);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ result_array.get_offsets().push_back(pair_column.size());
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ // Both fields are nullable in the planned type: a consumer reads a field through
+ // element_at, which is always nullable, and named_struct then records nullable fields.
+ // Produced entries never contain a NULL field.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(value_type),
+ make_nullable(std::make_shared<DataTypeInt64>())},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ return std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ }
+
+private:
+ /// The value of the row in `column`. The returned view is valid while the column is alive.
+ static ValueRef value_at(const IColumn& column, size_t row_num) {
+ if constexpr (IS_STRING) {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column).get_data_at(
+ row_num);
+ } else {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column)
+ .get_data()[row_num];
+ }
+ }
+
+ /// Exactly the comparison the value column would apply: CompareFloat for FLOAT/DOUBLE through
+ /// Compare::compare (the column uses the same helper), the same memcmp helper ColumnString
+ /// uses for strings, and the column's scale-independent ordering for every other type.
+ static int compare_values(ValueRef lhs, ValueRef rhs) {
+ if constexpr (IS_STRING) {
+ return memcmp_small_allow_overflow15(
+ reinterpret_cast<const uint8_t*>(lhs.data), lhs.size,
+ reinterpret_cast<const uint8_t*>(rhs.data), rhs.size);
+ } else {
+ return Compare::compare(lhs, rhs);
+ }
+ }
+
+ static void insert_value(IColumn& column, ValueRef value) {
+ if constexpr (IS_STRING) {
+ assert_cast<ColumnString&>(column).insert_data(value.data, value.size);
+ } else {
+ assert_cast<ColumnType&>(column).get_data().push_back(value);
+ }
+ }
+
+ /// True when `cmp` puts the left operand before the right one in the aggregate's order.
+ static bool comes_before(int cmp) { return IsMin ? cmp < 0 : cmp > 0; }
+
+ /// True when `cmp`, which compares a value with the worst retained one, puts it after it: the
+ /// value is worse and cannot enter the list.
+ static bool is_worse(int cmp) { return IsMin ? cmp > 0 : cmp < 0; }
+
+ /// splitmix64. The priorities only have to be independent enough to keep the treap balanced;
+ /// a per-state sequence keeps the shape reproducible for the same insertion order.
+ static uint32_t next_priority(State& state) {
+ uint64_t z = (state.rng += 0x9E3779B97F4A7C15ULL);
+ z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9ULL;
+ z = (z ^ (z >> 27)) * 0x94D049BB133111EBULL;
+ return static_cast<uint32_t>(z ^ (z >> 31));
+ }
+
+ static int compare_slots(const State& state, int32_t lhs, int32_t rhs) {
+ return compare_values(state.values.at(lhs), state.values.at(rhs));
+ }
+
+ /// Appends (value, count) as the next slot.
+ static int32_t append_slot(State& state, ValueRef value, int64_t count) {
+ auto slot = static_cast<int32_t>(state.slots.size());
+ DCHECK_LT(slot, std::numeric_limits<int32_t>::max());
+ state.values.append(value);
+ state.slots.push_back(Slot {count, State::NO_SLOT, State::NO_SLOT, next_priority(state)});
+ ++state.live;
+ return slot;
+ }
+
+ static int32_t append_slot(State& state, const IColumn& column, size_t row_num, int64_t count) {
+ return append_slot(state, value_at(column, row_num), count);
+ }
+
+ /// Rotates the subtree rooted at `node` right: the left child becomes the root and `node` its
+ /// right child. The in-order sequence is unchanged, so the treap stays ordered.
+ static int32_t rotate_right(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].left;
+ state.slots[node].left = state.slots[pivot].right;
+ state.slots[pivot].right = node;
+ return pivot;
+ }
+
+ static int32_t rotate_left(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].right;
+ state.slots[node].right = state.slots[pivot].left;
+ state.slots[pivot].left = node;
+ return pivot;
+ }
+
+ /// One descent that performs the whole per-row update: an equal value bumps its count and
+ /// reports no insertion; a missing value is linked as a new slot at the leaf and rotated up
+ /// while the recursion unwinds. `inserted_slot` receives the new slot, NO_SLOT on a hit.
+ static int32_t add_or_increment(State& state, int32_t node, const IColumn& column,
+ size_t row_num, int32_t* inserted_slot) {
+ if (node == State::NO_SLOT) {
+ *inserted_slot = append_slot(state, column, row_num, 1);
+ return *inserted_slot;
+ }
+ int cmp = compare_values(state.values.at(node), value_at(column, row_num));
+ if (cmp == 0) {
+ state.slots[node].count = checked_add(state.slots[node].count, 1);
+ return node;
+ }
+ if (comes_before(cmp)) {
+ // The stored value is the better one, so the input belongs in the right subtree.
+ state.slots[node].right = add_or_increment(state, state.slots[node].right, column,
+ row_num, inserted_slot);
+ if (state.slots[state.slots[node].right].priority > state.slots[node].priority) {
+ return rotate_left(state, node);
+ }
+ } else {
+ state.slots[node].left =
+ add_or_increment(state, state.slots[node].left, column, row_num, inserted_slot);
+ if (state.slots[state.slots[node].left].priority > state.slots[node].priority) {
+ return rotate_right(state, node);
+ }
+ }
+ return node;
+ }
+
+ /// Removes the last entry in the aggregate's order (the largest value for MIN) and returns the
+ /// new subtree root. That entry has no right child, so its left subtree takes its place; the
+ /// child's priority is below the parent's, so no rotation is needed.
+ static int32_t erase_worst(State& state, int32_t node, int32_t* removed) {
+ if (state.slots[node].right == State::NO_SLOT) {
+ *removed = node;
+ return state.slots[node].left;
+ }
+ state.slots[node].right = erase_worst(state, state.slots[node].right, removed);
+ return node;
+ }
+
+ static void refresh_worst(State& state) {
+ int32_t node = state.root;
+ while (node != State::NO_SLOT && state.slots[node].right != State::NO_SLOT) {
+ node = state.slots[node].right;
+ }
+ state.worst = node;
+ }
+
+ /// The first live slot in the aggregate's order, i.e. the best value the state holds.
+ static int32_t best_slot(const State& state) {
+ int32_t node = state.root;
+ while (node != State::NO_SLOT && state.slots[node].left != State::NO_SLOT) {
+ node = state.slots[node].left;
+ }
+ return node;
+ }
+
+ /// Slots in the aggregate's order, i.e. the order of the result array.
+ static void collect_in_order(const State& state, DorisVector<int32_t>& slots) {
+ slots.reserve(static_cast<size_t>(state.live));
+ DorisVector<int32_t> stack;
+ stack.reserve(64);
+ int32_t node = state.root;
+ while (node != State::NO_SLOT || !stack.empty()) {
+ while (node != State::NO_SLOT) {
+ stack.push_back(node);
+ node = state.slots[node].left;
+ }
+ node = stack.back();
+ stack.pop_back();
+ // Only live slots are reachable from the root; an evicted slot has no row count.
+ DCHECK_GE(state.slots[node].count, 0);
+ slots.push_back(node);
+ node = state.slots[node].right;
+ }
+ DCHECK_EQ(static_cast<int64_t>(slots.size()), state.live);
+ }
+
+ /// Rebuilds the tree from `values`, already in the aggregate's order, and the matching counts.
+ /// The treap is built in O(size) as a Cartesian tree over the fresh priorities.
+ static void assign_sorted(State& state, ValueStore values, DorisVector<int64_t> counts) {
+ const size_t size = counts.size();
+ DCHECK_EQ(values.size(), size);
+ state.values.swap(values);
+ state.slots.assign(size, Slot {});
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].count = counts[slot];
+ }
+ state.live = static_cast<int64_t>(size);
+ state.dead = 0;
+ DorisVector<int32_t> stack;
+ stack.reserve(size);
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].priority = next_priority(state);
+ int32_t last = State::NO_SLOT;
+ while (!stack.empty() &&
+ state.slots[stack.back()].priority < state.slots[slot].priority) {
+ last = stack.back();
+ stack.pop_back();
+ }
+ state.slots[slot].left = last;
+ if (!stack.empty()) {
+ state.slots[stack.back()].right = static_cast<int32_t>(slot);
+ }
+ stack.push_back(static_cast<int32_t>(slot));
+ }
+ state.root = stack.empty() ? State::NO_SLOT : stack.front();
+ refresh_worst(state);
+ }
+
+ /// Reclaims the slots evicted since the last compaction. The O(k) rebuild is charged to the
+ /// k/2 evictions that produced the dead slots, so it stays O(1) amortized per row.
+ static void compact(State& state) {
+ DorisVector<int32_t> slots;
+ collect_in_order(state, slots);
+ ValueStore values;
+ values.reserve(slots.size());
+ DorisVector<int64_t> counts;
+ counts.reserve(slots.size());
+ for (int32_t slot : slots) {
+ values.append(state.values.at(slot));
+ counts.push_back(state.slots[slot].count);
+ }
+ assign_sorted(state, std::move(values), std::move(counts));
+ }
+
+ static void check_k(State& state, const IColumn& count_column, size_t row_num) {
+ const IColumn* nested = &count_column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&count_column)) {
+ if (UNLIKELY(nullable->is_null_at(row_num))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-null "
+ "constant");
+ }
+ nested = &nullable->get_nested_column();
+ }
+ int64_t value =
+ assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(*nested).get_element(
+ row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-negative "
+ "constant, but got {}",
+ value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "min_topk/max_topk value count overflow");
+ }
+ return result;
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(State& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (state.live == 0) {
+ state.root = append_slot(state, column, row_num, 1);
+ state.worst = state.root;
+ return;
+ }
+ int cmp = compare_values(value_at(column, row_num), state.values.at(0));
+ if (cmp == 0) {
+ state.slots[0].count = checked_add(state.slots[0].count, 1);
+ } else if (comes_before(cmp)) {
+ // The new value is better: replace the tracked extreme. The slot and the
+ // single-node tree stay in place, only the stored value changes.
+ state.values.clear();
+ state.values.append(value_at(column, row_num));
+ state.slots[0].count = 1;
+ }
+ return;
+ }
+
+ // Dead slots are reclaimed before they can outnumber the live ones, and evicted string
+ // payload before it dwarfs the live one; see compact().
+ if (state.dead > state.live || state.values.dead_bytes() > state.values.live_bytes()) {
+ compact(state);
+ }
+
+ if (state.live >= state.k) {
+ // A full list rejects or absorbs most rows: comparing with the eviction candidate
+ // costs one comparison, while the tree descent would cost log2(k).
+ int cmp = compare_values(value_at(column, row_num), state.values.at(state.worst));
+ if (cmp == 0) {
+ state.slots[state.worst].count = checked_add(state.slots[state.worst].count, 1);
+ return;
+ }
+ if (is_worse(cmp)) {
+ return;
+ }
+ }
+
+ int32_t inserted = State::NO_SLOT;
+ state.root = add_or_increment(state, state.root, column, row_num, &inserted);
+ if (inserted == State::NO_SLOT) {
+ return;
+ }
+ if (state.live <= state.k) {
+ // Not over capacity: the new entry competes with the cached worst for the eviction
+ // candidate role.
+ if (state.worst == State::NO_SLOT ||
+ is_worse(compare_slots(state, inserted, state.worst))) {
+ state.worst = inserted;
+ }
+ return;
+ }
+ // The list was full and the value is new. It beat the cached worst above, so it cannot be
+ // the worst itself; the old worst is still the rightmost entry and leaves the list.
+ int32_t removed = State::NO_SLOT;
+ state.root = erase_worst(state, state.root, &removed);
+ DCHECK_EQ(removed, state.worst);
+ state.values.on_evict(static_cast<size_t>(removed));
+ state.slots[removed].count = State::DEAD_COUNT;
+ --state.live;
+ ++state.dead;
+ refresh_worst(state);
+ }
+
+ /// k == 0: the state holds at most the extreme value and its count.
+ static void merge_extreme(State& state, const State& other) {
+ if (other.live == 0) {
+ return;
+ }
+ if (state.live == 0) {
+ state.root = append_slot(state, other.values.at(0), other.slots[0].count);
+ state.worst = state.root;
+ return;
+ }
+ int cmp = compare_values(state.values.at(0), other.values.at(0));
+ if (cmp == 0) {
+ state.slots[0].count = checked_add(state.slots[0].count, other.slots[0].count);
+ } else if (is_worse(cmp)) {
+ state.values.clear();
+ state.values.append(other.values.at(0));
+ state.slots[0].count = other.slots[0].count;
+ }
+ }
+
+ /// Merges two ordered lists, summing the counts of equal values and keeping the k best.
+ ///
+ /// The merge stops as soon as k entries are produced. Both inputs are best-first and hold
+ /// distinct values, and equal values have already been combined, so every remaining value is
+ /// worse than all retained ones: it can neither enter the result nor overflow it. Stopping
+ /// here also avoids materializing a tail that would be truncated away.
+ static void merge_values(State& state, const State& other) {
+ if (other.live == 0) {
+ // The other partial state saw no non-NULL row.
+ return;
+ }
+ if (state.live >= state.k) {
+ // A full state already holds the k best values. If every value of the other state is
+ // worse than the worst one retained here, the merge cannot change the result, so skip
+ // the collect/rebuild. An equal value would raise a count, hence the strict test.
+ // `_state`/`_merge`/`_union` merges one singleton per row, and this turns each one
+ // into a single comparison once the list is full.
+ int cmp =
+ compare_values(other.values.at(best_slot(other)), state.values.at(state.worst));
+ if (is_worse(cmp)) {
+ return;
+ }
+ }
+ DorisVector<int32_t> lhs_slots;
+ DorisVector<int32_t> rhs_slots;
+ collect_in_order(state, lhs_slots);
+ collect_in_order(other, rhs_slots);
+ const auto limit = static_cast<size_t>(state.k);
+ ValueStore merged_values;
+ merged_values.reserve(std::min(limit, lhs_slots.size() + rhs_slots.size()));
+ DorisVector<int64_t> merged_counts;
+ merged_counts.reserve(std::min(limit, lhs_slots.size() + rhs_slots.size()));
+ size_t lhs_index = 0;
+ size_t rhs_index = 0;
+ while (merged_counts.size() < limit &&
+ (lhs_index < lhs_slots.size() || rhs_index < rhs_slots.size())) {
+ if (rhs_index == rhs_slots.size()) {
+ merged_values.append(state.values.at(lhs_slots[lhs_index]));
+ merged_counts.push_back(state.slots[lhs_slots[lhs_index]].count);
+ ++lhs_index;
+ } else if (lhs_index == lhs_slots.size()) {
+ merged_values.append(other.values.at(rhs_slots[rhs_index]));
+ merged_counts.push_back(other.slots[rhs_slots[rhs_index]].count);
+ ++rhs_index;
+ } else {
+ ValueRef lhs_value = state.values.at(lhs_slots[lhs_index]);
+ ValueRef rhs_value = other.values.at(rhs_slots[rhs_index]);
+ int cmp = compare_values(lhs_value, rhs_value);
+ // Both lists are ordered best-first (ascending for MIN, descending for MAX).
+ const bool lhs_is_better = comes_before(cmp);
+ if (lhs_is_better) {
+ merged_values.append(lhs_value);
+ merged_counts.push_back(state.slots[lhs_slots[lhs_index]].count);
+ ++lhs_index;
+ } else if (cmp != 0) {
+ // Not better and not equal: the right-hand value is better.
+ merged_values.append(rhs_value);
+ merged_counts.push_back(other.slots[rhs_slots[rhs_index]].count);
+ ++rhs_index;
+ } else {
+ merged_values.append(lhs_value);
+ merged_counts.push_back(checked_add(state.slots[lhs_slots[lhs_index]].count,
+ other.slots[rhs_slots[rhs_index]].count));
+ ++lhs_index;
+ ++rhs_index;
+ }
+ }
+ }
+ assign_sorted(state, std::move(merged_values), std::move(merged_counts));
Review Comment:
Completed in 41e89a60347: a single-entry RHS no longer goes through the collect/rebuild at all. It now runs the add path: an already retained value (including an interior one) bumps its count, otherwise the value is inserted and the old worst is evicted, in O(log k); the merge-side eviction also runs the compaction trigger so dead slots and payload stay bounded. MergeStringStates covers the interior singleton ({a,b,c} + {b} at k=3) and the insert-and-evict singleton.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6475342542
# BE Regression && UT Coverage Report
Increment line coverage `96.15% (574/597)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_5c8beb3b4bc041961f08e39e9b5aae6284a64cd6_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_5c8beb3b4bc041961f08e39e9b5aae6284a64cd6_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.56% (36377/46902) |
| Line Coverage | 62.79% (410902/654414) |
| Region Coverage | 59.16% (347256/586971) |
| Branch Coverage | 60.15% (161087/267794) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4401309789
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,786 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ // The payload is proportional to the slot count, which the caller already compares.
+ size_t live_bytes() const { return 0; }
+ size_t dead_bytes() const { return 0; }
+ void on_evict(size_t /*slot*/) {}
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ size_t offset = 0;
+ size_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ /// PaddedPODArray, not PODArray: comparisons go through
+ /// memcmp_small_allow_overflow15, which loads a full 16-byte vector even for a one-byte
+ /// string, the same tail allowance ColumnString's chars buffer provides.
+ PaddedPODArray<char, 32> bytes;
+ DorisVector<Span> spans;
+ // Sum of the live slots' payload and of the payload evicted since the last compaction. The
+ // slot counters alone cannot bound the memory: replacing 1024 1 MiB strings with short ones
+ // leaves dead == live by slot count while ~1 GiB of discarded bytes stay in the arena.
+ size_t live_payload_bytes = 0;
+ size_t dead_payload_bytes = 0;
+
+ size_t size() const { return spans.size(); }
+ size_t live_bytes() const { return live_payload_bytes; }
+ size_t dead_bytes() const { return dead_payload_bytes; }
+ void reserve(size_t n) { spans.reserve(n); }
+ void clear() {
+ bytes.clear();
+ spans.clear();
+ live_payload_bytes = 0;
+ dead_payload_bytes = 0;
+ }
+ StringRef at(size_t slot) const {
+ const Span& span = spans[slot];
+ return StringRef(bytes.data() + span.offset, span.length);
+ }
+ void append(StringRef value) {
+ spans.push_back({bytes.size(), value.size});
+ live_payload_bytes += value.size;
+ if (value.size != 0) {
+ bytes.insert(value.data, value.data + value.size);
+ }
+ }
+ /// Called when `slot` is evicted: its payload stays in the arena until the next compaction.
+ void on_evict(size_t slot) {
+ live_payload_bytes -= spans[slot].length;
+ dead_payload_bytes += spans[slot].length;
+ }
+ void swap(StringTopKValueStore& other) noexcept {
+ bytes.swap(other.bytes);
+ spans.swap(other.spans);
+ std::swap(live_payload_bytes, other.live_payload_bytes);
+ std::swap(dead_payload_bytes, other.dead_payload_bytes);
+ }
+};
+
+/**
+ * State shared by min_topk and max_topk, for one value type.
+ *
+ * The distinct values sit in slots (see the stores above), with one Slot record per value carrying
+ * the row count and the links of a treap ordered best-first (ascending values for MIN, descending
+ * for MAX). A row costs one O(log k) descent: it either finds its value and bumps the count or
+ * links a new slot at the leaf. A sorted vector would shift O(k) elements for the same insert.
+ *
+ * The treap stores explicit child links because, unlike a heap, its shape is not the complete tree
+ * that an array index can describe: rotations move entries, and the in-order sequence is the value
+ * order rather than the arrival order. A heap could evict and append in O(log k), but it cannot
+ * find an existing value without a hash index or a linear scan, and it cannot produce the sorted
+ * result without sorting. The ordered tree is the structure that keeps find, insert and eviction
+ * all at O(log k) with comparisons only.
+ *
+ * An entry that is evicted keeps its slot and value until the next compaction. The tree never
+ * points at a dead slot.
+ *
+ * With k == 0 the state holds at most the single extreme value and its row count; the caller
+ * derives truncation from the list length, so the state carries no separate bound.
+ */
+template <PrimitiveType T>
+struct AggregateTopKCountData {
+ using CppType = typename PrimitiveTypeTraits<T>::CppType;
+ static constexpr bool IS_STRING = is_string_type(T);
+ using ValueStore =
+ std::conditional_t<IS_STRING, StringTopKValueStore, FixedTopKValueStore<CppType>>;
+ using ValueRef = std::conditional_t<IS_STRING, StringRef, const CppType&>;
+
+ static constexpr int32_t NO_SLOT = -1;
+ // Row counts are never negative, so this cannot collide with a live entry.
+ static constexpr int64_t DEAD_COUNT = -1;
+
+ /// One slot's metadata; the value itself lives in `values` at the same index.
+ struct Slot {
+ int64_t count = 0;
+ int32_t left = NO_SLOT;
+ int32_t right = NO_SLOT;
+ uint32_t priority = 0;
+ };
+
+ bool has_k = false;
+ int64_t k = 0;
+ // Indexed by slot; an evicted slot stays as a dead entry until compaction.
+ ValueStore values;
+ DorisVector<Slot> slots;
+ int32_t root = NO_SLOT;
+ // The last live slot in the aggregate's order: the value an incoming row has to beat to enter
+ // a full list. Cached so that rejecting a worse value costs one comparison, not a descent.
+ int32_t worst = NO_SLOT;
+ int64_t live = 0;
+ // Slots evicted since the last compaction.
+ int64_t dead = 0;
+ uint64_t rng = 0x243F6A8885A308D3ULL;
+
+ AggregateTopKCountData() = default;
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * min_topk(x, k) / max_topk(x, k).
+ *
+ * Result: ARRAY<STRUCT<value T, count BIGINT>>. NULL inputs are ignored: add validates k for
+ * every row and then skips a NULL x, so the aggregate deliberately does not use the
+ * nullable-argument wrapper. The array is ordered by value (ascending for MIN), holds at most k
+ * distinct values with count > 0, and carries no truncation flag: the caller infers it from the
+ * length (a list of exactly k entries is treated as possibly truncated).
+ *
+ * With k = 0 the array holds at most the extreme value and its row count, which lets the
+ * append-only IVM path compute MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopK final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData<T>,
+ AggregateFunctionTopK<T, IsMin>, false>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using State = AggregateTopKCountData<T>;
+ using Base = IAggregateFunctionDataHelper<State, AggregateFunctionTopK<T, IsMin>, false>;
+ using Slot = typename State::Slot;
+ using ValueStore = typename State::ValueStore;
+ using ValueRef = typename State::ValueRef;
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ static constexpr bool IS_STRING = State::IS_STRING;
+ static constexpr int32_t NO_SLOT = State::NO_SLOT;
+
+ explicit AggregateFunctionTopK(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "min_topk" : "max_topk"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(remove_nullable(this->argument_types[0]));
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is read for every row, including rows where x is NULL: an evaluated count that is
+ // NULL or out of range must fail the query instead of being skipped like a NULL value.
+ check_k(state, *columns[1], row_num);
+ const IColumn* value_column = columns[0];
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(value_column)) {
+ if (nullable->is_null_at(row_num)) {
+ return;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ add_value(state, *value_column, row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw any row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else if (UNLIKELY(state.k != other.k)) {
+ // Ordinary partial/final aggregation always shares the same constant k, but the
+ // public _state/_merge/_union combinators do not encode k in the AggState type, so a
+ // UNION ALL of states built with different literal k can reach this merge.
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} states have incompatible counts: {} vs {}", get_name(), state.k,
+ other.k);
+ }
+ if (state.k == 0) {
+ merge_extreme(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ // The slots are in arrival order and may include dead ones, while the serialized form is
+ // the sorted (value, count) list the reader rebuilds the tree from.
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ auto sorted_values = remove_nullable(this->argument_types[0])->create_column();
+ sorted_values->reserve(order.size());
+ DorisVector<int64_t> sorted_counts;
+ sorted_counts.reserve(order.size());
+ for (int32_t slot : order) {
+ insert_value(*sorted_values, state.values.at(slot));
+ sorted_counts.push_back(state.slots[slot].count);
+ }
+ auto size = static_cast<int64_t>(sorted_counts.size());
+ buf.write_binary(size);
+ write_column(buf, *sorted_values);
+ if (size != 0) {
+ // An empty list keeps an empty vector, whose data() is null; passing it to memcpy is
+ // undefined even for a zero length.
+ buf.write(reinterpret_cast<const char*>(sorted_counts.data()),
+ sorted_counts.size() * sizeof(int64_t));
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ int64_t size = 0;
+ buf.read_binary(size);
+ auto values = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(values->size()), size);
+ DorisVector<int64_t> counts(static_cast<size_t>(size));
+ if (size != 0) {
+ buf.read(reinterpret_cast<char*>(counts.data()), size * sizeof(int64_t));
+ }
+ ValueStore value_store;
+ value_store.reserve(static_cast<size_t>(size));
+ for (int64_t row = 0; row != size; ++row) {
+ value_store.append(value_at(*values, static_cast<size_t>(row)));
+ }
+ assign_sorted(state, std::move(value_store), std::move(counts));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_array = assert_cast<ColumnArray&>(to);
+ // The array element and both fields are nullable in the planned type, but a produced entry
+ // never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(result_array.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = assert_cast<ColumnNullable&>(pair_column.get_column(0));
+ auto& count_column = assert_cast<ColumnNullable&>(pair_column.get_column(1));
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ for (int32_t slot : order) {
+ insert_value(value_column.get_nested_column(), state.values.at(slot));
+ value_column.get_null_map_data().push_back(0);
+ assert_cast<ColumnInt64&>(count_column.get_nested_column())
+ .get_data()
+ .push_back(state.slots[slot].count);
+ count_column.get_null_map_data().push_back(0);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ result_array.get_offsets().push_back(pair_column.size());
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ // Both fields are nullable in the planned type: a consumer reads a field through
+ // element_at, which is always nullable, and named_struct then records nullable fields.
+ // Produced entries never contain a NULL field.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(value_type),
+ make_nullable(std::make_shared<DataTypeInt64>())},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ return std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ }
+
+private:
+ /// The value of the row in `column`. The returned view is valid while the column is alive.
+ static ValueRef value_at(const IColumn& column, size_t row_num) {
+ if constexpr (IS_STRING) {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column).get_data_at(
+ row_num);
+ } else {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column)
+ .get_data()[row_num];
+ }
+ }
+
+ /// Exactly the comparison the value column would apply: CompareFloat for FLOAT/DOUBLE through
+ /// Compare::compare (the column uses the same helper), the same memcmp helper ColumnString
+ /// uses for strings, and the column's scale-independent ordering for every other type.
+ static int compare_values(ValueRef lhs, ValueRef rhs) {
+ if constexpr (IS_STRING) {
+ return memcmp_small_allow_overflow15(
+ reinterpret_cast<const uint8_t*>(lhs.data), lhs.size,
+ reinterpret_cast<const uint8_t*>(rhs.data), rhs.size);
+ } else {
+ return Compare::compare(lhs, rhs);
+ }
+ }
+
+ static void insert_value(IColumn& column, ValueRef value) {
+ if constexpr (IS_STRING) {
+ assert_cast<ColumnString&>(column).insert_data(value.data, value.size);
+ } else {
+ assert_cast<ColumnType&>(column).get_data().push_back(value);
+ }
+ }
+
+ /// True when `cmp` puts the left operand before the right one in the aggregate's order.
+ static bool comes_before(int cmp) { return IsMin ? cmp < 0 : cmp > 0; }
+
+ /// True when `cmp`, which compares a value with the worst retained one, puts it after it: the
+ /// value is worse and cannot enter the list.
+ static bool is_worse(int cmp) { return IsMin ? cmp > 0 : cmp < 0; }
+
+ /// splitmix64. The priorities only have to be independent enough to keep the treap balanced;
+ /// a per-state sequence keeps the shape reproducible for the same insertion order.
+ static uint32_t next_priority(State& state) {
+ uint64_t z = (state.rng += 0x9E3779B97F4A7C15ULL);
+ z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9ULL;
+ z = (z ^ (z >> 27)) * 0x94D049BB133111EBULL;
+ return static_cast<uint32_t>(z ^ (z >> 31));
+ }
+
+ static int compare_slots(const State& state, int32_t lhs, int32_t rhs) {
+ return compare_values(state.values.at(lhs), state.values.at(rhs));
+ }
+
+ /// Appends (value, count) as the next slot.
+ static int32_t append_slot(State& state, ValueRef value, int64_t count) {
+ auto slot = static_cast<int32_t>(state.slots.size());
+ DCHECK_LT(slot, std::numeric_limits<int32_t>::max());
+ state.values.append(value);
+ state.slots.push_back(Slot {count, State::NO_SLOT, State::NO_SLOT, next_priority(state)});
+ ++state.live;
+ return slot;
+ }
+
+ static int32_t append_slot(State& state, const IColumn& column, size_t row_num, int64_t count) {
+ return append_slot(state, value_at(column, row_num), count);
+ }
+
+ /// Rotates the subtree rooted at `node` right: the left child becomes the root and `node` its
+ /// right child. The in-order sequence is unchanged, so the treap stays ordered.
+ static int32_t rotate_right(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].left;
+ state.slots[node].left = state.slots[pivot].right;
+ state.slots[pivot].right = node;
+ return pivot;
+ }
+
+ static int32_t rotate_left(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].right;
+ state.slots[node].right = state.slots[pivot].left;
+ state.slots[pivot].left = node;
+ return pivot;
+ }
+
+ /// One descent that performs the whole per-row update: an equal value bumps its count and
+ /// reports no insertion; a missing value is linked as a new slot at the leaf and rotated up
+ /// while the recursion unwinds. `inserted_slot` receives the new slot, NO_SLOT on a hit.
+ static int32_t add_or_increment(State& state, int32_t node, const IColumn& column,
+ size_t row_num, int32_t* inserted_slot) {
+ if (node == State::NO_SLOT) {
+ *inserted_slot = append_slot(state, column, row_num, 1);
+ return *inserted_slot;
+ }
+ int cmp = compare_values(state.values.at(node), value_at(column, row_num));
+ if (cmp == 0) {
+ state.slots[node].count = checked_add(state.slots[node].count, 1);
+ return node;
+ }
+ if (comes_before(cmp)) {
+ // The stored value is the better one, so the input belongs in the right subtree.
+ state.slots[node].right = add_or_increment(state, state.slots[node].right, column,
+ row_num, inserted_slot);
+ if (state.slots[state.slots[node].right].priority > state.slots[node].priority) {
+ return rotate_left(state, node);
+ }
+ } else {
+ state.slots[node].left =
+ add_or_increment(state, state.slots[node].left, column, row_num, inserted_slot);
+ if (state.slots[state.slots[node].left].priority > state.slots[node].priority) {
+ return rotate_right(state, node);
+ }
+ }
+ return node;
+ }
+
+ /// Removes the last entry in the aggregate's order (the largest value for MIN) and returns the
+ /// new subtree root. That entry has no right child, so its left subtree takes its place; the
+ /// child's priority is below the parent's, so no rotation is needed.
+ static int32_t erase_worst(State& state, int32_t node, int32_t* removed) {
+ if (state.slots[node].right == State::NO_SLOT) {
+ *removed = node;
+ return state.slots[node].left;
+ }
+ state.slots[node].right = erase_worst(state, state.slots[node].right, removed);
+ return node;
+ }
+
+ static void refresh_worst(State& state) {
+ int32_t node = state.root;
+ while (node != State::NO_SLOT && state.slots[node].right != State::NO_SLOT) {
+ node = state.slots[node].right;
+ }
+ state.worst = node;
+ }
+
+ /// The first live slot in the aggregate's order, i.e. the best value the state holds.
+ static int32_t best_slot(const State& state) {
+ int32_t node = state.root;
+ while (node != State::NO_SLOT && state.slots[node].left != State::NO_SLOT) {
+ node = state.slots[node].left;
+ }
+ return node;
+ }
+
+ /// Slots in the aggregate's order, i.e. the order of the result array.
+ static void collect_in_order(const State& state, DorisVector<int32_t>& slots) {
+ slots.reserve(static_cast<size_t>(state.live));
+ DorisVector<int32_t> stack;
+ stack.reserve(64);
+ int32_t node = state.root;
+ while (node != State::NO_SLOT || !stack.empty()) {
+ while (node != State::NO_SLOT) {
+ stack.push_back(node);
+ node = state.slots[node].left;
+ }
+ node = stack.back();
+ stack.pop_back();
+ // Only live slots are reachable from the root; an evicted slot has no row count.
+ DCHECK_GE(state.slots[node].count, 0);
+ slots.push_back(node);
+ node = state.slots[node].right;
+ }
+ DCHECK_EQ(static_cast<int64_t>(slots.size()), state.live);
+ }
+
+ /// Rebuilds the tree from `values`, already in the aggregate's order, and the matching counts.
+ /// The treap is built in O(size) as a Cartesian tree over the fresh priorities.
+ static void assign_sorted(State& state, ValueStore values, DorisVector<int64_t> counts) {
+ const size_t size = counts.size();
+ DCHECK_EQ(values.size(), size);
+ state.values.swap(values);
+ state.slots.assign(size, Slot {});
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].count = counts[slot];
+ }
+ state.live = static_cast<int64_t>(size);
+ state.dead = 0;
+ DorisVector<int32_t> stack;
+ stack.reserve(size);
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].priority = next_priority(state);
+ int32_t last = State::NO_SLOT;
+ while (!stack.empty() &&
+ state.slots[stack.back()].priority < state.slots[slot].priority) {
+ last = stack.back();
+ stack.pop_back();
+ }
+ state.slots[slot].left = last;
+ if (!stack.empty()) {
+ state.slots[stack.back()].right = static_cast<int32_t>(slot);
+ }
+ stack.push_back(static_cast<int32_t>(slot));
+ }
+ state.root = stack.empty() ? State::NO_SLOT : stack.front();
+ refresh_worst(state);
+ }
+
+ /// Reclaims the slots evicted since the last compaction. The O(k) rebuild is charged to the
+ /// k/2 evictions that produced the dead slots, so it stays O(1) amortized per row.
+ static void compact(State& state) {
+ DorisVector<int32_t> slots;
+ collect_in_order(state, slots);
+ ValueStore values;
+ values.reserve(slots.size());
+ DorisVector<int64_t> counts;
+ counts.reserve(slots.size());
+ for (int32_t slot : slots) {
+ values.append(state.values.at(slot));
+ counts.push_back(state.slots[slot].count);
+ }
+ assign_sorted(state, std::move(values), std::move(counts));
+ }
+
+ static void check_k(State& state, const IColumn& count_column, size_t row_num) {
+ const IColumn* nested = &count_column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&count_column)) {
+ if (UNLIKELY(nullable->is_null_at(row_num))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-null "
+ "constant");
+ }
+ nested = &nullable->get_nested_column();
+ }
+ int64_t value =
+ assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(*nested).get_element(
+ row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-negative "
+ "constant, but got {}",
+ value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "min_topk/max_topk value count overflow");
+ }
+ return result;
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(State& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (state.live == 0) {
+ state.root = append_slot(state, column, row_num, 1);
+ state.worst = state.root;
+ return;
+ }
+ int cmp = compare_values(value_at(column, row_num), state.values.at(0));
+ if (cmp == 0) {
+ state.slots[0].count = checked_add(state.slots[0].count, 1);
+ } else if (comes_before(cmp)) {
+ // The new value is better: replace the tracked extreme. The slot and the
+ // single-node tree stay in place, only the stored value changes.
+ state.values.clear();
+ state.values.append(value_at(column, row_num));
+ state.slots[0].count = 1;
+ }
+ return;
+ }
+
+ // Dead slots are reclaimed before they can outnumber the live ones, and evicted string
+ // payload before it dwarfs the live one; see compact().
+ if (state.dead > state.live || state.values.dead_bytes() > state.values.live_bytes()) {
+ compact(state);
+ }
+
+ if (state.live >= state.k) {
+ // A full list rejects or absorbs most rows: comparing with the eviction candidate
+ // costs one comparison, while the tree descent would cost log2(k).
+ int cmp = compare_values(value_at(column, row_num), state.values.at(state.worst));
+ if (cmp == 0) {
+ state.slots[state.worst].count = checked_add(state.slots[state.worst].count, 1);
+ return;
+ }
+ if (is_worse(cmp)) {
+ return;
+ }
+ }
+
+ int32_t inserted = State::NO_SLOT;
+ state.root = add_or_increment(state, state.root, column, row_num, &inserted);
+ if (inserted == State::NO_SLOT) {
+ return;
+ }
+ if (state.live <= state.k) {
+ // Not over capacity: the new entry competes with the cached worst for the eviction
+ // candidate role.
+ if (state.worst == State::NO_SLOT ||
+ is_worse(compare_slots(state, inserted, state.worst))) {
+ state.worst = inserted;
+ }
+ return;
+ }
+ // The list was full and the value is new. It beat the cached worst above, so it cannot be
+ // the worst itself; the old worst is still the rightmost entry and leaves the list.
+ int32_t removed = State::NO_SLOT;
+ state.root = erase_worst(state, state.root, &removed);
+ DCHECK_EQ(removed, state.worst);
+ state.values.on_evict(static_cast<size_t>(removed));
+ state.slots[removed].count = State::DEAD_COUNT;
+ --state.live;
+ ++state.dead;
+ refresh_worst(state);
+ }
+
+ /// k == 0: the state holds at most the extreme value and its count.
+ static void merge_extreme(State& state, const State& other) {
+ if (other.live == 0) {
+ return;
+ }
+ if (state.live == 0) {
+ state.root = append_slot(state, other.values.at(0), other.slots[0].count);
+ state.worst = state.root;
+ return;
+ }
+ int cmp = compare_values(state.values.at(0), other.values.at(0));
+ if (cmp == 0) {
+ state.slots[0].count = checked_add(state.slots[0].count, other.slots[0].count);
+ } else if (is_worse(cmp)) {
+ state.values.clear();
+ state.values.append(other.values.at(0));
+ state.slots[0].count = other.slots[0].count;
+ }
+ }
+
+ /// Merges two ordered lists, summing the counts of equal values and keeping the k best.
+ ///
+ /// The merge stops as soon as k entries are produced. Both inputs are best-first and hold
+ /// distinct values, and equal values have already been combined, so every remaining value is
+ /// worse than all retained ones: it can neither enter the result nor overflow it. Stopping
+ /// here also avoids materializing a tail that would be truncated away.
+ static void merge_values(State& state, const State& other) {
+ if (other.live == 0) {
+ // The other partial state saw no non-NULL row.
+ return;
+ }
+ if (state.live >= state.k) {
+ // A full state already holds the k best values. If every value of the other state is
+ // worse than the worst one retained here, the merge cannot change the result, so skip
+ // the collect/rebuild. An equal value would raise a count, hence the strict test.
+ // `_state`/`_merge`/`_union` merges one singleton per row, and this turns each one
+ // into a single comparison once the list is full.
+ int cmp =
+ compare_values(other.values.at(best_slot(other)), state.values.at(state.worst));
+ if (is_worse(cmp)) {
+ return;
+ }
+ }
+ DorisVector<int32_t> lhs_slots;
+ DorisVector<int32_t> rhs_slots;
+ collect_in_order(state, lhs_slots);
+ collect_in_order(other, rhs_slots);
+ const auto limit = static_cast<size_t>(state.k);
+ ValueStore merged_values;
+ merged_values.reserve(std::min(limit, lhs_slots.size() + rhs_slots.size()));
+ DorisVector<int64_t> merged_counts;
+ merged_counts.reserve(std::min(limit, lhs_slots.size() + rhs_slots.size()));
+ size_t lhs_index = 0;
+ size_t rhs_index = 0;
+ while (merged_counts.size() < limit &&
+ (lhs_index < lhs_slots.size() || rhs_index < rhs_slots.size())) {
+ if (rhs_index == rhs_slots.size()) {
+ merged_values.append(state.values.at(lhs_slots[lhs_index]));
+ merged_counts.push_back(state.slots[lhs_slots[lhs_index]].count);
+ ++lhs_index;
+ } else if (lhs_index == lhs_slots.size()) {
+ merged_values.append(other.values.at(rhs_slots[rhs_index]));
+ merged_counts.push_back(other.slots[rhs_slots[rhs_index]].count);
+ ++rhs_index;
+ } else {
+ ValueRef lhs_value = state.values.at(lhs_slots[lhs_index]);
+ ValueRef rhs_value = other.values.at(rhs_slots[rhs_index]);
+ int cmp = compare_values(lhs_value, rhs_value);
+ // Both lists are ordered best-first (ascending for MIN, descending for MAX).
+ const bool lhs_is_better = comes_before(cmp);
+ if (lhs_is_better) {
+ merged_values.append(lhs_value);
+ merged_counts.push_back(state.slots[lhs_slots[lhs_index]].count);
+ ++lhs_index;
+ } else if (cmp != 0) {
+ // Not better and not equal: the right-hand value is better.
+ merged_values.append(rhs_value);
+ merged_counts.push_back(other.slots[rhs_slots[rhs_index]].count);
+ ++rhs_index;
+ } else {
+ merged_values.append(lhs_value);
+ merged_counts.push_back(checked_add(state.slots[lhs_slots[lhs_index]].count,
+ other.slots[rhs_slots[rhs_index]].count));
+ ++lhs_index;
+ ++rhs_index;
+ }
+ }
+ }
+ assign_sorted(state, std::move(merged_values), std::move(merged_counts));
Review Comment:
Residual case closed in 5c8beb3b4bc: the single-entry RHS path now runs for any state size, not only when live >= k. A retained value (interior included) bumps its count, a new value is inserted, and the worst is evicted only when the list was full; the below-k append also keeps the cached worst in sync, covered by MergeIntoAnUnfullStateKeepsTheWorstCache.
##########
be/src/exprs/aggregate/aggregate_function_topk.h:
##########
@@ -0,0 +1,741 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <type_traits>
+#include <utility>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/custom_allocator.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/memcmp_small.h"
+#include "core/pod_array.h"
+#include "core/string_buffer.hpp"
+#include "core/string_ref.h"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// The values of one group, in arrival order, one entry per slot.
+///
+/// They deliberately do not live in a Doris column: a column's container allocates a 4 KiB initial
+/// chunk (PaddedPODArray) as soon as it holds one value, so one column per group costs ~4 KiB per
+/// group regardless of k. That dominates the memory of a GROUP BY with millions of groups, while
+/// these stores grow with the number of values actually retained.
+template <typename T>
+struct FixedTopKValueStore {
+ DorisVector<T> values;
+
+ size_t size() const { return values.size(); }
+ void reserve(size_t n) { values.reserve(n); }
+ void clear() { values.clear(); }
+ const T& at(size_t slot) const { return values[slot]; }
+ void append(const T& value) { values.push_back(value); }
+ void swap(FixedTopKValueStore& other) noexcept { values.swap(other.values); }
+};
+
+/// A StringRef cannot be stored: the arena reallocates and would invalidate the pointers. A slot
+/// keeps an offset/length into the bytes instead and materializes a view on access.
+struct StringTopKValueStore {
+ struct Span {
+ uint32_t offset = 0;
+ uint32_t length = 0;
+ };
+
+ /// The initial allocation is small on purpose; see the comment on FixedTopKValueStore.
+ PODArray<char, 32> bytes;
+ DorisVector<Span> spans;
+
+ size_t size() const { return spans.size(); }
+ void reserve(size_t n) { spans.reserve(n); }
+ void clear() {
+ bytes.clear();
+ spans.clear();
+ }
+ StringRef at(size_t slot) const {
+ const Span& span = spans[slot];
+ return StringRef(bytes.data() + span.offset, span.length);
+ }
+ void append(StringRef value) {
+ DCHECK_LE(bytes.size() + value.size, std::numeric_limits<uint32_t>::max());
+ DCHECK_LE(value.size, std::numeric_limits<uint32_t>::max());
+ spans.push_back({static_cast<uint32_t>(bytes.size()), static_cast<uint32_t>(value.size)});
+ if (value.size != 0) {
+ bytes.insert(value.data, value.data + value.size);
+ }
+ }
+ void swap(StringTopKValueStore& other) noexcept {
+ bytes.swap(other.bytes);
+ spans.swap(other.spans);
+ }
+};
+
+/**
+ * State shared by min_topk and max_topk, for one value type.
+ *
+ * The distinct values sit in slots (see the stores above), with one Slot record per value carrying
+ * the row count and the links of a treap ordered best-first (ascending values for MIN, descending
+ * for MAX). A row costs one O(log k) descent: it either finds its value and bumps the count or
+ * links a new slot at the leaf. A sorted vector would shift O(k) elements for the same insert.
+ *
+ * The treap stores explicit child links because, unlike a heap, its shape is not the complete tree
+ * that an array index can describe: rotations move entries, and the in-order sequence is the value
+ * order rather than the arrival order. A heap could evict and append in O(log k), but it cannot
+ * find an existing value without a hash index or a linear scan, and it cannot produce the sorted
+ * result without sorting. The ordered tree is the structure that keeps find, insert and eviction
+ * all at O(log k) with comparisons only.
+ *
+ * An entry that is evicted keeps its slot and value until the next compaction. The tree never
+ * points at a dead slot.
+ *
+ * With k == 0 the state holds at most the single extreme value and its row count; the caller
+ * derives truncation from the list length, so the state carries no separate bound.
+ */
+template <PrimitiveType T>
+struct AggregateTopKCountData {
+ using CppType = typename PrimitiveTypeTraits<T>::CppType;
+ static constexpr bool IS_STRING = is_string_type(T);
+ using ValueStore =
+ std::conditional_t<IS_STRING, StringTopKValueStore, FixedTopKValueStore<CppType>>;
+ using ValueRef = std::conditional_t<IS_STRING, StringRef, const CppType&>;
+
+ static constexpr int32_t NO_SLOT = -1;
+ // Row counts are never negative, so this cannot collide with a live entry.
+ static constexpr int64_t DEAD_COUNT = -1;
+
+ /// One slot's metadata; the value itself lives in `values` at the same index.
+ struct Slot {
+ int64_t count = 0;
+ int32_t left = NO_SLOT;
+ int32_t right = NO_SLOT;
+ uint32_t priority = 0;
+ };
+
+ bool has_k = false;
+ int64_t k = 0;
+ // Indexed by slot; an evicted slot stays as a dead entry until compaction.
+ ValueStore values;
+ DorisVector<Slot> slots;
+ int32_t root = NO_SLOT;
+ // The last live slot in the aggregate's order: the value an incoming row has to beat to enter
+ // a full list. Cached so that rejecting a worse value costs one comparison, not a descent.
+ int32_t worst = NO_SLOT;
+ int64_t live = 0;
+ // Slots evicted since the last compaction.
+ int64_t dead = 0;
+ uint64_t rng = 0x243F6A8885A308D3ULL;
+
+ AggregateTopKCountData() = default;
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * min_topk(x, k) / max_topk(x, k).
+ *
+ * Result: ARRAY<STRUCT<value T, count BIGINT>>. NULL inputs are ignored: add validates k for
+ * every row and then skips a NULL x, so the aggregate deliberately does not use the
+ * nullable-argument wrapper. The array is ordered by value (ascending for MIN), holds at most k
+ * distinct values with count > 0, and carries no truncation flag: the caller infers it from the
+ * length (a list of exactly k entries is treated as possibly truncated).
+ *
+ * With k = 0 the array holds at most the extreme value and its row count, which lets the
+ * append-only IVM path compute MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopK final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData<T>,
+ AggregateFunctionTopK<T, IsMin>, false>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using State = AggregateTopKCountData<T>;
+ using Base = IAggregateFunctionDataHelper<State, AggregateFunctionTopK<T, IsMin>, false>;
+ using Slot = typename State::Slot;
+ using ValueStore = typename State::ValueStore;
+ using ValueRef = typename State::ValueRef;
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ static constexpr bool IS_STRING = State::IS_STRING;
+ static constexpr int32_t NO_SLOT = State::NO_SLOT;
+
+ explicit AggregateFunctionTopK(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "min_topk" : "max_topk"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(remove_nullable(this->argument_types[0]));
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is read for every row, including rows where x is NULL: an evaluated count that is
+ // NULL or out of range must fail the query instead of being skipped like a NULL value.
+ check_k(state, *columns[1], row_num);
+ const IColumn* value_column = columns[0];
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(value_column)) {
+ if (nullable->is_null_at(row_num)) {
+ return;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ add_value(state, *value_column, row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw any row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else if (UNLIKELY(state.k != other.k)) {
+ // Ordinary partial/final aggregation always shares the same constant k, but the
+ // public _state/_merge/_union combinators do not encode k in the AggState type, so a
+ // UNION ALL of states built with different literal k can reach this merge.
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "{} states have incompatible counts: {} vs {}", get_name(), state.k,
+ other.k);
+ }
+ if (state.k == 0) {
+ merge_extreme(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ // The slots are in arrival order and may include dead ones, while the serialized form is
+ // the sorted (value, count) list the reader rebuilds the tree from.
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ auto sorted_values = remove_nullable(this->argument_types[0])->create_column();
+ sorted_values->reserve(order.size());
+ DorisVector<int64_t> sorted_counts;
+ sorted_counts.reserve(order.size());
+ for (int32_t slot : order) {
+ insert_value(*sorted_values, state.values.at(slot));
+ sorted_counts.push_back(state.slots[slot].count);
+ }
+ auto size = static_cast<int64_t>(sorted_counts.size());
+ buf.write_binary(size);
+ write_column(buf, *sorted_values);
+ if (size != 0) {
+ // An empty list keeps an empty vector, whose data() is null; passing it to memcpy is
+ // undefined even for a zero length.
+ buf.write(reinterpret_cast<const char*>(sorted_counts.data()),
+ sorted_counts.size() * sizeof(int64_t));
+ }
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ int64_t size = 0;
+ buf.read_binary(size);
+ auto values = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(values->size()), size);
+ DorisVector<int64_t> counts(static_cast<size_t>(size));
+ if (size != 0) {
+ buf.read(reinterpret_cast<char*>(counts.data()), size * sizeof(int64_t));
+ }
+ ValueStore value_store;
+ value_store.reserve(static_cast<size_t>(size));
+ for (int64_t row = 0; row != size; ++row) {
+ value_store.append(value_at(*values, static_cast<size_t>(row)));
+ }
+ assign_sorted(state, std::move(value_store), std::move(counts));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_array = assert_cast<ColumnArray&>(to);
+ // The array element and both fields are nullable in the planned type, but a produced entry
+ // never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(result_array.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = assert_cast<ColumnNullable&>(pair_column.get_column(0));
+ auto& count_column = assert_cast<ColumnNullable&>(pair_column.get_column(1));
+ DorisVector<int32_t> order;
+ collect_in_order(state, order);
+ for (int32_t slot : order) {
+ insert_value(value_column.get_nested_column(), state.values.at(slot));
+ value_column.get_null_map_data().push_back(0);
+ assert_cast<ColumnInt64&>(count_column.get_nested_column())
+ .get_data()
+ .push_back(state.slots[slot].count);
+ count_column.get_null_map_data().push_back(0);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ result_array.get_offsets().push_back(pair_column.size());
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ // Both fields are nullable in the planned type: a consumer reads a field through
+ // element_at, which is always nullable, and named_struct then records nullable fields.
+ // Produced entries never contain a NULL field.
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {make_nullable(value_type),
+ make_nullable(std::make_shared<DataTypeInt64>())},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ return std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ }
+
+private:
+ /// The value of the row in `column`. The returned view is valid while the column is alive.
+ static ValueRef value_at(const IColumn& column, size_t row_num) {
+ if constexpr (IS_STRING) {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column).get_data_at(
+ row_num);
+ } else {
+ return assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(column)
+ .get_data()[row_num];
+ }
+ }
+
+ /// Exactly the comparison the value column would apply: CompareFloat for FLOAT/DOUBLE through
+ /// Compare::compare (the column uses the same helper), the same memcmp helper ColumnString
+ /// uses for strings, and the column's scale-independent ordering for every other type.
+ static int compare_values(ValueRef lhs, ValueRef rhs) {
+ if constexpr (IS_STRING) {
+ return memcmp_small_allow_overflow15(
+ reinterpret_cast<const uint8_t*>(lhs.data), lhs.size,
+ reinterpret_cast<const uint8_t*>(rhs.data), rhs.size);
+ } else {
+ return Compare::compare(lhs, rhs);
+ }
+ }
+
+ static void insert_value(IColumn& column, ValueRef value) {
+ if constexpr (IS_STRING) {
+ assert_cast<ColumnString&>(column).insert_data(value.data, value.size);
+ } else {
+ assert_cast<ColumnType&>(column).get_data().push_back(value);
+ }
+ }
+
+ /// True when `cmp` puts the left operand before the right one in the aggregate's order.
+ static bool comes_before(int cmp) { return IsMin ? cmp < 0 : cmp > 0; }
+
+ /// True when `cmp`, which compares a value with the worst retained one, puts it after it: the
+ /// value is worse and cannot enter the list.
+ static bool is_worse(int cmp) { return IsMin ? cmp > 0 : cmp < 0; }
+
+ /// splitmix64. The priorities only have to be independent enough to keep the treap balanced;
+ /// a per-state sequence keeps the shape reproducible for the same insertion order.
+ static uint32_t next_priority(State& state) {
+ uint64_t z = (state.rng += 0x9E3779B97F4A7C15ULL);
+ z = (z ^ (z >> 30)) * 0xBF58476D1CE4E5B9ULL;
+ z = (z ^ (z >> 27)) * 0x94D049BB133111EBULL;
+ return static_cast<uint32_t>(z ^ (z >> 31));
+ }
+
+ static int compare_slots(const State& state, int32_t lhs, int32_t rhs) {
+ return compare_values(state.values.at(lhs), state.values.at(rhs));
+ }
+
+ /// Appends (value, count) as the next slot.
+ static int32_t append_slot(State& state, ValueRef value, int64_t count) {
+ auto slot = static_cast<int32_t>(state.slots.size());
+ DCHECK_LT(slot, std::numeric_limits<int32_t>::max());
+ state.values.append(value);
+ state.slots.push_back(Slot {count, State::NO_SLOT, State::NO_SLOT, next_priority(state)});
+ ++state.live;
+ return slot;
+ }
+
+ static int32_t append_slot(State& state, const IColumn& column, size_t row_num, int64_t count) {
+ return append_slot(state, value_at(column, row_num), count);
+ }
+
+ /// Rotates the subtree rooted at `node` right: the left child becomes the root and `node` its
+ /// right child. The in-order sequence is unchanged, so the treap stays ordered.
+ static int32_t rotate_right(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].left;
+ state.slots[node].left = state.slots[pivot].right;
+ state.slots[pivot].right = node;
+ return pivot;
+ }
+
+ static int32_t rotate_left(State& state, int32_t node) {
+ int32_t pivot = state.slots[node].right;
+ state.slots[node].right = state.slots[pivot].left;
+ state.slots[pivot].left = node;
+ return pivot;
+ }
+
+ /// One descent that performs the whole per-row update: an equal value bumps its count and
+ /// reports no insertion; a missing value is linked as a new slot at the leaf and rotated up
+ /// while the recursion unwinds. `inserted_slot` receives the new slot, NO_SLOT on a hit.
+ static int32_t add_or_increment(State& state, int32_t node, const IColumn& column,
+ size_t row_num, int32_t* inserted_slot) {
+ if (node == State::NO_SLOT) {
+ *inserted_slot = append_slot(state, column, row_num, 1);
+ return *inserted_slot;
+ }
+ int cmp = compare_values(state.values.at(node), value_at(column, row_num));
+ if (cmp == 0) {
+ state.slots[node].count = checked_add(state.slots[node].count, 1);
+ return node;
+ }
+ if (comes_before(cmp)) {
+ // The stored value is the better one, so the input belongs in the right subtree.
+ state.slots[node].right = add_or_increment(state, state.slots[node].right, column,
+ row_num, inserted_slot);
+ if (state.slots[state.slots[node].right].priority > state.slots[node].priority) {
+ return rotate_left(state, node);
+ }
+ } else {
+ state.slots[node].left =
+ add_or_increment(state, state.slots[node].left, column, row_num, inserted_slot);
+ if (state.slots[state.slots[node].left].priority > state.slots[node].priority) {
+ return rotate_right(state, node);
+ }
+ }
+ return node;
+ }
+
+ /// Removes the last entry in the aggregate's order (the largest value for MIN) and returns the
+ /// new subtree root. That entry has no right child, so its left subtree takes its place; the
+ /// child's priority is below the parent's, so no rotation is needed.
+ static int32_t erase_worst(State& state, int32_t node, int32_t* removed) {
+ if (state.slots[node].right == State::NO_SLOT) {
+ *removed = node;
+ return state.slots[node].left;
+ }
+ state.slots[node].right = erase_worst(state, state.slots[node].right, removed);
+ return node;
+ }
+
+ static void refresh_worst(State& state) {
+ int32_t node = state.root;
+ while (node != State::NO_SLOT && state.slots[node].right != State::NO_SLOT) {
+ node = state.slots[node].right;
+ }
+ state.worst = node;
+ }
+
+ /// Slots in the aggregate's order, i.e. the order of the result array.
+ static void collect_in_order(const State& state, DorisVector<int32_t>& slots) {
+ slots.reserve(static_cast<size_t>(state.live));
+ DorisVector<int32_t> stack;
+ stack.reserve(64);
+ int32_t node = state.root;
+ while (node != State::NO_SLOT || !stack.empty()) {
+ while (node != State::NO_SLOT) {
+ stack.push_back(node);
+ node = state.slots[node].left;
+ }
+ node = stack.back();
+ stack.pop_back();
+ // Only live slots are reachable from the root; an evicted slot has no row count.
+ DCHECK_GE(state.slots[node].count, 0);
+ slots.push_back(node);
+ node = state.slots[node].right;
+ }
+ DCHECK_EQ(static_cast<int64_t>(slots.size()), state.live);
+ }
+
+ /// Rebuilds the tree from `values`, already in the aggregate's order, and the matching counts.
+ /// The treap is built in O(size) as a Cartesian tree over the fresh priorities.
+ static void assign_sorted(State& state, ValueStore values, DorisVector<int64_t> counts) {
+ const size_t size = counts.size();
+ DCHECK_EQ(values.size(), size);
+ state.values.swap(values);
+ state.slots.assign(size, Slot {});
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].count = counts[slot];
+ }
+ state.live = static_cast<int64_t>(size);
+ state.dead = 0;
+ DorisVector<int32_t> stack;
+ stack.reserve(size);
+ for (size_t slot = 0; slot != size; ++slot) {
+ state.slots[slot].priority = next_priority(state);
+ int32_t last = State::NO_SLOT;
+ while (!stack.empty() &&
+ state.slots[stack.back()].priority < state.slots[slot].priority) {
+ last = stack.back();
+ stack.pop_back();
+ }
+ state.slots[slot].left = last;
+ if (!stack.empty()) {
+ state.slots[stack.back()].right = static_cast<int32_t>(slot);
+ }
+ stack.push_back(static_cast<int32_t>(slot));
+ }
+ state.root = stack.empty() ? State::NO_SLOT : stack.front();
+ refresh_worst(state);
+ }
+
+ /// Reclaims the slots evicted since the last compaction. The O(k) rebuild is charged to the
+ /// k/2 evictions that produced the dead slots, so it stays O(1) amortized per row.
+ static void compact(State& state) {
+ DorisVector<int32_t> slots;
+ collect_in_order(state, slots);
+ ValueStore values;
+ values.reserve(slots.size());
+ DorisVector<int64_t> counts;
+ counts.reserve(slots.size());
+ for (int32_t slot : slots) {
+ values.append(state.values.at(slot));
+ counts.push_back(state.slots[slot].count);
+ }
+ assign_sorted(state, std::move(values), std::move(counts));
+ }
+
+ static void check_k(State& state, const IColumn& count_column, size_t row_num) {
+ const IColumn* nested = &count_column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&count_column)) {
+ if (UNLIKELY(nullable->is_null_at(row_num))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-null "
+ "constant");
+ }
+ nested = &nullable->get_nested_column();
+ }
+ int64_t value =
+ assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(*nested).get_element(
+ row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of min_topk and max_topk must be a non-negative "
+ "constant, but got {}",
+ value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "min_topk/max_topk value count overflow");
+ }
+ return result;
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(State& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (state.live == 0) {
+ state.root = append_slot(state, column, row_num, 1);
+ state.worst = state.root;
+ return;
+ }
+ int cmp = compare_values(value_at(column, row_num), state.values.at(0));
+ if (cmp == 0) {
+ state.slots[0].count = checked_add(state.slots[0].count, 1);
+ } else if (comes_before(cmp)) {
+ // The new value is better: replace the tracked extreme. The slot and the
+ // single-node tree stay in place, only the stored value changes.
+ state.values.clear();
+ state.values.append(value_at(column, row_num));
+ state.slots[0].count = 1;
+ }
+ return;
+ }
+
+ // Dead slots are reclaimed before they can outnumber the live ones; see compact().
+ if (state.dead > state.live) {
Review Comment:
Residual case closed in 5c8beb3b4bc: the compaction trigger is now checked immediately after an eviction as well, in both add_value and the singleton merge, so the last eviction of a group reclaims the discarded payload instead of keeping it until another row arrives.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6492497849
Correction to the memory table above, and a re-measurement.
The table above varied the number of rows per group at a fixed **total row count** (10M rows), so the 8-rows-per-group case also had eight times fewer groups (1.25M instead of 10M). That is not the shape the change is about. The correct comparison keeps the **group count fixed** and grows the rows per group, i.e. 10M groups with 1 row each (10M rows) against 10M groups with 8 rows each (80M rows).
Setup for the numbers below: release build, single BE restarted before every query (cold page cache on both sides), `enable_sql_cache=false`, `exec_mem_limit` raised, RSS peak delta of one query. The `MIN` runs are unaffected by this change, so the differences between their before/after values show the method's noise, roughly ±10%.
**BIGINT (10M groups)**
| rows per group | total rows | query | before | after |
| --- | --- | --- | --- | --- |
| 1 | 10M | `MIN(v)` | 956 MB | 1060 MB |
| 1 | 10M | `min_topk(v, 8)` | 1851 MB | 1877 MB |
| 8 | 80M | `MIN(v)` | 2044 MB | 1933 MB |
| 8 | 80M | `min_topk(v, 8)` | 5183 MB | 5071 MB |
**VARCHAR (10M groups)**
| rows per group | total rows | query | before | after |
| --- | --- | --- | --- | --- |
| 1 | 10M | `MIN(v)` | 1419 MB | 1348 MB |
| 1 | 10M | `min_topk(v, 8)` | 3085 MB | 3200 MB |
| 8 | 80M | `MIN(v)` | 2931 MB | 2842 MB |
| 8 | 80M | `min_topk(v, 8)` | 8395 MB | 8346 MB |
State kept on top of `MIN`, per group:
| value type | rows per group | before | after |
| --- | --- | --- | --- |
| BIGINT | 1 | ~90 B | ~82 B |
| BIGINT | 8 | ~314 B | ~314 B |
| VARCHAR | 1 | ~167 B | ~185 B |
| VARCHAR | 8 | ~546 B | ~550 B |
What this says: in a release build the change is at the edge of, or below, what this method resolves. At eight distinct values per group it changes nothing measurable, because the slots dominate and a slot costs the same either way (24-byte metadata plus the 8-byte value against one 32-byte slot carrying the value inline). At one value per group the state is ~24 bytes and one heap allocation smaller, which is the whole effect and is about the size of the measurement noise.
The larger gain reported earlier (1959 MB to 606 MB at 10M groups) was measured on a debug/ASAN build, where every allocation carries a redzone, so the second vector per group is what dominated. The change is worth keeping for that build, and because it removes a container, but it should not be sold as a release-memory win.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6496656982
# FE UT Coverage Report
Increment line coverage ` 28.07% (48/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_29fb3c47891613984533a1602cf3c9e6d15e2bb2/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_29fb3c47891613984533a1602cf3c9e6d15e2bb2/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6498468822
<details>
<summary>TPC-DS: <b>Total hot run time: 151336 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 29fb3c47891613984533a1602cf3c9e6d15e2bb2, data reload: false
query5 4310 597 437 437
query6 426 212 192 192
query7 4807 431 224 224
query8 320 163 149 149
query9 8799 3246 3240 3240
query10 481 308 265 265
query11 6029 3851 3570 3570
query12 155 98 91 91
query13 1253 437 359 359
query14 6117 3946 3641 3641
query14_1 3417 3432 3413 3413
query15 224 204 181 181
query16 942 446 375 375
query17 869 657 532 532
query18 2426 425 315 315
query19 204 173 144 144
query20 85 87 82 82
query21 208 130 109 109
query22 12982 13133 12833 12833
query23 13372 12950 12479 12479
query23_1 12524 12579 12655 12579
query24 7358 848 474 474
query24_1 488 491 497 491
query25 541 430 384 384
query26 1270 260 143 143
query27 2761 443 288 288
query28 4605 2075 2062 2062
query29 1577 601 470 470
query30 302 222 184 184
query31 1022 923 819 819
query32 164 96 92 92
query33 531 311 244 244
query34 895 869 520 520
query35 740 749 659 659
query36 831 826 712 712
query37 132 102 83 83
query38 1776 1758 1668 1668
query39 788 786 735 735
query39_1 730 727 741 727
query40 213 115 95 95
query41 70 67 72 67
query42 88 87 87 87
query43 345 366 322 322
query44 1323 713 709 709
query45 180 185 169 169
query46 880 969 573 573
query47 3025 2847 2627 2627
query48 296 303 229 229
query49 569 397 291 291
query50 665 263 210 210
query51 10596 10532 10396 10396
query52 81 78 69 69
query53 193 207 150 150
query54 230 205 178 178
query55 67 64 63 63
query56 232 227 207 207
query57 1687 1618 1565 1565
query58 280 261 278 261
query59 2206 2158 2000 2000
query60 299 244 227 227
query61 145 142 146 142
query62 390 380 277 277
query63 191 158 158 158
query64 2738 958 763 763
query65 3318 3258 3255 3255
query66 1787 416 304 304
query67 20192 20170 20008 20008
query68 2993 1014 621 621
query69 380 272 242 242
query70 837 800 783 783
query71 291 237 214 214
query72 2695 2562 2275 2275
query73 530 531 297 297
query74 4655 4498 4312 4312
query75 2322 2265 1920 1920
query76 2310 1011 587 587
query77 338 370 267 267
query78 9927 9693 9173 9173
query79 1552 838 543 543
query80 1234 436 346 346
query81 590 320 273 273
query82 875 133 103 103
query83 309 200 179 179
query84 311 124 104 104
query85 859 434 367 367
query86 396 239 234 234
query87 1977 1962 1782 1782
query88 3874 2781 2789 2781
query89 339 281 243 243
query90 1857 166 172 166
query91 166 153 130 130
query92 96 90 89 89
query93 1142 997 583 583
query94 679 313 299 299
query95 570 342 321 321
query96 680 477 241 241
query97 2477 2451 2347 2347
query98 164 153 144 144
query99 761 756 635 635
Total cold run time: 236556 ms
Total hot run time: 151336 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6406608760
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6407895181
# FE Regression Coverage Report
Increment line coverage ` 17.48% (139/795)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6432419393
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4384810800
##########
be/src/exprs/lambda_function/varray_sorted_merge_function.cpp:
##########
@@ -0,0 +1,377 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <algorithm>
+#include <map>
+#include <memory>
+#include <set>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "exprs/lambda_function/lambda_execution_context.h"
+#include "exprs/lambda_function/lambda_function.h"
+#include "exprs/lambda_function/lambda_function_factory.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vlambda_function_expr.h"
+
+namespace doris {
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc)
+ *
+ * Merges two arrays that are already sorted by the elements' natural order. The merge lambda
+ * receives the two current candidates and returns the combined element; returning NULL means the
+ * two elements must not be combined, so both are kept. Each input key occurs at most once per
+ * array, so a key appears at most twice across the two arrays and the lambda is called at most
+ * once per key.
+ */
+class ArraySortedMergeFunction : public LambdaFunction {
+ ENABLE_FACTORY_CREATOR(ArraySortedMergeFunction);
+
+public:
+ ~ArraySortedMergeFunction() override = default;
+
+ static constexpr auto name = "array_sorted_merge";
+
+ static LambdaFunctionPtr create() { return std::make_shared<ArraySortedMergeFunction>(); }
+
+ std::string get_name() const override { return name; }
+
+ /// Rejects a nested lambda that captures one of this lambda's arguments: a merge argument
+ /// stands for the candidate of one comparison step, which is not part of the supported
+ /// contract for a nested lambda. Nested lambdas may still use their own arguments
+ /// (shadowing) and may capture enclosing lambda arguments.
+ static Status validate_nested_captures(const VExprSPtr& expr,
+ const std::vector<std::string>& argument_names,
+ bool inside_nested_lambda,
+ std::vector<std::string> shadowed_names) {
+ if (expr->is_column_ref()) {
+ const std::string& column_name = expr->expr_name();
+ if (std::ranges::find(shadowed_names, column_name) != shadowed_names.end()) {
+ return Status::OK();
+ }
+ if (inside_nested_lambda &&
+ std::ranges::find(argument_names, column_name) != argument_names.end()) {
+ return Status::NotSupported(
+ "array_sorted_merge merge lambda argument '{}' cannot be captured by a "
+ "nested lambda",
+ column_name);
+ }
+ return Status::OK();
+ }
+ if (expr->node_type() == TExprNodeType::LAMBDA_FUNCTION_EXPR) {
+ const auto* lambda = assert_cast<const VLambdaFunctionExpr*>(expr.get());
+ if (lambda->has_argument_names()) {
+ for (const auto& argument_name : lambda->argument_names()) {
+ shadowed_names.emplace_back(argument_name);
+ }
+ }
+ return validate_nested_captures(lambda->get_child(0), argument_names, true,
+ std::move(shadowed_names));
+ }
+ for (const auto& child : expr->children()) {
+ RETURN_IF_ERROR(validate_nested_captures(child, argument_names, inside_nested_lambda,
+ shadowed_names));
+ }
+ return Status::OK();
+ }
+
+ Status prepare(RuntimeState* state, const VExprSPtrs& children) override {
+ RETURN_IF_ERROR(LambdaFunction::prepare(state, children));
+ DCHECK_EQ(children.size(), 4);
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ if (!lambda_expr->has_argument_names()) {
+ return Status::OK();
+ }
+ // A nested lambda that captures one of this lambda's arguments has no defined meaning
+ // (the argument is the candidate of a single comparison step); reject it during prepare
+ // instead of resolving it against some column at runtime.
+ return validate_nested_captures(children[0]->get_child(0), lambda_expr->argument_names(),
+ false, {});
+ }
+
+ Status execute(VExprContext* context, const Block* block, const Selector* expr_selector,
+ size_t count, ColumnPtr& result_column, const DataTypePtr& result_type,
+ const VExprSPtrs& children) const override {
+ DCHECK_EQ(children.size(), 4);
+
+ const ColumnArray* arrays[2] = {nullptr, nullptr};
+ const IColumn* data[2] = {nullptr, nullptr};
+ DataTypePtr nested_types[2];
+ ColumnPtr holders[2];
+ const ColumnUInt8::Container* nullmaps[2] = {nullptr, nullptr};
+ for (int i = 0; i < 2; ++i) {
+ RETURN_IF_ERROR(children[i + 1]->execute_column(context, block, expr_selector, count,
+ holders[i]));
+ holders[i] = holders[i]->convert_to_full_column_if_const();
+ DataTypePtr type = children[i + 1]->execute_type(block);
+ const IColumn* array_column = holders[i].get();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(array_column)) {
+ nullmaps[i] = &nullable->get_null_map_column().get_data();
+ array_column = &nullable->get_nested_column();
+ type = assert_cast<const DataTypeNullable&>(*type).get_nested_type();
+ }
+ arrays[i] = assert_cast<const ColumnArray*>(array_column);
+ data[i] = arrays[i]->get_data_ptr().get();
+ nested_types[i] = assert_cast<const DataTypeArray&>(*type).get_nested_type();
+ }
+
+ ColumnPtr flag_holder;
+ RETURN_IF_ERROR(
+ children[3]->execute_column(context, block, expr_selector, count, flag_holder));
+ flag_holder = flag_holder->convert_to_full_column_if_const();
+ const IColumn* flag_column = flag_holder.get();
+ const ColumnUInt8::Container* flag_nullmap = nullptr;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(flag_column)) {
+ flag_nullmap = &nullable->get_null_map_column().get_data();
+ flag_column = &nullable->get_nested_column();
+ }
+ const auto& flags = assert_cast<const ColumnUInt8&>(*flag_column).get_data();
+
+ const size_t rows = arrays[0]->size();
+ auto result_nested = nested_types[0]->create_column();
+ auto result_offsets = ColumnArray::ColumnOffsets::create();
+ auto& offsets_data = result_offsets->get_data();
+ offsets_data.reserve(rows);
+ auto result_nullmap = ColumnUInt8::create(rows, 0);
+ auto& result_nullmap_data = result_nullmap->get_data();
+
+ // The lambda may capture enclosing columns (for example a table column compared against
+ // the candidates). Materialize every captured input at its original position in the
+ // one-row lambda block, so slot refs read the current row; the two merge candidates
+ // take the positions after the last captured column and are bound by name.
+ std::set<int> captured_ids;
+ children[0]->collect_slot_column_ids(captured_ids);
+ // An enclosing lambda's argument is a name reference rather than a slot, and this
+ // function executes its lambda against a block of its own that hides the parent frames.
+ // Publish the visible binding values under their names in that block, so the reference
+ // resolves by name like the merge arguments do.
+ std::vector<LambdaExecutionContext::Binding> enclosing_bindings;
+ context->lambda_execution_context().collect_visible_bindings(enclosing_bindings);
+ if ((!captured_ids.empty() || !enclosing_bindings.empty()) && block == nullptr) {
+ return Status::InternalError(
+ "array_sorted_merge lambda references outer columns but has no input block");
+ }
+ std::map<int, ColumnPtr> captured_columns;
+ for (int id : captured_ids) {
+ if (id < 0 || static_cast<size_t>(id) >= block->columns()) {
+ return Status::InternalError(
+ "array_sorted_merge lambda input column id {} is outside input block", id);
+ }
+ captured_columns[id] =
+ block->get_by_position(id).column->convert_to_full_column_if_const();
Review Comment:
Fixed in 5309c29db2b: the private merge block now also copies the columns a captured virtual slot's expression reads. `VirtualSlotRef::collect_slot_column_ids` only reports the virtual column itself, so before this the block kept the `ColumnNothing` placeholder for `v` and padded `lim`'s position with the `UInt8 temp` column; the lazy `abs(lim)` then read a wrong-typed column. The new `collect_virtual_capture_inputs` walks the lambda for `VirtualSlotRef` nodes, adds the slot ids of their virtual expressions recursively, and those columns are copied at their original ids with their real types.
Reproduced before the fix with the reduced query and `experimental_enable_virtual_slot_for_cse = true`: `abs's argument does not match the expected data type.` The same query now returns `2`, and `order_qt_virtual_column_capture` in `test_array_sorted_merge` pins the pushed-filter CSE case (DUP table `t(lim, l, r)` with `(2,[1],[1])`).
Note this covers `array_sorted_merge`; a general fix could instead make `VirtualSlotRef::collect_slot_column_ids` include its expression's inputs so every higher-order function gets the same behavior, but that changes the contract for all callers, so it is left for a separate change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6431549068
cloud_p0 failed on a UBSan report in `be.out` while serializing an empty top-K state:
```
../src/core/pod_array.h:518:29: runtime error: null pointer passed as argument 2, which is declared to never be null
#3 AggregateFunctionTopK<...>::serialize(...) aggregate_function_topk.h:156
```
An empty state keeps an empty `counts` vector, whose `data()` is null, and `serialize`/`deserialize` passed it to `BufferWritable::write` / `BufferReadable::read`. `memcpy` rejects a null argument even for a zero length.
Fixed in 3a22241d0d5: the counts payload is written and read only when the list is non-empty. The value column payload always carries a size prefix, so only the counts needed the guard. `TopKTest.SerializeDeserializeRoundTrip` now round-trips an empty state as well.
The BE UT build cannot catch this class of issue on its own: `CXX_FLAGS_ASAN_UT` enables `-fsanitize=address` only, while the full ASAN build used by cloud_p0 adds `-fsanitize=undefined`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4358132125
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArraySortedMerge.java:
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.scalar;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.ArrayItemReference;
+import org.apache.doris.nereids.trees.expressions.Cast;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.functions.ImplicitlyCastableSignature;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.ArrayType;
+import org.apache.doris.nereids.types.BooleanType;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.types.LambdaType;
+import org.apache.doris.nereids.types.coercion.FollowToAnyDataType;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc): merge two arrays that are already sorted.
+ *
+ * <pre>
+ * array_sorted_merge((a, b) -> ..., arr1, arr2, true)
+ * </pre>
+ *
+ * <p>The inputs must already be sorted by the elements' natural order in the direction given by
+ * {@code is_asc}; the function only merges them. The merge lambda receives the two current
+ * candidates as {@code (a, b)} and returns the combined element; returning NULL means the two
+ * elements must not be combined, so both are kept. Each key occurs at most once per input, so a
+ * key appears at most twice across the two arrays and the lambda is called at most once per key.
+ *
+ * <p>A NULL input array makes the result NULL, like the other higher-order array functions. The
+ * ascending flag must be a non-null constant; it deliberately does not take part in nullable
+ * propagation, which would fold a NULL flag into a NULL result instead of failing the analysis.
+ */
+public class ArraySortedMerge extends ScalarFunction
+ implements ImplicitlyCastableSignature {
+
+ // The bound form: the lambda plus the ascending flag; the arrays live in the lambda argument
+ // references, like the other lambda functions.
+ public static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
+ FunctionSignature.ret(new FollowToAnyDataType(0))
+ .args(LambdaType.INSTANCE, BooleanType.INSTANCE));
+
+ /**
+ * The extra arguments are variadic so that the function builder resolved with only the lambda
+ * (the higher-order lookup) can also build the bound form that keeps the ascending flag.
+ */
+ public ArraySortedMerge(Expression lambda, Expression... extra) {
+ super("array_sorted_merge",
+ ImmutableList.<Expression>builder().add(lambda).add(extra).build());
+ }
+
+ /** constructor for withChildren and reuse signature */
+ private ArraySortedMerge(ScalarFunctionParams functionParams) {
+ super(functionParams);
+ }
+
+ @Override
+ public List<FunctionSignature> getSignatures() {
+ return SIGNATURES;
+ }
+
+ @Override
+ public ArraySortedMerge withChildren(List<Expression> children) {
+ Preconditions.checkArgument(children.size() == 2);
+ return new ArraySortedMerge(getFunctionParams(children));
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ if (!(child(0) instanceof Lambda)) {
+ throw new AnalysisException(getName() + " requires a merge lambda as the first argument");
+ }
+ Lambda lambda = (Lambda) child(0);
+ if (lambda.getLambdaArgumentNames().size() != 2) {
+ throw new AnalysisException(getName() + " requires a two-argument merge lambda, but got "
+ + lambda.getLambdaArgumentNames().size());
+ }
+ ArrayType left = requireArray(lambda, 0);
+ ArrayType right = requireArray(lambda, 1);
+ if (!left.getItemType().equals(right.getItemType())) {
+ throw new AnalysisException(getName()
+ + " requires both arrays to have the same element type, but got "
+ + left.getItemType().toSql() + " and " + right.getItemType().toSql());
+ }
+ if (!lambda.getRetType().equals(left.getItemType())) {
Review Comment:
Fixed in e250347cd12: the pair fields of min_topk/max_topk are now declared nullable, which is what a lambda can actually produce from them. An array element is always nullable, so `element_at` yields a nullable field and `named_struct` records it as nullable; a required field made the element-type equality fail, and `CheckCast` correctly refuses the nullable -> required cast that would otherwise be needed.
The aggregate still never writes a NULL field: the BE return type mirrors the nullable fields, and `insert_result_into` writes zero null maps for the pair element and for both fields. Nothing is cast, and the `CheckCast` safety rule is untouched.
The direct composition now works for both aggregates:
```sql
SELECT array_sorted_merge(
(a, b) -> IF(element_at(a, 'value') = element_at(b, 'value'),
named_struct('value', element_at(a, 'value'),
'count', element_at(a, 'count') + element_at(b, 'count')),
NULL),
s, s, true)
FROM (SELECT min_topk(i, 2) AS s FROM (SELECT 1 AS i) t) x
-- [{"value":1, "count":2}]
```
Covered by `order_qt_topk_output_merge` in `test_array_sorted_merge`; `TopKFunctionSignatureTest` and the BE `TopKTest.*` unit tests were updated to assert the nullable pair layout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6346413871
# FE UT Coverage Report
Increment line coverage ` 29.88% (49/164)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_a7420809fc69a4ea822cfd1f29ed9f8b03afbb3a/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_a7420809fc69a4ea822cfd1f29ed9f8b03afbb3a/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6349847838
<details>
<summary>ClickBench: <b>Total hot run time: 24.47 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d, data reload: false
query1 0.00 0.00 0.00
query2 0.10 0.05 0.06
query3 0.27 0.09 0.09
query4 1.62 0.10 0.10
query5 0.23 0.24 0.24
query6 1.18 0.69 0.66
query7 0.04 0.01 0.01
query8 0.06 0.03 0.03
query9 0.40 0.34 0.36
query10 0.61 0.58 0.56
query11 0.20 0.14 0.14
query12 0.18 0.14 0.15
query13 0.50 0.50 0.48
query14 1.05 1.02 1.01
query15 0.62 0.60 0.62
query16 0.35 0.34 0.34
query17 1.15 1.11 1.09
query18 0.21 0.20 0.20
query19 2.11 1.95 1.99
query20 0.02 0.01 0.02
query21 15.49 0.21 0.14
query22 4.94 0.05 0.06
query23 16.26 0.32 0.12
query24 2.76 0.94 0.61
query25 0.14 0.05 0.05
query26 1.65 0.22 0.16
query27 0.06 0.04 0.05
query28 4.35 0.72 0.37
query29 12.50 4.20 3.33
query30 0.28 0.13 0.10
query31 2.77 0.56 0.33
query32 3.22 0.60 0.50
query33 3.14 3.15 3.21
query34 15.39 4.01 3.38
query35 3.33 3.33 3.36
query36 0.57 0.44 0.42
query37 0.09 0.06 0.07
query38 0.05 0.04 0.04
query39 0.04 0.02 0.02
query40 0.18 0.15 0.14
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.05 0.03 0.04
Total cold run time: 98.29 s
Total hot run time: 24.47 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6352739211
<details>
<summary>TPC-H: <b>Total hot run time: 28494 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1e321f5e25ce0dd2d9f174971bb0cf1004ccbe3d, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17632 5101 5064 5064
q2 2164 300 253 253
q3 10223 1380 825 825
q4 4685 464 336 336
q5 7436 803 527 527
q6 185 191 159 159
q7 750 803 627 627
q8 9241 1337 1069 1069
q9 5505 4211 4211 4211
q10 6832 1294 1069 1069
q11 430 244 228 228
q12 644 411 295 295
q13 18044 3561 3023 3023
q14 267 271 255 255
q15 q16 735 715 664 664
q17 1186 743 571 571
q18 7074 6132 6080 6080
q19 1104 964 575 575
q20 370 338 228 228
q21 5024 2176 2273 2176
q22 349 297 259 259
Total cold run time: 99880 ms
Total hot run time: 28494 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5596 5597 5638 5597
q2 688 573 523 523
q3 5241 5430 4927 4927
q4 2009 2161 1574 1574
q5 5263 5130 5593 5130
q6 256 201 171 171
q7 2284 2026 1827 1827
q8 3027 2642 2670 2642
q9 8089 7949 7935 7935
q10 3744 3639 3311 3311
q11 620 452 419 419
q12 703 713 540 540
q13 3551 3848 3258 3258
q14 328 342 285 285
q15 q16 696 705 641 641
q17 8240 7202 6879 6879
q18 13019 12191 12860 12191
q19 905 781 788 781
q20 2163 2157 1901 1901
q21 5721 4795 4765 4765
q22 486 453 419 419
Total cold run time: 72629 ms
Total hot run time: 65716 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6361366709
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6367727329
# FE Regression Coverage Report
Increment line coverage ` 81.29% (139/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4357073249
##########
be/src/exprs/lambda_function/varray_sorted_merge_function.cpp:
##########
@@ -0,0 +1,215 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <memory>
+#include <string>
+#include <utility>
+
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "exprs/lambda_function/lambda_execution_context.h"
+#include "exprs/lambda_function/lambda_function.h"
+#include "exprs/lambda_function/lambda_function_factory.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "exprs/vlambda_function_expr.h"
+
+namespace doris {
+
+/**
+ * array_sorted_merge(merge_fn, arr1, arr2, is_asc)
+ *
+ * Merges two arrays that are already sorted by the elements' natural order. The merge lambda
+ * receives the two current candidates and returns the combined element; returning NULL means the
+ * two elements must not be combined, so both are kept. Each input key occurs at most once per
+ * array, so a key appears at most twice across the two arrays and the lambda is called at most
+ * once per key.
+ */
+class ArraySortedMergeFunction : public LambdaFunction {
+ ENABLE_FACTORY_CREATOR(ArraySortedMergeFunction);
+
+public:
+ ~ArraySortedMergeFunction() override = default;
+
+ static constexpr auto name = "array_sorted_merge";
+
+ static LambdaFunctionPtr create() { return std::make_shared<ArraySortedMergeFunction>(); }
+
+ std::string get_name() const override { return name; }
+
+ Status execute(VExprContext* context, const Block* block, const Selector* expr_selector,
+ size_t count, ColumnPtr& result_column, const DataTypePtr& result_type,
+ const VExprSPtrs& children) const override {
+ DCHECK_EQ(children.size(), 4);
+
+ const ColumnArray* arrays[2] = {nullptr, nullptr};
+ const IColumn* data[2] = {nullptr, nullptr};
+ DataTypePtr nested_types[2];
+ ColumnPtr holders[2];
+ const ColumnUInt8::Container* nullmaps[2] = {nullptr, nullptr};
+ for (int i = 0; i < 2; ++i) {
+ RETURN_IF_ERROR(children[i + 1]->execute_column(context, block, expr_selector, count,
+ holders[i]));
+ holders[i] = holders[i]->convert_to_full_column_if_const();
+ DataTypePtr type = children[i + 1]->execute_type(block);
+ const IColumn* array_column = holders[i].get();
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(array_column)) {
+ nullmaps[i] = &nullable->get_null_map_column().get_data();
+ array_column = &nullable->get_nested_column();
+ type = assert_cast<const DataTypeNullable&>(*type).get_nested_type();
+ }
+ arrays[i] = assert_cast<const ColumnArray*>(array_column);
+ data[i] = arrays[i]->get_data_ptr().get();
+ nested_types[i] = assert_cast<const DataTypeArray&>(*type).get_nested_type();
+ }
+
+ ColumnPtr flag_holder;
+ RETURN_IF_ERROR(
+ children[3]->execute_column(context, block, expr_selector, count, flag_holder));
+ flag_holder = flag_holder->convert_to_full_column_if_const();
+ const IColumn* flag_column = flag_holder.get();
+ const ColumnUInt8::Container* flag_nullmap = nullptr;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(flag_column)) {
+ flag_nullmap = &nullable->get_null_map_column().get_data();
+ flag_column = &nullable->get_nested_column();
+ }
+ const auto& flags = assert_cast<const ColumnUInt8&>(*flag_column).get_data();
+
+ const size_t rows = arrays[0]->size();
+ auto result_nested = nested_types[0]->create_column();
+ auto result_offsets = ColumnArray::ColumnOffsets::create();
+ auto& offsets_data = result_offsets->get_data();
+ offsets_data.reserve(rows);
+ auto result_nullmap = ColumnUInt8::create(rows, 0);
+ auto& result_nullmap_data = result_nullmap->get_data();
+
+ // Two one-row columns hold the lambda arguments of the current merge step. They carry
+ // the lambda argument names because the column refs resolve by name when no
+ // position-based gap is set.
+ const auto* lambda_expr = assert_cast<const VLambdaFunctionExpr*>(children[0].get());
+ const std::vector<std::string>* argument_names =
+ lambda_expr->has_argument_names() ? &lambda_expr->argument_names() : nullptr;
+ Block lambda_block;
+ for (int i = 0; i < 2; ++i) {
+ const std::string argument_name =
+ argument_names != nullptr && i < argument_names->size()
+ ? (*argument_names)[i]
+ : "merge_arg_" + std::to_string(i);
+ lambda_block.insert(ColumnWithTypeAndName(nested_types[i]->create_column(),
+ nested_types[i], argument_name));
+ }
+ // The lambda arguments are bound by position (column ids 0 and 1), like array_sort's
+ // comparator, so the body reads these two one-row columns.
+ IColumn* argument_columns[2] = {
+ lambda_block.get_by_position(0).column->assert_mutable().get(),
+ lambda_block.get_by_position(1).column->assert_mutable().get()};
+
+ LambdaExecutionContext::Frame lambda_frame;
+ lambda_frame.bind_by_name = false;
+ lambda_frame.parent_bindings_visible = false;
+ LambdaExecutionContext::FrameGuard lambda_frame_guard(context->lambda_execution_context(),
+ std::move(lambda_frame));
+
+ const auto& offsets0 = arrays[0]->get_offsets();
+ const auto& offsets1 = arrays[1]->get_offsets();
+ const int lambda_result_base = static_cast<int>(lambda_block.columns());
+
+ for (size_t row = 0; row < rows; ++row) {
+ const bool row_is_null = (nullmaps[0] != nullptr && (*nullmaps[0])[row] != 0) ||
+ (nullmaps[1] != nullptr && (*nullmaps[1])[row] != 0);
+ if (row_is_null) {
+ result_nullmap_data[row] = 1;
+ offsets_data.push_back(result_nested->size());
+ continue;
+ }
+ if (flag_nullmap != nullptr && (*flag_nullmap)[row] != 0) {
+ return Status::InvalidArgument(
+ "array_sorted_merge requires a non-null ascending flag");
+ }
+ const bool ascending = flags[row] != 0;
+
+ size_t left = row == 0 ? 0 : offsets0[row - 1];
+ const size_t left_end = offsets0[row];
+ size_t right = row == 0 ? 0 : offsets1[row - 1];
+ const size_t right_end = offsets1[row];
+
+ while (left < left_end || right < right_end) {
+ if (left == left_end) {
+ result_nested->insert_from(*data[1], right++);
+ continue;
+ }
+ if (right == right_end) {
+ result_nested->insert_from(*data[0], left++);
+ continue;
+ }
+
+ argument_columns[0]->clear();
+ argument_columns[0]->insert_from(*data[0], left);
+ argument_columns[1]->clear();
+ argument_columns[1]->insert_from(*data[1], right);
+
+ int lambda_res_id = lambda_result_base;
+ RETURN_IF_ERROR(children[0]->execute(context, &lambda_block, &lambda_res_id));
Review Comment:
Fixed in 51bd4e1021e: the merge frame now rebases every binding it can see onto the private block instead of hiding them. It publishes the enclosing bindings with their positions in that block, then the two merge arguments, so a same-named `a`/`b` shadows an outer binding. `parent_bindings_visible` stays false, because the parent frame positions do not describe this block.
A lambda nested inside the merge body therefore resolves `x` through the merge frame: the nested function collects that binding position and materializes it from the one-row merge block, and the value is cut at the same source row as the arrays. The reduced plan
```sql
SELECT array_map(x -> array_sorted_merge(
(a, b) -> IF(a = b AND a < element_at(array_map(y -> x, [1]), 1), a, NULL),
CAST([1] AS ARRAY<INT>), CAST([1] AS ARRAY<INT>), true), [0, 2])
```
now returns `[[1, 1], [1]]`, pinned by `order_qt_capture_enclosing_lambda_nested` in `test_array_sorted_merge`. Nested captures of the merge-local `a`/`b` are still rejected during prepare, and the legacy positional frame is unchanged when the lambda carries no argument names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6338688003
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6338689426
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6344998494
<details>
<summary>ClickBench: <b>Total hot run time: 24.43 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a7420809fc69a4ea822cfd1f29ed9f8b03afbb3a, data reload: false
query1 0.01 0.01 0.00
query2 0.10 0.05 0.05
query3 0.27 0.08 0.09
query4 1.61 0.10 0.10
query5 0.25 0.24 0.25
query6 1.16 0.69 0.68
query7 0.04 0.01 0.01
query8 0.06 0.05 0.04
query9 0.41 0.36 0.36
query10 0.58 0.54 0.61
query11 0.20 0.14 0.14
query12 0.18 0.15 0.15
query13 0.49 0.50 0.52
query14 1.04 1.01 1.02
query15 0.64 0.61 0.61
query16 0.35 0.36 0.35
query17 1.14 1.13 1.17
query18 0.20 0.19 0.19
query19 2.05 1.99 1.98
query20 0.02 0.02 0.01
query21 15.52 0.20 0.14
query22 4.80 0.05 0.06
query23 16.27 0.31 0.13
query24 2.77 0.59 0.43
query25 0.11 0.05 0.04
query26 1.36 0.21 0.15
query27 0.04 0.05 0.05
query28 3.63 0.76 0.40
query29 12.48 4.17 3.27
query30 0.29 0.13 0.10
query31 2.78 0.55 0.33
query32 3.22 0.60 0.51
query33 3.19 3.20 3.15
query34 15.33 4.03 3.40
query35 3.37 3.35 3.35
query36 0.55 0.43 0.41
query37 0.09 0.07 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.04
query40 0.17 0.15 0.14
query41 0.09 0.03 0.03
query42 0.03 0.02 0.02
query43 0.04 0.03 0.04
Total cold run time: 97.02 s
Total hot run time: 24.43 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6337086631
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6341230595
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6472282714
<details>
<summary>ClickBench: <b>Total hot run time: 24.43 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5c8beb3b4bc041961f08e39e9b5aae6284a64cd6, data reload: false
query1 0.01 0.01 0.01
query2 0.09 0.05 0.04
query3 0.27 0.09 0.07
query4 1.61 0.10 0.09
query5 0.24 0.25 0.24
query6 1.17 0.68 0.67
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.41 0.36 0.34
query10 0.58 0.59 0.57
query11 0.20 0.14 0.15
query12 0.18 0.15 0.16
query13 0.48 0.48 0.50
query14 1.05 1.01 1.02
query15 0.63 0.60 0.61
query16 0.36 0.34 0.35
query17 1.15 1.14 1.12
query18 0.20 0.19 0.20
query19 2.08 2.01 1.94
query20 0.02 0.01 0.01
query21 15.49 0.19 0.14
query22 4.90 0.05 0.05
query23 16.27 0.32 0.13
query24 2.84 0.52 0.45
query25 0.12 0.06 0.07
query26 1.69 0.22 0.15
query27 0.05 0.05 0.03
query28 3.74 0.76 0.41
query29 12.49 4.16 3.26
query30 0.28 0.15 0.10
query31 2.77 0.57 0.32
query32 3.22 0.60 0.51
query33 3.28 3.20 3.24
query34 15.33 4.00 3.38
query35 3.32 3.38 3.37
query36 0.54 0.46 0.43
query37 0.09 0.06 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.15 0.15
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.04
Total cold run time: 97.67 s
Total hot run time: 24.43 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6475766516
# FE Regression Coverage Report
Increment line coverage ` 76.37% (139/182)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_5c8beb3b4bc041961f08e39e9b5aae6284a64cd6_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_5c8beb3b4bc041961f08e39e9b5aae6284a64cd6_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6466774580
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6469961597
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6467487799
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6494026016
Codex automated review failed and did not complete.
Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/40848908479
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6360256804
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6361621940
# FE UT Coverage Report
Increment line coverage ` 28.07% (48/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_51bd4e1021eb209791aee5937061ac8805784483/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_51bd4e1021eb209791aee5937061ac8805784483/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6362185320
<details>
<summary>TPC-H: <b>Total hot run time: 28675 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 51bd4e1021eb209791aee5937061ac8805784483, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17626 5153 5049 5049
q2 2180 338 260 260
q3 10149 1378 816 816
q4 4679 462 335 335
q5 7428 804 522 522
q6 178 189 159 159
q7 749 822 592 592
q8 9253 1272 1045 1045
q9 5465 4187 4198 4187
q10 6840 1308 1015 1015
q11 443 241 226 226
q12 641 394 279 279
q13 18130 3515 3047 3047
q14 281 273 247 247
q15 q16 742 712 658 658
q17 1220 757 572 572
q18 6994 6165 6114 6114
q19 1098 982 615 615
q20 384 343 226 226
q21 5234 2439 2424 2424
q22 393 342 287 287
Total cold run time: 100107 ms
Total hot run time: 28675 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5964 5943 6122 5943
q2 737 639 567 567
q3 5626 5752 5305 5305
q4 2022 2177 1588 1588
q5 5833 5811 5680 5680
q6 244 181 148 148
q7 2248 2039 1751 1751
q8 3002 2632 2675 2632
q9 8007 7827 7522 7522
q10 3624 3580 3141 3141
q11 577 414 386 386
q12 655 664 495 495
q13 3229 3631 3055 3055
q14 287 294 281 281
q15 q16 670 688 613 613
q17 7832 7025 6861 6861
q18 13050 12137 12849 12137
q19 933 775 835 775
q20 2181 2162 1941 1941
q21 5633 4575 4747 4575
q22 511 444 458 444
Total cold run time: 72865 ms
Total hot run time: 65840 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4348382232
##########
be/src/exprs/function/function_topk_merge.cpp:
##########
@@ -0,0 +1,482 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <glog/logging.h>
+
+#include <cstddef>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "common/status.h"
+#include "core/assert_cast.h"
+#include "core/block/block.h"
+#include "core/block/column_numbers.h"
+#include "core/block/column_with_type_and_name.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/types.h"
+#include "exprs/function/function.h"
+#include "exprs/function/simple_function_factory.h"
+
+namespace doris {
+class FunctionContext;
+} // namespace doris
+
+namespace doris {
+
+namespace {
+
+/// Upper bound of k. Keep in sync with the aggregate's MAX_TOPK_K
+/// (aggregate_function_topk_count.h) and FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/// One row of an ARRAY<STRUCT<value, count>> argument. A NULL array is an empty view.
+struct TopKArrayView {
+ const IColumn* values = nullptr;
+ const ColumnInt64* counts = nullptr;
+ /// NULL bitmap of the array element (the pair struct).
+ const NullMap* null_map = nullptr;
+ /// NULL bitmap of the `value` field; NULL values are not part of the contract.
+ const NullMap* value_null_map = nullptr;
+ /// NULL bitmap of the `count` field; NULL counts are not part of the contract.
+ const NullMap* count_null_map = nullptr;
+ size_t begin = 0;
+ size_t end = 0;
+};
+
+TopKArrayView make_array_view(const IColumn& column, size_t row) {
+ TopKArrayView view;
+ const IColumn* array_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ array_column = &nullable->get_nested_column();
+ }
+ const auto& array = assert_cast<const ColumnArray&>(*array_column);
+ // offsets[i] is the cumulative size after row i, so row's range is [offsets[row-1], offsets[row]).
+ // offset_at(0) reads offsets[-1], which PaddedPODArray keeps as 0.
+ view.begin = array.offset_at(static_cast<ssize_t>(row));
+ view.end = array.offset_at(static_cast<ssize_t>(row) + 1);
+
+ const IColumn* pair_column = &array.get_data();
+ if (const auto* element_nullable = check_and_get_column<ColumnNullable>(pair_column)) {
+ view.null_map = &element_nullable->get_null_map_data();
+ pair_column = &element_nullable->get_nested_column();
+ }
+ const auto& pairs = assert_cast<const ColumnStruct&>(*pair_column);
+ view.values = &pairs.get_column(0);
+ // A cast may widen the `value` field to a nullable column; the entries themselves are never
+ // NULL, so only the nested column is used.
+ if (const auto* value_nullable = check_and_get_column<ColumnNullable>(view.values)) {
+ view.value_null_map = &value_nullable->get_null_map_data();
+ view.values = &value_nullable->get_nested_column();
+ }
+ // The same holds for `count`: a nested cast may have made the field nullable.
+ const auto* count_column = &pairs.get_column(1);
+ if (const auto* count_nullable = check_and_get_column<ColumnNullable>(count_column)) {
+ view.count_null_map = &count_nullable->get_null_map_data();
+ count_column = &count_nullable->get_nested_column();
+ }
+ view.counts = &assert_cast<const ColumnInt64&>(*count_column);
+ return view;
+}
+
+/// One scalar bound argument: NULL means the corresponding side was not truncated. `row` is the
+/// index of the current input row inside `value`: bounds are ordinary per-row columns, so a block
+/// can carry a different bound in every row.
+struct TopKBoundView {
+ const IColumn* value = nullptr;
+ size_t row = 0;
+ bool is_null = true;
+};
+
+TopKBoundView make_bound_view(const IColumn& column, size_t row) {
+ TopKBoundView view;
+ const IColumn* value_column = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ if (nullable->is_null_at(row)) {
+ return view;
+ }
+ value_column = &nullable->get_nested_column();
+ }
+ view.value = value_column;
+ view.row = row;
+ view.is_null = false;
+ return view;
+}
+
+int64_t read_int64(const IColumn& column, size_t row, const char* argument_name) {
+ const IColumn* nested = &column;
+ if (const auto* nullable = check_and_get_column<ColumnNullable>(&column)) {
+ // The contract declares these arguments as non-NULL BIGINT.
+ DORIS_CHECK(!nullable->is_null_at(row)) << argument_name << " must not be NULL";
Review Comment:
Please replace the `DORIS_CHECK` in `read_int64()` with an explicit `INVALID_ARGUMENT` exception:
```cpp
if (nullable->is_null_at(row)) {
throw Exception(ErrorCode::INVALID_ARGUMENT,
"{} must not be NULL", argument_name);
}
```
NULL `n`, `dn`, or `k` can reach this helper through the public SQL signature. They are invalid argument values and should fail the query with a normal parameter error. The current assertion reaches `LOG(FATAL)` in builds without `NDEBUG`, terminating the BE; release builds report `FATAL_ERROR`. Throwing `INVALID_ARGUMENT` also follows the existing `check_k()` and overflow-checking error-handling pattern.
Please cover NULL values for all three counters in both MIN/MAX variants, including nullable column inputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6366430963
# FE Regression Coverage Report
Increment line coverage ` 81.29% (139/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6367457391
# FE UT Coverage Report
Increment line coverage ` 28.07% (48/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6368558986
# BE Regression && UT Coverage Report
Increment line coverage `81.76% (269/329)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_e250347cd120427a408a4db7a0a55e83da7b098f_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 68.40% (31863/46585) |
| Line Coverage | 53.06% (345074/650355) |
| Region Coverage | 48.42% (282023/582490) |
| Branch Coverage | 49.90% (132269/265083) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6341166921
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6336822467
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6336830887
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #69456:
URL: https://github.com/apache/doris/pull/69456#discussion_r4348246466
##########
be/src/exprs/aggregate/aggregate_function_topk_count.h:
##########
@@ -0,0 +1,428 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <vector>
+
+#include "common/check.h"
+#include "common/compare.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/column/column.h"
+#include "core/column/column_array.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/primitive_type.h"
+#include "core/string_buffer.hpp"
+#include "core/types.h"
+#include "exprs/aggregate/aggregate_function.h"
+
+namespace doris {
+
+/// Upper bound of k, shared with FE TopKCountFunction.MAX_TOPK_K.
+constexpr int64_t MAX_TOPK_K = 65536;
+
+/**
+ * State shared by topk_min_count and topk_max_count.
+ *
+ * The entries are held as two parallel structures: a column with the distinct values (sorted
+ * ascending for MIN / descending for MAX, at most k entries) and their non-negative int64 counts.
+ * Only values with count > 0 are kept. `bound` is a one-row column with the extreme value seen
+ * when k == 0, or the k-th value once the list was truncated when k >= 1; `has_bound` is the
+ * truncation flag, and for k >= 1 `bound IS NULL` means "the list holds every distinct value".
+ */
+struct AggregateTopKCountData {
+ const DataTypePtr value_type;
+ bool has_k = false;
+ int64_t k = 0;
+ bool has_bound = false;
+ MutableColumnPtr bound_column;
+ MutableColumnPtr value_column;
+ std::vector<int64_t> counts;
+
+ explicit AggregateTopKCountData(const DataTypes& argument_types_)
+ : value_type(argument_types_[0]),
+ bound_column(value_type->create_column()),
+ value_column(value_type->create_column()) {}
+
+ AggregateTopKCountData(const AggregateTopKCountData&) = delete;
+ AggregateTopKCountData& operator=(const AggregateTopKCountData&) = delete;
+};
+
+/**
+ * topk_min_count(x, k) / topk_max_count(x, k).
+ *
+ * Result: STRUCT<values ARRAY<STRUCT<value T, count BIGINT>>, bound T NULL>. NULL inputs are
+ * ignored (the nullable-argument wrapper never calls add for them). `values` is ordered by value
+ * (ascending for MIN), holds at most k distinct values, and `bound` is the truncation point: the
+ * k-th value when more than k distinct values were seen, otherwise NULL.
+ *
+ * With k = 0 no list is built: `values` is empty and `bound` is the extreme value seen on this
+ * side (NULL when there was no non-NULL input), which lets the append-only IVM path compute
+ * MIN/MAX from the delta extreme alone.
+ */
+template <PrimitiveType T, bool IsMin>
+class AggregateFunctionTopKCount final
+ : public IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopKCount<T, IsMin>, true>,
+ public MultiExpression,
+ public NotNullableAggregateFunction {
+public:
+ using Base = IAggregateFunctionDataHelper<AggregateTopKCountData,
+ AggregateFunctionTopKCount<T, IsMin>, true>;
+
+ explicit AggregateFunctionTopKCount(const DataTypes& argument_types_) : Base(argument_types_) {
+ DCHECK_EQ(argument_types_.size(), 2);
+ }
+
+ String get_name() const override { return IsMin ? "topk_min_count" : "topk_max_count"; }
+
+ DataTypePtr get_return_type() const override {
+ return make_return_type(this->argument_types[0]);
+ }
+
+ void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+ Arena& arena) const override {
+ auto& state = Base::data(place);
+ // k is a constant expression, but only the first row carries it: the state remembers it so
+ // that merge can check both sides used the same capacity. A state without has_k is empty.
+ check_k(state, *columns[1], row_num);
+ add_value(state, *columns[0], row_num);
+ }
+
+ void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ const auto& other = Base::data(rhs);
+ // A partial state without has_k never saw a non-NULL row, so it carries no information.
+ if (!other.has_k) {
+ return;
+ }
+ if (!state.has_k) {
+ state.has_k = true;
+ state.k = other.k;
+ } else {
+ // Both sides are partial states of the same constant k.
+ DCHECK_EQ(state.k, other.k);
+ }
+ if (state.k == 0) {
+ merge_bound(state, other);
+ return;
+ }
+ merge_values(state, other);
+ }
+
+ void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+ const auto& state = Base::data(place);
+ buf.write_binary(state.has_k);
+ buf.write_binary(state.k);
+ buf.write_binary(state.has_bound);
+ if (state.has_bound) {
+ write_column(buf, *state.bound_column);
+ }
+ auto size = static_cast<int64_t>(state.counts.size());
+ buf.write_binary(size);
+ write_column(buf, *state.value_column);
+ buf.write(reinterpret_cast<const char*>(state.counts.data()),
+ state.counts.size() * sizeof(int64_t));
+ }
+
+ void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+ Arena&) const override {
+ auto& state = Base::data(place);
+ buf.read_binary(state.has_k);
+ buf.read_binary(state.k);
+ buf.read_binary(state.has_bound);
+ if (state.has_bound) {
+ state.bound_column = read_column(buf);
+ } else {
+ state.bound_column = state.value_type->create_column();
+ }
+ int64_t size = 0;
+ buf.read_binary(size);
+ state.value_column = read_column(buf);
+ DCHECK_EQ(static_cast<int64_t>(state.value_column->size()), size);
+ state.counts.resize(size);
+ buf.read(reinterpret_cast<char*>(state.counts.data()), size * sizeof(int64_t));
+ }
+
+ void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+ const auto& state = Base::data(place);
+ auto& result_struct = assert_cast<ColumnStruct&>(to);
+ auto& values_column = assert_cast<ColumnArray&>(result_struct.get_column(0));
+ // The array element is nullable in the planned type (ARRAY<STRUCT<...>> defaults to
+ // contains_null = true), but a produced entry never is.
+ auto& pair_nullable = assert_cast<ColumnNullable&>(values_column.get_data());
+ auto& pair_column = assert_cast<ColumnStruct&>(pair_nullable.get_nested_column());
+ auto& value_column = pair_column.get_column(0);
+ auto& count_column = assert_cast<ColumnInt64&>(pair_column.get_column(1));
+ for (size_t i = 0; i != state.counts.size(); ++i) {
+ value_column.insert_from(*state.value_column, i);
+ count_column.get_data().push_back(state.counts[i]);
+ pair_nullable.get_null_map_data().push_back(0);
+ }
+ values_column.get_offsets().push_back(pair_column.size());
+
+ auto& bound_column = assert_cast<ColumnNullable&>(result_struct.get_column(1));
+ if (state.has_bound) {
+ bound_column.get_nested_column().insert_from(*state.bound_column, 0);
+ bound_column.get_null_map_data().push_back(0);
+ } else {
+ bound_column.get_nested_column().insert_default();
+ bound_column.get_null_map_data().push_back(1);
+ }
+ }
+
+ static DataTypePtr make_return_type(const DataTypePtr& value_type) {
+ auto pair_type = std::make_shared<DataTypeStruct>(
+ DataTypes {value_type, std::make_shared<DataTypeInt64>()},
+ Strings {"value", "count"});
+ // ArrayType defaults contains_null to true on the FE side, so the pair element is nullable
+ // in the planned type even though the aggregate never produces a NULL entry.
+ auto values_type = std::make_shared<DataTypeArray>(make_nullable(pair_type));
+ return std::make_shared<DataTypeStruct>(DataTypes {values_type, make_nullable(value_type)},
+ Strings {"values", "bound"});
+ }
+
+private:
+ static void check_k(AggregateTopKCountData& state, const IColumn& count_column,
+ size_t row_num) {
+ DCHECK(!count_column.is_nullable());
+ int64_t value = assert_cast<const ColumnInt64&, TypeCheckOnRelease::DISABLE>(count_column)
+ .get_element(row_num);
+ if (!state.has_k) {
+ if (UNLIKELY(value < 0 || value > MAX_TOPK_K)) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "the count of topk_min_count and topk_max_count must be a constant "
+ "between 0 and {}, but got {}",
+ MAX_TOPK_K, value);
+ }
+ state.has_k = true;
+ state.k = value;
+ } else {
+ // k comes from the same constant expression on every row and every partial state.
+ DCHECK_EQ(state.k, value);
+ }
+ }
+
+ /// Ordering of the values. FLOAT/DOUBLE use CompareFloat so that NaN sorts greatest and is
+ /// distinct from every number, which keeps the list mergeable and the top-K deterministic;
+ /// every other type uses the column's compare_at, i.e. exactly the MIN/MAX comparison.
+ static int compare_at(const IColumn& lhs, size_t lhs_row, const IColumn& rhs, size_t rhs_row) {
+ if constexpr (is_float_or_double(T)) {
+ using ColumnType = typename PrimitiveTypeTraits<T>::ColumnType;
+ const auto& left =
+ assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(lhs).get_data();
+ const auto& right =
+ assert_cast<const ColumnType&, TypeCheckOnRelease::DISABLE>(rhs).get_data();
+ return CompareFloat(left[lhs_row], right[rhs_row]);
+ } else {
+ return lhs.compare_at(lhs_row, rhs_row, rhs, 1);
+ }
+ }
+
+ static int64_t checked_add(int64_t lhs, int64_t rhs) {
+ int64_t result = 0;
+ if (UNLIKELY(__builtin_add_overflow(lhs, rhs, &result))) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT,
+ "topk_min_count/topk_max_count value count overflow");
+ }
+ return result;
+ }
+
+ static void set_bound_from_value(AggregateTopKCountData& state, size_t index) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(*state.value_column, index);
+ state.has_bound = true;
+ }
+
+ /// Inserts (value, count) at `pos`, keeping the value column sorted by value.
+ static void insert_entry(AggregateTopKCountData& state, size_t pos, const IColumn& column,
+ size_t row_num, int64_t count) {
+ auto new_values = state.value_column->clone_empty();
+ new_values->reserve(state.value_column->size() + 1);
+ for (size_t i = 0; i < pos; ++i) {
+ new_values->insert_from(*state.value_column, i);
+ }
+ new_values->insert_from(column, row_num);
+ for (size_t i = pos; i < state.value_column->size(); ++i) {
+ new_values->insert_from(*state.value_column, i);
+ }
+ state.value_column = std::move(new_values);
+ state.counts.insert(state.counts.begin() + pos, count);
+ }
+
+ /// k == 0 fast path: only the extreme value is tracked.
+ static void add_value(AggregateTopKCountData& state, const IColumn& column, size_t row_num) {
+ if (state.k == 0) {
+ if (!state.has_bound) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(column, row_num);
+ state.has_bound = true;
+ return;
+ }
+ int cmp = compare_at(column, row_num, *state.bound_column, 0);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ state.bound_column->clear();
+ state.bound_column->insert_from(column, row_num);
+ }
+ return;
+ }
+
+ // The list is ascending for MIN and descending for MAX, so the insertion point is where
+ // the stored values stop being better than the new one.
+ size_t left = 0;
+ size_t right = state.value_column->size();
+ while (left < right) {
+ size_t mid = left + (right - left) / 2;
+ int cmp = compare_at(*state.value_column, mid, column, row_num);
+ if (IsMin ? cmp < 0 : cmp > 0) {
+ left = mid + 1;
+ } else {
+ right = mid;
+ }
+ }
+ if (left < state.value_column->size() &&
+ compare_at(*state.value_column, left, column, row_num) == 0) {
+ state.counts[left] = checked_add(state.counts[left], 1);
+ return;
+ }
+
+ insert_entry(state, left, column, row_num, 1);
Review Comment:
The full-list rejection fast path is fixed at `e4acad346d`. There is still an avoidable rebuild while filling the candidate list: `insert_entry()` clones the value column and copies every existing value even when `pos == state.value_column->size()`.
For `topk_min_count(x, 65536)` with distinct ascending input `1..65536`, every insertion is at the end and the list has room. Filling one group's state therefore copies `0 + 1 + ... + 65535 = 2,147,450,880` existing elements, plus repeated column allocations. MAX with descending input has the symmetric case. This is a source-derived operation count, not a measured runtime result.
Please append directly before the clone/copy path:
```cpp
if (pos == state.value_column->size()) {
state.value_column->insert_from(column, row_num);
state.counts.push_back(count);
return;
}
```
The state already owns a mutable value column, and `insert_from()` appends to it. The caller already rejects worse values when the list is full, so this handles the remaining tail-insertion case without rebuilding the retained prefix. A benchmark with sorted input and large k would quantify the benefit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](ivm) Add TOPK_MIN_COUNT/TOPK_MAX_COUNT and TOPK_MERGE_MIN/MAX functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6336832339
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6518931496
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6516900585
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6433263463
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6431551471
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6498729826
# FE Regression Coverage Report
Increment line coverage ` 81.29% (139/171)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_29fb3c47891613984533a1602cf3c9e6d15e2bb2_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_29fb3c47891613984533a1602cf3c9e6d15e2bb2_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6513606108
Codex automated review failed and did not complete.
Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/40988566702
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6435614814
# FE Regression Coverage Report
Increment line coverage ` 76.80% (139/181)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69456_5309c29db2b57c559711d38b6621f5c934c95d3a_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69456_5309c29db2b57c559711d38b6621f5c934c95d3a_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6470646057
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6492262537
<details>
<summary>ClickBench: <b>Total hot run time: 24.27 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d9d17e663629a2bc41b45deb64a2e384b1f41df2, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.04
query3 0.26 0.09 0.09
query4 1.66 0.11 0.10
query5 0.27 0.24 0.24
query6 1.17 0.68 0.69
query7 0.04 0.01 0.00
query8 0.05 0.04 0.04
query9 0.40 0.37 0.34
query10 0.58 0.57 0.60
query11 0.20 0.14 0.14
query12 0.19 0.14 0.14
query13 0.48 0.50 0.49
query14 1.03 1.01 1.02
query15 0.63 0.59 0.60
query16 0.34 0.34 0.31
query17 1.16 1.16 1.18
query18 0.22 0.18 0.20
query19 2.13 2.00 1.95
query20 0.02 0.01 0.01
query21 15.47 0.19 0.15
query22 4.97 0.06 0.05
query23 16.27 0.33 0.13
query24 3.00 0.46 0.36
query25 0.10 0.06 0.03
query26 0.89 0.21 0.16
query27 0.05 0.05 0.04
query28 4.02 0.87 0.36
query29 12.51 4.11 3.24
query30 0.29 0.12 0.11
query31 2.78 0.56 0.33
query32 3.23 0.60 0.50
query33 3.15 3.28 3.21
query34 15.47 4.08 3.39
query35 3.31 3.36 3.34
query36 0.55 0.44 0.43
query37 0.09 0.06 0.07
query38 0.05 0.03 0.04
query39 0.04 0.03 0.03
query40 0.18 0.15 0.14
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.05 0.04 0.03
Total cold run time: 97.53 s
Total hot run time: 24.27 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6494661866
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6468187600
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](function) Add MIN_TOPK/MAX_TOPK and ARRAY_SORTED_MERGE functions [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69456:
URL: https://github.com/apache/doris/pull/69456#issuecomment-6498579913
<details>
<summary>ClickBench: <b>Total hot run time: 24.38 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 29fb3c47891613984533a1602cf3c9e6d15e2bb2, data reload: false
query1 0.01 0.01 0.01
query2 0.11 0.05 0.04
query3 0.26 0.08 0.08
query4 1.60 0.10 0.10
query5 0.23 0.23 0.24
query6 1.17 0.69 0.69
query7 0.04 0.01 0.00
query8 0.06 0.04 0.04
query9 0.40 0.35 0.35
query10 0.59 0.61 0.57
query11 0.20 0.14 0.13
query12 0.19 0.15 0.15
query13 0.50 0.49 0.49
query14 1.01 1.03 1.00
query15 0.63 0.60 0.61
query16 0.34 0.34 0.36
query17 1.15 1.15 1.13
query18 0.21 0.20 0.19
query19 2.12 2.00 2.01
query20 0.02 0.01 0.01
query21 15.50 0.19 0.13
query22 4.86 0.05 0.06
query23 16.26 0.32 0.14
query24 3.02 0.51 0.42
query25 0.14 0.05 0.06
query26 1.12 0.21 0.16
query27 0.05 0.05 0.05
query28 3.88 0.82 0.37
query29 12.46 4.13 3.29
query30 0.28 0.12 0.10
query31 2.77 0.54 0.32
query32 3.22 0.58 0.50
query33 3.15 3.22 3.17
query34 15.34 4.01 3.39
query35 3.34 3.32 3.36
query36 0.56 0.44 0.42
query37 0.10 0.06 0.07
query38 0.04 0.04 0.03
query39 0.03 0.03 0.03
query40 0.18 0.14 0.14
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.04
Total cold run time: 97.3 s
Total hot run time: 24.38 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org