You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "linrrzqqq (via GitHub)" <gi...@apache.org> on 2026/04/15 07:18:30 UTC

[PR] [Opt](func) Improve AggFunc Percentile performance [doris]

linrrzqqq opened a new pull request, #62520:
URL: https://github.com/apache/doris/pull/62520

   ### What problem does this PR solve?
   
   Issue Number: close #xxx
   
   Related PR: #xxx
   
   Problem Summary:
   
   ### Key Optimizations
   - Reworked exact percentile state into a simpler `values + PercentileLevels + inited_flag` layout.
   - Removed the old Counts-based exact percentile path and replaced it with a lightweight `PercentileLevels` metadata structure.
   - Reduced add-path overhead by directly appending values into a contiguous buffer and initializing percentile levels only once.
   - Optimized multi-quantile finalize by sorting quantiles through permutation once, then reusing incremental nth_element selection.
   - Simplified serialization / merge logic to avoid extra exact-state overhead.
   
   ### Performance
   before:
   ```text
   Doris> select percentile(FUniqID, 0.91) from hits_100m;
   +---------------------------+
   | percentile(FUniqID, 0.91) |
   +---------------------------+
   |     8.749975206154981e+18 |
   +---------------------------+
   1 row in set (6.149 sec)
   
   Doris> select percentile_array(FUniqID, [0.12, 0.23, 0.34, 0.45, 0.56]) from hits_100m;
   +------------------------------------------------------------------------------------------------+
   | percentile_array(FUniqID, [0.12, 0.23, 0.34, 0.45, 0.56])                                      |
   +------------------------------------------------------------------------------------------------+
   | [0, 5.097819197233503e+18, 5.68906185719365e+18, 6.281280528073601e+18, 6.869583632113269e+18] |
   +------------------------------------------------------------------------------------------------+
   1 row in set (28.984 sec)
   
   Doris> select percentile_array(FUniqID, [0.12, 0.23, 0.34, 0.45, 0.56, 0.67, 0.78, 0.89, 0.91]) from hits_100m;
   ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[E-3113]string column length is too large: total_length=7200000149, element_number=0, rows=0
   ```
   
   now
   ```text
   Doris> select percentile(FUniqID, 0.91) from hits_100m;
   +---------------------------+
   | percentile(FUniqID, 0.91) |
   +---------------------------+
   |     8.749975206154981e+18 |
   +---------------------------+
   1 row in set (2.107 sec)
   
   Doris> select percentile_array(FUniqID, [0.12, 0.23, 0.34, 0.45, 0.56]) from hits_100m;
   +------------------------------------------------------------------------------------------------+
   | percentile_array(FUniqID, [0.12, 0.23, 0.34, 0.45, 0.56])                                      |
   +------------------------------------------------------------------------------------------------+
   | [0, 5.097819197233503e+18, 5.68906185719365e+18, 6.281280528073601e+18, 6.869583632113269e+18] |
   +------------------------------------------------------------------------------------------------+
   1 row in set (5.903 sec)
   
   Doris> select percentile_array(FUniqID, [0.12, 0.23, 0.34, 0.45, 0.56, 0.67, 0.78, 0.89, 0.91]) from hits_100m;
   +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
   | percentile_array(FUniqID, [0.12, 0.23, 0.34, 0.45, 0.56, 0.67, 0.78, 0.89, 0.91])                                                                                                          |
   +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
   | [0, 5.097819197233503e+18, 5.68906185719365e+18, 6.281280528073601e+18, 6.869583632113269e+18, 7.459044216605808e+18, 8.044902013692362e+18, 8.640523572825436e+18, 8.749975206154981e+18] |
   +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
   1 row in set (8.866 sec)
   ```
   
   ### Release note
   
   None
   
   ### Check List (For Author)
   
   - Test <!-- At least one of them must be included. -->
       - [ ] Regression test
       - [x] Unit Test
       - [ ] Manual test (add detailed scripts or steps below)
       - [ ] No need to test or manual test. Explain why:
           - [ ] This is a refactor/code format and no logic has been changed.
           - [ ] Previous test can cover this change.
           - [ ] No code files have been changed.
           - [ ] Other reason <!-- Add your reason?  -->
   
   - Behavior changed:
       - [x] No.
       - [ ] Yes. <!-- Explain the behavior change -->
   
   - Does this need documentation?
       - [x] No.
       - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4295316131

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4295316817

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4280160544

   # BE Regression && UT Coverage Report
   Increment line coverage `85.10% (257/302)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62520_27e26098db0ce456a42c70dd890d08e8dfbdd59e_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62520_27e26098db0ce456a42c70dd890d08e8dfbdd59e_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 71.86% (26897/37430) |
   | Line Coverage     | 55.07% (285744/518911) |
   | Region Coverage   | 52.06% (236210/453743) |
   | Branch Coverage   | 53.45% (102008/190830) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4278574900

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4304185737

   # FE Regression Coverage Report
   Increment line coverage `  100.00% (5/5)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62520_ac8c96277ce1fa8881d3890362b9dd5a6cc52f24_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62520_ac8c96277ce1fa8881d3890362b9dd5a6cc52f24_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4251029045

   # BE Regression && UT Coverage Report
   Increment line coverage `82.76% (192/232)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62520_809d8bcace5d8da87823bb14fe57754752d24ddc_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62520_809d8bcace5d8da87823bb14fe57754752d24ddc_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 73.68% (27430/37227) |
   | Line Coverage     | 57.31% (296122/516680) |
   | Region Coverage   | 54.60% (246968/452351) |
   | Branch Coverage   | 56.23% (106962/190230) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4266772505

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4278747314

   # Cloud UT Coverage Report
   Increment line coverage ` ` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/27e26098db0ce456a42c70dd890d08e8dfbdd59e_27e26098db0ce456a42c70dd890d08e8dfbdd59e_cloud/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/27e26098db0ce456a42c70dd890d08e8dfbdd59e_27e26098db0ce456a42c70dd890d08e8dfbdd59e_cloud/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 78.02% (1842/2361) |
   | Line Coverage     | 64.61% (32924/50958) |
   | Region Coverage   | 65.20% (16321/25034) |
   | Branch Coverage   | 55.76% (8708/15616) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4304611962

   # BE Regression && UT Coverage Report
   Increment line coverage `86.09% (260/302)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62520_ac8c96277ce1fa8881d3890362b9dd5a6cc52f24_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62520_ac8c96277ce1fa8881d3890362b9dd5a6cc52f24_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 71.61% (26836/37476) |
   | Line Coverage     | 53.96% (280367/519576) |
   | Region Coverage   | 47.33% (215645/455596) |
   | Branch Coverage   | 50.64% (97591/192726) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4293437736

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on code in PR #62520:
URL: https://github.com/apache/doris/pull/62520#discussion_r3097631944


##########
be/src/exprs/aggregate/aggregate_function_percentile.h:
##########
@@ -436,6 +429,194 @@ struct PercentileState {
     }
 };
 
+template <PrimitiveType T>
+struct PercentileExactState {
+    using ValueType = typename PrimitiveTypeTraits<T>::CppType;
+    static constexpr size_t bytes_in_arena = 64 - sizeof(PODArray<ValueType>);
+    using Array = PODArrayWithStackMemory<ValueType, bytes_in_arena>;
+
+    void add_single_range(const ValueType* data, size_t count, double quantile) {
+        if (!inited_flag) {
+            _set_single_level(quantile);
+            inited_flag = true;
+        }
+        _append(data, count);
+    }
+
+    void add_many_range(const ValueType* data, size_t count,
+                        const PaddedPODArray<Float64>& quantiles_data, const NullMap& null_maps,
+                        size_t start, int64_t arg_size) {
+        if (!inited_flag) {
+            _set_many_levels(quantiles_data, null_maps, start, arg_size);
+            inited_flag = true;
+        }
+        if (levels.empty()) {
+            return;
+        }
+        _append(data, count);
+    }
+
+    void write(BufferWritable& buf) const {
+        buf.write_binary(inited_flag);
+        if (!inited_flag) {
+            return;
+        }
+
+        levels.write(buf);
+        size_t size = values.size();
+        buf.write_binary(size);
+        if (size > 0) {
+            buf.write(reinterpret_cast<const char*>(values.data()), sizeof(ValueType) * size);
+        }
+    }
+
+    void read(BufferReadable& buf) {
+        reset();
+        buf.read_binary(inited_flag);
+        if (!inited_flag) {
+            return;
+        }
+
+        levels.read(buf);
+        size_t size = 0;
+        buf.read_binary(size);
+        values.resize(size);
+        if (size > 0) {
+            auto raw = buf.read(sizeof(ValueType) * size);
+            memcpy(values.data(), raw.data, raw.size);
+        }
+    }
+
+    void merge(const PercentileExactState& rhs) {
+        if (!rhs.inited_flag) {
+            return;
+        }
+
+        if (!inited_flag) {
+            levels = rhs.levels;
+            inited_flag = true;
+        } else {
+            levels.merge(rhs.levels);
+        }
+        _append(rhs.values.data(), rhs.values.size());
+    }
+
+    void reset() {
+        values.clear();
+        levels.clear();
+        inited_flag = false;
+    }
+
+    double get() const {
+        if (!inited_flag || levels.empty() || values.empty()) {
+            return 0.0;
+        }
+
+        DCHECK_EQ(levels.quantiles.size(), 1);
+        return _get_float(levels.quantiles[0]);
+    }
+
+    void insert_result_into(IColumn& to) const {
+        auto& column_data = assert_cast<ColumnFloat64&>(to).get_data();
+        if (!inited_flag || levels.empty() || values.empty()) {
+            return;
+        }
+
+        size_t old_size = column_data.size();
+        size_t size = levels.quantiles.size();
+        column_data.resize(old_size + size);
+        auto* result = column_data.data() + old_size;
+
+        if (values.size() == 1) {
+            for (size_t i = 0; i < size; ++i) {
+                result[i] = static_cast<double>(values.front());
+            }
+            return;
+        }
+
+        size_t prev_index = 0;
+        const auto& quantiles = levels.quantiles;
+        const auto& permutation = levels.permutation;
+        for (size_t i = 0; i < size; ++i) {
+            auto level_index = permutation[i];
+            auto level = quantiles[level_index];
+            double u = static_cast<double>(values.size() - 1) * level;
+            auto index = static_cast<size_t>(u);
+
+            if (index + 1 >= values.size()) {
+                result[level_index] =
+                        static_cast<double>(*std::max_element(values.begin(), values.end()));
+            } else {
+                std::nth_element(values.begin() + prev_index, values.begin() + index, values.end());
+                auto* nth_elem = std::min_element(values.begin() + index + 1, values.end());
+                result[level_index] =
+                        static_cast<double>(values[index]) +
+                        (u - static_cast<double>(index)) * (static_cast<double>(*nth_elem) -
+                                                            static_cast<double>(values[index]));
+                prev_index = index;
+            }
+        }
+    }
+
+private:
+    void _set_single_level(double quantile) {
+        DCHECK(levels.empty());
+        check_quantile(quantile);
+        levels.quantiles.push_back(quantile);
+        levels.permutation.push_back(0);
+    }
+
+    void _set_many_levels(const PaddedPODArray<Float64>& quantiles_data, const NullMap& null_maps,
+                          size_t start, int64_t arg_size) {
+        DCHECK(levels.empty());
+        size_t size = cast_set<size_t>(arg_size);
+        levels.quantiles.resize(size);
+        levels.permutation.resize(size);
+        for (size_t i = 0; i < size; ++i) {
+            if (null_maps[start + i]) {
+                throw Exception(ErrorCode::INVALID_ARGUMENT,
+                                "quantiles in func percentile_array should not have null");
+            }
+            check_quantile(quantiles_data[start + i]);
+            levels.quantiles[i] = quantiles_data[start + i];
+            levels.permutation[i] = i;
+        }
+        levels.sort_permutation();

Review Comment:
   updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4264998037

   /review
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4250069188

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4258955642

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #62520:
URL: https://github.com/apache/doris/pull/62520#discussion_r3096657733


##########
be/src/exprs/aggregate/aggregate_function_percentile.h:
##########
@@ -436,6 +429,194 @@ struct PercentileState {
     }
 };
 
+template <PrimitiveType T>
+struct PercentileExactState {
+    using ValueType = typename PrimitiveTypeTraits<T>::CppType;
+    static constexpr size_t bytes_in_arena = 64 - sizeof(PODArray<ValueType>);
+    using Array = PODArrayWithStackMemory<ValueType, bytes_in_arena>;
+
+    void add_single_range(const ValueType* data, size_t count, double quantile) {
+        if (!inited_flag) {
+            _set_single_level(quantile);
+            inited_flag = true;
+        }
+        _append(data, count);
+    }
+
+    void add_many_range(const ValueType* data, size_t count,
+                        const PaddedPODArray<Float64>& quantiles_data, const NullMap& null_maps,
+                        size_t start, int64_t arg_size) {
+        if (!inited_flag) {
+            _set_many_levels(quantiles_data, null_maps, start, arg_size);
+            inited_flag = true;
+        }
+        if (levels.empty()) {
+            return;
+        }
+        _append(data, count);
+    }
+
+    void write(BufferWritable& buf) const {
+        buf.write_binary(inited_flag);
+        if (!inited_flag) {
+            return;
+        }
+
+        levels.write(buf);
+        size_t size = values.size();
+        buf.write_binary(size);
+        if (size > 0) {
+            buf.write(reinterpret_cast<const char*>(values.data()), sizeof(ValueType) * size);
+        }
+    }
+
+    void read(BufferReadable& buf) {
+        reset();
+        buf.read_binary(inited_flag);
+        if (!inited_flag) {
+            return;
+        }
+
+        levels.read(buf);
+        size_t size = 0;
+        buf.read_binary(size);
+        values.resize(size);
+        if (size > 0) {
+            auto raw = buf.read(sizeof(ValueType) * size);
+            memcpy(values.data(), raw.data, raw.size);
+        }
+    }
+
+    void merge(const PercentileExactState& rhs) {
+        if (!rhs.inited_flag) {
+            return;
+        }
+
+        if (!inited_flag) {
+            levels = rhs.levels;
+            inited_flag = true;
+        } else {
+            levels.merge(rhs.levels);
+        }
+        _append(rhs.values.data(), rhs.values.size());
+    }
+
+    void reset() {
+        values.clear();
+        levels.clear();
+        inited_flag = false;
+    }
+
+    double get() const {
+        if (!inited_flag || levels.empty() || values.empty()) {
+            return 0.0;
+        }
+
+        DCHECK_EQ(levels.quantiles.size(), 1);
+        return _get_float(levels.quantiles[0]);
+    }
+
+    void insert_result_into(IColumn& to) const {
+        auto& column_data = assert_cast<ColumnFloat64&>(to).get_data();
+        if (!inited_flag || levels.empty() || values.empty()) {

Review Comment:
   为什么上面的get返回的是0.0,这里是没有return呢



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4273162221

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4273714841

   # BE Regression && UT Coverage Report
   Increment line coverage `85.10% (257/302)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62520_689e2b61e2f51b6f52968ff7bbf27e9b15ba222b_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62520_689e2b61e2f51b6f52968ff7bbf27e9b15ba222b_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 73.72% (27535/37352) |
   | Line Coverage     | 57.46% (298038/518686) |
   | Region Coverage   | 54.71% (248211/453708) |
   | Branch Coverage   | 56.26% (107328/190776) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62520:
URL: https://github.com/apache/doris/pull/62520#discussion_r3123647014


##########
be/src/exprs/aggregate/aggregate_function_simple_factory.h:
##########
@@ -135,6 +135,15 @@ class AggregateFunctionSimpleFactory {
         if (function_alias.contains(name)) {
             name_str = function_alias[name];
         }
+
+        if (attr.new_version_percentile) {

Review Comment:
   `temporary_function_update(be_version, name_str)` still runs before this `new_version_percentile` remap. That means a FE that temporarily lowers `be_exec_version` for rolling upgrade or compatibility mode will still hit the old `percentile` and `percentile_array` restriction before these public names can be rewritten to `percentile_v2` and `percentile_array_v2`. Since FE now always sets `new_version_percentile=true`, this remap needs to happen before the old-version compatibility lookup. The scalar-function factory already uses that ordering for the `unix_timestamp` upgrade path.



##########
be/src/exprs/aggregate/aggregate_function_percentile.h:
##########
@@ -568,4 +748,194 @@ class AggregateFunctionPercentileArray final
     }
 };
 
-} // namespace doris
\ No newline at end of file
+template <PrimitiveType T>
+class AggregateFunctionPercentileV2 final
+        : public IAggregateFunctionDataHelper<PercentileExactState<T>,
+                                              AggregateFunctionPercentileV2<T>>,
+          MultiExpression,
+          NullableAggregateFunction {
+public:
+    using ColVecType = typename PrimitiveTypeTraits<T>::ColumnType;
+    using Base =
+            IAggregateFunctionDataHelper<PercentileExactState<T>, AggregateFunctionPercentileV2<T>>;
+    AggregateFunctionPercentileV2(const DataTypes& argument_types_) : Base(argument_types_) {}
+
+    String get_name() const override { return "percentile_v2"; }
+
+    DataTypePtr get_return_type() const override { return std::make_shared<DataTypeFloat64>(); }
+
+    void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+             Arena&) const override {
+        const auto& sources =
+                assert_cast<const ColVecType&, TypeCheckOnRelease::DISABLE>(*columns[0]);
+        const auto& quantile =
+                assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(*columns[1]);
+        AggregateFunctionPercentileV2::data(place).add_single_range(&sources.get_data()[row_num], 1,
+                                                                    quantile.get_data()[0]);
+    }
+
+    void add_batch_single_place(size_t batch_size, AggregateDataPtr place, const IColumn** columns,
+                                Arena&) const override {
+        const auto& sources =
+                assert_cast<const ColVecType&, TypeCheckOnRelease::DISABLE>(*columns[0]);
+        const auto& quantile =
+                assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(*columns[1]);
+        DCHECK_EQ(sources.get_data().size(), batch_size);
+        AggregateFunctionPercentileV2::data(place).add_single_range(
+                sources.get_data().data(), batch_size, quantile.get_data()[0]);
+    }
+
+    void add_batch_range(size_t batch_begin, size_t batch_end, AggregateDataPtr place,
+                         const IColumn** columns, Arena&, bool has_null) override {
+        const auto& sources =
+                assert_cast<const ColVecType&, TypeCheckOnRelease::DISABLE>(*columns[0]);
+        const auto& quantile =
+                assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(*columns[1]);
+        DCHECK(!has_null);
+        AggregateFunctionPercentileV2::data(place).add_single_range(
+                sources.get_data().data() + batch_begin, batch_end - batch_begin + 1,
+                quantile.get_data()[0]);
+    }
+
+    void reset(AggregateDataPtr __restrict place) const override {
+        AggregateFunctionPercentileV2::data(place).reset();
+    }
+
+    void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+               Arena&) const override {
+        AggregateFunctionPercentileV2::data(place).merge(AggregateFunctionPercentileV2::data(rhs));
+    }
+
+    void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+        AggregateFunctionPercentileV2::data(place).write(buf);
+    }
+
+    void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+                     Arena&) const override {
+        AggregateFunctionPercentileV2::data(place).read(buf);
+    }
+
+    void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+        auto& col = assert_cast<ColumnFloat64&>(to);
+        col.insert_value(AggregateFunctionPercentileV2::data(place).get());
+    }
+};
+
+template <PrimitiveType T>
+class AggregateFunctionPercentileArrayV2 final
+        : public IAggregateFunctionDataHelper<PercentileExactState<T>,
+                                              AggregateFunctionPercentileArrayV2<T>>,
+          MultiExpression,
+          NotNullableAggregateFunction {
+public:
+    using ColVecType = typename PrimitiveTypeTraits<T>::ColumnType;
+    using Base = IAggregateFunctionDataHelper<PercentileExactState<T>,
+                                              AggregateFunctionPercentileArrayV2<T>>;
+    AggregateFunctionPercentileArrayV2(const DataTypes& argument_types_) : Base(argument_types_) {}
+
+    String get_name() const override { return "percentile_array_v2"; }
+
+    DataTypePtr get_return_type() const override {
+        return std::make_shared<DataTypeArray>(make_nullable(std::make_shared<DataTypeFloat64>()));
+    }
+
+    void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+             Arena&) const override {
+        const auto& sources =
+                assert_cast<const ColVecType&, TypeCheckOnRelease::DISABLE>(*columns[0]);
+        const auto& quantile_array =
+                assert_cast<const ColumnArray&, TypeCheckOnRelease::DISABLE>(*columns[1]);
+        const auto& offset_column_data = quantile_array.get_offsets();
+        const auto& null_maps = assert_cast<const ColumnNullable&, TypeCheckOnRelease::DISABLE>(
+                                        quantile_array.get_data())
+                                        .get_null_map_data();
+        const auto& nested_column = assert_cast<const ColumnNullable&, TypeCheckOnRelease::DISABLE>(
+                                            quantile_array.get_data())
+                                            .get_nested_column();
+        const auto& nested_column_data =
+                assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(nested_column);
+        size_t start = row_num == 0 ? 0 : offset_column_data[row_num - 1];
+        AggregateFunctionPercentileArrayV2::data(place).add_many_range(
+                &sources.get_data()[row_num], 1, nested_column_data.get_data(), null_maps, start,
+                cast_set<int64_t>(offset_column_data[row_num] - start));
+    }
+
+    void add_batch_single_place(size_t batch_size, AggregateDataPtr place, const IColumn** columns,
+                                Arena&) const override {
+        const auto& sources =
+                assert_cast<const ColVecType&, TypeCheckOnRelease::DISABLE>(*columns[0]);
+        const auto& quantile_array =
+                assert_cast<const ColumnArray&, TypeCheckOnRelease::DISABLE>(*columns[1]);
+        const auto& offset_column_data = quantile_array.get_offsets();
+        const auto& null_maps = assert_cast<const ColumnNullable&, TypeCheckOnRelease::DISABLE>(
+                                        quantile_array.get_data())
+                                        .get_null_map_data();
+        const auto& nested_column = assert_cast<const ColumnNullable&, TypeCheckOnRelease::DISABLE>(
+                                            quantile_array.get_data())
+                                            .get_nested_column();
+        const auto& nested_column_data =
+                assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(nested_column);
+        DCHECK_EQ(sources.get_data().size(), batch_size);
+        AggregateFunctionPercentileArrayV2::data(place).add_many_range(

Review Comment:
   This batch fast path assumes every row in the block shares row 0 values for the quantile array: it always passes nested quantiles and null-map data starting at offset 0, and it uses `offset_column_data[0]` as the array length for the whole batch. That is only safe if FE guarantees a constant second argument. Nereids does in `PercentileArray.checkLegalityBeforeTypeCoercion()`, but legacy FE still has no such check, and `SessionVariable.toThrift()` now enables `new_version_percentile=true` globally. A legacy no-`GROUP BY` query like `select percentile_array(v, arr_col) from t` will therefore go through `AggFnEvaluator::execute_single_add()` and silently use the first row levels for the whole block. The old `percentile_array` implementation did not override `add_batch_single_place()`, so this is a new regression.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on code in PR #62520:
URL: https://github.com/apache/doris/pull/62520#discussion_r3098644140


##########
be/src/exprs/aggregate/aggregate_function_percentile.cpp:
##########
@@ -74,8 +82,28 @@ void register_percentile_approx_old_function(AggregateFunctionSimpleFactory& fac
 }
 
 void register_aggregate_function_percentile_old(AggregateFunctionSimpleFactory& factory) {
-    BeExecVersionManager::registe_restrict_function_compatibility("percentile");
-    BeExecVersionManager::registe_restrict_function_compatibility("percentile_array");
+    using creator = creator_with_type_list<TYPE_TINYINT, TYPE_SMALLINT, TYPE_INT, TYPE_BIGINT,
+                                           TYPE_LARGEINT, TYPE_FLOAT, TYPE_DOUBLE>;
+    factory.register_alternative_function("percentile",

Review Comment:
   use `query_option` based upgrade



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62520:
URL: https://github.com/apache/doris/pull/62520#discussion_r3092530406


##########
fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinAggregateFunctions.java:
##########
@@ -173,11 +175,13 @@ private BuiltinAggregateFunctions() {
                 agg(OrthogonalBitmapIntersect.class, "orthogonal_bitmap_intersect"),
                 agg(OrthogonalBitmapIntersectCount.class, "orthogonal_bitmap_intersect_count"),
                 agg(OrthogonalBitmapUnionCount.class, "orthogonal_bitmap_union_count"),

Review Comment:
   Once `percentile` / `percentile_cont` bind to `PercentileV2` here, the main optimization path stops triggering: `MergePercentileToArray.collectFuncMap()` still only matches the v1 `Percentile` class. So `select percentile(x, 0.1), percentile(x, 0.2) ...` (and the `median()` rewrite, which now also produces `PercentileV2`) no longer merge into a single `percentile_array_v2(...)` state, which defeats the main performance goal of this PR for the default Nereids path.



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/MergePercentileToArray.java:
##########
@@ -96,14 +96,14 @@ private List<AggregateFunction> getPercentileArrays(Map<DistinctAndExpr, List<Ag
                 percentArray = new Array(percentList.toArray(new Expression[0]));
             }
 
-            PercentileArray percentileArray;
             Expression secondArg = allPercentIsLiteral
                     ? TypeCoercionUtils.castIfNotSameType(percentArrayLiteral, ArrayType.of(DoubleType.INSTANCE))
                     : TypeCoercionUtils.castIfNotSameType(percentArray, ArrayType.of(DoubleType.INSTANCE));
+            AggregateFunction percentileArray;
             if (entry.getKey().isDistinct) {

Review Comment:
   `collectFuncMap()` still only collects `Percentile`, which after this PR is the explicit `percentile_v1` class. Replacing those functions with `PercentileArrayV2` changes semantics: `select percentile_v1(x, 0.1), percentile_v1(x, 0.9) ...` will now be planned as `percentile_array_v2(...)`, so the result no longer matches the explicit compatibility function the user asked for. The merged aggregate needs to stay on the v1 implementation when the collected functions are `Percentile`.



##########
be/src/exprs/aggregate/aggregate_function_percentile.cpp:
##########
@@ -74,8 +83,16 @@ void register_percentile_approx_old_function(AggregateFunctionSimpleFactory& fac
 }
 
 void register_aggregate_function_percentile_old(AggregateFunctionSimpleFactory& factory) {
-    BeExecVersionManager::registe_restrict_function_compatibility("percentile");
-    BeExecVersionManager::registe_restrict_function_compatibility("percentile_array");
+    using creator = creator_with_type_list<TYPE_TINYINT, TYPE_SMALLINT, TYPE_INT, TYPE_BIGINT,
+                                           TYPE_LARGEINT, TYPE_FLOAT, TYPE_DOUBLE>;
+    factory.register_alternative_function("percentile",

Review Comment:
   The old-version registration here only covers `percentile` / `percentile_array`. After the FE changes in this PR, Nereids serializes the default spellings as `percentile_v2` / `percentile_array_v2` (`ExpressionTranslator.visitAggregateFunction()` uses `function.getName()`), so `be_exec_version=6` no longer forces the old implementation for ordinary `percentile(...)` queries. That breaks the rolling-upgrade compatibility story for the default path.



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/types/AggStateType.java:
##########
@@ -49,7 +49,9 @@ public class AggStateType extends DataType {
             .put("any", "any_value")
             .put("char_length", "character_length")
             .put("stddev_pop", "stddev")
-            .put("percentile_cont", "percentile")
+            .put("percentile_cont", "percentile_v2")
+            .put("percentile", "percentile_v2")

Review Comment:
   Canonicalizing existing `agg_state` names from `percentile` / `percentile_array` to the v2 names here breaks stored percentile states. `DataType.fromCatalogType()` reconstructs catalog `AggStateType`s through this constructor, and `PlanTranslatorContext` later re-emits that normalized type to BE. For an old `agg_state<percentile(...)>` column or MV, BE union/merge will deserialize persisted v1 bytes with the v2 nested aggregate function, bypassing the old `be_exec_version` compatibility path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #62520:
URL: https://github.com/apache/doris/pull/62520#discussion_r3096594870


##########
be/src/util/percentile_util.h:
##########
@@ -223,4 +237,61 @@ class Counts {
     std::vector<PODArray<Ty>> _sorted_nums_vec;
 };
 
+class PercentileLevels {
+public:
+    void merge(const PercentileLevels& rhs) {
+        if (rhs.empty()) {
+            return;
+        }
+
+        if (empty()) {
+            quantiles = rhs.quantiles;
+            permutation = rhs.permutation;
+            return;
+        }
+
+        DCHECK_EQ(quantiles.size(), rhs.quantiles.size());
+        for (size_t i = 0; i < quantiles.size(); ++i) {
+            DCHECK_EQ(quantiles[i], rhs.quantiles[i]);
+        }
+    }
+
+    void write(BufferWritable& buf) const {
+        int size_num = cast_set<int>(quantiles.size());
+        buf.write_binary(size_num);
+        for (const auto& quantile : quantiles) {
+            buf.write_binary(quantile);
+        }
+    }
+
+    void read(BufferReadable& buf) {
+        int size_num = 0;
+        buf.read_binary(size_num);
+
+        quantiles.resize(size_num);
+        permutation.resize(size_num);
+        for (int i = 0; i < size_num; ++i) {
+            buf.read_binary(quantiles[i]);
+            permutation[i] = cast_set<size_t>(i);
+        }
+
+        sort_permutation();
+    }
+
+    void clear() {
+        quantiles.clear();
+        permutation.clear();
+    }
+
+    bool empty() const { return quantiles.empty(); }
+
+    void sort_permutation() {
+        pdqsort(permutation.begin(), permutation.end(),
+                [this](size_t lhs, size_t rhs) { return quantiles[lhs] < quantiles[rhs]; });
+    }
+
+    std::vector<double> quantiles;
+    std::vector<size_t> permutation;

Review Comment:
   这个为什么只有size_t呢?原来的函数不是支持tinyint到double吗?为什么这里是一个size_t呢?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4251037571

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #62520:
URL: https://github.com/apache/doris/pull/62520#discussion_r3096682557


##########
be/src/exprs/aggregate/aggregate_function_percentile.h:
##########
@@ -436,6 +429,194 @@ struct PercentileState {
     }
 };
 
+template <PrimitiveType T>
+struct PercentileExactState {
+    using ValueType = typename PrimitiveTypeTraits<T>::CppType;
+    static constexpr size_t bytes_in_arena = 64 - sizeof(PODArray<ValueType>);
+    using Array = PODArrayWithStackMemory<ValueType, bytes_in_arena>;
+
+    void add_single_range(const ValueType* data, size_t count, double quantile) {
+        if (!inited_flag) {
+            _set_single_level(quantile);
+            inited_flag = true;
+        }
+        _append(data, count);
+    }
+
+    void add_many_range(const ValueType* data, size_t count,
+                        const PaddedPODArray<Float64>& quantiles_data, const NullMap& null_maps,
+                        size_t start, int64_t arg_size) {
+        if (!inited_flag) {
+            _set_many_levels(quantiles_data, null_maps, start, arg_size);
+            inited_flag = true;
+        }
+        if (levels.empty()) {
+            return;
+        }
+        _append(data, count);
+    }
+
+    void write(BufferWritable& buf) const {
+        buf.write_binary(inited_flag);
+        if (!inited_flag) {
+            return;
+        }
+
+        levels.write(buf);
+        size_t size = values.size();
+        buf.write_binary(size);
+        if (size > 0) {
+            buf.write(reinterpret_cast<const char*>(values.data()), sizeof(ValueType) * size);
+        }
+    }
+
+    void read(BufferReadable& buf) {
+        reset();
+        buf.read_binary(inited_flag);
+        if (!inited_flag) {
+            return;
+        }
+
+        levels.read(buf);
+        size_t size = 0;
+        buf.read_binary(size);
+        values.resize(size);
+        if (size > 0) {
+            auto raw = buf.read(sizeof(ValueType) * size);
+            memcpy(values.data(), raw.data, raw.size);
+        }
+    }
+
+    void merge(const PercentileExactState& rhs) {
+        if (!rhs.inited_flag) {
+            return;
+        }
+
+        if (!inited_flag) {
+            levels = rhs.levels;
+            inited_flag = true;
+        } else {
+            levels.merge(rhs.levels);
+        }
+        _append(rhs.values.data(), rhs.values.size());
+    }
+
+    void reset() {
+        values.clear();
+        levels.clear();
+        inited_flag = false;
+    }
+
+    double get() const {
+        if (!inited_flag || levels.empty() || values.empty()) {
+            return 0.0;
+        }
+
+        DCHECK_EQ(levels.quantiles.size(), 1);
+        return _get_float(levels.quantiles[0]);
+    }
+
+    void insert_result_into(IColumn& to) const {
+        auto& column_data = assert_cast<ColumnFloat64&>(to).get_data();
+        if (!inited_flag || levels.empty() || values.empty()) {
+            return;
+        }
+
+        size_t old_size = column_data.size();
+        size_t size = levels.quantiles.size();
+        column_data.resize(old_size + size);
+        auto* result = column_data.data() + old_size;
+
+        if (values.size() == 1) {
+            for (size_t i = 0; i < size; ++i) {
+                result[i] = static_cast<double>(values.front());
+            }
+            return;
+        }
+
+        size_t prev_index = 0;
+        const auto& quantiles = levels.quantiles;
+        const auto& permutation = levels.permutation;
+        for (size_t i = 0; i < size; ++i) {
+            auto level_index = permutation[i];
+            auto level = quantiles[level_index];
+            double u = static_cast<double>(values.size() - 1) * level;
+            auto index = static_cast<size_t>(u);
+
+            if (index + 1 >= values.size()) {
+                result[level_index] =
+                        static_cast<double>(*std::max_element(values.begin(), values.end()));
+            } else {
+                std::nth_element(values.begin() + prev_index, values.begin() + index, values.end());
+                auto* nth_elem = std::min_element(values.begin() + index + 1, values.end());
+                result[level_index] =
+                        static_cast<double>(values[index]) +
+                        (u - static_cast<double>(index)) * (static_cast<double>(*nth_elem) -
+                                                            static_cast<double>(values[index]));
+                prev_index = index;
+            }
+        }
+    }
+
+private:
+    void _set_single_level(double quantile) {
+        DCHECK(levels.empty());
+        check_quantile(quantile);
+        levels.quantiles.push_back(quantile);
+        levels.permutation.push_back(0);
+    }
+
+    void _set_many_levels(const PaddedPODArray<Float64>& quantiles_data, const NullMap& null_maps,
+                          size_t start, int64_t arg_size) {
+        DCHECK(levels.empty());
+        size_t size = cast_set<size_t>(arg_size);
+        levels.quantiles.resize(size);
+        levels.permutation.resize(size);
+        for (size_t i = 0; i < size; ++i) {
+            if (null_maps[start + i]) {
+                throw Exception(ErrorCode::INVALID_ARGUMENT,
+                                "quantiles in func percentile_array should not have null");
+            }
+            check_quantile(quantiles_data[start + i]);
+            levels.quantiles[i] = quantiles_data[start + i];
+            levels.permutation[i] = i;
+        }
+        levels.sort_permutation();

Review Comment:
   我们应该每次排序吗?是不是应该最后读取的时候排序一次就行了



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #62520:
URL: https://github.com/apache/doris/pull/62520#discussion_r3096630812


##########
be/src/exprs/aggregate/aggregate_function_percentile.h:
##########
@@ -436,6 +429,194 @@ struct PercentileState {
     }
 };
 
+template <PrimitiveType T>
+struct PercentileExactState {
+    using ValueType = typename PrimitiveTypeTraits<T>::CppType;
+    static constexpr size_t bytes_in_arena = 64 - sizeof(PODArray<ValueType>);
+    using Array = PODArrayWithStackMemory<ValueType, bytes_in_arena>;
+
+    void add_single_range(const ValueType* data, size_t count, double quantile) {
+        if (!inited_flag) {
+            _set_single_level(quantile);
+            inited_flag = true;
+        }
+        _append(data, count);
+    }
+
+    void add_many_range(const ValueType* data, size_t count,
+                        const PaddedPODArray<Float64>& quantiles_data, const NullMap& null_maps,
+                        size_t start, int64_t arg_size) {
+        if (!inited_flag) {
+            _set_many_levels(quantiles_data, null_maps, start, arg_size);
+            inited_flag = true;
+        }
+        if (levels.empty()) {
+            return;
+        }
+        _append(data, count);
+    }
+
+    void write(BufferWritable& buf) const {
+        buf.write_binary(inited_flag);
+        if (!inited_flag) {
+            return;
+        }
+
+        levels.write(buf);
+        size_t size = values.size();
+        buf.write_binary(size);
+        if (size > 0) {
+            buf.write(reinterpret_cast<const char*>(values.data()), sizeof(ValueType) * size);
+        }
+    }
+
+    void read(BufferReadable& buf) {
+        reset();
+        buf.read_binary(inited_flag);
+        if (!inited_flag) {
+            return;
+        }
+
+        levels.read(buf);
+        size_t size = 0;
+        buf.read_binary(size);
+        values.resize(size);
+        if (size > 0) {
+            auto raw = buf.read(sizeof(ValueType) * size);
+            memcpy(values.data(), raw.data, raw.size);
+        }
+    }
+
+    void merge(const PercentileExactState& rhs) {
+        if (!rhs.inited_flag) {
+            return;
+        }
+
+        if (!inited_flag) {
+            levels = rhs.levels;
+            inited_flag = true;
+        } else {
+            levels.merge(rhs.levels);
+        }
+        _append(rhs.values.data(), rhs.values.size());
+    }
+
+    void reset() {
+        values.clear();
+        levels.clear();
+        inited_flag = false;
+    }
+
+    double get() const {
+        if (!inited_flag || levels.empty() || values.empty()) {
+            return 0.0;
+        }
+
+        DCHECK_EQ(levels.quantiles.size(), 1);
+        return _get_float(levels.quantiles[0]);
+    }
+
+    void insert_result_into(IColumn& to) const {
+        auto& column_data = assert_cast<ColumnFloat64&>(to).get_data();
+        if (!inited_flag || levels.empty() || values.empty()) {
+            return;
+        }
+
+        size_t old_size = column_data.size();
+        size_t size = levels.quantiles.size();
+        column_data.resize(old_size + size);
+        auto* result = column_data.data() + old_size;
+
+        if (values.size() == 1) {
+            for (size_t i = 0; i < size; ++i) {
+                result[i] = static_cast<double>(values.front());
+            }
+            return;
+        }
+
+        size_t prev_index = 0;
+        const auto& quantiles = levels.quantiles;
+        const auto& permutation = levels.permutation;
+        for (size_t i = 0; i < size; ++i) {
+            auto level_index = permutation[i];
+            auto level = quantiles[level_index];
+            double u = static_cast<double>(values.size() - 1) * level;
+            auto index = static_cast<size_t>(u);
+
+            if (index + 1 >= values.size()) {
+                result[level_index] =
+                        static_cast<double>(*std::max_element(values.begin(), values.end()));
+            } else {
+                std::nth_element(values.begin() + prev_index, values.begin() + index, values.end());
+                auto* nth_elem = std::min_element(values.begin() + index + 1, values.end());
+                result[level_index] =
+                        static_cast<double>(values[index]) +
+                        (u - static_cast<double>(index)) * (static_cast<double>(*nth_elem) -
+                                                            static_cast<double>(values[index]));
+                prev_index = index;
+            }
+        }
+    }
+
+private:
+    void _set_single_level(double quantile) {
+        DCHECK(levels.empty());
+        check_quantile(quantile);
+        levels.quantiles.push_back(quantile);
+        levels.permutation.push_back(0);
+    }
+
+    void _set_many_levels(const PaddedPODArray<Float64>& quantiles_data, const NullMap& null_maps,
+                          size_t start, int64_t arg_size) {
+        DCHECK(levels.empty());
+        size_t size = cast_set<size_t>(arg_size);
+        levels.quantiles.resize(size);
+        levels.permutation.resize(size);
+        for (size_t i = 0; i < size; ++i) {
+            if (null_maps[start + i]) {
+                throw Exception(ErrorCode::INVALID_ARGUMENT,
+                                "quantiles in func percentile_array should not have null");
+            }
+            check_quantile(quantiles_data[start + i]);
+            levels.quantiles[i] = quantiles_data[start + i];
+            levels.permutation[i] = i;
+        }
+        levels.sort_permutation();
+    }
+
+    void _append(const ValueType* data, size_t count) {
+        if (count == 0) {
+            return;
+        }
+        values.reserve(values.size() + count);
+        values.insert_assume_reserved(data, data + count);
+    }
+
+    double _get_float(double quantile) const {

Review Comment:
   why not call `_get_result` or `_get_double`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4273689825

   # FE Regression Coverage Report
   Increment line coverage `  84.21% (64/76)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62520_689e2b61e2f51b6f52968ff7bbf27e9b15ba222b_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62520_689e2b61e2f51b6f52968ff7bbf27e9b15ba222b_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb merged PR #62520:
URL: https://github.com/apache/doris/pull/62520


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4296637791

   # FE Regression Coverage Report
   Increment line coverage `  100.00% (5/5)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62520_f0fb942a44e76d55ee84aec1750abae1c2196ecb_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62520_f0fb942a44e76d55ee84aec1750abae1c2196ecb_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4303098668

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4280170035

   # FE Regression Coverage Report
   Increment line coverage `  58.72% (64/109)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62520_27e26098db0ce456a42c70dd890d08e8dfbdd59e_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62520_27e26098db0ce456a42c70dd890d08e8dfbdd59e_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4250034516

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on code in PR #62520:
URL: https://github.com/apache/doris/pull/62520#discussion_r3097587226


##########
be/src/exprs/aggregate/aggregate_function_percentile.h:
##########
@@ -436,6 +429,194 @@ struct PercentileState {
     }
 };
 
+template <PrimitiveType T>
+struct PercentileExactState {
+    using ValueType = typename PrimitiveTypeTraits<T>::CppType;
+    static constexpr size_t bytes_in_arena = 64 - sizeof(PODArray<ValueType>);
+    using Array = PODArrayWithStackMemory<ValueType, bytes_in_arena>;
+
+    void add_single_range(const ValueType* data, size_t count, double quantile) {
+        if (!inited_flag) {
+            _set_single_level(quantile);
+            inited_flag = true;
+        }
+        _append(data, count);
+    }
+
+    void add_many_range(const ValueType* data, size_t count,
+                        const PaddedPODArray<Float64>& quantiles_data, const NullMap& null_maps,
+                        size_t start, int64_t arg_size) {
+        if (!inited_flag) {
+            _set_many_levels(quantiles_data, null_maps, start, arg_size);
+            inited_flag = true;
+        }
+        if (levels.empty()) {
+            return;
+        }
+        _append(data, count);
+    }
+
+    void write(BufferWritable& buf) const {
+        buf.write_binary(inited_flag);
+        if (!inited_flag) {
+            return;
+        }
+
+        levels.write(buf);
+        size_t size = values.size();
+        buf.write_binary(size);
+        if (size > 0) {
+            buf.write(reinterpret_cast<const char*>(values.data()), sizeof(ValueType) * size);
+        }
+    }
+
+    void read(BufferReadable& buf) {
+        reset();
+        buf.read_binary(inited_flag);
+        if (!inited_flag) {
+            return;
+        }
+
+        levels.read(buf);
+        size_t size = 0;
+        buf.read_binary(size);
+        values.resize(size);
+        if (size > 0) {
+            auto raw = buf.read(sizeof(ValueType) * size);
+            memcpy(values.data(), raw.data, raw.size);
+        }
+    }
+
+    void merge(const PercentileExactState& rhs) {
+        if (!rhs.inited_flag) {
+            return;
+        }
+
+        if (!inited_flag) {
+            levels = rhs.levels;
+            inited_flag = true;
+        } else {
+            levels.merge(rhs.levels);
+        }
+        _append(rhs.values.data(), rhs.values.size());
+    }
+
+    void reset() {
+        values.clear();
+        levels.clear();
+        inited_flag = false;
+    }
+
+    double get() const {
+        if (!inited_flag || levels.empty() || values.empty()) {
+            return 0.0;
+        }
+
+        DCHECK_EQ(levels.quantiles.size(), 1);
+        return _get_float(levels.quantiles[0]);
+    }
+
+    void insert_result_into(IColumn& to) const {
+        auto& column_data = assert_cast<ColumnFloat64&>(to).get_data();
+        if (!inited_flag || levels.empty() || values.empty()) {

Review Comment:
   行为不一致
   ```text
   Doris> SELECT percentile(num, 0.5) FROM test_percent;
   +----------------------+
   | percentile(num, 0.5) |
   +----------------------+
   |                 NULL |
   +----------------------+
   1 row in set (0.228 sec)
   
   Doris> SELECT percentile_array(num, [0.1, 0.5]) FROM test_percent;
   +-----------------------------------+
   | percentile_array(num, [0.1, 0.5]) |
   +-----------------------------------+
   | []                                |
   +-----------------------------------+
   1 row in set (0.127 sec)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62520:
URL: https://github.com/apache/doris/pull/62520#discussion_r3097774121


##########
fe/fe-core/src/main/java/org/apache/doris/nereids/types/AggStateType.java:
##########
@@ -49,7 +49,6 @@ public class AggStateType extends DataType {
             .put("any", "any_value")
             .put("char_length", "character_length")
             .put("stddev_pop", "stddev")
-            .put("percentile_cont", "percentile")

Review Comment:
   Binding `percentile`/`percentile_cont` to `PercentileV2` while removing the agg-state canonicalization here breaks the `*_state/*_merge/*_union` path for existing `agg_state<percentile(...)>` and `agg_state<percentile_cont(...)>` schemas. `StateCombinator` / `MergeCombinator` derive the runtime function name from `nested.getName()` (`percentile_v2` now), but `LogicalPlanBuilder` still parses those column types as `percentile` / `percentile_cont`, `CheckCast.checkWithLooseAggState()` only accepts equal function names, and BE merge/union checks `type_function_name + suffix` exactly in `vectorized_agg_fn.cpp`. A concrete failure is `CREATE TABLE t (s agg_state<percentile(int,double)> generic); INSERT INTO t SELECT percentile_state(k, 0.5) ...` or `SELECT percentile_merge(s) FROM t`, which now becomes `percentile_v2_*` on the query side and no longer matches the stored agg-state name. `map_agg` / `window_funnel` keep alias canonicalization for this reason, so percentile needs the same
  treatment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62520:
URL: https://github.com/apache/doris/pull/62520#discussion_r3121605330


##########
fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinAggregateFunctions.java:
##########
@@ -173,11 +175,13 @@ private BuiltinAggregateFunctions() {
                 agg(OrthogonalBitmapIntersect.class, "orthogonal_bitmap_intersect"),
                 agg(OrthogonalBitmapIntersectCount.class, "orthogonal_bitmap_intersect_count"),
                 agg(OrthogonalBitmapUnionCount.class, "orthogonal_bitmap_union_count"),
-                agg(Percentile.class, "percentile", "percentile_cont"),
+                agg(Percentile.class, "percentile_v1"),

Review Comment:
   `percentile_v1` / `percentile_array_v1` are not real v1 escape hatches yet. `BuiltinFunctionBuilder` still instantiates `Percentile` / `PercentileArray`, whose runtime names are `percentile` / `percentile_array`; `ExpressionTranslator` serializes those names, and `SessionVariable.toThrift()` now always sets `new_version_percentile=true`, so BE remaps them straight back to the v2 implementation. A concrete repro is `select percentile_v1(pk, 0.5) ...`: it still executes v2.



##########
fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinAggregateFunctions.java:
##########
@@ -173,11 +175,13 @@ private BuiltinAggregateFunctions() {
                 agg(OrthogonalBitmapIntersect.class, "orthogonal_bitmap_intersect"),
                 agg(OrthogonalBitmapIntersectCount.class, "orthogonal_bitmap_intersect_count"),
                 agg(OrthogonalBitmapUnionCount.class, "orthogonal_bitmap_union_count"),
-                agg(Percentile.class, "percentile", "percentile_cont"),
+                agg(Percentile.class, "percentile_v1"),
+                agg(PercentileV2.class, "percentile_v2", "percentile", "percentile_cont"),

Review Comment:
   Binding the public `percentile` / `percentile_array` spellings to `PercentileV2` / `PercentileArrayV2` means Nereids now serializes `percentile_v2` / `percentile_array_v2` into the plan (`ExpressionTranslator` uses `function.getName()`). Old BEs at the base revision do not register those symbols, and `new_version_percentile` only rewrites old names to new ones on the BE side. So ordinary `percentile(...)` queries from a new FE can still fail on an old BE during rolling upgrade.



##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/MedianConvert.java:
##########
@@ -39,7 +39,7 @@ public class MedianConvert implements ExpressionPatternRuleFactory {
     public List<ExpressionPatternMatcher<? extends Expression>> buildRules() {
         return ImmutableList.of(
                 matchesType(Median.class).then(median ->
-                    new Percentile(median.child(0), DoubleLiteral.of(0.5))
+                    new PercentileV2(median.child(0), DoubleLiteral.of(0.5))

Review Comment:
   This still drops `median.isDistinct()` when rewriting. `median(distinct x)` becomes `percentile_v2(x, 0.5)` instead of `percentile_v2(distinct x, 0.5)`, which is a correctness bug. For example, on values `(1, 1, 100)` the distinct median is `50.5`, but the rewritten non-distinct form returns `1`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62520:
URL: https://github.com/apache/doris/pull/62520#discussion_r3123639748


##########
be/src/exprs/aggregate/aggregate_function_simple_factory.h:
##########
@@ -135,6 +135,15 @@ class AggregateFunctionSimpleFactory {
         if (function_alias.contains(name)) {
             name_str = function_alias[name];
         }
+
+        if (attr.new_version_percentile) {

Review Comment:
    still runs before this  remap. That means a FE that temporarily lowers  for rolling upgrade / compatibility mode will still hit the old  /  restriction before these public names can be rewritten to  / . Since FE now always sets , the remap needs to happen before the old-version compatibility lookup (the scalar-function factory already uses that ordering for ).



##########
be/src/exprs/aggregate/aggregate_function_percentile.h:
##########
@@ -568,4 +748,194 @@ class AggregateFunctionPercentileArray final
     }
 };
 
-} // namespace doris
\ No newline at end of file
+template <PrimitiveType T>
+class AggregateFunctionPercentileV2 final
+        : public IAggregateFunctionDataHelper<PercentileExactState<T>,
+                                              AggregateFunctionPercentileV2<T>>,
+          MultiExpression,
+          NullableAggregateFunction {
+public:
+    using ColVecType = typename PrimitiveTypeTraits<T>::ColumnType;
+    using Base =
+            IAggregateFunctionDataHelper<PercentileExactState<T>, AggregateFunctionPercentileV2<T>>;
+    AggregateFunctionPercentileV2(const DataTypes& argument_types_) : Base(argument_types_) {}
+
+    String get_name() const override { return "percentile_v2"; }
+
+    DataTypePtr get_return_type() const override { return std::make_shared<DataTypeFloat64>(); }
+
+    void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+             Arena&) const override {
+        const auto& sources =
+                assert_cast<const ColVecType&, TypeCheckOnRelease::DISABLE>(*columns[0]);
+        const auto& quantile =
+                assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(*columns[1]);
+        AggregateFunctionPercentileV2::data(place).add_single_range(&sources.get_data()[row_num], 1,
+                                                                    quantile.get_data()[0]);
+    }
+
+    void add_batch_single_place(size_t batch_size, AggregateDataPtr place, const IColumn** columns,
+                                Arena&) const override {
+        const auto& sources =
+                assert_cast<const ColVecType&, TypeCheckOnRelease::DISABLE>(*columns[0]);
+        const auto& quantile =
+                assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(*columns[1]);
+        DCHECK_EQ(sources.get_data().size(), batch_size);
+        AggregateFunctionPercentileV2::data(place).add_single_range(
+                sources.get_data().data(), batch_size, quantile.get_data()[0]);
+    }
+
+    void add_batch_range(size_t batch_begin, size_t batch_end, AggregateDataPtr place,
+                         const IColumn** columns, Arena&, bool has_null) override {
+        const auto& sources =
+                assert_cast<const ColVecType&, TypeCheckOnRelease::DISABLE>(*columns[0]);
+        const auto& quantile =
+                assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(*columns[1]);
+        DCHECK(!has_null);
+        AggregateFunctionPercentileV2::data(place).add_single_range(
+                sources.get_data().data() + batch_begin, batch_end - batch_begin + 1,
+                quantile.get_data()[0]);
+    }
+
+    void reset(AggregateDataPtr __restrict place) const override {
+        AggregateFunctionPercentileV2::data(place).reset();
+    }
+
+    void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs,
+               Arena&) const override {
+        AggregateFunctionPercentileV2::data(place).merge(AggregateFunctionPercentileV2::data(rhs));
+    }
+
+    void serialize(ConstAggregateDataPtr __restrict place, BufferWritable& buf) const override {
+        AggregateFunctionPercentileV2::data(place).write(buf);
+    }
+
+    void deserialize(AggregateDataPtr __restrict place, BufferReadable& buf,
+                     Arena&) const override {
+        AggregateFunctionPercentileV2::data(place).read(buf);
+    }
+
+    void insert_result_into(ConstAggregateDataPtr __restrict place, IColumn& to) const override {
+        auto& col = assert_cast<ColumnFloat64&>(to);
+        col.insert_value(AggregateFunctionPercentileV2::data(place).get());
+    }
+};
+
+template <PrimitiveType T>
+class AggregateFunctionPercentileArrayV2 final
+        : public IAggregateFunctionDataHelper<PercentileExactState<T>,
+                                              AggregateFunctionPercentileArrayV2<T>>,
+          MultiExpression,
+          NotNullableAggregateFunction {
+public:
+    using ColVecType = typename PrimitiveTypeTraits<T>::ColumnType;
+    using Base = IAggregateFunctionDataHelper<PercentileExactState<T>,
+                                              AggregateFunctionPercentileArrayV2<T>>;
+    AggregateFunctionPercentileArrayV2(const DataTypes& argument_types_) : Base(argument_types_) {}
+
+    String get_name() const override { return "percentile_array_v2"; }
+
+    DataTypePtr get_return_type() const override {
+        return std::make_shared<DataTypeArray>(make_nullable(std::make_shared<DataTypeFloat64>()));
+    }
+
+    void add(AggregateDataPtr __restrict place, const IColumn** columns, ssize_t row_num,
+             Arena&) const override {
+        const auto& sources =
+                assert_cast<const ColVecType&, TypeCheckOnRelease::DISABLE>(*columns[0]);
+        const auto& quantile_array =
+                assert_cast<const ColumnArray&, TypeCheckOnRelease::DISABLE>(*columns[1]);
+        const auto& offset_column_data = quantile_array.get_offsets();
+        const auto& null_maps = assert_cast<const ColumnNullable&, TypeCheckOnRelease::DISABLE>(
+                                        quantile_array.get_data())
+                                        .get_null_map_data();
+        const auto& nested_column = assert_cast<const ColumnNullable&, TypeCheckOnRelease::DISABLE>(
+                                            quantile_array.get_data())
+                                            .get_nested_column();
+        const auto& nested_column_data =
+                assert_cast<const ColumnFloat64&, TypeCheckOnRelease::DISABLE>(nested_column);

Review Comment:
   This batch fast path assumes every row in the block shares row 0's quantile array: it always feeds  the nested quantiles/null-map starting at offset 0 and uses  as the array length for the whole batch. That is only safe if FE guarantees a constant second argument. Nereids does (), but legacy FE still has no such check, and  now enables  globally. So a legacy no- query like  will silently use the first row's levels for the whole block via . The old  implementation did not override , so this is a new regression.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4273283827

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4279075038

   # FE UT Coverage Report
   Increment line coverage `  71.05% (54/76)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/62520_27e26098db0ce456a42c70dd890d08e8dfbdd59e/fe_increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/62520_27e26098db0ce456a42c70dd890d08e8dfbdd59e/fe_report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "linrrzqqq (via GitHub)" <gi...@apache.org>.
linrrzqqq commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4309916230

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62520:
URL: https://github.com/apache/doris/pull/62520#discussion_r3085604403


##########
be/src/exprs/aggregate/aggregate_function_percentile.h:
##########
@@ -323,112 +319,191 @@ class AggregateFunctionPercentileApproxWeightedFourParams final
 };
 
 template <PrimitiveType T>
-struct PercentileState {
-    mutable std::vector<Counts<typename PrimitiveTypeTraits<T>::CppType>> vec_counts;
-    std::vector<double> vec_quantile {-1};
-    bool inited_flag = false;
+class PercentileState {
+public:
+    using ValueType = typename PrimitiveTypeTraits<T>::CppType;
+    static constexpr size_t bytes_in_arena = 64 - sizeof(PODArray<ValueType>);
+    using Array = PODArrayWithStackMemory<ValueType, bytes_in_arena>;
+
+    void add_single_range(const ValueType* data, size_t count, double quantile) {
+        if (!inited_flag) {
+            _set_single_level(quantile);
+            inited_flag = true;
+        }
+        _append(data, count);
+    }
+
+    void add_many_range(const ValueType* data, size_t count,
+                        const PaddedPODArray<Float64>& quantiles_data, const NullMap& null_maps,
+                        size_t start, int64_t arg_size) {
+        if (!inited_flag) {
+            _set_many_levels(quantiles_data, null_maps, start, arg_size);
+            inited_flag = true;
+        }
+        if (levels.empty()) {
+            return;
+        }
+        _append(data, count);
+    }
 
     void write(BufferWritable& buf) const {
         buf.write_binary(inited_flag);
         if (!inited_flag) {
             return;
         }
-        int size_num = cast_set<int>(vec_quantile.size());
-        buf.write_binary(size_num);
-        for (const auto& quantile : vec_quantile) {
-            buf.write_binary(quantile);
-        }
-        for (auto& counts : vec_counts) {
-            counts.serialize(buf);
+
+        levels.write(buf);
+        size_t size = values.size();

Review Comment:
   This changes the serialized layout of `PercentileState` in a backward-incompatible way.
   
   Before this PR, the state was encoded as:
   `inited_flag + quantile list + serialized Counts objects`
   
   After this PR, it becomes:
   `inited_flag + quantile list + raw value count + raw values`
   
   `percentile` / `percentile_array` states are not just transient in-memory objects. They are also persisted and reloaded through `AggState` (`DataTypeAggState` -> storage reader -> `deserialize()`), and can be exchanged between BEs as serialized aggregate states. With the new `read()` implementation, any previously stored state bytes will interpret the old `Counts` payload as the new `size_t size` field and then read the wrong number of bytes into `values`.
   
   I don't see any `be_exec_version` bump/gate, dual-format reader, or other compatibility path here, so this looks like a storage / wire-format break rather than a pure internal refactor. Please either preserve the old format, or add explicit versioned compatibility handling plus tests that cover reading old serialized percentile states.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [Opt](func) Improve AggFunc Percentile performance [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62520:
URL: https://github.com/apache/doris/pull/62520#issuecomment-4310460743

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org