(doris) branch master updated: [fix](hll) shrink chars buffer in hll_to_base64 to match offsets (#62797)

[pa...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

BiteTheDDDDt pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new e87bccbd98d [fix](hll) shrink chars buffer in hll_to_base64 to match offsets (#62797)
e87bccbd98d is described below

commit e87bccbd98d90886cd96ea468e4a306f6462f52c
Author: Pxl <xl...@selectdb.com>
AuthorDate: Mon Apr 27 10:09:03 2026 +0800

    [fix](hll) shrink chars buffer in hll_to_base64 to match offsets (#62797)
    
    
    
    HllToBase64::vector() in be/src/exprs/function/function_hll.cpp pre-allocates the output chars buffer using HyperLogLog::max_serialized_size() (for SPARSE/FULL HLL the upper bound is 1 + HLL_REGISTERS_COUNT = 16385 bytes), but the inner loop only advances encoded_offset by the actual base64-encoded length of each row's serialized HLL. After the loop, offsets.back() == encoded_offset reflects the real bytes written, while chars.size() is still the over-allocated upper bound.
    
    This violates ColumnString's invariant chars.size() == offsets.back(), which is asserted by ColumnString::sanity_check_simple() (see be/src/core/column/column_string.cpp:153). In debug / ASAN builds this throws Internal error: chars.size(): X, offset[N-1]: Y. In release builds it leaks tail bytes into memory accounting and any downstream consumer that relies on chars.size().
    
    The bug is most easily reproducible with SPARSE HLLs: max_serialized_size() returns 16385 but the real serialized form is 1 + 4 + 3 * num_non_zero_registers, so the gap between allocated and used bytes is large. The existing tests only covered EXPLICIT HLL (hll_hash('abc')) and EMPTY HLL where max == real, so they did not exercise this path.
    
    Fix: shrink chars to the real encoded_offset after the encoding loop completes.
---
 be/src/exprs/function/function_hll.cpp                              | 4 ++++
 .../query_p0/sql_functions/hll_functions/test_hll_functions.out     | 5 +++++
 .../query_p0/sql_functions/hll_functions/test_hll_functions.groovy  | 6 ++++++
 3 files changed, 15 insertions(+)

diff --git a/be/src/exprs/function/function_hll.cpp b/be/src/exprs/function/function_hll.cpp
index ca9b29336ad..435cd8c8d77 100644
--- a/be/src/exprs/function/function_hll.cpp
+++ b/be/src/exprs/function/function_hll.cpp
@@ -303,6 +303,10 @@ struct HllToBase64 {
             encoded_offset += outlen;
             offsets[i] = cast_set<uint32_t>(encoded_offset);
         }
+        // chars was sized using max_serialized_size(); the actual encoded length
+        // can be smaller (e.g. sparse HLL), so shrink chars to keep
+        // offsets.back() == chars.size() and satisfy ColumnString::sanity_check.
+        chars.resize(encoded_offset);
         return Status::OK();
     }
 };
diff --git a/regression-test/data/query_p0/sql_functions/hll_functions/test_hll_functions.out b/regression-test/data/query_p0/sql_functions/hll_functions/test_hll_functions.out
index e9012565746..b34c741473e 100644
--- a/regression-test/data/query_p0/sql_functions/hll_functions/test_hll_functions.out
+++ b/regression-test/data/query_p0/sql_functions/hll_functions/test_hll_functions.out
@@ -33,3 +33,8 @@ AQEC5XSzrpDsdw==
 -- !const_select --
 AQEyl7yFZerf2A==
 
+-- !sparse_roundtrip --
+true
+
+-- !sparse_len_positive --
+true
diff --git a/regression-test/suites/query_p0/sql_functions/hll_functions/test_hll_functions.groovy b/regression-test/suites/query_p0/sql_functions/hll_functions/test_hll_functions.groovy
index aac99c29bd9..90226543d55 100644
--- a/regression-test/suites/query_p0/sql_functions/hll_functions/test_hll_functions.groovy
+++ b/regression-test/suites/query_p0/sql_functions/hll_functions/test_hll_functions.groovy
@@ -74,4 +74,10 @@ suite("test_hll_functions") {
         sql """SELECT hll_from_base64(CAST('AgAAAAAAAABwAAAAAAAAAA==' AS VARCHAR(100))) AS result;"""
         exception "[RUNTIME_ERROR]hll_from_base64 decode failed";
     }
+
+    // Cover sparse/full HLL hll_to_base64: max_serialized_size is much larger
+    // than the real serialized size, which previously left ColumnString chars
+    // oversized vs offsets and tripped sanity_check.
+    qt_sparse_roundtrip "select hll_cardinality(hll_from_base64(hll_to_base64(hll_union(hll_hash(number))))) > 400 from numbers(\"number\"=\"500\");"
+    qt_sparse_len_positive "select length(hll_to_base64(hll_union(hll_hash(number)))) > 0 from numbers(\"number\"=\"500\");"
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org