You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "taiyang-li (via GitHub)" <gi...@apache.org> on 2025/01/16 11:01:31 UTC
[PR] [GLUTEN-8528][CH]Support approx count distinct [incubator-gluten]
taiyang-li opened a new pull request, #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
(Fixes: \#8528)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2647416241
> Run Gluten ClickHouse CI on ARM
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2649898184
Run Gluten ClickHouse CI on ARM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2647331093
Run Gluten ClickHouse CI on ARM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2647358136
Run Gluten ClickHouse CI on ARM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2636324016
@CodiumAI-Agent /review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "CodiumAI-Agent (via GitHub)" <gi...@apache.org>.
CodiumAI-Agent commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2636325426
## PR Reviewer Guide π
Here are some key observations to aid the review process:
<table>
<tr><td>
**π« Ticket compliance analysis πΆ**
**[8528](https://github.com/apache/incubator-gluten/issues/8528) - Partially compliant**
Compliant requirements:
- Support the `approx_count_distinct` function.
Non-compliant requirements:
[]
Requires further human verification:
[]
</td></tr>
<tr><td>β±οΈ <strong>Estimated effort to review</strong>: 4 π΅π΅π΅π΅βͺ</td></tr>
<tr><td>π§ͺ <strong>PR contains tests</strong></td></tr>
<tr><td>π <strong>No security concerns identified</strong></td></tr>
<tr><td>β‘ <strong>Recommended focus areas for review</strong><br><br>
<details><summary><a href='https://github.com/apache/incubator-gluten/pull/8550/files#diff-a600bba8c3e169329dafd07a3806300a4be4371e1808dea9389a2606293fcdd2R272-R289'><strong>Possible Issue</strong></a>
Verify that the logic for handling `HyperLogLogPlusPlus` in the `Partial` and `Complete` aggregation modes is correct, especially the addition of `relativeSDLiteral` to the nodes.</summary>
```scala
val nodes = aggregateFunc.children.toList.map(
expr => {
ExpressionConverter
.replaceWithExpressionTransformer(expr, child.output)
.doTransform(args)
})
val extraNodes = aggregateFunc match {
case hll: HyperLogLogPlusPlus =>
val relativeSDLiteral = Literal(hll.relativeSD)
Seq(
ExpressionConverter
.replaceWithExpressionTransformer(relativeSDLiteral, child.output)
.doTransform(args))
case _ => Seq.empty
}
nodes ++ extraNodes
```
</details>
<details><summary><a href='https://github.com/apache/incubator-gluten/pull/8550/files#diff-a4849d6dba3b9258b586813a72c5a2ec05f82b223c741611c662506615a221eeR75-R141'><strong>Edge Case Handling</strong></a>
Ensure that the parser correctly handles all edge cases, such as invalid argument counts or non-literal second arguments for `approx_count_distinct`.</summary>
```c++
Array parseFunctionParameters(
const CommonFunctionInfo & func_info, ActionsDAG::NodeRawConstPtrs & arg_nodes, ActionsDAG & actions_dag) const override
{
if (func_info.phase == substrait::AGGREGATION_PHASE_INITIAL_TO_INTERMEDIATE
|| func_info.phase == substrait::AGGREGATION_PHASE_INITIAL_TO_RESULT
|| func_info.phase == substrait::AGGREGATION_PHASE_UNSPECIFIED)
{
const auto & arguments = func_info.arguments;
const size_t num_args = arguments.size();
const size_t num_nodes = arg_nodes.size();
if (num_args != num_nodes || num_args > 2 || num_args < 1 || num_nodes > 2 || num_nodes < 1)
throw Exception(
ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH,
"Function {} takes 1 or 2 arguments in phase {}",
getName(),
magic_enum::enum_name(func_info.phase));
Array params;
if (num_args == 2)
{
const auto & relative_sd_arg = arguments[1].value();
if (relative_sd_arg.has_literal())
{
auto [_, field] = parseLiteral(relative_sd_arg.literal());
params.push_back(std::move(field));
}
else
throw Exception(ErrorCodes::BAD_ARGUMENTS, "Second argument of function {} must be literal", getName());
}
else
{
params.push_back(0.05);
}
const auto & expr_arg = arg_nodes[0];
const auto * is_null_node = toFunctionNode(actions_dag, "isNull", {expr_arg});
const auto * hash_node = toFunctionNode(actions_dag, "sparkXxHash64", {expr_arg});
const auto * null_node
= addColumnToActionsDAG(actions_dag, std::make_shared<DataTypeNullable>(std::make_shared<DataTypeUInt64>()), {});
const auto * if_node = toFunctionNode(actions_dag, "if", {is_null_node, null_node, hash_node});
/// Replace the first argument expr with if(isNull(expr), null, sparkXxHash64(expr))
arg_nodes[0] = if_node;
arg_nodes.resize(1);
return params;
}
else
{
if (arg_nodes.size() != 1)
throw Exception(
ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH,
"Function {} takes 1 argument in phase {}",
getName(),
magic_enum::enum_name(func_info.phase));
const auto & result_type = arg_nodes[0]->result_type;
const auto * aggregate_function_type = checkAndGetDataType<DataTypeAggregateFunction>(result_type.get());
if (!aggregate_function_type)
throw Exception(
ErrorCodes::BAD_ARGUMENTS,
"The first argument type of function {} in phase {} must be AggregateFunction, but is {}",
getName(),
magic_enum::enum_name(func_info.phase),
result_type->getName());
return aggregate_function_type->getParameters();
}
```
</details>
<details><summary><a href='https://github.com/apache/incubator-gluten/pull/8550/files#diff-a248447f92eab84bf5b4420bec4999aa77dd1b0d9e7d6b87efe6c8b22ad46822R1-R72'><strong>Test Coverage</strong></a>
Confirm that the tests cover all critical scenarios for `HyperLogLogPlusPlus`, including serialization, merging, and edge cases.</summary>
```c++
#include <gtest/gtest.h>
#include <AggregateFunctions/AggregateFunctionUniqHyperLogLogPlusPlus.h>
#include "IO/ReadBufferFromString.h"
using namespace DB;
static std::vector<UInt64> random_uint64s
= {17956993516945311251ULL,
4306050051188505054ULL,
14289061765075743502ULL,
16763375724458316157ULL,
6144297519955185930ULL,
18446472757487308114ULL,
16923578592198257123ULL,
13557354668567515845ULL,
15328387702200001967ULL,
15878166530370497646ULL};
static void initSmallHLL(HyperLogLogPlusPlusData & hll)
{
for (auto x : random_uint64s)
hll.add(x);
}
static void initLargeHLL(HyperLogLogPlusPlusData & hll)
{
for (auto x : random_uint64s)
{
for (size_t i = 0; i < 100; ++i)
hll.add(x * (i+1));
}
}
TEST(HyperLogLogPlusPlusDataTest, Small)
{
HyperLogLogPlusPlusData hll;
initSmallHLL(hll);
EXPECT_EQ(hll.query(), 10);
}
TEST(HyperLogLogPlusPlusDataTest, Large)
{
HyperLogLogPlusPlusData hll;
initLargeHLL(hll);
EXPECT_EQ(hll.query(), 806);
}
TEST(HyperLogLogPlusPlusDataTest, Merge) {
HyperLogLogPlusPlusData hll1;
initSmallHLL(hll1);
HyperLogLogPlusPlusData hll2;
initLargeHLL(hll2);
hll1.merge(hll2);
EXPECT_EQ(hll1.query(), 806);
}
TEST(HyperLogLogPlusPlusDataTest, SerializeAndDeserialize) {
HyperLogLogPlusPlusData hll1;
initLargeHLL(hll1);
WriteBufferFromOwnString write_buffer;
hll1.serialize(write_buffer);
ReadBufferFromString read_buffer(write_buffer.str());
HyperLogLogPlusPlusData hll2;
hll2.deserialize(read_buffer);
EXPECT_EQ(hll2.query(), 806);
}
```
</details>
</td></tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "zhanglistar (via GitHub)" <gi...@apache.org>.
zhanglistar commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2647384324
Lets' enable spark hll UT to see what will happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2595219035
Run Gluten ClickHouse CI on ARM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2595218496
https://github.com/apache/incubator-gluten/issues/8528
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2653177578
> Better add some comment about HLLPP for later feature readers.
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "CodiumAI-Agent (via GitHub)" <gi...@apache.org>.
CodiumAI-Agent commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2647319645
**[Persistent review](https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2636325426)** updated to latest commit https://github.com/apache/incubator-gluten/commit/92e9224195b563b58d4956e848547323c133e4a4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on code in PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#discussion_r1953929584
##########
backends-clickhouse/src/main/scala/org/apache/gluten/execution/CHHashAggregateExecTransformer.scala:
##########
@@ -277,12 +277,24 @@ case class CHHashAggregateExecTransformer(
val childrenNodeList = new util.ArrayList[ExpressionNode]()
val childrenNodes = aggExpr.mode match {
case Partial | Complete =>
- aggregateFunc.children.toList.map(
+ val nodes = aggregateFunc.children.toList.map(
expr => {
ExpressionConverter
.replaceWithExpressionTransformer(expr, childOutput)
.doTransform(args)
})
+
+ val extraNodes = aggregateFunc match {
+ case hll: HyperLogLogPlusPlus =>
Review Comment:
If these cases become more, it should be put into a function to generate the nodes of an aggregate function. Anyway it seems OK at present
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li merged PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "zhanglistar (via GitHub)" <gi...@apache.org>.
zhanglistar commented on code in PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#discussion_r1948799571
##########
cpp-ch/local-engine/Parser/aggregate_function_parser/ApproxCountDistinctFunctionParser.cpp:
##########
@@ -0,0 +1,148 @@
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include <Parser/AggregateFunctionParser.h>
+#include <DataTypes/DataTypeNullable.h>
+#include <Poco/Logger.h>
+#include <Common/logger_useful.h>
+#include "DataTypes/DataTypeAggregateFunction.h"
Review Comment:
Use `<>` instead of `"` in include clause
##########
cpp-ch/local-engine/AggregateFunctions/AggregateFunctionUniqHyperLogLogPlusPlus.h:
##########
@@ -0,0 +1,3089 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <AggregateFunctions/IAggregateFunction.h>
+#include <AggregateFunctions/AggregateFunctionFactory.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <DataTypes/IDataType.h>
+#include <DataTypes/DataTypeAggregateFunction.h>
+#include <Common/FieldVisitorConvertToNumber.h>
+#include <Common/FieldVisitors.h>
+#include <Common/HashTable/Hash.h>
+#include <Common/HyperLogLogWithSmallSetOptimization.h>
+#include <Parsers/NullsAction.h>
+namespace DB
+{
+
+namespace ErrorCodes
+{
+extern const int PARAMETER_OUT_OF_BOUND;
+extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+extern const int INCORRECT_DATA;
+}
+
+struct HyperLogLogPlusPlusData
+{
+ explicit HyperLogLogPlusPlusData(double relative_sd_ = 0.05)
+ : relative_sd(relative_sd_)
+ , p(static_cast<UInt64>(std::ceil(2.0 * std::log(1.106 / relative_sd) / std::log(2.0))))
+ , idx_shift(64 - p)
+ , w_padding(1ULL << (p - 1))
+ , m(1ULL << p)
+ , num_words(m / REGISTERS_PER_WORD + 1)
+ , alpha_mm(computeAlphaMM())
+ {
+ if (p < 4)
+ throw Exception(
+ ErrorCodes::PARAMETER_OUT_OF_BOUND,
+ "HLL++ requires at least 4 bits for addressing instead of {}. Use a lower error, at most 39%",
+ p);
+
+ if (p > 25)
+ throw Exception(
+ ErrorCodes::PARAMETER_OUT_OF_BOUND,
+ "HLL++ requires at most 25 bits for addressing instead of {} to avoid allocating too much memory",
+ p);
+
+ // std::cout << "relative_sd:" << relative_sd << " p:" << p << " m:" << m << " num_words:" << num_words << " alpha_mm:" << alpha_mm
+ // << std::endl;
+ registers = PaddedPODArray<UInt64>(num_words, 0); // Initialize registers with zeros
+ }
+
+ void serialize(WriteBuffer & buf) const
+ {
+ writeBinaryLittleEndian(relative_sd, buf);
+
+ writeBinaryLittleEndian(registers.size(), buf);
+ for (const auto & word : registers)
+ writeBinaryLittleEndian(word, buf);
+ }
+
+ void deserialize(ReadBuffer & buf)
+ {
+ Float64 relative_rd_read;
+ readBinaryLittleEndian(relative_rd_read, buf);
+ if (relative_rd_read != relative_sd)
+ throw Exception(
+ ErrorCodes::INCORRECT_DATA, "The relative standard deviation {} isn't the expected one {}", relative_rd_read, relative_sd);
+
+ size_t registers_size = 0;
+ readBinaryLittleEndian(registers_size, buf);
+ if (registers_size != registers.size())
+ throw Exception(
+ ErrorCodes::INCORRECT_DATA, "The number of registers {} isn't the expected one {}", registers_size, registers.size());
+
+ for (size_t i = 0; i < registers_size; ++i)
+ readBinaryLittleEndian(registers[i], buf);
+ }
+
+ void add(UInt64 value)
+ {
+ // std::cout << "add:" << value << std::endl;
+ UInt64 x = value;
+ UInt64 idx = x >> idx_shift;
+ UInt64 w = (x << p) | w_padding;
+ UInt64 pw = __builtin_clzll(w) + 1;
+
+ UInt64 word_offset = idx / REGISTERS_PER_WORD;
+ UInt64 word = registers[word_offset];
+
+ UInt64 shift = (idx - word_offset * REGISTERS_PER_WORD) * REGISTER_SIZE;
+ UInt64 mask = REGISTER_WORD_MASK << shift;
+ UInt64 midx = (word & mask) >> shift;
+ // std::cout << "bucket[" << std::setw(3) << std::setfill('0') << idx << "]=" << midx << " input:" << value << " pw:" << pw;
+ if (pw > midx)
+ {
+ // std::cout << " updated";
+ registers[word_offset] = (word & ~mask) | (pw << shift);
+ }
+ // std::cout << std::endl;
+ }
+
+ void merge(const HyperLogLogPlusPlusData & other)
+ {
+ size_t i = 0; // global register index
+ for (size_t word_i = 0; word_i < num_words; ++word_i)
+ {
+ UInt64 word1 = registers[word_i];
+ UInt64 word2 = other.registers[word_i];
+ UInt64 mask = REGISTER_WORD_MASK;
+ UInt64 merged_word = 0;
+ for (size_t register_i = 0; i < m && register_i < REGISTERS_PER_WORD; ++register_i, ++i)
+ {
+ merged_word |= std::max((word1 & mask), (word2 & mask));
+ mask <<= REGISTER_SIZE;
+ }
+ registers[word_i] = merged_word;
+ }
+ }
+
+ UInt64 query() const
+ {
+ double z_inverse = 0.0;
+ UInt64 v = 0;
+
+ size_t i = 0; // global register index
+ for (size_t word_i = 0; word_i < num_words; ++word_i)
+ {
+ UInt64 word = registers[word_i];
+ for (size_t register_i = 0; i < m && register_i < REGISTERS_PER_WORD; ++register_i, ++i)
+ {
+ UInt64 midx = (word >> (register_i * REGISTER_SIZE)) & REGISTER_WORD_MASK;
+ // std::cout << "bucket[" << std::setw(3) << std::setfill('0') << i << "]=" << midx << std::endl;
+ z_inverse += 1.0 / (1ULL << midx);
+
+ if (midx == 0)
+ ++v;
+ }
+ }
+ // std::cout << "num_zero:" << v << std::endl;
+
+ double e = alpha_mm / z_inverse;
+ // std::cout << "e:" << e << std::endl;
+
+ double e_bias_corrected = e;
+ if (p < 19 && e < 5.0 * m)
+ e_bias_corrected = e - estimateBias(e);
+
+ // std::cout << "e_bias_corrected:" << e_bias_corrected << std::endl;
+
+ if (v > 0)
+ {
+ double h = m * std::log(static_cast<double>(m) / v);
+ if ((p < 19 && h <= THRESHOLDS[p - 4]) || e <= 2.5 * m)
+ {
+ // std::cout << "h:" << h << std::endl;
+ return static_cast<UInt64>(std::round(h));
+ }
+ else
+ return static_cast<UInt64>(std::round(e_bias_corrected));
+ }
+ else
+ return static_cast<UInt64>(std::round(e_bias_corrected));
+ }
+
+private:
+ Float64 computeAlphaMM() const
+ {
+ // Compute alpha * m * m based on the value of m
+ if (p == 4)
+ return 0.673 * m * m;
+ else if (p == 5)
+ return 0.697 * m * m;
+ else if (p == 6)
+ return 0.709 * m * m;
+ else
+ return (0.7213 / (1 + 1.079 / m)) * m * m;
+ }
+
+
+ /// Estimate the bias using the raw estimates with their respective biases from the HLL++
+ /// appendix. We currently use KNN interpolation to determine the bias (as suggested in the
+ /// paper).
+ double estimateBias(double e) const
+ {
+ const auto & estimates = RAW_ESTIMATE_DATA[p - 4];
+ size_t num_estimates = estimates.size();
+
+ // Binary search to find the nearest estimate index
+ auto nearest_estimate_it = std::lower_bound(estimates.begin(), estimates.end(), e);
+ size_t nearest_estimate_index = std::distance(estimates.begin(), nearest_estimate_it);
+
+ // Distance metric: square of the difference
+ auto distance = [&](size_t i)
+ {
+ double diff = e - estimates[i];
+ return diff * diff;
+ };
+
+ // Adjust bounds
+ size_t low = nearest_estimate_index + 1 > K ? nearest_estimate_index - K + 1 : 0;
+ size_t high = std::min(low + K, num_estimates);
+ while (high < num_estimates && distance(high) < distance(low))
+ {
+ ++low;
+ ++high;
+ }
+
+ // Sum biases in the interval
+ const auto & biases = BIAS_DATA[p - 4];
+ double bias_sum = 0.0;
+ for (size_t i = low; i < high; ++i)
+ bias_sum += biases[i];
+
+ // Calculate the bias
+ return bias_sum / (high - low);
+ }
+
+ /// Defines the maximum relative standard deviation allowed.
+ const Float64 relative_sd;
+
+ /// HLL++ uses 'p' bits for bucket addressing.
+ const UInt64 p;
+
+ /// Shift used to extract the index of the register from the hashed value.
+ /// This assumes the use of 64-bit hashcodes.
+ const UInt64 idx_shift;
+
+ /// Value to pad the 'w' value with before the number of leading zeros is determined.
+ const UInt64 w_padding;
+
+ /// The number of registers used.
+ const UInt64 m;
+
+ /// The number of words used to store the registers. We use Longs for storage because this is the
+ /// most compact way of storage; Spark aligns to 8-byte words or uses Long wrappers.
+
+ /// We only store whole registers per word in order to prevent overly complex bitwise operations.
+ /// In practice this means we only use 60 out of 64 bits.
+ const UInt64 num_words;
+
+ /// The pre-calculated combination of: alpha * m * m
+ /// 'alpha' corrects the raw cardinality estimate 'Z'. See the FlFuGaMe07 paper for its derivation.
+ const double alpha_mm;
+
+ /// The number of bits that is required per register.
+ ///
+ /// This number is determined by the maximum number of leading binary zeros a hashcode can
+ /// produce. This is equal to the number of bits the hashcode returns. The current
+ /// implementation uses a 64-bit hashcode, this means 6-bits are (at most) needed to store the
+ /// number of leading zeros.
+ static constexpr UInt64 REGISTER_SIZE = 6;
+ static constexpr UInt64 REGISTER_WORD_MASK = (1ULL << REGISTER_SIZE) - 1;
+
+ /// The size of a word used for storing registers: 64 Bits.
+ static constexpr UInt64 WORD_SIZE = 64;
+
+ static constexpr UInt64 REGISTERS_PER_WORD = WORD_SIZE / REGISTER_SIZE;
+
+ /// Number of points used for interpolating the bias value.
+ static constexpr UInt64 K = 6;
+
+ inline static const std::vector<std::vector<double>> BIAS_DATA = {
+ // precision 4
+ {10,
Review Comment:
θΏιζ ΌεΌεδΈε§ζ―θ‘10δΈͺε
η΄ γ
##########
cpp-ch/local-engine/AggregateFunctions/AggregateFunctionUniqHyperLogLogPlusPlus.h:
##########
@@ -0,0 +1,3089 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <AggregateFunctions/IAggregateFunction.h>
+#include <AggregateFunctions/AggregateFunctionFactory.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <DataTypes/IDataType.h>
+#include <DataTypes/DataTypeAggregateFunction.h>
+#include <Common/FieldVisitorConvertToNumber.h>
+#include <Common/FieldVisitors.h>
+#include <Common/HashTable/Hash.h>
+#include <Common/HyperLogLogWithSmallSetOptimization.h>
+#include <Parsers/NullsAction.h>
+namespace DB
+{
+
+namespace ErrorCodes
+{
+extern const int PARAMETER_OUT_OF_BOUND;
+extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+extern const int INCORRECT_DATA;
+}
+
+struct HyperLogLogPlusPlusData
+{
+ explicit HyperLogLogPlusPlusData(double relative_sd_ = 0.05)
Review Comment:
Use Float64 for consistency
##########
cpp-ch/local-engine/Parser/AggregateFunctionParser.cpp:
##########
@@ -25,6 +25,7 @@
#include <Parser/TypeParser.h>
#include <Common/CHUtil.h>
#include <Common/Exception.h>
+#include <Common/logger_useful.h>
Review Comment:
Useless header
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on code in PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#discussion_r1950215441
##########
cpp-ch/local-engine/AggregateFunctions/AggregateFunctionUniqHyperLogLogPlusPlus.h:
##########
@@ -0,0 +1,3089 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <AggregateFunctions/IAggregateFunction.h>
+#include <AggregateFunctions/AggregateFunctionFactory.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <DataTypes/IDataType.h>
+#include <DataTypes/DataTypeAggregateFunction.h>
+#include <Common/FieldVisitorConvertToNumber.h>
+#include <Common/FieldVisitors.h>
+#include <Common/HashTable/Hash.h>
+#include <Common/HyperLogLogWithSmallSetOptimization.h>
+#include <Parsers/NullsAction.h>
+namespace DB
+{
+
+namespace ErrorCodes
+{
+extern const int PARAMETER_OUT_OF_BOUND;
+extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH;
+extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+extern const int INCORRECT_DATA;
+}
+
+struct HyperLogLogPlusPlusData
+{
+ explicit HyperLogLogPlusPlusData(double relative_sd_ = 0.05)
+ : relative_sd(relative_sd_)
+ , p(static_cast<UInt64>(std::ceil(2.0 * std::log(1.106 / relative_sd) / std::log(2.0))))
+ , idx_shift(64 - p)
+ , w_padding(1ULL << (p - 1))
+ , m(1ULL << p)
+ , num_words(m / REGISTERS_PER_WORD + 1)
+ , alpha_mm(computeAlphaMM())
+ {
+ if (p < 4)
+ throw Exception(
+ ErrorCodes::PARAMETER_OUT_OF_BOUND,
+ "HLL++ requires at least 4 bits for addressing instead of {}. Use a lower error, at most 39%",
+ p);
+
+ if (p > 25)
+ throw Exception(
+ ErrorCodes::PARAMETER_OUT_OF_BOUND,
+ "HLL++ requires at most 25 bits for addressing instead of {} to avoid allocating too much memory",
+ p);
+
+ // std::cout << "relative_sd:" << relative_sd << " p:" << p << " m:" << m << " num_words:" << num_words << " alpha_mm:" << alpha_mm
+ // << std::endl;
+ registers = PaddedPODArray<UInt64>(num_words, 0); // Initialize registers with zeros
+ }
+
+ void serialize(WriteBuffer & buf) const
+ {
+ writeBinaryLittleEndian(relative_sd, buf);
+
+ writeBinaryLittleEndian(registers.size(), buf);
+ for (const auto & word : registers)
+ writeBinaryLittleEndian(word, buf);
+ }
+
+ void deserialize(ReadBuffer & buf)
+ {
+ Float64 relative_rd_read;
+ readBinaryLittleEndian(relative_rd_read, buf);
+ if (relative_rd_read != relative_sd)
+ throw Exception(
+ ErrorCodes::INCORRECT_DATA, "The relative standard deviation {} isn't the expected one {}", relative_rd_read, relative_sd);
+
+ size_t registers_size = 0;
+ readBinaryLittleEndian(registers_size, buf);
+ if (registers_size != registers.size())
+ throw Exception(
+ ErrorCodes::INCORRECT_DATA, "The number of registers {} isn't the expected one {}", registers_size, registers.size());
+
+ for (size_t i = 0; i < registers_size; ++i)
+ readBinaryLittleEndian(registers[i], buf);
+ }
+
+ void add(UInt64 value)
+ {
+ // std::cout << "add:" << value << std::endl;
+ UInt64 x = value;
+ UInt64 idx = x >> idx_shift;
+ UInt64 w = (x << p) | w_padding;
+ UInt64 pw = __builtin_clzll(w) + 1;
+
+ UInt64 word_offset = idx / REGISTERS_PER_WORD;
+ UInt64 word = registers[word_offset];
+
+ UInt64 shift = (idx - word_offset * REGISTERS_PER_WORD) * REGISTER_SIZE;
+ UInt64 mask = REGISTER_WORD_MASK << shift;
+ UInt64 midx = (word & mask) >> shift;
+ // std::cout << "bucket[" << std::setw(3) << std::setfill('0') << idx << "]=" << midx << " input:" << value << " pw:" << pw;
+ if (pw > midx)
+ {
+ // std::cout << " updated";
+ registers[word_offset] = (word & ~mask) | (pw << shift);
+ }
+ // std::cout << std::endl;
+ }
+
+ void merge(const HyperLogLogPlusPlusData & other)
+ {
+ size_t i = 0; // global register index
+ for (size_t word_i = 0; word_i < num_words; ++word_i)
+ {
+ UInt64 word1 = registers[word_i];
+ UInt64 word2 = other.registers[word_i];
+ UInt64 mask = REGISTER_WORD_MASK;
+ UInt64 merged_word = 0;
+ for (size_t register_i = 0; i < m && register_i < REGISTERS_PER_WORD; ++register_i, ++i)
+ {
+ merged_word |= std::max((word1 & mask), (word2 & mask));
+ mask <<= REGISTER_SIZE;
+ }
+ registers[word_i] = merged_word;
+ }
+ }
+
+ UInt64 query() const
+ {
+ double z_inverse = 0.0;
+ UInt64 v = 0;
+
+ size_t i = 0; // global register index
+ for (size_t word_i = 0; word_i < num_words; ++word_i)
+ {
+ UInt64 word = registers[word_i];
+ for (size_t register_i = 0; i < m && register_i < REGISTERS_PER_WORD; ++register_i, ++i)
+ {
+ UInt64 midx = (word >> (register_i * REGISTER_SIZE)) & REGISTER_WORD_MASK;
+ // std::cout << "bucket[" << std::setw(3) << std::setfill('0') << i << "]=" << midx << std::endl;
+ z_inverse += 1.0 / (1ULL << midx);
+
+ if (midx == 0)
+ ++v;
+ }
+ }
+ // std::cout << "num_zero:" << v << std::endl;
+
+ double e = alpha_mm / z_inverse;
+ // std::cout << "e:" << e << std::endl;
+
+ double e_bias_corrected = e;
+ if (p < 19 && e < 5.0 * m)
+ e_bias_corrected = e - estimateBias(e);
+
+ // std::cout << "e_bias_corrected:" << e_bias_corrected << std::endl;
+
+ if (v > 0)
+ {
+ double h = m * std::log(static_cast<double>(m) / v);
+ if ((p < 19 && h <= THRESHOLDS[p - 4]) || e <= 2.5 * m)
+ {
+ // std::cout << "h:" << h << std::endl;
+ return static_cast<UInt64>(std::round(h));
+ }
+ else
+ return static_cast<UInt64>(std::round(e_bias_corrected));
+ }
+ else
+ return static_cast<UInt64>(std::round(e_bias_corrected));
+ }
+
+private:
+ Float64 computeAlphaMM() const
+ {
+ // Compute alpha * m * m based on the value of m
+ if (p == 4)
+ return 0.673 * m * m;
+ else if (p == 5)
+ return 0.697 * m * m;
+ else if (p == 6)
+ return 0.709 * m * m;
+ else
+ return (0.7213 / (1 + 1.079 / m)) * m * m;
+ }
+
+
+ /// Estimate the bias using the raw estimates with their respective biases from the HLL++
+ /// appendix. We currently use KNN interpolation to determine the bias (as suggested in the
+ /// paper).
+ double estimateBias(double e) const
+ {
+ const auto & estimates = RAW_ESTIMATE_DATA[p - 4];
+ size_t num_estimates = estimates.size();
+
+ // Binary search to find the nearest estimate index
+ auto nearest_estimate_it = std::lower_bound(estimates.begin(), estimates.end(), e);
+ size_t nearest_estimate_index = std::distance(estimates.begin(), nearest_estimate_it);
+
+ // Distance metric: square of the difference
+ auto distance = [&](size_t i)
+ {
+ double diff = e - estimates[i];
+ return diff * diff;
+ };
+
+ // Adjust bounds
+ size_t low = nearest_estimate_index + 1 > K ? nearest_estimate_index - K + 1 : 0;
+ size_t high = std::min(low + K, num_estimates);
+ while (high < num_estimates && distance(high) < distance(low))
+ {
+ ++low;
+ ++high;
+ }
+
+ // Sum biases in the interval
+ const auto & biases = BIAS_DATA[p - 4];
+ double bias_sum = 0.0;
+ for (size_t i = low; i < high; ++i)
+ bias_sum += biases[i];
+
+ // Calculate the bias
+ return bias_sum / (high - low);
+ }
+
+ /// Defines the maximum relative standard deviation allowed.
+ const Float64 relative_sd;
+
+ /// HLL++ uses 'p' bits for bucket addressing.
+ const UInt64 p;
+
+ /// Shift used to extract the index of the register from the hashed value.
+ /// This assumes the use of 64-bit hashcodes.
+ const UInt64 idx_shift;
+
+ /// Value to pad the 'w' value with before the number of leading zeros is determined.
+ const UInt64 w_padding;
+
+ /// The number of registers used.
+ const UInt64 m;
+
+ /// The number of words used to store the registers. We use Longs for storage because this is the
+ /// most compact way of storage; Spark aligns to 8-byte words or uses Long wrappers.
+
+ /// We only store whole registers per word in order to prevent overly complex bitwise operations.
+ /// In practice this means we only use 60 out of 64 bits.
+ const UInt64 num_words;
+
+ /// The pre-calculated combination of: alpha * m * m
+ /// 'alpha' corrects the raw cardinality estimate 'Z'. See the FlFuGaMe07 paper for its derivation.
+ const double alpha_mm;
+
+ /// The number of bits that is required per register.
+ ///
+ /// This number is determined by the maximum number of leading binary zeros a hashcode can
+ /// produce. This is equal to the number of bits the hashcode returns. The current
+ /// implementation uses a 64-bit hashcode, this means 6-bits are (at most) needed to store the
+ /// number of leading zeros.
+ static constexpr UInt64 REGISTER_SIZE = 6;
+ static constexpr UInt64 REGISTER_WORD_MASK = (1ULL << REGISTER_SIZE) - 1;
+
+ /// The size of a word used for storing registers: 64 Bits.
+ static constexpr UInt64 WORD_SIZE = 64;
+
+ static constexpr UInt64 REGISTERS_PER_WORD = WORD_SIZE / REGISTER_SIZE;
+
+ /// Number of points used for interpolating the bias value.
+ static constexpr UInt64 K = 6;
+
+ inline static const std::vector<std::vector<double>> BIAS_DATA = {
+ // precision 4
+ {10,
Review Comment:
It's already formatted by clang-format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on code in PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#discussion_r1950215779
##########
cpp-ch/local-engine/Parser/AggregateFunctionParser.cpp:
##########
@@ -25,6 +25,7 @@
#include <Parser/TypeParser.h>
#include <Common/CHUtil.h>
#include <Common/Exception.h>
+#include <Common/logger_useful.h>
Review Comment:
Required by `LOG_INFO`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2653177620
Run Gluten ClickHouse CI on ARM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2655276746
Run Gluten ClickHouse CI on ARM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "taiyang-li (via GitHub)" <gi...@apache.org>.
taiyang-li commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2647317891
``` sql
0: jdbc:hive2://localhost:10000/> select approx_count_distinct(id, 0.001), approx_count_distinct(id, 0.01), approx_count_distinct(id, 0.1) from range(1000);
+----------------------------+----------------------------+----------------------------+
| approx_count_distinct(id) | approx_count_distinct(id) | approx_count_distinct(id) |
+----------------------------+----------------------------+----------------------------+
| 999 | 996 | 928 |
+----------------------------+----------------------------+----------------------------+
1 row selected (5.82 seconds)
0: jdbc:hive2://localhost:10000/>
0: jdbc:hive2://localhost:10000/> set spark.gluten.enabled = false;
+-----------------------+--------+
| key | value |
+-----------------------+--------+
| spark.gluten.enabled | false |
+-----------------------+--------+
1 row selected (0.137 seconds)
0: jdbc:hive2://localhost:10000/> select approx_count_distinct(id, 0.001), approx_count_distinct(id, 0.01), approx_count_distinct(id, 0.1) from range(1000);
+----------------------------+----------------------------+----------------------------+
| approx_count_distinct(id) | approx_count_distinct(id) | approx_count_distinct(id) |
+----------------------------+----------------------------+----------------------------+
| 999 | 996 | 928 |
+----------------------------+----------------------------+----------------------------+
1 row selected (149.915 seconds)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2649592547
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2647393719
Run Gluten ClickHouse CI on ARM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2647340622
Run Gluten ClickHouse CI on ARM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2647417265
Run Gluten ClickHouse CI on ARM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2658283790
@taiyang-li it's my fault that we dont't run spark 35 ut for arm. this pr cause spark 35 ut failed.
see https://github.com/apache/incubator-gluten/issues/8728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-8528][CH]Support approx_count_distinct [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #8550:
URL: https://github.com/apache/incubator-gluten/pull/8550#issuecomment-2652555230
Run Gluten ClickHouse CI on ARM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org