You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "malinjawi (via GitHub)" <gi...@apache.org> on 2026/06/01 12:16:14 UTC
[PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
malinjawi opened a new pull request, #12214:
URL: https://github.com/apache/gluten/pull/12214
What changes are proposed in this pull request?
This PR is the next split for Delta deletion-vector MoR support. It adds the native bitmap primitive needed by later DELETE DV work, without changing DELETE routing or enabling native bitmap construction in the command path yet.
Main changes:
- extend `RoaringBitmapArray` for Delta Portable-format deletion-vector payloads
- add bounded deserialization using CRoaring portable deserialize sizing before `readSafe`
- add native `bitmapaggregator` support for Delta row-index aggregation
- wire the aggregate name through Gluten expression/substrait planning
- add focused native tests for bitmap serialization/deserialization and aggregate behavior
- add `delta_bitmap_benchmark` with construction, partial-merge, and deserialize/probe cases
This PR is intentionally primitive-only:
- no DELETE command routing changes
- no DML row-index scan planning changes
- no plain Parquet target scan optimization
- no native bitmap aggregation enabled as the default DELETE path
Those pieces remain in follow-up split PRs after the primitive and benchmark shape are reviewed.
How was this patch tested?
Post-rebase validation on top of current `upstream/main` (`33be6fb8bf703ac16eae3c75efa919a97d9cdf5a`):
- `git diff --check upstream/main...HEAD`
- `env JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home PATH=/opt/homebrew/opt/openjdk@17/bin:$PATH ./build/mvn test-compile -pl backends-velox -am -Pjava-17,spark-3.5,backends-velox,hadoop-3.3,spark-ut,delta -DskipTests`
- `env JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home PATH=/opt/homebrew/opt/openjdk@17/bin:$PATH ./build/mvn test-compile -pl backends-velox -am -Pjava-17,spark-4.0,scala-2.13,backends-velox,hadoop-3.3,spark-ut,delta -DskipTests`
Focused standalone native validation from the same diff before the final rebase:
- standalone `RoaringBitmapArrayTest`: passed all 9 focused tests
- Delta JVM compatibility: JVM-generated sparse-gap portable fixture for values `1`, `7`, and `1 << 33` is read by native code; native compact portable payload for the same values is read by a Delta 3.3.2 JVM helper with cardinality `3`, all expected contains checks, and last value `8589934592`
- standalone `delta_bitmap_benchmark` construction/merge output: `/tmp/delta_bitmap_benchmark_delete_construction.json`
- standalone `delta_bitmap_benchmark` read/probe output: `/tmp/delta_bitmap_benchmark_read_probe.json`
Benchmark highlights from the standalone run:
- contiguous 1M build+serialize: `7.91 ms`, `132.5M rows/s`
- sparse 1M build+serialize: `9.99 ms`, `105.0M rows/s`
- clustered 1M build+serialize: `10.10 ms`, `103.9M rows/s`
- multi-bucket 256K build+serialize: `2.28 ms`, `114.9M rows/s`
- sparse 1M merge from 64 partials: `1.12 ms`
- contiguous round-robin merge from 64 partials: `1.32 ms`
- sparse deserialize+probe: `487 us` for an 8,192-probe sample
Notes:
- Normal local Gluten C++ target validation is currently blocked by local Velox/build-tree setup issues, so this draft PR is opened to get the regular native CI signal.
- `clang-format` was not available on this local machine after the final rebase; C++ format CI should validate formatting.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: IBM BOB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12214:
URL: https://github.com/apache/gluten/pull/12214#issuecomment-4728093893
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #12214:
URL: https://github.com/apache/gluten/pull/12214#discussion_r3421877638
##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/AggregateFunctionsBuilder.scala:
##########
@@ -62,4 +66,14 @@ object AggregateFunctionsBuilder {
}
}
}
+
+ private def isPortableDeltaBitmapAggregator(aggregateFunc: AggregateFunction): Boolean = {
+ aggregateFunc.prettyName == ExpressionNames.BITMAP_AGGREGATOR &&
+ Try {
+ aggregateFunc.getClass
+ .getMethod("serializationFormatString")
+ .invoke(aggregateFunc)
+ .asInstanceOf[String]
+ }.toOption.contains("Portable")
+ }
Review Comment:
Would you explain a bit more about the method? Why reflection is needed, etc.
##########
cpp/velox/operators/functions/DeltaBitmapAggregator.h:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+
+namespace gluten {
+
+void registerDeltaBitmapAggregator(
Review Comment:
Can we add a new source folder `operators/functions/delta`, for all Delta functions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12214:
URL: https://github.com/apache/gluten/pull/12214#issuecomment-4720648020
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12214:
URL: https://github.com/apache/gluten/pull/12214#issuecomment-4720782482
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #12214:
URL: https://github.com/apache/gluten/pull/12214#issuecomment-5037682330
Contributes to #11901
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12214:
URL: https://github.com/apache/gluten/pull/12214#issuecomment-4592709406
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12214:
URL: https://github.com/apache/gluten/pull/12214#issuecomment-4717064466
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12214:
URL: https://github.com/apache/gluten/pull/12214#issuecomment-4708016555
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on code in PR #12214:
URL: https://github.com/apache/gluten/pull/12214#discussion_r3426991500
##########
cpp/velox/operators/functions/delta/DeltaBitmapAggregator.h:
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+
+namespace gluten {
+
+void registerDeltaBitmapAggregator(
Review Comment:
Done, moved them into operators/functions/delta/ and fixed up the include + test + CMake (cb2f28772). Future Delta functions can land there too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12214:
URL: https://github.com/apache/gluten/pull/12214#issuecomment-4592628803
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12214:
URL: https://github.com/apache/gluten/pull/12214#issuecomment-4592559031
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer merged PR #12214:
URL: https://github.com/apache/gluten/pull/12214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on code in PR #12214:
URL: https://github.com/apache/gluten/pull/12214#discussion_r3426989935
##########
gluten-substrait/src/main/scala/org/apache/gluten/expression/AggregateFunctionsBuilder.scala:
##########
@@ -62,4 +66,14 @@ object AggregateFunctionsBuilder {
}
}
}
+
+ private def isPortableDeltaBitmapAggregator(aggregateFunc: AggregateFunction): Boolean = {
+ aggregateFunc.prettyName == ExpressionNames.BITMAP_AGGREGATOR &&
+ Try {
+ aggregateFunc.getClass
+ .getMethod("serializationFormatString")
+ .invoke(aggregateFunc)
+ .asInstanceOf[String]
+ }.toOption.contains("Portable")
+ }
Review Comment:
Yeah, dug into this and the reflection wasn't actually buying us anything, so I dropped it it's just a prettyName match now (cb2f28772).
The Portable check was me being paranoid. Delta only ever builds this aggregator in one spot (DMLWithDeletionVectorsHelper) and always with the Portable format, so it never actually filtered anything. And if a weird non-Portable payload ever did show up, the native deserialize already throws on the magic number so the real check is on the C++ side anyway. Bonus: gluten-substrait no longer needs to know anything about delta-spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #12214:
URL: https://github.com/apache/gluten/pull/12214#discussion_r3426628061
##########
cpp/velox/operators/functions/delta/DeltaBitmapAggregator.cc:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "operators/functions/delta/DeltaBitmapAggregator.h"
+
+#include <memory>
+#include <optional>
+#include <tuple>
+#include <vector>
+
+#include "compute/delta/RoaringBitmapArray.h"
+#include "velox/exec/SimpleAggregateAdapter.h"
+#include "velox/expression/FunctionSignature.h"
+#include "velox/type/SimpleFunctionApi.h"
+
+using namespace facebook::velox;
+
+namespace gluten {
+namespace {
+
+class DeltaBitmapAggregatorFunction {
+ public:
+ using InputType = Row<int64_t>;
+ using IntermediateType = Varbinary;
+ using OutputType = Row<int64_t, int64_t, Varbinary>;
+
+ static constexpr bool default_null_behavior_ = false;
+
+ static void addRowIndex(delta::RoaringBitmapArray& bitmap, int64_t value) {
+ VELOX_CHECK_GE(value, 0, "Delta bitmap row index cannot be negative: {}", value);
+ bitmap.addSafe(static_cast<uint64_t>(value));
+ }
+
+ static bool toIntermediate(exec::out_type<IntermediateType>& out, exec::optional_arg_type<int64_t> value) {
+ delta::RoaringBitmapArray bitmap;
+ if (value.has_value()) {
+ addRowIndex(bitmap, value.value());
+ }
+ const auto serialized = bitmap.serializeToString();
+ out.copy_from(StringView(serialized.data(), serialized.size()));
+ return true;
+ }
+
+ struct AccumulatorType {
+ static constexpr bool use_external_memory_ = true;
+
+ explicit AccumulatorType(HashStringAllocator* /* allocator */, DeltaBitmapAggregatorFunction* /* fn */) {}
+
+ bool addInput(HashStringAllocator* /* allocator */, exec::optional_arg_type<int64_t> value) {
+ if (!value.has_value()) {
+ return false;
+ }
+ addRowIndex(bitmap, value.value());
+ return true;
+ }
+
+ bool combine(HashStringAllocator* /* allocator */, exec::optional_arg_type<IntermediateType> other) {
+ if (!other.has_value()) {
+ return false;
+ }
+ const auto serialized = other.value();
+ delta::RoaringBitmapArray otherBitmap;
+ otherBitmap.deserialize(serialized.data(), serialized.size());
+ bitmap.merge(otherBitmap);
+ return true;
+ }
+
+ bool writeIntermediateResult(bool /* nonNullGroup */, exec::out_type<IntermediateType>& out) {
+ const auto serialized = bitmap.serializeToString();
+ out.copy_from(StringView(serialized.data(), serialized.size()));
+ return true;
+ }
+
+ bool writeFinalResult(bool /* nonNullGroup */, exec::out_type<OutputType>& out) {
+ const auto serialized = bitmap.serializeToString(true);
+ const auto last = bitmap.last();
+ out.copy_from(std::make_tuple(
+ static_cast<int64_t>(bitmap.cardinality()),
+ last.has_value() ? std::optional<int64_t>(static_cast<int64_t>(*last)) : std::nullopt,
+ StringView(serialized.data(), serialized.size())));
+ return true;
+ }
+
+ delta::RoaringBitmapArray bitmap;
+ };
+};
+
+} // namespace
+
+void registerDeltaBitmapAggregator(const std::string& prefix, bool withCompanionFunctions, bool overwrite) {
+ std::vector<std::shared_ptr<exec::AggregateFunctionSignature>> signatures{
+ exec::AggregateFunctionSignatureBuilder()
+ .argumentType("bigint")
+ .intermediateType("varbinary")
+ .returnType("row(bigint,bigint,varbinary)")
+ .build()};
+
+ exec::registerAggregateFunction(
+ prefix + "bitmapaggregator",
+ std::move(signatures),
+ [](core::AggregationNode::Step step,
+ const std::vector<TypePtr>& argTypes,
+ const TypePtr& resultType,
+ const core::QueryConfig& /* config */) -> std::unique_ptr<exec::Aggregate> {
+ VELOX_CHECK_EQ(argTypes.size(), 1, "bitmapaggregator takes one argument");
+ VELOX_CHECK_EQ(argTypes[0]->kind(), exec::isRawInput(step) ? TypeKind::BIGINT : TypeKind::VARBINARY);
Review Comment:
`bitmapaggregator` is registered here, but Velox plan validation still gates aggregate support via a hardcoded allow-list (cpp/velox/substrait/SubstraitToVeloxPlanValidator.cc:1294-1327). That allow-list currently includes `bitmap_construct_agg` but not `bitmapaggregator`, so Spark-side planning will likely treat this aggregate as unsupported and fall back instead of executing natively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [VL][Delta] Add native Delta bitmap aggregation support [gluten]
Posted by "malinjawi (via GitHub)" <gi...@apache.org>.
malinjawi commented on code in PR #12214:
URL: https://github.com/apache/gluten/pull/12214#discussion_r3427004188
##########
cpp/velox/operators/functions/delta/DeltaBitmapAggregator.cc:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "operators/functions/delta/DeltaBitmapAggregator.h"
+
+#include <memory>
+#include <optional>
+#include <tuple>
+#include <vector>
+
+#include "compute/delta/RoaringBitmapArray.h"
+#include "velox/exec/SimpleAggregateAdapter.h"
+#include "velox/expression/FunctionSignature.h"
+#include "velox/type/SimpleFunctionApi.h"
+
+using namespace facebook::velox;
+
+namespace gluten {
+namespace {
+
+class DeltaBitmapAggregatorFunction {
+ public:
+ using InputType = Row<int64_t>;
+ using IntermediateType = Varbinary;
+ using OutputType = Row<int64_t, int64_t, Varbinary>;
+
+ static constexpr bool default_null_behavior_ = false;
+
+ static void addRowIndex(delta::RoaringBitmapArray& bitmap, int64_t value) {
+ VELOX_CHECK_GE(value, 0, "Delta bitmap row index cannot be negative: {}", value);
+ bitmap.addSafe(static_cast<uint64_t>(value));
+ }
+
+ static bool toIntermediate(exec::out_type<IntermediateType>& out, exec::optional_arg_type<int64_t> value) {
+ delta::RoaringBitmapArray bitmap;
+ if (value.has_value()) {
+ addRowIndex(bitmap, value.value());
+ }
+ const auto serialized = bitmap.serializeToString();
+ out.copy_from(StringView(serialized.data(), serialized.size()));
+ return true;
+ }
+
+ struct AccumulatorType {
+ static constexpr bool use_external_memory_ = true;
+
+ explicit AccumulatorType(HashStringAllocator* /* allocator */, DeltaBitmapAggregatorFunction* /* fn */) {}
+
+ bool addInput(HashStringAllocator* /* allocator */, exec::optional_arg_type<int64_t> value) {
+ if (!value.has_value()) {
+ return false;
+ }
+ addRowIndex(bitmap, value.value());
+ return true;
+ }
+
+ bool combine(HashStringAllocator* /* allocator */, exec::optional_arg_type<IntermediateType> other) {
+ if (!other.has_value()) {
+ return false;
+ }
+ const auto serialized = other.value();
+ delta::RoaringBitmapArray otherBitmap;
+ otherBitmap.deserialize(serialized.data(), serialized.size());
+ bitmap.merge(otherBitmap);
+ return true;
+ }
+
+ bool writeIntermediateResult(bool /* nonNullGroup */, exec::out_type<IntermediateType>& out) {
+ const auto serialized = bitmap.serializeToString();
+ out.copy_from(StringView(serialized.data(), serialized.size()));
+ return true;
+ }
+
+ bool writeFinalResult(bool /* nonNullGroup */, exec::out_type<OutputType>& out) {
+ const auto serialized = bitmap.serializeToString(true);
+ const auto last = bitmap.last();
+ out.copy_from(std::make_tuple(
+ static_cast<int64_t>(bitmap.cardinality()),
+ last.has_value() ? std::optional<int64_t>(static_cast<int64_t>(*last)) : std::nullopt,
+ StringView(serialized.data(), serialized.size())));
+ return true;
+ }
+
+ delta::RoaringBitmapArray bitmap;
+ };
+};
+
+} // namespace
+
+void registerDeltaBitmapAggregator(const std::string& prefix, bool withCompanionFunctions, bool overwrite) {
+ std::vector<std::shared_ptr<exec::AggregateFunctionSignature>> signatures{
+ exec::AggregateFunctionSignatureBuilder()
+ .argumentType("bigint")
+ .intermediateType("varbinary")
+ .returnType("row(bigint,bigint,varbinary)")
+ .build()};
+
+ exec::registerAggregateFunction(
+ prefix + "bitmapaggregator",
+ std::move(signatures),
+ [](core::AggregationNode::Step step,
+ const std::vector<TypePtr>& argTypes,
+ const TypePtr& resultType,
+ const core::QueryConfig& /* config */) -> std::unique_ptr<exec::Aggregate> {
+ VELOX_CHECK_EQ(argTypes.size(), 1, "bitmapaggregator takes one argument");
+ VELOX_CHECK_EQ(argTypes[0]->kind(), exec::isRawInput(step) ? TypeKind::BIGINT : TypeKind::VARBINARY);
Review Comment:
fixed in 302653952
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org