You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "Prajwal-banakar (via GitHub)" <gi...@apache.org> on 2026/06/16 16:22:25 UTC

[PR] [FIP-37] Add bitmap scalar functions and register via FlussCatalog [fluss]

Prajwal-banakar opened a new pull request, #3492:
URL: https://github.com/apache/fluss/pull/3492

   
   ### Purpose
   
   Linked issue: Part of #3289
   
   This PR completes the Phase 1 implementation of FIP-37 by adding the six RoaringBitmap scalar functions. It builds on the infrastructure from PR #3319  and the aggregate functions from PR #3398 
   
   These functions are registered as built-in functions in `FlussCatalog`, allowing users to call them directly in Flink SQL without manual `CREATE FUNCTION` statements.
   
   ### Brief change log
   
   **New files in `fluss-flink/fluss-flink-common`:**
   
   *   `functions/bitmap/RbCardinalityFunction.java`: `rb_cardinality(BYTES) -> BIGINT` — Returns the number of distinct integers in the bitmap.
   *   `functions/bitmap/RbBuildFunction.java`: `rb_build(ARRAY<INT>) -> BYTES` — Constructs a bitmap from an array of integers.
   *   `functions/bitmap/RbContainsFunction.java`: `rb_contains(BYTES, INT) -> BOOLEAN` — Checks if a bitmap contains a specific integer.
   *   `functions/bitmap/RbToArrayFunction.java`: `rb_to_array(BYTES) -> ARRAY<INT>` — Converts a bitmap into an array of its integers.
   *   `functions/bitmap/RbOrFunction.java`: `rb_or(BYTES, BYTES) -> BYTES` — Computes the union of two bitmaps.
   *   `functions/bitmap/RbAndFunction.java`: `rb_and(BYTES, BYTES) -> BYTES` — Computes the intersection of two bitmaps.
   *   `test/java/.../functions/bitmap/RbScalarFunctionsTest.java`: Unit tests for all six scalar functions, covering correctness, null handling, and edge cases.
   
   **Modified files:**
   
   *   `flink/catalog/FlinkCatalog.java`: Registers the six new scalar functions in the `BUILTIN_BITMAP_FUNCTIONS` map.
   *   `test/.../catalog/FlinkCatalogTest.java`: Adds a unit test to verify the scalar functions are correctly registered.
   *   `test/.../functions/bitmap/RbFunctionsCatalogITCase.java`: Adds a new integration test (`testScalarFunctions`) that exercises all scalar functions through a `TableEnvironment`, validating the end-to-end catalog registration and execution path.
   
   ### Tests
   
   *   **Unit tests:** `RbScalarFunctionsTest` contains comprehensive tests for each new scalar function.
   *   **Catalog integration:** `FlinkCatalogTest` verifies that the functions are discoverable through the catalog API.
   *   **End-to-end SQL ITCase:** `RbFunctionsCatalogITCase` now includes a test that runs SQL queries with `rb_cardinality`, `rb_build`, `rb_contains`, `rb_to_array`, `rb_or`, and `rb_and` against a live Flink cluster.
   
   **Verified locally:**
   
   *   `./mvnw spotless:apply` — 0 violations
   *   `./mvnw test -pl fluss-flink/fluss-flink-common -Dtest="RbScalarFunctionsTest,FlinkCatalogTest"` — BUILD SUCCESS
   *   `./mvnw verify -pl fluss-flink/fluss-flink-common -Dit.test=RbFunctionsCatalogITCase` — BUILD SUCCESS
   
   ### API and Format
   
   This change is purely additive and does not affect any storage formats or wire protocols. The functions operate on `BYTES` columns containing standard RoaringBitmap serialized data.
   
   ### Documentation
   
   User-facing documentation will be created in a follow-up PR now that the full set of Phase 1 functions is complete.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FIP-37] Add bitmap scalar functions and register via FlussCatalog [fluss]

Posted by "platinumhamburg (via GitHub)" <gi...@apache.org>.
platinumhamburg commented on code in PR #3492:
URL: https://github.com/apache/fluss/pull/3492#discussion_r3517899049


##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/functions/bitmap/RbOrFunction.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.flink.functions.bitmap;
+
+import org.apache.flink.table.functions.ScalarFunction;
+import org.roaringbitmap.RoaringBitmap;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/**
+ * {@code rb_or(left BYTES, right BYTES) -> BYTES}
+ *
+ * <p>Returns the bitwise OR (union) of two serialized {@link RoaringBitmap} values. If one argument
+ * is null, the other is returned as-is. Returns {@code null} only if both are null.
+ */
+public class RbOrFunction extends ScalarFunction {
+
+    /**
+     * @param leftBytes serialized left bitmap
+     * @param rightBytes serialized right bitmap
+     * @return union of left and right, or null if both are null
+     */
+    @Nullable
+    public byte[] eval(@Nullable byte[] leftBytes, @Nullable byte[] rightBytes) throws IOException {
+        if (leftBytes == null && rightBytes == null) {
+            return null;
+        }
+        if (leftBytes == null) {
+            return rightBytes;
+        }
+        if (rightBytes == null) {

Review Comment:
   rb_or applies the aggregate-style NULL-as-ignored behavior to a scalar function. FIP-37 explicitly documents that rb_or_agg ignores NULL inputs, but also distinguishes NULL from empty bitmap. Unless scalar rb_or is specified to treat NULL as identity, this should probably return NULL when either argument is NULL, or the scalar NULL semantics need to be documented and tested.



##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/functions/bitmap/RbBuildFunction.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.flink.functions.bitmap;
+
+import org.apache.flink.table.functions.ScalarFunction;
+import org.roaringbitmap.RoaringBitmap;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/**
+ * {@code rb_build(v1 INT, v2 INT, ...) -> BYTES}
+ *
+ * <p>Builds a serialized {@link RoaringBitmap} from a variadic list of 32-bit integer values within
+ * a single row. Unlike {@code rb_build_agg}, this function operates on individual column values in
+ * the same row rather than aggregating across rows. Null values are ignored. Returns {@code null}
+ * if all inputs are null or no inputs are provided.
+ */
+public class RbBuildFunction extends ScalarFunction {
+
+    /**
+     * @param values variadic integer values to add to the bitmap; null values are ignored
+     * @return serialized bitmap, or null if all inputs are null
+     */
+    @Nullable
+    public byte[] eval(@Nullable Integer... values) throws IOException {

Review Comment:
   It looks like the implementation does not match the advertised signature for `rb_build`.
   
   The PR description says `rb_build(ARRAY<INT>) -> BYTES`, but the UDF currently exposes:
   ```java
   public byte[] eval(@Nullable Integer... values)
   ```
   In Flink UDF type inference, Java varargs are treated as a varying sequence of arguments rather than a single ARRAY argument. So this supports SQL like:
   ```sql
   rb_build(1, 2, 3)
   ```
   but it is different from:
   ```sql
   rb_build(ARRAY[1, 2, 3])
   ```
   If the intended API is rb_build(ARRAY<INT>) -> BYTES, could we change the implementation to accept a non-varargs array/list-style parameter and add an SQL test for rb_build(ARRAY[1, 2, 3]) or an array column? Otherwise, the PR description and function Javadoc should document the variadic signature instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FIP-37] Add bitmap scalar functions and register via FlussCatalog [fluss]

Posted by "polyzos (via GitHub)" <gi...@apache.org>.
polyzos commented on PR #3492:
URL: https://github.com/apache/fluss/pull/3492#issuecomment-5036586400

   @Prajwal-banakar I will merge once the CI passes. Do we need any documentation here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FIP-37] Add bitmap scalar functions and register via FlussCatalog [fluss]

Posted by "Prajwal-banakar (via GitHub)" <gi...@apache.org>.
Prajwal-banakar commented on PR #3492:
URL: https://github.com/apache/fluss/pull/3492#issuecomment-4880929801

   Hi @platinumhamburg Thanks for the feedback, adressed the comments PTAL another look when you  have ssome time, Thanks again!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FIP-37] Add bitmap scalar functions and register via FlussCatalog [fluss]

Posted by "Prajwal-banakar (via GitHub)" <gi...@apache.org>.
Prajwal-banakar commented on PR #3492:
URL: https://github.com/apache/fluss/pull/3492#issuecomment-5042188557

   Hi @polyzos, no docs needed to block this merge. All three Phase 1 PRs committed to a follow-up documentation PR once the full function set was complete, and that's what I'll open next.
   
   For the documentation itself, I see two approaches and wanted your input on which fits better:
   
   1. Add a new section to the existing website/docs/table-design/merge-engines/aggregation.md under a "RoaringBitmap SQL Functions" heading. Simpler, keeps everything bitmap-related in one place.
   
   2. Create a new standalone page website/docs/table-design/bitmap-functions.md. Cleaner separation since the FIP-37 functions are Flink SQL UDFs registered via FlussCatalog, which is a different concept from the server-side rbm32/rbm64 merge engine aggregators already documented there.
   
   Worth noting that @platinumhamburg 's  [UV deduplication](https://fluss.apache.org/blog/roaringbitmap-uv-deduplication/) blog post already covers the end-to-end workflow well, so the docs page can reference that and stay focused on the function reference (signatures, null semantics, examples).
   
   Happy to go with whichever approach you prefer, just let me know!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FIP-37] Add bitmap scalar functions and register via FlussCatalog [fluss]

Posted by "platinumhamburg (via GitHub)" <gi...@apache.org>.
platinumhamburg commented on PR #3492:
URL: https://github.com/apache/fluss/pull/3492#issuecomment-4889602768

   > Hi @platinumhamburg Thanks for the feedback, adressed the comments PTAL another look when you have some time, Thanks again!
   
   @Prajwal-banakar Thanks, LGTM. @polyzos @wuchong Would you like to take another look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FIP-37] Add bitmap scalar functions and register via FlussCatalog [fluss]

Posted by "Prajwal-banakar (via GitHub)" <gi...@apache.org>.
Prajwal-banakar commented on code in PR #3492:
URL: https://github.com/apache/fluss/pull/3492#discussion_r3522686439


##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/functions/bitmap/RbOrFunction.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.flink.functions.bitmap;
+
+import org.apache.flink.table.functions.ScalarFunction;
+import org.roaringbitmap.RoaringBitmap;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/**
+ * {@code rb_or(left BYTES, right BYTES) -> BYTES}
+ *
+ * <p>Returns the bitwise OR (union) of two serialized {@link RoaringBitmap} values. If one argument
+ * is null, the other is returned as-is. Returns {@code null} only if both are null.
+ */
+public class RbOrFunction extends ScalarFunction {
+
+    /**
+     * @param leftBytes serialized left bitmap
+     * @param rightBytes serialized right bitmap
+     * @return union of left and right, or null if both are null
+     */
+    @Nullable
+    public byte[] eval(@Nullable byte[] leftBytes, @Nullable byte[] rightBytes) throws IOException {
+        if (leftBytes == null && rightBytes == null) {
+            return null;
+        }
+        if (leftBytes == null) {
+            return rightBytes;
+        }
+        if (rightBytes == null) {

Review Comment:
   I've updated rb_or to return NULL if either input is NULL, aligning it with standard SQL behavior for scalar functions. The corresponding unit tests have been updated to verify this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FIP-37] Add bitmap scalar functions and register via FlussCatalog [fluss]

Posted by "Prajwal-banakar (via GitHub)" <gi...@apache.org>.
Prajwal-banakar commented on PR #3492:
URL: https://github.com/apache/fluss/pull/3492#issuecomment-4722101808

   Hi @polyzos , could you please help review here!?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FIP-37] Add bitmap scalar functions and register via FlussCatalog [fluss]

Posted by "polyzos (via GitHub)" <gi...@apache.org>.
polyzos commented on PR #3492:
URL: https://github.com/apache/fluss/pull/3492#issuecomment-5042298881

   @Prajwal-banakar, what I'm thinking - if it makes sense to you - is to go with option 1 and then just create a small follow-up blog if you want to, just to show your work and explain how it differs now in terms of "how it works".
   
   Whatever works best for you, happy to support either


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FIP-37] Add bitmap scalar functions and register via FlussCatalog [fluss]

Posted by "polyzos (via GitHub)" <gi...@apache.org>.
polyzos merged PR #3492:
URL: https://github.com/apache/fluss/pull/3492


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FIP-37] Add bitmap scalar functions and register via FlussCatalog [fluss]

Posted by "Prajwal-banakar (via GitHub)" <gi...@apache.org>.
Prajwal-banakar commented on code in PR #3492:
URL: https://github.com/apache/fluss/pull/3492#discussion_r3522687307


##########
fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/functions/bitmap/RbBuildFunction.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.fluss.flink.functions.bitmap;
+
+import org.apache.flink.table.functions.ScalarFunction;
+import org.roaringbitmap.RoaringBitmap;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/**
+ * {@code rb_build(v1 INT, v2 INT, ...) -> BYTES}
+ *
+ * <p>Builds a serialized {@link RoaringBitmap} from a variadic list of 32-bit integer values within
+ * a single row. Unlike {@code rb_build_agg}, this function operates on individual column values in
+ * the same row rather than aggregating across rows. Null values are ignored. Returns {@code null}
+ * if all inputs are null or no inputs are provided.
+ */
+public class RbBuildFunction extends ScalarFunction {
+
+    /**
+     * @param values variadic integer values to add to the bitmap; null values are ignored
+     * @return serialized bitmap, or null if all inputs are null
+     */
+    @Nullable
+    public byte[] eval(@Nullable Integer... values) throws IOException {

Review Comment:
   I've corrected the signature to align with the documented rb_build(ARRAY<INT>) behavior. The eval method now accepts Integer[], and I've added an integration test case in RbFunctionsCatalogITCase using rb_build(ARRAY[...]) to confirm it works as expected.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org