You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jiangxt2 (via GitHub)" <gi...@apache.org> on 2026/08/18 03:53:31 UTC
[PR] [SPARK-58831][SQL] Add bitmap scalar set operation functions [spark]
jiangxt2 opened a new pull request, #58066:
URL: https://github.com/apache/spark/pull/58066
### What changes were proposed in this pull request?
This PR adds four scalar set operations for Spark's flat bitmap representation:
- `bitmap_and`
- `bitmap_or`
- `bitmap_andnot`
- `bitmap_xor`
Each function operates on two `BINARY` values from the same row. The implementation:
- accepts inputs up to 4096 bytes;
- treats missing trailing bytes as zero;
- returns a newly allocated, fixed-size 4096-byte bitmap;
- propagates `NULL` inputs;
- rejects oversized inputs with the structured `BITMAP_INPUT_TOO_LARGE` error;
- does not mutate either input.
The functions are implemented as Catalyst-native binary expressions with both interpreted
and whole-stage codegen paths. They are registered in the SQL function registry and exposed
through the Scala API, PySpark classic, and Spark Connect.
This PR also adds SQL documentation, Python documentation, a structured error definition,
Connect plan and schema golden files, and tests for the new APIs.
### Why are the changes needed?
Spark provides functions for constructing, aggregating, and counting flat bitmaps, but it
does not provide scalar set operations for combining two bitmap values from the same row.
Without native scalar functions, users need UDFs or external bitmap processing to compute
intersection, union, difference, or symmetric difference between precomputed bitmap columns.
Native Catalyst expressions provide consistent SQL semantics, structured validation, and
code generation, and allow these operations to compose naturally with the existing bitmap
aggregate and count functions.
### Does this PR introduce _any_ user-facing change?
Yes. It adds four new SQL, Scala, PySpark classic, and Spark Connect functions:
```sql
SELECT bitmap_and(left_bitmap, right_bitmap);
SELECT bitmap_or(left_bitmap, right_bitmap);
SELECT bitmap_andnot(left_bitmap, right_bitmap);
SELECT bitmap_xor(left_bitmap, right_bitmap);
```
`bitmap_andnot(left, right)` is directional and returns the bits present in `left` but not
in `right`.
No existing function behavior is changed.
### How was this patch tested?
The following targeted suites and build checks passed:
- `build/sbt "catalyst/testOnly org.apache.spark.sql.catalyst.expressions.BitmapExpressionUtilsSuite"`
(10 tests)
- `build/sbt "sql/testOnly org.apache.spark.sql.BitmapExpressionsQuerySuite"`
(19 tests, including interpreted and codegen execution)
- `build/sbt "sql/testOnly org.apache.spark.sql.ExpressionsSchemaSuite"`
- `build/sbt "connect-client-jvm/testOnly org.apache.spark.sql.PlanGenerationTestSuite -- -z 'function bitmap'"`
(9 tests)
- `build/sbt -Phive package`
- `python/run-tests --testnames "pyspark.sql.tests.test_functions FunctionsTests.test_bitmap_scalar_functions"`
- `python/run-tests --testnames "pyspark.sql.tests.connect.test_connect_plan SparkConnectPlanTests.test_bitmap_scalar_functions"`
- Targeted PySpark doctests for the four new functions (12 tests)
The tests cover variable-length and empty inputs, the 4096-byte boundary, input immutability,
null propagation, invalid types, oversized inputs on either side, interpreted and codegen
execution, aggregation composition, grouped queries, PySpark classic, and Spark Connect.
Scalafmt, Ruff, JSON validation, golden-file consistency checks, and `git diff --check` also
passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex and Claude AI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58831][SQL] Add bitmap scalar set operation functions [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #58066: [SPARK-58831][SQL] Add bitmap scalar set operation functions
URL: https://github.com/apache/spark/pull/58066
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58831][SQL] Add bitmap scalar set operation functions [spark]
Posted by "jiangxt2 (via GitHub)" <gi...@apache.org>.
jiangxt2 commented on PR #58066:
URL: https://github.com/apache/spark/pull/58066#issuecomment-5340774384
Hi @uros-b, would you mind taking a look at this PR when you have time? All CI checks are green. Thanks! ^_^
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58831][SQL] Add bitmap scalar set operation functions [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #58066:
URL: https://github.com/apache/spark/pull/58066#issuecomment-5356962474
can you resolve merge conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58831][SQL] Add bitmap scalar set operation functions [spark]
Posted by "jiangxt2 (via GitHub)" <gi...@apache.org>.
jiangxt2 commented on PR #58066:
URL: https://github.com/apache/spark/pull/58066#issuecomment-5357026839
> can you resolve merge conflicts?
Sure, I’ll take care of the merge conflicts and push an updated branch shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58831][SQL] Add bitmap scalar set operation functions [spark]
Posted by "jiangxt2 (via GitHub)" <gi...@apache.org>.
jiangxt2 commented on PR #58066:
URL: https://github.com/apache/spark/pull/58066#issuecomment-5363768800
Hi @cloud-fan I resolved the merge conflicts and pushed the updated branch. The PR is now mergeable again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58831][SQL] Add bitmap scalar set operation functions [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #58066:
URL: https://github.com/apache/spark/pull/58066#issuecomment-5364493532
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/cc75de63a45694f8e70c60cd3387ddf8a83cf80b
- merged into branch-4.x https://github.com/apache/spark/commit/d20be69d9c07d0195288f73361db63c6d189701b
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58831][SQL] Add bitmap scalar set operation functions [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #58066:
URL: https://github.com/apache/spark/pull/58066#issuecomment-5348611631
LGTM, thank you @jiangxt2! Adding @cloud-fan / @MaxGekk for additional review on Catalyst expressions / SQL area, and @HyukjinKwon / @zhengruifeng / @Yicong-Huang for PySpark function bindings and Connect.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58831][SQL] Add bitmap scalar set operation functions [spark]
Posted by "jiangxt2 (via GitHub)" <gi...@apache.org>.
jiangxt2 commented on PR #58066:
URL: https://github.com/apache/spark/pull/58066#issuecomment-5339562501
Additional ecosystem references:
For reference, the same four bitmap set operations are also available in several analytical SQL engines:
- [Flink 2.3](https://nightlies.apache.org/flink/flink-docs-release-2.3/docs/sql/functions/built-in-functions/#bitmap-functions) exposes `BITMAP_AND`, `BITMAP_OR`, `BITMAP_ANDNOT`, and `BITMAP_XOR` over its native `BITMAP` type.
- [ClickHouse](https://clickhouse.com/docs/reference/functions/regular-functions/bitmap-functions) has provided `bitmapAnd`, `bitmapOr`, `bitmapAndnot`, and `bitmapXor` since v20.1.
- Apache Doris provides `BITMAP_AND`, `BITMAP_OR`, `BITMAP_AND_NOT`, and `BITMAP_XOR`, and documents bitmap use cases such as [exact distinct counting and audience segmentation](https://doris.apache.org/docs/4.x/query-acceleration/distinct-counts/bitmap-precise-deduplication/).
- [StarRocks](https://docs.starrocks.io/docs/category/bitmap/) provides the corresponding `bitmap_and`, `bitmap_or`, `bitmap_andnot`, and `bitmap_xor` functions and documents bitmap use cases such as exact distinct counting and retention analysis.
These engines use different physical bitmap representations, so this comparison does not imply binary-format compatibility. Spark continues to use its existing 4096-byte `BINARY` flat bitmap, and both operands must use the same bit-position mapping and, when bucketed, represent the same bucket.
These ecosystem references support the completeness and naming of the proposed API. The four scalar functions allow users to compute row-wise intersection, union, difference, and symmetric difference between precomputed bitmaps without implementing equivalent UDFs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org