[jira] [Updated] (SPARK-58186) Add bitmap_contains function

["jiangxintong (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

jiangxintong updated SPARK-58186:
---------------------------------
    Description: 
h3. Problem

Spark's flat-bitmap infrastructure provides functions for building and merging bitmaps
({{bitmap_construct_agg}}, {{bitmap_or_agg}}, {{bitmap_and_agg}}, {{bitmap_count}},
{{bitmap_bucket_number}}, and {{bitmap_bit_position}}), but lacks a predicate to test whether a
specific bit is set. Without this, precomputed bitmaps cannot be used directly as membership
predicates in SQL queries.

Related bitmap membership operations exist in other analytical databases, but their argument
contracts are not necessarily identical to Spark's flat-bitmap mapping. This proposal is defined
around Spark's existing bucket-number and bit-position functions.

h3. Proposed Function

|| Function || Signature || Description ||
| bitmap_contains | bitmap_contains(BINARY, numeric) -> BOOLEAN | Returns true if the given bit position is set in one bitmap bucket, and false otherwise. The numeric position is explicitly cast to BIGINT before the lookup. |

h3. Examples

{code:sql}
SELECT bitmap_contains(X '01', 0L);
-- true

SELECT bitmap_contains(X '01', 1L);
-- false

SELECT bitmap_contains(X '10', 4L);
-- true

-- Practical usage with bitmap_construct_agg
SELECT e.*
FROM events e
JOIN segment_bitmaps b
  ON bitmap_bucket_number(e.user_id) = b.bucket
WHERE bitmap_contains(b.bm, bitmap_bit_position(e.user_id));
{code}

h3. Semantics

* The first argument must be a BINARY bitmap. The second argument accepts Spark numeric types and
  is explicitly cast to BIGINT using standard Spark cast semantics.
* The function checks a bit position within one bucket. It does not accept a raw business value;
  callers should use {{bitmap_bucket_number}} and {{bitmap_bit_position}} when mapping values to
  Spark's flat-bitmap representation.
* For a non-NULL bitmap, the function returns false for a negative position, a position greater
  than or equal to 32768, or a position beyond the actual byte-array length. A standard flat
  bitmap bucket contains 4096 bytes and 32768 bits; shorter binary arrays are read safely.
* NULL bitmap or position inputs, including untyped NULL literals accepted by the function's input
  contract, produce SQL NULL.
* Finite fractional positions are truncated toward zero by the BIGINT cast. Invalid or
  out-of-range numeric values follow Spark cast semantics; ANSI mode controls whether a failing
  cast raises a cast error. Non-numeric, non-NULL position types are rejected during analysis.

h3. Notes

* This function operates on Spark's existing flat-bitmap representation. It does not introduce a
  new bitmap storage format, replace bucket calculation, or change the physical join plan.
* The function completes the flat-bitmap expression family alongside the existing construct,
  merge, count, bucket-number, and bit-position functions.


  was:
h3. Problem

Spark's flat-bitmap infrastructure provides functions for building and merging bitmaps (bitmap_construct_agg, bitmap_or_agg, bitmap_and_agg, bitmap_count, bitmap_bucket_number, bitmap_bit_position), but lacks a predicate to test whether a specific bit is set. Without this, precomputed bitmaps cannot be used for membership filtering — the primary use case of bitmap indexes in analytical databases.

Other databases with this capability:
- ClickHouse: bitmapContains(bitmap, value)
- Doris: bitmap_contains(bitmap, value)
- StarRocks: bitmap_contains(bitmap, value)

h3. Proposed Function

|| Function || Signature || Description ||
| bitmap_contains | bitmap_contains(BINARY, LONG) -> BOOLEAN | Returns true if the bit at the given position is set in the bitmap. Returns false for out-of-range positions. |

h3. Examples

{code:sql}
> SELECT bitmap_contains(X '01', 0L);
 true

> SELECT bitmap_contains(X '01', 1L);
 false

> SELECT bitmap_contains(X '10', 4L);
 true

-- Practical usage with bitmap_construct_agg
SELECT e.*
FROM events e
JOIN segment_bitmaps b ON bitmap_bucket_number(e.user_id) = b.bucket
WHERE bitmap_contains(b.bm, bitmap_bit_position(e.user_id));
{code}

h3. Semantics

- Input: (BINARY bitmap, LONG bit_position)
- Output: BOOLEAN
- Returns false for out-of-range positions (negative or >= 32768), safe by default
- NULL bitmap or NULL position → NULL result (standard SQL NULL propagation)
- Works with bitmaps of any length (not limited to the standard 4096-byte bitmap)

h3. Notes

- This function operates on Spark's flat-bitmap format (fixed 4KB per bucket), not RoaringBitmap. The 2-step mapping (bucket_number + bit_position) is still required.
- Completes the flat-bitmap expression family alongside the existing construct/merge/count functions.


> Add bitmap_contains function
> ----------------------------
>
>                 Key: SPARK-58186
>                 URL: https://issues.apache.org/jira/browse/SPARK-58186
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: jiangxintong
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Problem
> Spark's flat-bitmap infrastructure provides functions for building and merging bitmaps
> ({{bitmap_construct_agg}}, {{bitmap_or_agg}}, {{bitmap_and_agg}}, {{bitmap_count}},
> {{bitmap_bucket_number}}, and {{bitmap_bit_position}}), but lacks a predicate to test whether a
> specific bit is set. Without this, precomputed bitmaps cannot be used directly as membership
> predicates in SQL queries.
> Related bitmap membership operations exist in other analytical databases, but their argument
> contracts are not necessarily identical to Spark's flat-bitmap mapping. This proposal is defined
> around Spark's existing bucket-number and bit-position functions.
> h3. Proposed Function
> || Function || Signature || Description ||
> | bitmap_contains | bitmap_contains(BINARY, numeric) -> BOOLEAN | Returns true if the given bit position is set in one bitmap bucket, and false otherwise. The numeric position is explicitly cast to BIGINT before the lookup. |
> h3. Examples
> {code:sql}
> SELECT bitmap_contains(X '01', 0L);
> -- true
> SELECT bitmap_contains(X '01', 1L);
> -- false
> SELECT bitmap_contains(X '10', 4L);
> -- true
> -- Practical usage with bitmap_construct_agg
> SELECT e.*
> FROM events e
> JOIN segment_bitmaps b
>   ON bitmap_bucket_number(e.user_id) = b.bucket
> WHERE bitmap_contains(b.bm, bitmap_bit_position(e.user_id));
> {code}
> h3. Semantics
> * The first argument must be a BINARY bitmap. The second argument accepts Spark numeric types and
>   is explicitly cast to BIGINT using standard Spark cast semantics.
> * The function checks a bit position within one bucket. It does not accept a raw business value;
>   callers should use {{bitmap_bucket_number}} and {{bitmap_bit_position}} when mapping values to
>   Spark's flat-bitmap representation.
> * For a non-NULL bitmap, the function returns false for a negative position, a position greater
>   than or equal to 32768, or a position beyond the actual byte-array length. A standard flat
>   bitmap bucket contains 4096 bytes and 32768 bits; shorter binary arrays are read safely.
> * NULL bitmap or position inputs, including untyped NULL literals accepted by the function's input
>   contract, produce SQL NULL.
> * Finite fractional positions are truncated toward zero by the BIGINT cast. Invalid or
>   out-of-range numeric values follow Spark cast semantics; ANSI mode controls whether a failing
>   cast raises a cast error. Non-numeric, non-NULL position types are rejected during analysis.
> h3. Notes
> * This function operates on Spark's existing flat-bitmap representation. It does not introduce a
>   new bitmap storage format, replace bucket calculation, or change the physical join plan.
> * The function completes the flat-bitmap expression family alongside the existing construct,
>   merge, count, bucket-number, and bit-position functions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org