You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by dylanhz <dy...@163.com> on 2025/11/21 03:07:22 UTC

[DISCUSS] FLIP-556 Introduce BITMAP Data Type

Hi everyone,


I would like to start a discussion about FLIP-556 Introduce BITMAP Data Type[1]. 


Flink currently has no native, compressed data type for large integer sets, forcing users to rely on external libraries like RoaringBitmap via UDFs. 
This limits performance, maintainability, and integration with Flink’s type system and SQL engine. 
We propose adding a built‑in BITMAP type based on RoaringBitmap to provide compact storage, exact deduplication, and efficient set operations (AND, OR, XOR) directly within Flink.


I have had some initial discussions with @Lincoln Lee and @Jark Wu regarding this FLIP.
Looking forward to your feedback and suggestions.


[1] https://docs.google.com/document/d/1YNgIt93iFboogHMoKbDD4LjP5UrfqtF65hitGKRtKMs/edit?usp=sharing



--

Best regards,
dylanhz

Re: [DISCUSS] FLIP-556 Introduce BITMAP Data Type

Posted by dylanhz <dy...@163.com>.
Hello everyone,

This is a final call for feedback on this FLIP. 
The discussion has been open for about three weeks.

If there is no further feedback, I will start the vote on December 12th.


Best regards,
dylanhz




> 2025年12月9日 22:42,Lincoln Lee <li...@gmail.com> 写道:
> 
> Thanks for the updates!
> 
> Agree with *1 & 2*  merging the two functions and using BIGINT return type.
> 
> *3 & 4* will be useful optimizations.
> 
> *5*’s rename is also reasonable and keeps the naming clean.
> 
> If there’s no further feedback, I suggest we can start the vote in a few
> days.
> 
> 
> Best,
> Lincoln Lee
> 
> 
> dylanhz <dy...@163.com> 于2025年12月9日周二 11:41写道:
> 
>> Hi everyone,
>> 
>> 
>> I've made some updates to this FLIP, including:
>> 1. Removed BITMAP_LONG_CARDINALITY.
>> 2. Changed the result type of BITMAP_CARDINALITY from INT to BIGINT.
>> 3. Added 4 cardinality agg functions to optimize performance by avoiding
>> unnecessary bitmap clones.
>> 4. Added a work item, "Add a rewrite rule to eliminate unnecessary bitmap
>> clones for cardinality calculation".
>> 5. Renamed the internal implementation class from RoaringBitmap32Data to
>> RoaringBitmapData for consistency with the BITMAP type name.
>> 
>> 
>> Please review the updated FLIP and share your feedback. Looking forward to
>> your thoughts!
>> 
>> 
>> 
>> --
>> 
>> Best regards,
>> dylanhz
>> 
>> 
>> 
>> At 2025-11-21 11:07:22, "dylanhz" <dy...@163.com> wrote:
>> 
>> Hi everyone,
>> 
>> 
>> I would like to start a discussion about FLIP-556 Introduce BITMAP Data
>> Type[1].
>> 
>> 
>> Flink currently has no native, compressed data type for large integer
>> sets, forcing users to rely on external libraries like RoaringBitmap via
>> UDFs.
>> This limits performance, maintainability, and integration with Flink’s
>> type system and SQL engine.
>> We propose adding a built‑in BITMAP type based on RoaringBitmap to provide
>> compact storage, exact deduplication, and efficient set operations (AND,
>> OR, XOR) directly within Flink.
>> 
>> 
>> I have had some initial discussions with @Lincoln Lee and @Jark Wu
>> regarding this FLIP.
>> Looking forward to your feedback and suggestions.
>> 
>> 
>> [1]
>> https://docs.google.com/document/d/1YNgIt93iFboogHMoKbDD4LjP5UrfqtF65hitGKRtKMs/edit?usp=sharing
>> 
>> 
>> 
>> --
>> 
>> Best regards,
>> dylanhz


Re: [DISCUSS] FLIP-556 Introduce BITMAP Data Type

Posted by Lincoln Lee <li...@gmail.com>.
Thanks for the updates!

Agree with *1 & 2*  merging the two functions and using BIGINT return type.

*3 & 4* will be useful optimizations.

*5*’s rename is also reasonable and keeps the naming clean.

If there’s no further feedback, I suggest we can start the vote in a few
days.


Best,
Lincoln Lee


dylanhz <dy...@163.com> 于2025年12月9日周二 11:41写道:

> Hi everyone,
>
>
> I've made some updates to this FLIP, including:
> 1. Removed BITMAP_LONG_CARDINALITY.
> 2. Changed the result type of BITMAP_CARDINALITY from INT to BIGINT.
> 3. Added 4 cardinality agg functions to optimize performance by avoiding
> unnecessary bitmap clones.
> 4. Added a work item, "Add a rewrite rule to eliminate unnecessary bitmap
> clones for cardinality calculation".
> 5. Renamed the internal implementation class from RoaringBitmap32Data to
> RoaringBitmapData for consistency with the BITMAP type name.
>
>
> Please review the updated FLIP and share your feedback. Looking forward to
> your thoughts!
>
>
>
> --
>
> Best regards,
> dylanhz
>
>
>
> At 2025-11-21 11:07:22, "dylanhz" <dy...@163.com> wrote:
>
> Hi everyone,
>
>
> I would like to start a discussion about FLIP-556 Introduce BITMAP Data
> Type[1].
>
>
> Flink currently has no native, compressed data type for large integer
> sets, forcing users to rely on external libraries like RoaringBitmap via
> UDFs.
> This limits performance, maintainability, and integration with Flink’s
> type system and SQL engine.
> We propose adding a built‑in BITMAP type based on RoaringBitmap to provide
> compact storage, exact deduplication, and efficient set operations (AND,
> OR, XOR) directly within Flink.
>
>
> I have had some initial discussions with @Lincoln Lee and @Jark Wu
> regarding this FLIP.
> Looking forward to your feedback and suggestions.
>
>
> [1]
> https://docs.google.com/document/d/1YNgIt93iFboogHMoKbDD4LjP5UrfqtF65hitGKRtKMs/edit?usp=sharing
>
>
>
> --
>
> Best regards,
> dylanhz

Re:[DISCUSS] FLIP-556 Introduce BITMAP Data Type

Posted by dylanhz <dy...@163.com>.
Hi everyone,


I've made some updates to this FLIP, including:
1. Removed BITMAP_LONG_CARDINALITY.
2. Changed the result type of BITMAP_CARDINALITY from INT to BIGINT.
3. Added 4 cardinality agg functions to optimize performance by avoiding unnecessary bitmap clones.
4. Added a work item, "Add a rewrite rule to eliminate unnecessary bitmap clones for cardinality calculation".
5. Renamed the internal implementation class from RoaringBitmap32Data to RoaringBitmapData for consistency with the BITMAP type name.


Please review the updated FLIP and share your feedback. Looking forward to your thoughts!



--

Best regards,
dylanhz



At 2025-11-21 11:07:22, "dylanhz" <dy...@163.com> wrote:

Hi everyone,


I would like to start a discussion about FLIP-556 Introduce BITMAP Data Type[1]. 


Flink currently has no native, compressed data type for large integer sets, forcing users to rely on external libraries like RoaringBitmap via UDFs. 
This limits performance, maintainability, and integration with Flink’s type system and SQL engine. 
We propose adding a built‑in BITMAP type based on RoaringBitmap to provide compact storage, exact deduplication, and efficient set operations (AND, OR, XOR) directly within Flink.


I have had some initial discussions with @Lincoln Lee and @Jark Wu regarding this FLIP.
Looking forward to your feedback and suggestions.


[1] https://docs.google.com/document/d/1YNgIt93iFboogHMoKbDD4LjP5UrfqtF65hitGKRtKMs/edit?usp=sharing



--

Best regards,
dylanhz

Re: [DISCUSS] FLIP-556 Introduce BITMAP Data Type

Posted by dylanhz <dy...@163.com>.
Hello everyone,

This is the final call for feedback on this FLIP. 
If there is no further feedback, I will start the vote tomorrow.


Best regards,
dylanhz




> 2025年12月11日 10:43,dylanhz <dy...@163.com> 写道:
> 
> Hi David, 
> 
> Thanks for your review! Here are the clarifications:
> 
> 1. You are right about the value range, the parenthesis excludes the upper boundary.
> 
> 2. Yes, we support sparsely populated bitmaps. We use RoaringBitmap[1] for internal implementation, which efficiently handles both sparse and dense data via adaptive compression (Array/Bitmap/Run containers).
> 
> 3.1 Operations like BITMAP_AND are set algebra operations (intersection/union) on the integers, not bit-level operations. Here is an example:
> ```
> SELECT BITMAP_OR(BITMAP_BUILD(ARRAY[1,2,3]), BITMAP_BUILD(ARRAY[3,6,9]))
>> {1,2,3,6,9}
> ``` 
> 
> 3.2 Bitmap level null semantics are stated in section “BITMAP Semantics”. Any integer value is either in (bit 1) or not in (bit 0) the bitmap, there is no “null” state.
> 
> 3.3 Bitmap stores integers only, so you have to do mapping before storing non-integer values.
> As for the timestamps example, I’m not sure why and how you store them in a bitmap, could you explain a little more?
> 
> [1] https://github.com/RoaringBitmap/RoaringFormatSpec
> 
> 
> Best regards,
> dylanhz
> 
> 
> 
> 
>> 2025年12月10日 20:21,David Radley <da...@uk.ibm.com> 写道:
>> 
>> Hi,
>> Looks good. A couple of thoughts :
>> 
>> *
>> It says “Only integers in the logical range [0, 2^32) are supported”. I assume the top value should be 2^32-1.
>> *
>> I assume we can have sparsely populated bitmaps.
>> *
>> When we do an AND or OR, I assume this is at the bit level, not at the integer level. Would a null value at position 6 be the same as 32 0’s for the purposes of these operations? It would be useful to show some examples around this for example around timestamps
>> 
>> Kind regards, David.
>> 
>> 
>> From: dylanhz <dy...@163.com>
>> Date: Wednesday, 26 November 2025 at 12:09
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-556 Introduce BITMAP Data Type
>> 
>> You are right that there is a binding, but I want to clarify: Bitmap is bound to Roaring as the external serialization format, not the internal implementation. The internal implementation can be changed independently without affecting users.
>> 
>> If users need other serialization formats in the future, we can add a format parameter to Bitmap#toBytes and Bitmap#fromBytes methods, as well as their corresponding built-in functions. Users can then work with the BYTES type directly to serialize/deserialize in their preferred format when exchanging data with external systems.
>> 
>> 
>> Best regards,
>> dylanhz
>> 
>> 
>>> 2025年11月26日 19:16,Xuyang <xy...@163.com> 写道:
>>> 
>>> +1 for this feature.
>>> Looks good to see the support for bitmaps to enable Flink handling computations in extremely high-dimensional scenarios. After reviewing the entire FLIP, I have one question:
>>> Regarding the Bitmap#toBytes interface, I noticed that it will output bytes in RoaringBitmap format default. Does this imply a strong binding to the internal implementation of RoaringBitmap? For the writers on the sink table, they need to be aware that the bytes are in RoaringBitmap format, right?
>>> 
>>> 
>>> 
>>> --
>>> 
>>>  Best!
>>>  Xuyang
>>> 
>>> 
>>> 
>>> 在 2025-11-24 18:09:25,"Lincoln Lee" <li...@gmail.com> 写道:
>>>> +1 for this feature! Expanding the bitmap type will help users unlock more
>>>> computation scenarios and integrate more easily with external systems.
>>>> 
>>>> 
>>>> Best,
>>>> Lincoln Lee
>>>> 
>>>> 
>>>> dylanhz <dy...@163.com> 于2025年11月21日周五 11:07写道:
>>>> 
>>>>> Hi everyone,
>>>>> 
>>>>> 
>>>>> I would like to start a discussion about FLIP-556 Introduce BITMAP Data
>>>>> Type[1].
>>>>> 
>>>>> 
>>>>> Flink currently has no native, compressed data type for large integer
>>>>> sets, forcing users to rely on external libraries like RoaringBitmap via
>>>>> UDFs.
>>>>> This limits performance, maintainability, and integration with Flink’s
>>>>> type system and SQL engine.
>>>>> We propose adding a built‑in BITMAP type based on RoaringBitmap to provide
>>>>> compact storage, exact deduplication, and efficient set operations (AND,
>>>>> OR, XOR) directly within Flink.
>>>>> 
>>>>> 
>>>>> I have had some initial discussions with @Lincoln Lee and @Jark Wu
>>>>> regarding this FLIP.
>>>>> Looking forward to your feedback and suggestions.
>>>>> 
>>>>> 
>>>>> [1]
>>>>> https://docs.google.com/document/d/1YNgIt93iFboogHMoKbDD4LjP5UrfqtF65hitGKRtKMs/edit?usp=sharing
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> 
>>>>> Best regards,
>>>>> dylanhz
>> 
>> 
>> Unless otherwise stated above:
>> 
>> IBM United Kingdom Limited
>> Registered in England and Wales with number 741598
>> Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN
> 


Re: [DISCUSS] FLIP-556 Introduce BITMAP Data Type

Posted by dylanhz <dy...@163.com>.
Hi David, 

Thanks for your review! Here are the clarifications:

1. You are right about the value range, the parenthesis excludes the upper boundary.

2. Yes, we support sparsely populated bitmaps. We use RoaringBitmap[1] for internal implementation, which efficiently handles both sparse and dense data via adaptive compression (Array/Bitmap/Run containers).

3.1 Operations like BITMAP_AND are set algebra operations (intersection/union) on the integers, not bit-level operations. Here is an example:
```
SELECT BITMAP_OR(BITMAP_BUILD(ARRAY[1,2,3]), BITMAP_BUILD(ARRAY[3,6,9]))
> {1,2,3,6,9}
``` 

3.2 Bitmap level null semantics are stated in section “BITMAP Semantics”. Any integer value is either in (bit 1) or not in (bit 0) the bitmap, there is no “null” state.

3.3 Bitmap stores integers only, so you have to do mapping before storing non-integer values.
As for the timestamps example, I’m not sure why and how you store them in a bitmap, could you explain a little more?

[1] https://github.com/RoaringBitmap/RoaringFormatSpec


Best regards,
dylanhz




> 2025年12月10日 20:21,David Radley <da...@uk.ibm.com> 写道:
> 
> Hi,
> Looks good. A couple of thoughts :
> 
>  *
> It says “Only integers in the logical range [0, 2^32) are supported”. I assume the top value should be 2^32-1.
>  *
> I assume we can have sparsely populated bitmaps.
>  *
> When we do an AND or OR, I assume this is at the bit level, not at the integer level. Would a null value at position 6 be the same as 32 0’s for the purposes of these operations? It would be useful to show some examples around this for example around timestamps
> 
> Kind regards, David.
> 
> 
> From: dylanhz <dy...@163.com>
> Date: Wednesday, 26 November 2025 at 12:09
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-556 Introduce BITMAP Data Type
> 
> You are right that there is a binding, but I want to clarify: Bitmap is bound to Roaring as the external serialization format, not the internal implementation. The internal implementation can be changed independently without affecting users.
> 
> If users need other serialization formats in the future, we can add a format parameter to Bitmap#toBytes and Bitmap#fromBytes methods, as well as their corresponding built-in functions. Users can then work with the BYTES type directly to serialize/deserialize in their preferred format when exchanging data with external systems.
> 
> 
> Best regards,
> dylanhz
> 
> 
>> 2025年11月26日 19:16,Xuyang <xy...@163.com> 写道:
>> 
>> +1 for this feature.
>> Looks good to see the support for bitmaps to enable Flink handling computations in extremely high-dimensional scenarios. After reviewing the entire FLIP, I have one question:
>> Regarding the Bitmap#toBytes interface, I noticed that it will output bytes in RoaringBitmap format default. Does this imply a strong binding to the internal implementation of RoaringBitmap? For the writers on the sink table, they need to be aware that the bytes are in RoaringBitmap format, right?
>> 
>> 
>> 
>> --
>> 
>>   Best!
>>   Xuyang
>> 
>> 
>> 
>> 在 2025-11-24 18:09:25,"Lincoln Lee" <li...@gmail.com> 写道:
>>> +1 for this feature! Expanding the bitmap type will help users unlock more
>>> computation scenarios and integrate more easily with external systems.
>>> 
>>> 
>>> Best,
>>> Lincoln Lee
>>> 
>>> 
>>> dylanhz <dy...@163.com> 于2025年11月21日周五 11:07写道:
>>> 
>>>> Hi everyone,
>>>> 
>>>> 
>>>> I would like to start a discussion about FLIP-556 Introduce BITMAP Data
>>>> Type[1].
>>>> 
>>>> 
>>>> Flink currently has no native, compressed data type for large integer
>>>> sets, forcing users to rely on external libraries like RoaringBitmap via
>>>> UDFs.
>>>> This limits performance, maintainability, and integration with Flink’s
>>>> type system and SQL engine.
>>>> We propose adding a built‑in BITMAP type based on RoaringBitmap to provide
>>>> compact storage, exact deduplication, and efficient set operations (AND,
>>>> OR, XOR) directly within Flink.
>>>> 
>>>> 
>>>> I have had some initial discussions with @Lincoln Lee and @Jark Wu
>>>> regarding this FLIP.
>>>> Looking forward to your feedback and suggestions.
>>>> 
>>>> 
>>>> [1]
>>>> https://docs.google.com/document/d/1YNgIt93iFboogHMoKbDD4LjP5UrfqtF65hitGKRtKMs/edit?usp=sharing
>>>> 
>>>> 
>>>> 
>>>> --
>>>> 
>>>> Best regards,
>>>> dylanhz
> 
> 
> Unless otherwise stated above:
> 
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN


RE: [DISCUSS] FLIP-556 Introduce BITMAP Data Type

Posted by David Radley <da...@uk.ibm.com>.
Hi,
Looks good. A couple of thoughts :

  *
It says “Only integers in the logical range [0, 2^32) are supported”. I assume the top value should be 2^32-1.
  *
I assume we can have sparsely populated bitmaps.
  *
When we do an AND or OR, I assume this is at the bit level, not at the integer level. Would a null value at position 6 be the same as 32 0’s for the purposes of these operations? It would be useful to show some examples around this for example around timestamps

Kind regards, David.


From: dylanhz <dy...@163.com>
Date: Wednesday, 26 November 2025 at 12:09
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-556 Introduce BITMAP Data Type

You are right that there is a binding, but I want to clarify: Bitmap is bound to Roaring as the external serialization format, not the internal implementation. The internal implementation can be changed independently without affecting users.

If users need other serialization formats in the future, we can add a format parameter to Bitmap#toBytes and Bitmap#fromBytes methods, as well as their corresponding built-in functions. Users can then work with the BYTES type directly to serialize/deserialize in their preferred format when exchanging data with external systems.


Best regards,
dylanhz


> 2025年11月26日 19:16,Xuyang <xy...@163.com> 写道:
>
> +1 for this feature.
> Looks good to see the support for bitmaps to enable Flink handling computations in extremely high-dimensional scenarios. After reviewing the entire FLIP, I have one question:
> Regarding the Bitmap#toBytes interface, I noticed that it will output bytes in RoaringBitmap format default. Does this imply a strong binding to the internal implementation of RoaringBitmap? For the writers on the sink table, they need to be aware that the bytes are in RoaringBitmap format, right?
>
>
>
> --
>
>    Best!
>    Xuyang
>
>
>
> 在 2025-11-24 18:09:25,"Lincoln Lee" <li...@gmail.com> 写道:
>> +1 for this feature! Expanding the bitmap type will help users unlock more
>> computation scenarios and integrate more easily with external systems.
>>
>>
>> Best,
>> Lincoln Lee
>>
>>
>> dylanhz <dy...@163.com> 于2025年11月21日周五 11:07写道:
>>
>>> Hi everyone,
>>>
>>>
>>> I would like to start a discussion about FLIP-556 Introduce BITMAP Data
>>> Type[1].
>>>
>>>
>>> Flink currently has no native, compressed data type for large integer
>>> sets, forcing users to rely on external libraries like RoaringBitmap via
>>> UDFs.
>>> This limits performance, maintainability, and integration with Flink’s
>>> type system and SQL engine.
>>> We propose adding a built‑in BITMAP type based on RoaringBitmap to provide
>>> compact storage, exact deduplication, and efficient set operations (AND,
>>> OR, XOR) directly within Flink.
>>>
>>>
>>> I have had some initial discussions with @Lincoln Lee and @Jark Wu
>>> regarding this FLIP.
>>> Looking forward to your feedback and suggestions.
>>>
>>>
>>> [1]
>>> https://docs.google.com/document/d/1YNgIt93iFboogHMoKbDD4LjP5UrfqtF65hitGKRtKMs/edit?usp=sharing
>>>
>>>
>>>
>>> --
>>>
>>> Best regards,
>>> dylanhz


Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN

Re: [DISCUSS] FLIP-556 Introduce BITMAP Data Type

Posted by dylanhz <dy...@163.com>.
You are right that there is a binding, but I want to clarify: Bitmap is bound to Roaring as the external serialization format, not the internal implementation. The internal implementation can be changed independently without affecting users.

If users need other serialization formats in the future, we can add a format parameter to Bitmap#toBytes and Bitmap#fromBytes methods, as well as their corresponding built-in functions. Users can then work with the BYTES type directly to serialize/deserialize in their preferred format when exchanging data with external systems.


Best regards,
dylanhz


> 2025年11月26日 19:16,Xuyang <xy...@163.com> 写道:
> 
> +1 for this feature. 
> Looks good to see the support for bitmaps to enable Flink handling computations in extremely high-dimensional scenarios. After reviewing the entire FLIP, I have one question:
> Regarding the Bitmap#toBytes interface, I noticed that it will output bytes in RoaringBitmap format default. Does this imply a strong binding to the internal implementation of RoaringBitmap? For the writers on the sink table, they need to be aware that the bytes are in RoaringBitmap format, right?
> 
> 
> 
> --
> 
>    Best!
>    Xuyang
> 
> 
> 
> 在 2025-11-24 18:09:25,"Lincoln Lee" <li...@gmail.com> 写道:
>> +1 for this feature! Expanding the bitmap type will help users unlock more
>> computation scenarios and integrate more easily with external systems.
>> 
>> 
>> Best,
>> Lincoln Lee
>> 
>> 
>> dylanhz <dy...@163.com> 于2025年11月21日周五 11:07写道:
>> 
>>> Hi everyone,
>>> 
>>> 
>>> I would like to start a discussion about FLIP-556 Introduce BITMAP Data
>>> Type[1].
>>> 
>>> 
>>> Flink currently has no native, compressed data type for large integer
>>> sets, forcing users to rely on external libraries like RoaringBitmap via
>>> UDFs.
>>> This limits performance, maintainability, and integration with Flink’s
>>> type system and SQL engine.
>>> We propose adding a built‑in BITMAP type based on RoaringBitmap to provide
>>> compact storage, exact deduplication, and efficient set operations (AND,
>>> OR, XOR) directly within Flink.
>>> 
>>> 
>>> I have had some initial discussions with @Lincoln Lee and @Jark Wu
>>> regarding this FLIP.
>>> Looking forward to your feedback and suggestions.
>>> 
>>> 
>>> [1]
>>> https://docs.google.com/document/d/1YNgIt93iFboogHMoKbDD4LjP5UrfqtF65hitGKRtKMs/edit?usp=sharing
>>> 
>>> 
>>> 
>>> --
>>> 
>>> Best regards,
>>> dylanhz


Re:Re: [DISCUSS] FLIP-556 Introduce BITMAP Data Type

Posted by Xuyang <xy...@163.com>.
+1 for this feature. 
Looks good to see the support for bitmaps to enable Flink handling computations in extremely high-dimensional scenarios. After reviewing the entire FLIP, I have one question:
Regarding the Bitmap#toBytes interface, I noticed that it will output bytes in RoaringBitmap format default. Does this imply a strong binding to the internal implementation of RoaringBitmap? For the writers on the sink table, they need to be aware that the bytes are in RoaringBitmap format, right?



--

    Best!
    Xuyang



在 2025-11-24 18:09:25,"Lincoln Lee" <li...@gmail.com> 写道:
>+1 for this feature! Expanding the bitmap type will help users unlock more
>computation scenarios and integrate more easily with external systems.
>
>
>Best,
>Lincoln Lee
>
>
>dylanhz <dy...@163.com> 于2025年11月21日周五 11:07写道:
>
>> Hi everyone,
>>
>>
>> I would like to start a discussion about FLIP-556 Introduce BITMAP Data
>> Type[1].
>>
>>
>> Flink currently has no native, compressed data type for large integer
>> sets, forcing users to rely on external libraries like RoaringBitmap via
>> UDFs.
>> This limits performance, maintainability, and integration with Flink’s
>> type system and SQL engine.
>> We propose adding a built‑in BITMAP type based on RoaringBitmap to provide
>> compact storage, exact deduplication, and efficient set operations (AND,
>> OR, XOR) directly within Flink.
>>
>>
>> I have had some initial discussions with @Lincoln Lee and @Jark Wu
>> regarding this FLIP.
>> Looking forward to your feedback and suggestions.
>>
>>
>> [1]
>> https://docs.google.com/document/d/1YNgIt93iFboogHMoKbDD4LjP5UrfqtF65hitGKRtKMs/edit?usp=sharing
>>
>>
>>
>> --
>>
>> Best regards,
>> dylanhz

Re: [DISCUSS] FLIP-556 Introduce BITMAP Data Type

Posted by Lincoln Lee <li...@gmail.com>.
+1 for this feature! Expanding the bitmap type will help users unlock more
computation scenarios and integrate more easily with external systems.


Best,
Lincoln Lee


dylanhz <dy...@163.com> 于2025年11月21日周五 11:07写道:

> Hi everyone,
>
>
> I would like to start a discussion about FLIP-556 Introduce BITMAP Data
> Type[1].
>
>
> Flink currently has no native, compressed data type for large integer
> sets, forcing users to rely on external libraries like RoaringBitmap via
> UDFs.
> This limits performance, maintainability, and integration with Flink’s
> type system and SQL engine.
> We propose adding a built‑in BITMAP type based on RoaringBitmap to provide
> compact storage, exact deduplication, and efficient set operations (AND,
> OR, XOR) directly within Flink.
>
>
> I have had some initial discussions with @Lincoln Lee and @Jark Wu
> regarding this FLIP.
> Looking forward to your feedback and suggestions.
>
>
> [1]
> https://docs.google.com/document/d/1YNgIt93iFboogHMoKbDD4LjP5UrfqtF65hitGKRtKMs/edit?usp=sharing
>
>
>
> --
>
> Best regards,
> dylanhz