[DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Prajwal Banakar <pr...@gmail.com>]

Hi everyone,

I would like to start a discussion on the proposal for Native Bitmap
Integration & Stateless Pushdown Aggregation.

This proposal enables end-to-end native support for the BITMAP type in
Fluss and integrates it with the existing aggregation merge engine to
support server-side bitmap union pushdown. The goal is to reduce network
transfer and offload DISTINCT-style aggregation from Flink to the
TabletServer.

Key highlights of the proposal include:

- Type System: Promoting BITMAP to a first-class logical type.
- UDF Suite: Introducing BITMAP_BUILD, BITMAP_OR_AGG, and
BITMAP_CARDINALITY (aligned with FLIP-556 and StarRocks semantics).
- Optimizer: Planner-based pushdown via applyAggregates in the Flink
connector.
- Safety: No changes to LogRecordBatch or WAL, making this strictly
additive and migration-free.

You can find the full proposal document here:
https://docs.google.com/document/d/1sDhfkmo-w-UTvo2n3rsY1lytSSryswfkI83cSdka8s0/edit?usp=sharing

I would appreciate feedback on the public interfaces, pushdown constraints,
and overall scope.

Best regards,
Prajwal Banakar

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Jark Wu <im...@gmail.com>]

Hi Prajwal,

Thanks for the update. The new FIP looks good to me. +1 to start the vote.

It would be great if Yang Wang could take a final look, given his
contributions to the RoaringBitmap aggregate merge engine and the
related blog post.

Best,
Jark

On Tue, 21 Apr 2026 at 13:58, Prajwal Banakar
<pr...@gmail.com> wrote:
>
> Hi Jark,
>
> Thank you for the thorough and precise feedback. I agree with all four
> points you raised and have updated the proposal accordingly.
>
> 1. On deferring the BITMAP type:
> Agreed. Since FLIP-556 is not yet available in Flink, the in-memory
> benefits of a native type cannot be realized as UDFs must operate on byte[]
> regardless of the logical type. I have removed Layer 1 and will keep bitmap
> columns as BYTES. We can revisit the BITMAP type in a dedicated FIP once
> FLIP-556 lands.
>
> 2. On the RB_ prefix:
> Agreed. Using the "RB_" prefix avoids future conflicts with Flink
> built-ins, aligns with existing rbm32/64 naming in Fluss, and ensures
> compatibility with pg_roaringbitmap conventions. All functions have been
> renamed.
>
> 3. On expanding the function set:
> Thank you for the references. I have added rb_build, rb_contains, and
> rb_to_array to Phase 1, and rb_xor_agg, rb_xor, and rb_andnot to Phase 2. I
> have also deferred rb_build_agg_64 due to the lack of standardized 64-bit
> serialization formats, consistent with the scope of FLIP-556.
>
> 3. On deferring pushdown:
> Agreed. The concerns regarding full table scans on the TabletServer and
> potential memory pressure are valid. I have deferred Layer 3 to future work
> and documented these architectural concerns.
>
> FIP-37 is now a focused single-layer proposal implementing 10 RoaringBitmap
> SQL functions registered via FlussCatalog on BYTES columns. The updated
> document here:
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-37%3A+Native+RoaringBitmap+Integration+for+Apache+Fluss
>
> Best regards,
> Prajwal Banakar
>
> On Sun, 19 Apr 2026 at 21:23, Jark Wu <im...@gmail.com> wrote:
>
> > Add the missing links in my previous email
> >
> > [1] https://github.com/flink-extended/flink-roaringbitmap
> > [2] https://github.com/ChenHuajun/pg_roaringbitmap
> >
> > On Sun, 19 Apr 2026 at 23:51, Jark Wu <im...@gmail.com> wrote:
> >
> > > Hi Prajwal,
> > >
> > > Thanks for the proposal! RoaringBitmap SQL functions are a valuable
> > > addition to Fluss. I have a few suggestions:
> > >
> > > *1. Defer BITMAP Type Introduction*
> > >
> > > I suggest excluding the BITMAP type (Layer 1) from this FIP and keeping
> > > bitmap columns as BYTES.
> > >
> > > The primary benefit of a native BITMAP type — as proposed in Flink's
> > > FLIP-556 — is that functions can pass Bitmap Java objects in-memory
> > between
> > > operators without serialization/deserialization at each step. However,
> > > since FLIP-556 is still under discussion and not yet available in Flink,
> > > the Flink UDFs we register can only operate on byte[] (BYTES). On the
> > Fluss
> > > server side, the merge engine aggregators always deserialize from byte[],
> > > perform the OR, and serialize back — regardless of whether the logical
> > type
> > > is BYTES or BITMAP.
> > >
> > > Introducing the BITMAP type offers limited immediate benefits, while
> > > ensuring compatibility is a complex issue that warrants separate
> > > discussion. This includes addressing how type conversions between Fluss
> > and
> > > various engines bridge to BITMAP without breaking existing compatibility,
> > > which is not currently covered in this FIP. Therefore, I propose
> > deferring
> > > the introduction of the BITMAP type to simplify and accelerate the
> > approval
> > > of the current FIP.
> > >
> > > The FIP can focus on what matters most: SQL functions operating on BYTES
> > > columns that is roaringbitmap binary format underlying. We can revisit
> > the
> > > BITMAP type once FLIP-556 has been released in Flink.
> > >
> > > *2. Function Naming: Use "RB_" Prefix Instead of "BITMAP_"*
> > >
> > > I suggest using the "RB_" prefix rather than "BITMAP_". Reasons:
> > >
> > > (1) Intentional differentiation from Flink built-ins. When FLIP-556
> > > eventually lands, Flink will have its own BITMAP_* built-in functions
> > > operating on a native BITMAP type. Our functions operate on BYTES and are
> > > registered via FlussCatalog. Using RB_ avoids ambiguity and accidental
> > > invocation of the wrong function at runtime.
> > > (2) Consistent with Fluss conventions. Fluss already uses rbm32/rbm64 as
> > > aggregation function identifiers in the merge engine. RB_ aligns
> > naturally
> > > with this existing naming.
> > > (3) PostgreSQL protocol compatibility. Fluss has future plans for PG
> > > protocol compatibility. The de facto standard for RoaringBitmap in
> > > PostgreSQL (pg_roaringbitmap) uses rb_ prefix. Adopting this now avoids a
> > > rename later.
> > >
> > > *3. Expand the Function Set*
> > >
> > > Referencing the function sets from flink-roaringbitmap[1] and
> > > pg_roaringbitmap[2], I suggest the following additions:
> > >
> > > Phase 1 (add these — they are very commonly used):
> > >
> > > * rb_build(ARRAY<INT>) → BYTES — construct a bitmap from an array without
> > > aggregation
> > > * rb_contains(BYTES, INT) → BOOLEAN — check if a bitmap contains a value
> > > * rb_to_array(BYTES) → ARRAY<INT> — convert bitmap back to array
> > > (debugging, integration)
> > >
> > > Phase 2 (add these to complete set algebra):
> > >
> > > * rb_xor_agg(BYTES) → BYTES — XOR aggregation
> > > * rb_xor(BYTES, BYTES) → BYTES — XOR of two bitmaps
> > > * rb_andnot(BYTES, BYTES) → BYTES — difference (AND NOT) of two bitmaps
> > >
> > > Also, consider deferring "bitmap_build_agg_64" — the 64-bit RoaringBitmap
> > > serialization format is not standardized, which limits cross-engine
> > > interoperability. FLIP-556 explicitly scoped to 32-bit only for this
> > reason.
> > >
> > > *4. Pushdown Optimization Needs Careful Scoping*
> > >
> > > The bitmap OR aggregation pushdown is the riskiest part of this proposal.
> > > Unlike count(*) (metadata-only), bitmap aggregation pushdown requires the
> > > TabletServer to scan all rows, deserialize each bitmap, and perform
> > > OR-merge. Key concerns:
> > >
> > > * Full table scan on server: Could starve normal read/write operations on
> > > the same tablet.
> > > * Memory pressure: Large bitmaps + concurrent queries could cause OOM. No
> > > memory budget or backpressure is described.
> > > * Server is not a query engine: Currently, TabletServer is designed for
> > > low-latency read/write, not OLAP-style full scans.
> > >
> > > Reading all the bytes and computing in the Flink/StarRocks may be more
> > > efficient and stable for now. I suggest deferring pushdown to future
> > work.
> > >
> > > Overall, this is a solid proposal. Narrowing the scope (skip BITMAP type,
> > > use RB_ prefix, expand functions, carefully scope pushdown) would make it
> > > more incremental and lower-risk.
> > >
> > > Best regards,
> > > Jark Wu
> > >
> >

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Prajwal Banakar <pr...@gmail.com>]

Hi Jark,

Thank you for the thorough and precise feedback. I agree with all four
points you raised and have updated the proposal accordingly.

1. On deferring the BITMAP type:
Agreed. Since FLIP-556 is not yet available in Flink, the in-memory
benefits of a native type cannot be realized as UDFs must operate on byte[]
regardless of the logical type. I have removed Layer 1 and will keep bitmap
columns as BYTES. We can revisit the BITMAP type in a dedicated FIP once
FLIP-556 lands.

2. On the RB_ prefix:
Agreed. Using the "RB_" prefix avoids future conflicts with Flink
built-ins, aligns with existing rbm32/64 naming in Fluss, and ensures
compatibility with pg_roaringbitmap conventions. All functions have been
renamed.

3. On expanding the function set:
Thank you for the references. I have added rb_build, rb_contains, and
rb_to_array to Phase 1, and rb_xor_agg, rb_xor, and rb_andnot to Phase 2. I
have also deferred rb_build_agg_64 due to the lack of standardized 64-bit
serialization formats, consistent with the scope of FLIP-556.

3. On deferring pushdown:
Agreed. The concerns regarding full table scans on the TabletServer and
potential memory pressure are valid. I have deferred Layer 3 to future work
and documented these architectural concerns.

FIP-37 is now a focused single-layer proposal implementing 10 RoaringBitmap
SQL functions registered via FlussCatalog on BYTES columns. The updated
document here:
https://cwiki.apache.org/confluence/display/FLUSS/FIP-37%3A+Native+RoaringBitmap+Integration+for+Apache+Fluss

Best regards,
Prajwal Banakar

On Sun, 19 Apr 2026 at 21:23, Jark Wu <im...@gmail.com> wrote:

> Add the missing links in my previous email
>
> [1] https://github.com/flink-extended/flink-roaringbitmap
> [2] https://github.com/ChenHuajun/pg_roaringbitmap
>
> On Sun, 19 Apr 2026 at 23:51, Jark Wu <im...@gmail.com> wrote:
>
> > Hi Prajwal,
> >
> > Thanks for the proposal! RoaringBitmap SQL functions are a valuable
> > addition to Fluss. I have a few suggestions:
> >
> > *1. Defer BITMAP Type Introduction*
> >
> > I suggest excluding the BITMAP type (Layer 1) from this FIP and keeping
> > bitmap columns as BYTES.
> >
> > The primary benefit of a native BITMAP type — as proposed in Flink's
> > FLIP-556 — is that functions can pass Bitmap Java objects in-memory
> between
> > operators without serialization/deserialization at each step. However,
> > since FLIP-556 is still under discussion and not yet available in Flink,
> > the Flink UDFs we register can only operate on byte[] (BYTES). On the
> Fluss
> > server side, the merge engine aggregators always deserialize from byte[],
> > perform the OR, and serialize back — regardless of whether the logical
> type
> > is BYTES or BITMAP.
> >
> > Introducing the BITMAP type offers limited immediate benefits, while
> > ensuring compatibility is a complex issue that warrants separate
> > discussion. This includes addressing how type conversions between Fluss
> and
> > various engines bridge to BITMAP without breaking existing compatibility,
> > which is not currently covered in this FIP. Therefore, I propose
> deferring
> > the introduction of the BITMAP type to simplify and accelerate the
> approval
> > of the current FIP.
> >
> > The FIP can focus on what matters most: SQL functions operating on BYTES
> > columns that is roaringbitmap binary format underlying. We can revisit
> the
> > BITMAP type once FLIP-556 has been released in Flink.
> >
> > *2. Function Naming: Use "RB_" Prefix Instead of "BITMAP_"*
> >
> > I suggest using the "RB_" prefix rather than "BITMAP_". Reasons:
> >
> > (1) Intentional differentiation from Flink built-ins. When FLIP-556
> > eventually lands, Flink will have its own BITMAP_* built-in functions
> > operating on a native BITMAP type. Our functions operate on BYTES and are
> > registered via FlussCatalog. Using RB_ avoids ambiguity and accidental
> > invocation of the wrong function at runtime.
> > (2) Consistent with Fluss conventions. Fluss already uses rbm32/rbm64 as
> > aggregation function identifiers in the merge engine. RB_ aligns
> naturally
> > with this existing naming.
> > (3) PostgreSQL protocol compatibility. Fluss has future plans for PG
> > protocol compatibility. The de facto standard for RoaringBitmap in
> > PostgreSQL (pg_roaringbitmap) uses rb_ prefix. Adopting this now avoids a
> > rename later.
> >
> > *3. Expand the Function Set*
> >
> > Referencing the function sets from flink-roaringbitmap[1] and
> > pg_roaringbitmap[2], I suggest the following additions:
> >
> > Phase 1 (add these — they are very commonly used):
> >
> > * rb_build(ARRAY<INT>) → BYTES — construct a bitmap from an array without
> > aggregation
> > * rb_contains(BYTES, INT) → BOOLEAN — check if a bitmap contains a value
> > * rb_to_array(BYTES) → ARRAY<INT> — convert bitmap back to array
> > (debugging, integration)
> >
> > Phase 2 (add these to complete set algebra):
> >
> > * rb_xor_agg(BYTES) → BYTES — XOR aggregation
> > * rb_xor(BYTES, BYTES) → BYTES — XOR of two bitmaps
> > * rb_andnot(BYTES, BYTES) → BYTES — difference (AND NOT) of two bitmaps
> >
> > Also, consider deferring "bitmap_build_agg_64" — the 64-bit RoaringBitmap
> > serialization format is not standardized, which limits cross-engine
> > interoperability. FLIP-556 explicitly scoped to 32-bit only for this
> reason.
> >
> > *4. Pushdown Optimization Needs Careful Scoping*
> >
> > The bitmap OR aggregation pushdown is the riskiest part of this proposal.
> > Unlike count(*) (metadata-only), bitmap aggregation pushdown requires the
> > TabletServer to scan all rows, deserialize each bitmap, and perform
> > OR-merge. Key concerns:
> >
> > * Full table scan on server: Could starve normal read/write operations on
> > the same tablet.
> > * Memory pressure: Large bitmaps + concurrent queries could cause OOM. No
> > memory budget or backpressure is described.
> > * Server is not a query engine: Currently, TabletServer is designed for
> > low-latency read/write, not OLAP-style full scans.
> >
> > Reading all the bytes and computing in the Flink/StarRocks may be more
> > efficient and stable for now. I suggest deferring pushdown to future
> work.
> >
> > Overall, this is a solid proposal. Narrowing the scope (skip BITMAP type,
> > use RB_ prefix, expand functions, carefully scope pushdown) would make it
> > more incremental and lower-risk.
> >
> > Best regards,
> > Jark Wu
> >
>

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Jark Wu <im...@gmail.com>]

Add the missing links in my previous email

[1] https://github.com/flink-extended/flink-roaringbitmap
[2] https://github.com/ChenHuajun/pg_roaringbitmap

On Sun, 19 Apr 2026 at 23:51, Jark Wu <im...@gmail.com> wrote:

> Hi Prajwal,
>
> Thanks for the proposal! RoaringBitmap SQL functions are a valuable
> addition to Fluss. I have a few suggestions:
>
> *1. Defer BITMAP Type Introduction*
>
> I suggest excluding the BITMAP type (Layer 1) from this FIP and keeping
> bitmap columns as BYTES.
>
> The primary benefit of a native BITMAP type — as proposed in Flink's
> FLIP-556 — is that functions can pass Bitmap Java objects in-memory between
> operators without serialization/deserialization at each step. However,
> since FLIP-556 is still under discussion and not yet available in Flink,
> the Flink UDFs we register can only operate on byte[] (BYTES). On the Fluss
> server side, the merge engine aggregators always deserialize from byte[],
> perform the OR, and serialize back — regardless of whether the logical type
> is BYTES or BITMAP.
>
> Introducing the BITMAP type offers limited immediate benefits, while
> ensuring compatibility is a complex issue that warrants separate
> discussion. This includes addressing how type conversions between Fluss and
> various engines bridge to BITMAP without breaking existing compatibility,
> which is not currently covered in this FIP. Therefore, I propose deferring
> the introduction of the BITMAP type to simplify and accelerate the approval
> of the current FIP.
>
> The FIP can focus on what matters most: SQL functions operating on BYTES
> columns that is roaringbitmap binary format underlying. We can revisit the
> BITMAP type once FLIP-556 has been released in Flink.
>
> *2. Function Naming: Use "RB_" Prefix Instead of "BITMAP_"*
>
> I suggest using the "RB_" prefix rather than "BITMAP_". Reasons:
>
> (1) Intentional differentiation from Flink built-ins. When FLIP-556
> eventually lands, Flink will have its own BITMAP_* built-in functions
> operating on a native BITMAP type. Our functions operate on BYTES and are
> registered via FlussCatalog. Using RB_ avoids ambiguity and accidental
> invocation of the wrong function at runtime.
> (2) Consistent with Fluss conventions. Fluss already uses rbm32/rbm64 as
> aggregation function identifiers in the merge engine. RB_ aligns naturally
> with this existing naming.
> (3) PostgreSQL protocol compatibility. Fluss has future plans for PG
> protocol compatibility. The de facto standard for RoaringBitmap in
> PostgreSQL (pg_roaringbitmap) uses rb_ prefix. Adopting this now avoids a
> rename later.
>
> *3. Expand the Function Set*
>
> Referencing the function sets from flink-roaringbitmap[1] and
> pg_roaringbitmap[2], I suggest the following additions:
>
> Phase 1 (add these — they are very commonly used):
>
> * rb_build(ARRAY<INT>) → BYTES — construct a bitmap from an array without
> aggregation
> * rb_contains(BYTES, INT) → BOOLEAN — check if a bitmap contains a value
> * rb_to_array(BYTES) → ARRAY<INT> — convert bitmap back to array
> (debugging, integration)
>
> Phase 2 (add these to complete set algebra):
>
> * rb_xor_agg(BYTES) → BYTES — XOR aggregation
> * rb_xor(BYTES, BYTES) → BYTES — XOR of two bitmaps
> * rb_andnot(BYTES, BYTES) → BYTES — difference (AND NOT) of two bitmaps
>
> Also, consider deferring "bitmap_build_agg_64" — the 64-bit RoaringBitmap
> serialization format is not standardized, which limits cross-engine
> interoperability. FLIP-556 explicitly scoped to 32-bit only for this reason.
>
> *4. Pushdown Optimization Needs Careful Scoping*
>
> The bitmap OR aggregation pushdown is the riskiest part of this proposal.
> Unlike count(*) (metadata-only), bitmap aggregation pushdown requires the
> TabletServer to scan all rows, deserialize each bitmap, and perform
> OR-merge. Key concerns:
>
> * Full table scan on server: Could starve normal read/write operations on
> the same tablet.
> * Memory pressure: Large bitmaps + concurrent queries could cause OOM. No
> memory budget or backpressure is described.
> * Server is not a query engine: Currently, TabletServer is designed for
> low-latency read/write, not OLAP-style full scans.
>
> Reading all the bytes and computing in the Flink/StarRocks may be more
> efficient and stable for now. I suggest deferring pushdown to future work.
>
> Overall, this is a solid proposal. Narrowing the scope (skip BITMAP type,
> use RB_ prefix, expand functions, carefully scope pushdown) would make it
> more incremental and lower-risk.
>
> Best regards,
> Jark Wu
>

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Jark Wu <im...@gmail.com>]

Hi Prajwal,

Thanks for the proposal! RoaringBitmap SQL functions are a valuable
addition to Fluss. I have a few suggestions:

*1. Defer BITMAP Type Introduction*

I suggest excluding the BITMAP type (Layer 1) from this FIP and keeping
bitmap columns as BYTES.

The primary benefit of a native BITMAP type — as proposed in Flink's
FLIP-556 — is that functions can pass Bitmap Java objects in-memory between
operators without serialization/deserialization at each step. However,
since FLIP-556 is still under discussion and not yet available in Flink,
the Flink UDFs we register can only operate on byte[] (BYTES). On the Fluss
server side, the merge engine aggregators always deserialize from byte[],
perform the OR, and serialize back — regardless of whether the logical type
is BYTES or BITMAP.

Introducing the BITMAP type offers limited immediate benefits, while
ensuring compatibility is a complex issue that warrants separate
discussion. This includes addressing how type conversions between Fluss and
various engines bridge to BITMAP without breaking existing compatibility,
which is not currently covered in this FIP. Therefore, I propose deferring
the introduction of the BITMAP type to simplify and accelerate the approval
of the current FIP.

The FIP can focus on what matters most: SQL functions operating on BYTES
columns that is roaringbitmap binary format underlying. We can revisit the
BITMAP type once FLIP-556 has been released in Flink.

*2. Function Naming: Use "RB_" Prefix Instead of "BITMAP_"*

I suggest using the "RB_" prefix rather than "BITMAP_". Reasons:

(1) Intentional differentiation from Flink built-ins. When FLIP-556
eventually lands, Flink will have its own BITMAP_* built-in functions
operating on a native BITMAP type. Our functions operate on BYTES and are
registered via FlussCatalog. Using RB_ avoids ambiguity and accidental
invocation of the wrong function at runtime.
(2) Consistent with Fluss conventions. Fluss already uses rbm32/rbm64 as
aggregation function identifiers in the merge engine. RB_ aligns naturally
with this existing naming.
(3) PostgreSQL protocol compatibility. Fluss has future plans for PG
protocol compatibility. The de facto standard for RoaringBitmap in
PostgreSQL (pg_roaringbitmap) uses rb_ prefix. Adopting this now avoids a
rename later.

*3. Expand the Function Set*

Referencing the function sets from flink-roaringbitmap[1] and
pg_roaringbitmap[2], I suggest the following additions:

Phase 1 (add these — they are very commonly used):

* rb_build(ARRAY<INT>) → BYTES — construct a bitmap from an array without
aggregation
* rb_contains(BYTES, INT) → BOOLEAN — check if a bitmap contains a value
* rb_to_array(BYTES) → ARRAY<INT> — convert bitmap back to array
(debugging, integration)

Phase 2 (add these to complete set algebra):

* rb_xor_agg(BYTES) → BYTES — XOR aggregation
* rb_xor(BYTES, BYTES) → BYTES — XOR of two bitmaps
* rb_andnot(BYTES, BYTES) → BYTES — difference (AND NOT) of two bitmaps

Also, consider deferring "bitmap_build_agg_64" — the 64-bit RoaringBitmap
serialization format is not standardized, which limits cross-engine
interoperability. FLIP-556 explicitly scoped to 32-bit only for this reason.

*4. Pushdown Optimization Needs Careful Scoping*

The bitmap OR aggregation pushdown is the riskiest part of this proposal.
Unlike count(*) (metadata-only), bitmap aggregation pushdown requires the
TabletServer to scan all rows, deserialize each bitmap, and perform
OR-merge. Key concerns:

* Full table scan on server: Could starve normal read/write operations on
the same tablet.
* Memory pressure: Large bitmaps + concurrent queries could cause OOM. No
memory budget or backpressure is described.
* Server is not a query engine: Currently, TabletServer is designed for
low-latency read/write, not OLAP-style full scans.

Reading all the bytes and computing in the Flink/StarRocks may be more
efficient and stable for now. I suggest deferring pushdown to future work.

Overall, this is a solid proposal. Narrowing the scope (skip BITMAP type,
use RB_ prefix, expand functions, carefully scope pushdown) would make it
more incremental and lower-risk.

Best regards,
Jark Wu

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Prajwal Banakar <pr...@gmail.com>]

Hi everyone,

Yang Wang's blog post on Real-Time UV Deduplication is now live on the
Fluss website and explicitly references FIP-37 as the future direction.
This validates the motivation behind the proposal and demonstrates strong
community interest.

I look forward to driving FIP-37 toward implementation.

Blog link: https://fluss.apache.org/blog/roaringbitmap-uv-deduplication/

Best regards,
Prajwal Banakar

On Mon, 6 Apr 2026 at 16:37, Giannis Polyzos <ip...@gmail.com> wrote:

> That’s great.. I will also share my write up with the UDFs I implemented
> and if there is something missing from the repo after you introduce the
> UDFs maybe we can add them.
>
> I will ping you to take a look once I create the blog post PR 🙏
>
> Again exciting feature and looking forward to this.
>
> Best,
> Giannis
>
> On Mon, 6 Apr 2026 at 1:17 PM, Prajwal Banakar <prajwal.banakara@gmail.com
> >
> wrote:
>
> > Hi Giannis,
> >
> > Thank you for the detailed and encouraging feedback. I have addressed
> your
> > points as follows:
> >
> > 1. DataTypeVisitor Breaking Change: I agree. To avoid breaking external
> > implementors, I will provide a default implementation that throws an
> > UnsupportedOperationException with a clear message. This is now
> documented
> > in the FIP under Layer 1.
> >
> > 2. NULL Semantics: Thank you for catching the contradiction. The intended
> > behavior is BITMAP_CARDINALITY(NULL) returns NULL (missing value), while
> > BITMAP_CARDINALITY(empty_bitmap) returns 0. I have added an explicit
> > clarification to the FIP to distinguish these cases.
> >
> > 3. 64-bit Scope: Agreed. Since FieldRoaringBitmap64Agg already exists
> > server-side, I have added BITMAP_BUILD_AGG_64 (BIGINT → BITMAP) as a
> Phase
> > 2 deliverable in the updated function table.
> >
> > 4. BITMAP_AND_AGG Safety: I have added a note stating that BITMAP_AND_AGG
> > will execute entirely in Flink without server-side pushdown support in
> this
> > release. I also clarified that combining it with the aggregation merge
> > engine may produce unexpected results during compaction.
> >
> > Regarding BITMAP_TO_STRING, this is a great idea for debugging. I have
> > mentioned this on my GSOC proposal and also added it to the "Future Work"
> > section of the FIP and attributed the suggestion.
> >
> > I also wanted to mention that Jark and Yang Wang have set up an official
> > flink-extended/flink-roaringbitmap repository for the external UDF
> library.
> > I have an open PR there including rb_cardinality, rb_or_agg, and
> > rb_build_agg. This may complement your current work and serve as a bridge
> > until native support lands in Fluss.
> > Link to the repo : https://github.com/flink-extended/flink-roaringbitmap
> >
> > Best regards,
> > Prajwal Banakar
> >
> > On Mon, 6 Apr 2026 at 12:52, Giannis Polyzos <ip...@gmail.com>
> > wrote:
> >
> > > Hi Prajwal,
> > >
> > > This is a great proposal. I have been creating end-to-end real-time
> > profile
> > > use cases with Fluss features, and I had to create a few UDFs to better
> > > interact with BM.
> > > The FIP solves this structural problem as it makes BITMAP a first-class
> > > type, promotes those same operations into the catalog so no UDF jar is
> > > needed, and adds server-side pushdown so BITMAP_OR_AGG queries avoid
> > > materializing per-row data in Flink entirely.
> > >
> > > The overall direction is solid, the three-layer decomposition is clean,
> > and
> > > the backward compatibility story is well thought out. I have a few
> > points I
> > > would like to discuss before moving forward, mostly around design
> > decisions
> > > that I think are worth aligning on at the proposal stage.
> > >
> > > 1. DataTypeVisitor is @PublicStable but im wondering whether this can
> be
> > a
> > > breaking change. Adding a new method to this interface would likely
> break
> > > any external implementors. It would be great if the proposal could
> settle
> > > on an approach: a default fallback method, an abstract base class users
> > can
> > > extend, or a deferral to the next major version. Any of these would
> work.
> > >
> > > 2. Null semantics are a bit contradicted within the proposal itself.
> The
> > > public interface section states BITMAP_CARDINALITY(NULL) → NULL, while
> > the
> > > reference implementation section returns 0 for null input. Could you
> > > clarify which behavior is the intended one?
> > >
> > > 3. BITMAP_BUILD_AGG accepts only INT, while the server already supports
> > > rbm64 for BIGINT. Given that most real-world entity IDs exceed
> > > Integer.MAX_VALUE and the 64-bit aggregator already exists server-side,
> > it
> > > would be worth clarifying whether the initial scope is intentionally
> > > limited to 32-bit and, if so, the reasoning behind that choice.
> > >
> > > 4. BITMAP_AND_AGG is included in the public interface but has no
> > > server-side aggregator and no pushdown support. I'm reluctant that
> > exposing
> > > this in the public API without a working server-side counterpart could
> > lead
> > > to incorrect results when users combine it with the merge engine. Would
> > it
> > > be possible to add a clear section defining exactly when it is safe to
> > use?
> > >
> > > Some food for thought on my side is whether it makes sense (based on my
> > > examples) to consider adding a function that converts a bitmap to a
> > string
> > > so it's human-readable for users debugging. This can be
> > >
> > > For example, take a bitmap and output something like
> > (bitmap_to_string??):
> > > Output format: "count=3 [1001, 1002, 1007]."
> > >
> > > This can be out of scope of this FIP, and more of a future improvement,
> > if
> > > it resonates.
> > > I just wanted to bring this to your attention, just in case you think
> it
> > > makes sense, because it's something that helped me a lot with my
> > examples.
> > >
> > > Looking forward to this
> > >
> > > Best,
> > > Giannis
> > >
> > >
> > >
> > > On Mon, Apr 6, 2026 at 9:12 AM Prajwal Banakar <
> > prajwal.banakara@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I have created the formal Confluence page for FIP-37 [1].
> > > >
> > > > As this is my first time creating a FIP page, I would greatly
> > appreciate
> > > > any feedback or suggestions for improvement. The discussion will
> > continue
> > > > on this thread.
> > > >
> > > > Looking forward to your thoughts.
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-37%3A+Native+RoaringBitmap+Integration+for+Apache+Fluss
> > > >
> > > > Best regards,
> > > > Prajwal Banakar
> > > >
> > > > On Fri, 3 Apr 2026 at 19:22, Giannis Polyzos <ip...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Prajwal,
> > > > > It’s probably a mistake that needs fixing.
> > > > >
> > > > > Feel free to use FIP-37
> > > > >
> > > > > Best,
> > > > > Giannis
> > > > >
> > > > > On Fri, 3 Apr 2026 at 4:49 PM, Prajwal Banakar <
> > > > prajwal.banakara@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi dev's,
> > > > > >
> > > > > > I am currently creating the Confluence page for this proposal.
> > > > > >
> > > > > > I noticed that FIP-35 is not currently listed on the wiki. Could
> > you
> > > > > please
> > > > > > confirm if this number is available for use, or if I should
> > assigned
> > > a
> > > > > > different one?
> > > > > >
> > > > > > Best regards,
> > > > > > Prajwal Banakar
> > > > > >
> > > > > > On Wed, 11 Mar 2026 at 22:56, Prajwal Banakar <
> > > > > prajwal.banakara@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Keith,
> > > > > > >
> > > > > > > Thank you for the follow-up.
> > > > > > >
> > > > > > > You are correct that FieldRoaringBitmap64Agg already exists in
> > > > > > > fluss-server. I have updated the proposal accordingly. To
> > clarify,
> > > > the
> > > > > > > 32-bit scope is intended to keep the initial type system and
> SQL
> > > > > function
> > > > > > > surface focused and deliverable, rather than being a limitation
> > of
> > > > the
> > > > > > > aggregator itself. Since the server-side aggregator is already
> in
> > > > > place,
> > > > > > > RBM64 will be a natural, low-risk follow-on once the type
> system
> > > and
> > > > > > > pushdown infrastructure are established.
> > > > > > >
> > > > > > > I have also removed the misleading motivation paragraph as you
> > > > > suggested.
> > > > > > > The updated document is available at the same link.
> > Additionally, I
> > > > > would
> > > > > > > welcome Yang's input on the alignment with FIP-21.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Prajwal Banakar
> > > > > > >
> > > > > > > On Wed, 11 Mar 2026 at 17:37, Keith Lee <
> > > leekeiabstraction@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Prajwal,
> > > > > > >>
> > > > > > >> Thank you for addressing / answering the questions.
> > > > > > >>
> > > > > > >> > This proposal adds the missing bridge: a proper BITMAP DDL
> > type,
> > > > SQL
> > > > > > >> functions (BITMAP_BUILD, BITMAP_OR_AGG, BITMAP_CARDINALITY),
> and
> > > > > > pushdown
> > > > > > >> via applyAggregates(). The storage-side aggregation logic
> > already
> > > > > > exists;
> > > > > > >> this proposal makes it accessible end-to-end
> > > > > > >>
> > > > > > >> 1. That makes sense. I think the motivation section should
> lead
> > > with
> > > > > > that
> > > > > > >> and remove the following as it can be misleading given that
> rbm
> > is
> > > > > > >> supported by aggregation merge engine: “users requiring
> > > > > high-cardinality
> > > > > > >> unique counting (e.g., UV analytics) must execute Client-Side
> > > > > > Aggregation.
> > > > > > >> The TabletServer is forced to send massive amounts of raw
> > > > > LogRecordBatch
> > > > > > >> rows over the network to a Flink cluster for evaluation. This
> > > > results
> > > > > in
> > > > > > >> unnecessary network transfer and prevents efficient
> utilization
> > of
> > > > the
> > > > > > >> existing aggregation merge engine.”
> > > > > > >>
> > > > > > >> 2. That makes sense. Thank you for the context.
> > > > > > >>
> > > > > > >> 3.
> > > > > > >>
> > > > > > >> > RBM64 requires a fundamentally different internal
> structure; a
> > > map
> > > > > of
> > > > > > >> RBM32 chunks which increases implementation and serialization
> > > > > complexity
> > > > > > >> significantly.
> > > > > > >>
> > > > > > >> My understanding is that the proposal wires existing
> > > > > > >> FieldRoaringBitmap32Agg to support rbm32.
> > FieldRoaringBitmap64Agg
> > > > > should
> > > > > > >> already exist and handle the complexity that you mentioned?
> > > > > > >>
> > > > > > >> Additionally, it might be good for Yang to review / provide
> > input
> > > on
> > > > > > this
> > > > > > >> given his work on FIP-21.
> > > > > > >>
> > > > > > >> Best regards
> > > > > > >>
> > > > > > >> Keith Lee
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, 11 Mar 2026 at 05:49, Prajwal Banakar <
> > > > > > prajwal.banakara@gmail.com
> > > > > > >> >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi Keith, thank you for the detailed feedback.
> > > > > > >> >
> > > > > > >> > 1. On motivation vs existing aggregation merge engine: The
> > > > > aggregation
> > > > > > >> > merge engine in 0.9 supports rbm32/rbm64 at the storage
> level,
> > > but
> > > > > > >> BITMAP
> > > > > > >> > is not yet a first-class type in the DDL or type system.
> Users
> > > > today
> > > > > > >> must
> > > > > > >> > declare the column as BYTES (as shown in the 0.9 release
> > > example:
> > > > > > >> uv_bitmap
> > > > > > >> > BYTES), and there are no SQL functions to build, merge, or
> > query
> > > > > > bitmaps
> > > > > > >> > from Flink SQL. This proposal adds the missing bridge: a
> > proper
> > > > > BITMAP
> > > > > > >> DDL
> > > > > > >> > type, SQL functions (BITMAP_BUILD, BITMAP_OR_AGG,
> > > > > BITMAP_CARDINALITY),
> > > > > > >> and
> > > > > > >> > pushdown via applyAggregates(). The storage-side aggregation
> > > logic
> > > > > > >> already
> > > > > > >> > exists; this proposal makes it accessible end-to-end.
> > > > > > >> >
> > > > > > >> > 2. On NULL semantics: BITMAP_OR(bitmap, NULL) returns NULL
> > > > following
> > > > > > >> > standard SQL scalar function semantics where NULL inputs
> > > propagate
> > > > > to
> > > > > > >> NULL
> > > > > > >> > outputs. BITMAP_OR_AGG follows aggregate function convention
> > > > > > consistent
> > > > > > >> > with how SUM and AVG behave, where NULLs in individual rows
> > are
> > > > > > skipped
> > > > > > >> and
> > > > > > >> > only a fully NULL input set returns NULL. This distinction
> > > follows
> > > > > > >> FLIP-556
> > > > > > >> > and StarRocks semantics.
> > > > > > >> >
> > > > > > >> > 3. On 32-bit scope: The proposal is scoped to 32-bit
> initially
> > > > > because
> > > > > > >> > RoaringBitmap32 covers integer values up to 2^32 (~4
> billion),
> > > > which
> > > > > > is
> > > > > > >> > sufficient for most user ID and session ID use cases. RBM64
> > > > > requires a
> > > > > > >> > fundamentally different internal structure; a map of RBM32
> > > chunks
> > > > > > which
> > > > > > >> > increases implementation and serialization complexity
> > > > significantly.
> > > > > > >> > Starting with 32-bit keeps the initial scope focused and
> > > > > deliverable.
> > > > > > >> RBM64
> > > > > > >> > support is listed as a Could-Have in the MoSCoW deliverables
> > and
> > > > can
> > > > > > >> follow
> > > > > > >> > in a subsequent iteration.
> > > > > > >> >
> > > > > > >> > Best regards,
> > > > > > >> >
> > > > > > >> > Prajwal Banakar
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, 11 Mar 2026 at 01:34, Keith Lee <
> > > > > leekeiabstraction@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Hello Prajwal,
> > > > > > >> > >
> > > > > > >> > > Thank you for the detailed proposal. I enjoyed reading it
> > and
> > > > > have a
> > > > > > >> few
> > > > > > >> > > questions/comments.
> > > > > > >> > >
> > > > > > >> > > 1. On motivation, can you provide context on how this
> > differs
> > > > with
> > > > > > >> > > aggregation merge engine’s roaring bitmap implementation
> > [1]?
> > > > > > >> > Specifically,
> > > > > > >> > > motivation part states that “users requiring high
> > cardinality
> > > > > unique
> > > > > > >> > > counting … must execute client-side aggregation”.
> > Aggregation
> > > > > merge
> > > > > > >> > engine
> > > > > > >> > > performs aggregation on server-side. The motivation
> section
> > > > should
> > > > > > >> > clarify
> > > > > > >> > > how the proposed changes improve or complement aggregation
> > > merge
> > > > > > >> engine,
> > > > > > >> > > which seems to have been considered as Section 2
> references
> > > > FIP-21
> > > > > > >> > > Aggregation Merge Engine. Adding this context will help
> > > readers
> > > > > > >> > understand
> > > > > > >> > > the motivation of the proposal better.
> > > > > > >> > >
> > > > > > >> > > 2. Can you clarify the NULL semantics section specifically
> > on
> > > > the
> > > > > > >> > decision
> > > > > > >> > > on why BITMAP_OR(bitmap, NULL) returns NULL but
> > BITMAP_OR_AGG
> > > > only
> > > > > > >> > returns
> > > > > > >> > > null when all rows are NULL?
> > > > > > >> > >
> > > > > > >> > > 3. Why is the scope limited to 32 bit bitmaps? Adding the
> > > > > rationale
> > > > > > >> > behind
> > > > > > >> > > these e.g. how (if any) support of 64bit bitmaps would
> > > increase
> > > > > > >> > > implementation complexity. Articulating these may help
> other
> > > > > > >> contributors
> > > > > > >> > > understand the complexity and perhaps come up with
> > suggestions
> > > > on
> > > > > > how
> > > > > > >> to
> > > > > > >> > > address them.
> > > > > > >> > >
> > > > > > >> > > Best regards
> > > > > > >> > >
> > > > > > >> > > Keith Lee
> > > > > > >> > >
> > > > > > >> > > [1]
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://fluss.apache.org/blog/releases/0.9/#2-storage-level-processing--semantics
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Mon, 9 Mar 2026 at 05:31, Prajwal Banakar <
> > > > > > >> prajwal.banakara@gmail.com
> > > > > > >> > >
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Hi Devs,
> > > > > > >> > > >
> > > > > > >> > > > I have pushed a working prototype to my public fork
> > > > > demonstrating
> > > > > > >> the
> > > > > > >> > > > BitmapType integrated with FieldRoaringBitmap32Agg. This
> > > > > includes
> > > > > > >> four
> > > > > > >> > > > passing unit tests.
> > > > > > >> > > >
> > > > > > >> > > > The link to the prototype is available in the Google
> Doc,
> > > and
> > > > > you
> > > > > > >> can
> > > > > > >> > > also
> > > > > > >> > > > find it here:
> > > > > > >> > > >
> > > > > > >>
> > > >
> https://github.com/Prajwal-banakar/fluss/tree/RoaringBitmap-prototype
> > > > > > >> > > >
> > > > > > >> > > > The Google Doc link remains the same. I look forward to
> > your
> > > > > > >> feedback.
> > > > > > >> > > >
> > > > > > >> > > > Best regards,
> > > > > > >> > > >
> > > > > > >> > > > Prajwal Banakar
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On Sun, 1 Mar, 2026, 11:49 am Prajwal Banakar, <
> > > > > > >> > > prajwal.banakara@gmail.com
> > > > > > >> > > > >
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Hi everyone,
> > > > > > >> > > > >
> > > > > > >> > > > > I would like to start a discussion on the proposal for
> > > > Native
> > > > > > >> Bitmap
> > > > > > >> > > > > Integration & Stateless Pushdown Aggregation.
> > > > > > >> > > > >
> > > > > > >> > > > > This proposal enables end-to-end native support for
> the
> > > > BITMAP
> > > > > > >> type
> > > > > > >> > in
> > > > > > >> > > > > Fluss and integrates it with the existing aggregation
> > > merge
> > > > > > >> engine to
> > > > > > >> > > > > support server-side bitmap union pushdown. The goal is
> > to
> > > > > reduce
> > > > > > >> > > network
> > > > > > >> > > > > transfer and offload DISTINCT-style aggregation from
> > Flink
> > > > to
> > > > > > the
> > > > > > >> > > > > TabletServer.
> > > > > > >> > > > >
> > > > > > >> > > > > Key highlights of the proposal include:
> > > > > > >> > > > >
> > > > > > >> > > > > - Type System: Promoting BITMAP to a first-class
> logical
> > > > type.
> > > > > > >> > > > > - UDF Suite: Introducing BITMAP_BUILD, BITMAP_OR_AGG,
> > and
> > > > > > >> > > > > BITMAP_CARDINALITY (aligned with FLIP-556 and
> StarRocks
> > > > > > >> semantics).
> > > > > > >> > > > > - Optimizer: Planner-based pushdown via
> applyAggregates
> > in
> > > > the
> > > > > > >> Flink
> > > > > > >> > > > > connector.
> > > > > > >> > > > > - Safety: No changes to LogRecordBatch or WAL, making
> > this
> > > > > > >> strictly
> > > > > > >> > > > > additive and migration-free.
> > > > > > >> > > > >
> > > > > > >> > > > > You can find the full proposal document here:
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1sDhfkmo-w-UTvo2n3rsY1lytSSryswfkI83cSdka8s0/edit?usp=sharing
> > > > > > >> > > > >
> > > > > > >> > > > > I would appreciate feedback on the public interfaces,
> > > > pushdown
> > > > > > >> > > > > constraints, and overall scope.
> > > > > > >> > > > >
> > > > > > >> > > > > Best regards,
> > > > > > >> > > > > Prajwal Banakar
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Giannis Polyzos <ip...@gmail.com>]

That’s great.. I will also share my write up with the UDFs I implemented
and if there is something missing from the repo after you introduce the
UDFs maybe we can add them.

I will ping you to take a look once I create the blog post PR 🙏

Again exciting feature and looking forward to this.

Best,
Giannis

On Mon, 6 Apr 2026 at 1:17 PM, Prajwal Banakar <pr...@gmail.com>
wrote:

> Hi Giannis,
>
> Thank you for the detailed and encouraging feedback. I have addressed your
> points as follows:
>
> 1. DataTypeVisitor Breaking Change: I agree. To avoid breaking external
> implementors, I will provide a default implementation that throws an
> UnsupportedOperationException with a clear message. This is now documented
> in the FIP under Layer 1.
>
> 2. NULL Semantics: Thank you for catching the contradiction. The intended
> behavior is BITMAP_CARDINALITY(NULL) returns NULL (missing value), while
> BITMAP_CARDINALITY(empty_bitmap) returns 0. I have added an explicit
> clarification to the FIP to distinguish these cases.
>
> 3. 64-bit Scope: Agreed. Since FieldRoaringBitmap64Agg already exists
> server-side, I have added BITMAP_BUILD_AGG_64 (BIGINT → BITMAP) as a Phase
> 2 deliverable in the updated function table.
>
> 4. BITMAP_AND_AGG Safety: I have added a note stating that BITMAP_AND_AGG
> will execute entirely in Flink without server-side pushdown support in this
> release. I also clarified that combining it with the aggregation merge
> engine may produce unexpected results during compaction.
>
> Regarding BITMAP_TO_STRING, this is a great idea for debugging. I have
> mentioned this on my GSOC proposal and also added it to the "Future Work"
> section of the FIP and attributed the suggestion.
>
> I also wanted to mention that Jark and Yang Wang have set up an official
> flink-extended/flink-roaringbitmap repository for the external UDF library.
> I have an open PR there including rb_cardinality, rb_or_agg, and
> rb_build_agg. This may complement your current work and serve as a bridge
> until native support lands in Fluss.
> Link to the repo : https://github.com/flink-extended/flink-roaringbitmap
>
> Best regards,
> Prajwal Banakar
>
> On Mon, 6 Apr 2026 at 12:52, Giannis Polyzos <ip...@gmail.com>
> wrote:
>
> > Hi Prajwal,
> >
> > This is a great proposal. I have been creating end-to-end real-time
> profile
> > use cases with Fluss features, and I had to create a few UDFs to better
> > interact with BM.
> > The FIP solves this structural problem as it makes BITMAP a first-class
> > type, promotes those same operations into the catalog so no UDF jar is
> > needed, and adds server-side pushdown so BITMAP_OR_AGG queries avoid
> > materializing per-row data in Flink entirely.
> >
> > The overall direction is solid, the three-layer decomposition is clean,
> and
> > the backward compatibility story is well thought out. I have a few
> points I
> > would like to discuss before moving forward, mostly around design
> decisions
> > that I think are worth aligning on at the proposal stage.
> >
> > 1. DataTypeVisitor is @PublicStable but im wondering whether this can be
> a
> > breaking change. Adding a new method to this interface would likely break
> > any external implementors. It would be great if the proposal could settle
> > on an approach: a default fallback method, an abstract base class users
> can
> > extend, or a deferral to the next major version. Any of these would work.
> >
> > 2. Null semantics are a bit contradicted within the proposal itself. The
> > public interface section states BITMAP_CARDINALITY(NULL) → NULL, while
> the
> > reference implementation section returns 0 for null input. Could you
> > clarify which behavior is the intended one?
> >
> > 3. BITMAP_BUILD_AGG accepts only INT, while the server already supports
> > rbm64 for BIGINT. Given that most real-world entity IDs exceed
> > Integer.MAX_VALUE and the 64-bit aggregator already exists server-side,
> it
> > would be worth clarifying whether the initial scope is intentionally
> > limited to 32-bit and, if so, the reasoning behind that choice.
> >
> > 4. BITMAP_AND_AGG is included in the public interface but has no
> > server-side aggregator and no pushdown support. I'm reluctant that
> exposing
> > this in the public API without a working server-side counterpart could
> lead
> > to incorrect results when users combine it with the merge engine. Would
> it
> > be possible to add a clear section defining exactly when it is safe to
> use?
> >
> > Some food for thought on my side is whether it makes sense (based on my
> > examples) to consider adding a function that converts a bitmap to a
> string
> > so it's human-readable for users debugging. This can be
> >
> > For example, take a bitmap and output something like
> (bitmap_to_string??):
> > Output format: "count=3 [1001, 1002, 1007]."
> >
> > This can be out of scope of this FIP, and more of a future improvement,
> if
> > it resonates.
> > I just wanted to bring this to your attention, just in case you think it
> > makes sense, because it's something that helped me a lot with my
> examples.
> >
> > Looking forward to this
> >
> > Best,
> > Giannis
> >
> >
> >
> > On Mon, Apr 6, 2026 at 9:12 AM Prajwal Banakar <
> prajwal.banakara@gmail.com
> > >
> > wrote:
> >
> > > Hi everyone,
> > >
> > > I have created the formal Confluence page for FIP-37 [1].
> > >
> > > As this is my first time creating a FIP page, I would greatly
> appreciate
> > > any feedback or suggestions for improvement. The discussion will
> continue
> > > on this thread.
> > >
> > > Looking forward to your thoughts.
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-37%3A+Native+RoaringBitmap+Integration+for+Apache+Fluss
> > >
> > > Best regards,
> > > Prajwal Banakar
> > >
> > > On Fri, 3 Apr 2026 at 19:22, Giannis Polyzos <ip...@gmail.com>
> > > wrote:
> > >
> > > > Hi Prajwal,
> > > > It’s probably a mistake that needs fixing.
> > > >
> > > > Feel free to use FIP-37
> > > >
> > > > Best,
> > > > Giannis
> > > >
> > > > On Fri, 3 Apr 2026 at 4:49 PM, Prajwal Banakar <
> > > prajwal.banakara@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi dev's,
> > > > >
> > > > > I am currently creating the Confluence page for this proposal.
> > > > >
> > > > > I noticed that FIP-35 is not currently listed on the wiki. Could
> you
> > > > please
> > > > > confirm if this number is available for use, or if I should
> assigned
> > a
> > > > > different one?
> > > > >
> > > > > Best regards,
> > > > > Prajwal Banakar
> > > > >
> > > > > On Wed, 11 Mar 2026 at 22:56, Prajwal Banakar <
> > > > prajwal.banakara@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Keith,
> > > > > >
> > > > > > Thank you for the follow-up.
> > > > > >
> > > > > > You are correct that FieldRoaringBitmap64Agg already exists in
> > > > > > fluss-server. I have updated the proposal accordingly. To
> clarify,
> > > the
> > > > > > 32-bit scope is intended to keep the initial type system and SQL
> > > > function
> > > > > > surface focused and deliverable, rather than being a limitation
> of
> > > the
> > > > > > aggregator itself. Since the server-side aggregator is already in
> > > > place,
> > > > > > RBM64 will be a natural, low-risk follow-on once the type system
> > and
> > > > > > pushdown infrastructure are established.
> > > > > >
> > > > > > I have also removed the misleading motivation paragraph as you
> > > > suggested.
> > > > > > The updated document is available at the same link.
> Additionally, I
> > > > would
> > > > > > welcome Yang's input on the alignment with FIP-21.
> > > > > >
> > > > > > Best regards,
> > > > > > Prajwal Banakar
> > > > > >
> > > > > > On Wed, 11 Mar 2026 at 17:37, Keith Lee <
> > leekeiabstraction@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Prajwal,
> > > > > >>
> > > > > >> Thank you for addressing / answering the questions.
> > > > > >>
> > > > > >> > This proposal adds the missing bridge: a proper BITMAP DDL
> type,
> > > SQL
> > > > > >> functions (BITMAP_BUILD, BITMAP_OR_AGG, BITMAP_CARDINALITY), and
> > > > > pushdown
> > > > > >> via applyAggregates(). The storage-side aggregation logic
> already
> > > > > exists;
> > > > > >> this proposal makes it accessible end-to-end
> > > > > >>
> > > > > >> 1. That makes sense. I think the motivation section should lead
> > with
> > > > > that
> > > > > >> and remove the following as it can be misleading given that rbm
> is
> > > > > >> supported by aggregation merge engine: “users requiring
> > > > high-cardinality
> > > > > >> unique counting (e.g., UV analytics) must execute Client-Side
> > > > > Aggregation.
> > > > > >> The TabletServer is forced to send massive amounts of raw
> > > > LogRecordBatch
> > > > > >> rows over the network to a Flink cluster for evaluation. This
> > > results
> > > > in
> > > > > >> unnecessary network transfer and prevents efficient utilization
> of
> > > the
> > > > > >> existing aggregation merge engine.”
> > > > > >>
> > > > > >> 2. That makes sense. Thank you for the context.
> > > > > >>
> > > > > >> 3.
> > > > > >>
> > > > > >> > RBM64 requires a fundamentally different internal structure; a
> > map
> > > > of
> > > > > >> RBM32 chunks which increases implementation and serialization
> > > > complexity
> > > > > >> significantly.
> > > > > >>
> > > > > >> My understanding is that the proposal wires existing
> > > > > >> FieldRoaringBitmap32Agg to support rbm32.
> FieldRoaringBitmap64Agg
> > > > should
> > > > > >> already exist and handle the complexity that you mentioned?
> > > > > >>
> > > > > >> Additionally, it might be good for Yang to review / provide
> input
> > on
> > > > > this
> > > > > >> given his work on FIP-21.
> > > > > >>
> > > > > >> Best regards
> > > > > >>
> > > > > >> Keith Lee
> > > > > >>
> > > > > >>
> > > > > >> On Wed, 11 Mar 2026 at 05:49, Prajwal Banakar <
> > > > > prajwal.banakara@gmail.com
> > > > > >> >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Keith, thank you for the detailed feedback.
> > > > > >> >
> > > > > >> > 1. On motivation vs existing aggregation merge engine: The
> > > > aggregation
> > > > > >> > merge engine in 0.9 supports rbm32/rbm64 at the storage level,
> > but
> > > > > >> BITMAP
> > > > > >> > is not yet a first-class type in the DDL or type system. Users
> > > today
> > > > > >> must
> > > > > >> > declare the column as BYTES (as shown in the 0.9 release
> > example:
> > > > > >> uv_bitmap
> > > > > >> > BYTES), and there are no SQL functions to build, merge, or
> query
> > > > > bitmaps
> > > > > >> > from Flink SQL. This proposal adds the missing bridge: a
> proper
> > > > BITMAP
> > > > > >> DDL
> > > > > >> > type, SQL functions (BITMAP_BUILD, BITMAP_OR_AGG,
> > > > BITMAP_CARDINALITY),
> > > > > >> and
> > > > > >> > pushdown via applyAggregates(). The storage-side aggregation
> > logic
> > > > > >> already
> > > > > >> > exists; this proposal makes it accessible end-to-end.
> > > > > >> >
> > > > > >> > 2. On NULL semantics: BITMAP_OR(bitmap, NULL) returns NULL
> > > following
> > > > > >> > standard SQL scalar function semantics where NULL inputs
> > propagate
> > > > to
> > > > > >> NULL
> > > > > >> > outputs. BITMAP_OR_AGG follows aggregate function convention
> > > > > consistent
> > > > > >> > with how SUM and AVG behave, where NULLs in individual rows
> are
> > > > > skipped
> > > > > >> and
> > > > > >> > only a fully NULL input set returns NULL. This distinction
> > follows
> > > > > >> FLIP-556
> > > > > >> > and StarRocks semantics.
> > > > > >> >
> > > > > >> > 3. On 32-bit scope: The proposal is scoped to 32-bit initially
> > > > because
> > > > > >> > RoaringBitmap32 covers integer values up to 2^32 (~4 billion),
> > > which
> > > > > is
> > > > > >> > sufficient for most user ID and session ID use cases. RBM64
> > > > requires a
> > > > > >> > fundamentally different internal structure; a map of RBM32
> > chunks
> > > > > which
> > > > > >> > increases implementation and serialization complexity
> > > significantly.
> > > > > >> > Starting with 32-bit keeps the initial scope focused and
> > > > deliverable.
> > > > > >> RBM64
> > > > > >> > support is listed as a Could-Have in the MoSCoW deliverables
> and
> > > can
> > > > > >> follow
> > > > > >> > in a subsequent iteration.
> > > > > >> >
> > > > > >> > Best regards,
> > > > > >> >
> > > > > >> > Prajwal Banakar
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, 11 Mar 2026 at 01:34, Keith Lee <
> > > > leekeiabstraction@gmail.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Hello Prajwal,
> > > > > >> > >
> > > > > >> > > Thank you for the detailed proposal. I enjoyed reading it
> and
> > > > have a
> > > > > >> few
> > > > > >> > > questions/comments.
> > > > > >> > >
> > > > > >> > > 1. On motivation, can you provide context on how this
> differs
> > > with
> > > > > >> > > aggregation merge engine’s roaring bitmap implementation
> [1]?
> > > > > >> > Specifically,
> > > > > >> > > motivation part states that “users requiring high
> cardinality
> > > > unique
> > > > > >> > > counting … must execute client-side aggregation”.
> Aggregation
> > > > merge
> > > > > >> > engine
> > > > > >> > > performs aggregation on server-side. The motivation section
> > > should
> > > > > >> > clarify
> > > > > >> > > how the proposed changes improve or complement aggregation
> > merge
> > > > > >> engine,
> > > > > >> > > which seems to have been considered as Section 2 references
> > > FIP-21
> > > > > >> > > Aggregation Merge Engine. Adding this context will help
> > readers
> > > > > >> > understand
> > > > > >> > > the motivation of the proposal better.
> > > > > >> > >
> > > > > >> > > 2. Can you clarify the NULL semantics section specifically
> on
> > > the
> > > > > >> > decision
> > > > > >> > > on why BITMAP_OR(bitmap, NULL) returns NULL but
> BITMAP_OR_AGG
> > > only
> > > > > >> > returns
> > > > > >> > > null when all rows are NULL?
> > > > > >> > >
> > > > > >> > > 3. Why is the scope limited to 32 bit bitmaps? Adding the
> > > > rationale
> > > > > >> > behind
> > > > > >> > > these e.g. how (if any) support of 64bit bitmaps would
> > increase
> > > > > >> > > implementation complexity. Articulating these may help other
> > > > > >> contributors
> > > > > >> > > understand the complexity and perhaps come up with
> suggestions
> > > on
> > > > > how
> > > > > >> to
> > > > > >> > > address them.
> > > > > >> > >
> > > > > >> > > Best regards
> > > > > >> > >
> > > > > >> > > Keith Lee
> > > > > >> > >
> > > > > >> > > [1]
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://fluss.apache.org/blog/releases/0.9/#2-storage-level-processing--semantics
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Mon, 9 Mar 2026 at 05:31, Prajwal Banakar <
> > > > > >> prajwal.banakara@gmail.com
> > > > > >> > >
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > Hi Devs,
> > > > > >> > > >
> > > > > >> > > > I have pushed a working prototype to my public fork
> > > > demonstrating
> > > > > >> the
> > > > > >> > > > BitmapType integrated with FieldRoaringBitmap32Agg. This
> > > > includes
> > > > > >> four
> > > > > >> > > > passing unit tests.
> > > > > >> > > >
> > > > > >> > > > The link to the prototype is available in the Google Doc,
> > and
> > > > you
> > > > > >> can
> > > > > >> > > also
> > > > > >> > > > find it here:
> > > > > >> > > >
> > > > > >>
> > > https://github.com/Prajwal-banakar/fluss/tree/RoaringBitmap-prototype
> > > > > >> > > >
> > > > > >> > > > The Google Doc link remains the same. I look forward to
> your
> > > > > >> feedback.
> > > > > >> > > >
> > > > > >> > > > Best regards,
> > > > > >> > > >
> > > > > >> > > > Prajwal Banakar
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Sun, 1 Mar, 2026, 11:49 am Prajwal Banakar, <
> > > > > >> > > prajwal.banakara@gmail.com
> > > > > >> > > > >
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi everyone,
> > > > > >> > > > >
> > > > > >> > > > > I would like to start a discussion on the proposal for
> > > Native
> > > > > >> Bitmap
> > > > > >> > > > > Integration & Stateless Pushdown Aggregation.
> > > > > >> > > > >
> > > > > >> > > > > This proposal enables end-to-end native support for the
> > > BITMAP
> > > > > >> type
> > > > > >> > in
> > > > > >> > > > > Fluss and integrates it with the existing aggregation
> > merge
> > > > > >> engine to
> > > > > >> > > > > support server-side bitmap union pushdown. The goal is
> to
> > > > reduce
> > > > > >> > > network
> > > > > >> > > > > transfer and offload DISTINCT-style aggregation from
> Flink
> > > to
> > > > > the
> > > > > >> > > > > TabletServer.
> > > > > >> > > > >
> > > > > >> > > > > Key highlights of the proposal include:
> > > > > >> > > > >
> > > > > >> > > > > - Type System: Promoting BITMAP to a first-class logical
> > > type.
> > > > > >> > > > > - UDF Suite: Introducing BITMAP_BUILD, BITMAP_OR_AGG,
> and
> > > > > >> > > > > BITMAP_CARDINALITY (aligned with FLIP-556 and StarRocks
> > > > > >> semantics).
> > > > > >> > > > > - Optimizer: Planner-based pushdown via applyAggregates
> in
> > > the
> > > > > >> Flink
> > > > > >> > > > > connector.
> > > > > >> > > > > - Safety: No changes to LogRecordBatch or WAL, making
> this
> > > > > >> strictly
> > > > > >> > > > > additive and migration-free.
> > > > > >> > > > >
> > > > > >> > > > > You can find the full proposal document here:
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1sDhfkmo-w-UTvo2n3rsY1lytSSryswfkI83cSdka8s0/edit?usp=sharing
> > > > > >> > > > >
> > > > > >> > > > > I would appreciate feedback on the public interfaces,
> > > pushdown
> > > > > >> > > > > constraints, and overall scope.
> > > > > >> > > > >
> > > > > >> > > > > Best regards,
> > > > > >> > > > > Prajwal Banakar
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Prajwal Banakar <pr...@gmail.com>]

Hi Giannis,

Thank you for the detailed and encouraging feedback. I have addressed your
points as follows:

1. DataTypeVisitor Breaking Change: I agree. To avoid breaking external
implementors, I will provide a default implementation that throws an
UnsupportedOperationException with a clear message. This is now documented
in the FIP under Layer 1.

2. NULL Semantics: Thank you for catching the contradiction. The intended
behavior is BITMAP_CARDINALITY(NULL) returns NULL (missing value), while
BITMAP_CARDINALITY(empty_bitmap) returns 0. I have added an explicit
clarification to the FIP to distinguish these cases.

3. 64-bit Scope: Agreed. Since FieldRoaringBitmap64Agg already exists
server-side, I have added BITMAP_BUILD_AGG_64 (BIGINT → BITMAP) as a Phase
2 deliverable in the updated function table.

4. BITMAP_AND_AGG Safety: I have added a note stating that BITMAP_AND_AGG
will execute entirely in Flink without server-side pushdown support in this
release. I also clarified that combining it with the aggregation merge
engine may produce unexpected results during compaction.

Regarding BITMAP_TO_STRING, this is a great idea for debugging. I have
mentioned this on my GSOC proposal and also added it to the "Future Work"
section of the FIP and attributed the suggestion.

I also wanted to mention that Jark and Yang Wang have set up an official
flink-extended/flink-roaringbitmap repository for the external UDF library.
I have an open PR there including rb_cardinality, rb_or_agg, and
rb_build_agg. This may complement your current work and serve as a bridge
until native support lands in Fluss.
Link to the repo : https://github.com/flink-extended/flink-roaringbitmap

Best regards,
Prajwal Banakar

On Mon, 6 Apr 2026 at 12:52, Giannis Polyzos <ip...@gmail.com> wrote:

> Hi Prajwal,
>
> This is a great proposal. I have been creating end-to-end real-time profile
> use cases with Fluss features, and I had to create a few UDFs to better
> interact with BM.
> The FIP solves this structural problem as it makes BITMAP a first-class
> type, promotes those same operations into the catalog so no UDF jar is
> needed, and adds server-side pushdown so BITMAP_OR_AGG queries avoid
> materializing per-row data in Flink entirely.
>
> The overall direction is solid, the three-layer decomposition is clean, and
> the backward compatibility story is well thought out. I have a few points I
> would like to discuss before moving forward, mostly around design decisions
> that I think are worth aligning on at the proposal stage.
>
> 1. DataTypeVisitor is @PublicStable but im wondering whether this can be a
> breaking change. Adding a new method to this interface would likely break
> any external implementors. It would be great if the proposal could settle
> on an approach: a default fallback method, an abstract base class users can
> extend, or a deferral to the next major version. Any of these would work.
>
> 2. Null semantics are a bit contradicted within the proposal itself. The
> public interface section states BITMAP_CARDINALITY(NULL) → NULL, while the
> reference implementation section returns 0 for null input. Could you
> clarify which behavior is the intended one?
>
> 3. BITMAP_BUILD_AGG accepts only INT, while the server already supports
> rbm64 for BIGINT. Given that most real-world entity IDs exceed
> Integer.MAX_VALUE and the 64-bit aggregator already exists server-side, it
> would be worth clarifying whether the initial scope is intentionally
> limited to 32-bit and, if so, the reasoning behind that choice.
>
> 4. BITMAP_AND_AGG is included in the public interface but has no
> server-side aggregator and no pushdown support. I'm reluctant that exposing
> this in the public API without a working server-side counterpart could lead
> to incorrect results when users combine it with the merge engine. Would it
> be possible to add a clear section defining exactly when it is safe to use?
>
> Some food for thought on my side is whether it makes sense (based on my
> examples) to consider adding a function that converts a bitmap to a string
> so it's human-readable for users debugging. This can be
>
> For example, take a bitmap and output something like (bitmap_to_string??):
> Output format: "count=3 [1001, 1002, 1007]."
>
> This can be out of scope of this FIP, and more of a future improvement, if
> it resonates.
> I just wanted to bring this to your attention, just in case you think it
> makes sense, because it's something that helped me a lot with my examples.
>
> Looking forward to this
>
> Best,
> Giannis
>
>
>
> On Mon, Apr 6, 2026 at 9:12 AM Prajwal Banakar <prajwal.banakara@gmail.com
> >
> wrote:
>
> > Hi everyone,
> >
> > I have created the formal Confluence page for FIP-37 [1].
> >
> > As this is my first time creating a FIP page, I would greatly appreciate
> > any feedback or suggestions for improvement. The discussion will continue
> > on this thread.
> >
> > Looking forward to your thoughts.
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-37%3A+Native+RoaringBitmap+Integration+for+Apache+Fluss
> >
> > Best regards,
> > Prajwal Banakar
> >
> > On Fri, 3 Apr 2026 at 19:22, Giannis Polyzos <ip...@gmail.com>
> > wrote:
> >
> > > Hi Prajwal,
> > > It’s probably a mistake that needs fixing.
> > >
> > > Feel free to use FIP-37
> > >
> > > Best,
> > > Giannis
> > >
> > > On Fri, 3 Apr 2026 at 4:49 PM, Prajwal Banakar <
> > prajwal.banakara@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi dev's,
> > > >
> > > > I am currently creating the Confluence page for this proposal.
> > > >
> > > > I noticed that FIP-35 is not currently listed on the wiki. Could you
> > > please
> > > > confirm if this number is available for use, or if I should assigned
> a
> > > > different one?
> > > >
> > > > Best regards,
> > > > Prajwal Banakar
> > > >
> > > > On Wed, 11 Mar 2026 at 22:56, Prajwal Banakar <
> > > prajwal.banakara@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Keith,
> > > > >
> > > > > Thank you for the follow-up.
> > > > >
> > > > > You are correct that FieldRoaringBitmap64Agg already exists in
> > > > > fluss-server. I have updated the proposal accordingly. To clarify,
> > the
> > > > > 32-bit scope is intended to keep the initial type system and SQL
> > > function
> > > > > surface focused and deliverable, rather than being a limitation of
> > the
> > > > > aggregator itself. Since the server-side aggregator is already in
> > > place,
> > > > > RBM64 will be a natural, low-risk follow-on once the type system
> and
> > > > > pushdown infrastructure are established.
> > > > >
> > > > > I have also removed the misleading motivation paragraph as you
> > > suggested.
> > > > > The updated document is available at the same link. Additionally, I
> > > would
> > > > > welcome Yang's input on the alignment with FIP-21.
> > > > >
> > > > > Best regards,
> > > > > Prajwal Banakar
> > > > >
> > > > > On Wed, 11 Mar 2026 at 17:37, Keith Lee <
> leekeiabstraction@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > >> Hi Prajwal,
> > > > >>
> > > > >> Thank you for addressing / answering the questions.
> > > > >>
> > > > >> > This proposal adds the missing bridge: a proper BITMAP DDL type,
> > SQL
> > > > >> functions (BITMAP_BUILD, BITMAP_OR_AGG, BITMAP_CARDINALITY), and
> > > > pushdown
> > > > >> via applyAggregates(). The storage-side aggregation logic already
> > > > exists;
> > > > >> this proposal makes it accessible end-to-end
> > > > >>
> > > > >> 1. That makes sense. I think the motivation section should lead
> with
> > > > that
> > > > >> and remove the following as it can be misleading given that rbm is
> > > > >> supported by aggregation merge engine: “users requiring
> > > high-cardinality
> > > > >> unique counting (e.g., UV analytics) must execute Client-Side
> > > > Aggregation.
> > > > >> The TabletServer is forced to send massive amounts of raw
> > > LogRecordBatch
> > > > >> rows over the network to a Flink cluster for evaluation. This
> > results
> > > in
> > > > >> unnecessary network transfer and prevents efficient utilization of
> > the
> > > > >> existing aggregation merge engine.”
> > > > >>
> > > > >> 2. That makes sense. Thank you for the context.
> > > > >>
> > > > >> 3.
> > > > >>
> > > > >> > RBM64 requires a fundamentally different internal structure; a
> map
> > > of
> > > > >> RBM32 chunks which increases implementation and serialization
> > > complexity
> > > > >> significantly.
> > > > >>
> > > > >> My understanding is that the proposal wires existing
> > > > >> FieldRoaringBitmap32Agg to support rbm32. FieldRoaringBitmap64Agg
> > > should
> > > > >> already exist and handle the complexity that you mentioned?
> > > > >>
> > > > >> Additionally, it might be good for Yang to review / provide input
> on
> > > > this
> > > > >> given his work on FIP-21.
> > > > >>
> > > > >> Best regards
> > > > >>
> > > > >> Keith Lee
> > > > >>
> > > > >>
> > > > >> On Wed, 11 Mar 2026 at 05:49, Prajwal Banakar <
> > > > prajwal.banakara@gmail.com
> > > > >> >
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Keith, thank you for the detailed feedback.
> > > > >> >
> > > > >> > 1. On motivation vs existing aggregation merge engine: The
> > > aggregation
> > > > >> > merge engine in 0.9 supports rbm32/rbm64 at the storage level,
> but
> > > > >> BITMAP
> > > > >> > is not yet a first-class type in the DDL or type system. Users
> > today
> > > > >> must
> > > > >> > declare the column as BYTES (as shown in the 0.9 release
> example:
> > > > >> uv_bitmap
> > > > >> > BYTES), and there are no SQL functions to build, merge, or query
> > > > bitmaps
> > > > >> > from Flink SQL. This proposal adds the missing bridge: a proper
> > > BITMAP
> > > > >> DDL
> > > > >> > type, SQL functions (BITMAP_BUILD, BITMAP_OR_AGG,
> > > BITMAP_CARDINALITY),
> > > > >> and
> > > > >> > pushdown via applyAggregates(). The storage-side aggregation
> logic
> > > > >> already
> > > > >> > exists; this proposal makes it accessible end-to-end.
> > > > >> >
> > > > >> > 2. On NULL semantics: BITMAP_OR(bitmap, NULL) returns NULL
> > following
> > > > >> > standard SQL scalar function semantics where NULL inputs
> propagate
> > > to
> > > > >> NULL
> > > > >> > outputs. BITMAP_OR_AGG follows aggregate function convention
> > > > consistent
> > > > >> > with how SUM and AVG behave, where NULLs in individual rows are
> > > > skipped
> > > > >> and
> > > > >> > only a fully NULL input set returns NULL. This distinction
> follows
> > > > >> FLIP-556
> > > > >> > and StarRocks semantics.
> > > > >> >
> > > > >> > 3. On 32-bit scope: The proposal is scoped to 32-bit initially
> > > because
> > > > >> > RoaringBitmap32 covers integer values up to 2^32 (~4 billion),
> > which
> > > > is
> > > > >> > sufficient for most user ID and session ID use cases. RBM64
> > > requires a
> > > > >> > fundamentally different internal structure; a map of RBM32
> chunks
> > > > which
> > > > >> > increases implementation and serialization complexity
> > significantly.
> > > > >> > Starting with 32-bit keeps the initial scope focused and
> > > deliverable.
> > > > >> RBM64
> > > > >> > support is listed as a Could-Have in the MoSCoW deliverables and
> > can
> > > > >> follow
> > > > >> > in a subsequent iteration.
> > > > >> >
> > > > >> > Best regards,
> > > > >> >
> > > > >> > Prajwal Banakar
> > > > >> >
> > > > >> >
> > > > >> > On Wed, 11 Mar 2026 at 01:34, Keith Lee <
> > > leekeiabstraction@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Hello Prajwal,
> > > > >> > >
> > > > >> > > Thank you for the detailed proposal. I enjoyed reading it and
> > > have a
> > > > >> few
> > > > >> > > questions/comments.
> > > > >> > >
> > > > >> > > 1. On motivation, can you provide context on how this differs
> > with
> > > > >> > > aggregation merge engine’s roaring bitmap implementation [1]?
> > > > >> > Specifically,
> > > > >> > > motivation part states that “users requiring high cardinality
> > > unique
> > > > >> > > counting … must execute client-side aggregation”. Aggregation
> > > merge
> > > > >> > engine
> > > > >> > > performs aggregation on server-side. The motivation section
> > should
> > > > >> > clarify
> > > > >> > > how the proposed changes improve or complement aggregation
> merge
> > > > >> engine,
> > > > >> > > which seems to have been considered as Section 2 references
> > FIP-21
> > > > >> > > Aggregation Merge Engine. Adding this context will help
> readers
> > > > >> > understand
> > > > >> > > the motivation of the proposal better.
> > > > >> > >
> > > > >> > > 2. Can you clarify the NULL semantics section specifically on
> > the
> > > > >> > decision
> > > > >> > > on why BITMAP_OR(bitmap, NULL) returns NULL but BITMAP_OR_AGG
> > only
> > > > >> > returns
> > > > >> > > null when all rows are NULL?
> > > > >> > >
> > > > >> > > 3. Why is the scope limited to 32 bit bitmaps? Adding the
> > > rationale
> > > > >> > behind
> > > > >> > > these e.g. how (if any) support of 64bit bitmaps would
> increase
> > > > >> > > implementation complexity. Articulating these may help other
> > > > >> contributors
> > > > >> > > understand the complexity and perhaps come up with suggestions
> > on
> > > > how
> > > > >> to
> > > > >> > > address them.
> > > > >> > >
> > > > >> > > Best regards
> > > > >> > >
> > > > >> > > Keith Lee
> > > > >> > >
> > > > >> > > [1]
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://fluss.apache.org/blog/releases/0.9/#2-storage-level-processing--semantics
> > > > >> > >
> > > > >> > >
> > > > >> > > On Mon, 9 Mar 2026 at 05:31, Prajwal Banakar <
> > > > >> prajwal.banakara@gmail.com
> > > > >> > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Hi Devs,
> > > > >> > > >
> > > > >> > > > I have pushed a working prototype to my public fork
> > > demonstrating
> > > > >> the
> > > > >> > > > BitmapType integrated with FieldRoaringBitmap32Agg. This
> > > includes
> > > > >> four
> > > > >> > > > passing unit tests.
> > > > >> > > >
> > > > >> > > > The link to the prototype is available in the Google Doc,
> and
> > > you
> > > > >> can
> > > > >> > > also
> > > > >> > > > find it here:
> > > > >> > > >
> > > > >>
> > https://github.com/Prajwal-banakar/fluss/tree/RoaringBitmap-prototype
> > > > >> > > >
> > > > >> > > > The Google Doc link remains the same. I look forward to your
> > > > >> feedback.
> > > > >> > > >
> > > > >> > > > Best regards,
> > > > >> > > >
> > > > >> > > > Prajwal Banakar
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Sun, 1 Mar, 2026, 11:49 am Prajwal Banakar, <
> > > > >> > > prajwal.banakara@gmail.com
> > > > >> > > > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi everyone,
> > > > >> > > > >
> > > > >> > > > > I would like to start a discussion on the proposal for
> > Native
> > > > >> Bitmap
> > > > >> > > > > Integration & Stateless Pushdown Aggregation.
> > > > >> > > > >
> > > > >> > > > > This proposal enables end-to-end native support for the
> > BITMAP
> > > > >> type
> > > > >> > in
> > > > >> > > > > Fluss and integrates it with the existing aggregation
> merge
> > > > >> engine to
> > > > >> > > > > support server-side bitmap union pushdown. The goal is to
> > > reduce
> > > > >> > > network
> > > > >> > > > > transfer and offload DISTINCT-style aggregation from Flink
> > to
> > > > the
> > > > >> > > > > TabletServer.
> > > > >> > > > >
> > > > >> > > > > Key highlights of the proposal include:
> > > > >> > > > >
> > > > >> > > > > - Type System: Promoting BITMAP to a first-class logical
> > type.
> > > > >> > > > > - UDF Suite: Introducing BITMAP_BUILD, BITMAP_OR_AGG, and
> > > > >> > > > > BITMAP_CARDINALITY (aligned with FLIP-556 and StarRocks
> > > > >> semantics).
> > > > >> > > > > - Optimizer: Planner-based pushdown via applyAggregates in
> > the
> > > > >> Flink
> > > > >> > > > > connector.
> > > > >> > > > > - Safety: No changes to LogRecordBatch or WAL, making this
> > > > >> strictly
> > > > >> > > > > additive and migration-free.
> > > > >> > > > >
> > > > >> > > > > You can find the full proposal document here:
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://docs.google.com/document/d/1sDhfkmo-w-UTvo2n3rsY1lytSSryswfkI83cSdka8s0/edit?usp=sharing
> > > > >> > > > >
> > > > >> > > > > I would appreciate feedback on the public interfaces,
> > pushdown
> > > > >> > > > > constraints, and overall scope.
> > > > >> > > > >
> > > > >> > > > > Best regards,
> > > > >> > > > > Prajwal Banakar
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Giannis Polyzos <ip...@gmail.com>]

Hi Prajwal,

This is a great proposal. I have been creating end-to-end real-time profile
use cases with Fluss features, and I had to create a few UDFs to better
interact with BM.
The FIP solves this structural problem as it makes BITMAP a first-class
type, promotes those same operations into the catalog so no UDF jar is
needed, and adds server-side pushdown so BITMAP_OR_AGG queries avoid
materializing per-row data in Flink entirely.

The overall direction is solid, the three-layer decomposition is clean, and
the backward compatibility story is well thought out. I have a few points I
would like to discuss before moving forward, mostly around design decisions
that I think are worth aligning on at the proposal stage.

1. DataTypeVisitor is @PublicStable but im wondering whether this can be a
breaking change. Adding a new method to this interface would likely break
any external implementors. It would be great if the proposal could settle
on an approach: a default fallback method, an abstract base class users can
extend, or a deferral to the next major version. Any of these would work.

2. Null semantics are a bit contradicted within the proposal itself. The
public interface section states BITMAP_CARDINALITY(NULL) → NULL, while the
reference implementation section returns 0 for null input. Could you
clarify which behavior is the intended one?

3. BITMAP_BUILD_AGG accepts only INT, while the server already supports
rbm64 for BIGINT. Given that most real-world entity IDs exceed
Integer.MAX_VALUE and the 64-bit aggregator already exists server-side, it
would be worth clarifying whether the initial scope is intentionally
limited to 32-bit and, if so, the reasoning behind that choice.

4. BITMAP_AND_AGG is included in the public interface but has no
server-side aggregator and no pushdown support. I'm reluctant that exposing
this in the public API without a working server-side counterpart could lead
to incorrect results when users combine it with the merge engine. Would it
be possible to add a clear section defining exactly when it is safe to use?

Some food for thought on my side is whether it makes sense (based on my
examples) to consider adding a function that converts a bitmap to a string
so it's human-readable for users debugging. This can be

For example, take a bitmap and output something like (bitmap_to_string??):
Output format: "count=3 [1001, 1002, 1007]."

This can be out of scope of this FIP, and more of a future improvement, if
it resonates.
I just wanted to bring this to your attention, just in case you think it
makes sense, because it's something that helped me a lot with my examples.

Looking forward to this

Best,
Giannis



On Mon, Apr 6, 2026 at 9:12 AM Prajwal Banakar <pr...@gmail.com>
wrote:

> Hi everyone,
>
> I have created the formal Confluence page for FIP-37 [1].
>
> As this is my first time creating a FIP page, I would greatly appreciate
> any feedback or suggestions for improvement. The discussion will continue
> on this thread.
>
> Looking forward to your thoughts.
> [1]
>
> https://cwiki.apache.org/confluence/display/FLUSS/FIP-37%3A+Native+RoaringBitmap+Integration+for+Apache+Fluss
>
> Best regards,
> Prajwal Banakar
>
> On Fri, 3 Apr 2026 at 19:22, Giannis Polyzos <ip...@gmail.com>
> wrote:
>
> > Hi Prajwal,
> > It’s probably a mistake that needs fixing.
> >
> > Feel free to use FIP-37
> >
> > Best,
> > Giannis
> >
> > On Fri, 3 Apr 2026 at 4:49 PM, Prajwal Banakar <
> prajwal.banakara@gmail.com
> > >
> > wrote:
> >
> > > Hi dev's,
> > >
> > > I am currently creating the Confluence page for this proposal.
> > >
> > > I noticed that FIP-35 is not currently listed on the wiki. Could you
> > please
> > > confirm if this number is available for use, or if I should assigned a
> > > different one?
> > >
> > > Best regards,
> > > Prajwal Banakar
> > >
> > > On Wed, 11 Mar 2026 at 22:56, Prajwal Banakar <
> > prajwal.banakara@gmail.com>
> > > wrote:
> > >
> > > > Hi Keith,
> > > >
> > > > Thank you for the follow-up.
> > > >
> > > > You are correct that FieldRoaringBitmap64Agg already exists in
> > > > fluss-server. I have updated the proposal accordingly. To clarify,
> the
> > > > 32-bit scope is intended to keep the initial type system and SQL
> > function
> > > > surface focused and deliverable, rather than being a limitation of
> the
> > > > aggregator itself. Since the server-side aggregator is already in
> > place,
> > > > RBM64 will be a natural, low-risk follow-on once the type system and
> > > > pushdown infrastructure are established.
> > > >
> > > > I have also removed the misleading motivation paragraph as you
> > suggested.
> > > > The updated document is available at the same link. Additionally, I
> > would
> > > > welcome Yang's input on the alignment with FIP-21.
> > > >
> > > > Best regards,
> > > > Prajwal Banakar
> > > >
> > > > On Wed, 11 Mar 2026 at 17:37, Keith Lee <leekeiabstraction@gmail.com
> >
> > > > wrote:
> > > >
> > > >> Hi Prajwal,
> > > >>
> > > >> Thank you for addressing / answering the questions.
> > > >>
> > > >> > This proposal adds the missing bridge: a proper BITMAP DDL type,
> SQL
> > > >> functions (BITMAP_BUILD, BITMAP_OR_AGG, BITMAP_CARDINALITY), and
> > > pushdown
> > > >> via applyAggregates(). The storage-side aggregation logic already
> > > exists;
> > > >> this proposal makes it accessible end-to-end
> > > >>
> > > >> 1. That makes sense. I think the motivation section should lead with
> > > that
> > > >> and remove the following as it can be misleading given that rbm is
> > > >> supported by aggregation merge engine: “users requiring
> > high-cardinality
> > > >> unique counting (e.g., UV analytics) must execute Client-Side
> > > Aggregation.
> > > >> The TabletServer is forced to send massive amounts of raw
> > LogRecordBatch
> > > >> rows over the network to a Flink cluster for evaluation. This
> results
> > in
> > > >> unnecessary network transfer and prevents efficient utilization of
> the
> > > >> existing aggregation merge engine.”
> > > >>
> > > >> 2. That makes sense. Thank you for the context.
> > > >>
> > > >> 3.
> > > >>
> > > >> > RBM64 requires a fundamentally different internal structure; a map
> > of
> > > >> RBM32 chunks which increases implementation and serialization
> > complexity
> > > >> significantly.
> > > >>
> > > >> My understanding is that the proposal wires existing
> > > >> FieldRoaringBitmap32Agg to support rbm32. FieldRoaringBitmap64Agg
> > should
> > > >> already exist and handle the complexity that you mentioned?
> > > >>
> > > >> Additionally, it might be good for Yang to review / provide input on
> > > this
> > > >> given his work on FIP-21.
> > > >>
> > > >> Best regards
> > > >>
> > > >> Keith Lee
> > > >>
> > > >>
> > > >> On Wed, 11 Mar 2026 at 05:49, Prajwal Banakar <
> > > prajwal.banakara@gmail.com
> > > >> >
> > > >> wrote:
> > > >>
> > > >> > Hi Keith, thank you for the detailed feedback.
> > > >> >
> > > >> > 1. On motivation vs existing aggregation merge engine: The
> > aggregation
> > > >> > merge engine in 0.9 supports rbm32/rbm64 at the storage level, but
> > > >> BITMAP
> > > >> > is not yet a first-class type in the DDL or type system. Users
> today
> > > >> must
> > > >> > declare the column as BYTES (as shown in the 0.9 release example:
> > > >> uv_bitmap
> > > >> > BYTES), and there are no SQL functions to build, merge, or query
> > > bitmaps
> > > >> > from Flink SQL. This proposal adds the missing bridge: a proper
> > BITMAP
> > > >> DDL
> > > >> > type, SQL functions (BITMAP_BUILD, BITMAP_OR_AGG,
> > BITMAP_CARDINALITY),
> > > >> and
> > > >> > pushdown via applyAggregates(). The storage-side aggregation logic
> > > >> already
> > > >> > exists; this proposal makes it accessible end-to-end.
> > > >> >
> > > >> > 2. On NULL semantics: BITMAP_OR(bitmap, NULL) returns NULL
> following
> > > >> > standard SQL scalar function semantics where NULL inputs propagate
> > to
> > > >> NULL
> > > >> > outputs. BITMAP_OR_AGG follows aggregate function convention
> > > consistent
> > > >> > with how SUM and AVG behave, where NULLs in individual rows are
> > > skipped
> > > >> and
> > > >> > only a fully NULL input set returns NULL. This distinction follows
> > > >> FLIP-556
> > > >> > and StarRocks semantics.
> > > >> >
> > > >> > 3. On 32-bit scope: The proposal is scoped to 32-bit initially
> > because
> > > >> > RoaringBitmap32 covers integer values up to 2^32 (~4 billion),
> which
> > > is
> > > >> > sufficient for most user ID and session ID use cases. RBM64
> > requires a
> > > >> > fundamentally different internal structure; a map of RBM32 chunks
> > > which
> > > >> > increases implementation and serialization complexity
> significantly.
> > > >> > Starting with 32-bit keeps the initial scope focused and
> > deliverable.
> > > >> RBM64
> > > >> > support is listed as a Could-Have in the MoSCoW deliverables and
> can
> > > >> follow
> > > >> > in a subsequent iteration.
> > > >> >
> > > >> > Best regards,
> > > >> >
> > > >> > Prajwal Banakar
> > > >> >
> > > >> >
> > > >> > On Wed, 11 Mar 2026 at 01:34, Keith Lee <
> > leekeiabstraction@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hello Prajwal,
> > > >> > >
> > > >> > > Thank you for the detailed proposal. I enjoyed reading it and
> > have a
> > > >> few
> > > >> > > questions/comments.
> > > >> > >
> > > >> > > 1. On motivation, can you provide context on how this differs
> with
> > > >> > > aggregation merge engine’s roaring bitmap implementation [1]?
> > > >> > Specifically,
> > > >> > > motivation part states that “users requiring high cardinality
> > unique
> > > >> > > counting … must execute client-side aggregation”. Aggregation
> > merge
> > > >> > engine
> > > >> > > performs aggregation on server-side. The motivation section
> should
> > > >> > clarify
> > > >> > > how the proposed changes improve or complement aggregation merge
> > > >> engine,
> > > >> > > which seems to have been considered as Section 2 references
> FIP-21
> > > >> > > Aggregation Merge Engine. Adding this context will help readers
> > > >> > understand
> > > >> > > the motivation of the proposal better.
> > > >> > >
> > > >> > > 2. Can you clarify the NULL semantics section specifically on
> the
> > > >> > decision
> > > >> > > on why BITMAP_OR(bitmap, NULL) returns NULL but BITMAP_OR_AGG
> only
> > > >> > returns
> > > >> > > null when all rows are NULL?
> > > >> > >
> > > >> > > 3. Why is the scope limited to 32 bit bitmaps? Adding the
> > rationale
> > > >> > behind
> > > >> > > these e.g. how (if any) support of 64bit bitmaps would increase
> > > >> > > implementation complexity. Articulating these may help other
> > > >> contributors
> > > >> > > understand the complexity and perhaps come up with suggestions
> on
> > > how
> > > >> to
> > > >> > > address them.
> > > >> > >
> > > >> > > Best regards
> > > >> > >
> > > >> > > Keith Lee
> > > >> > >
> > > >> > > [1]
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://fluss.apache.org/blog/releases/0.9/#2-storage-level-processing--semantics
> > > >> > >
> > > >> > >
> > > >> > > On Mon, 9 Mar 2026 at 05:31, Prajwal Banakar <
> > > >> prajwal.banakara@gmail.com
> > > >> > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hi Devs,
> > > >> > > >
> > > >> > > > I have pushed a working prototype to my public fork
> > demonstrating
> > > >> the
> > > >> > > > BitmapType integrated with FieldRoaringBitmap32Agg. This
> > includes
> > > >> four
> > > >> > > > passing unit tests.
> > > >> > > >
> > > >> > > > The link to the prototype is available in the Google Doc, and
> > you
> > > >> can
> > > >> > > also
> > > >> > > > find it here:
> > > >> > > >
> > > >>
> https://github.com/Prajwal-banakar/fluss/tree/RoaringBitmap-prototype
> > > >> > > >
> > > >> > > > The Google Doc link remains the same. I look forward to your
> > > >> feedback.
> > > >> > > >
> > > >> > > > Best regards,
> > > >> > > >
> > > >> > > > Prajwal Banakar
> > > >> > > >
> > > >> > > >
> > > >> > > > On Sun, 1 Mar, 2026, 11:49 am Prajwal Banakar, <
> > > >> > > prajwal.banakara@gmail.com
> > > >> > > > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi everyone,
> > > >> > > > >
> > > >> > > > > I would like to start a discussion on the proposal for
> Native
> > > >> Bitmap
> > > >> > > > > Integration & Stateless Pushdown Aggregation.
> > > >> > > > >
> > > >> > > > > This proposal enables end-to-end native support for the
> BITMAP
> > > >> type
> > > >> > in
> > > >> > > > > Fluss and integrates it with the existing aggregation merge
> > > >> engine to
> > > >> > > > > support server-side bitmap union pushdown. The goal is to
> > reduce
> > > >> > > network
> > > >> > > > > transfer and offload DISTINCT-style aggregation from Flink
> to
> > > the
> > > >> > > > > TabletServer.
> > > >> > > > >
> > > >> > > > > Key highlights of the proposal include:
> > > >> > > > >
> > > >> > > > > - Type System: Promoting BITMAP to a first-class logical
> type.
> > > >> > > > > - UDF Suite: Introducing BITMAP_BUILD, BITMAP_OR_AGG, and
> > > >> > > > > BITMAP_CARDINALITY (aligned with FLIP-556 and StarRocks
> > > >> semantics).
> > > >> > > > > - Optimizer: Planner-based pushdown via applyAggregates in
> the
> > > >> Flink
> > > >> > > > > connector.
> > > >> > > > > - Safety: No changes to LogRecordBatch or WAL, making this
> > > >> strictly
> > > >> > > > > additive and migration-free.
> > > >> > > > >
> > > >> > > > > You can find the full proposal document here:
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://docs.google.com/document/d/1sDhfkmo-w-UTvo2n3rsY1lytSSryswfkI83cSdka8s0/edit?usp=sharing
> > > >> > > > >
> > > >> > > > > I would appreciate feedback on the public interfaces,
> pushdown
> > > >> > > > > constraints, and overall scope.
> > > >> > > > >
> > > >> > > > > Best regards,
> > > >> > > > > Prajwal Banakar
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Prajwal Banakar <pr...@gmail.com>]

Hi everyone,

I have created the formal Confluence page for FIP-37 [1].

As this is my first time creating a FIP page, I would greatly appreciate
any feedback or suggestions for improvement. The discussion will continue
on this thread.

Looking forward to your thoughts.
[1]
https://cwiki.apache.org/confluence/display/FLUSS/FIP-37%3A+Native+RoaringBitmap+Integration+for+Apache+Fluss

Best regards,
Prajwal Banakar

On Fri, 3 Apr 2026 at 19:22, Giannis Polyzos <ip...@gmail.com> wrote:

> Hi Prajwal,
> It’s probably a mistake that needs fixing.
>
> Feel free to use FIP-37
>
> Best,
> Giannis
>
> On Fri, 3 Apr 2026 at 4:49 PM, Prajwal Banakar <prajwal.banakara@gmail.com
> >
> wrote:
>
> > Hi dev's,
> >
> > I am currently creating the Confluence page for this proposal.
> >
> > I noticed that FIP-35 is not currently listed on the wiki. Could you
> please
> > confirm if this number is available for use, or if I should assigned a
> > different one?
> >
> > Best regards,
> > Prajwal Banakar
> >
> > On Wed, 11 Mar 2026 at 22:56, Prajwal Banakar <
> prajwal.banakara@gmail.com>
> > wrote:
> >
> > > Hi Keith,
> > >
> > > Thank you for the follow-up.
> > >
> > > You are correct that FieldRoaringBitmap64Agg already exists in
> > > fluss-server. I have updated the proposal accordingly. To clarify, the
> > > 32-bit scope is intended to keep the initial type system and SQL
> function
> > > surface focused and deliverable, rather than being a limitation of the
> > > aggregator itself. Since the server-side aggregator is already in
> place,
> > > RBM64 will be a natural, low-risk follow-on once the type system and
> > > pushdown infrastructure are established.
> > >
> > > I have also removed the misleading motivation paragraph as you
> suggested.
> > > The updated document is available at the same link. Additionally, I
> would
> > > welcome Yang's input on the alignment with FIP-21.
> > >
> > > Best regards,
> > > Prajwal Banakar
> > >
> > > On Wed, 11 Mar 2026 at 17:37, Keith Lee <le...@gmail.com>
> > > wrote:
> > >
> > >> Hi Prajwal,
> > >>
> > >> Thank you for addressing / answering the questions.
> > >>
> > >> > This proposal adds the missing bridge: a proper BITMAP DDL type, SQL
> > >> functions (BITMAP_BUILD, BITMAP_OR_AGG, BITMAP_CARDINALITY), and
> > pushdown
> > >> via applyAggregates(). The storage-side aggregation logic already
> > exists;
> > >> this proposal makes it accessible end-to-end
> > >>
> > >> 1. That makes sense. I think the motivation section should lead with
> > that
> > >> and remove the following as it can be misleading given that rbm is
> > >> supported by aggregation merge engine: “users requiring
> high-cardinality
> > >> unique counting (e.g., UV analytics) must execute Client-Side
> > Aggregation.
> > >> The TabletServer is forced to send massive amounts of raw
> LogRecordBatch
> > >> rows over the network to a Flink cluster for evaluation. This results
> in
> > >> unnecessary network transfer and prevents efficient utilization of the
> > >> existing aggregation merge engine.”
> > >>
> > >> 2. That makes sense. Thank you for the context.
> > >>
> > >> 3.
> > >>
> > >> > RBM64 requires a fundamentally different internal structure; a map
> of
> > >> RBM32 chunks which increases implementation and serialization
> complexity
> > >> significantly.
> > >>
> > >> My understanding is that the proposal wires existing
> > >> FieldRoaringBitmap32Agg to support rbm32. FieldRoaringBitmap64Agg
> should
> > >> already exist and handle the complexity that you mentioned?
> > >>
> > >> Additionally, it might be good for Yang to review / provide input on
> > this
> > >> given his work on FIP-21.
> > >>
> > >> Best regards
> > >>
> > >> Keith Lee
> > >>
> > >>
> > >> On Wed, 11 Mar 2026 at 05:49, Prajwal Banakar <
> > prajwal.banakara@gmail.com
> > >> >
> > >> wrote:
> > >>
> > >> > Hi Keith, thank you for the detailed feedback.
> > >> >
> > >> > 1. On motivation vs existing aggregation merge engine: The
> aggregation
> > >> > merge engine in 0.9 supports rbm32/rbm64 at the storage level, but
> > >> BITMAP
> > >> > is not yet a first-class type in the DDL or type system. Users today
> > >> must
> > >> > declare the column as BYTES (as shown in the 0.9 release example:
> > >> uv_bitmap
> > >> > BYTES), and there are no SQL functions to build, merge, or query
> > bitmaps
> > >> > from Flink SQL. This proposal adds the missing bridge: a proper
> BITMAP
> > >> DDL
> > >> > type, SQL functions (BITMAP_BUILD, BITMAP_OR_AGG,
> BITMAP_CARDINALITY),
> > >> and
> > >> > pushdown via applyAggregates(). The storage-side aggregation logic
> > >> already
> > >> > exists; this proposal makes it accessible end-to-end.
> > >> >
> > >> > 2. On NULL semantics: BITMAP_OR(bitmap, NULL) returns NULL following
> > >> > standard SQL scalar function semantics where NULL inputs propagate
> to
> > >> NULL
> > >> > outputs. BITMAP_OR_AGG follows aggregate function convention
> > consistent
> > >> > with how SUM and AVG behave, where NULLs in individual rows are
> > skipped
> > >> and
> > >> > only a fully NULL input set returns NULL. This distinction follows
> > >> FLIP-556
> > >> > and StarRocks semantics.
> > >> >
> > >> > 3. On 32-bit scope: The proposal is scoped to 32-bit initially
> because
> > >> > RoaringBitmap32 covers integer values up to 2^32 (~4 billion), which
> > is
> > >> > sufficient for most user ID and session ID use cases. RBM64
> requires a
> > >> > fundamentally different internal structure; a map of RBM32 chunks
> > which
> > >> > increases implementation and serialization complexity significantly.
> > >> > Starting with 32-bit keeps the initial scope focused and
> deliverable.
> > >> RBM64
> > >> > support is listed as a Could-Have in the MoSCoW deliverables and can
> > >> follow
> > >> > in a subsequent iteration.
> > >> >
> > >> > Best regards,
> > >> >
> > >> > Prajwal Banakar
> > >> >
> > >> >
> > >> > On Wed, 11 Mar 2026 at 01:34, Keith Lee <
> leekeiabstraction@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Hello Prajwal,
> > >> > >
> > >> > > Thank you for the detailed proposal. I enjoyed reading it and
> have a
> > >> few
> > >> > > questions/comments.
> > >> > >
> > >> > > 1. On motivation, can you provide context on how this differs with
> > >> > > aggregation merge engine’s roaring bitmap implementation [1]?
> > >> > Specifically,
> > >> > > motivation part states that “users requiring high cardinality
> unique
> > >> > > counting … must execute client-side aggregation”. Aggregation
> merge
> > >> > engine
> > >> > > performs aggregation on server-side. The motivation section should
> > >> > clarify
> > >> > > how the proposed changes improve or complement aggregation merge
> > >> engine,
> > >> > > which seems to have been considered as Section 2 references FIP-21
> > >> > > Aggregation Merge Engine. Adding this context will help readers
> > >> > understand
> > >> > > the motivation of the proposal better.
> > >> > >
> > >> > > 2. Can you clarify the NULL semantics section specifically on the
> > >> > decision
> > >> > > on why BITMAP_OR(bitmap, NULL) returns NULL but BITMAP_OR_AGG only
> > >> > returns
> > >> > > null when all rows are NULL?
> > >> > >
> > >> > > 3. Why is the scope limited to 32 bit bitmaps? Adding the
> rationale
> > >> > behind
> > >> > > these e.g. how (if any) support of 64bit bitmaps would increase
> > >> > > implementation complexity. Articulating these may help other
> > >> contributors
> > >> > > understand the complexity and perhaps come up with suggestions on
> > how
> > >> to
> > >> > > address them.
> > >> > >
> > >> > > Best regards
> > >> > >
> > >> > > Keith Lee
> > >> > >
> > >> > > [1]
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://fluss.apache.org/blog/releases/0.9/#2-storage-level-processing--semantics
> > >> > >
> > >> > >
> > >> > > On Mon, 9 Mar 2026 at 05:31, Prajwal Banakar <
> > >> prajwal.banakara@gmail.com
> > >> > >
> > >> > > wrote:
> > >> > >
> > >> > > > Hi Devs,
> > >> > > >
> > >> > > > I have pushed a working prototype to my public fork
> demonstrating
> > >> the
> > >> > > > BitmapType integrated with FieldRoaringBitmap32Agg. This
> includes
> > >> four
> > >> > > > passing unit tests.
> > >> > > >
> > >> > > > The link to the prototype is available in the Google Doc, and
> you
> > >> can
> > >> > > also
> > >> > > > find it here:
> > >> > > >
> > >> https://github.com/Prajwal-banakar/fluss/tree/RoaringBitmap-prototype
> > >> > > >
> > >> > > > The Google Doc link remains the same. I look forward to your
> > >> feedback.
> > >> > > >
> > >> > > > Best regards,
> > >> > > >
> > >> > > > Prajwal Banakar
> > >> > > >
> > >> > > >
> > >> > > > On Sun, 1 Mar, 2026, 11:49 am Prajwal Banakar, <
> > >> > > prajwal.banakara@gmail.com
> > >> > > > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi everyone,
> > >> > > > >
> > >> > > > > I would like to start a discussion on the proposal for Native
> > >> Bitmap
> > >> > > > > Integration & Stateless Pushdown Aggregation.
> > >> > > > >
> > >> > > > > This proposal enables end-to-end native support for the BITMAP
> > >> type
> > >> > in
> > >> > > > > Fluss and integrates it with the existing aggregation merge
> > >> engine to
> > >> > > > > support server-side bitmap union pushdown. The goal is to
> reduce
> > >> > > network
> > >> > > > > transfer and offload DISTINCT-style aggregation from Flink to
> > the
> > >> > > > > TabletServer.
> > >> > > > >
> > >> > > > > Key highlights of the proposal include:
> > >> > > > >
> > >> > > > > - Type System: Promoting BITMAP to a first-class logical type.
> > >> > > > > - UDF Suite: Introducing BITMAP_BUILD, BITMAP_OR_AGG, and
> > >> > > > > BITMAP_CARDINALITY (aligned with FLIP-556 and StarRocks
> > >> semantics).
> > >> > > > > - Optimizer: Planner-based pushdown via applyAggregates in the
> > >> Flink
> > >> > > > > connector.
> > >> > > > > - Safety: No changes to LogRecordBatch or WAL, making this
> > >> strictly
> > >> > > > > additive and migration-free.
> > >> > > > >
> > >> > > > > You can find the full proposal document here:
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://docs.google.com/document/d/1sDhfkmo-w-UTvo2n3rsY1lytSSryswfkI83cSdka8s0/edit?usp=sharing
> > >> > > > >
> > >> > > > > I would appreciate feedback on the public interfaces, pushdown
> > >> > > > > constraints, and overall scope.
> > >> > > > >
> > >> > > > > Best regards,
> > >> > > > > Prajwal Banakar
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Giannis Polyzos <ip...@gmail.com>]

Hi Prajwal,
It’s probably a mistake that needs fixing.

Feel free to use FIP-37

Best,
Giannis

On Fri, 3 Apr 2026 at 4:49 PM, Prajwal Banakar <pr...@gmail.com>
wrote:

> Hi dev's,
>
> I am currently creating the Confluence page for this proposal.
>
> I noticed that FIP-35 is not currently listed on the wiki. Could you please
> confirm if this number is available for use, or if I should assigned a
> different one?
>
> Best regards,
> Prajwal Banakar
>
> On Wed, 11 Mar 2026 at 22:56, Prajwal Banakar <pr...@gmail.com>
> wrote:
>
> > Hi Keith,
> >
> > Thank you for the follow-up.
> >
> > You are correct that FieldRoaringBitmap64Agg already exists in
> > fluss-server. I have updated the proposal accordingly. To clarify, the
> > 32-bit scope is intended to keep the initial type system and SQL function
> > surface focused and deliverable, rather than being a limitation of the
> > aggregator itself. Since the server-side aggregator is already in place,
> > RBM64 will be a natural, low-risk follow-on once the type system and
> > pushdown infrastructure are established.
> >
> > I have also removed the misleading motivation paragraph as you suggested.
> > The updated document is available at the same link. Additionally, I would
> > welcome Yang's input on the alignment with FIP-21.
> >
> > Best regards,
> > Prajwal Banakar
> >
> > On Wed, 11 Mar 2026 at 17:37, Keith Lee <le...@gmail.com>
> > wrote:
> >
> >> Hi Prajwal,
> >>
> >> Thank you for addressing / answering the questions.
> >>
> >> > This proposal adds the missing bridge: a proper BITMAP DDL type, SQL
> >> functions (BITMAP_BUILD, BITMAP_OR_AGG, BITMAP_CARDINALITY), and
> pushdown
> >> via applyAggregates(). The storage-side aggregation logic already
> exists;
> >> this proposal makes it accessible end-to-end
> >>
> >> 1. That makes sense. I think the motivation section should lead with
> that
> >> and remove the following as it can be misleading given that rbm is
> >> supported by aggregation merge engine: “users requiring high-cardinality
> >> unique counting (e.g., UV analytics) must execute Client-Side
> Aggregation.
> >> The TabletServer is forced to send massive amounts of raw LogRecordBatch
> >> rows over the network to a Flink cluster for evaluation. This results in
> >> unnecessary network transfer and prevents efficient utilization of the
> >> existing aggregation merge engine.”
> >>
> >> 2. That makes sense. Thank you for the context.
> >>
> >> 3.
> >>
> >> > RBM64 requires a fundamentally different internal structure; a map of
> >> RBM32 chunks which increases implementation and serialization complexity
> >> significantly.
> >>
> >> My understanding is that the proposal wires existing
> >> FieldRoaringBitmap32Agg to support rbm32. FieldRoaringBitmap64Agg should
> >> already exist and handle the complexity that you mentioned?
> >>
> >> Additionally, it might be good for Yang to review / provide input on
> this
> >> given his work on FIP-21.
> >>
> >> Best regards
> >>
> >> Keith Lee
> >>
> >>
> >> On Wed, 11 Mar 2026 at 05:49, Prajwal Banakar <
> prajwal.banakara@gmail.com
> >> >
> >> wrote:
> >>
> >> > Hi Keith, thank you for the detailed feedback.
> >> >
> >> > 1. On motivation vs existing aggregation merge engine: The aggregation
> >> > merge engine in 0.9 supports rbm32/rbm64 at the storage level, but
> >> BITMAP
> >> > is not yet a first-class type in the DDL or type system. Users today
> >> must
> >> > declare the column as BYTES (as shown in the 0.9 release example:
> >> uv_bitmap
> >> > BYTES), and there are no SQL functions to build, merge, or query
> bitmaps
> >> > from Flink SQL. This proposal adds the missing bridge: a proper BITMAP
> >> DDL
> >> > type, SQL functions (BITMAP_BUILD, BITMAP_OR_AGG, BITMAP_CARDINALITY),
> >> and
> >> > pushdown via applyAggregates(). The storage-side aggregation logic
> >> already
> >> > exists; this proposal makes it accessible end-to-end.
> >> >
> >> > 2. On NULL semantics: BITMAP_OR(bitmap, NULL) returns NULL following
> >> > standard SQL scalar function semantics where NULL inputs propagate to
> >> NULL
> >> > outputs. BITMAP_OR_AGG follows aggregate function convention
> consistent
> >> > with how SUM and AVG behave, where NULLs in individual rows are
> skipped
> >> and
> >> > only a fully NULL input set returns NULL. This distinction follows
> >> FLIP-556
> >> > and StarRocks semantics.
> >> >
> >> > 3. On 32-bit scope: The proposal is scoped to 32-bit initially because
> >> > RoaringBitmap32 covers integer values up to 2^32 (~4 billion), which
> is
> >> > sufficient for most user ID and session ID use cases. RBM64 requires a
> >> > fundamentally different internal structure; a map of RBM32 chunks
> which
> >> > increases implementation and serialization complexity significantly.
> >> > Starting with 32-bit keeps the initial scope focused and deliverable.
> >> RBM64
> >> > support is listed as a Could-Have in the MoSCoW deliverables and can
> >> follow
> >> > in a subsequent iteration.
> >> >
> >> > Best regards,
> >> >
> >> > Prajwal Banakar
> >> >
> >> >
> >> > On Wed, 11 Mar 2026 at 01:34, Keith Lee <le...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hello Prajwal,
> >> > >
> >> > > Thank you for the detailed proposal. I enjoyed reading it and have a
> >> few
> >> > > questions/comments.
> >> > >
> >> > > 1. On motivation, can you provide context on how this differs with
> >> > > aggregation merge engine’s roaring bitmap implementation [1]?
> >> > Specifically,
> >> > > motivation part states that “users requiring high cardinality unique
> >> > > counting … must execute client-side aggregation”. Aggregation merge
> >> > engine
> >> > > performs aggregation on server-side. The motivation section should
> >> > clarify
> >> > > how the proposed changes improve or complement aggregation merge
> >> engine,
> >> > > which seems to have been considered as Section 2 references FIP-21
> >> > > Aggregation Merge Engine. Adding this context will help readers
> >> > understand
> >> > > the motivation of the proposal better.
> >> > >
> >> > > 2. Can you clarify the NULL semantics section specifically on the
> >> > decision
> >> > > on why BITMAP_OR(bitmap, NULL) returns NULL but BITMAP_OR_AGG only
> >> > returns
> >> > > null when all rows are NULL?
> >> > >
> >> > > 3. Why is the scope limited to 32 bit bitmaps? Adding the rationale
> >> > behind
> >> > > these e.g. how (if any) support of 64bit bitmaps would increase
> >> > > implementation complexity. Articulating these may help other
> >> contributors
> >> > > understand the complexity and perhaps come up with suggestions on
> how
> >> to
> >> > > address them.
> >> > >
> >> > > Best regards
> >> > >
> >> > > Keith Lee
> >> > >
> >> > > [1]
> >> > >
> >> > >
> >> >
> >>
> https://fluss.apache.org/blog/releases/0.9/#2-storage-level-processing--semantics
> >> > >
> >> > >
> >> > > On Mon, 9 Mar 2026 at 05:31, Prajwal Banakar <
> >> prajwal.banakara@gmail.com
> >> > >
> >> > > wrote:
> >> > >
> >> > > > Hi Devs,
> >> > > >
> >> > > > I have pushed a working prototype to my public fork demonstrating
> >> the
> >> > > > BitmapType integrated with FieldRoaringBitmap32Agg. This includes
> >> four
> >> > > > passing unit tests.
> >> > > >
> >> > > > The link to the prototype is available in the Google Doc, and you
> >> can
> >> > > also
> >> > > > find it here:
> >> > > >
> >> https://github.com/Prajwal-banakar/fluss/tree/RoaringBitmap-prototype
> >> > > >
> >> > > > The Google Doc link remains the same. I look forward to your
> >> feedback.
> >> > > >
> >> > > > Best regards,
> >> > > >
> >> > > > Prajwal Banakar
> >> > > >
> >> > > >
> >> > > > On Sun, 1 Mar, 2026, 11:49 am Prajwal Banakar, <
> >> > > prajwal.banakara@gmail.com
> >> > > > >
> >> > > > wrote:
> >> > > >
> >> > > > > Hi everyone,
> >> > > > >
> >> > > > > I would like to start a discussion on the proposal for Native
> >> Bitmap
> >> > > > > Integration & Stateless Pushdown Aggregation.
> >> > > > >
> >> > > > > This proposal enables end-to-end native support for the BITMAP
> >> type
> >> > in
> >> > > > > Fluss and integrates it with the existing aggregation merge
> >> engine to
> >> > > > > support server-side bitmap union pushdown. The goal is to reduce
> >> > > network
> >> > > > > transfer and offload DISTINCT-style aggregation from Flink to
> the
> >> > > > > TabletServer.
> >> > > > >
> >> > > > > Key highlights of the proposal include:
> >> > > > >
> >> > > > > - Type System: Promoting BITMAP to a first-class logical type.
> >> > > > > - UDF Suite: Introducing BITMAP_BUILD, BITMAP_OR_AGG, and
> >> > > > > BITMAP_CARDINALITY (aligned with FLIP-556 and StarRocks
> >> semantics).
> >> > > > > - Optimizer: Planner-based pushdown via applyAggregates in the
> >> Flink
> >> > > > > connector.
> >> > > > > - Safety: No changes to LogRecordBatch or WAL, making this
> >> strictly
> >> > > > > additive and migration-free.
> >> > > > >
> >> > > > > You can find the full proposal document here:
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://docs.google.com/document/d/1sDhfkmo-w-UTvo2n3rsY1lytSSryswfkI83cSdka8s0/edit?usp=sharing
> >> > > > >
> >> > > > > I would appreciate feedback on the public interfaces, pushdown
> >> > > > > constraints, and overall scope.
> >> > > > >
> >> > > > > Best regards,
> >> > > > > Prajwal Banakar
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Prajwal Banakar <pr...@gmail.com>]

Hi dev's,

I am currently creating the Confluence page for this proposal.

I noticed that FIP-35 is not currently listed on the wiki. Could you please
confirm if this number is available for use, or if I should assigned a
different one?

Best regards,
Prajwal Banakar

On Wed, 11 Mar 2026 at 22:56, Prajwal Banakar <pr...@gmail.com>
wrote:

> Hi Keith,
>
> Thank you for the follow-up.
>
> You are correct that FieldRoaringBitmap64Agg already exists in
> fluss-server. I have updated the proposal accordingly. To clarify, the
> 32-bit scope is intended to keep the initial type system and SQL function
> surface focused and deliverable, rather than being a limitation of the
> aggregator itself. Since the server-side aggregator is already in place,
> RBM64 will be a natural, low-risk follow-on once the type system and
> pushdown infrastructure are established.
>
> I have also removed the misleading motivation paragraph as you suggested.
> The updated document is available at the same link. Additionally, I would
> welcome Yang's input on the alignment with FIP-21.
>
> Best regards,
> Prajwal Banakar
>
> On Wed, 11 Mar 2026 at 17:37, Keith Lee <le...@gmail.com>
> wrote:
>
>> Hi Prajwal,
>>
>> Thank you for addressing / answering the questions.
>>
>> > This proposal adds the missing bridge: a proper BITMAP DDL type, SQL
>> functions (BITMAP_BUILD, BITMAP_OR_AGG, BITMAP_CARDINALITY), and pushdown
>> via applyAggregates(). The storage-side aggregation logic already exists;
>> this proposal makes it accessible end-to-end
>>
>> 1. That makes sense. I think the motivation section should lead with that
>> and remove the following as it can be misleading given that rbm is
>> supported by aggregation merge engine: “users requiring high-cardinality
>> unique counting (e.g., UV analytics) must execute Client-Side Aggregation.
>> The TabletServer is forced to send massive amounts of raw LogRecordBatch
>> rows over the network to a Flink cluster for evaluation. This results in
>> unnecessary network transfer and prevents efficient utilization of the
>> existing aggregation merge engine.”
>>
>> 2. That makes sense. Thank you for the context.
>>
>> 3.
>>
>> > RBM64 requires a fundamentally different internal structure; a map of
>> RBM32 chunks which increases implementation and serialization complexity
>> significantly.
>>
>> My understanding is that the proposal wires existing
>> FieldRoaringBitmap32Agg to support rbm32. FieldRoaringBitmap64Agg should
>> already exist and handle the complexity that you mentioned?
>>
>> Additionally, it might be good for Yang to review / provide input on this
>> given his work on FIP-21.
>>
>> Best regards
>>
>> Keith Lee
>>
>>
>> On Wed, 11 Mar 2026 at 05:49, Prajwal Banakar <prajwal.banakara@gmail.com
>> >
>> wrote:
>>
>> > Hi Keith, thank you for the detailed feedback.
>> >
>> > 1. On motivation vs existing aggregation merge engine: The aggregation
>> > merge engine in 0.9 supports rbm32/rbm64 at the storage level, but
>> BITMAP
>> > is not yet a first-class type in the DDL or type system. Users today
>> must
>> > declare the column as BYTES (as shown in the 0.9 release example:
>> uv_bitmap
>> > BYTES), and there are no SQL functions to build, merge, or query bitmaps
>> > from Flink SQL. This proposal adds the missing bridge: a proper BITMAP
>> DDL
>> > type, SQL functions (BITMAP_BUILD, BITMAP_OR_AGG, BITMAP_CARDINALITY),
>> and
>> > pushdown via applyAggregates(). The storage-side aggregation logic
>> already
>> > exists; this proposal makes it accessible end-to-end.
>> >
>> > 2. On NULL semantics: BITMAP_OR(bitmap, NULL) returns NULL following
>> > standard SQL scalar function semantics where NULL inputs propagate to
>> NULL
>> > outputs. BITMAP_OR_AGG follows aggregate function convention consistent
>> > with how SUM and AVG behave, where NULLs in individual rows are skipped
>> and
>> > only a fully NULL input set returns NULL. This distinction follows
>> FLIP-556
>> > and StarRocks semantics.
>> >
>> > 3. On 32-bit scope: The proposal is scoped to 32-bit initially because
>> > RoaringBitmap32 covers integer values up to 2^32 (~4 billion), which is
>> > sufficient for most user ID and session ID use cases. RBM64 requires a
>> > fundamentally different internal structure; a map of RBM32 chunks which
>> > increases implementation and serialization complexity significantly.
>> > Starting with 32-bit keeps the initial scope focused and deliverable.
>> RBM64
>> > support is listed as a Could-Have in the MoSCoW deliverables and can
>> follow
>> > in a subsequent iteration.
>> >
>> > Best regards,
>> >
>> > Prajwal Banakar
>> >
>> >
>> > On Wed, 11 Mar 2026 at 01:34, Keith Lee <le...@gmail.com>
>> > wrote:
>> >
>> > > Hello Prajwal,
>> > >
>> > > Thank you for the detailed proposal. I enjoyed reading it and have a
>> few
>> > > questions/comments.
>> > >
>> > > 1. On motivation, can you provide context on how this differs with
>> > > aggregation merge engine’s roaring bitmap implementation [1]?
>> > Specifically,
>> > > motivation part states that “users requiring high cardinality unique
>> > > counting … must execute client-side aggregation”. Aggregation merge
>> > engine
>> > > performs aggregation on server-side. The motivation section should
>> > clarify
>> > > how the proposed changes improve or complement aggregation merge
>> engine,
>> > > which seems to have been considered as Section 2 references FIP-21
>> > > Aggregation Merge Engine. Adding this context will help readers
>> > understand
>> > > the motivation of the proposal better.
>> > >
>> > > 2. Can you clarify the NULL semantics section specifically on the
>> > decision
>> > > on why BITMAP_OR(bitmap, NULL) returns NULL but BITMAP_OR_AGG only
>> > returns
>> > > null when all rows are NULL?
>> > >
>> > > 3. Why is the scope limited to 32 bit bitmaps? Adding the rationale
>> > behind
>> > > these e.g. how (if any) support of 64bit bitmaps would increase
>> > > implementation complexity. Articulating these may help other
>> contributors
>> > > understand the complexity and perhaps come up with suggestions on how
>> to
>> > > address them.
>> > >
>> > > Best regards
>> > >
>> > > Keith Lee
>> > >
>> > > [1]
>> > >
>> > >
>> >
>> https://fluss.apache.org/blog/releases/0.9/#2-storage-level-processing--semantics
>> > >
>> > >
>> > > On Mon, 9 Mar 2026 at 05:31, Prajwal Banakar <
>> prajwal.banakara@gmail.com
>> > >
>> > > wrote:
>> > >
>> > > > Hi Devs,
>> > > >
>> > > > I have pushed a working prototype to my public fork demonstrating
>> the
>> > > > BitmapType integrated with FieldRoaringBitmap32Agg. This includes
>> four
>> > > > passing unit tests.
>> > > >
>> > > > The link to the prototype is available in the Google Doc, and you
>> can
>> > > also
>> > > > find it here:
>> > > >
>> https://github.com/Prajwal-banakar/fluss/tree/RoaringBitmap-prototype
>> > > >
>> > > > The Google Doc link remains the same. I look forward to your
>> feedback.
>> > > >
>> > > > Best regards,
>> > > >
>> > > > Prajwal Banakar
>> > > >
>> > > >
>> > > > On Sun, 1 Mar, 2026, 11:49 am Prajwal Banakar, <
>> > > prajwal.banakara@gmail.com
>> > > > >
>> > > > wrote:
>> > > >
>> > > > > Hi everyone,
>> > > > >
>> > > > > I would like to start a discussion on the proposal for Native
>> Bitmap
>> > > > > Integration & Stateless Pushdown Aggregation.
>> > > > >
>> > > > > This proposal enables end-to-end native support for the BITMAP
>> type
>> > in
>> > > > > Fluss and integrates it with the existing aggregation merge
>> engine to
>> > > > > support server-side bitmap union pushdown. The goal is to reduce
>> > > network
>> > > > > transfer and offload DISTINCT-style aggregation from Flink to the
>> > > > > TabletServer.
>> > > > >
>> > > > > Key highlights of the proposal include:
>> > > > >
>> > > > > - Type System: Promoting BITMAP to a first-class logical type.
>> > > > > - UDF Suite: Introducing BITMAP_BUILD, BITMAP_OR_AGG, and
>> > > > > BITMAP_CARDINALITY (aligned with FLIP-556 and StarRocks
>> semantics).
>> > > > > - Optimizer: Planner-based pushdown via applyAggregates in the
>> Flink
>> > > > > connector.
>> > > > > - Safety: No changes to LogRecordBatch or WAL, making this
>> strictly
>> > > > > additive and migration-free.
>> > > > >
>> > > > > You can find the full proposal document here:
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1sDhfkmo-w-UTvo2n3rsY1lytSSryswfkI83cSdka8s0/edit?usp=sharing
>> > > > >
>> > > > > I would appreciate feedback on the public interfaces, pushdown
>> > > > > constraints, and overall scope.
>> > > > >
>> > > > > Best regards,
>> > > > > Prajwal Banakar
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Prajwal Banakar <pr...@gmail.com>]

Hi Keith,

Thank you for the follow-up.

You are correct that FieldRoaringBitmap64Agg already exists in
fluss-server. I have updated the proposal accordingly. To clarify, the
32-bit scope is intended to keep the initial type system and SQL function
surface focused and deliverable, rather than being a limitation of the
aggregator itself. Since the server-side aggregator is already in place,
RBM64 will be a natural, low-risk follow-on once the type system and
pushdown infrastructure are established.

I have also removed the misleading motivation paragraph as you suggested.
The updated document is available at the same link. Additionally, I would
welcome Yang's input on the alignment with FIP-21.

Best regards,
Prajwal Banakar

On Wed, 11 Mar 2026 at 17:37, Keith Lee <le...@gmail.com> wrote:

> Hi Prajwal,
>
> Thank you for addressing / answering the questions.
>
> > This proposal adds the missing bridge: a proper BITMAP DDL type, SQL
> functions (BITMAP_BUILD, BITMAP_OR_AGG, BITMAP_CARDINALITY), and pushdown
> via applyAggregates(). The storage-side aggregation logic already exists;
> this proposal makes it accessible end-to-end
>
> 1. That makes sense. I think the motivation section should lead with that
> and remove the following as it can be misleading given that rbm is
> supported by aggregation merge engine: “users requiring high-cardinality
> unique counting (e.g., UV analytics) must execute Client-Side Aggregation.
> The TabletServer is forced to send massive amounts of raw LogRecordBatch
> rows over the network to a Flink cluster for evaluation. This results in
> unnecessary network transfer and prevents efficient utilization of the
> existing aggregation merge engine.”
>
> 2. That makes sense. Thank you for the context.
>
> 3.
>
> > RBM64 requires a fundamentally different internal structure; a map of
> RBM32 chunks which increases implementation and serialization complexity
> significantly.
>
> My understanding is that the proposal wires existing
> FieldRoaringBitmap32Agg to support rbm32. FieldRoaringBitmap64Agg should
> already exist and handle the complexity that you mentioned?
>
> Additionally, it might be good for Yang to review / provide input on this
> given his work on FIP-21.
>
> Best regards
>
> Keith Lee
>
>
> On Wed, 11 Mar 2026 at 05:49, Prajwal Banakar <pr...@gmail.com>
> wrote:
>
> > Hi Keith, thank you for the detailed feedback.
> >
> > 1. On motivation vs existing aggregation merge engine: The aggregation
> > merge engine in 0.9 supports rbm32/rbm64 at the storage level, but BITMAP
> > is not yet a first-class type in the DDL or type system. Users today must
> > declare the column as BYTES (as shown in the 0.9 release example:
> uv_bitmap
> > BYTES), and there are no SQL functions to build, merge, or query bitmaps
> > from Flink SQL. This proposal adds the missing bridge: a proper BITMAP
> DDL
> > type, SQL functions (BITMAP_BUILD, BITMAP_OR_AGG, BITMAP_CARDINALITY),
> and
> > pushdown via applyAggregates(). The storage-side aggregation logic
> already
> > exists; this proposal makes it accessible end-to-end.
> >
> > 2. On NULL semantics: BITMAP_OR(bitmap, NULL) returns NULL following
> > standard SQL scalar function semantics where NULL inputs propagate to
> NULL
> > outputs. BITMAP_OR_AGG follows aggregate function convention consistent
> > with how SUM and AVG behave, where NULLs in individual rows are skipped
> and
> > only a fully NULL input set returns NULL. This distinction follows
> FLIP-556
> > and StarRocks semantics.
> >
> > 3. On 32-bit scope: The proposal is scoped to 32-bit initially because
> > RoaringBitmap32 covers integer values up to 2^32 (~4 billion), which is
> > sufficient for most user ID and session ID use cases. RBM64 requires a
> > fundamentally different internal structure; a map of RBM32 chunks which
> > increases implementation and serialization complexity significantly.
> > Starting with 32-bit keeps the initial scope focused and deliverable.
> RBM64
> > support is listed as a Could-Have in the MoSCoW deliverables and can
> follow
> > in a subsequent iteration.
> >
> > Best regards,
> >
> > Prajwal Banakar
> >
> >
> > On Wed, 11 Mar 2026 at 01:34, Keith Lee <le...@gmail.com>
> > wrote:
> >
> > > Hello Prajwal,
> > >
> > > Thank you for the detailed proposal. I enjoyed reading it and have a
> few
> > > questions/comments.
> > >
> > > 1. On motivation, can you provide context on how this differs with
> > > aggregation merge engine’s roaring bitmap implementation [1]?
> > Specifically,
> > > motivation part states that “users requiring high cardinality unique
> > > counting … must execute client-side aggregation”. Aggregation merge
> > engine
> > > performs aggregation on server-side. The motivation section should
> > clarify
> > > how the proposed changes improve or complement aggregation merge
> engine,
> > > which seems to have been considered as Section 2 references FIP-21
> > > Aggregation Merge Engine. Adding this context will help readers
> > understand
> > > the motivation of the proposal better.
> > >
> > > 2. Can you clarify the NULL semantics section specifically on the
> > decision
> > > on why BITMAP_OR(bitmap, NULL) returns NULL but BITMAP_OR_AGG only
> > returns
> > > null when all rows are NULL?
> > >
> > > 3. Why is the scope limited to 32 bit bitmaps? Adding the rationale
> > behind
> > > these e.g. how (if any) support of 64bit bitmaps would increase
> > > implementation complexity. Articulating these may help other
> contributors
> > > understand the complexity and perhaps come up with suggestions on how
> to
> > > address them.
> > >
> > > Best regards
> > >
> > > Keith Lee
> > >
> > > [1]
> > >
> > >
> >
> https://fluss.apache.org/blog/releases/0.9/#2-storage-level-processing--semantics
> > >
> > >
> > > On Mon, 9 Mar 2026 at 05:31, Prajwal Banakar <
> prajwal.banakara@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Devs,
> > > >
> > > > I have pushed a working prototype to my public fork demonstrating the
> > > > BitmapType integrated with FieldRoaringBitmap32Agg. This includes
> four
> > > > passing unit tests.
> > > >
> > > > The link to the prototype is available in the Google Doc, and you can
> > > also
> > > > find it here:
> > > >
> https://github.com/Prajwal-banakar/fluss/tree/RoaringBitmap-prototype
> > > >
> > > > The Google Doc link remains the same. I look forward to your
> feedback.
> > > >
> > > > Best regards,
> > > >
> > > > Prajwal Banakar
> > > >
> > > >
> > > > On Sun, 1 Mar, 2026, 11:49 am Prajwal Banakar, <
> > > prajwal.banakara@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I would like to start a discussion on the proposal for Native
> Bitmap
> > > > > Integration & Stateless Pushdown Aggregation.
> > > > >
> > > > > This proposal enables end-to-end native support for the BITMAP type
> > in
> > > > > Fluss and integrates it with the existing aggregation merge engine
> to
> > > > > support server-side bitmap union pushdown. The goal is to reduce
> > > network
> > > > > transfer and offload DISTINCT-style aggregation from Flink to the
> > > > > TabletServer.
> > > > >
> > > > > Key highlights of the proposal include:
> > > > >
> > > > > - Type System: Promoting BITMAP to a first-class logical type.
> > > > > - UDF Suite: Introducing BITMAP_BUILD, BITMAP_OR_AGG, and
> > > > > BITMAP_CARDINALITY (aligned with FLIP-556 and StarRocks semantics).
> > > > > - Optimizer: Planner-based pushdown via applyAggregates in the
> Flink
> > > > > connector.
> > > > > - Safety: No changes to LogRecordBatch or WAL, making this strictly
> > > > > additive and migration-free.
> > > > >
> > > > > You can find the full proposal document here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1sDhfkmo-w-UTvo2n3rsY1lytSSryswfkI83cSdka8s0/edit?usp=sharing
> > > > >
> > > > > I would appreciate feedback on the public interfaces, pushdown
> > > > > constraints, and overall scope.
> > > > >
> > > > > Best regards,
> > > > > Prajwal Banakar
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Keith Lee <le...@gmail.com>]

Hi Prajwal,

Thank you for addressing / answering the questions.

> This proposal adds the missing bridge: a proper BITMAP DDL type, SQL
functions (BITMAP_BUILD, BITMAP_OR_AGG, BITMAP_CARDINALITY), and pushdown
via applyAggregates(). The storage-side aggregation logic already exists;
this proposal makes it accessible end-to-end

1. That makes sense. I think the motivation section should lead with that
and remove the following as it can be misleading given that rbm is
supported by aggregation merge engine: “users requiring high-cardinality
unique counting (e.g., UV analytics) must execute Client-Side Aggregation.
The TabletServer is forced to send massive amounts of raw LogRecordBatch
rows over the network to a Flink cluster for evaluation. This results in
unnecessary network transfer and prevents efficient utilization of the
existing aggregation merge engine.”

2. That makes sense. Thank you for the context.

3.

> RBM64 requires a fundamentally different internal structure; a map of
RBM32 chunks which increases implementation and serialization complexity
significantly.

My understanding is that the proposal wires existing
FieldRoaringBitmap32Agg to support rbm32. FieldRoaringBitmap64Agg should
already exist and handle the complexity that you mentioned?

Additionally, it might be good for Yang to review / provide input on this
given his work on FIP-21.

Best regards

Keith Lee


On Wed, 11 Mar 2026 at 05:49, Prajwal Banakar <pr...@gmail.com>
wrote:

> Hi Keith, thank you for the detailed feedback.
>
> 1. On motivation vs existing aggregation merge engine: The aggregation
> merge engine in 0.9 supports rbm32/rbm64 at the storage level, but BITMAP
> is not yet a first-class type in the DDL or type system. Users today must
> declare the column as BYTES (as shown in the 0.9 release example: uv_bitmap
> BYTES), and there are no SQL functions to build, merge, or query bitmaps
> from Flink SQL. This proposal adds the missing bridge: a proper BITMAP DDL
> type, SQL functions (BITMAP_BUILD, BITMAP_OR_AGG, BITMAP_CARDINALITY), and
> pushdown via applyAggregates(). The storage-side aggregation logic already
> exists; this proposal makes it accessible end-to-end.
>
> 2. On NULL semantics: BITMAP_OR(bitmap, NULL) returns NULL following
> standard SQL scalar function semantics where NULL inputs propagate to NULL
> outputs. BITMAP_OR_AGG follows aggregate function convention consistent
> with how SUM and AVG behave, where NULLs in individual rows are skipped and
> only a fully NULL input set returns NULL. This distinction follows FLIP-556
> and StarRocks semantics.
>
> 3. On 32-bit scope: The proposal is scoped to 32-bit initially because
> RoaringBitmap32 covers integer values up to 2^32 (~4 billion), which is
> sufficient for most user ID and session ID use cases. RBM64 requires a
> fundamentally different internal structure; a map of RBM32 chunks which
> increases implementation and serialization complexity significantly.
> Starting with 32-bit keeps the initial scope focused and deliverable. RBM64
> support is listed as a Could-Have in the MoSCoW deliverables and can follow
> in a subsequent iteration.
>
> Best regards,
>
> Prajwal Banakar
>
>
> On Wed, 11 Mar 2026 at 01:34, Keith Lee <le...@gmail.com>
> wrote:
>
> > Hello Prajwal,
> >
> > Thank you for the detailed proposal. I enjoyed reading it and have a few
> > questions/comments.
> >
> > 1. On motivation, can you provide context on how this differs with
> > aggregation merge engine’s roaring bitmap implementation [1]?
> Specifically,
> > motivation part states that “users requiring high cardinality unique
> > counting … must execute client-side aggregation”. Aggregation merge
> engine
> > performs aggregation on server-side. The motivation section should
> clarify
> > how the proposed changes improve or complement aggregation merge engine,
> > which seems to have been considered as Section 2 references FIP-21
> > Aggregation Merge Engine. Adding this context will help readers
> understand
> > the motivation of the proposal better.
> >
> > 2. Can you clarify the NULL semantics section specifically on the
> decision
> > on why BITMAP_OR(bitmap, NULL) returns NULL but BITMAP_OR_AGG only
> returns
> > null when all rows are NULL?
> >
> > 3. Why is the scope limited to 32 bit bitmaps? Adding the rationale
> behind
> > these e.g. how (if any) support of 64bit bitmaps would increase
> > implementation complexity. Articulating these may help other contributors
> > understand the complexity and perhaps come up with suggestions on how to
> > address them.
> >
> > Best regards
> >
> > Keith Lee
> >
> > [1]
> >
> >
> https://fluss.apache.org/blog/releases/0.9/#2-storage-level-processing--semantics
> >
> >
> > On Mon, 9 Mar 2026 at 05:31, Prajwal Banakar <prajwal.banakara@gmail.com
> >
> > wrote:
> >
> > > Hi Devs,
> > >
> > > I have pushed a working prototype to my public fork demonstrating the
> > > BitmapType integrated with FieldRoaringBitmap32Agg. This includes four
> > > passing unit tests.
> > >
> > > The link to the prototype is available in the Google Doc, and you can
> > also
> > > find it here:
> > > https://github.com/Prajwal-banakar/fluss/tree/RoaringBitmap-prototype
> > >
> > > The Google Doc link remains the same. I look forward to your feedback.
> > >
> > > Best regards,
> > >
> > > Prajwal Banakar
> > >
> > >
> > > On Sun, 1 Mar, 2026, 11:49 am Prajwal Banakar, <
> > prajwal.banakara@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I would like to start a discussion on the proposal for Native Bitmap
> > > > Integration & Stateless Pushdown Aggregation.
> > > >
> > > > This proposal enables end-to-end native support for the BITMAP type
> in
> > > > Fluss and integrates it with the existing aggregation merge engine to
> > > > support server-side bitmap union pushdown. The goal is to reduce
> > network
> > > > transfer and offload DISTINCT-style aggregation from Flink to the
> > > > TabletServer.
> > > >
> > > > Key highlights of the proposal include:
> > > >
> > > > - Type System: Promoting BITMAP to a first-class logical type.
> > > > - UDF Suite: Introducing BITMAP_BUILD, BITMAP_OR_AGG, and
> > > > BITMAP_CARDINALITY (aligned with FLIP-556 and StarRocks semantics).
> > > > - Optimizer: Planner-based pushdown via applyAggregates in the Flink
> > > > connector.
> > > > - Safety: No changes to LogRecordBatch or WAL, making this strictly
> > > > additive and migration-free.
> > > >
> > > > You can find the full proposal document here:
> > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1sDhfkmo-w-UTvo2n3rsY1lytSSryswfkI83cSdka8s0/edit?usp=sharing
> > > >
> > > > I would appreciate feedback on the public interfaces, pushdown
> > > > constraints, and overall scope.
> > > >
> > > > Best regards,
> > > > Prajwal Banakar
> > > >
> > >
> >
>

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Prajwal Banakar <pr...@gmail.com>]

Hi Keith, thank you for the detailed feedback.

1. On motivation vs existing aggregation merge engine: The aggregation
merge engine in 0.9 supports rbm32/rbm64 at the storage level, but BITMAP
is not yet a first-class type in the DDL or type system. Users today must
declare the column as BYTES (as shown in the 0.9 release example: uv_bitmap
BYTES), and there are no SQL functions to build, merge, or query bitmaps
from Flink SQL. This proposal adds the missing bridge: a proper BITMAP DDL
type, SQL functions (BITMAP_BUILD, BITMAP_OR_AGG, BITMAP_CARDINALITY), and
pushdown via applyAggregates(). The storage-side aggregation logic already
exists; this proposal makes it accessible end-to-end.

2. On NULL semantics: BITMAP_OR(bitmap, NULL) returns NULL following
standard SQL scalar function semantics where NULL inputs propagate to NULL
outputs. BITMAP_OR_AGG follows aggregate function convention consistent
with how SUM and AVG behave, where NULLs in individual rows are skipped and
only a fully NULL input set returns NULL. This distinction follows FLIP-556
and StarRocks semantics.

3. On 32-bit scope: The proposal is scoped to 32-bit initially because
RoaringBitmap32 covers integer values up to 2^32 (~4 billion), which is
sufficient for most user ID and session ID use cases. RBM64 requires a
fundamentally different internal structure; a map of RBM32 chunks which
increases implementation and serialization complexity significantly.
Starting with 32-bit keeps the initial scope focused and deliverable. RBM64
support is listed as a Could-Have in the MoSCoW deliverables and can follow
in a subsequent iteration.

Best regards,

Prajwal Banakar


On Wed, 11 Mar 2026 at 01:34, Keith Lee <le...@gmail.com> wrote:

> Hello Prajwal,
>
> Thank you for the detailed proposal. I enjoyed reading it and have a few
> questions/comments.
>
> 1. On motivation, can you provide context on how this differs with
> aggregation merge engine’s roaring bitmap implementation [1]? Specifically,
> motivation part states that “users requiring high cardinality unique
> counting … must execute client-side aggregation”. Aggregation merge engine
> performs aggregation on server-side. The motivation section should clarify
> how the proposed changes improve or complement aggregation merge engine,
> which seems to have been considered as Section 2 references FIP-21
> Aggregation Merge Engine. Adding this context will help readers understand
> the motivation of the proposal better.
>
> 2. Can you clarify the NULL semantics section specifically on the decision
> on why BITMAP_OR(bitmap, NULL) returns NULL but BITMAP_OR_AGG only returns
> null when all rows are NULL?
>
> 3. Why is the scope limited to 32 bit bitmaps? Adding the rationale behind
> these e.g. how (if any) support of 64bit bitmaps would increase
> implementation complexity. Articulating these may help other contributors
> understand the complexity and perhaps come up with suggestions on how to
> address them.
>
> Best regards
>
> Keith Lee
>
> [1]
>
> https://fluss.apache.org/blog/releases/0.9/#2-storage-level-processing--semantics
>
>
> On Mon, 9 Mar 2026 at 05:31, Prajwal Banakar <pr...@gmail.com>
> wrote:
>
> > Hi Devs,
> >
> > I have pushed a working prototype to my public fork demonstrating the
> > BitmapType integrated with FieldRoaringBitmap32Agg. This includes four
> > passing unit tests.
> >
> > The link to the prototype is available in the Google Doc, and you can
> also
> > find it here:
> > https://github.com/Prajwal-banakar/fluss/tree/RoaringBitmap-prototype
> >
> > The Google Doc link remains the same. I look forward to your feedback.
> >
> > Best regards,
> >
> > Prajwal Banakar
> >
> >
> > On Sun, 1 Mar, 2026, 11:49 am Prajwal Banakar, <
> prajwal.banakara@gmail.com
> > >
> > wrote:
> >
> > > Hi everyone,
> > >
> > > I would like to start a discussion on the proposal for Native Bitmap
> > > Integration & Stateless Pushdown Aggregation.
> > >
> > > This proposal enables end-to-end native support for the BITMAP type in
> > > Fluss and integrates it with the existing aggregation merge engine to
> > > support server-side bitmap union pushdown. The goal is to reduce
> network
> > > transfer and offload DISTINCT-style aggregation from Flink to the
> > > TabletServer.
> > >
> > > Key highlights of the proposal include:
> > >
> > > - Type System: Promoting BITMAP to a first-class logical type.
> > > - UDF Suite: Introducing BITMAP_BUILD, BITMAP_OR_AGG, and
> > > BITMAP_CARDINALITY (aligned with FLIP-556 and StarRocks semantics).
> > > - Optimizer: Planner-based pushdown via applyAggregates in the Flink
> > > connector.
> > > - Safety: No changes to LogRecordBatch or WAL, making this strictly
> > > additive and migration-free.
> > >
> > > You can find the full proposal document here:
> > >
> > >
> >
> https://docs.google.com/document/d/1sDhfkmo-w-UTvo2n3rsY1lytSSryswfkI83cSdka8s0/edit?usp=sharing
> > >
> > > I would appreciate feedback on the public interfaces, pushdown
> > > constraints, and overall scope.
> > >
> > > Best regards,
> > > Prajwal Banakar
> > >
> >
>

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Keith Lee <le...@gmail.com>]

Hello Prajwal,

Thank you for the detailed proposal. I enjoyed reading it and have a few
questions/comments.

1. On motivation, can you provide context on how this differs with
aggregation merge engine’s roaring bitmap implementation [1]? Specifically,
motivation part states that “users requiring high cardinality unique
counting … must execute client-side aggregation”. Aggregation merge engine
performs aggregation on server-side. The motivation section should clarify
how the proposed changes improve or complement aggregation merge engine,
which seems to have been considered as Section 2 references FIP-21
Aggregation Merge Engine. Adding this context will help readers understand
the motivation of the proposal better.

2. Can you clarify the NULL semantics section specifically on the decision
on why BITMAP_OR(bitmap, NULL) returns NULL but BITMAP_OR_AGG only returns
null when all rows are NULL?

3. Why is the scope limited to 32 bit bitmaps? Adding the rationale behind
these e.g. how (if any) support of 64bit bitmaps would increase
implementation complexity. Articulating these may help other contributors
understand the complexity and perhaps come up with suggestions on how to
address them.

Best regards

Keith Lee

[1]
https://fluss.apache.org/blog/releases/0.9/#2-storage-level-processing--semantics


On Mon, 9 Mar 2026 at 05:31, Prajwal Banakar <pr...@gmail.com>
wrote:

> Hi Devs,
>
> I have pushed a working prototype to my public fork demonstrating the
> BitmapType integrated with FieldRoaringBitmap32Agg. This includes four
> passing unit tests.
>
> The link to the prototype is available in the Google Doc, and you can also
> find it here:
> https://github.com/Prajwal-banakar/fluss/tree/RoaringBitmap-prototype
>
> The Google Doc link remains the same. I look forward to your feedback.
>
> Best regards,
>
> Prajwal Banakar
>
>
> On Sun, 1 Mar, 2026, 11:49 am Prajwal Banakar, <prajwal.banakara@gmail.com
> >
> wrote:
>
> > Hi everyone,
> >
> > I would like to start a discussion on the proposal for Native Bitmap
> > Integration & Stateless Pushdown Aggregation.
> >
> > This proposal enables end-to-end native support for the BITMAP type in
> > Fluss and integrates it with the existing aggregation merge engine to
> > support server-side bitmap union pushdown. The goal is to reduce network
> > transfer and offload DISTINCT-style aggregation from Flink to the
> > TabletServer.
> >
> > Key highlights of the proposal include:
> >
> > - Type System: Promoting BITMAP to a first-class logical type.
> > - UDF Suite: Introducing BITMAP_BUILD, BITMAP_OR_AGG, and
> > BITMAP_CARDINALITY (aligned with FLIP-556 and StarRocks semantics).
> > - Optimizer: Planner-based pushdown via applyAggregates in the Flink
> > connector.
> > - Safety: No changes to LogRecordBatch or WAL, making this strictly
> > additive and migration-free.
> >
> > You can find the full proposal document here:
> >
> >
> https://docs.google.com/document/d/1sDhfkmo-w-UTvo2n3rsY1lytSSryswfkI83cSdka8s0/edit?usp=sharing
> >
> > I would appreciate feedback on the public interfaces, pushdown
> > constraints, and overall scope.
> >
> > Best regards,
> > Prajwal Banakar
> >
>

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Prajwal Banakar <pr...@gmail.com>]

Hi Devs,

I have pushed a working prototype to my public fork demonstrating the
BitmapType integrated with FieldRoaringBitmap32Agg. This includes four
passing unit tests.

The link to the prototype is available in the Google Doc, and you can also
find it here:
https://github.com/Prajwal-banakar/fluss/tree/RoaringBitmap-prototype

The Google Doc link remains the same. I look forward to your feedback.

Best regards,

Prajwal Banakar


On Sun, 1 Mar, 2026, 11:49 am Prajwal Banakar, <pr...@gmail.com>
wrote:

> Hi everyone,
>
> I would like to start a discussion on the proposal for Native Bitmap
> Integration & Stateless Pushdown Aggregation.
>
> This proposal enables end-to-end native support for the BITMAP type in
> Fluss and integrates it with the existing aggregation merge engine to
> support server-side bitmap union pushdown. The goal is to reduce network
> transfer and offload DISTINCT-style aggregation from Flink to the
> TabletServer.
>
> Key highlights of the proposal include:
>
> - Type System: Promoting BITMAP to a first-class logical type.
> - UDF Suite: Introducing BITMAP_BUILD, BITMAP_OR_AGG, and
> BITMAP_CARDINALITY (aligned with FLIP-556 and StarRocks semantics).
> - Optimizer: Planner-based pushdown via applyAggregates in the Flink
> connector.
> - Safety: No changes to LogRecordBatch or WAL, making this strictly
> additive and migration-free.
>
> You can find the full proposal document here:
>
> https://docs.google.com/document/d/1sDhfkmo-w-UTvo2n3rsY1lytSSryswfkI83cSdka8s0/edit?usp=sharing
>
> I would appreciate feedback on the public interfaces, pushdown
> constraints, and overall scope.
>
> Best regards,
> Prajwal Banakar
>

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Prajwal Banakar <pr...@gmail.com>]

Hi Yang,

Thank you for the review and for your kind words.

Regarding the "rb64_" prefix, I agree. Using "rb64_" (e.g., rb64_build_agg,
rb64_cardinality) aligns well with pg_roaringbitmap conventions and creates
a clean separation between the 32-bit and 64-bit function families. I have
updated the "Future Work" section to reflect this naming direction.

I also agree with adding rb_or and rb_and scalars to Phase 1. Since these
are fundamental set operations, they belong in Phase 1 alongside the other
scalar functions. I have added rb_or(BYTES, BYTES) → BYTES and
rb_and(BYTES, BYTES) → BYTES to the Phase 1 function table.

Best regards,
Prajwal Banakar

On Thu, 23 Apr 2026 at 11:28, Yang Wang <pl...@gmail.com> wrote:

> Hi Prajwal,
>
> Thank you for your outstanding work in driving this forward. I fully agree
> with the conclusions reached in our recent discussions. I just have two
> minor suggestions to add:
>
> For the 64-bit function family, I suggest using the 'rb64_' prefix (e.g.,
> rb64_build_agg). This aligns well with PostgreSQL’s naming conventions.
> Looking ahead, we could adopt a consistent style for type handling as
> well—for instance, using BITMAP to refer to the 32-bit type and BITMAP64
> for the 64-bit encoded type.
>
> Additionally, for the functions introduced in Phase 1, it would be
> beneficial to include scalar versions of rb_and and rb_or.
>
> Best regards,
> Yang
>
> Prajwal Banakar <pr...@gmail.com> 于2026年3月1日周日 14:20写道：
>
> > Hi everyone,
> >
> > I would like to start a discussion on the proposal for Native Bitmap
> > Integration & Stateless Pushdown Aggregation.
> >
> > This proposal enables end-to-end native support for the BITMAP type in
> > Fluss and integrates it with the existing aggregation merge engine to
> > support server-side bitmap union pushdown. The goal is to reduce network
> > transfer and offload DISTINCT-style aggregation from Flink to the
> > TabletServer.
> >
> > Key highlights of the proposal include:
> >
> > - Type System: Promoting BITMAP to a first-class logical type.
> > - UDF Suite: Introducing BITMAP_BUILD, BITMAP_OR_AGG, and
> > BITMAP_CARDINALITY (aligned with FLIP-556 and StarRocks semantics).
> > - Optimizer: Planner-based pushdown via applyAggregates in the Flink
> > connector.
> > - Safety: No changes to LogRecordBatch or WAL, making this strictly
> > additive and migration-free.
> >
> > You can find the full proposal document here:
> >
> >
> https://docs.google.com/document/d/1sDhfkmo-w-UTvo2n3rsY1lytSSryswfkI83cSdka8s0/edit?usp=sharing
> >
> > I would appreciate feedback on the public interfaces, pushdown
> constraints,
> > and overall scope.
> >
> > Best regards,
> > Prajwal Banakar
> >
>

Re: [DISCUSS] Proposal: Native Bitmap Integration & Stateless Pushdown Aggregation

[Yang Wang <pl...@gmail.com>]

Hi Prajwal,

Thank you for your outstanding work in driving this forward. I fully agree
with the conclusions reached in our recent discussions. I just have two
minor suggestions to add:

For the 64-bit function family, I suggest using the 'rb64_' prefix (e.g.,
rb64_build_agg). This aligns well with PostgreSQL’s naming conventions.
Looking ahead, we could adopt a consistent style for type handling as
well—for instance, using BITMAP to refer to the 32-bit type and BITMAP64
for the 64-bit encoded type.

Additionally, for the functions introduced in Phase 1, it would be
beneficial to include scalar versions of rb_and and rb_or.

Best regards,
Yang

Prajwal Banakar <pr...@gmail.com> 于2026年3月1日周日 14:20写道：

> Hi everyone,
>
> I would like to start a discussion on the proposal for Native Bitmap
> Integration & Stateless Pushdown Aggregation.
>
> This proposal enables end-to-end native support for the BITMAP type in
> Fluss and integrates it with the existing aggregation merge engine to
> support server-side bitmap union pushdown. The goal is to reduce network
> transfer and offload DISTINCT-style aggregation from Flink to the
> TabletServer.
>
> Key highlights of the proposal include:
>
> - Type System: Promoting BITMAP to a first-class logical type.
> - UDF Suite: Introducing BITMAP_BUILD, BITMAP_OR_AGG, and
> BITMAP_CARDINALITY (aligned with FLIP-556 and StarRocks semantics).
> - Optimizer: Planner-based pushdown via applyAggregates in the Flink
> connector.
> - Safety: No changes to LogRecordBatch or WAL, making this strictly
> additive and migration-free.
>
> You can find the full proposal document here:
>
> https://docs.google.com/document/d/1sDhfkmo-w-UTvo2n3rsY1lytSSryswfkI83cSdka8s0/edit?usp=sharing
>
> I would appreciate feedback on the public interfaces, pushdown constraints,
> and overall scope.
>
> Best regards,
> Prajwal Banakar
>