You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gsoc@community.apache.org by "Jark Wu (Jira)" <ji...@apache.org> on 2026/03/18 06:54:00 UTC
[jira] [Updated] (GSOC-331) Apache Fluss (Incubating): Native RoaringBitmap Integration for Apache Fluss
[ https://issues.apache.org/jira/browse/GSOC-331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jark Wu updated GSOC-331:
-------------------------
Description:
*Synopsis*
Apache Fluss currently incorporates the BITMAP data type within its metadata layer, but it remains inaccessible to end-users as it is trapped in the UnsupportedKeyword enum. While the aggregation merge engine in Fluss 0.9 supports rbm32/rbm64 at the storage level, BITMAP is not yet a first-class type. Users must currently declare bitmap columns as BYTES.
This GSoC project aims to enable end-to-end native support for the BITMAP data type to allow efficient server-side unique counting. By shifting the computational burden from the client side to the storage side, we can reduce network I/O and CPU utilization for high-cardinality DISTINCT-style aggregations. The project will introduce a proper BITMAP DDL type, SQL functions, and pushdown optimization via applyAggregates().
*Benefits to Community*
1. **Network I/O Efficiency**: With bitmap pushdown, only one serialized RoaringBitmap is transferred per group instead of all raw rows, reducing network cost from O(N) to O(G).
2. **CPU Utilization Reduction**: Heavy unique counting computation is offloaded to the Fluss TabletServer's native merge engine, reducing Flink TaskManager CPU overhead.
3. **Ecosystem Interoperability**: By using the standard RoaringBitmap binary serialization format, Fluss ensures bitmap data remains accessible to downstream consumers such as Flink, StarRocks, and Doris without requiring proprietary Fluss-specific headers or custom decoders.
4. **UV Analytics Optimization**: Enables efficient Unique Visitor analytics workflows with pre-aggregated bitmap fragments that can be efficiently merged on the storage side.
*Deliverables*
The student will deliver the following components:
1. **Type System Enablement (fluss-common)**
- Introduce BitmapType as a new logical type in fluss-common
- Extend DataTypeParser to support the BITMAP keyword in CREATE TABLE statements
- Define BITMAP type properties (nullable, not orderable, cannot be used as primary key or partition key)
2. **Server-Side Aggregation Integration (fluss-server)**
- Wire FieldRoaringBitmap32Agg to the new BITMAP logical type
- Extend FieldRoaringBitmap32AggFactory to accept DataTypeRoot.BITMAP in addition to DataTypeRoot.BYTES
- Update AggFunctionType.getSupportedDataTypeRoots() accordingly
- Perform comprehensive audit of AggregationMergeEngine for BITMAP type handling
3. **Flink Connector Bridge (fluss-flink)**
- Implement SQL UDFs: BITMAP_BUILD_AGG, BITMAP_OR_AGG, BITMAP_CARDINALITY, BITMAP_FROM_BYTES, BITMAP_TO_BYTES
- Extend PbDataTypeRoot with BITMAP = 16 for RPC message support
- Implement applyAggregates() pushdown optimization for BITMAP_OR_AGG
- Handle graceful fallback to Flink-side aggregation when pushdown is not applicable
4. **Testing & Documentation**
- Functional unit tests for BitmapType and FieldRoaringBitmap32Agg
- End-to-end integration tests (BitmapPushdownITCase) in flink-flink-common module
- Performance benchmarks measuring network I/O and CPU utilization improvements
- User documentation and SQL reference guides
**Required Skills**
- Proficiency in Java programming
- Understanding of distributed systems and data processing concepts
- Familiarity with Apache Flink or similar stream processing frameworks
- Knowledge of SQL and query optimization is a plus
- Experience with bitmap data structures (RoaringBitmap) is advantageous
- Good communication skills for community collaboration
*Difficulty Level*
Medium to Hard
This project requires understanding of multiple layers in the Fluss stack (common, server, flink connector) and involves type system changes, aggregation engine integration, and query optimization. A working prototype demonstrating BitmapType integration with FieldRoaringBitmap32Agg is available to help the student get started.
*Mentors*
The Apache Fluss community will provide experienced mentors for this project. Mentor details will be announced on the Fluss dev mailing list before the GSoC application period.
**Future Work (Stretch Goals)**
- Native 64-bit support (BITMAP64 type) for IDs exceeding 32-bit range
- Advanced conversion functions: BITMAP_TO_ARRAY, BITMAP_TO_STRING, BITMAP_XOR_AGG
- Reverse materialization: UNNEST_BITMAP to explode bitmaps back into individual integer rows
**Name and Contact Information**
Project: Apache Fluss (Incubating)
Website: https://fluss.apache.org
Mailing List: dev@fluss.apache.org
GitHub: https://github.com/apache/fluss
was:
**Synopsis**
Apache Fluss currently incorporates the BITMAP data type within its metadata layer, but it remains inaccessible to end-users as it is trapped in the UnsupportedKeyword enum. While the aggregation merge engine in Fluss 0.9 supports rbm32/rbm64 at the storage level, BITMAP is not yet a first-class type. Users must currently declare bitmap columns as BYTES.
This GSoC project aims to enable end-to-end native support for the BITMAP data type to allow efficient server-side unique counting. By shifting the computational burden from the client side to the storage side, we can reduce network I/O and CPU utilization for high-cardinality DISTINCT-style aggregations. The project will introduce a proper BITMAP DDL type, SQL functions, and pushdown optimization via applyAggregates().
**Benefits to Community**
1. **Network I/O Efficiency**: With bitmap pushdown, only one serialized RoaringBitmap is transferred per group instead of all raw rows, reducing network cost from O(N) to O(G).
2. **CPU Utilization Reduction**: Heavy unique counting computation is offloaded to the Fluss TabletServer's native merge engine, reducing Flink TaskManager CPU overhead.
3. **Ecosystem Interoperability**: By using the standard RoaringBitmap binary serialization format, Fluss ensures bitmap data remains accessible to downstream consumers such as Flink, StarRocks, and Doris without requiring proprietary Fluss-specific headers or custom decoders.
4. **UV Analytics Optimization**: Enables efficient Unique Visitor analytics workflows with pre-aggregated bitmap fragments that can be efficiently merged on the storage side.
**Deliverables**
The student will deliver the following components:
1. **Type System Enablement (fluss-common)**
- Introduce BitmapType as a new logical type in fluss-common
- Extend DataTypeParser to support the BITMAP keyword in CREATE TABLE statements
- Define BITMAP type properties (nullable, not orderable, cannot be used as primary key or partition key)
2. **Server-Side Aggregation Integration (fluss-server)**
- Wire FieldRoaringBitmap32Agg to the new BITMAP logical type
- Extend FieldRoaringBitmap32AggFactory to accept DataTypeRoot.BITMAP in addition to DataTypeRoot.BYTES
- Update AggFunctionType.getSupportedDataTypeRoots() accordingly
- Perform comprehensive audit of AggregationMergeEngine for BITMAP type handling
3. **Flink Connector Bridge (fluss-flink)**
- Implement SQL UDFs: BITMAP_BUILD_AGG, BITMAP_OR_AGG, BITMAP_CARDINALITY, BITMAP_FROM_BYTES, BITMAP_TO_BYTES
- Extend PbDataTypeRoot with BITMAP = 16 for RPC message support
- Implement applyAggregates() pushdown optimization for BITMAP_OR_AGG
- Handle graceful fallback to Flink-side aggregation when pushdown is not applicable
4. **Testing & Documentation**
- Functional unit tests for BitmapType and FieldRoaringBitmap32Agg
- End-to-end integration tests (BitmapPushdownITCase) in flink-flink-common module
- Performance benchmarks measuring network I/O and CPU utilization improvements
- User documentation and SQL reference guides
**Required Skills**
- Proficiency in Java programming
- Understanding of distributed systems and data processing concepts
- Familiarity with Apache Flink or similar stream processing frameworks
- Knowledge of SQL and query optimization is a plus
- Experience with bitmap data structures (RoaringBitmap) is advantageous
- Good communication skills for community collaboration
**Difficulty Level**
Medium to Hard
This project requires understanding of multiple layers in the Fluss stack (common, server, flink connector) and involves type system changes, aggregation engine integration, and query optimization. A working prototype demonstrating BitmapType integration with FieldRoaringBitmap32Agg is available to help the student get started.
**Mentors**
The Apache Fluss community will provide experienced mentors for this project. Mentor details will be announced on the Fluss dev mailing list before the GSoC application period.
**Related Work**
1. FIP-21 (Aggregation Merge Engine, release 0.9) - Established the FieldAggregator SPI and includes RBM32/RBM64 in AggFunctionType
2. FLIP-556 (Flink Improvement Proposal for BITMAP Data Type) - Function naming and null semantics conventions
3. RoaringBitmap specification - Standard binary serialization format for interoperability
4. Working prototype available at: https://github.com/Prajwal-banakar/fluss/tree/RoaringBitmap-prototype
**Future Work (Stretch Goals)**
- Native 64-bit support (BITMAP64 type) for IDs exceeding 32-bit range
- Advanced conversion functions: BITMAP_TO_ARRAY, BITMAP_TO_STRING, BITMAP_XOR_AGG
- Reverse materialization: UNNEST_BITMAP to explode bitmaps back into individual integer rows
**Name and Contact Information**
Project: Apache Fluss (Incubating)
Website: https://fluss.apache.org
Mailing List: dev@fluss.apache.org
GitHub: https://github.com/apache/fluss
> Apache Fluss (Incubating): Native RoaringBitmap Integration for Apache Fluss
> ----------------------------------------------------------------------------
>
> Key: GSOC-331
> URL: https://issues.apache.org/jira/browse/GSOC-331
> Project: Comdev GSOC
> Issue Type: New Feature
> Reporter: Jark Wu
> Priority: Major
>
> *Synopsis*
> Apache Fluss currently incorporates the BITMAP data type within its metadata layer, but it remains inaccessible to end-users as it is trapped in the UnsupportedKeyword enum. While the aggregation merge engine in Fluss 0.9 supports rbm32/rbm64 at the storage level, BITMAP is not yet a first-class type. Users must currently declare bitmap columns as BYTES.
> This GSoC project aims to enable end-to-end native support for the BITMAP data type to allow efficient server-side unique counting. By shifting the computational burden from the client side to the storage side, we can reduce network I/O and CPU utilization for high-cardinality DISTINCT-style aggregations. The project will introduce a proper BITMAP DDL type, SQL functions, and pushdown optimization via applyAggregates().
> *Benefits to Community*
> 1. **Network I/O Efficiency**: With bitmap pushdown, only one serialized RoaringBitmap is transferred per group instead of all raw rows, reducing network cost from O(N) to O(G).
> 2. **CPU Utilization Reduction**: Heavy unique counting computation is offloaded to the Fluss TabletServer's native merge engine, reducing Flink TaskManager CPU overhead.
> 3. **Ecosystem Interoperability**: By using the standard RoaringBitmap binary serialization format, Fluss ensures bitmap data remains accessible to downstream consumers such as Flink, StarRocks, and Doris without requiring proprietary Fluss-specific headers or custom decoders.
> 4. **UV Analytics Optimization**: Enables efficient Unique Visitor analytics workflows with pre-aggregated bitmap fragments that can be efficiently merged on the storage side.
> *Deliverables*
> The student will deliver the following components:
> 1. **Type System Enablement (fluss-common)**
> - Introduce BitmapType as a new logical type in fluss-common
> - Extend DataTypeParser to support the BITMAP keyword in CREATE TABLE statements
> - Define BITMAP type properties (nullable, not orderable, cannot be used as primary key or partition key)
> 2. **Server-Side Aggregation Integration (fluss-server)**
> - Wire FieldRoaringBitmap32Agg to the new BITMAP logical type
> - Extend FieldRoaringBitmap32AggFactory to accept DataTypeRoot.BITMAP in addition to DataTypeRoot.BYTES
> - Update AggFunctionType.getSupportedDataTypeRoots() accordingly
> - Perform comprehensive audit of AggregationMergeEngine for BITMAP type handling
> 3. **Flink Connector Bridge (fluss-flink)**
> - Implement SQL UDFs: BITMAP_BUILD_AGG, BITMAP_OR_AGG, BITMAP_CARDINALITY, BITMAP_FROM_BYTES, BITMAP_TO_BYTES
> - Extend PbDataTypeRoot with BITMAP = 16 for RPC message support
> - Implement applyAggregates() pushdown optimization for BITMAP_OR_AGG
> - Handle graceful fallback to Flink-side aggregation when pushdown is not applicable
> 4. **Testing & Documentation**
> - Functional unit tests for BitmapType and FieldRoaringBitmap32Agg
> - End-to-end integration tests (BitmapPushdownITCase) in flink-flink-common module
> - Performance benchmarks measuring network I/O and CPU utilization improvements
> - User documentation and SQL reference guides
> **Required Skills**
> - Proficiency in Java programming
> - Understanding of distributed systems and data processing concepts
> - Familiarity with Apache Flink or similar stream processing frameworks
> - Knowledge of SQL and query optimization is a plus
> - Experience with bitmap data structures (RoaringBitmap) is advantageous
> - Good communication skills for community collaboration
> *Difficulty Level*
> Medium to Hard
> This project requires understanding of multiple layers in the Fluss stack (common, server, flink connector) and involves type system changes, aggregation engine integration, and query optimization. A working prototype demonstrating BitmapType integration with FieldRoaringBitmap32Agg is available to help the student get started.
> *Mentors*
> The Apache Fluss community will provide experienced mentors for this project. Mentor details will be announced on the Fluss dev mailing list before the GSoC application period.
> **Future Work (Stretch Goals)**
> - Native 64-bit support (BITMAP64 type) for IDs exceeding 32-bit range
> - Advanced conversion functions: BITMAP_TO_ARRAY, BITMAP_TO_STRING, BITMAP_XOR_AGG
> - Reverse materialization: UNNEST_BITMAP to explode bitmaps back into individual integer rows
> **Name and Contact Information**
> Project: Apache Fluss (Incubating)
> Website: https://fluss.apache.org
> Mailing List: dev@fluss.apache.org
> GitHub: https://github.com/apache/fluss
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: gsoc-unsubscribe@community.apache.org
For additional commands, e-mail: gsoc-help@community.apache.org