[I] [Feature] Add Apache DataSketches HLL sketch aggregation support [fluss]

["rahulbsw (via GitHub)" <gi...@apache.org>]

rahulbsw opened a new issue, #3331:
URL: https://github.com/apache/fluss/issues/3331

   ### Motivation
   
   Fluss aggregation merge-engine tables currently support exact numeric, string, boolean, value-selection, and RoaringBitmap byte aggregations. The aggregation documentation does not currently expose a native approximate distinct-count aggregation, such as HLL/HyperLogLog, for incremental real-time analytics.
   
   Approximate distinct counting is useful for workloads such as daily active users, unique devices, unique sessions, and feature-store metrics. In these workloads, exact `COUNT(DISTINCT ...)` can be expensive to maintain continuously, while sketches provide compact mergeable state.
   
   Apache DataSketches provides sketch implementations with defined serialization and merge semantics. HLL sketch support fits Fluss' aggregation merge engine model because Fluss can store serialized sketch bytes and merge them per primary key, similar to the existing `rbm32` and `rbm64` serialized RoaringBitmap aggregation functions.
   
   ### Solution
   
   Add a new aggregation merge function named `hll_sketch`.
   
   The function should:
   
   - Support only `BYTES` columns.
   - Treat input values as serialized Apache DataSketches HLL sketches.
   - Merge two non-null sketches using the DataSketches HLL union API.
   - Return compact serialized sketch bytes.
   - Ignore nulls consistently with existing byte aggregations such as `rbm32` and `rbm64`.
   - Validate function options through the existing aggregation function parameter validation path.
   
   Example DDL:
   
   ```sql
   CREATE TABLE daily_user_metrics (
       metric_day STRING,
       user_hll BYTES,
       PRIMARY KEY (metric_day) NOT ENFORCED
   ) WITH (
       'table.merge-engine' = 'aggregation',
       'fields.user_hll.agg' = 'hll_sketch'
   );
   ```
   
   This first change should focus on storage-level sketch merging. Users would still write pre-serialized HLL sketch bytes from a client or compute engine.
   
   Potential follow-up work:
   
   - Add Flink SQL helper functions such as `fluss_hll_build(value)` and `fluss_hll_estimate(bytes)`.
   - Add Spark SQL helper functions with the same semantics.
   - Add KLL-based percentile support with a `kll_double_sketch` merge function.
   
   ### Anything else?
   
   The implementation can follow the existing RoaringBitmap aggregation pattern:
   
   - Add `HLL_SKETCH` to `AggFunctionType`.
   - Add a Java API factory method such as `AggFunctions.HLL_SKETCH()`.
   - Add a server-side `FieldHllSketchAgg` and SPI factory.
   - Add unit tests in the aggregation row merger test suite.
   - Document the function in the aggregation merge-engine docs.
   
   Dependency and packaging notes:
   
   - DataSketches Java depends on DataSketches Memory, so the build and binary license metadata should be checked.
   - `LICENSE-bin` already mentions Apache DataSketches, but the actual module dependencies still need to be added and verified.
   
   I searched existing issues for DataSketches, HLL, KLL, `approx_count_distinct`, and `approx_percentile` and did not find a matching Fluss feature request.
   
   I am willing to submit a PR for the first `hll_sketch` implementation after maintainers confirm the API name and scope.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org