[I] Optimize client bucket-level metrics [fluss]

["loserwang1024 (via GitHub)" <gi...@apache.org>]

loserwang1024 opened a new issue, #3826:
URL: https://github.com/apache/fluss/issues/3826

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
   
   
   ### Motivation
   
   Currently, Fluss exposes client-side bucket-level metrics during Flink job execution. For jobs with many buckets or large-scale deployments, these high-cardinality metrics consume a significant amount of metrics storage and lead to high observability costs.
   
   
   Need to reduce the default metrics cardinality and storage cost, while still allowing users to expose bucket-level metrics when they need fine-grained observability or troubleshooting.
   
   
   ### Solution
   
   Phase 1: Remove the default exposure of client-side bucket-level metrics to reduce metrics storage pressure for existing Flink jobs.
   
   Phase 2: Add a configuration option to let users enable bucket-level metrics on demand. The option should be disabled by default.
   
   
   ### Anything else?
   
   _No response_
   
   ### Willingness to contribute
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Optimize client bucket-level metrics [fluss]

["Zhuoxi2000 (via GitHub)" <gi...@apache.org>]

Zhuoxi2000 commented on issue #3826:
URL: https://github.com/apache/fluss/issues/3826#issuecomment-5304560605

   I’d like to work on this, high cardinality client metrics is something I’ve run into operationally on large Flink deployments, so the motivation makes sense to me.
   
   My understanding is that the goal is to keep bucket-level metrics available for debugging without paying the cardinality cost by default. I’m thinking:
   
   1. Aggregate the current bucket metrics at table level by default, using sum/max depending on the metric.
   2. Keep per-bucket metrics behind an opt-in client option, e.g. `client.metrics.bucket-level.enabled=false`.
   3. Call this out in the release notes since dashboards relying on bucket labels will change.
   
   One thing I’d like to check while implementing is whether any metric, such as per bucket lag, is important enough to keep bucket-level by default.
   
   Could you assign this to me?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org