[I] [metrics] Histogram samples never age out, leaving stale percentiles on low-frequency metrics [fluss]

["fxbing (via GitHub)" <gi...@apache.org>]

fxbing opened a new issue, #4139:
URL: https://github.com/apache/fluss/issues/4139

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
   
   
   ### Fluss version
   
   main (development)
   
   ### Please describe the bug 🐞
   
   `DescriptiveStatisticsHistogram` is the only `Histogram` implementation used by Fluss metric groups (13 construction sites: tablet-server flush latencies, lake lookup time, coordinator queue/process time, RPC request histograms, client writer/scanner). It keeps a fixed-size circular buffer of values with no time dimension: a sample is evicted only when enough newer samples arrive to push it out. The same implementation is present in v0.9.1-incubating.
   
   On low-frequency metrics this makes percentiles stick for hours after a short anomaly, and on metrics that stop receiving samples entirely (idle table, tablet reassigned away) stale percentiles never expire until restart.
   
   Concrete example: `logFlushLatencyMs` is a low-frequency metric by design — `log.flush.interval-messages` defaults to `Long.MAX_VALUE`, so flushes happen only on infrequent paths (segment roll, close). In one observed low-traffic deployment, this metric was updated at about 1.8 samples/min. In the deterministic 1024-sample reproduction, P99 drops below 1 second only after 1,015 subsequent normal samples — about 9.4 hours at that observed rate. During the first hour shown by the reproduction, P99 remains at 2,000 ms across scrapes while `_count` increases, so alerting on P99 fires for hours although the server is healthy. The same applies to all window sizes in use (64 / 100 / 1024).
   
   Reproduction is deterministic and takes <1s against current source (verified on main @ f09de856b): fill a `DescriptiveStatisticsHistogram` with normal samples, add a few percent of slow samples, then keep feeding normal samples — P99 stays at the slow value until 1,015 more samples arrive.
   
   Expected behavior: histogram samples age out by time, so that (a) a short burst of slow samples fades from percentiles within a bounded duration, and (b) metrics without new samples stop reporting stale values.
   
   ### Solution
   
   Give the histogram a dual bound: **current samples = last `maxSamples` ∩ last `maxAge`**.
   
   - Keep a timestamp array next to the existing value ring in `DescriptiveStatisticsHistogram`; evict expired samples in both `update()` and `getStatistics()`, so scraping alone expires stale values even when no new samples arrive. `getCount()` stays cumulative.
   - Add a `(maxSamples, maxAge)` constructor, keep the existing single-arg constructor unchanged, and migrate the 13 internal construction sites to a shared initial `maxAge` proposal of 60 minutes, subject to maintainer feedback, while retaining the existing caps of 64/100/1024. At the observed ~1.8 samples/min this retains about 108 recent values; each sample expires within 60 minutes, so stale contribution is bounded to 60 minutes after the last slow sample. High-frequency metrics keep behaving as today because the sample cap dominates.
   - No new dependency; metric names, the `HistogramStatistics` contract, and the Prometheus export format stay unchanged.
   
   Alternatives considered: Dropwizard `ExponentiallyDecayingReservoir` never expires without new samples (verified: P99 unchanged after 60 idle minutes) and its effective sample count is rate-dependent (Kafka's α=0.015 yields ~2 effective samples at 1.8 samples/min); `SlidingTimeWindowArrayReservoir` has no sample cap, a memory risk for high-frequency RPC histograms; Prometheus `Summary` age buckets live in the reporter module and cannot back the reporter-agnostic `Histogram` contract (JMX/InfluxDB/Flink); HdrHistogram/Micrometer still need time-bucket rotation on top. The dual-bound semantics follow Kafka's `SampledStat` (event window + time window).
   
   Out of scope, to be tracked separately: RocksDB native histogram statistics are exported as lifetime gauges and need their own handling; exporting Prometheus `_bucket` series for server-side `histogram_quantile` aggregation is an independent change.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org