[I] offlineSegmentDelayHours reports up to one checker period of extra delay because the stored max end time only refreshes on the validation run [pinot]

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz opened a new issue, #19970:
URL: https://github.com/apache/pinot/issues/19970

   ## Summary
   
   `pinot_controller_offlineSegmentDelayHours` (and `lastPushTimeDelayHours`) can report a delay up to one checker period (24h by default) higher than the real delay. The gauge computes `now - stored max segment end time` on each scrape, which is correct. But the stored value only changes when `OfflineSegmentValidationManager` runs. A segment uploaded right after a run has no effect on the gauge until the next run.
   
   ## How it works today
   
   - `OfflineSegmentValidationManager.validateOfflineSegmentPush()` reads the ZK metadata of all segments of the table, computes the max `endTimeMs` and max push time, and stores them with `ValidationMetrics.updateOfflineSegmentDelayGauge()` / `updateLastPushTimeGauge()`.
   - `ValidationMetrics.CurrentTimeMillisDeltaGaugeHours.value()` returns `(System.currentTimeMillis() - storedValue) / 1h` on each read.
   - The task runs every `controller.offline.segment.interval.checker.frequencyPeriod`, which is `24h` by default.
   
   As a result, the gauge is a sawtooth: between two runs it increases by 1h every hour, even when segments arrive on time.
   
   ```
   reported delay
        |      /|      /|      /|
        |    /  |    /  |    /  |      <- peak = real delay + up to 24h
        |  /    |  /    |  /    |
        |/      |/      |/      |      <- value right after each run = real delay
        +-------+-------+-------+----> time
               run     run     run
   ```
   
   ## Example
   
   A hybrid table gets one OFFLINE segment per day from SegmentImportTask at 00:20 UTC. The validation task runs at 14:32 UTC.
   
   - At 14:07 UTC the newest segment was already in ZK, uploaded at 00:20 UTC, with an end time about 70h old.
   - The gauge still used the end time of the previous segment and reported 96.05h.
   - At the 14:32 UTC run the gauge dropped to 69.98h.
   
   An alert at `>= 96` fired for 25 minutes on data that was not late. For 30 days the daily peak was 92–95h and the real delay was 68–72h on every day.
   
   ## Suggested fix
   
   This is a suggestion. Other approaches are fine.
   
   Update the stored max end time and max push time when the controller accepts a new or refreshed OFFLINE segment. The upload path already has the new `SegmentZKMetadata`, so the update is `stored = max(stored, newEndTimeMs)` with no ZK scan. `OfflineSegmentValidationManager` keeps its full recompute, which corrects the value after segment deletion or retention, when the max can decrease.
   
   Alternatives:
   
   - Lower the default checker period. This is configuration only, but it adds ZK reads for clusters with many segments, and it reduces the error without removing it.
   - Document the behavior and tell users to alert on `min_over_time(...[<checker period>])`. This is what we do today as a mitigation.
   
   ## Workaround
   
   Alert on `min_over_time(pinot_controller_offlineSegmentDelayHours_Value[24h])`, which reads the value right after the last run. The cost is that a real stall shows up to 24h later.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

Re: [I] offlineSegmentDelayHours reports up to one checker period of extra delay because the stored max end time only refreshes on the validation run [pinot]

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz closed issue #19970: offlineSegmentDelayHours reports up to one checker period of extra delay because the stored max end time only refreshes on the validation run
URL: https://github.com/apache/pinot/issues/19970


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org