[PR] fix(prometheus): flush expired slab memory in exporter timer [apisix]

["sihyeonn (via GitHub)" <gi...@apache.org>]

sihyeonn opened a new pull request, #13195:
URL: https://github.com/apache/apisix/pull/13195

   ## Summary
   
   When metrics are configured with an `expire` value, nginx's slab allocator marks entries as logically expired but does **not** automatically return the underlying slab pages to the free-space pool. As a result, `apisix_shared_dict_free_space_bytes` for `prometheus-metrics` decreases monotonically over time — slabs are only reclaimed when explicitly flushed.
   
   ## Root Cause
   
   `ngx.shared.DICT:flush_expired()` must be called explicitly to reclaim slab memory from expired entries. Without it:
   
   - Time series expire logically (reads return nil after `expire` seconds)
   - But the slab memory is **not** returned to free space
   - `free_space_bytes` trends toward zero regardless of actual active time-series count
   
   This can be observed by comparing `free_space_bytes` with the active time-series count: the count fluctuates (e.g. drops significantly during low-traffic periods) while free space never recovers — even after most entries have expired.
   
   ## Fix
   
   Call `dict:flush_expired(1000)` inside `exporter_timer`, which already runs every `refresh_interval` (default 15 s) in the privileged agent process.
   
   ```lua
   local prom_dict = ngx.shared["prometheus-metrics"]
   if prom_dict then
       prom_dict:flush_expired(1000)
   end
   ```
   
   **Why `max_count=1000`**: Without a limit, a single flush call could hold the shared-dict write lock for an extended time if many expired entries have accumulated. Limiting to 1000 per cycle keeps the lock time well under 10 ms in practice, while remaining entries are flushed in subsequent timer ticks (every 15 s).
   
   The call runs in the privileged agent process, which is separate from worker request-handling processes, so the brief write-lock has minimal impact on request throughput.
   
   ## Checklist
   
   - [x] No functional change to metric collection or rendering
   - [x] Compatible with existing `expire` metric configuration
   - [x] Works with any `refresh_interval` setting


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(prometheus): flush expired slab memory in exporter timer [apisix]

["sihyeonn (via GitHub)" <gi...@apache.org>]

sihyeonn closed pull request #13195: fix(prometheus): flush expired slab memory in exporter timer
URL: https://github.com/apache/apisix/pull/13195


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(prometheus): flush expired slab memory in exporter timer [apisix]

["x1117 (via GitHub)" <gi...@apache.org>]

x1117 commented on PR #13195:
URL: https://github.com/apache/apisix/pull/13195#issuecomment-4628605136

   Hi, I'm also experiencing this issue. Has it been fixed now?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(prometheus): flush expired slab memory in exporter timer [apisix]

["sihyeonn (via GitHub)" <gi...@apache.org>]

sihyeonn commented on PR #13195:
URL: https://github.com/apache/apisix/pull/13195#issuecomment-4263968085

   https://github.com/api7/nginx-lua-prometheus/pull/13


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org