You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "nzw921rx (via GitHub)" <gi...@apache.org> on 2026/08/12 06:39:04 UTC

[I] [Feature][Prometheus] Migrate PrometheusWriter to the engine-level FlushSignal mechanism [seatunnel]

nzw921rx opened a new issue, #11773:
URL: https://github.com/apache/seatunnel/issues/11773

   ### Search before asking
   
   - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
   
   
   ### Description
   
   `PrometheusWriter` currently creates and manages its own `ScheduledExecutorService` to periodically flush buffered metrics according to the connector-level `flush_interval` option.
   
   This requires the connector to manage an additional background thread and its lifecycle. It can also introduce concurrency between scheduled flushes and the normal Sink write, checkpoint, and close paths.
   
   Zeta now provides engine-level timer flush through `FlushSignal` and `SinkWriter.Context#registerFlushAction`. Prometheus Sink should migrate to this mechanism and stop maintaining its own scheduler.
   
   The expected registration is similar to:
   
   ```java
   if (context != null) {
       context.registerFlushAction(this::flush);
   }
   ```
   
   ### Proposed Changes
   
   1. Pass `SinkWriter.Context` from `PrometheusSink#createWriter()` to `PrometheusWriter`.
   2. Register the Prometheus flush action through `context.registerFlushAction(this::flush)`.
   3. Remove the connector-owned `ScheduledExecutorService`, `ScheduledFuture`, `flushSchedule()`, scheduler startup, and scheduler shutdown logic.
   4. Use `sink.flush.interval` in the job `env` block as the timer interval.
   5. Deprecate or remove the connector-level `flush_interval` option as appropriate, with a clear compatibility and migration note.
   6. Preserve the existing `batch_size` trigger and the final flush in `close()`.
   7. Ensure failures from the registered flush action are propagated to the engine instead of being silently treated as successful flushes.
   8. Update the English and Chinese Prometheus Sink documentation. The documentation should also state that engine-level timer flush is currently supported only by Zeta.
   
   ### Test Requirements
   
   Add a Zeta E2E test that:
   
   - enables engine-level timer flush through `sink.flush.interval` in the job `env` block;
   - configures a `batch_size` larger than the number of records produced;
   - sends a small number of records and then keeps the Source running but idle;
   - disables checkpoint-triggered flushing, or configures a checkpoint interval longer than the test assertion window;
   - verifies that the Prometheus mock endpoint receives the buffered records before the Writer is closed.
   
   The test should cover the complete flow:
   
   `Source timer → FlushSignal → Sink → PrometheusWriter.flush()`
   
   ### Usage Scenario
   
   In low-throughput or temporarily idle streaming jobs, Prometheus Sink may buffer fewer records than `batch_size` for a long time.
   
   With engine-level timer flush, these metrics can be sent without waiting for another record, a checkpoint, or Writer shutdown. The flush action is executed in the normal Sink input-processing path, avoiding concurrency between a connector-owned background thread and the Sink lifecycle.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Prometheus] Migrate PrometheusWriter to the engine-level FlushSignal mechanism [seatunnel]

Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on issue #11773:
URL: https://github.com/apache/seatunnel/issues/11773#issuecomment-5263746459

   Hi, I'd like to work on this. I'll open a PR soon. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Prometheus] Migrate PrometheusWriter to the engine-level FlushSignal mechanism [seatunnel]

Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on issue #11773:
URL: https://github.com/apache/seatunnel/issues/11773#issuecomment-5266360950

   I opened a PR for this: #11778. It moves the Prometheus sink to the engine-level FlushSignal (registerFlushAction), removes the connector's own scheduler thread, makes flush failures propagate, and deprecates the `flush_interval` option in favor of `sink.flush.interval`. Docs (EN + ZH) and a unit test are included. Please take a look, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Prometheus] Migrate PrometheusWriter to the engine-level FlushSignal mechanism [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo closed issue #11773: [Feature][Prometheus] Migrate PrometheusWriter to the engine-level FlushSignal mechanism
URL: https://github.com/apache/seatunnel/issues/11773


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org