You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "surafel58 (via GitHub)" <gi...@apache.org> on 2026/08/12 11:43:48 UTC

[PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

surafel58 opened a new pull request, #11778:
URL: https://github.com/apache/seatunnel/pull/11778

   ### Purpose of this pull request
   
   Closes #11773.
   
   `PrometheusWriter` used to create and manage its own `ScheduledExecutorService` to
   periodically flush buffered metrics based on the connector-level `flush_interval`
   option. This meant the connector owned an extra background thread whose lifecycle
   could run concurrently with the normal Sink write / checkpoint / close paths.
   
   Zeta already provides engine-level timer flush through `FlushSignal` and
   `SinkWriter.Context#registerFlushAction`. This PR migrates the Prometheus Sink to
   that mechanism:
   
   - `PrometheusSink#createWriter()` now passes `SinkWriter.Context` to `PrometheusWriter`.
   - The writer opts in with `context.registerFlushAction(this::flush)`.
   - The connector-owned `ScheduledExecutorService`, `ScheduledFuture`, and
     `flushSchedule()` (plus their startup/shutdown) are removed.
   - The `batch_size` trigger and the final flush in `close()` are preserved.
   - A failed flush is now propagated to the engine instead of being logged and the
     batch silently dropped, so a flush that did not succeed is no longer treated as a
     successful flush.
   - The connector-level `flush_interval` option is deprecated (kept for backward
     compatibility, but ignored). Users should set the engine-level `sink.flush.interval`
     in the job `env` block instead. If `flush_interval` is still configured, the writer
     logs a one-time warning pointing to the new option.
   
   The engine invokes the registered action on the normal Sink input-processing path,
   so there is no connector-owned scheduler thread and no concurrency with the Sink
   lifecycle. As documented, engine-level timer flush is currently supported only by Zeta.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes.
   
   - `flush_interval` (connector option) is deprecated and no longer starts a background
     flush thread. Existing jobs that set it keep validating (the option is still
     accepted) but the value is ignored; a warning is logged. To keep timer-based
     flushing, set `sink.flush.interval` in the job `env` block (Zeta only).
   - The English and Chinese Prometheus Sink docs are updated with a new "Timer Flush"
     section and the deprecation note, and the streaming example now uses
     `sink.flush.interval`.
   
   ### How was this patch tested?
   
   Added `PrometheusWriterTest` (unit test), which verifies:
   
   1. the writer registers a flush action via `SinkWriter.Context#registerFlushAction`,
      and the buffered records are sent only when that action is invoked (the flush
      signal), not on every write;
   2. a failed flush (non-`204` response) propagates a `PrometheusConnectorException`
      instead of being swallowed.
   
   Ran locally with JDK 8:
   
   ```
   ./mvnw -pl seatunnel-connectors-v2/connector-prometheus -am install -DskipTests
   ./mvnw -pl seatunnel-connectors-v2/connector-prometheus test -Dtest=PrometheusWriterTest
   => Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
   ```
   
   `spotless:check` passes for the module.
   
   Note on E2E: the engine already covers the `Source timer -> FlushSignal -> Sink ->
   writer.flush()` path generically in `TimerFlushIT`
   (`seatunnel-engine-e2e/connector-seatunnel-e2e-base`), and this PR adds a connector
   unit test for the Prometheus registration/propagation behavior. I have not added a
   Prometheus-specific Zeta E2E in this PR because it needs a Docker-based Prometheus
   mock endpoint that I could not run reliably in my environment. I am happy to add a
   `PrometheusTimerFlushIT` (modeled on the existing `PrometheusIT` container setup and
   the other connectors' `*TimerFlushIT`) as a follow-up or in this PR if the reviewers
   prefer.
   
   ### Check list
   
   * [x] If necessary, please update the documentation to describe the new feature. (EN + ZH Prometheus sink docs updated.)
   * [ ] If any new Jar binary package adding in your PR, please add License Notice — N/A, no new dependencies.
   * [ ] Update `incompatible-changes.md` — the `flush_interval` option is deprecated but still accepted, so no hard incompatibility; happy to add a note if reviewers want.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#issuecomment-5266621749

   Pushed the changes:
   
   - Removed the flush_interval option from the writer, config, options, and option rule.
   - Added the removal to incompatible-changes.md (EN and ZH) and dropped it from the EN and ZH docs.
   - Added PrometheusTimerFlushIT and prometheus_timer_flush.conf under connector-prometheus-e2e. A streaming FakeSource emits one metric row and stays idle, batch_size is larger than the row count, and the checkpoint interval is long, so the row only reaches Prometheus through the engine timer flush. The test checks the metric arrives while the job is still running, and is limited to Zeta.
   
   The connector unit tests pass locally. The E2E needs Docker, so it runs on CI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#issuecomment-5283085599

   Thanks for your contribution! This is a solid addition to the project, and your attention to detail really shows. Contributors like you are what keep this community moving forward. Feel free to keep the PRs coming — we're always glad to have you here!
   
   Please ignore the following message if you have joined in the Slack~
   ```
   To facilitate focused communication among contributors, we’ve created a brand-new dedicated #contributors channel in official Slack workspace.
   Slack workspace link: https://s.apache.org/seatunnel-slack
   
   This channel is designed for technical discussions, feature design brainstorming, roadmap sync and direct communication with project core Contributors. Everyone who contributes code, docs, test cases or community ideas is warmly invited to join, share thoughts and collaborate together under The Apache Way. Welcome to join 😄
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on code in PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#discussion_r3766115733


##########
seatunnel-connectors-v2/connector-prometheus/src/main/java/org/apache/seatunnel/connectors/seatunnel/prometheus/sink/PrometheusWriter.java:
##########
@@ -42,57 +44,44 @@
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
-import java.util.concurrent.Executors;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.ScheduledFuture;
-import java.util.concurrent.TimeUnit;
 
 @Slf4j
 public class PrometheusWriter extends HttpSinkWriter {
     private final List<Point> batchList;
-    private volatile Exception flushException;
     private final Integer batchSize;
-    private final long flushInterval;
-    private PrometheusSinkConfig sinkConfig;
+    private final PrometheusSinkConfig sinkConfig;
     private final Serializer serializer;
     protected final HttpClientProvider httpClient;
-    private ScheduledExecutorService executor;
-    private ScheduledFuture scheduledFuture;
 
     public PrometheusWriter(
             SeaTunnelRowType seaTunnelRowType,
             HttpParameter httpParameter,
-            ReadonlyConfig pluginConfig) {
+            ReadonlyConfig pluginConfig,
+            SinkWriter.Context context) {
 
         super(seaTunnelRowType, httpParameter);
         this.batchList = new ArrayList<>();
         this.sinkConfig = PrometheusSinkConfig.loadConfig(pluginConfig);
         this.batchSize = sinkConfig.getBatchSize();
-        this.flushInterval = sinkConfig.getFlushInterval();
         this.serializer =
                 new PrometheusSerializer(
                         seaTunnelRowType,
                         sinkConfig.getKeyTimestamp(),
                         sinkConfig.getKeyLabel(),
                         sinkConfig.getKeyValue());
         this.httpClient = new HttpClientProvider(httpParameter);
-        if (flushInterval > 0) {
-            log.info("start schedule submit message,interval:{}", flushInterval);
-            this.executor =
-                    Executors.newScheduledThreadPool(
-                            1,
-                            runnable -> {
-                                Thread thread = new Thread(runnable);
-                                thread.setDaemon(true);
-                                thread.setName("Prometheus-Metric-Sender");
-                                return thread;
-                            });
-            this.scheduledFuture =
-                    executor.scheduleAtFixedRate(
-                            this::flushSchedule,
-                            flushInterval,
-                            flushInterval,
-                            TimeUnit.MILLISECONDS);
+        if (pluginConfig.getOptional(PrometheusSinkOptions.FLUSH_INTERVAL).isPresent()) {

Review Comment:
   The semantics of the actual parameters have been destructively replaced, and I think we should clean up this parameter to avoid ambiguity🎉



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#issuecomment-5283238928

   Thank you. I am glad to contribute, and I will join the community on Slack as well. Looking forward to helping out more.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#issuecomment-5266508136

   I will add a complete E2E test under seatunnel-e2e/connector-prometheus-e2e, using ElasticsearchTimerFlushIT as the reference.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#issuecomment-5280247893

   Thank you @nzw921rx and @DanielLeens for the careful reviews and the clear feedback. I really appreciate the time you put into this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on code in PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#discussion_r3766092340


##########
docs/zh/connectors/sink/Prometheus.md:
##########
@@ -51,7 +51,7 @@ Prometheus 数据接收器把上游数据写入 Prometheus remote write API。
 | retry_backoff_multiplier_ms | Int    | 否       | 100    | 重试退避时间倍数,单位毫秒。 |
 | retry_backoff_max_ms        | Int    | 否       | 10000  | 最大重试退避时间,单位毫秒。 |
 | batch_size                  | Int    | 否       | 1024   | 写入 Prometheus 前最多缓存的行数。 |
-| flush_interval              | Long   | 否       | 300000 | 最大刷新间隔,单位毫秒。 |
+| flush_interval              | Long   | 否       | 300000 | 已废弃且不再生效。连接器不再启动自己的刷新线程。请改为在作业的 `env` 中设置引擎级的 `sink.flush.interval`(详见[定时刷新](#定时刷新))。 |

Review Comment:
   I think we can remove this parameter and document it as an incompatible change in `incompatible-changes.md`, while keeping the English and Chinese versions in sync.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#issuecomment-5278293251

   Thanks for the deep review. Pushed 9608d8a1 with the fixes:
   
   - Issue 1 (Spark/Flink): documented the limit in incompatible-changes.md and the Timer Flush section. sink.flush.interval is Zeta-only; on Spark and Flink there is no periodic flush, so the buffer is sent by batch_size and on close. I kept flush_interval removed (per the earlier review and the other FlushSignal-migrated connectors) and documented the capability loss instead of re-adding a connector thread.
   - Issue 2: corrected the validation wording. A leftover flush_interval key only fails under --check/--dry-run; a direct run ignores it. Added a one-time WARN in the writer when the raw config still has flush_interval.
   - Issue 3: removed the 'on checkpoint' claim from the Timer Flush docs (EN/ZH), since PrometheusWriter does not override prepareCommit().
   - Issue 6: close() now closes the writer's own HttpClientProvider (it shadowed the parent field and leaked).
   - Issue 7: added a comment explaining the context null-check.
   - 2.2: added a unit test that the buffer is still flushed on close when the engine never invokes the flush action (Spark/Flink path).
   
   The red Build is the flaky windows engine test (RestApiSubmitJobConfigShadeDecryptTest / Node failed to start), unrelated to this change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#issuecomment-5289883773

   Heads up on the latest force-push: it is only a rebase onto the current dev to resolve a conflict, no new work on this PR. The follow-up #11797 (apply the batch_size default in loadConfig) was merged first, and both PRs touched PrometheusSinkConfig.loadConfig(), so they overlapped. I kept #11797's batch_size fix from dev and dropped the flush_interval block from this PR. No functional change to this PR's logic, and the connector tests still pass.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#issuecomment-5266458382

   Please add a complete E2E test case under `seatunnel-e2e/connector-prometheus-e2e`. I think `ElasticsearchTimerFlushIT` can be used as a reference.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#issuecomment-5266380505

   <img width="2920" height="1230" alt="image" src="https://github.com/user-attachments/assets/40b953d6-9034-4e29-9353-d3d6be832c38" />
   Thank you for submitting the PR quickly. Please follow the instructions to enable CI🚀


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#issuecomment-5283299865

   You're very welcome, @surafel58 — thanks for engaging so thoroughly with the feedback across every round, including the harder Spark/Flink scoping question. From my side there's nothing left open: head is still `9608d8a1`, CI (`Build`) is green, and my last review's "Ready to merge" stands. The remaining `BLOCKED` state is just the standard maintainer-approval gate, not a technical item pending from me. Nice work on this one.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#issuecomment-5292879550

   Thanks for the detailed re-review @DanielLeens. Pushed e88a0034 with all three non-blocking nits addressed:
   
   - Issue 1 (close() exception masking): reworked close() to keep the flush failure as the primary exception and attach any cleanup failures with addSuppressed, so a client-close IOException can no longer replace the 'Writing records to prometheus failed' message. Added a unit test (closeShouldKeepFlushExceptionWhenHttpClientCloseAlsoThrows) that fails the flush and the client close together and asserts the flush exception surfaces with the IOException suppressed.
   - Issue 2 (stale test comment): reworded the batch_size comment in PrometheusWriterTest to state the real mechanism (unset batch_size now resolves to the default 1024 from #11797, and each test writes a single row well below it).
   - Issue 3 (warning wording): changed 'one-time warning' to 'once per sink writer at startup' in the EN and ZH incompatible-changes entries, and hoisted the removed 'flush_interval' key into a named constant.
   
   Issue 4 (expected conflict with #11738): noted, I will resolve that deliberately whichever of the two lands second, keeping this PR's flush_interval removals and #11738's batch_size lines both intact.
   
   Connector module tests pass locally (9 tests). This push will re-run CI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo merged PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#issuecomment-5295548347

   Thank you @davidzollo for merging, and thank you @nzw921rx and @DanielLeens for the thorough reviews across every round. I learned a lot from the feedback, and it made the change much better. Happy to keep contributing, and I have joined the Slack.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on code in PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#discussion_r3766179581


##########
seatunnel-connectors-v2/connector-prometheus/src/main/java/org/apache/seatunnel/connectors/seatunnel/prometheus/sink/PrometheusWriter.java:
##########
@@ -42,57 +44,44 @@
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
-import java.util.concurrent.Executors;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.ScheduledFuture;
-import java.util.concurrent.TimeUnit;
 
 @Slf4j
 public class PrometheusWriter extends HttpSinkWriter {
     private final List<Point> batchList;
-    private volatile Exception flushException;
     private final Integer batchSize;
-    private final long flushInterval;
-    private PrometheusSinkConfig sinkConfig;
+    private final PrometheusSinkConfig sinkConfig;
     private final Serializer serializer;
     protected final HttpClientProvider httpClient;
-    private ScheduledExecutorService executor;
-    private ScheduledFuture scheduledFuture;
 
     public PrometheusWriter(
             SeaTunnelRowType seaTunnelRowType,
             HttpParameter httpParameter,
-            ReadonlyConfig pluginConfig) {
+            ReadonlyConfig pluginConfig,
+            SinkWriter.Context context) {
 
         super(seaTunnelRowType, httpParameter);
         this.batchList = new ArrayList<>();
         this.sinkConfig = PrometheusSinkConfig.loadConfig(pluginConfig);
         this.batchSize = sinkConfig.getBatchSize();
-        this.flushInterval = sinkConfig.getFlushInterval();
         this.serializer =
                 new PrometheusSerializer(
                         seaTunnelRowType,
                         sinkConfig.getKeyTimestamp(),
                         sinkConfig.getKeyLabel(),
                         sinkConfig.getKeyValue());
         this.httpClient = new HttpClientProvider(httpParameter);
-        if (flushInterval > 0) {
-            log.info("start schedule submit message,interval:{}", flushInterval);
-            this.executor =
-                    Executors.newScheduledThreadPool(
-                            1,
-                            runnable -> {
-                                Thread thread = new Thread(runnable);
-                                thread.setDaemon(true);
-                                thread.setName("Prometheus-Metric-Sender");
-                                return thread;
-                            });
-            this.scheduledFuture =
-                    executor.scheduleAtFixedRate(
-                            this::flushSchedule,
-                            flushInterval,
-                            flushInterval,
-                            TimeUnit.MILLISECONDS);
+        if (pluginConfig.getOptional(PrometheusSinkOptions.FLUSH_INTERVAL).isPresent()) {

Review Comment:
   I will remove the flush_interval option completely from PrometheusWriter, PrometheusSinkConfig, PrometheusSinkOptions, and the option rule, so the parameter is no longer there.



##########
docs/zh/connectors/sink/Prometheus.md:
##########
@@ -51,7 +51,7 @@ Prometheus 数据接收器把上游数据写入 Prometheus remote write API。
 | retry_backoff_multiplier_ms | Int    | 否       | 100    | 重试退避时间倍数,单位毫秒。 |
 | retry_backoff_max_ms        | Int    | 否       | 10000  | 最大重试退避时间,单位毫秒。 |
 | batch_size                  | Int    | 否       | 1024   | 写入 Prometheus 前最多缓存的行数。 |
-| flush_interval              | Long   | 否       | 300000 | 最大刷新间隔,单位毫秒。 |
+| flush_interval              | Long   | 否       | 300000 | 已废弃且不再生效。连接器不再启动自己的刷新线程。请改为在作业的 `env` 中设置引擎级的 `sink.flush.interval`(详见[定时刷新](#定时刷新))。 |

Review Comment:
   I will remove this parameter, add an entry for it in incompatible-changes.md, and keep the English and Chinese docs in sync.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Feature][Connector-V2][Prometheus] Migrate PrometheusWriter to engine-level FlushSignal [seatunnel]

Posted by "surafel58 (via GitHub)" <gi...@apache.org>.
surafel58 commented on code in PR #11778:
URL: https://github.com/apache/seatunnel/pull/11778#discussion_r3766179822


##########
docs/zh/connectors/sink/Prometheus.md:
##########
@@ -51,7 +51,7 @@ Prometheus 数据接收器把上游数据写入 Prometheus remote write API。
 | retry_backoff_multiplier_ms | Int    | 否       | 100    | 重试退避时间倍数,单位毫秒。 |
 | retry_backoff_max_ms        | Int    | 否       | 10000  | 最大重试退避时间,单位毫秒。 |
 | batch_size                  | Int    | 否       | 1024   | 写入 Prometheus 前最多缓存的行数。 |
-| flush_interval              | Long   | 否       | 300000 | 最大刷新间隔,单位毫秒。 |
+| flush_interval              | Long   | 否       | 300000 | 已废弃且不再生效。连接器不再启动自己的刷新线程。请改为在作业的 `env` 中设置引擎级的 `sink.flush.interval`(详见[定时刷新](#定时刷新))。 |

Review Comment:
   Got it, I will remove this parameter, add an entry for it in incompatible-changes.md, and keep the English and Chinese docs in sync.



##########
seatunnel-connectors-v2/connector-prometheus/src/main/java/org/apache/seatunnel/connectors/seatunnel/prometheus/sink/PrometheusWriter.java:
##########
@@ -42,57 +44,44 @@
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
-import java.util.concurrent.Executors;
-import java.util.concurrent.ScheduledExecutorService;
-import java.util.concurrent.ScheduledFuture;
-import java.util.concurrent.TimeUnit;
 
 @Slf4j
 public class PrometheusWriter extends HttpSinkWriter {
     private final List<Point> batchList;
-    private volatile Exception flushException;
     private final Integer batchSize;
-    private final long flushInterval;
-    private PrometheusSinkConfig sinkConfig;
+    private final PrometheusSinkConfig sinkConfig;
     private final Serializer serializer;
     protected final HttpClientProvider httpClient;
-    private ScheduledExecutorService executor;
-    private ScheduledFuture scheduledFuture;
 
     public PrometheusWriter(
             SeaTunnelRowType seaTunnelRowType,
             HttpParameter httpParameter,
-            ReadonlyConfig pluginConfig) {
+            ReadonlyConfig pluginConfig,
+            SinkWriter.Context context) {
 
         super(seaTunnelRowType, httpParameter);
         this.batchList = new ArrayList<>();
         this.sinkConfig = PrometheusSinkConfig.loadConfig(pluginConfig);
         this.batchSize = sinkConfig.getBatchSize();
-        this.flushInterval = sinkConfig.getFlushInterval();
         this.serializer =
                 new PrometheusSerializer(
                         seaTunnelRowType,
                         sinkConfig.getKeyTimestamp(),
                         sinkConfig.getKeyLabel(),
                         sinkConfig.getKeyValue());
         this.httpClient = new HttpClientProvider(httpParameter);
-        if (flushInterval > 0) {
-            log.info("start schedule submit message,interval:{}", flushInterval);
-            this.executor =
-                    Executors.newScheduledThreadPool(
-                            1,
-                            runnable -> {
-                                Thread thread = new Thread(runnable);
-                                thread.setDaemon(true);
-                                thread.setName("Prometheus-Metric-Sender");
-                                return thread;
-                            });
-            this.scheduledFuture =
-                    executor.scheduleAtFixedRate(
-                            this::flushSchedule,
-                            flushInterval,
-                            flushInterval,
-                            TimeUnit.MILLISECONDS);
+        if (pluginConfig.getOptional(PrometheusSinkOptions.FLUSH_INTERVAL).isPresent()) {

Review Comment:
   On it, I will remove the flush_interval option completely from PrometheusWriter, PrometheusSinkConfig, PrometheusSinkOptions, and the option rule, so the parameter is no longer there.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org