You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "QiChunHui (via GitHub)" <gi...@apache.org> on 2026/06/02 09:06:30 UTC

[I] [Bug] [Zeta] Batch job occasionally hangs in RUNNING forever under high-frequency job submission [seatunnel]

QiChunHui opened a new issue, #10997:
URL: https://github.com/apache/seatunnel/issues/10997

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
   
   
   ### What happened
   
   In production we hit an intermittent problem: a **BATCH** job (JDBC MySQL source → JDBC
   `MultiTableSink`) finished reading all source data and the bounded reader closed normally, but the
   whole job then **stays in `RUNNING` forever and never turns `FINISHED`**. It cannot finish on its
   own — the only way to clear it is a manual cancel or a master restart.
   
   This does not happen every time; it shows up intermittently under a high job-submission load.
   Below are the clues we collected. We'd like the community's help to analyze the possible causes and
   directions for improvement.
   
   Environment & workload characteristics
   
   - SeaTunnel version: 2.3.13, Zeta engine.
   - Deployment: separated master/worker, 2 masters (HA 32C+48G heap) + 8 workers(3 32C+48G heap + 5 16C+20G heap).
   - **Workload**: the platform has **1000+ data-collection jobs**, submitted periodically by a
     scheduler, with **on average several dozen job submissions per minute** (higher at peak). So the
     cluster runs continuously under "sustained high-frequency submission + many small concurrent jobs".
   - The job that hung is itself trivial: a single bounded table read, BATCH mode, and **no
     `checkpoint.interval` configured (i.e. checkpoint disabled)**.
   
   ### SeaTunnel Version
   
   SeaTunnel version: 2.3.13, Zeta engine
   
   ### SeaTunnel Config
   
   ```conf
   for master:
   
   -- seatunnel.yaml
   seatunnel:
     engine:
       classloader-cache-mode: true
       history-job-expire-minutes: 30
       backup-count: 0
       queue-type: blockingqueue
       job-metrics-backup-interval: 60
       print-execution-info-interval: 60
       print-job-metrics-info-interval: 1800
       job-metrics-partition-count: 4
       job-schedule-strategy: WAIT
       slot-service:
         dynamic-slot: false
       checkpoint:
         interval: 60000
         timeout: 600000
         storage:
           ...
       telemetry:
         metric:
           enabled: false
         logs:
           scheduled-deletion-enable: false
       http:
         enable-http: true
         port: 8010
         enable-dynamic-port: false
   
   -- hazelcast-master.yaml
   hazelcast:
     cluster-name: platform
     network:
       rest-api:
         enabled: false
         endpoint-groups:
           CLUSTER_WRITE:
             enabled: true
           DATA:
             enabled: true
       join:
         ...
       port:
         auto-increment: false
         port: 5801
     properties:
       hazelcast.invocation.max.retry.count: 20
       hazelcast.tcp.join.port.try.count: 30
       hazelcast.logging.type: log4j2
       hazelcast.operation.generic.thread.count: 50
       hazelcast.operation.call.timeout.millis: 60000
       hazelcast.operation.backup.timeout.millis: 60000
       hazelcast.heartbeat.failuredetector.type: phi-accrual
       hazelcast.heartbeat.interval.seconds: 5
       hazelcast.max.no.heartbeat.seconds: 300
       hazelcast.heartbeat.phiaccrual.failuredetector.threshold: 10
       hazelcast.heartbeat.phiaccrual.failuredetector.sample.size: 200
       hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis: 100
       hazelcast.operation.thread.count: 32
       hazelcast.io.thread.count: 32
     member-attributes:
       group:
         type: string
         value: default
       team:
         type: string
         value: default
   
   for worker:
   
   -- seatunnel.yaml
   seatunnel:
     engine:
       classloader-cache-mode: true
       history-job-expire-minutes: 30
       backup-count: 0
       queue-type: blockingqueue
       job-metrics-backup-interval: 60
       print-execution-info-interval: 60
       print-job-metrics-info-interval: 1800
       job-metrics-partition-count: 4
       job-schedule-strategy: WAIT
       slot-service:
         dynamic-slot: false
         slot-num: 96
         slot-allocate-strategy: slot_ratio
       checkpoint:
         interval: 60000
         timeout: 600000
         storage:
           ...
       telemetry:
         metric:
           enabled: false
         logs:
           scheduled-deletion-enable: false
       http:
         enable-http: true
         port: 8010
         enable-dynamic-port: false
   
   -- hazelcast-worker.yaml
   hazelcast:
     cluster-name: platform
     network:
       join:
         ...
       port:
         auto-increment: false
         port: 5802
     properties:
       hazelcast.invocation.max.retry.count: 20
       hazelcast.tcp.join.port.try.count: 30
       hazelcast.logging.type: log4j2
       hazelcast.io.thread.count: 32
       hazelcast.operation.generic.thread.count: 50
       hazelcast.operation.call.timeout.millis: 60000
       hazelcast.operation.backup.timeout.millis: 60000
       hazelcast.heartbeat.failuredetector.type: phi-accrual
       hazelcast.heartbeat.interval.seconds: 5
       hazelcast.max.no.heartbeat.seconds: 300
       hazelcast.heartbeat.phiaccrual.failuredetector.threshold: 10
       hazelcast.heartbeat.phiaccrual.failuredetector.sample.size: 200
       hazelcast.heartbeat.phiaccrual.failuredetector.min.std.dev.millis: 100
       hazelcast.operation.thread.count: 32
     member-attributes:
       group:
         type: string
         value: default
       team:
         type: string
         value: default
   ```
   
   ### Running Command
   
   ```shell
   start up: ./bin/seatunnel-cluster.sh -d -r master  ./bin/seatunnel-cluster.sh -d -r worker
   submit job: /submit-job API use the JSON config
   ```
   
   ### Error Exception
   
   ```log
   ## Clues we found
   
   ### Clue 1: state of the three task threads while hung (from jstack)
   
   | Role | Thread-stack location | Meaning |
   |---|---|---|
   | Source reader (taskGroup 2) | looping in `SourceFlowLifeCycle.collect:194` (sleep) | reader's internal `prepareClose=true`, but the **task-level `prepareCloseStatus=false`, still in RUNNING** |
   | Sink side (taskGroup 2) | blocked in `IntermediateBlockingQueue.collect:54` (poll) | waiting forever for data / a close marker on the intermediate queue that never arrives |
   | SplitEnumerator (taskGroup 1) | `SourceSplitEnumeratorTask.stateProcess:350` (PREPARE_CLOSE) | already detected all readers finished, already sent `LastCheckpointNotifyOperation` to master, waiting for `closeCalled` |
   
   The source finished reading at `17:48:42`; after that there is **almost no further log** for this
   job on either worker or master.
   
   ### Clue 2: how a bounded BATCH job is supposed to finish (for reference)
   
   Normally a bounded BATCH job is closed by a "final checkpoint":
   
   1. reader finishes → `signalNoMoreElement()` → notifies the enumerator;
   2. enumerator detects all readers finished → sends `LastCheckpointNotifyOperation` to master →
      `CheckpointCoordinator.readyToClose()`;
   3. master triggers the final checkpoint `tryTriggerPendingCheckpoint(COMPLETED_POINT_TYPE)`;
   4. this close barrier flows enumerator → reader → sink, each acks, and it sets `prepareCloseStatus`
      to true on every task, advancing each task `RUNNING → PREPARE_CLOSE → CLOSED`;
   5. once all complete, master tells the tasks to close and the job finishes.
   
   In the hung job, step 3 appears to have happened (the enumerator is already in PREPARE_CLOSE), but
   the **source reader's `prepareCloseStatus` was never set** — i.e. the close-barrier handshake never
   completed / never reached the reader.
   
   ### Clue 3: in the same time window, worker↔master Hazelcast communication was massively timing out and the connection was flapping
   
   In the logs of the worker that ran the source/sink (172.18.1.145) and the active master
   (172.18.1.143), **right in the window when the job tried to close (around source close at
   `17:48:42`)**, cluster-internal communication was badly degraded:
   
   - the master logged **1419 `Slow operation detected: ReportMetricsOperation`** warnings within one
     hour; `ReportMetricsOperation` (partitionId=-1) was hogging almost all generic-operation threads;
   - several operations from the worker to the master were **rejected by the 60s
     operation-call-timeout**, e.g. `JobEventReportOperation` (17:46:36, 17:48:03, with the connection
     already `alive=false`) and `ReportMetricsOperation` (17:48:59, `ERROR ... failed to update metrics`);
   - the TCP cluster connection worker(145)↔master(143) was **re-initialized 6 times between 17:39 and
     17:56** (each reconnect drops the operations in flight on that connection); meanwhile **no
     Hazelcast cluster membership change occurred** (no master failover) — only the connection flapped;
   - shortly after, `SourceRegisterOperation` was flagged as a slow operation (17:50:37) and then
     **failed after 30 retries** (17:51:27), confirming operations were being lost during this period.
   
   ### Clue 4: the close handshake relies on a chain of Hazelcast operations
   
   The close flow in step 2 is, in implementation, a chain of cross-node Hazelcast operations:
   `LastCheckpointNotifyOperation` (worker→master), `CheckpointBarrierTriggerOperation`
   (master→enumerator), `BarrierFlowOperation` (enumerator→reader), `TaskAcknowledgeOperation`
   (reader→master). If any one of these is lost (in a connection drop / 60s-timeout rejection) and is
   not retried, the close flow cannot complete.
   
   We also noticed that when `checkpoint.interval` is not configured (checkpoint disabled), this final
   checkpoint used to close the bounded source does not seem to have the same timeout detection that a
   normal checkpoint has (this is only an observation; we're not sure whether it is by design).
   
   ## Our questions / what we'd like the community to look at
   
   1. Under our "1000+ jobs, several dozen submissions per minute" high-frequency scenario,
      `ReportMetricsOperation` and similar internal operations saturate the master's operation threads
      and cause the worker↔master connection to repeatedly time out / reconnect — is this a known
      scalability bottleneck? How does the community usually mitigate it?
   2. For this phenomenon, what does the community think are the **possible root causes** and the
      **directions for future optimization**? (Whether at the config, deployment, or engine level —
      we'd love to hear any thoughts.)
   ```
   
   ### Zeta or Flink or Spark Version
   
   _No response_
   
   ### Java or Scala Version
   
   Java: 8
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Zeta] Batch job occasionally hangs in RUNNING forever under high-frequency job submission [seatunnel]

Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on issue #10997:
URL: https://github.com/apache/seatunnel/issues/10997#issuecomment-4624341923

   Thanks for reporting this issue.
   As a quick mitigation, since the current configuration uses `job-metrics-partition-count=4`, it may be worth testing a much larger value, such as 1000 or 2000, on the master side, to reduce metrics IMap contention under high job-submission load.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Zeta] Batch job occasionally hangs in RUNNING forever under high-frequency job submission [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10997:
URL: https://github.com/apache/seatunnel/issues/10997#issuecomment-4627269639

   I rechecked the scenario you described, and this still looks worth tracking as a real engine-side completion problem, but not the same path as #10836.
   
   The important difference is the one you already called out: #10836 fixes the master-failover case where completion bookkeeping is lost after leadership takeover. In your report, there was no master switch, so that fix alone does not explain a batch job staying in `RUNNING` after the bounded reader has already finished.
   
   At this point, the most likely direction is the master-side finalization / bookkeeping path under sustained high-frequency submission pressure, rather than the bounded source itself. The mitigation suggested by @dybyte (`job-metrics-partition-count` much larger than `4`) is also worth testing, because your current cluster shape is heavily skewed toward many small concurrent jobs.
   
   To narrow this down further, could you please add these three pieces of information:
   1. whether the same workload still reproduces on the latest `dev` branch build
   2. the master log around the exact time the job should transition from `RUNNING` to `FINISHED` (especially `JobMaster` / finalization / close-related lines)
   3. whether increasing `job-metrics-partition-count` to a much larger value changes the reproduction rate
   
   If you can share those, we can separate "capacity / contention side effect" from a deeper completion-state bug much more confidently.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Zeta] Batch job occasionally hangs in RUNNING forever under high-frequency job submission [seatunnel]

Posted by "nielifeng (via GitHub)" <gi...@apache.org>.
nielifeng commented on issue #10997:
URL: https://github.com/apache/seatunnel/issues/10997#issuecomment-5086473326

   I completed a controlled validation of this issue on current `dev` commit
   `dbead9b` using a separated Zeta Master/Worker deployment.
   
   ### Patch direction
   
   The candidate fix changes only `ReportMetricsOperation` scheduling:
   
   ```java
   public class ReportMetricsOperation extends TracingOperation
           implements IdentifiedDataSerializable, Offloadable {
   
       @Override
       public String getExecutorName() {
           return Offloadable.OFFLOADABLE_EXECUTOR;
       }
   }
   ```
   
   The payload, serializer ID, invocation semantics, metrics storage, and
   `SeaTunnelServer.updateMetrics()` behavior remain unchanged.
   
   ### Results
   
   - A focused regression confirms that `ReportMetricsOperation` is offloaded from
     Hazelcast generic-operation threads and uses the built-in offloadable executor.
   - The existing metrics IMap integration path still stores metrics correctly.
   - Matched unpatched and patched bounded-batch runs completed without metrics
     failure/interruption, failed jobs, or persistent pending jobs.
   - A controlled reduced generic-operation-thread comparison also completed on
     both versions; the patched run kept the metrics operation off generic threads.
   - Both versions reached the same workload-capacity knee at a higher submission
     rate, so that behavior was treated as capacity evidence rather than a product
     failure.
   - A longer patched run remained healthy before a separately planned Worker fault.
   
   The original production-style permanent RUNNING hang was **not directly
   reproduced** in this lab environment. Therefore, the cluster result is
   directional evidence for isolating metrics work from control-plane generic
   threads, not a definitive before/after reproduction of the reported hang.
   
   The planned Worker restart exposed an independent checkpoint recovery NPE in
   `CheckpointCoordinator.isNoErrorCompleted()`. It is unrelated to
   `ReportMetricsOperation` offloading and is tracked separately in #11563.
   
   Local targeted tests for the metrics offload and the independent checkpoint
   regression pass. No PR has been submitted yet.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Zeta] Batch job occasionally hangs in RUNNING forever under high-frequency job submission [seatunnel]

Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on issue #10997:
URL: https://github.com/apache/seatunnel/issues/10997#issuecomment-4602450997

   @QiChunHui Thank you for your report. 
   This issue should have already been addressed by this PR https://github.com/apache/seatunnel/pull/10836 repair
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Zeta] Batch job occasionally hangs in RUNNING forever under high-frequency job submission [seatunnel]

Posted by "QiChunHui (via GitHub)" <gi...@apache.org>.
QiChunHui commented on issue #10997:
URL: https://github.com/apache/seatunnel/issues/10997#issuecomment-4609733406

   > [@QiChunHui](https://github.com/QiChunHui)感谢您的报告。 这个问题应该已经在 PR [#10836](https://github.com/apache/seatunnel/pull/10836)修复中解决了。
   
   Thanks @nzw921rx. I don't think #10836 applies to my case.
   
   #10836 only triggers during master failover, and the fix persists the readyToCloseStartingTask set that a new master loses on takeover.
   
   But no master switch happened in my case. As reported, the worker-to-master TCP connection was re-initialized 6 times without any cluster membership change — same master throughout, no leader election. The failover path that #10836 fixes is never hit here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Zeta] Batch job occasionally hangs in RUNNING forever under high-frequency job submission [seatunnel]

Posted by "nzw921rx (via GitHub)" <gi...@apache.org>.
nzw921rx commented on issue #10997:
URL: https://github.com/apache/seatunnel/issues/10997#issuecomment-4610069096

   > 
   > Thanks [@nzw921rx](https://github.com/nzw921rx). I don't think [#10836](https://github.com/apache/seatunnel/pull/10836) applies to my case.
   > 
   > [#10836](https://github.com/apache/seatunnel/pull/10836) only triggers during master failover, and the fix persists the readyToCloseStartingTask set that a new master loses on takeover.
   > 
   > But no master switch happened in my case. As reported, the worker-to-master TCP connection was re-initialized 6 times without any cluster membership change — same master throughout, no leader election. The failover path that [#10836](https://github.com/apache/seatunnel/pull/10836) fixes is never hit here.
   
   1. you are right, Please try adjusting the following parameters to see if they can alleviate the issue
   
   ```yaml
   job-metrics-backup-interval: 300           # 60s → 300s, reducing scheduled reporting by 80%
   hazelcast.operation.generic.thread.count: 100 # 50 → 100, expanding thread pool
   ```
   It is also recommended to configure 'checkpoint. interval' (such as 300000ms) for BATCH job, so that the final checkpoint will have timeout protection, and even if the barrier is lost, it will not be permanently suspended.
   
   2. I analyzed the root cause from the source code level. The core issue is that the `ReportMetricsOperation` implementation exhausts the Hazelcast generic-operation thread pool under large-scale workloads, which in turn triggers connection flapping.
   
      The specific mechanism: `TaskExecutionService.collectLocalMetricsMap()` serializes and sends both `finishedExecutionContexts` (completed tasks) and `executionContexts` (running tasks) in full to the master on every report, using a blocking `invoke.get()` call. Under your 1000+ job high-frequency submission/completion scenario, the 8 workers' scheduled reports (every 60s) plus the event-driven reports triggered on each task completion result in 40-60 large-payload `ReportMetricsOperation`s hitting the master's 50 generic threads per minute. These operations execute slowly due to distributed IMap writes, saturating the thread pool. Once full, critical operations like `CheckpointBarrierTriggerOperation` and `BarrierFlowOperation` queue up and time out, ultimately triggering connection rebuilds that discard all in-flight operations.
   
       your logs — 1419 slow operation warnings in one hour and 6 connection rebuilds — perfectly confirm this chain.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] [Zeta] Batch job occasionally hangs in RUNNING forever under high-frequency job submission [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10997:
URL: https://github.com/apache/seatunnel/issues/10997#issuecomment-5098260301

   Thanks for the controlled validation and for splitting the findings.
   
   I agree with the current classification: the original #10997 remains a real Zeta control-plane / completion-path scalability issue, but the latest evidence narrows the immediate patch to isolating metrics reporting from Hazelcast generic-operation threads rather than proving a deterministic before/after reproduction of the permanent `RUNNING` hang.
   
   PR #11564 is now the right review path for the `ReportMetricsOperation` part. The important boundary is that it keeps the metrics payload, serializer, storage, and `SeaTunnelServer.updateMetrics()` behavior unchanged, while moving the expensive work off the generic-operation thread. The separate Worker-restart checkpoint recovery NPE is correctly tracked in #11563 and #11565.
   
   I removed the stale `help wanted` label because there is now an active same-root-cause PR. Next, let us use #11564 current-head Build and maintainer review as the implementation gate, and keep this issue open until we know whether that patch is sufficient or whether final-checkpoint retry / timeout handling still needs a follow-up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org