You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "DanielLeens (via GitHub)" <gi...@apache.org> on 2026/08/05 11:32:44 UTC
[I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
DanielLeens opened a new issue, #11663:
URL: https://github.com/apache/seatunnel/issues/11663
### Search before asking
- [x] I had searched in the feature issues and found no similar feature requirement.
### Description
SeaTunnel Engine already has the core building blocks for resource-aware scheduling:
- `slot-service.dynamic-slot` controls whether the engine uses dynamic slots.
- `slot-service.slot-allocate-strategy` supports `RANDOM`, `SLOT_RATIO`, and `SYSTEM_LOAD`.
- `SYSTEM_LOAD` can use heartbeat-reported CPU and memory utilization to prefer lower-load workers during slot allocation.
- Engine telemetry can expose Prometheus/OpenMetrics data from `/hazelcast/rest/instance/metrics`.
- Job-level throughput metrics such as source/sink QPS and bytes-per-second are available through REST API v2 `/job-info/:jobId`.
However, SeaTunnel does not currently provide a built-in autoscaling control loop similar to Flink Autoscaler. In production, users who run Zeta on Kubernetes can only build their own scaling policy through external HPA/custom metrics, and users outside Kubernetes do not have a standard SeaTunnel-native scaling mechanism.
It would be useful to add a first-class Zeta autoscaling design and implementation that can evaluate cluster load, slot pressure, pending jobs, and job throughput metrics, then recommend or trigger safe worker/slot capacity changes.
Possible scope:
1. Add an autoscaler module for SeaTunnel Engine / Zeta.
2. Define autoscaler inputs, for example:
- worker CPU and memory utilization,
- slot usage and pending slot requests,
- pending job diagnostics,
- job-level source/sink QPS and BPS,
- queue/backpressure or throughput trend metrics when available.
3. Define scaling decisions:
- scale out when pending jobs or slot pressure remain high for a configurable stabilization window,
- scale in only when workers stay underutilized and no running job would be disrupted,
- avoid oscillation through cooldown, min/max workers, and hysteresis.
4. Expose autoscaler state and decisions through metrics and REST API, so users can observe why a scaling action was or was not taken.
5. Provide a Kubernetes integration path. For example, SeaTunnel could initially expose autoscaling recommendations/custom metrics that HPA or an operator can consume, and later support a controller/operator-managed implementation.
6. Keep backward compatibility: the feature should be disabled by default and should not change current scheduling behavior unless explicitly enabled.
### Usage Scenario
A typical streaming cluster may have different workload patterns during the day:
- During peak hours, source QPS/BPS increases, pending slot requests accumulate, or sink throughput becomes the bottleneck.
- During low-traffic hours, workers remain idle for a long period.
Today, users must manually adjust worker replicas or maintain an external HPA policy without a SeaTunnel-aware view of job topology, slots, and throughput. A built-in autoscaler would make Zeta easier to run in production because scaling decisions could be based on SeaTunnel's own resource and job metrics instead of only generic CPU/memory usage.
Expected benefits:
- Better utilization of Zeta worker resources.
- Faster recovery from workload spikes or accumulated pending jobs.
- Lower operational cost during idle periods.
- A standard scaling model for Kubernetes and non-Kubernetes deployments.
- Clear observability of scaling decisions for operations teams.
### Related issues
No exact duplicate was found during the issue search. This is related to the existing Zeta slot/resource management and telemetry areas, including dynamic slots, slot allocation strategy, Prometheus/OpenMetrics telemetry, and REST API v2 job information.
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #11663:
URL: https://github.com/apache/seatunnel/issues/11663#issuecomment-5215603453
This is a substantial design effort — could you draft a design doc (SEIP-style) covering architecture (where the autoscaler loop runs, e.g., master node), metric collection, decision policy, and configuration surface? That would help move the discussion forward.
Please clarify the initial target environment: will the first iteration only emit scaling recommendations (metrics/REST) for external actuators like Kubernetes HPA/operator, or also actively provision/terminate workers? Actuation differs significantly between K8s and bare-metal/standalone deployments.
Scale-in safety needs careful design: how will the autoscaler drain a worker without disrupting running jobs? Please describe interaction with checkpointing/task restoration, and whether slot decommissioning (blocking new allocations on a draining worker) will be a prerequisite.
<!-- streview-comment:65 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
Posted by "CryoThrust (via GitHub)" <gi...@apache.org>.
CryoThrust commented on issue #11663:
URL: https://github.com/apache/seatunnel/issues/11663#issuecomment-5536217306
A design point that may help keep phase-1 reviewable: separate the recommendation loop from actuation. The first slice could expose a deterministic recommendation record (target worker/slot delta, observed utilization window, confidence, cooldown, and reason) without changing membership. Before enabling actuation, the design should define: (1) whether utilization is aggregated per worker or per slot, including skew handling; (2) the exact control-loop inputs and sampling/averaging window; (3) a fencing/version token so a stale recommendation cannot scale after a newer topology change; (4) scale-in drain/decommission semantics for running pipelines; and (5) checkpoint/restore behavior when capacity changes mid-recovery. This would let the recommendation path be tested independently from cluster membership changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
Posted by "LeonxLJX (via GitHub)" <gi...@apache.org>.
LeonxLJX commented on issue #11663:
URL: https://github.com/apache/seatunnel/issues/11663#issuecomment-5493477786
I'd like to take this one (`[Feature][Zeta] Add autoscaling support for SeaTunnel Engine`). I'll look into the root cause and follow up with a PR. (claiming via @LeonxLJX)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
Posted by "JeremyXin (via GitHub)" <gi...@apache.org>.
JeremyXin commented on issue #11663:
URL: https://github.com/apache/seatunnel/issues/11663#issuecomment-5492637323
> I'd like to take this one (`[Feature][Zeta] Add autoscaling support for SeaTunnel Engine`). I'll look into the root cause and follow up with a PR. (claiming via [@LeonxLJX](https://github.com/LeonxLJX))
Thanks for paying attention to this issue. I am already designing and implementing this problem. I will submit the STIP and the corresponding implementation pr. Welcome to assist in reviewing the related PRs in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
Posted by "JeremyXin (via GitHub)" <gi...@apache.org>.
JeremyXin commented on issue #11663:
URL: https://github.com/apache/seatunnel/issues/11663#issuecomment-5493601732
@LeonxLJX Hi, Thanks for your attention to this issue. I have taken this issue and already thought through, designed the initial implementation plan. Welcome to assist in reviewing the related PRs in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
Posted by "LeonxLJX (via GitHub)" <gi...@apache.org>.
LeonxLJX commented on issue #11663:
URL: https://github.com/apache/seatunnel/issues/11663#issuecomment-5493503384
I'd like to take this one (`[Feature][Zeta] Add autoscaling support for SeaTunnel Engine`). I'll dig into the root cause and follow up with a PR shortly. (claiming via @LeonxLJX)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
Posted by "JeremyXin (via GitHub)" <gi...@apache.org>.
JeremyXin commented on issue #11663:
URL: https://github.com/apache/seatunnel/issues/11663#issuecomment-5192799795
@davidzollo Hi, I want to try and take on this task, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
Posted by "LeonxLJX (via GitHub)" <gi...@apache.org>.
LeonxLJX commented on issue #11663:
URL: https://github.com/apache/seatunnel/issues/11663#issuecomment-5502694791
Thanks for the heads-up @JeremyXin! Since you've already taken this and worked out the implementation plan, I'll step back rather than duplicate the effort.
One thing I'd flag while you're designing it, in case it's useful: for autoscaling it's worth deciding early whether scaling decisions are made on **slot-level utilization** or **worker-level utilization**, because the two can disagree — a worker can look busy on slots while being idle on CPU/memory (or vice versa) depending on whether tasks are CPU- or IO-bound. Pinning that down up front tends to avoid churn in the policy later.
Happy to review the PR when it's up if that's helpful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #11663:
URL: https://github.com/apache/seatunnel/issues/11663#issuecomment-5366254711
@JeremyXin — that plan sounds right, thanks. The focus areas you listed (phase-1 scope and whether it is recommendation-only, control-loop inputs and decision policy, the actuation boundary per environment, and the scale-in safety model with draining/decommission and checkpoint/restore interaction) match the questions we need settled before implementation starts. Once the STIP is up, please link it here so we can review it and narrow the first reviewable phase.
@zhang-arvin — thanks for the interest, but this issue is already assigned to @JeremyXin, who is preparing the design proposal. Once the STIP is posted, feedback on the design would be very welcome.
<!-- streview-comment:435 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
Posted by "LeonxLJX (via GitHub)" <gi...@apache.org>.
LeonxLJX commented on issue #11663:
URL: https://github.com/apache/seatunnel/issues/11663#issuecomment-5492524991
I'd like to take this one (`[Feature][Zeta] Add autoscaling support for SeaTunnel Engine`). I'll look into the root cause and follow up with a PR. (claiming via @LeonxLJX)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on issue #11663:
URL: https://github.com/apache/seatunnel/issues/11663#issuecomment-5351136963
I'd like to work on this issue. Could you please assign it to me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
Posted by "JeremyXin (via GitHub)" <gi...@apache.org>.
JeremyXin commented on issue #11663:
URL: https://github.com/apache/seatunnel/issues/11663#issuecomment-5313941116
Thanks for the guidance.
I will first analyze the current requirement and existing Zeta capabilities around slot scheduling, worker/resource management,
metrics, and recovery behavior, then prepare an STIP/design proposal before starting implementation.
The design draft will focus on:
- the phase-1 scope and whether it should be recommendation-only first,
- the control-loop inputs and decision policy,
- the actuation boundary for different environments,
- and the scale-in safety model, including draining/decommission and checkpoint/restore interaction.
After the STIP is reviewed and the scope is aligned, I will move on to implementation in incremental phases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #11663:
URL: https://github.com/apache/seatunnel/issues/11663#issuecomment-5203022967
This is a significant architectural change; please raise it on the dev mailing list and consider drafting a SEIP/design doc covering the control loop, metric inputs, decision policy, and actuation mechanism before implementation starts.
Could you clarify the actuation scope: will the autoscaler only emit recommendations (metrics/REST) that external systems like Kubernetes HPA or an operator act on, or will it directly provision/decommission workers? Direct provisioning outside Kubernetes needs a pluggable executor design.
Scale-in safety needs special attention in the design: how running jobs/pipelines on a to-be-removed worker are drained or rescheduled (checkpoint-based restore vs. graceful task migration), plus cooldown/hysteresis defaults to avoid oscillation. A phased plan (metrics + advisory first, actuation later) would make review easier.
<!-- streview-comment:56 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11663:
URL: https://github.com/apache/seatunnel/issues/11663#issuecomment-5210722685
Thanks for picking this up.
This is the right kind of topic to treat as a design-first feature rather than jumping directly into implementation. The main questions to settle up front are still:
1. whether the first phase is recommendation-only or includes direct worker actuation;
2. how scale-in remains safe for running jobs;
3. what metrics and cooldown rules form the stable control-loop contract.
Please continue with the design/STIP path first, then we can use that to narrow the implementation scope into a reviewable first phase. I've assigned the issue to you so the ownership is visible while the design is being refined.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Zeta] Add autoscaling support for SeaTunnel Engine workers and slots [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11663:
URL: https://github.com/apache/seatunnel/issues/11663#issuecomment-5541737290
Thanks @CryoThrust. The recommendation-versus-actuation boundary is the right way to keep the first slice reviewable. I checked current dev and the open PR set: there is still no built-in AutoScaler or active autoscaling implementation, so this umbrella should remain design-first.
Please use #12087 as the formal phased STIP and keep this issue as the umbrella. The STIP now needs to make the active-master fencing token, recommendation generation, sample freshness and time-window semantics, stale/duplicate rejection, and scale-in drain dependency explicit. Phase 1 should publish idempotent recommendations only and must not alter membership or running-job topology. Implementation should begin only after those contracts and handover/restart tests are agreed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org