You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "gangavhi (via GitHub)" <gi...@apache.org> on 2026/07/12 20:48:52 UTC
[PR] [kubernetes] Graceful TaskManager termination + fix wrong-signal TODO in ResourceManager [flink]
gangavhi opened a new pull request, #28729:
URL: https://github.com/apache/flink/pull/28729
## What is the purpose of the change
On Kubernetes, a TaskManager pod is handled the same way whether it's lost voluntarily (node
drain, spot/preemptible reclaim, rolling upgrade) or crashes outright: the ResourceManager only
learns it's gone once the heartbeat times out (50s by default), and `SIGTERM` tears the process
down immediately with no chance for running tasks to make further progress.
This PR adds an opt-in `kubernetes.taskmanager.termination-grace-period` option that gives a
TaskManager pod a real, bounded grace window on voluntary eviction, and separately fixes a
long-standing `TODO` in `ResourceManager` where the "notify the TaskExecutor to stop" step existed
but sent the wrong instruction.
**Not a Jira ticket yet** — happy to file one (or have this PR converted into the trigger for one)
based on maintainer feedback on scope/approach; wanted to put the code up for review first rather
than guess at a title before there's been any discussion.
## Brief change log
- `kubernetes.taskmanager.termination-grace-period` (new `ConfigOption<Duration>`) +
`TerminationGracePeriodDecorator`: sets `terminationGracePeriodSeconds` and a `preStop` hook on
the TaskManager pod, never overriding a value already present in the user's pod template.
- The `preStop` hook sends `SIGUSR2` to the main process, which `TaskManagerRunner` routes to the
new `TaskExecutor#prepareForTermination()`: disconnects from the ResourceManager proactively
instead of waiting to be timed out, so the slot is freed immediately. The hook then sleeps for
most of the remaining grace period before returning (Kubernetes only sends `SIGTERM` once
`preStop` returns), so already-running tasks keep making progress in the meantime.
- `ResourceManager#closeTaskManagerConnection` carried
`// TODO :: suggest failed task executor to stop itself`. It already calls back into the
presumed-dead TaskExecutor via `TaskExecutorGateway#disconnectResourceManager` — but that RPC
tells the TaskExecutor to *reconnect*, which is the wrong instruction for a TaskExecutor the
ResourceManager has already unregistered and told every JobMaster to fail. Adds
`TaskExecutorGateway#fenceAndStop`, called here instead: tells the TaskExecutor to stop trying
to reconnect and to proactively fail its own currently-running tasks.
- Docs: new "Graceful TaskManager Termination" section in `native_kubernetes.md`; regenerated
`kubernetes_config_configuration.html`.
Deliberately **out of scope**, to keep this reviewable: equivalent JobManager-side graceful
shutdown (needs its own `SIGUSR2` handler first — sending it unhandled terminates a process by
POSIX default, the opposite of graceful), and a TaskManager-side proactive self-fencing/isolation
timeout for the case where the ResourceManager's own notification can't be delivered at all (a
true network partition) — that changes failure-detection *timing*, not just teardown *messaging*,
and seems like it deserves its own dev@ discussion rather than being bundled here.
## Verifying this change
- New unit tests: `TerminationGracePeriodDecoratorTest` (pod-spec assertions: option unset is a
no-op, grace period + `preStop` set correctly, neither overrides an existing pod-template value,
sleep duration clamps to zero for very short grace periods), plus two new tests in
`TaskExecutorTest` covering `prepareForTermination()` (disconnects and does not reconnect) and
`fenceAndStop` (fails locally-running tasks and disconnects the JobMaster connection, reusing
the existing `runJobManagerHeartbeatTest` harness).
- Full `flink-kubernetes` module test suite and the full `TaskExecutorTest` /
`TaskManagerRunnerTest` classes pass (checked for regressions since a shared gateway interface,
`TaskExecutorGateway`, gained a new method).
- `./mvnw spotless:check` and `./mvnw checkstyle:check` clean on touched modules.
- Manually verified end-to-end on a local `kind` cluster with a build of this branch: submitted a
checkpointed streaming job, then `kubectl delete pod` on the TaskManager. Confirmed via
timestamped logs that the ResourceManager's proactive-disconnect log line lands within ~1s of
the `SIGUSR2` signal (well before the 50s heartbeat timeout would fire), and that the pod (and
its task) stays alive for the configured grace window before `SIGTERM` arrives — compared
against a baseline session cluster without the option set, where `SIGTERM` fires the instant
`kubectl delete pod` is issued today.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes) — adds
one new `ConfigOption` to `KubernetesConfigOptions` (`@PublicEvolving`); purely additive,
defaults to today's behavior when unset.
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes) — TaskManager pod teardown on Kubernetes; see change log above.
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs, generated config reference)
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (Claude Code / Claude Sonnet 5)
<!-- Generated-by: Claude Code (Claude Sonnet 5) -->
I worked with Claude Code through the design, implementation, and verification of this change,
including tracing the existing heartbeat/RM/TM code paths to scope it, and reviewing the kind
cluster test results (which surfaced and required fixing two real bugs during development: a
`preStop` script that fired the signal but didn't actually block for the grace period, and a race
condition in one of the new unit tests). I'm responsible for the correctness of everything in this
PR regardless of tooling used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [kubernetes] Graceful TaskManager termination + fix wrong-signal TODO in ResourceManager [flink]
Posted by "gangavhi (via GitHub)" <gi...@apache.org>.
gangavhi commented on PR #28729:
URL: https://github.com/apache/flink/pull/28729#issuecomment-5104900390
> Can you file a JIRA for this change and work on fixing the CI failures before opening the PR for review.
Thank you, I have created issue - https://issues.apache.org/jira/browse/FLINK-40252
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [kubernetes] Graceful TaskManager termination + fix wrong-signal TODO in ResourceManager [flink]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28729:
URL: https://github.com/apache/flink/pull/28729#issuecomment-6435577264
This PR is being marked as stale since it has not had any activity in the last 90 days.
If you would like to keep this PR alive, please leave a comment asking for a review.
If the PR has merge conflicts, update it with the latest from the base branch.
If you are having difficulty finding a reviewer, please reach out to the
community, contact details can be found here: https://flink.apache.org/what-is-flink/community/
If this PR is no longer valid or desired, please feel free to close it.
If no activity occurs in the next 30 days, it will be automatically closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [kubernetes] Graceful TaskManager termination + fix wrong-signal TODO in ResourceManager [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28729:
URL: https://github.com/apache/flink/pull/28729#issuecomment-4952740222
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b190e69a89542df1843c103613f09d9f98aa68ae",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b190e69a89542df1843c103613f09d9f98aa68ae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b190e69a89542df1843c103613f09d9f98aa68ae UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [kubernetes] Graceful TaskManager termination + fix wrong-signal TODO in ResourceManager [flink]
Posted by "spuru9 (via GitHub)" <gi...@apache.org>.
spuru9 commented on PR #28729:
URL: https://github.com/apache/flink/pull/28729#issuecomment-4955297494
Can you file a JIRA for this change and work on fixing the CI failures before opening the PR for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org