You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "FrankYang0529 (via GitHub)" <gi...@apache.org> on 2026/10/14 05:00:20 UTC
[PR] Make Java SDK context signal-aware [airflow]
FrankYang0529 opened a new pull request, #74690:
URL: https://github.com/apache/airflow/pull/74690
### Why
- When Airflow wants a running task to stop, for example after a user marks the task instance as failed, the supervisor sends the task's process SIGTERM. If the process is still running after `[core] killed_task_cleanup_time` seconds, the supervisor sends SIGKILL. The Go and TypeScript SDKs pass that SIGTERM to the task through its context, so the task can stop work it started, such as a job in an external system.
### How
- While a task runs, the SDK replaces the JVM's SIGINT and SIGTERM handlers through `sun.misc.Signal`. It puts the old handlers back after the supervisor has acknowledged the task's result.
- `Context.isCancelled()` and `Context.awaitCancellation(Duration)` tell a task that Airflow asked it to stop. `awaitCancellation` can replace `Thread.sleep` in a polling loop.
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
For user-facing UI changes, please attach before/after screenshots (or a short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
closes: #72680
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [X] Yes - Claude Code
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make Java SDK context signal-aware [airflow]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on PR #74690:
URL: https://github.com/apache/airflow/pull/74690#issuecomment-6381110572
I created a follow-up PR to fix the hang cause. https://github.com/apache/airflow/pull/75538
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make Java SDK context signal-aware [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #74690:
URL: https://github.com/apache/airflow/pull/74690#discussion_r4333120653
##########
java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Context.kt:
##########
@@ -127,6 +129,54 @@ data class Context(
/** Registration of the executing task; resolves wired data inputs. */
internal var taskDef: TaskDef? = null
+ // The value is read once, when the context is made. A context that read Cancellation.running on
+ // every call would stop seeing the stop request once the server sets the field back to null, for
+ // example in a shutdown hook.
Review Comment:
Nit: the server sets the field back to null in the `finally` of `runTaskAndReport`, not in a shutdown hook.
```suggestion
// The value is read once, when the context is made. The server sets Cancellation.running back to
// null once it is done with the task, so a context that read the field on every call would stop
// seeing the stop request in code that runs after that, such as a JVM shutdown hook.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make Java SDK context signal-aware [airflow]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #74690:
URL: https://github.com/apache/airflow/pull/74690#discussion_r4332520808
##########
java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Context.kt:
##########
@@ -127,10 +129,49 @@ data class Context(
/** Registration of the executing task; resolves wired data inputs. */
internal var taskDef: TaskDef? = null
+ internal var cancellation = Cancellation()
+
+ /**
+ * Whether Airflow has asked this task to stop.
+ *
+ * Airflow asks a running task to stop when, for example, a user marks its task
+ * instance as failed. The supervisor, which is the Airflow process that started
+ * the JVM, then sends the JVM SIGTERM. If the JVM is still running after a
+ * grace period, the supervisor sends SIGKILL.
+ *
+ * Once a task has called this method or [awaitCancellation], the SDK catches
+ * the first SIGTERM or SIGINT. The SDK records the stop request instead of
+ * letting the JVM exit, so the task can stop its work before SIGKILL arrives.
+ * From then on the task must keep checking until it returns.
+ *
+ * If a task has called neither method, or if a second signal arrives, the JVM
+ * handles the signal as it would without the SDK. By default the JVM runs its
+ * shutdown hooks and exits.
+ *
+ * A context made with [copy] does not see the stop request.
Review Comment:
Changed to share the same `Cancellation` instance for the `Context`.
##########
java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Context.kt:
##########
@@ -127,10 +129,49 @@ data class Context(
/** Registration of the executing task; resolves wired data inputs. */
internal var taskDef: TaskDef? = null
+ internal var cancellation = Cancellation()
Review Comment:
Thanks for the review. I agree that only the instance from `runTaskAndReport` was connected to the signal trap, so a caller that forgot to pass it would get an unconnected one without any error.
I used another way further than making the parameter required. A task JVM runs one task, and SIGTERM reaches the whole JVM anyway. The SDK therefore keeps the running task's `Cancellation` in one JVM-wide field, `Cancellation.running`. `runTaskAndReport` creates the instance, stores it in that field before it opens the trap. Set the field back to null when it is done with the task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make Java SDK context signal-aware [airflow]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on PR #74690:
URL: https://github.com/apache/airflow/pull/74690#issuecomment-6354904610
Sorry for that. I will take a look today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make Java SDK context signal-aware [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #74690:
URL: https://github.com/apache/airflow/pull/74690#issuecomment-6336347300
### Backport successfully created: v3-4-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-4-test</td>
<td><a href="https://github.com/apache/airflow/pull/75386"><img src="https://img.shields.io/badge/PR-75386-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make Java SDK context signal-aware [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #74690:
URL: https://github.com/apache/airflow/pull/74690#issuecomment-6347005202
The K8S Lang-SDK test (`test_lang_sdk_mixed_language`) has been hanging intermittently since this PR merged: it failed in 9 of the 21 Tests (AMD) runs created after 08:23 UTC today that ran it, and passed in every run created before. Each time, one Java task stays `running` until the 600s test timeout.
In the hung task's pod log (KinD logs artifact of run https://github.com/apache/airflow/actions/runs/39791718953, pod `lang-sdk-mixed-language-java-transform-pjzecsi6`):
- The JVM sends its first request, `GetVariable(key='my_variable')` (id 0), at 14:44:06.99.
- The Python supervisor receives it, gets the variable from the API server (200 OK), and keeps heartbeating for the next 10 minutes.
- The JVM never logs `Received` for the reply, and logs nothing else afterwards.
So the JVM looks stuck right after its first request to the supervisor. This PR changed `Server.kt`, `Context.kt` and `Logger.kt` around the task run, which is where it stops. Could you take a look? A thread dump of a hung task would show where it blocks.
---
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make Java SDK context signal-aware [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #74690:
URL: https://github.com/apache/airflow/pull/74690#discussion_r4323132929
##########
java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Context.kt:
##########
@@ -127,10 +129,49 @@ data class Context(
/** Registration of the executing task; resolves wired data inputs. */
internal var taskDef: TaskDef? = null
+ internal var cancellation = Cancellation()
Review Comment:
How about constructing the `Cancellation()` once instead of constructing in several places?
Here're more context that I ask agent to double check:
---
Only the instance in `runTaskAndReport` (Server.kt) is connected to the signal trap. The others are defaults: this field, `Context.from` (line 174), `TaskRunner.runTask` (Task.kt line 101, never used since both wrappers pass it), and `runTask(bundle, request, client)` (Task.kt line 170, only `TaskTest` relies on it).
If a future caller forgets to pass the server's instance, it gets an unconnected one with no error. `isCancelled()` then marks the wrong instance as watched, and SIGTERM goes to the JVM.
Could we create it once in `runTaskAndReport` and make the parameter required in `Context.from` and both `runTask` functions? Tests can pass `Cancellation()` explicitly. This field's initializer also allocates an instance that `Context.from` replaces right away.
##########
java-sdk/sdk/src/main/kotlin/org/apache/airflow/sdk/Context.kt:
##########
@@ -127,10 +129,49 @@ data class Context(
/** Registration of the executing task; resolves wired data inputs. */
internal var taskDef: TaskDef? = null
+ internal var cancellation = Cancellation()
+
+ /**
+ * Whether Airflow has asked this task to stop.
+ *
+ * Airflow asks a running task to stop when, for example, a user marks its task
+ * instance as failed. The supervisor, which is the Airflow process that started
+ * the JVM, then sends the JVM SIGTERM. If the JVM is still running after a
+ * grace period, the supervisor sends SIGKILL.
+ *
+ * Once a task has called this method or [awaitCancellation], the SDK catches
+ * the first SIGTERM or SIGINT. The SDK records the stop request instead of
+ * letting the JVM exit, so the task can stop its work before SIGKILL arrives.
+ * From then on the task must keep checking until it returns.
+ *
+ * If a task has called neither method, or if a second signal arrives, the JVM
+ * handles the signal as it would without the SDK. By default the JVM runs its
+ * shutdown hooks and exits.
+ *
+ * A context made with [copy] does not see the stop request.
Review Comment:
Would it better to block the `copy` method at all or the share the same `Cancellation` instance for the `Context`?
---
Since is a public data class, `copy()` is easy to reach for, and the copy gets its own `Cancellation` that the signal trap never sees. If a task checks only through the copy, the real instance is never marked as watched, so SIGTERM goes to the JVM. Is documenting it enough, or should the copy share the original instance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Make Java SDK context signal-aware [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 merged PR #74690:
URL: https://github.com/apache/airflow/pull/74690
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org