You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "seanmuth (via GitHub)" <gi...@apache.org> on 2026/07/17 22:07:57 UTC

[D] Proposal: opt-in process isolation for JVM-backed providers (JDBC / jaydebeapi) [airflow]

GitHub user seanmuth created a discussion: Proposal: opt-in process isolation for JVM-backed providers (JDBC / jaydebeapi)

## Summary

The JDBC provider (and any provider that embeds a JVM via JPype/jaydebeapi) runs the JVM *inside* the worker's task process. This makes a whole class of native faults impossible to handle gracefully: a native crash or hang in the driver corrupts or wedges the task process, and there is no in-process remedy. I'd like to propose an opt-in process-isolation execution mode for these providers and gather input on the right shape.

## Why this can't be solved in-process today

- A `SIGSEGV` (or other native abort) in the JDBC driver / JVM is not a Java exception. It never surfaces through JPype as something `try/except` in a task or in the hook can catch. It corrupts the process itself.
- JPype's JVM is a per-process singleton: started once, shared by every task in that process, and not restartable after it crashes. One native fault leaves a poisoned JVM that every subsequent task in that process inherits.
- It is not fork-safe, so naive `multiprocessing` workarounds are unreliable.

The provider already carries scar tissue from this in-process model: `get_conn` had to be made threadsafe to avoid "JVM is already started" (#44718), Java exceptions are suppressed with warnings around autocommit (#43786), and there is recurring JPype packaging fragility (#23847, #65532). These are tactical patches around a structural issue: the JVM has no isolation from the task process.

## Observed real-world impact

A production user running Sybase via the jConnect JDBC driver hit repeated `SIGSEGV`s in the JVM. Because the segfault killed a native thread but the parent Python task survived and kept heartbeating, the task never transitioned to a terminal state. Airflow's zombie detection is heartbeat-based, so it never reaped these tasks. They sat "Running" for 13+ hours, blocking their workload's cadence, and only recovered when manually marked failed. A native fault that should have been a single retryable task failure instead became a silent, process-poisoning hang.

## Proposal

Add an opt-in isolation mode for JVM-backed execution, so a native fault is contained to a disposable process and converted into a normal, retryable task failure. Two candidate shapes:

1. **Per-call subprocess.** Simple, and already partially achievable today via `ExternalPythonOperator`, but each call pays full JVM cold-start (1-2s+), which is prohibitive for high-frequency, short-running queries.
2. **Persistent isolated JVM worker (daemon).** A long-lived sidecar process hosts the JVM and services queries over IPC. This preserves warm-JVM/connection reuse while giving crash containment: if the daemon dies, the parent sees it, fails the task cleanly, and respawns a fresh JVM for the next run. More work, but the version that actually fits latency-sensitive workloads.

## Open questions

- Is this better as an execution mode on the existing hook/operator (e.g. `execution_isolation="daemon"`) or a distinct operator? An execution mode keeps users on the operator they already know and could generalize to any JPype/native-backed hook.
- The Task Execution Interface (AIP-72 / Task SDK, accepted and shipped in Airflow 3) already moves task execution behind a defined, language-agnostic interface with a supervisor over a task subprocess. That supervisor catches a full task-process crash, but not this case: a native-thread `SIGSEGV` leaves the task subprocess alive and heartbeating, and the JVM still lives inside that subprocess. Does a JVM-isolation boundary belong as an extension of that interface, or is it better kept local to the JVM-backed providers?
- Appetite for a reference implementation of the daemon model, or is per-call subprocess isolation an acceptable first step?

Would appreciate maintainer input on the shape before anyone invests in an implementation.

🤖 Drafted with Claude Code (Claude Opus 4.8, 1M context)

GitHub link: https://github.com/apache/airflow/discussions/70055

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Proposal: opt-in process isolation for JVM-backed providers (JDBC / jaydebeapi) [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.

GitHub user uranusjr added a comment to the discussion: Proposal: opt-in process isolation for JVM-backed providers (JDBC / jaydebeapi)

I would go further and argue that isolation should be the default mode. In general, the cleaner and safer mode should be the easy one to reach, while dangerous with potential performance should be opt-in. It is of course already not the case, but personally I think that’s a strong enough argument to +1 on the new mode, plus a deprecation/migration period to make the new mode the default.

GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-17679134

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation) [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.

GitHub user kaxil added a comment to the discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)

The reframed proposal makes sense to me. I checked the Task SDK: `execution_timeout` is enforced in-process with SIGALRM, and the Python-level handler can only run once the main thread re-enters the interpreter loop. If the main thread is stuck inside a native call that never returns to the interpreter (the JPype case, whether or not the GIL is released), or the extension retries EINTR internally, that never happens and the timeout silently doesn't fire. And since the supervisor heartbeats on the task's behalf for as long as the child process is alive, heartbeat-based detection can't see it either. So a task that's alive but permanently wedged is invisible today.

Where I'd simplify the shape: I wouldn't start with a new arm/disarm primitive for hook authors. It needs every provider to be instrumented before it covers anything, and hook authors can't pick good deadlines anyway; a legitimately slow 4-hour query and a wedged one look identical from inside the hook. The DAG author is the one who knows expected runtime, and they already declare it: `execution_timeout`. So the smaller first step is to make the supervisor enforce `execution_timeout` as a backstop, and there's already an open issue for it: #53337. The liveness case here is a good argument for prioritizing it. The task runner knows the timeout once it has parsed the task and already has a message channel to the supervisor (though as Ash notes on that issue, the timeout could also ride the Run request), so this is one new message plus one clock check in the supervisor's existing poll loop, reusing the supervisor's kill escalation. Set the supervisor deadline to `execution_timeout` plus
a grace period and well-behaved tasks keep exactly today's semantics (`AirflowTaskTimeout` raised in-process, `on_kill` runs); the supervisor only steps in when the in-process timeout has failed to fire. There is precedent in the SDK: the callback supervisor does the same thing for `callback_execution_timeout`, a wall-clock check in the monitor loop that kills the subprocess.

This would be Task SDK / Airflow 3 only, since 2.x is EOL. On AIP-86: deadline alerts are dag-run-level and alerting-only, while this is worker-local enforcement, so they're complementary, but a proposal using the word "deadline" should say so explicitly.

Per-call arming could ride on the same mechanism later if finer granularity turns out to be needed. The JVM daemon idea I'd keep provider-local; per-task process isolation already exists, and the liveness backstop is the part that belongs in core. One caveat: `execution_timeout` doesn't span deferrals today (#69678 tracks that separately), and supervisor enforcement doesn't change that, since the process exits at deferral time.

GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-17856562

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation) [airflow]

Posted by "seanmuth (via GitHub)" <gi...@apache.org>.

GitHub user seanmuth edited a discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)

## Summary

## Why this can't be solved in-process today

## Observed real-world impact

## Proposal

Add an opt-in isolation mode for JVM-backed execution, so a native fault is contained to a disposable process and converted into a normal, retryable task failure. Two candidate shapes:

## Open questions

Would appreciate maintainer input on the shape before anyone invests in an implementation.

🤖 Drafted with Claude Code (Claude Opus 4.8, 1M context)

GitHub link: https://github.com/apache/airflow/discussions/70055

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation) [airflow]

Posted by "ashb (via GitHub)" <gi...@apache.org>.

GitHub user ashb added a comment to the discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)

> any native-extension-backed hook (pymongo, paramiko/cryptography-backed SSH, other JNI bridges) has the same class of risk, a native call that hangs without releasing the GIL is invisible to anything running inside that same process.

Those isn't true in general, as most other native extensions are better behaved and release the GIL when they aren't running python code - and only require it when necessary.

Which isn't to say there aren't other ways of achieving alive-but-deadlocked state which is invisible to the supervisor 

GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-17706119

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation) [airflow]

Posted by "seanmuth (via GitHub)" <gi...@apache.org>.

GitHub user seanmuth edited a comment on the discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)

Second, independent case for the same blind spot: OpenSSL fork-orphaned lock ([#71707](https://github.com/apache/airflow/issues/71707), fix merged as [#72164](https://github.com/apache/airflow/pull/72164), shipping 3.3.2/3.4.0). Task SDK's supervisor forks the task process via a bare `os.fork()`; if any supervisor thread (OTel/Sentry export, google-auth refresh, secrets-backend polling — confirmed across three separate customer incidents) is mid-critical-section on OpenSSL 3.x's provider-store lock at that instant, the lock is orphaned into the child permanently — no thread survives fork to release it. Same blind spot you both already established for JPype: `execution_timeout`'s SIGALRM never fires (the thread's stuck in native code, never returns to the interpreter loop), and heartbeating stays healthy regardless. Real-world result: task pods alive, heartbeating, and frozen for 3+ days before manual intervention.

+1 on making the supervisor enforce `execution_timeout` as a backstop (#53337) — it's the right default-on mechanism and this case is more corroborating evidence for it.

Where I'd push for keeping the no-progress/low-CPU angle alive as a second, complementary layer rather than closing it out once #53337 lands: `execution_timeout` only bounds a wedge to whatever the DAG author declared, and two things are true in practice for this specific failure mode. Plenty of deployments don't set it at all. And where it is set, it's commonly hours, not just minutes, sized for the task's legitimate runtime rather than for detecting a hang. The OpenSSL deadlock happens at the instant of fork, before the task has done any real work — so a task wedged this way is knowably stuck almost immediately, and a multi-hour `execution_timeout` still leaves it sitting, heartbeating, doing nothing, for most of that window. The two backstops catch different things: `execution_timeout` catches "ran longer than the DAG author's declared bound," which is correct and sufficient for most hangs; a no-progress/low-CPU check catches "this specific process has been doing zero work since
  shortly after it started," independent of whatever timeout is or isn't configured. A genuinely wedged process here shows a CPU flatline from very early on, not just "slow" — confirmed directly against a live wedge in this investigation — which is a different and earlier-available signal than elapsed wall-clock time alone.

Not proposing this as a replacement for #53337 or as fully specified yet — just think it's worth continued exploration alongside it, since the two are complementary defense-in-depth rather than competing designs.

---
Drafted-by: Claude Sonnet 5; reviewed by @seanmuth before posting

GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-18475241

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation) [airflow]

Posted by "seanmuth (via GitHub)" <gi...@apache.org>.

GitHub user seanmuth edited a comment on the discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)

Second, independent case for the same blind spot: OpenSSL fork-orphaned lock ([#71707](https://github.com/apache/airflow/issues/71707), fix merged as [#72164](https://github.com/apache/airflow/pull/72164), shipping 3.3.2/3.4.0). Task SDK's supervisor forks the task process via a bare `os.fork()`; if any supervisor thread (OTel/Sentry export, google-auth refresh, secrets-backend polling — confirmed across three separate customer incidents) is mid-critical-section on OpenSSL 3.x's provider-store lock at that instant, the lock is orphaned into the child permanently — no thread survives fork to release it. Same blind spot you both already established for JPype: `execution_timeout`'s SIGALRM never fires (the thread's stuck in native code, never returns to the interpreter loop), and heartbeating stays healthy regardless. Real-world result: task pods alive, heartbeating, and frozen for 3+ days before manual intervention.

+1 on making the supervisor enforce `execution_timeout` as a backstop (#53337) — it's the right default-on mechanism and this case is more corroborating evidence for it.

Where I'd push for keeping the no-progress/low-CPU angle alive as a second, complementary layer rather than closing it out once #53337 lands: `execution_timeout` only bounds a wedge to whatever the DAG author declared, and two things are true in practice for this specific failure mode. Plenty of deployments don't set it at all. And where it is set, it's commonly hours, not minutes, sized for the task's legitimate runtime rather than for detecting a hang. The OpenSSL deadlock happens at the instant of fork, before the task has done any real work — so a task wedged this way is knowably stuck almost immediately, and a multi-hour `execution_timeout` still leaves it sitting, heartbeating, doing nothing, for most of that window. The two backstops catch different things: `execution_timeout` catches "ran longer than the DAG author's declared bound," which is correct and sufficient for most hangs; a no-progress/low-CPU check catches "this specific process has been doing zero work since shor
 tly after it started," independent of whatever timeout is or isn't configured. A genuinely wedged process here shows a CPU flatline from very early on, not just "slow" — confirmed directly against a live wedge in this investigation — which is a different and earlier-available signal than elapsed wall-clock time alone.

Not proposing this as a replacement for #53337 or as fully specified yet — just think it's worth continued exploration alongside it, since the two are complementary defense-in-depth rather than competing designs.

---
Drafted-by: Claude Sonnet 5; reviewed by @seanmuth before posting

GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-18475241

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation) [airflow]

Posted by "seanmuth (via GitHub)" <gi...@apache.org>.

GitHub user seanmuth edited a comment on the discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)

Second, independent case for the same blind spot: OpenSSL fork-orphaned lock (ZD98087/ZD98601, [#71707](https://github.com/apache/airflow/issues/71707), fix merged as [#72164](https://github.com/apache/airflow/pull/72164), shipping 3.3.2/3.4.0). Task SDK's supervisor forks the task process via a bare `os.fork()`; if any supervisor thread (OTel/Sentry export, google-auth refresh, secrets-backend polling — confirmed across three separate customer incidents) is mid-critical-section on OpenSSL 3.x's provider-store lock at that instant, the lock is orphaned into the child permanently — no thread survives fork to release it. Same blind spot you both already established for JPype: `execution_timeout`'s SIGALRM never fires (the thread's stuck in native code, never returns to the interpreter loop), and heartbeating stays healthy regardless. Real-world result: task pods alive, heartbeating, and frozen for 3+ days before manual intervention.

+1 on making the supervisor enforce `execution_timeout` as a backstop (#53337) — it's the right default-on mechanism and this case is more corroborating evidence for it.

Where I'd push for keeping the no-progress/low-CPU angle alive as a second, complementary layer rather than closing it out once #53337 lands: `execution_timeout` only bounds a wedge to whatever the DAG author declared, and two things are true in practice for this specific failure mode. Plenty of deployments don't set it at all — Moloco's hadn't. And where it is set, it's commonly hours, not minutes, sized for the task's legitimate runtime rather than for detecting a hang. The OpenSSL deadlock happens at the instant of fork, before the task has done any real work — so a task wedged this way is knowably stuck almost immediately, and a multi-hour `execution_timeout` still leaves it sitting, heartbeating, doing nothing, for most of that window. The two backstops catch different things: `execution_timeout` catches "ran longer than the DAG author's declared bound," which is correct and sufficient for most hangs; a no-progress/low-CPU check catches "this specific process has been doing 
 zero work since shortly after it started," independent of whatever timeout is or isn't configured. A genuinely wedged process here shows a CPU flatline from very early on, not just "slow" — confirmed directly against a live wedge in this investigation — which is a different and earlier-available signal than elapsed wall-clock time alone.

Not proposing this as a replacement for #53337 or as fully specified yet — just think it's worth continued exploration alongside it, since the two are complementary defense-in-depth rather than competing designs.

---
Drafted-by: Claude Sonnet 5; reviewed by @seanmuth before posting

GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-18475241

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation) [airflow]

Posted by "seanmuth (via GitHub)" <gi...@apache.org>.

GitHub user seanmuth edited a comment on the discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)

Second, independent case for the same blind spot: OpenSSL fork-orphaned lock ([#71707](https://github.com/apache/airflow/issues/71707), fix merged as [#72164](https://github.com/apache/airflow/pull/72164), shipping 3.3.2/3.4.0). Task SDK's supervisor forks the task process via a bare `os.fork()`; if any supervisor thread (OTel/Sentry export, google-auth refresh, secrets-backend polling — confirmed across three separate customer incidents) is mid-critical-section on OpenSSL 3.x's provider-store lock at that instant, the lock is orphaned into the child permanently — no thread survives fork to release it. Same blind spot you both already established for JPype: `execution_timeout`'s SIGALRM never fires (the thread's stuck in native code, never returns to the interpreter loop), and heartbeating stays healthy regardless. Real-world result: task pods alive, heartbeating, and frozen for 3+ days before manual intervention.

+1 on making the supervisor enforce `execution_timeout` as a backstop (#53337) — it's the right default-on mechanism and this case is more corroborating evidence for it.

Where I'd push for keeping the no-progress/low-CPU angle alive as a second, complementary layer rather than closing it out once #53337 lands: `execution_timeout` only bounds a wedge to whatever the DAG author declared, and two things are true in practice for this specific failure mode. Plenty of deployments don't set it at all — Moloco's hadn't. And where it is set, it's commonly hours, not minutes, sized for the task's legitimate runtime rather than for detecting a hang. The OpenSSL deadlock happens at the instant of fork, before the task has done any real work — so a task wedged this way is knowably stuck almost immediately, and a multi-hour `execution_timeout` still leaves it sitting, heartbeating, doing nothing, for most of that window. The two backstops catch different things: `execution_timeout` catches "ran longer than the DAG author's declared bound," which is correct and sufficient for most hangs; a no-progress/low-CPU check catches "this specific process has been doing 
 zero work since shortly after it started," independent of whatever timeout is or isn't configured. A genuinely wedged process here shows a CPU flatline from very early on, not just "slow" — confirmed directly against a live wedge in this investigation — which is a different and earlier-available signal than elapsed wall-clock time alone.

Not proposing this as a replacement for #53337 or as fully specified yet — just think it's worth continued exploration alongside it, since the two are complementary defense-in-depth rather than competing designs.

---
Drafted-by: Claude Sonnet 5; reviewed by @seanmuth before posting

GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-18475241

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation) [airflow]

Posted by "seanmuth (via GitHub)" <gi...@apache.org>.

GitHub user seanmuth added a comment to the discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)

**Second, independent case for the same gap: OpenSSL fork-orphaned lock (ZD98087/ZD98601, [#71707](https://github.com/apache/airflow/issues/71707))**

Different native library, same structural failure mode as the JVM case above, and it sharpens why an in-process remedy can't generalize.

**Mechanism:** OpenSSL 3.x's provider-store `pthread_rwlock` has no fork handler. Task SDK's supervisor spawns the task process via a bare `os.fork()` (`WatchedSubprocess.start()`); if any thread in the supervisor (OTel/Sentry export, google-auth token refresh, secrets-backend polling — all confirmed independently across three separate customer incidents) is mid-critical-section on that lock at the instant of fork, the lock is orphaned into the child in a permanently locked state — no thread survives fork to ever release it. The next OpenSSL call the task makes (typically its first outbound HTTPS call) blocks forever. No exception, no OOM, no crash.

**Why this can't be caught in-process, and why that generalizes beyond this one bug:** `execution_timeout` is implemented as `SIGALRM` + `signal.setitimer` (`airflow.sdk.execution_time.timeout.TimeoutPosix`) — an in-process signal-based timeout. Two independent reasons it cannot fire against this class of hang: CPython only dispatches a pending signal handler when control returns to the bytecode eval loop, and a thread blocked inside a native call (`SSL_CTX_new_ex` → `pthread_rwlock_wrlock`) never returns there; and even if the signal does interrupt the underlying `futex_wait` at the kernel level, glibc's rwlock wait loop treats that as a spurious wakeup and retries rather than propagating a failure. Heartbeating doesn't help either — it's a separate channel that stays healthy regardless of whether the task thread is making progress, same as the zombie-detection gap described above for the JVM case. Real-world result: task pods sat alive, heartbeating, and completely frozen for
  as long as 3+ days before manual intervention.

**Why I think this belongs at the supervisor level, not as two more per-provider patches:** the JDBC/JVM case and this one have nothing in common at the library level, but they fail identically from the supervisor's point of view — a child process that is alive, heartbeating, and making zero forward progress, for a reason no in-process signal can observe or interrupt. A supervisor-level check (wall-clock time since the last real progress signal from the child, cross-checked against near-zero child CPU sampled externally, terminating unconditionally via `SIGKILL` past a threshold) doesn't need to know anything about JVMs or OpenSSL to catch either case. Notably, the supervisor is a safe place to add this: the fork hazard here is asymmetric — the *parent's* copy of the lock isn't stuck, only the child's is, since the thread holding it in the parent survives and releases it normally moments later — so the supervisor can never be wedged by the same event that wedges its child.

To be clear, this isn't a substitute for fixing either root cause (fork+exec for the task process on Linux is already merged, [#72164](https://github.com/apache/airflow/pull/72164), shipping in 3.3.2 and 3.4.0; JVM isolation is the right fix for the JDBC case above) — it's a backstop that bounds the blast radius to a bounded retry instead of a silent multi-day hang, regardless of which native library is at fault next time.

---
Drafted-by: Claude Sonnet 5; reviewed by @seanmuth before posting

GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-18475241

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation) [airflow]

Posted by "seanmuth (via GitHub)" <gi...@apache.org>.

GitHub user seanmuth edited a comment on the discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)

Fair correction, and you're right, I overreached there. Most well-behaved C extensions do release the GIL around blocking calls, that's the standard pattern for exactly the reason you're describing, so pymongo and paramiko/cryptography aren't in the same boat as JPype by default. JPype specifically is the outlier here, I checked their native source directly, there's no GIL-release pattern anywhere in it, and their own tracker has a GIL-deadlock issue where the maintainer describes their threading model as "hopelessly opaque."

Your second point is the one that actually fixes my framing, though. The real justification for a supervisor-level check isn't "the GIL problem generalizes to everything," it's that alive-but-not-progressing is a broader failure class than any single mechanism, a cross-thread deadlock, an infinite loop, a GIL-cooperative I/O call whose remote end went silent, all of these are just as invisible to a supervisor that only checks process liveness as JPype's case is, they just get there differently. JPype's GIL behavior is the most severe instance, since it also rules out a same-process mitigation, but it's not the general argument for why this belongs at the supervisor layer.

Concretely, we see this with some regularity in customer deployments as SSL/TLS handshake hangs, a well-behaved, GIL-releasing library, but the handshake itself just never completes if the remote end goes quiet mid-negotiation. There are timeout safeguards for most of that today, but they're per-call and per-library, so coverage is uneven, and a supervisor-level backstop would catch what slips through regardless of which library or mechanism is involved. That's a better foundation for the proposal than what I wrote, thanks for the catch.

🤖 Drafted with Claude Code (Claude Sonnet 5)

GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-17706366

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Proposal: opt-in process isolation for JVM-backed providers (JDBC / jaydebeapi) [airflow]

Posted by "seanmuth (via GitHub)" <gi...@apache.org>.

GitHub user seanmuth added a comment to the discussion: Proposal: opt-in process isolation for JVM-backed providers (JDBC / jaydebeapi)

@uranusjr @ashb following up on both of your points, and reframing this proposal based on where the investigation actually led.

The original framing (JDBC-specific process isolation) missed the real gap. Ash's question about the process tree was the right challenge: each task already runs in its own forked process, via `StandardTaskRunner` in AF2 and the Task SDK's `WatchedSubprocess`/`supervise_task` in AF3. That per-task isolation is already there and already works for the clean-crash case.

What isn't there is liveness enforcement *within* that isolation. The supervisor currently only checks whether the child process has exited. It has no way to tell "alive and making progress" from "alive and permanently wedged." A native fault that corrupts a thread but doesn't kill the process (confirmed for the JDBC/JPype case: JPype never releases the GIL around a JNI call, so if that specific call hangs, nothing else in the process, including any same-process watchdog thread, can run either) leaves the task heartbeating and Running indefinitely. That's a supervisor-level gap, not a process-isolation gap, and it's why isolation alone, however it's implemented, doesn't close it.

So the concrete proposal, in the spirit of "should be default" rather than JDBC-specific:

**Add a generic in-flight deadline primitive that any task can arm, enforced by the supervisor.** Shape:

- A shared value, created by the supervisor before it forks the child (so it's naturally inherited in AF2, or riding on AF3's existing pre-fork socketpairs, no new IPC setup needed), representing "the deadline by which the current unit of work should have returned control."
- A hook/operator author arms it before a risky blocking call (e.g. a JDBC query) and disarms it immediately after the call returns, success or a normal exception either way.
- The supervisor's existing poll loop, the same one that already checks whether the child has exited, gains one more check: if the deadline is armed and expired while the child is still alive, `SIGKILL` it.
- Killing the child this way lands in the exact same failure/retry path Airflow already uses for any other child death (a normal exit, an exception, a kernel OOM-kill), so there's no new correctness surface on the retry side, only the *decision to kill* is new.

Why this has to live at the supervisor layer and not in a provider package or a same-process thread: only the supervisor is structurally guaranteed to be a separate process with its own GIL that never touches whatever native resource the task is using. This generalizes past JDBC too, any native-extension-backed hook (pymongo, paramiko/cryptography-backed SSH, other JNI bridges) has the same class of risk, a native call that hangs without releasing the GIL is invisible to anything running inside that same process.

The honest cost: this needs coordinated changes across the task-runner layer (owns the fork, needs to set up the shared primitive first) and the supervisor's own poll loop (needs the extra check), so it's a core change, not something a provider can ship alone. Scope-wise this generalizes cleanly: task_runner is orthogonal to executor choice, so it applies to Celery, Kubernetes, Local, and Sequential executors alike, in both AF2 and AF3, since they all funnel through the same shared dispatch. It does not apply to `DebugExecutor`, which intentionally bypasses this for in-process debugging, or to executors with a fundamentally different worker architecture that doesn't route through the standard task-runner/supervisor layer at all.

This doesn't replace the daemon-isolation idea from the original post, that's still the stronger fix for the specific JVM-in-process case. But this is a smaller, more general first step that gets most of the value (a wedged task becomes a detected, retryable failure instead of a silent multi-hour hang) without touching provider or isolation architecture at all, and it's the piece that actually makes "isolation as default" mean something operationally, since isolation without liveness enforcement doesn't catch this failure mode either way.

🤖 Drafted with Claude Code (Claude Sonnet 5)

GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-17706013

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation) [airflow]

Posted by "seanmuth (via GitHub)" <gi...@apache.org>.

GitHub user seanmuth added a comment to the discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)

Fair correction, and you're right, I overreached there. Most well-behaved C extensions do release the GIL around blocking calls, that's the standard pattern for exactly the reason you're describing, so pymongo and paramiko/cryptography aren't in the same boat as JPype by default. JPype specifically is the outlier here, I checked their native source directly, there's no GIL-release pattern anywhere in it, and their own tracker has a GIL-deadlock issue where the maintainer describes their threading model as "hopelessly opaque."

Your second point is the one that actually fixes my framing, though. The real justification for a supervisor-level check isn't "the GIL problem generalizes to everything," it's that alive-but-not-progressing is a broader failure class than any single mechanism, a cross-thread deadlock, an infinite loop, a GIL-cooperative I/O call whose remote end went silent, all of these are just as invisible to a supervisor that only checks process liveness as JPype's case is, they just get there differently. JPype's GIL behavior is the most severe instance, since it also rules out a same-process mitigation, but it's not the general argument for why this belongs at the supervisor layer.

Concretely, we see this with some regularity in customer deployments as SSL/TLS handshake hangs, a well-behaved, GIL-releasing library, but the handshake itself just never completes if the remote end goes quiet mid-negotiation. There are timeout safeguards for most of that today, but they're per-call and per-library, so coverage is uneven, and a supervisor-level backstop would catch what slips through regardless of which library or mechanism is involved. That's a better foundation for the proposal than what I wrote, thanks for the catch.

GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-17706366

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org

Re: [D] Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation) [airflow]

Posted by "seanmuth (via GitHub)" <gi...@apache.org>.

GitHub user seanmuth added a comment to the discussion: Proposal: supervisor-level liveness enforcement for tasks with blocking native calls (was: JVM process isolation)

Third instance of the same blind spot, different mechanism: datadog's `dogstatsd` client registers an `os.register_at_fork(after_in_child=...)` handler by default, and every branch of it calls `log.debug(...)` unconditionally — including the "everything's disabled" branch — regardless of what Airflow's own config tells it to disable. That `log.debug()` can block acquiring a `StreamHandler`'s own lock, orphaned into the child forever if some other thread held it at the instant of `fork()`.

This is a different failure shape than the OpenSSL case, and it changes the picture, not just adds a data point: `os.register_at_fork()` callbacks run synchronously, *inside* the `fork()` syscall itself, before any subsequent code — including a planned `execv()` — regains control. That means the fork+exec mitigation discussed above (#72164, and the backstop it enables) doesn't cover this one: the deadlock happens before exec is ever reached, not after. What does structurally close the gap is `os.posix_spawn()`, which never runs those callbacks at all — no `fork()` call exists on that path for CPython's binding to hook, and glibc's own implementation (2.24+) uses `clone(CLONE_VM|CLONE_VFORK)`, not `fork()`. That's now merged as #73503 (with a fix proposed upstream in the library itself, DataDog/datadogpy#989).

On scope: DAG processor and triggerer are structurally exposed to the same fork-time deadlock classes discussed here (OpenSSL and the atfork case), since both fork/spawn their child through the same `WatchedSubprocess` path — but that's a confirmed structural risk, not something we've actually seen in the wild. Every real incident behind this discussion is task execution. Checking whether this proposal's arm/disarm mechanism could actually extend to them, rather than assuming it can't: the DAG processor is one file per forked child, no pooling or reuse (`DagFileProcessorManager._create_process()` → `DagFileProcessorProcess.start()`, one child, one `DagFileParseRequest`, per parse) — killing a child on a missed deadline costs exactly one file's parse, the same cost profile as task execution. The triggerer is different: one process runs many concurrent triggers on a single event loop, so killing it does affect other in-flight triggers — but that's already a safe, existing failu
 re mode, not a new cost the proposal would introduce. `Trigger` rows are persisted in the DB keyed by `triggerer_id`, and `Trigger.assign_unassigned()` already reassigns any trigger whose triggerer isn't alive (by `Job` heartbeat) to a live one — the same recovery path that already runs today for a crashed or OOM-killed triggerer. Separately, on which executors this covers on the task-execution side: Celery, Kubernetes, Local, and Sequential all funnel through the same task-runner/supervisor layer, so `LocalExecutor` is technically in scope too — though hitting the fork-rate needed to turn a rare per-fork race into a practical incident needs production-scale scheduling volume, which is more a deployment/Runtime consideration than something this discussion needs to weigh in on.

Given that, do DP and triggerer already have a working backstop for this specific case today, independent of this proposal? Checked both: DAG processor does. `DagFileProcessorManager._kill_timed_out_processors()` checks wall-clock elapsed since each child's `start_time` (set at fork/spawn) against `[dag_processor] dag_file_processor_timeout` (default 50s) and unconditionally `SIGKILL`s — it doesn't depend on the child doing or reporting anything, so it fires the same whether the child is slow or wedged at fork. That gap is already closed.

The triggerer has an analogous-looking watchdog with a real hole for this exact case, though. `TriggerRunnerSupervisor.heartbeat()` checks elapsed time since the child's last message against `[triggerer] runner_health_check_threshold` (default 30s, added 3.2.2) and stops heartbeating the Job if it's been silent too long — same idea on the surface. But that clock (`_last_runner_comms`) is deliberately seeded to `+inf` until the first message arrives, specifically to avoid false-positiving on slow/cold-start hosts. `elapsed = now - inf` never exceeds the threshold. A child wedged at or immediately after fork — before it ever sends a first message, exactly the atfork/OpenSSL class — leaves this watchdog permanently silent.

Possible hardening worth a separate look: seed the "no message yet" case off the child's `start_time` (already available — same field DP's timeout already uses) instead of `+inf`, but compare it against a larger, still-finite cold-start allowance rather than the normal threshold — e.g. `runner_health_check_threshold * 5` (150s by default) until the first message arrives, then the normal 30s after. That keeps the original intent (no false positives on legitimately slow cold starts — cold-start cost we've measured elsewhere in this investigation is under a second, so 150s is generous headroom) while giving a fork-time wedge a real, finite deadline instead of an infinite one. Small, self-contained fix, independent of everything else here — curious whether this is considered worthwhile hardening or if there's a reason the current `+inf` behavior is preferred.

Directly relevant to the execution_timeout piece: Andrushika's note on #72493 (https://github.com/apache/airflow/pull/72493#issuecomment-5728745614) confirms the supervisor doesn't know `execution_timeout` until the Dag is parsed, and needs its own config value to cover anything before that point, left for a separate PR. That's exactly the window both the OpenSSL and atfork wedges land in — before `execute()`, often before any task code runs at all — so even once #53337 lands, it won't reach this failure class without that follow-up. More corroboration for keeping the no-progress/low-CPU angle alive as a genuinely separate layer, not something #53337 subsumes.

One more thing worth pre-empting: does `task_queued_timeout` already provide a backstop here, since the child can't respond to any signal while wedged? In practice, no — and not just partially. The scheduler's stuck-in-queued check only looks at TIs still in `state == QUEUED`, but the `queued → running` transition is made by the *supervisor*, immediately after `fork()`/`posix_spawn()` returns a PID to the parent — it doesn't wait on the child to execute anything. So a task wedged this way is typically already `RUNNING` in the DB within milliseconds of the fork, before `task_queued_timeout`'s check interval (120s, by default) could ever observe it sitting in `QUEUED`. Same blind spot as the OpenSSL case, and for the same root reason: the DB state machine here is driven by "the parent got a PID back," not "the child made any progress."

Drafted-by: Claude Sonnet 5; reviewed by @seanmuth before posting

GitHub link: https://github.com/apache/airflow/discussions/70055#discussioncomment-18554584

----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org