You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2026/08/17 10:48:24 UTC
[I] Task subprocess can deadlock permanently when Linux fork() inherits a held OpenSSL lock from the supervisor [airflow]
amoghrajesh opened a new issue, #71707:
URL: https://github.com/apache/airflow/issues/71707
**Under which category would you file this issue?** Task SDK
**Apache Airflow version:** main (current `task-sdk/src/airflow/sdk/execution_time/supervisor.py`, where `_FORK_EXEC_PLATFORMS = {"darwin"}` is unchanged) — likely affects any 3.x release with the same gate.
**What happened and how to reproduce it?**
Issue Description:
Tasks that make outbound HTTPS calls (cloud-provider hooks/operators built on `google-auth`/`urllib3`/`requests` — e.g. `GKEStartPodOperator` or a Google-provider sensor) can hang indefinitely with no further progress. `execution_timeout` does not recover the task (that's a separate, related problem tracked in #53337 — this issue is about the hang itself).
`py-spy` dumps of a hung worker, taken minutes apart, are byte-identical — confirming a true deadlock, not a slow call. Every active thread in the process — the task's own thread, plus unrelated background threads (an OpenTelemetry metrics-exporter thread, a `google-auth` credential-refresh thread, an OpenLineage listener thread) — is frozen at the identical point, each building a fresh `SSLContext` for a *different* remote host:
```
__new__ (ssl.py:440)
create_urllib3_context (urllib3/util/ssl_.py:252)
_ssl_wrap_socket_and_match_hostname (urllib3/connection.py:928)
connect (urllib3/connection.py:812)
...
```
That rules out a network-specific hang — `SSLContext` construction itself is blocked, process-wide.
The task subprocess is created via a bare `os.fork()` in `ActivitySubprocess.start()` (`task-sdk/src/airflow/sdk/execution_time/supervisor.py`), since `_should_use_exec()` only returns `True` on `darwin`:
```python
_FORK_EXEC_PLATFORMS = {"darwin"}
def _should_use_exec() -> bool:
return sys.platform in _FORK_EXEC_PLATFORMS
```
The supervisor process is multithreaded by the time any task starts (OTel exporter threads, per-connection `google-auth` background refresh threads, an OpenLineage listener, etc. are normal background threads in a long-running worker). `os.fork()` copies the address space, but only the calling thread survives in the child. If any sibling thread was mid-way through OpenSSL's global lock (taken inside `SSL_CTX_new()`, which every `ssl.SSLContext()` call goes through) at the exact instant of the fork, the child inherits that lock already held, permanently — the thread that held it no longer exists to release it. Every later `SSLContext` construction in that child, from any thread, then blocks forever.
Steps to reproduce (a timing race, won't reproduce every attempt):
1. Run a task via the Task SDK on Linux, using a hook/operator that opens outbound HTTPS connections.
2. Ensure the supervisor has several concurrent background threads also doing HTTPS/TLS work at task-launch time (OTel exporter, one or more `google-auth` `Credentials` objects with regional-access-boundary refresh threads, OpenLineage listener).
3. Launch tasks in volume. Intermittently, a task hangs forever with no log output past its last HTTPS-triggering line.
4. `py-spy dump --pid <hung-pid> --subprocesses` while hung — every active thread shows the same `ssl.py:440` frame.
**What you think should happen instead?**
The task should complete or eventually be killed and retried — not hang indefinitely due to C-library lock state inherited from an unrelated background thread.
Same hazard class already fixed for macOS (#64874, #65691 — Objective-C runtime corruption after bare `fork()`) and hit again independently on Linux (#65942/#65943 — Edge Worker forking a 22-thread process, corrupting Python's own import-lock state). Here the poisoned lock is OpenSSL's, and the affected fork is the Task SDK's own `ActivitySubprocess.start()`, not an executor-level fork — so the existing `_should_use_exec()` gate doesn't cover it, since the original macOS fix's reasoning ("Linux's resolver has no ObjC dependency") is specific to that hazard and doesn't generalize to OpenSSL's own fork-unsafe locking.
Following the Edge Worker precedent (#65943), a plausible fix shape is an opt-in "fresh interpreter instead of bare fork" setting for `ActivitySubprocess.start()` on Linux, defaulting to current behavior to avoid a blanket performance regression — open question for maintainers whether that reuses `execute_tasks_new_python_interpreter` or needs a dedicated setting.
**Operating System:** Linux (the hazard is generic CPython/OpenSSL fork-safety, not distro-specific)
**Deployment:** Other Docker-based deployment
**Apache Airflow Provider(s):** google, cncf-kubernetes (trigger examples observed; not provider-specific — any HTTPS-heavy hook can be the victim)
**Anything else?**
- Related but distinct: #53337 (moving `execution_timeout` enforcement to the supervisor) would let a hang like this get killed and retried automatically — valuable, but doesn't fix the hang itself.
- Prior art for this hazard class: #64874, #65691 (macOS/ObjC), #65942/#65943 (Linux/Edge Worker, Python import locks). CPython itself emits `DeprecationWarning: This process is multi-threaded, use of fork() may lead to deadlocks in the child` in this exact scenario.
- Can share redacted `py-spy` dumps if useful for triage.
**Are you willing to submit PR?**
- [ ] Yes I am willing to submit a PR!
**Code of Conduct**
- [X] I agree to follow this project's Code of Conduct
---
Drafted-by: Claude Code (Sonnet 5); reviewed by @amoghrajesh before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task subprocess can deadlock permanently when Linux fork() inherits a held OpenSSL lock from the supervisor [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on issue #71707:
URL: https://github.com/apache/airflow/issues/71707#issuecomment-5598874534
In Airflow 2 we had a config option to say whether to use fork or exec for running tasks -- we could bring that back, as this is only a problem in a small number of deployments -- the vast majority are fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task subprocess can deadlock permanently when Linux fork() inherits a held OpenSSL lock from the supervisor [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed issue #71707: Task subprocess can deadlock permanently when Linux fork() inherits a held OpenSSL lock from the supervisor
URL: https://github.com/apache/airflow/issues/71707
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task subprocess can deadlock permanently when Linux fork() inherits a held OpenSSL lock from the supervisor [airflow]
Posted by "slice-soupam (via GitHub)" <gi...@apache.org>.
slice-soupam commented on issue #71707:
URL: https://github.com/apache/airflow/issues/71707#issuecomment-5590309664
<p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">The machinery to fix this already exists on <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">main</code> — it's just gated off for Linux:</p><div class="codeBlockWrapper_-a7MRw" style="position: relative; margin: 8px 0px; color
: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><button class="copyButton_CEmTFw copyButton_-a7MRw" title="Copy code" aria-label="Copy code to clipboard" style="color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; background: none 0% 0% / auto repeat scroll padding-box border-box rgb(34, 26, 15); border-color: rgba(127, 127, 127, 0.4); border-style: solid; border-width: 1px; border-image: none 100% / 1 / 0 stretch;
cursor: pointer; opacity: 0; display: flex; border-radius: 4px; justify-content: center; align-items: center; padding: 4px; transition: opacity 0.15s, background 0.15s; position: absolute; top: 4px; right: 4px;"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 20 20" fill="currentColor" aria-hidden="true" data-slot="icon" class="copyIcon_CEmTFw"><path fill-rule="evenodd" d="M15.988 3.012A2.25 2.25 0 0 1 18 5.25v6.5A2.25 2.25 0 0 1 15.75 14H13.5v-3.379a3 3 0 0 0-.879-2.121l-3.12-3.121a3 3 0 0 0-1.402-.791 2.252 2.252 0 0 1 1.913-1.576A2.25 2.25 0 0 1 12.25 1h1.5a2.25 2.25 0 0 1 2.238 2.012ZM11.5 3.25a.75.75 0 0 1 .75-.75h1.5a.75.75 0 0 1 .75.75v.25h-3v-.25Z" clip-rule="evenodd"></path><path d="M3.5 6A1.5 1.5 0 0 0 2 7.5v9A1.5 1.5 0 0 0 3.5 18h7a1.5 1.5 0 0 0 1.5-1.5v-5.879a1.5 1.5 0 0 0-.44-1.06L8.44 6.439A1.5 1.5 0 0 0 7.378 6H3.5Z"></path></svg></button><pre style="overflow-x: auto; white-space: pre; box-sizing: border-box; border-radius: 4px; max-width: 100%; margin: 0px; pad
ding: 8px;"><code class="language-python" style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 0px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">_FORK_EXEC_PLATFORMS = {"darwin"} # task-sdk/.../execution_time/supervisor.py
</code></pre></div><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">start()</code> can <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3
px; word-break: break-word; font-size: 0.9em;">os.execv</code> a fresh interpreter right after <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">os.fork()</code>. It was added for macOS's ObjC fork-unsafety (#65691), but it's the right remedy here too: <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">execv</code> replaces the child's address space, so an OpenSSL mutex left locked by a thread that doesn't exist in the child can't survive into it. And it's not new code — it already carries the task runner, DAG processor and triggerer on macOS, socketpair/FD handoff and process groups included.</p><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204,
204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">So: reuse it on Linux rather than add a second spawn mechanism. The catch is that <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">exec</code> forfeits copy-on-write sharing of the parent's imported Airflow, so each child pays a fresh interpreter start plus full re-import — in both latency (~1-3s) and RSS, since nothing is shared with the pa
rent any more. That cost is per-fork, so it scales with fork rate, and the fork rate differs by three orders of magnitude across components:</p><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong>High concurrent task load.</strong> Worst case. Every task instance pays the re-import, and because COW sharing is gone, each concurrent child now holds its own full copy of Airflow — so worker RSS grows with concurre
ncy instead of staying near-flat. Short tasks are hit hardest, since the overhead is a fixed cost against a small payload. Note the perverse incentive: deadlock probability rises with parent thread activity, so the deployments that most need this fix are exactly the ones that pay most for it.</p><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong>High DAG file count.</strong> Also significant, and I want to corr
ect something I'd assumed: the DAG processor forks once <em>per file per parse cycle</em> (<code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">_create_process</code>, bounded by <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">parsing_processes</code>). With thousands of files, adding an interpreter start to every parse stretches the whole loop, and longer loops mean staler DAGs and later scheduling. Re-import is only cheap relative to a <em>slow</em> parse; across many fast ones it dominates.</p><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif;
font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong>Triggerer.</strong> Forks once at startup. Overhead is unmeasurable.</p><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: r
gb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Which suggests gating by fork rate rather than by platform alone:</p>
Component | Fork rate | Proposal
-- | -- | --
Triggerer | once | enable on Linux unconditionally
DAG processor | per file, per cycle | config-gated, default off
Task execution | per task instance | config-gated, default off
<div class="codeBlockWrapper_-a7MRw" style="position: relative; margin: 8px 0px; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><button class="copyButton_CEmTFw copyButton_-a7MRw" title="Copy code" aria-label="Copy code to clipboard" style="color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; background: none 0% 0% / auto repeat scroll padding-box border-box rgb(34, 26, 15); border-color: rgba(127, 127, 127
, 0.4); border-style: solid; border-width: 1px; border-image: none 100% / 1 / 0 stretch; cursor: pointer; opacity: 0; display: flex; border-radius: 4px; justify-content: center; align-items: center; padding: 4px; transition: opacity 0.15s, background 0.15s; position: absolute; top: 4px; right: 4px;"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 20 20" fill="currentColor" aria-hidden="true" data-slot="icon" class="copyIcon_CEmTFw"><path fill-rule="evenodd" d="M15.988 3.012A2.25 2.25 0 0 1 18 5.25v6.5A2.25 2.25 0 0 1 15.75 14H13.5v-3.379a3 3 0 0 0-.879-2.121l-3.12-3.121a3 3 0 0 0-1.402-.791 2.252 2.252 0 0 1 1.913-1.576A2.25 2.25 0 0 1 12.25 1h1.5a2.25 2.25 0 0 1 2.238 2.012ZM11.5 3.25a.75.75 0 0 1 .75-.75h1.5a.75.75 0 0 1 .75.75v.25h-3v-.25Z" clip-rule="evenodd"></path><path d="M3.5 6A1.5 1.5 0 0 0 2 7.5v9A1.5 1.5 0 0 0 3.5 18h7a1.5 1.5 0 0 0 1.5-1.5v-5.879a1.5 1.5 0 0 0-.44-1.06L8.44 6.439A1.5 1.5 0 0 0 7.378 6H3.5Z"></path></svg></button><pre style="overflow-x: auto; white-s
pace: pre; box-sizing: border-box; border-radius: 4px; max-width: 100%; margin: 0px; padding: 8px;"><code class="language-python" style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 0px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">_FORK_EXEC_PLATFORMS = {"darwin", "linux"}
# darwin: unconditional (correctness). linux: unconditional for the triggerer,
# conf.getboolean("workers", "fork_exec_task_subprocess", fallback=False) and the
# dag_processor equivalent elsewhere. See #71707.
</code></pre></div><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">airflow.sdk.configuration.conf</code> is already the SDK-side accessor and <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba
(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">[workers]</code> an existing SDK-visible section, so no new config plumbing. Follows the <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">mp_forkserver_preload</code> precedent of exposing a start-method tradeoff rather than hardcoding it.</p><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb
(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong>Pros</strong></p><ul style="padding-inline-start: 2em; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><li style="unicode-bidi: plaintext;">Reuses a shipped, exercised path; prevents the deadlock rather than detecting it (unlike<span> </span><code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; borde
r-radius: 3px; word-break: break-word; font-size: 0.9em;">execution_timeout</code>, which only kills an already-lost task and can't cover the pre-<code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">execute()</code><span> </span>window).</li><li style="unicode-bidi: plaintext;">Default-off leaves unaffected deployments bit-for-bit unchanged — safe to backport, affected users flip one flag.</li><li style="unicode-bidi: plaintext;">Fixes the whole inherited-state class in the child, not just OpenSSL.</li></ul><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spaci
ng: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong>Cons</strong></p><ul style="padding-inline-start: 2em; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><li style="unicode-bidi: plaintext;">Opt-in means you hit the deadlock before you find the flag. Worth do
cumenting the symptom (stuck in<span> </span><code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">running</code>, no logs, no heartbeat progress) in the option description.</li><li style="unicode-bidi: plaintext;">When enabled at high concurrency the cost is real, in RSS as much as latency — the tradeoff is genuinely per-deployment, which is the argument for a flag over a default.</li><li style="unicode-bidi: plaintext;">Two more platform-conditional branches to keep tested.</li></ul><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><strong>Longer term</strong> the cleaner answer is to not fork from a threaded parent at all — a <code style="font-family: monospace; color: rgb(215, 186, 125); background-color: rgba(255, 255, 255, 0.1); padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size: 0.9em;">forkserver</code> forked before any threads start removes the entire inherited-lock class and amortises the import across children, which would address the high-concurrency cost too. Bigger change; shouldn't block the mitigation above.</p><p style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family: -apple-system, "system-ui", "Segoe UI", Roboto, sans-s
erif; font-size: 13px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(54, 39, 18); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;">Happy to put up a PR for the split above if the direction seems right.</p>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task subprocess can deadlock permanently when Linux fork() inherits a held OpenSSL lock from the supervisor [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #71707:
URL: https://github.com/apache/airflow/issues/71707#issuecomment-5601635207
> In Airflow 2 we had a config option to say whether to use fork or exec for running tasks -- we could bring that back, as this is only a problem in a small number of deployments -- the vast majority are fine.
https://github.com/apache/airflow/pull/72164 should mostly solve it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Task subprocess can deadlock permanently when Linux fork() inherits a held OpenSSL lock from the supervisor [airflow]
Posted by "seanmuth (via GitHub)" <gi...@apache.org>.
seanmuth commented on issue #71707:
URL: https://github.com/apache/airflow/issues/71707#issuecomment-5767004810
Also related issue, `fork+exec` doesn't guard against fork-unsafe `atfork` handlers, such as the `logging.debug(...)` calls in datadogpy: https://github.com/DataDog/datadogpy/pull/989
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org