[I] Extend macOS fork+exec fix to DAG processor and triggerer [airflow]

["kaxil (via GitHub)" <gi...@apache.org>]

kaxil opened a new issue, #65691:
URL: https://github.com/apache/airflow/issues/65691

   ## Context
   
   #64874 fixed the macOS `SIGSEGV`/`SIGABRT` crash during **task execution** by switching from bare `os.fork()` to `fork + exec` in the Task SDK supervisor for `ActivitySubprocess`. The same macOS fork-safety issue applies to the **DAG processor** and **triggerer**, which both use `WatchedSubprocess` and both run code paths that can trigger ObjC class initialization (DNS resolution, secret backends, HTTP clients, etc.).
   
   ## Problem
   
   `WatchedSubprocess` subclasses hit by this on macOS:
   
   - `DagFileProcessorProcess` (`airflow-core/src/airflow/dag_processing/processor.py`) -- parses DAG files on a forked child. DAG files frequently have top-level network calls (secret backends, connection lookups, variable fetches). We already observed the DAG processor ObjC crash during #64874 investigation -- the `InProcessExecutionAPI` spins up an `a2wsgi` background thread that initializes `httpx` / `ssl` / `Security.framework`.
   - `TriggerRunnerSupervisor` (`airflow-core/src/airflow/jobs/triggerer_job_runner.py`) -- runs user-defined triggers that poll APIs / watch queues / hit HTTP endpoints. Practically always network calls.
   
   #64874 intentionally scoped the fix to task execution so `_child_exec_main()` could hardcode `_subprocess_main` as the entry point. Both of these callers pass different `target`s:
   - DAG processor: `target=_parse_file_entrypoint`
   - Triggerer: `target=TriggerRunnerSupervisor.run_in_process`
   
   ## Proposed approach
   
   Generalize `_child_exec_main()` so it can rehydrate an arbitrary target across `execv`. Approach:
   
   1. Before `execv`, encode the target as a module:function string in an env var (e.g., `_AIRFLOW_CHILD_TARGET`). Class methods can be handled as `module:ClassName.method`.
   2. In `_child_exec_main()`, read the env var, `importlib.import_module` + `getattr` to rehydrate, then call `_fork_main(..., target)`.
   3. `DagFileProcessorProcess.start()` and `TriggerRunnerSupervisor.start()` compute `use_exec = sys.platform in _FORK_EXEC_PLATFORMS` and pass it through.
   
   ## Acceptance criteria
   
   - DAG processor runs without `SIGSEGV`/`SIGABRT` on macOS when DAG files trigger DNS / httpx / Security.framework at parse time.
   - Triggerer runs user triggers without fork-safety crashes on macOS.
   - `_child_exec_main()` does not hardcode `_subprocess_main`; it accepts any importable target.
   - No regression on Linux (bare fork path untouched for non-`_FORK_EXEC_PLATFORMS`).
   
   ## Related
   
   - #64874 (task execution fix, scoped this work)
   - https://github.com/python/cpython/issues/105912
   - https://github.com/apache/airflow/discussions/24463


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Extend macOS fork+exec fix to DAG processor and triggerer [airflow]

["kaxil (via GitHub)" <gi...@apache.org>]

kaxil closed issue #65691: Extend macOS fork+exec fix to DAG processor and triggerer
URL: https://github.com/apache/airflow/issues/65691


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org