[PR] Trace the dag-processor job instead of its parent process with memray [airflow]

["Eason09053360 (via GitHub)" <gi...@apache.org>]

Eason09053360 opened a new pull request, #72367:
URL: https://github.com/apache/airflow/pull/72367

   `@enable_memray_trace` was on the dag-processor's outer CLI function. Both paths
   that function takes — `run_command_with_daemon_option` and, under `--dev`,
   `run_with_reloader` — fork before the job runs, and `memray.Tracker` defaults to
   `follow_fork=False`. So the tracer stayed with the parent while every allocation
   worth measuring happened in the child, and an opted-in user got a capture file
   containing interpreter startup instead of the profile they asked for. 
   It fails silently: the file looks perfectly normal.
   
   The three sibling components already decorate their inner run function —
   `_run_scheduler_job`, `triggerer_run`, `_run_api_server`. This makes
   dag-processor match them by extracting `_run_dag_processor_job` and moving the
   decorator onto it.
   
   `set_component_mp_start_method()` and `_create_dag_processor_job_runner()`
   deliberately stay in the parent. The latter calls `validate_dag_bundle_arg`,
   which raises `SystemExit` on an unknown `--bundle-name`; moving it past the fork
   would bury that error in a daemonized child where the user never sees it.
   
   No behaviour change for anyone who has not opted in — `[profiling]
   memray_trace_components` is unset by default.
   
   The regression test asserts the invariant behaviourally rather than structurally:
   no `memray.Tracker` may be constructed while the CLI function runs, and exactly
   one must be constructed when the callback that lands on the far side of the fork
   is invoked. Verified failing against the unfixed code both whole-file and in isolation.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Claude Code (Opus 5)
   
   Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Trace the dag-processor job instead of its parent process with memray [airflow]

["potiuk (via GitHub)" <gi...@apache.org>]

potiuk merged PR #72367:
URL: https://github.com/apache/airflow/pull/72367


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Trace the dag-processor job instead of its parent process with memray [airflow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on PR #72367:
URL: https://github.com/apache/airflow/pull/72367#issuecomment-5574395192

   ### Backport successfully created: v3-3-test
   
   Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
   the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
   
   In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
   
   <table>
           <tr>
           <th>Status</th>
           <th>Branch</th>
           <th>Result</th>
       </tr>
       <tr>
           <td>✅</td>
           <td>v3-3-test</td>
           <td><a href="https://github.com/apache/airflow/pull/72661"><img src="https://img.shields.io/badge/PR-72661-blue" alt="PR Link"></a></td>
       </tr>
   </table>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org