You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Andrushika (via GitHub)" <gi...@apache.org> on 2026/09/05 17:12:08 UTC

[PR] Speed up breeze start by caching Python bytecode in a docker volume [airflow]

Andrushika opened a new pull request, #72567:
URL: https://github.com/apache/airflow/pull/72567

   ## Why
   
   Breeze sets `PYTHONDONTWRITEBYTECODE=true` in its containers, so every `airflow` process recompiles all mounted sources on import, through the bind mount. On macOS this is about 1s per CLI call and several seconds per component start. Writing `.pyc` next to the sources was disabled on purpose, it would leave root-owned files in the checkout.
   
   ## What
   
   Set `PYTHONPYCACHEPREFIX=/root/.cache/airflow-pycache` and mount an external docker volume `airflow-pycache-volume` there, same pattern as `mypy-cache-volume`. Only `breeze shell` and `breeze start-airflow` mount it and enable the cache. Other containers (`breeze testing`, CI) keep `PYTHONDONTWRITEBYTECODE=true`, so CI behaviour does not change. `breeze down --cleanup-pycache` removes the volume.
   
   ## Benchmark
   
   Measured inside the breeze container on macOS (Docker Desktop 28.4, 16 cores, `--backend postgres --dev-mode`, CI image python3.10). Time is until the api-server answers its first HTTP request, with scheduler, triggerer and dag-processor starting at the same time.
   
   | step | before | after, first run | after, second run |
   |---|---|---|---|
   | `airflow db migrate` | 3.8s | 4.1s | 2.2s |
   | `airflow api-server -d` first response | 10.8s | 6.7s | 6.6s |
   | `airflow version` | 2.1s | 2.4s | 1.2s |
   | start-airflow critical path in container | 16.8s | 10.8s | 8.8s |
   
   The first run is slightly slower because the prefix also replaces the pre-built `.pyc` of site-packages, they are compiled once into the volume.
   
   One known limit, same as any normal Python setup: `.pyc` freshness is checked by source mtime and size, so an edit within the same second that keeps the file size would reuse the old bytecode. That is why the cleanup flag exists.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Claude Code (Fable 5.1)
   
   Generated-by: Claude Code (Fable 5.1) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Speed up breeze start by caching Python bytecode in a docker volume [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

potiuk commented on code in PR #72567:
URL: https://github.com/apache/airflow/pull/72567#discussion_r4024510069


##########
dev/breeze/doc/03_developer_tasks.rst:
##########
@@ -399,6 +399,15 @@ in ``--from-ref`` and ``--to-ref`` flags.
     If the cache gets broken, run ``breeze down --cleanup-mypy-cache`` which wipes the docker
     volume and every per-hook ``.build/mypy-venvs/`` and ``.build/mypy-caches/`` directory.
 
+.. note::
+
+    Python bytecode (``.pyc``) compiled from the mounted sources inside the container is written
+    to the ``airflow-pycache-volume`` docker volume (``PYTHONPYCACHEPREFIX``) rather than next to
+    the sources, so it never shows up in your checkout but survives between ``breeze shell`` and
+    ``breeze start-airflow`` runs. This noticeably speeds up every ``airflow`` command and component
+    start-up, especially on macOS where reading sources through the bind mount is slow.
+    Run ``breeze down --cleanup-pycache`` to wipe the volume.

Review Comment:
   Non-blocking doc nit. Two facts a reader (me, an hour ago) will otherwise re-derive are missing here: that one volume is safe across Python versions, and that it is *not* isolated per worktree.
   
   ```suggestion
       The cache is safe to share across Python versions: ``.pyc`` file names keep the
       interpreter tag (``foo.cpython-310.pyc`` vs ``foo.cpython-312.pyc``), so bytecode is
       never reused across versions. It is shared across worktrees, though -- sources always
       mount at ``/opt/airflow`` and freshness is checked by source mtime and size, so
       alternating between worktrees keeps invalidating the other one's entries.
       Run ``breeze down --cleanup-pycache`` to wipe the volume.
   ```
   
   The worktree point is not a defect -- it is correct, just not free. Because breeze always mounts sources at `/opt/airflow`, two worktrees map onto the same cache paths, and since invalidation is source mtime + size, alternating between them keeps invalidating each other's entries. Same family as the same-second-edit limit you already documented.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Speed up breeze start by caching Python bytecode in a docker volume [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on PR #72567:
URL: https://github.com/apache/airflow/pull/72567#issuecomment-5715236081

   ### Backport failed to create: v3-3-test. View the failure log <a href='https://github.com/apache/airflow/actions/runs/35227791464'> Run details </a>
   
   Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
   the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
   
   In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
   
   <table>
       <tr>
           <th>Status</th>
           <th>Branch</th>
           <th>Result</th>
       </tr>
       <tr>
           <td>❌</td>
           <td>v3-3-test</td>
           <td><a href="https://github.com/apache/airflow/commit/8843de3ed17f932b4c47808bdfec1d527e7febdb"><img src='https://img.shields.io/badge/Commit-8843de3-red' alt='Commit Link'></a></td>
       </tr>
   </table>
   
   You can attempt to backport this manually by running:
   
   ```bash
   cherry_picker 8843de3 v3-3-test
   ```
   
   This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
   the files that need manual conflict resolution.
   
   After you have resolved the conflicts, you can continue the backport process by running:
   
   ```bash
   cherry_picker --continue
   ```
   
   If you don't have cherry-picker installed, see the [installation guide](https://github.com/apache/airflow/blob/main/dev/README.md#how-to-backport-pr-with-cherry-picker-cli).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Speed up breeze start by caching Python bytecode in a docker volume [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.

potiuk merged PR #72567:
URL: https://github.com/apache/airflow/pull/72567


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org