You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/03/11 21:33:39 UTC
[PR] Generate ERD diagrams at doc build time instead of committing them [airflow]
potiuk opened a new pull request, #63386:
URL: https://github.com/apache/airflow/pull/63386
The ERD diagram (airflow_erd.svg, ~320KB) was previously generated via a prek pre-commit hook and committed to the repository. This caused unnecessary merge conflicts and repo bloat every time migrations changed.
This PR replaces that with a `generate_erd` Sphinx extension that generates ERD diagrams at documentation build time directly from SQLAlchemy model metadata — no running database needed.
The diagram is also split into three separate per-package ERDs, each generated only when building docs for that package:
- **apache-airflow**: core models → `airflow_erd.svg` (54 tables)
- **apache-airflow-providers-fab**: FAB auth manager models → `fab_erd.svg` (11 tables)
- **apache-airflow-providers-edge3**: Edge3 worker models → `edge3_erd.svg` (3 tables)
On environments where eralchemy or graphviz system package is unavailable (e.g. local macOS without `brew install graphviz`), a placeholder SVG is written so the build still succeeds with a clear message.
## Summary
- Added `generate_erd` Sphinx extension that detects the package being built via `AIRFLOW_PACKAGE_NAME` and generates the relevant ERD from SQLAlchemy metadata
- Added the extension to both `airflow-core/docs/conf.py` and `provider_conf.py`
- Created `database-erd-ref.rst` pages for FAB and Edge3 providers, linked from their doc indexes
- Removed committed `airflow_erd.svg`, `airflow_erd.sha256` from the repo
- Removed `update-er-diagram` prek pre-commit hook and associated scripts
- Removed `checksumdir` dependency (only used by the removed script)
- Updated contributing docs to reflect the change
## Test plan
- [ ] Run `breeze build-docs apache-airflow` — verify core ERD is generated
- [ ] Run `breeze build-docs apache-airflow-providers-fab` — verify FAB ERD is generated
- [ ] Run `breeze build-docs apache-airflow-providers-edge3` — verify Edge3 ERD is generated
- [ ] Build docs on macOS without graphviz — verify placeholder SVG appears and build succeeds
- [ ] Build docs for an unrelated provider — verify no ERD generation is attempted
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Opus 4.6
Generated-by: Claude Opus 4.6 following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate ERD diagrams at doc build time instead of committing them [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63386:
URL: https://github.com/apache/airflow/pull/63386#issuecomment-4043325903
### Backport successfully created: v3-1-test
Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-1-test</td>
<td><a href="https://github.com/apache/airflow/pull/63399"><img src="https://img.shields.io/badge/PR-63399-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate ERD diagrams at doc build time instead of committing them [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #63386:
URL: https://github.com/apache/airflow/pull/63386#issuecomment-4043611718
Ha... Bad rebase: Complete change here https://github.com/apache/airflow/pull/63402
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate ERD diagrams at doc build time instead of committing them [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #63386:
URL: https://github.com/apache/airflow/pull/63386
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate ERD diagrams at doc build time instead of committing them [airflow]
Posted by "dstandish (via GitHub)" <gi...@apache.org>.
dstandish commented on PR #63386:
URL: https://github.com/apache/airflow/pull/63386#issuecomment-4042412561
love it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate ERD diagrams at doc build time instead of committing them [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #63386:
URL: https://github.com/apache/airflow/pull/63386#discussion_r2922010963
##########
devel-common/src/docs/utils/conf_constants.py:
##########
@@ -332,6 +332,7 @@ def get_google_intersphinx_mapping() -> dict[str, tuple[str, tuple[str]]]:
"*/airflow/executors/*",
"*/_internal*",
"*/node_modules/*",
+ "*/.pnpm-store/*",
Review Comment:
We only add `"*/.pnpm-store/*",` to `BASIC_AUTOAPI_IGNORE_PATTERNS` in this PR. Does it really make "Generate ERD" from commit time to doc build time? The change doesn't seem related to me, but maybe I'm wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate ERD diagrams at doc build time instead of committing them [airflow]
Posted by "jason810496 (via GitHub)" <gi...@apache.org>.
jason810496 commented on code in PR #63386:
URL: https://github.com/apache/airflow/pull/63386#discussion_r2922010963
##########
devel-common/src/docs/utils/conf_constants.py:
##########
@@ -332,6 +332,7 @@ def get_google_intersphinx_mapping() -> dict[str, tuple[str, tuple[str]]]:
"*/airflow/executors/*",
"*/_internal*",
"*/node_modules/*",
+ "*/.pnpm-store/*",
Review Comment:
We only add `"*/.pnpm-store/*",` to `BASIC_AUTOAPI_IGNORE_PATTERNS` in this PR. Does it really make "Generate ERD" from commit time to doc build time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Generate ERD diagrams at doc build time instead of committing them [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #63386:
URL: https://github.com/apache/airflow/pull/63386#discussion_r2922017609
##########
devel-common/src/docs/utils/conf_constants.py:
##########
@@ -332,6 +332,7 @@ def get_google_intersphinx_mapping() -> dict[str, tuple[str, tuple[str]]]:
"*/airflow/executors/*",
"*/_internal*",
"*/node_modules/*",
+ "*/.pnpm-store/*",
Review Comment:
Just realized that... That was bad rebase :) - rebased only last commit by mistake :)
This is complete change
https://github.com/apache/airflow/pull/63402
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org