You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jscheffl (via GitHub)" <gi...@apache.org> on 2026/05/16 20:55:18 UTC
[PR] Add Git Sparse Checkout to Git Dag Bundle [airflow]
jscheffl opened a new pull request, #67047:
URL: https://github.com/apache/airflow/pull/67047
Git Dag Budle does currently always a full clone of the Git repo. Not good in cases when you run Airflow on a big monorepo.
This PR adds support for Git Sparse Checkout.
I experimented a bit around, initial clone is still large as if first bare close in made with `--filter=blob:none` which would be optimal, then the local clones of the bare are not able to resolve the object SHAs and miss reference. Due to structure with clone from bare clone I have no idea to slim initial clone down. Might be future improvement for a Git expert. Or strategy with clone from bare clone need to be revised.
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ ] Yes (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Git Sparse Checkout to Git Dag Bundle [airflow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #67047:
URL: https://github.com/apache/airflow/pull/67047#discussion_r3255110462
##########
providers/git/tests/unit/git/bundles/test_git.py:
##########
@@ -839,6 +840,39 @@ def test_subdir(self, mock_githook, git_repo):
assert str(bundle.path).endswith(subdir)
assert {"some_new_file.py"} == files_in_repo
+ @mock.patch("airflow.providers.git.bundles.git.GitHook")
+ def test_sparse_checkout(self, mock_githook, git_repo):
+ repo_path, repo = git_repo
+ mock_githook.return_value.repo_url = repo_path
+
+ subdir = "some/subdir"
+ subdir_path = repo_path / subdir
+ subdir_path.mkdir(parents=True)
+ file_path = subdir_path / "some_relevant_file.py"
+ with open(file_path, "w") as f:
+ f.write("hello world")
+ otherdir = "other/dir"
+ otherdir_path = repo_path / otherdir
+ otherdir_path.mkdir(parents=True)
+ otherfile_path = otherdir_path / "some_other_file.py"
+ with open(otherfile_path, "w") as f:
+ f.write("hello world")
+
+ repo.index.add([file_path, otherfile_path])
+ repo.index.commit("Other commit")
+
+ bundle = GitDagBundle(
+ name="test-sparse",
+ git_conn_id=CONN_HTTPS,
+ tracking_ref=GIT_DEFAULT_BRANCH,
+ sparse_dirs=[subdir],
+ )
+ bundle.initialize()
+
+ files_in_repo = {f.name for f in bundle.path.glob("**/*.py") if f.is_file()}
+ assert "some_other_file.py" not in files_in_repo
+ assert "some_relevant_file.py" in files_in_repo
+
Review Comment:
The new test only covers the fresh, unversioned initial clone path. The bundle has additional execution paths affected by sparse checkout that are not exercised: (1) `self.version` is set, in which case `_initialize` creates a separate worktree under `versions_dir/<version>` and does `head.reset(index=True, working_tree=True)`, and (2) `prune_dotgit_folder=True` (the default), which deletes `.git` after checkout. Consider adding at least one test that combines `sparse_dirs` with a pinned `version` and/or with `prune_dotgit_folder=True` to guard against regressions in those paths.
##########
providers/git/docs/bundles/index.rst:
##########
@@ -34,9 +34,10 @@ Example of using the GitDagBundle:
"kwargs": {
"subdir": "dags",
"tracking_ref": "main",
- "refresh_interval": 3600
+ "refresh_interval": 3600,
"submodules": False,
Review Comment:
The example is wrapped in a JSON-formatted shell environment variable (`AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST='[ ... ]'`), so booleans must be lowercase JSON literals. This change correctly switches `True` → `true` for `prune_dotgit_folder`, but the adjacent `"submodules": False` is left as a Python-style capitalized boolean, making the snippet still invalid JSON. Please also lowercase `False` → `false` on line 38 so the documented example actually parses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Git Sparse Checkout to Git Dag Bundle [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #67047:
URL: https://github.com/apache/airflow/pull/67047
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add Git Sparse Checkout to Git Dag Bundle [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #67047:
URL: https://github.com/apache/airflow/pull/67047#discussion_r3255368098
##########
providers/git/tests/unit/git/bundles/test_git.py:
##########
@@ -839,6 +840,39 @@ def test_subdir(self, mock_githook, git_repo):
assert str(bundle.path).endswith(subdir)
assert {"some_new_file.py"} == files_in_repo
+ @mock.patch("airflow.providers.git.bundles.git.GitHook")
+ def test_sparse_checkout(self, mock_githook, git_repo):
+ repo_path, repo = git_repo
+ mock_githook.return_value.repo_url = repo_path
+
+ subdir = "some/subdir"
+ subdir_path = repo_path / subdir
+ subdir_path.mkdir(parents=True)
+ file_path = subdir_path / "some_relevant_file.py"
+ with open(file_path, "w") as f:
+ f.write("hello world")
+ otherdir = "other/dir"
+ otherdir_path = repo_path / otherdir
+ otherdir_path.mkdir(parents=True)
+ otherfile_path = otherdir_path / "some_other_file.py"
+ with open(otherfile_path, "w") as f:
+ f.write("hello world")
+
+ repo.index.add([file_path, otherfile_path])
+ repo.index.commit("Other commit")
+
+ bundle = GitDagBundle(
+ name="test-sparse",
+ git_conn_id=CONN_HTTPS,
+ tracking_ref=GIT_DEFAULT_BRANCH,
+ sparse_dirs=[subdir],
+ )
+ bundle.initialize()
+
+ files_in_repo = {f.name for f in bundle.path.glob("**/*.py") if f.is_file()}
+ assert "some_other_file.py" not in files_in_repo
+ assert "some_relevant_file.py" in files_in_repo
+
Review Comment:
Manually applied
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org