You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "antonio-mello-ai (via GitHub)" <gi...@apache.org> on 2026/03/12 18:52:51 UTC
[PR] fix(providers/git): force HTTP auth for public repos via extraHeader [airflow]
antonio-mello-ai opened a new pull request, #63471:
URL: https://github.com/apache/airflow/pull/63471
## Description
Git does not send credentials for public repositories because the server never responds with a 401 challenge. This causes Airflow's DAG bundle git fetches to hit **anonymous rate limits** even when valid credentials are configured in the connection.
### Root cause
Git's HTTPS behavior: it always tries anonymous access first and only sends credentials if the server responds with 401. Public repos don't issue 401, so credentials embedded in the URL are never sent.
### Fix
When an HTTP/HTTPS connection has an `auth_token`, set `GIT_CONFIG_*` environment variables to inject an `http.extraHeader` with a `Basic` auth token:
```
GIT_CONFIG_COUNT=1
GIT_CONFIG_KEY_0=http.extraHeader
GIT_CONFIG_VALUE_0=Authorization: Basic <base64(username:password)>
```
This forces git to send the `Authorization` header on every request (git >= 2.31), allowing authenticated rate limits to apply even for public repositories.
Also updated `_fetch_bare_repo` in `GitDagBundle` to pass **all** hook env vars via `custom_environment(**self.hook.env)` instead of only `GIT_SSH_COMMAND`. This ensures the auth header is used during both clone and fetch operations.
### Changes
- `GitHook._set_http_auth_env()`: New method that encodes credentials and sets `GIT_CONFIG_*` env vars.
- `GitHook._process_git_auth_url()`: Calls `_set_http_auth_env()` when HTTPS/HTTP + auth_token.
- `GitDagBundle._fetch_bare_repo()`: Passes all env vars (SSH + HTTP auth) instead of only SSH.
### Backward compatibility
- URL-embedded credentials are **kept** for backward compatibility (they still work for private repos via 401 challenge).
- SSH connections are not affected (no `GIT_CONFIG_*` vars are set for SSH URLs).
- The `GIT_CONFIG_*` env vars require git >= 2.31 (March 2021).
### Tests
- 4 new tests: HTTPS auth sets header, HTTP auth sets header, no auth skips header, SSH skips header.
- All 17 hook tests + 80 bundle tests pass.
Closes #54829
Co-Authored-By: Claude Opus 4.6 <no...@anthropic.com>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(providers/git): force HTTP auth for public repos via extraHeader [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #63471:
URL: https://github.com/apache/airflow/pull/63471#issuecomment-4299345464
@antonio-mello-ai Thanks for your earlier work on this PR — there are still two unresolved threads plus a CI failure that need your attention. This PR has been converted to **draft** because it does not yet meet our [Pull Request quality criteria](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-quality-criteria).
**Issues found:**
- :x: **Failing CI**: `Special tests / Latest Boto test (amazon)` is failing. Please investigate with `breeze testing providers-tests --test-type "Providers[amazon]"` and check for a Boto API change.
- :x: **Unresolved review comments** (2 threads): please walk through each unresolved review thread. **Even if a suggestion looks incorrect or irrelevant** — and some of them will be, especially any comments left by automated reviewers like GitHub Copilot — it is still the author's responsibility to respond: apply the fix, reply in-thread with a brief explanation of why the suggestion does not apply, or resolve the thread if the feedback is no longer relevant. Leaving threads unaddressed for weeks blocks the PR from moving forward.
**What to do next:**
- Walk through each unresolved review thread and respond as described above.
- Fix the LatestBoto regression and push.
- Make sure static checks pass locally: `prek run --from-ref main --stage pre-commit`.
- Mark the PR as "Ready for review" when you're done.
Converting a PR to draft is **not** a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the [Airflow Slack](https://s.apache.org/airflow-slack).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(providers/git): force HTTP auth for public repos via extraHeader [airflow]
Posted by "antonio-mello-ai (via GitHub)" <gi...@apache.org>.
antonio-mello-ai commented on PR #63471:
URL: https://github.com/apache/airflow/pull/63471#issuecomment-4233240920
Rebased on latest `main`. Zero conflicts, none of the recent commits touched `providers/git/`. Full `Providers[git]` suite (105 tests) passes locally.
One flake in CI: `test_example_dag_hashes_are_always_consistent[raw-serialized_dags]` under `LowestDeps`. The `compress` variant of the same test passes in the same job, so it reads as hash-order non-determinism in the minimum-pinned serialization libs rather than anything this PR touches (scope is just `providers/git/`).
Happy to rerun if that helps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(providers/git): force HTTP auth for public repos via extraHeader [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #63471: fix(providers/git): force HTTP auth for public repos via extraHeader
URL: https://github.com/apache/airflow/pull/63471
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(providers/git): force HTTP auth for public repos via extraHeader [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #63471:
URL: https://github.com/apache/airflow/pull/63471#issuecomment-4195296596
@antonio-mello-ai This PR has a few issues that need to be addressed before it can be reviewed — please see our [Pull Request quality criteria](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-quality-criteria).
**Issues found:**
- :x: **Merge conflicts**: This PR has merge conflicts with the `main` branch. Your branch is 44 commits behind `main`. Please rebase your branch (`git fetch origin && git rebase origin/main`), resolve the conflicts, and push again. See [contributing quick start](https://github.com/apache/airflow/blob/main/contributing-docs/03a_contributors_quick_start_beginners.rst).
**What to do next:**
- The comment informs you what you need to do.
- Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
- There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
- Maintainers will then proceed with a normal review.
There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the [Airflow Slack](https://s.apache.org/airflow-slack).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(providers/git): force HTTP auth for public repos via extraHeader [airflow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #63471:
URL: https://github.com/apache/airflow/pull/63471#discussion_r3048482267
##########
providers/git/src/airflow/providers/git/hooks/git.py:
##########
@@ -149,16 +150,35 @@ def _process_git_auth_url(self):
encoded_user = urlquote(self.user_name, safe="")
encoded_token = urlquote(self.auth_token, safe="")
self.repo_url = self.repo_url.replace("https://", f"https://{encoded_user}:{encoded_token}@", 1)
+ self._set_http_auth_env()
elif self.auth_token and self.repo_url.startswith("http://"):
encoded_user = urlquote(self.user_name, safe="")
encoded_token = urlquote(self.auth_token, safe="")
self.repo_url = self.repo_url.replace("http://", f"http://{encoded_user}:{encoded_token}@", 1)
+ self._set_http_auth_env()
elif self.repo_url.startswith("http://"):
# if no auth token, use the repo url as is
pass
elif not self.repo_url.startswith("git@") and not self.repo_url.startswith("https://"):
self.repo_url = os.path.expanduser(self.repo_url)
+ def _set_http_auth_env(self):
+ """
+ Set git config env vars to force HTTP authentication via extraHeader.
+
+ Git does not send credentials for public repositories since the server
+ does not respond with a 401 challenge. This forces the Authorization
+ header to be sent on every request, allowing authenticated rate limits.
+
+ Uses GIT_CONFIG_* environment variables (git >= 2.31) to inject an
+ ``http.extraHeader`` with a Basic auth token.
+ """
+ credentials = f"{self.user_name}:{self.auth_token}"
+ encoded = base64.b64encode(credentials.encode()).decode()
+ self.env["GIT_CONFIG_COUNT"] = "1"
+ self.env["GIT_CONFIG_KEY_0"] = "http.extraHeader"
+ self.env["GIT_CONFIG_VALUE_0"] = f"Authorization: Basic {encoded}"
Review Comment:
`http.extraHeader` is configured globally via `GIT_CONFIG_KEY_0=http.extraHeader`, so the Authorization header may be sent to *any* HTTP(S) endpoint Git contacts during this operation (redirects, alternates, or other HTTP remotes), which is a credential-leak risk. Consider scoping the config to the target host by using Git’s per-URL config form (e.g. `http.<url>.extraHeader` derived from the repo URL without embedded credentials), so the header is only attached for requests to that host/prefix.
##########
providers/git/src/airflow/providers/git/hooks/git.py:
##########
@@ -149,16 +150,35 @@ def _process_git_auth_url(self):
encoded_user = urlquote(self.user_name, safe="")
encoded_token = urlquote(self.auth_token, safe="")
self.repo_url = self.repo_url.replace("https://", f"https://{encoded_user}:{encoded_token}@", 1)
+ self._set_http_auth_env()
elif self.auth_token and self.repo_url.startswith("http://"):
encoded_user = urlquote(self.user_name, safe="")
encoded_token = urlquote(self.auth_token, safe="")
self.repo_url = self.repo_url.replace("http://", f"http://{encoded_user}:{encoded_token}@", 1)
+ self._set_http_auth_env()
elif self.repo_url.startswith("http://"):
# if no auth token, use the repo url as is
pass
elif not self.repo_url.startswith("git@") and not self.repo_url.startswith("https://"):
self.repo_url = os.path.expanduser(self.repo_url)
+ def _set_http_auth_env(self):
+ """
+ Set git config env vars to force HTTP authentication via extraHeader.
+
+ Git does not send credentials for public repositories since the server
+ does not respond with a 401 challenge. This forces the Authorization
+ header to be sent on every request, allowing authenticated rate limits.
+
+ Uses GIT_CONFIG_* environment variables (git >= 2.31) to inject an
+ ``http.extraHeader`` with a Basic auth token.
+ """
Review Comment:
This relies on `GIT_CONFIG_*` env vars which require git >= 2.31, but the code doesn’t detect/handle older git versions. In environments with older git, the new env vars will be ignored and public-repo auth will still not be forced (leaving the original rate-limit issue unresolved). Please consider adding a runtime fallback (e.g. using an older supported mechanism such as `GIT_CONFIG_PARAMETERS` / `-c` injection) or at least emitting a clear warning/error when an auth token is configured but the installed git is too old.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(providers/git): force HTTP auth for public repos via extraHeader [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #63471:
URL: https://github.com/apache/airflow/pull/63471#issuecomment-4300310903
Quick follow-up to the triage comment above — one clarification on the "Unresolved review comments" item:
Once you believe a thread has been addressed — whether by pushing a fix, or by replying in-thread with an explanation of why the suggestion doesn't apply — **please mark the thread as resolved yourself** by clicking the "Resolve conversation" button at the bottom of each thread. Reviewers don't auto-close their own threads, so an addressed-but-unresolved thread reads as "still waiting on the author" and keeps the PR from moving forward. The author doing the resolve-click is the expected convention on this project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(providers/git): force HTTP auth for public repos via extraHeader [airflow]
Posted by "antonio-mello-ai (via GitHub)" <gi...@apache.org>.
antonio-mello-ai commented on PR #63471:
URL: https://github.com/apache/airflow/pull/63471#issuecomment-4173821134
Friendly follow-up — this PR has been open for 20 days with all 133 CI checks passing and no conflicts. Would appreciate any feedback or direction. Happy to adjust anything needed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(providers/git): force HTTP auth for public repos via extraHeader [airflow]
Posted by "antonio-mello-ai (via GitHub)" <gi...@apache.org>.
antonio-mello-ai commented on PR #63471:
URL: https://github.com/apache/airflow/pull/63471#issuecomment-4115043452
Friendly ping -- this PR has been waiting for review for ~9 days. CI is green (133/133 checks passing), no conflicts. Happy to address any feedback. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(providers/git): force HTTP auth for public repos via extraHeader [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #63471:
URL: https://github.com/apache/airflow/pull/63471#issuecomment-5022606824
@antonio-mello-ai This draft PR has been inactive for 89 days since the last triage comment and no response from the author. Closing to keep the queue clean.
You are welcome to reopen this PR when you resume work, or to open a new one addressing the issues previously raised. There is no rush — take your time.
---
_Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this [two-stage triage process](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md#why-the-first-pass-is-automated) so that our maintainers' limited time is spent where it matters most: the conversation with you._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(providers/git): force HTTP auth for public repos via extraHeader [airflow]
Posted by "antonio-mello-ai (via GitHub)" <gi...@apache.org>.
antonio-mello-ai commented on code in PR #63471:
URL: https://github.com/apache/airflow/pull/63471#discussion_r3127820122
##########
providers/git/src/airflow/providers/git/hooks/git.py:
##########
@@ -149,16 +150,35 @@ def _process_git_auth_url(self):
encoded_user = urlquote(self.user_name, safe="")
encoded_token = urlquote(self.auth_token, safe="")
self.repo_url = self.repo_url.replace("https://", f"https://{encoded_user}:{encoded_token}@", 1)
+ self._set_http_auth_env()
elif self.auth_token and self.repo_url.startswith("http://"):
encoded_user = urlquote(self.user_name, safe="")
encoded_token = urlquote(self.auth_token, safe="")
self.repo_url = self.repo_url.replace("http://", f"http://{encoded_user}:{encoded_token}@", 1)
+ self._set_http_auth_env()
elif self.repo_url.startswith("http://"):
# if no auth token, use the repo url as is
pass
elif not self.repo_url.startswith("git@") and not self.repo_url.startswith("https://"):
self.repo_url = os.path.expanduser(self.repo_url)
+ def _set_http_auth_env(self):
+ """
+ Set git config env vars to force HTTP authentication via extraHeader.
+
+ Git does not send credentials for public repositories since the server
+ does not respond with a 401 challenge. This forces the Authorization
+ header to be sent on every request, allowing authenticated rate limits.
+
+ Uses GIT_CONFIG_* environment variables (git >= 2.31) to inject an
+ ``http.extraHeader`` with a Basic auth token.
+ """
+ credentials = f"{self.user_name}:{self.auth_token}"
+ encoded = base64.b64encode(credentials.encode()).decode()
+ self.env["GIT_CONFIG_COUNT"] = "1"
+ self.env["GIT_CONFIG_KEY_0"] = "http.extraHeader"
+ self.env["GIT_CONFIG_VALUE_0"] = f"Authorization: Basic {encoded}"
Review Comment:
Thanks. This is addressed in the current branch.
I changed the config from a global `http.extraHeader` to the per-URL form `http.<url>.extraHeader`, derived from the repo URL without embedded credentials, so the Authorization header is scoped to the target host/prefix instead of being sent globally.
That change is in `055930e245`, and the hook tests were updated to cover the URL-scoped behavior for both HTTP and HTTPS repository URLs.
##########
providers/git/src/airflow/providers/git/hooks/git.py:
##########
@@ -149,16 +150,35 @@ def _process_git_auth_url(self):
encoded_user = urlquote(self.user_name, safe="")
encoded_token = urlquote(self.auth_token, safe="")
self.repo_url = self.repo_url.replace("https://", f"https://{encoded_user}:{encoded_token}@", 1)
+ self._set_http_auth_env()
elif self.auth_token and self.repo_url.startswith("http://"):
encoded_user = urlquote(self.user_name, safe="")
encoded_token = urlquote(self.auth_token, safe="")
self.repo_url = self.repo_url.replace("http://", f"http://{encoded_user}:{encoded_token}@", 1)
+ self._set_http_auth_env()
elif self.repo_url.startswith("http://"):
# if no auth token, use the repo url as is
pass
elif not self.repo_url.startswith("git@") and not self.repo_url.startswith("https://"):
self.repo_url = os.path.expanduser(self.repo_url)
+ def _set_http_auth_env(self):
+ """
+ Set git config env vars to force HTTP authentication via extraHeader.
+
+ Git does not send credentials for public repositories since the server
+ does not respond with a 401 challenge. This forces the Authorization
+ header to be sent on every request, allowing authenticated rate limits.
+
+ Uses GIT_CONFIG_* environment variables (git >= 2.31) to inject an
+ ``http.extraHeader`` with a Basic auth token.
+ """
Review Comment:
Thanks. I addressed this in the current branch as well.
The hook now checks the installed git version before setting the `GIT_CONFIG_*` environment variables. When git is older than 2.31, it does not try to inject the extra header silently anymore and emits a clear warning instead.
That change is in `46a008a1c7e`, and there is a unit test covering the `git < 2.31` path (`test_old_git_does_not_set_extra_header`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org