You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jhgoebbert (via GitHub)" <gi...@apache.org> on 2025/08/22 11:26:17 UTC
[I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
jhgoebbert opened a new issue, #54829:
URL: https://github.com/apache/airflow/issues/54829
### Description
Hi everyone,
I recently came across the problem, that the git-fetch-requests of the dag-bundle hit the rate-limit of the git-server.
I found out that Airflow cannot **take advantage of a higher rate-limit for authorized git-requests** if the repo is public as it never sends the credentials to the server.
But Airflow could support to **force authentication** for a git connection type via a custom http.extraHeader as additional argument to the git command. This would allow authenticated Git operations even for public repositories.
### Use case/motivation
Git does not authenticate when syncing public repositories, even if login and password are set. This is due to Git’s HTTPS behavior: **it always tries anonymous access first** and only sends credentials if the server responds with a 401 Unauthorized.
=> Public repos typically do not issue a 401, so credentials are never used.
This is problematic because GitLab/GitHub apply different rate limits for authenticated vs anonymous requests. As a result, Airflow tasks can hit anonymous rate limits even when valid credentials are provided.
**A known workaround** is to use Git’s http.extraHeader option to force authentication:
```
B64=$(printf 'x:%s' "<PROJECT_ACCESS_TOKEN>" | base64)
git -c http.extraHeader="Authorization: Basic $B64" clone https://gitlab.com/<group>/<repo>.git
```
This ensures that all requests are authenticated from the first request, avoiding anonymous rate limits.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3214412457
CC @ephraimbuddy could you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "jhgoebbert (via GitHub)" <gi...@apache.org>.
jhgoebbert commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3214840763
https://docs.gitlab.com/administration/settings/git_http_rate_limits/
```
GitLab can enforce rate limits on both authenticated and unauthenticated Git HTTP requests to improve the security and durability of your web application.
```
The GitLab we are using has rate-limits enabled for unauthenticated Git HTTP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3219738054
> https://docs.gitlab.com/administration/settings/git_http_rate_limits/
>
> ```
> GitLab can enforce rate limits on both authenticated and unauthenticated Git HTTP requests to improve the security and durability of your web application.
> ```
>
> The GitLab we are using has rate-limits enabled for unauthenticated Git HTTP.
The link above looks like it's something you will set in the repo and not what Airflow should do or am I missing something?
For security reasons, I think that you should be the preferred way to force the authentication
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "jhgoebbert (via GitHub)" <gi...@apache.org>.
jhgoebbert commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3219781081
You can check this with
```
GIT_CURL_VERBOSE=1 git ls-remote https://<USERNAME>:<PASWORD>@gitlab.com/<NAMESPACE>/<REPO>.git
```
and search for authentication
```
Server auth using <...>
```
You will not find this line if the repo is public - so git accesses the server unauthorized even though you add a username+password.
The consequence is, that the server will serve with the rate-limits for unauthorized access.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3219896613
IMO, for security reasons, we cannot force auth header. Most we can do is to allow env vars that can be used per fetch/push, totally set by users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "jhgoebbert (via GitHub)" <gi...@apache.org>.
jhgoebbert commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3219913970
Ok, having the possibility to set `env` and `config-var` in `extra` would of cause make git-authentication to public repos already possible for experts.
Why do you think that sending the base64-encoded username+password in a dedicated header to the git-server is a security issue. This string would be build directly from the username and password values provided to the connection and the user would explicitly enable it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3214036353
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3219817565
Would it work for you if we allow users to provide extra environment variables instead of setting the extra header ourselves?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3455058236
is there a task left on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "jhgoebbert (via GitHub)" <gi...@apache.org>.
jhgoebbert commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3219842178
This would allow a lot of advanced `git`-features as one could not only use `--config-env` to force authentication.
(alternatively it would also be an option to explicitly add an extra-flag `force-auth-header` to allow forcing of authentication - which would make it easier to use for normal users)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "jhgoebbert (via GitHub)" <gi...@apache.org>.
jhgoebbert commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3219825468
It would work, if it would be possible to pass an extra environment to the command `git` via git's command-line argument `--config-env`.
```
export MY_HEADER="Authorization: Basic <BASE64>"
git --config-env=http.extraheader=MY_HEADER clone <repo>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "jhgoebbert (via GitHub)" <gi...@apache.org>.
jhgoebbert commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3214845430
For GitHub you can find this here:
https://docs.github.com/de/rest/using-the-rest-api/rate-limits-for-the-rest-api
`The primary rate limit for unauthenticated requests is 60 requests per hour.`
(I have only tested GitLab and can only refer to GitHub's documentation here)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3220044499
You can disable this behaviour @jhgoebbert - this is standard feature of git that can be configured via git config:
https://git-scm.com/docs/git-config#Documentation/git-config.txt-httpproactiveAuth (set to True)
> # http.proactiveAuth
> Attempt authentication without first making an unauthenticated attempt and receiving a 401 response. This can be used to > ensure that all requests are authenticated. If http.emptyAuth is set to true, this value has no effect.
> If the credential helper used specifies an authentication scheme (i.e., via the authtype field), that value will be used; if a username and password is provided without a scheme, then Basic authentication is used. The value of the option determines the scheme requested from the helper. Possible values are:
> basic - Request Basic authentication from the helper.
> auto - Allow the helper to pick an appropriate scheme.
> none - Disable proactive authentication.
> Note that TLS should always be used with this configuration, since otherwise it is easy to accidentally expose plaintext credentials if Basic authentication is selected.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "jhgoebbert (via GitHub)" <gi...@apache.org>.
jhgoebbert commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3457624138
Sorry, I had no time yet to fix this. But it is on my todo-list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "antonio-mello-ai (via GitHub)" <gi...@apache.org>.
antonio-mello-ai commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-4048951108
Hi @jhgoebbert @eladkal — is this still open? I'd be happy to pick it up if no one is actively working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-4048982546
We don't assign issues. the first PR to be ready will be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "jhgoebbert (via GitHub)" <gi...@apache.org>.
jhgoebbert commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3220207132
@potiuk Oh, great! That simplifies it a lot and we do not create/send the auth header ourselfs via `--config-env`.
The pure `git`-command would then look like this `git -c http.proactiveAuth=true clone https://<USERNAME:<PASSWD>@<REPO>`
@ephraimbuddy Yes, sure. I can look into this and create a pull-request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3220378041
> **note:** `http.proactiveAuth` is relative new (Git version 2.46 release on Jul 29, 2024) and was added in July 2024 here [git/git@610cbc1](https://github.com/git/git/commit/610cbc1dfb4ad947b82d7312c54c7987177f979f)
Good. More than year. That's old-ish by my account. Using newer version of software (especially such as git) is better for security, especially around authentication.
BTW. If you add anything here @jhgoebbert , my suggestion is to simply add a capability of easier passing configuration variables (any) to git - possibly even adding them via extras. Choosing just **this one** config is not a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3214816098
@jhgoebbert , can you point me to the documentation on this rate limiting?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "EncrypterParv (via GitHub)" <gi...@apache.org>.
EncrypterParv commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-6264396557
I looked into the previous attempts for this issue. PR #63471 used `http.extraHeader` and was closed as stale; PR #70204 replaced that with URL-scoped `http.<URL>.proactiveAuth=basic`, limited to HTTPS and Git 2.46+, and was also closed as stale.
I’m considering working from the #70204 approach rather than resubmitting the `extraHeader` patch. Would maintainers prefer that approach, or is there a reason it should not be revived? I’ll hold off on opening another PR until I understand the intended direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "jhgoebbert (via GitHub)" <gi...@apache.org>.
jhgoebbert commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-4048999248
@antonio-mello-ai Yes, please - go ahead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "jhgoebbert (via GitHub)" <gi...@apache.org>.
jhgoebbert commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3219753337
So GitLab/GitHub is not the issue here. The issue is on the side of the `git`-command itself.
The command `git` does not authenticate when syncing public repositories, even if login and password are set. This is due to the `git`-command’s HTTPS behavior: it always tries anonymous access first and only sends credentials if the server responds with a 401 Unauthorized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3220187589
> Ok, having the possibility to set `env` and `config-var` in `extra` would of cause make git-authentication to public repos already possible for experts.
>
> Why do you think that sending the base64-encoded username+password in a dedicated header to the git-server is a security issue. This string would be build directly from the username and password values provided to the connection and the user would explicitly enable it.
I'm not sure how it would be exploited but I feel like each time there's a request, there would be raw headers sent over to the url. Moreover, we use gitpython, if you can help implementing this feature, as you already know how to reproduce it, that would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "jhgoebbert (via GitHub)" <gi...@apache.org>.
jhgoebbert commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3220229746
**note:** `http.proactiveAuth` is relative new and was added in July 2024 here https://github.com/git/git/commit/610cbc1dfb4ad947b82d7312c54c7987177f979f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DAG-Bundle : Git connections ignore credentials for public repositories, causing anonymous rate limit issues [airflow]
Posted by "jhgoebbert (via GitHub)" <gi...@apache.org>.
jhgoebbert commented on issue #54829:
URL: https://github.com/apache/airflow/issues/54829#issuecomment-3220386363
> BTW. If you add anything here @jhgoebbert , my suggestion is to simply add a capability of easier passing configuration variables (any) to git - possibly even adding them via extras. Choosing just this one config is not a good idea.
Ok, I will do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org