You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vitstransky (via GitHub)" <gi...@apache.org> on 2025/11/05 14:39:35 UTC
[I] Use Google-signed OpenID Connect in Databricks Connection [airflow]
vitstransky opened a new issue, #57867:
URL: https://github.com/apache/airflow/issues/57867
### Description
Hello,
we have in Databricks mapped service accounts from GCP. The accounts can be authenticated with Google ID Tokens. Databricks Connection doesn't support this type of tokens (or I wasn't able to figure out how). It could be done similar to AAD token.
### Use case/motivation
- In Airflow, we have set GOOGLE_APPLICATION_CREDENTIALS to authenticate GCP service account A.
- In GCP and Databricks we have service account B.
- Service account A can impersonate service account B.
- Using AuthorizedSession with default credentials and https://docs.cloud.google.com/docs/authentication/get-id-token#impersonation we can get ID token for service account B at runtime.
- This token can be then used to authenticate Databricks API as service account B.
### Related issues
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use Google-signed OpenID Connect in Databricks Connection [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #57867:
URL: https://github.com/apache/airflow/issues/57867#issuecomment-3986427279
@hwang-cadent We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use Google-signed OpenID Connect in Databricks Connection [airflow]
Posted by "ynpreet (via GitHub)" <gi...@apache.org>.
ynpreet commented on issue #57867:
URL: https://github.com/apache/airflow/issues/57867#issuecomment-6298974427
Hi, I'd like to pick this up — I see the previous assignee was unassigned under the assignment policy, so I'm commenting here as suggested before opening a PR.
Approach, following the existing AAD token pattern in `BaseDatabricksHook`:
- New opt-in auth branch in `_get_token()` / `_a_get_token()` behind a `use_google_id_token` connection-extra flag, with the ID-token audience and an optional impersonated service account configured via connection extras.
- At runtime, obtain the Google ID token with google-auth: `google.auth.default()` + `AuthorizedSession` against the IAM Credentials `generateIdToken` endpoint when impersonating service account B, or `id_token.fetch_id_token` for the default credentials otherwise; cache the JWT and refresh it shortly before expiry.
- Unit tests mirroring the AAD token tests, with all google-auth calls mocked (no real GCP calls).
I'll open a PR once it's ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use Google-signed OpenID Connect in Databricks Connection [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #57867:
URL: https://github.com/apache/airflow/issues/57867#issuecomment-3491588422
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use Google-signed OpenID Connect in Databricks Connection [airflow]
Posted by "hwang-cadent (via GitHub)" <gi...@apache.org>.
hwang-cadent commented on issue #57867:
URL: https://github.com/apache/airflow/issues/57867#issuecomment-3517432237
@vitstransky Thank you for providing the code, The _is_jwt_token_valid() method is more accurate for ID tokens
I will consider to use it for Google ID tokens instead of OAuth validation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use Google-signed OpenID Connect in Databricks Connection [airflow]
Posted by "hwang-cadent (via GitHub)" <gi...@apache.org>.
hwang-cadent commented on issue #57867:
URL: https://github.com/apache/airflow/issues/57867#issuecomment-3512658284
Can I work on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use Google-signed OpenID Connect in Databricks Connection [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #57867:
URL: https://github.com/apache/airflow/issues/57867#issuecomment-3512817628
Feel free.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use Google-signed OpenID Connect in Databricks Connection [airflow]
Posted by "vitstransky (via GitHub)" <gi...@apache.org>.
vitstransky commented on issue #57867:
URL: https://github.com/apache/airflow/issues/57867#issuecomment-3515171514
Hello @hwang-cadent,
please find my working example attached. I hope you find it helpful.
[databricks_base.py](https://github.com/user-attachments/files/23470741/databricks_base.py)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Use Google-signed OpenID Connect in Databricks Connection [airflow]
Posted by "hwang-cadent (via GitHub)" <gi...@apache.org>.
hwang-cadent commented on issue #57867:
URL: https://github.com/apache/airflow/issues/57867#issuecomment-3517504913
> Hello [@hwang-cadent](https://github.com/hwang-cadent), please find my working example attached. I hope you find it helpful.
>
> [databricks_base.py](https://github.com/user-attachments/files/23470741/databricks_base.py)
and the fetch_id_token() for non-impersonated case, it could simplify the code path when no impersonation is needed. That's cool. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org