You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "fpopic (via GitHub)" <gi...@apache.org> on 2025/02/06 08:29:39 UTC
[PR] Fix hashicorp provider vault gcp auth [airflow]
fpopic opened a new pull request, #46505:
URL: https://github.com/apache/airflow/pull/46505
closes issue: https://github.com/apache/airflow/issues/17500
continuation of closed PR: https://github.com/apache/airflow/pull/35855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix hashicorp provider vault gcp auth [airflow]
Posted by "aaabramov (via GitHub)" <gi...@apache.org>.
aaabramov commented on code in PR #46505:
URL: https://github.com/apache/airflow/pull/46505#discussion_r1950581744
##########
providers/hashicorp/src/airflow/providers/hashicorp/_internal_client/vault_client.py:
##########
@@ -299,13 +306,47 @@ def _auth_gcp(self, _client: hvac.Client) -> None:
)
scopes = _get_scopes(self.gcp_scopes)
- credentials, _ = get_credentials_and_project_id(
+ credentials, project_id = get_credentials_and_project_id(
key_path=self.gcp_key_path, keyfile_dict=self.gcp_keyfile_dict, scopes=scopes
)
+
+ import time
+ import json
+ import googleapiclient
+
+ with open(self.gcp_key_path, 'r') as f:
Review Comment:
This will work only with a mounted credentials file. We also have to handle cases when Airflow is running using workload identity (or Composer).
Maybe we can reuse gcloud SDK to create a signed JWT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix hashicorp provider vault gcp auth [airflow]
Posted by "aaabramov (via GitHub)" <gi...@apache.org>.
aaabramov commented on code in PR #46505:
URL: https://github.com/apache/airflow/pull/46505#discussion_r1950581744
##########
providers/hashicorp/src/airflow/providers/hashicorp/_internal_client/vault_client.py:
##########
@@ -299,13 +306,47 @@ def _auth_gcp(self, _client: hvac.Client) -> None:
)
scopes = _get_scopes(self.gcp_scopes)
- credentials, _ = get_credentials_and_project_id(
+ credentials, project_id = get_credentials_and_project_id(
key_path=self.gcp_key_path, keyfile_dict=self.gcp_keyfile_dict, scopes=scopes
)
+
+ import time
+ import json
+ import googleapiclient
+
+ with open(self.gcp_key_path, 'r') as f:
Review Comment:
This will work only with a mounted credentials file. We also have to handle cases when Airflow is running using workload identity (or Composer).
Maybe we can reuse gcloud SDK to create a signed JWT?
Please correct me if I am wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix hashicorp provider vault gcp auth [airflow]
Posted by "fpopic (via GitHub)" <gi...@apache.org>.
fpopic commented on code in PR #46505:
URL: https://github.com/apache/airflow/pull/46505#discussion_r1951497666
##########
providers/hashicorp/src/airflow/providers/hashicorp/_internal_client/vault_client.py:
##########
@@ -299,13 +306,47 @@ def _auth_gcp(self, _client: hvac.Client) -> None:
)
scopes = _get_scopes(self.gcp_scopes)
- credentials, _ = get_credentials_and_project_id(
+ credentials, project_id = get_credentials_and_project_id(
key_path=self.gcp_key_path, keyfile_dict=self.gcp_keyfile_dict, scopes=scopes
)
+
+ import time
+ import json
+ import googleapiclient
+
+ with open(self.gcp_key_path, 'r') as f:
Review Comment:
@aaabramov, my basic understanding:
the code above is calling
https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/utils/credentials_provider.py#L416-L418
that tries the following methods
https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/utils/credentials_provider.py#L251-L297
once any crednetials is resolved this open(...) will just read the credential and use it for jwt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix hashicorp provider vault gcp auth [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #46505:
URL: https://github.com/apache/airflow/pull/46505#issuecomment-2795474749
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix hashicorp provider vault gcp auth [airflow]
Posted by "messam88 (via GitHub)" <gi...@apache.org>.
messam88 commented on PR #46505:
URL: https://github.com/apache/airflow/pull/46505#issuecomment-2658863862
Could you @potiuk @hussein-awala again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix hashicorp provider vault gcp auth [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #46505:
URL: https://github.com/apache/airflow/pull/46505#issuecomment-2663063008
I am afraid the tests are failing. Rebased it to check but look like problems to be solved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix hashicorp provider vault gcp auth [airflow]
Posted by "fpopic (via GitHub)" <gi...@apache.org>.
fpopic commented on code in PR #46505:
URL: https://github.com/apache/airflow/pull/46505#discussion_r1951497666
##########
providers/hashicorp/src/airflow/providers/hashicorp/_internal_client/vault_client.py:
##########
@@ -299,13 +306,47 @@ def _auth_gcp(self, _client: hvac.Client) -> None:
)
scopes = _get_scopes(self.gcp_scopes)
- credentials, _ = get_credentials_and_project_id(
+ credentials, project_id = get_credentials_and_project_id(
key_path=self.gcp_key_path, keyfile_dict=self.gcp_keyfile_dict, scopes=scopes
)
+
+ import time
+ import json
+ import googleapiclient
+
+ with open(self.gcp_key_path, 'r') as f:
Review Comment:
@aaabramov, my basic understanding:
the code above is calling
https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/utils/credentials_provider.py#L416-L418
which calls
https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/utils/credentials_provider.py#L416-L418
that tries the following methods
https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/utils/credentials_provider.py#L251-L297
once any crednetials is resolved this open(...) will just read the credential and use it for jwt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix hashicorp provider vault gcp auth [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #46505:
URL: https://github.com/apache/airflow/pull/46505#issuecomment-2639741714
A number of tests/checks to fix though :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix hashicorp provider vault gcp auth [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #46505: Fix hashicorp provider vault gcp auth
URL: https://github.com/apache/airflow/pull/46505
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix hashicorp provider vault gcp auth [airflow]
Posted by "fpopic (via GitHub)" <gi...@apache.org>.
fpopic commented on code in PR #46505:
URL: https://github.com/apache/airflow/pull/46505#discussion_r1951497666
##########
providers/hashicorp/src/airflow/providers/hashicorp/_internal_client/vault_client.py:
##########
@@ -299,13 +306,47 @@ def _auth_gcp(self, _client: hvac.Client) -> None:
)
scopes = _get_scopes(self.gcp_scopes)
- credentials, _ = get_credentials_and_project_id(
+ credentials, project_id = get_credentials_and_project_id(
key_path=self.gcp_key_path, keyfile_dict=self.gcp_keyfile_dict, scopes=scopes
)
+
+ import time
+ import json
+ import googleapiclient
+
+ with open(self.gcp_key_path, 'r') as f:
Review Comment:
@aaabramov, my basic understanding:
the code above is calling
https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/utils/credentials_provider.py#L416-L418
that tries the following methods
https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/utils/credentials_provider.py#L251-L297
once current credentials is resolved this `open(...)` will just read the current credential material and use it for jwt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix hashicorp provider vault gcp auth [airflow]
Posted by "fpopic (via GitHub)" <gi...@apache.org>.
fpopic commented on code in PR #46505:
URL: https://github.com/apache/airflow/pull/46505#discussion_r1951497666
##########
providers/hashicorp/src/airflow/providers/hashicorp/_internal_client/vault_client.py:
##########
@@ -299,13 +306,47 @@ def _auth_gcp(self, _client: hvac.Client) -> None:
)
scopes = _get_scopes(self.gcp_scopes)
- credentials, _ = get_credentials_and_project_id(
+ credentials, project_id = get_credentials_and_project_id(
key_path=self.gcp_key_path, keyfile_dict=self.gcp_keyfile_dict, scopes=scopes
)
+
+ import time
+ import json
+ import googleapiclient
+
+ with open(self.gcp_key_path, 'r') as f:
Review Comment:
@aaabramov, my basic understanding:
the code above is calling
https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/utils/credentials_provider.py#L416-L418
that tries the following methods
https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/utils/credentials_provider.py#L251-L297
once current credential is resolved this `open(...)` will just read the current credential material and use it for jwt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org