You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kiranpreet-kaur17 (via GitHub)" <gi...@apache.org> on 2026/01/29 16:33:59 UTC

[I] CloudSecretManagerBackend fails with ADC when explicit project_id provided (regression in 19.4.0) [airflow]

kiranpreet-kaur17 opened a new issue, #61217:
URL: https://github.com/apache/airflow/issues/61217

   ### Apache Airflow Provider(s)
   
   google
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-google==19.4.0
   
   ### Apache Airflow version
   
   3.x affected
   
   ### Operating System
   
   All
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   N/A
   
   ### What happened
   
   After upgrading to `apache-airflow-providers-google==19.4.0`, `CloudSecretManagerBackend` fails to initialize when using Application Default Credentials (ADC) without a default project configured in gcloud, even when an explicit `project_id` is provided in `BACKEND_KWARGS`.
   
   This issue was encountered in a local development environment.
   
   **Error:**
   ```
   airflow.exceptions.AirflowException: Project ID could not be determined from default credentials. Please provide `key_secret_project_id` parameter.
   ```
   
   **Configuration:**
   ```bash
   AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
   AIRFLOW__SECRETS__BACKEND_KWARGS='{"project_id":"my-project-id"}'
   
   # Using ADC without default project (common when working with multiple projects)
   gcloud auth application-default login
   gcloud config unset project
   ```
   
   This configuration worked fine in version 19.3.0 and earlier.
   
   ### What you think should happen instead
   
   
   `CloudSecretManagerBackend` should use the explicit `project_id` provided in `BACKEND_KWARGS`, regardless of whether ADC returns a default project. The validation should only fail if no project_id is available from ANY source.
   
   
   ### How to reproduce
   
   
   1. Install `apache-airflow-providers-google==19.4.0`
   2. Configure secrets backend:
      ```bash
      export AIRFLOW__SECRETS__BACKEND=airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
      export AIRFLOW__SECRETS__BACKEND_KWARGS='{"project_id":"my-project-id"}'
      ```
   3. Set up ADC without a default project:
      ```bash
      gcloud auth application-default login
      gcloud config unset project
      ```
   4. Start Airflow
   5. Observe the error during secrets backend initialization
   
   ### Anything else
   
   `CloudSecretManagerBackend` should use the explicit `project_id` provided in `BACKEND_KWARGS`, regardless of whether ADC returns a default project. The validation should only fail if no project_id is available from ANY source.
   
   **Root Cause:**
   
   Commit [`0a3ee82ba3`](https://github.com/apache/airflow/commit/0a3ee82ba3005c8e20e82be56b2320b45a519f0e) (PR #60146) added validation in `_CredentialProvider._get_credentials_using_adc()` that raises an exception if `google.auth.default()` returns `None` for `project_id`.
   
   The problem is that this validation happens **before** `CloudSecretManagerBackend.__init__` can use the explicit `project_id` parameter:
   
   ```python
   # In CloudSecretManagerBackend.__init__ (lines 120-137)
   self.credentials, self.project_id = get_credentials_and_project_id(...)
   # ↑ This raises exception in 19.4.0 before reaching the code below ↓
   
   # This code is never reached:
   if project_id:
       self.project_id = project_id  # Would have used explicit project_id!
   ```
   
   **Why the error message is misleading:**
   
   The error suggests using `key_secret_project_id`, but this parameter is for a different purpose (specifying which project contains the **credentials secret**, not which project contains the **Airflow secrets**). It doesn't solve the problem for users providing `project_id` in `BACKEND_KWARGS`.
   
   **Comparison:**
   
   Version 19.3.0 (working):
   ```python
   def _get_credentials_using_adc(self) -> tuple[Credentials, str]:
       credentials, project_id = google.auth.default(scopes=self.scopes)
       return credentials, project_id  # No validation
   ```
   
   Version 19.4.0 (broken):
   ```python
   def _get_credentials_using_adc(self) -> tuple[Credentials, str]:
       scopes = list(self.scopes) if self.scopes else None
       credentials, project_id = google.auth.default(scopes=scopes)
       if not project_id:  # Fails before caller can use explicit project_id
           raise AirflowException(...)
       return credentials, project_id
   ```
   
   **Proposed Fix:**
   
   Move the validation to the caller level, after checking all possible sources of `project_id`:
   
   ```python
   # In CloudSecretManagerBackend.__init__
   self.credentials, adc_project_id = get_credentials_and_project_id(...)
   
   # Use explicit project_id if provided, otherwise use ADC project_id
   if project_id:
       self.project_id = project_id
   else:
       self.project_id = adc_project_id
   
   # NOW validate after checking all sources
   if not self.project_id:
       raise AirflowException(
           "Project ID could not be determined. "
           "Please provide 'project_id' in backend configuration or ensure "
           "your credentials include a default project."
       )
   ```
   
   **Workaround:**
   
   Set the `GOOGLE_CLOUD_PROJECT` environment variable:
   ```bash
   export GOOGLE_CLOUD_PROJECT=my-project-id
   ```
   
   This makes `google.auth.default()` return a project_id, satisfying the validation.
   
   **Impact:**
   
   This is a regression that breaks a valid and common use case:
   - Using ADC for authentication (Google's recommended approach)
   - Working with multiple GCP projects without a default
   - Explicitly configuring project_id in secrets backend settings
   
   Affects both Airflow 2.x and 3.x users who upgrade to google provider 19.4.0+.
   
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CloudSecretManagerBackend fails with ADC when explicit project_id provided (regression in 19.4.0) [airflow]

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on issue #61217:
URL: https://github.com/apache/airflow/issues/61217#issuecomment-3818916950

   Thank you for reporting, feel free to raise a PR!
   I'll be happy to review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CloudSecretManagerBackend fails with ADC when explicit project_id provided (regression in 19.4.0) [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #61217: CloudSecretManagerBackend fails with ADC when explicit project_id provided (regression in 19.4.0)
URL: https://github.com/apache/airflow/issues/61217


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] CloudSecretManagerBackend fails with ADC when explicit project_id provided (regression in 19.4.0) [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #61217:
URL: https://github.com/apache/airflow/issues/61217#issuecomment-3818842480

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org