You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/02 12:58:25 UTC

[GitHub] [airflow] akerone opened a new issue #21940: Google Ads Hook broken for Composer 2 (and presumably others)

akerone opened a new issue #21940:
URL: https://github.com/apache/airflow/issues/21940


   ### Apache Airflow Provider(s)
   
   google
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-google==5.1.0
   
   ### Apache Airflow version
   
   2.1.4
   
   ### Operating System
   
   Google Cloud Composer (GKE Autopilot, composer-2.0.1-airflow-2.1.4)
   
   ### Deployment
   
   Composer
   
   ### Deployment details
   
   Using Composer 2 which uses Workload Identity as the way to interact with GCP services, rather than the traditional Service Account Key in the connection extras.
   
   ### What happened
   
   ## The Google Ads Hook is broken for Composer 2
   
   # Details
   Trying to use the airflow.providers.google.ads.hooks.ads.GoogleAdsHook on Composer 2 raises:
   `KeyError: 'extra__google_cloud_platform__keyfile_dict'`
   
   This is due to Composer 2 using Workload Identity rather than a Service Account Key in the connection extras. Creating a key is *not* an option for us, since it creates an avoidable security risk on a highly privileged SA.
   
   The `"path_to_private_key_file"` key is also completely unnecessary in the config YAML, and we've successfully called the REST API without it.
   
   ### What you expected to happen
   
   We expect the hook to work in Composer 2.
   
   As Composer 2 is being rolled out, this will happen to more and more users.
   
   A possible solution would make the path_to_private_key_file key optional, so the hook doesn't have to retrieve it if it's not necessary.
   
   ### How to reproduce
   
   1. Create a Composer 2 environment on GCP
   2. Create a connection to Google Ads
   3. Create a DAG with a task using the GoogleAdsHook that uses the default GCP connection provided by Composer
   4. Run it the dag
   
   ### Anything else
   
   We lost the logs so I can't provide a full stack trace, but this happens on line 206 of [airflow/providers/google/ads/hooks/ads.py](https://github.com/apache/airflow/blob/2c57ad4ff9ddde8102c62f2e25c2a2e82cceb3e7/airflow/providers/google/ads/hooks/ads.py) 
   
   There's *nothing* in the extras section of the GCP default connection in Composer 2, so  secret_conn.extra_dejson["extra__google_cloud_platform__keyfile_dict"] will always fail.
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #21940: Google Ads Hook broken for Composer 2 (and presumably others)

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21940:
URL: https://github.com/apache/airflow/issues/21940#issuecomment-1057153707


   Maybe you can provide a fix for that @akerone ? Otherwise you will have to wait if there is someone who would like to fix it (Airflow has many contributors and marking it as goo first issue might help, but if you need it - making PR and becoing one of the almost 2000 contributors it is the fastest way to fix it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #21940: Google Ads Hook broken for Composer 2 (and presumably others)

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21940:
URL: https://github.com/apache/airflow/issues/21940#issuecomment-1060074720


   Composer is not relevant for the community release. I think both cases should be supported via different keys.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] boring-cyborg[bot] commented on issue #21940: Google Ads Hook broken for Composer 2 (and presumably others)

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #21940:
URL: https://github.com/apache/airflow/issues/21940#issuecomment-1056901858


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #21940: Google Ads Hook broken for Composer 2 (and presumably others)

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21940:
URL: https://github.com/apache/airflow/issues/21940#issuecomment-1060074986


   It could be auto-detected - if one fails, you can fall-back to the other.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] TreyYi commented on issue #21940: Google Ads Hook broken for Composer 2 (and presumably others)

Posted by GitBox <gi...@apache.org>.
TreyYi commented on issue #21940:
URL: https://github.com/apache/airflow/issues/21940#issuecomment-1058497488


   I understand that "path_to_private_key_file" is not required since Cloud Composer 2 is implemented to use Workload Identity which is pre-configured by GCP, but still there could be a chance of using it in Cloud Composer 1 with Standard mode.
   
   @potiuk @akerone , would you mind provide some ideas of how to implement ads.py? 
   I was trying to add if statement like below but "extra__google_cloud_platform__keyfile_dict" is needed for Cloud Composer 1 with Standard mode.
   ```
       def _update_config_with_secret(self, secrets_temp: IO[str]) -> None:
           """
           Gets Google Cloud secret from connection and saves the contents to the temp file
           Updates google ads config with file path of the temp file containing the secret.
           Note, the secret is optional in extra since Cloud Composer 2 does not require path_to_private_key_file.
           (Workload Identity is used instead of Service Account)
           """
           secret_conn = self.get_connection(self.gcp_conn_id)
           
           if "extra__google_cloud_platform__keyfile_dict" in secret_conn.extra_dejson:
               secret = secret_conn.extra_dejson["extra__google_cloud_platform__keyfile_dict"]
               secrets_temp.write(secret)
               secrets_temp.flush()
   
               self.google_ads_config["path_to_private_key_file"] = secrets_temp.name
   ```
   
   Also, I am wondering how to distinguish whether GCP users use Composer 1 or 2..


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] TreyYi edited a comment on issue #21940: Google Ads Hook broken for Composer 2 (and presumably others)

Posted by GitBox <gi...@apache.org>.
TreyYi edited a comment on issue #21940:
URL: https://github.com/apache/airflow/issues/21940#issuecomment-1058497488


   I understand that "path_to_private_key_file" is not required since Cloud Composer 2 is implemented to use Workload Identity which is pre-configured by GCP, but still there could be a chance of using it in Cloud Composer 1 with Standard mode.
   
   @potiuk @kaxil @akerone , would you mind provide some ideas of how to implement ads.py? 
   I was trying to add if statement like below but "extra__google_cloud_platform__keyfile_dict" is needed for Cloud Composer 1 with Standard mode.
   ```
       def _update_config_with_secret(self, secrets_temp: IO[str]) -> None:
           """
           Gets Google Cloud secret from connection and saves the contents to the temp file
           Updates google ads config with file path of the temp file containing the secret.
           Note, the secret is optional in extra since Cloud Composer 2 does not require path_to_private_key_file.
           (Workload Identity is used instead of Service Account)
           """
           secret_conn = self.get_connection(self.gcp_conn_id)
           
           if "extra__google_cloud_platform__keyfile_dict" in secret_conn.extra_dejson:
               secret = secret_conn.extra_dejson["extra__google_cloud_platform__keyfile_dict"]
               secrets_temp.write(secret)
               secrets_temp.flush()
   
               self.google_ads_config["path_to_private_key_file"] = secrets_temp.name
   ```
   
   Also, I am wondering how to distinguish whether GCP users use Composer 1 or 2..


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org