You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "shahar1 (via GitHub)" <gi...@apache.org> on 2023/04/19 17:07:54 UTC

[GitHub] [airflow] shahar1 opened a new pull request, #30748: Remove deprecated "delegate_to" from GCP operators and hooks

shahar1 opened a new pull request, #30748:
URL: https://github.com/apache/airflow/pull/30748

   related: #9461, #29088
   
   ---
   
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #30748: Remove deprecated "delegate_to" from GCP operators and hooks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #30748:
URL: https://github.com/apache/airflow/pull/30748#discussion_r1172254804


##########
airflow/providers/google/cloud/hooks/gcs.py:
##########
@@ -152,16 +151,10 @@ class GCSHook(GoogleBaseHook):
     def __init__(
         self,
         gcp_conn_id: str = "google_cloud_default",
-        delegate_to: str | None = None,

Review Comment:
   I think we cannot **just** remove the parameter here, because this will break an important property of the providers we have that they can be upgraded independently.
   
   While I see why removing such deprecated parameter is needed at some point in time doing it this way has rather profound consequences.
   
   For example if somene would like to keep old AWS provider but would like to upgrade a google provider to the new version, this will break silently for AWS provider (which will not be upgraded) - even if delegate_to parameter has NOT been used.
   
   The problem is that old providers are already passing this parameter even if it has not been used.  It will default to None and None will passed down.
   
   So for exmple if you have old AWS provider and new Google provider,  and you do not have `delegate_to`  parameter in your operator, the `gcs_to_s3` provider will call the  GCSHook with "delegate_to" parameter set to `None` and this will crash. 
   
   This is far too invasive. Yes. It's ok to remove that feature, but it's not ok to break another provider's usage of this hook even if the feature has not been used.
   
   Proposal:
   
   I am ok with removing the delegate to parameter, but this change should be done differently.
   
   * In all Google hooks using `delegate_to`, remove the `delegate_to` parameter but add `**kwargs` . This will stop the Hooks from crashing even if delegate_to parameter has been passed to it from an older version of Amazon or other providers.
   
   * In those hooks convert the warning into error. Smth like:
   
   ```
   if kwargs.get('delegate_to') is not None:
        raise RuntimeError("The `delegate_to` parameter has been deprecated before and finally removed in this version of Google Provider. You MUST convert it to `impersonate_chain`).
   ```
   
   I think this should be left here pretty much indefinitely.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] shahar1 commented on a diff in pull request #30748: Remove deprecated "delegate_to" from GCP operators and hooks

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #30748:
URL: https://github.com/apache/airflow/pull/30748#discussion_r1172708442


##########
airflow/providers/google/cloud/hooks/gcs.py:
##########
@@ -152,16 +151,10 @@ class GCSHook(GoogleBaseHook):
     def __init__(
         self,
         gcp_conn_id: str = "google_cloud_default",
-        delegate_to: str | None = None,

Review Comment:
   Done, including tests for the `if` statements in each GCP hook



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] shahar1 commented on a diff in pull request #30748: Remove deprecated "delegate_to" from GCP operators and hooks

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #30748:
URL: https://github.com/apache/airflow/pull/30748#discussion_r1172708943


##########
airflow/providers/amazon/CHANGELOG.rst:
##########
@@ -24,6 +24,15 @@
 Changelog
 ---------
 
+8.0.0
+......
+
+Breaking changes
+~~~~~~~~~~~~~~~~
+
+* ``Remove deprecated 'delegate_to' param from transfers that interact with Google Cloud (#30748)``

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] shahar1 commented on pull request #30748: Remove deprecated "delegate_to" from GCP operators and hooks

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #30748:
URL: https://github.com/apache/airflow/pull/30748#issuecomment-1515796614

   > Need also the following:
   > 
   > 1. Add to all relevant provider.yaml with new breaking change version (Except google as there it's already set)
   > 2. Add entry in changelog of each provider that explains the breaking change for the specific provider. This is simply an entry of "Removed delegate_to parameter from operators x, y, z"
   
   Done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk merged pull request #30748: Remove deprecated "delegate_to" from GCP operators and hooks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #30748:
URL: https://github.com/apache/airflow/pull/30748


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] shahar1 commented on a diff in pull request #30748: Remove deprecated "delegate_to" from GCP operators and hooks

Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #30748:
URL: https://github.com/apache/airflow/pull/30748#discussion_r1172708442


##########
airflow/providers/google/cloud/hooks/gcs.py:
##########
@@ -152,16 +151,10 @@ class GCSHook(GoogleBaseHook):
     def __init__(
         self,
         gcp_conn_id: str = "google_cloud_default",
-        delegate_to: str | None = None,

Review Comment:
   Done, including tests for the `if` statements in all GCP hooks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #30748: Remove deprecated "delegate_to" from GCP operators and hooks

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #30748:
URL: https://github.com/apache/airflow/pull/30748#discussion_r1172254804


##########
airflow/providers/google/cloud/hooks/gcs.py:
##########
@@ -152,16 +151,10 @@ class GCSHook(GoogleBaseHook):
     def __init__(
         self,
         gcp_conn_id: str = "google_cloud_default",
-        delegate_to: str | None = None,

Review Comment:
   I think we cannot **just** remove the parameter here, because this will break an important property of the providers we have that they can be upgraded independently.
   
   While I see why removing such deprecated parameter is needed at some point in time doing it this way has rather profound consequences.
   
   For example if somene would like to keep old AWS provider but would like to upgrade a google provider to the new version, this will break silently for AWS provider (which will not be upgraded) - even if delegate_to parameter has NOT been used.
   
   The problem is that old providers are already passing this parameter even if it has not been used.  It will default to None and None will passed down.
   
   So for exmple if you have old AWS provider and new Google provider,  and you do not have `delegate_to`  parameter in your operator, the `gcs_to_s3` provider will call the  GCSHook with "delegate_to" parameter set to `None` and this will crash. 
   
   This is far too invasive. Yes. It's ok to remove that feature, but it's not ok to break another provider's usage of this hook even if the feature has not been used.
   
   Proposal:
   
   I am ok with removing the delegate to parameter, but this change should be done differently.
   
   * In all Google hooks using `delegate_to`, remove the `delegate_to` parameter but add `**kwargs` . This will stop the Hooks from crashing even if `delegate_to=None` parameter has been passed to it from an older version of Amazon or other providers.
   
   * In those hooks convert the warning into error. Smth like:
   
   ```
   if kwargs.get('delegate_to') is not None:
        raise RuntimeError("The `delegate_to` parameter has been deprecated before and finally removed in this version of Google Provider. You MUST convert it to `impersonate_chain`).
   ```
   
   I think this should be left here pretty much indefinitely.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] HammadASiddiqui commented on pull request #30748: Remove deprecated "delegate_to" from GCP operators and hooks

Posted by "HammadASiddiqui (via GitHub)" <gi...@apache.org>.
HammadASiddiqui commented on PR #30748:
URL: https://github.com/apache/airflow/pull/30748#issuecomment-1702991561

   Hi!
   Can someone please guide me on this discussion
   
   https://github.com/googleapis/google-api-python-client/discussions/2228


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on a diff in pull request #30748: Remove deprecated "delegate_to" from GCP operators and hooks

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #30748:
URL: https://github.com/apache/airflow/pull/30748#discussion_r1172192621


##########
airflow/providers/amazon/CHANGELOG.rst:
##########
@@ -24,6 +24,15 @@
 Changelog
 ---------
 
+8.0.0
+......
+
+Breaking changes
+~~~~~~~~~~~~~~~~
+
+* ``Remove deprecated 'delegate_to' param from transfers that interact with Google Cloud (#30748)``

Review Comment:
   This line will be added automatically by the release manager during release.
   What we want to describe via the PR is information about that the commit message itself does not cover.
   For example: what operators involve? what action users needs to take.. basically additional information that may help user to bridge the breaking changes faster



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org