You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/08 15:53:15 UTC
[GitHub] [airflow] vincbeck opened a new pull request, #25609: Remove deprecated modules from Amazon provider package
vincbeck opened a new pull request, #25609:
URL: https://github.com/apache/airflow/pull/25609
Follow up of #25543
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
Taragolis commented on PR #25609:
URL: https://github.com/apache/airflow/pull/25609#issuecomment-1208635776
Selected tests not relevant anymore and could be removed
**test_get_client_type_returns_a_boto3_client_of_the_requested_type**
https://github.com/apache/airflow/blob/72a6ac5e54b6ab682861537e37abcd97f405d911/tests/providers/amazon/aws/hooks/test_base_aws.py#L223-L232
**test_get_client_type_overwrite**
https://github.com/apache/airflow/blob/72a6ac5e54b6ab682861537e37abcd97f405d911/tests/providers/amazon/aws/hooks/test_base_aws.py#L245-L253
**test_get_client_type_deprecation_warning**
https://github.com/apache/airflow/blob/72a6ac5e54b6ab682861537e37abcd97f405d911/tests/providers/amazon/aws/hooks/test_base_aws.py#L255-L262
**test_get_resource_type_returns_a_boto3_resource_of_the_requested_type**
https://github.com/apache/airflow/blob/72a6ac5e54b6ab682861537e37abcd97f405d911/tests/providers/amazon/aws/hooks/test_base_aws.py#L264-L282
**test_get_resource_type_overwrite**
https://github.com/apache/airflow/blob/72a6ac5e54b6ab682861537e37abcd97f405d911/tests/providers/amazon/aws/hooks/test_base_aws.py#L304-L322
**test_get_resource_deprecation_warning**
https://github.com/apache/airflow/blob/72a6ac5e54b6ab682861537e37abcd97f405d911/tests/providers/amazon/aws/hooks/test_base_aws.py#L324-L331
---
**test_refreshable_credentials** uses removed attribute
Quick hotfix:
1. In hook specify `client_type='sts'`
https://github.com/apache/airflow/blob/72a6ac5e54b6ab682861537e37abcd97f405d911/tests/providers/amazon/aws/hooks/test_base_aws.py#L623-L625
2. `hook.get_client_type('sts')` -> `hook.get_client_type()`
https://github.com/apache/airflow/blob/72a6ac5e54b6ab682861537e37abcd97f405d911/tests/providers/amazon/aws/hooks/test_base_aws.py#L662-L664
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
vincbeck commented on PR #25609:
URL: https://github.com/apache/airflow/pull/25609#issuecomment-1209777748
YAY!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25609:
URL: https://github.com/apache/airflow/pull/25609#issuecomment-1209800641
🎉 🎉 🎉 🎉 🎉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on a diff in pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
vincbeck commented on code in PR #25609:
URL: https://github.com/apache/airflow/pull/25609#discussion_r941439870
##########
airflow/providers/amazon/aws/secrets/systems_manager.py:
##########
@@ -100,24 +98,6 @@ def get_conn_value(self, conn_id: str) -> Optional[str]:
return self._get_secret(self.connections_prefix, conn_id)
- def get_conn_uri(self, conn_id: str) -> Optional[str]:
- """
- Return URI representation of Connection conn_id.
-
- As of Airflow version 2.3.0 this method is deprecated.
-
- :param conn_id: the connection id
- :return: deserialized Connection
- """
- if get_airflow_version() >= (2, 3):
- warnings.warn(
- f"Method `{self.__class__.__name__}.get_conn_uri` is deprecated and will be removed "
- "in a future release. Please use method `get_conn_value` instead.",
- DeprecationWarning,
- stacklevel=2,
- )
- return self.get_conn_value(conn_id)
-
Review Comment:
Keeping it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25609:
URL: https://github.com/apache/airflow/pull/25609#issuecomment-1209747710
Just one - likely an intermittent error. Re-run it @vincbeck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #25609:
URL: https://github.com/apache/airflow/pull/25609#discussion_r941209559
##########
airflow/providers/amazon/aws/operators/aws_lambda.py:
##########
@@ -1,28 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-"""This module is deprecated. Please use :mod:`airflow.providers.amazon.aws.operators.lambda_function`."""
-
-import warnings
-
-from airflow.providers.amazon.aws.operators.lambda_function import AwsLambdaInvokeFunctionOperator # noqa
-
-warnings.warn(
- "This module is deprecated. Please use `airflow.providers.amazon.aws.operators.lambda_function`.",
Review Comment:
Same comment as https://github.com/apache/airflow/pull/25543#discussion_r938497233
are you sure you want to remove this module?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25609:
URL: https://github.com/apache/airflow/pull/25609#issuecomment-1209267915
I prepared the #25618 - but I can still include that one if we fix the issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #25609:
URL: https://github.com/apache/airflow/pull/25609
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on a diff in pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
vincbeck commented on code in PR #25609:
URL: https://github.com/apache/airflow/pull/25609#discussion_r941441132
##########
airflow/providers/amazon/aws/secrets/secrets_manager.py:
##########
@@ -367,24 +366,6 @@ def get_conn_value(self, conn_id: str) -> Optional[str]:
return connection
- def get_conn_uri(self, conn_id: str) -> Optional[str]:
- """
- Return URI representation of Connection conn_id.
-
- As of Airflow version 2.3.0 this method is deprecated.
-
- :param conn_id: the connection id
- :return: deserialized Connection
- """
- if get_airflow_version() >= (2, 3):
- warnings.warn(
- f"Method `{self.__class__.__name__}.get_conn_uri` is deprecated and will be removed "
- "in a future release. Please use method `get_conn_value` instead.",
- DeprecationWarning,
- stacklevel=2,
- )
- return self.get_conn_value(conn_id)
-
Review Comment:
Keeping it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25609:
URL: https://github.com/apache/airflow/pull/25609#issuecomment-1208575601
Yep. Waiting for the tests to complete :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #25609:
URL: https://github.com/apache/airflow/pull/25609#issuecomment-1209241875
CI is not happy :(
some failures related to `TestSsmSecrets`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
vincbeck commented on PR #25609:
URL: https://github.com/apache/airflow/pull/25609#issuecomment-1209701923
Some tests were canceled. Any idea @potiuk ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #25609:
URL: https://github.com/apache/airflow/pull/25609#discussion_r941227062
##########
airflow/providers/amazon/aws/secrets/secrets_manager.py:
##########
@@ -367,24 +366,6 @@ def get_conn_value(self, conn_id: str) -> Optional[str]:
return connection
- def get_conn_uri(self, conn_id: str) -> Optional[str]:
- """
- Return URI representation of Connection conn_id.
-
- As of Airflow version 2.3.0 this method is deprecated.
-
- :param conn_id: the connection id
- :return: deserialized Connection
- """
- if get_airflow_version() >= (2, 3):
- warnings.warn(
- f"Method `{self.__class__.__name__}.get_conn_uri` is deprecated and will be removed "
- "in a future release. Please use method `get_conn_value` instead.",
- DeprecationWarning,
- stacklevel=2,
- )
- return self.get_conn_value(conn_id)
-
Review Comment:
Same as for SSM Secret Backend
##########
airflow/providers/amazon/aws/secrets/systems_manager.py:
##########
@@ -100,24 +98,6 @@ def get_conn_value(self, conn_id: str) -> Optional[str]:
return self._get_secret(self.connections_prefix, conn_id)
- def get_conn_uri(self, conn_id: str) -> Optional[str]:
- """
- Return URI representation of Connection conn_id.
-
- As of Airflow version 2.3.0 this method is deprecated.
-
- :param conn_id: the connection id
- :return: deserialized Connection
- """
- if get_airflow_version() >= (2, 3):
- warnings.warn(
- f"Method `{self.__class__.__name__}.get_conn_uri` is deprecated and will be removed "
- "in a future release. Please use method `get_conn_value` instead.",
- DeprecationWarning,
- stacklevel=2,
- )
- return self.get_conn_value(conn_id)
-
Review Comment:
This is the reason why SSM Parameters Sectet Backend Task failed.
I don't think we can remove this method right now, otherwise it would use `BaseSecretsBackend.get_conn_uri` which raise NotImplementedError
Also Provider might use in Airflow 2.2 where this method not deprecated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25609:
URL: https://github.com/apache/airflow/pull/25609#issuecomment-1209544262
:crossed_fingers: @vincbeck - I think this is the last one I want to include in this wave of providers :) (I will make one more pass to see)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25609:
URL: https://github.com/apache/airflow/pull/25609#issuecomment-1208607994
stilll a number of tests failing :( @vincbeck
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
vincbeck commented on PR #25609:
URL: https://github.com/apache/airflow/pull/25609#issuecomment-1208645699
Awesome! Thank you so much @Taragolis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #25609:
URL: https://github.com/apache/airflow/pull/25609#discussion_r941233063
##########
airflow/providers/amazon/aws/transfers/s3_to_redshift.py:
##########
@@ -85,17 +84,6 @@ def __init__(
upsert_keys: Optional[List[str]] = None,
**kwargs,
) -> None:
-
- if 'truncate_table' in kwargs:
- warnings.warn(
- """`truncate_table` is deprecated. Please use `REPLACE` method.""",
- DeprecationWarning,
- stacklevel=2,
- )
- if kwargs['truncate_table']:
- method = 'REPLACE'
- kwargs.pop('truncate_table', None)
Review Comment:
**test_deprecated_truncate** - should be removed, since `truncate_table` not used anymore.
https://github.com/apache/airflow/blob/741c20770230c83a95f74fe7ad7cc9f95329f2cc/tests/providers/amazon/aws/transfers/test_s3_to_redshift.py#L125-L175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on a diff in pull request #25609: Remove deprecated modules from Amazon provider package
Posted by GitBox <gi...@apache.org>.
vincbeck commented on code in PR #25609:
URL: https://github.com/apache/airflow/pull/25609#discussion_r941434819
##########
airflow/providers/amazon/aws/operators/aws_lambda.py:
##########
@@ -1,28 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-"""This module is deprecated. Please use :mod:`airflow.providers.amazon.aws.operators.lambda_function`."""
-
-import warnings
-
-from airflow.providers.amazon.aws.operators.lambda_function import AwsLambdaInvokeFunctionOperator # noqa
-
-warnings.warn(
- "This module is deprecated. Please use `airflow.providers.amazon.aws.operators.lambda_function`.",
Review Comment:
Good point. Keeping it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org