You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/17 08:10:58 UTC
[GitHub] [airflow] bhavaniravi opened a new pull request #20370: 20139 sagemaker
bhavaniravi opened a new pull request #20370:
URL: https://github.com/apache/airflow/pull/20370
Fixes part of #20139
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #20370: 20139 sagemaker
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #20370:
URL: https://github.com/apache/airflow/pull/20370#discussion_r771916110
##########
File path: dev/provider_packages/prepare_provider_packages.py
##########
@@ -2154,11 +2154,13 @@ def summarise_total_vs_bad_and_warnings(total: int, bad: int, warns: List[warnin
"This module is deprecated. Please use `airflow.providers.amazon.aws.sensors.dms`.",
'This module is deprecated. Please use `airflow.providers.amazon.aws.operators.emr`.',
'This module is deprecated. Please use `airflow.providers.amazon.aws.sensors.emr`.',
- "This module is deprecated. Please use `airflow.providers.amazon.aws.hooks.redshift_cluster` "
- "or `airflow.providers.amazon.aws.hooks.redshift_sql` as appropriate.",
- "This module is deprecated. Please use `airflow.providers.amazon.aws.operators.redshift_sql` "
- "or `airflow.providers.amazon.aws.operators.redshift_cluster` as appropriate.",
- "This module is deprecated. Please use `airflow.providers.amazon.aws.sensors.redshift_cluster`.",
+ 'This module is deprecated. Please use `airflow.providers.amazon.aws.operators.sagemaker`.',
+ 'This module is deprecated. Please use `airflow.providers.amazon.aws.sensors.sagemaker`.',
+ 'This module is deprecated. Please use `airflow.hooks.redshift_sql` '
+ 'or `airflow.hooks.redshift_cluster` as appropriate.',
+ 'This module is deprecated. Please use `airflow.operators.redshift_sql` or '
+ '`airflow.operators.redshift_cluster` as appropriate.',
+ 'This module is deprecated. Please use `airflow.sensors.redshift_cluster`.',
Review comment:
These changes should be removed as it's not part of this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #20370: 20139 sagemaker
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #20370:
URL: https://github.com/apache/airflow/pull/20370#discussion_r771640752
##########
File path: airflow/providers/amazon/aws/operators/sagemaker.py
##########
@@ -0,0 +1,643 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+import sys
+from typing import List, Optional
+
+from botocore.exceptions import ClientError
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+from airflow.providers.amazon.aws.hooks.sagemaker import SageMakerHook
+
+if sys.version_info >= (3, 8):
+ from functools import cached_property
+else:
+ from cached_property import cached_property
Review comment:
We are not changing the code of the operators in this PR. This is just about classes names/files.
If the operator code needs some work it should be done seperatly.
Specifically here - in airflow code base we do this import from compact but in providers we cant because we dont want to limit the provider only to new airflow versions. Eventually this import will be replaced with:
`from airflow.compat.functools import cached_property`
For the moment we are OK here :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #20370: 20139 sagemaker
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #20370:
URL: https://github.com/apache/airflow/pull/20370#discussion_r771640752
##########
File path: airflow/providers/amazon/aws/operators/sagemaker.py
##########
@@ -0,0 +1,643 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+import sys
+from typing import List, Optional
+
+from botocore.exceptions import ClientError
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+from airflow.providers.amazon.aws.hooks.sagemaker import SageMakerHook
+
+if sys.version_info >= (3, 8):
+ from functools import cached_property
+else:
+ from cached_property import cached_property
Review comment:
We are not changing the code of the operators in this PR. This is just about classes names/files.
If the operator code needs some work it should be done seperatly.
Specifically here - in airflow code base we do this import from compat but in providers we cant because we dont want to limit the provider only to new airflow versions. Eventually this import will be replaced with:
`from airflow.compat.functools import cached_property`
For the moment we are OK here :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on a change in pull request #20370: 20139 sagemaker
Posted by GitBox <gi...@apache.org>.
ferruzzi commented on a change in pull request #20370:
URL: https://github.com/apache/airflow/pull/20370#discussion_r771599403
##########
File path: airflow/providers/amazon/aws/operators/sagemaker.py
##########
@@ -0,0 +1,643 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import json
+import sys
+from typing import List, Optional
+
+from botocore.exceptions import ClientError
+
+from airflow.exceptions import AirflowException
+from airflow.models import BaseOperator
+from airflow.providers.amazon.aws.hooks.base_aws import AwsBaseHook
+from airflow.providers.amazon.aws.hooks.sagemaker import SageMakerHook
+
+if sys.version_info >= (3, 8):
+ from functools import cached_property
+else:
+ from cached_property import cached_property
Review comment:
I have seen this also done as a try/except block elsewhere in the Airflow codebase. Do we have a preference on one form over another?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal merged pull request #20370: Organize Sagemaker classes in Amazon provider
Posted by GitBox <gi...@apache.org>.
eladkal merged pull request #20370:
URL: https://github.com/apache/airflow/pull/20370
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #20370: Organize Sagemaker classes in Amazon provider
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #20370:
URL: https://github.com/apache/airflow/pull/20370#discussion_r771916110
##########
File path: dev/provider_packages/prepare_provider_packages.py
##########
@@ -2154,11 +2154,13 @@ def summarise_total_vs_bad_and_warnings(total: int, bad: int, warns: List[warnin
"This module is deprecated. Please use `airflow.providers.amazon.aws.sensors.dms`.",
'This module is deprecated. Please use `airflow.providers.amazon.aws.operators.emr`.',
'This module is deprecated. Please use `airflow.providers.amazon.aws.sensors.emr`.',
- "This module is deprecated. Please use `airflow.providers.amazon.aws.hooks.redshift_cluster` "
- "or `airflow.providers.amazon.aws.hooks.redshift_sql` as appropriate.",
- "This module is deprecated. Please use `airflow.providers.amazon.aws.operators.redshift_sql` "
- "or `airflow.providers.amazon.aws.operators.redshift_cluster` as appropriate.",
- "This module is deprecated. Please use `airflow.providers.amazon.aws.sensors.redshift_cluster`.",
+ 'This module is deprecated. Please use `airflow.providers.amazon.aws.operators.sagemaker`.',
+ 'This module is deprecated. Please use `airflow.providers.amazon.aws.sensors.sagemaker`.',
+ 'This module is deprecated. Please use `airflow.hooks.redshift_sql` '
+ 'or `airflow.hooks.redshift_cluster` as appropriate.',
+ 'This module is deprecated. Please use `airflow.operators.redshift_sql` or '
+ '`airflow.operators.redshift_cluster` as appropriate.',
+ 'This module is deprecated. Please use `airflow.sensors.redshift_cluster`.',
Review comment:
The test is failing related to these changes. Probably some wrong entry edit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #20370: Organize Sagemaker classes in Amazon provider
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #20370:
URL: https://github.com/apache/airflow/pull/20370#issuecomment-998736090
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org