You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Adaverse (via GitHub)" <gi...@apache.org> on 2023/07/01 20:13:11 UTC
[GitHub] [airflow] Adaverse opened a new pull request, #32306: Moves `AzureBlobStorageToGCSOperator` from Azure to Google provider
Adaverse opened a new pull request, #32306:
URL: https://github.com/apache/airflow/pull/32306
Closes: #32285
- [X] Moved `AzureBlobStorageToGCSOperator` to Google provider
- [X] Deprecated `AzureBlobStorageToGCSOperator` in Azure provider
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Adaverse commented on a diff in pull request #32306: Moves `AzureBlobStorageToGCSOperator` from Azure to Google provider
Posted by "Adaverse (via GitHub)" <gi...@apache.org>.
Adaverse commented on code in PR #32306:
URL: https://github.com/apache/airflow/pull/32306#discussion_r1249470566
##########
docs/apache-airflow-providers-google/operators/transfer/azure_blob_to_gcs.rst:
##########
Review Comment:
Yes! It works with a few extra dots. Added to the `redirects.txt`, but of course, it would only redirect to the latest version and has to be removed in later versions.
```
transfer/azure_blob_to_gcs.rst ../../apache-airflow-providers-google/latest/operators/transfer/azure_blob_to_gcs.rst
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on a diff in pull request #32306: Moves `AzureBlobStorageToGCSOperator` from Azure to Google provider
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on code in PR #32306:
URL: https://github.com/apache/airflow/pull/32306#discussion_r1256349822
##########
airflow/providers/microsoft/azure/transfers/azure_blob_to_gcs.py:
##########
@@ -17,102 +17,26 @@
# under the License.
from __future__ import annotations
-import tempfile
-from typing import TYPE_CHECKING, Sequence
+import warnings
-from airflow.models import BaseOperator
-from airflow.providers.google.cloud.hooks.gcs import GCSHook
-from airflow.providers.microsoft.azure.hooks.wasb import WasbHook
+from airflow.exceptions import AirflowProviderDeprecationWarning
+from airflow.providers.google.cloud.transfers.azure_blob_to_gcs import (
+ AzureBlobStorageToGCSOperator as AzureBlobStorageToGCSOperatorFromGoogleProvider,
+)
-if TYPE_CHECKING:
- from airflow.utils.context import Context
-
-class AzureBlobStorageToGCSOperator(BaseOperator):
+class AzureBlobStorageToGCSOperator(AzureBlobStorageToGCSOperatorFromGoogleProvider):
"""
- Operator transfers data from Azure Blob Storage to specified bucket in Google Cloud Storage.
-
- .. seealso::
- For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:AzureBlobStorageToGCSOperator`
-
- :param wasb_conn_id: Reference to the wasb connection.
- :param gcp_conn_id: The connection ID to use when fetching connection info.
- :param blob_name: Name of the blob
- :param container_name: Name of the container
- :param bucket_name: The bucket to upload to
- :param object_name: The object name to set when uploading the file
- :param filename: The local file path to the file to be uploaded
- :param gzip: Option to compress local file or file data for upload
- :param impersonation_chain: Optional service account to impersonate using short-term
- credentials, or chained list of accounts required to get the access_token
- of the last account in the list, which will be impersonated in the request.
- If set as a string, the account must grant the originating account
- the Service Account Token Creator IAM role.
- If set as a sequence, the identities from the list must grant
- Service Account Token Creator IAM role to the directly preceding identity, with first
- account from the list granting this role to the originating account.
+ This class is deprecated.
+ Please use `airflow.providers.google.cloud.transfers.azure_blob_to_gcs.AzureBlobStorageToGCSOperator`.
Review Comment:
I've been working on enabling rule D205 in the stylechecks which enforces a blank line after the summary, if you don't mind.
```suggestion
This class is deprecated.
Please use `airflow.providers.google.cloud.transfers.azure_blob_to_gcs.AzureBlobStorageToGCSOperator`.
```
##########
airflow/providers/microsoft/azure/transfers/azure_blob_to_gcs.py:
##########
@@ -17,102 +17,26 @@
# under the License.
from __future__ import annotations
-import tempfile
-from typing import TYPE_CHECKING, Sequence
+import warnings
-from airflow.models import BaseOperator
-from airflow.providers.google.cloud.hooks.gcs import GCSHook
-from airflow.providers.microsoft.azure.hooks.wasb import WasbHook
+from airflow.exceptions import AirflowProviderDeprecationWarning
+from airflow.providers.google.cloud.transfers.azure_blob_to_gcs import (
+ AzureBlobStorageToGCSOperator as AzureBlobStorageToGCSOperatorFromGoogleProvider,
+)
-if TYPE_CHECKING:
- from airflow.utils.context import Context
-
-class AzureBlobStorageToGCSOperator(BaseOperator):
+class AzureBlobStorageToGCSOperator(AzureBlobStorageToGCSOperatorFromGoogleProvider):
"""
- Operator transfers data from Azure Blob Storage to specified bucket in Google Cloud Storage.
-
- .. seealso::
- For more information on how to use this operator, take a look at the guide:
- :ref:`howto/operator:AzureBlobStorageToGCSOperator`
-
- :param wasb_conn_id: Reference to the wasb connection.
- :param gcp_conn_id: The connection ID to use when fetching connection info.
- :param blob_name: Name of the blob
- :param container_name: Name of the container
- :param bucket_name: The bucket to upload to
- :param object_name: The object name to set when uploading the file
- :param filename: The local file path to the file to be uploaded
- :param gzip: Option to compress local file or file data for upload
- :param impersonation_chain: Optional service account to impersonate using short-term
- credentials, or chained list of accounts required to get the access_token
- of the last account in the list, which will be impersonated in the request.
- If set as a string, the account must grant the originating account
- the Service Account Token Creator IAM role.
- If set as a sequence, the identities from the list must grant
- Service Account Token Creator IAM role to the directly preceding identity, with first
- account from the list granting this role to the originating account.
+ This class is deprecated.
+ Please use `airflow.providers.google.cloud.transfers.azure_blob_to_gcs.AzureBlobStorageToGCSOperator`.
"""
- def __init__(
- self,
- *,
- wasb_conn_id="wasb_default",
- gcp_conn_id: str = "google_cloud_default",
- blob_name: str,
- container_name: str,
- bucket_name: str,
- object_name: str,
- filename: str,
- gzip: bool,
- impersonation_chain: str | Sequence[str] | None = None,
- **kwargs,
- ) -> None:
- super().__init__(**kwargs)
- self.wasb_conn_id = wasb_conn_id
- self.gcp_conn_id = gcp_conn_id
- self.blob_name = blob_name
- self.container_name = container_name
- self.bucket_name = bucket_name
- self.object_name = object_name
- self.filename = filename
- self.gzip = gzip
- self.impersonation_chain = impersonation_chain
-
- template_fields: Sequence[str] = (
- "blob_name",
- "container_name",
- "bucket_name",
- "object_name",
- "filename",
- )
-
- def execute(self, context: Context) -> str:
- azure_hook = WasbHook(wasb_conn_id=self.wasb_conn_id)
- gcs_hook = GCSHook(
- gcp_conn_id=self.gcp_conn_id,
- impersonation_chain=self.impersonation_chain,
+ def __init__(self, *args, **kwargs):
+ warnings.warn(
+ """This class is deprecated.
+ Please use
Review Comment:
```suggestion
"""This class is deprecated.
Please use
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Adaverse commented on a diff in pull request #32306: Moves `AzureBlobStorageToGCSOperator` from Azure to Google provider
Posted by "Adaverse (via GitHub)" <gi...@apache.org>.
Adaverse commented on code in PR #32306:
URL: https://github.com/apache/airflow/pull/32306#discussion_r1249470566
##########
docs/apache-airflow-providers-google/operators/transfer/azure_blob_to_gcs.rst:
##########
Review Comment:
Yes! It works with a few extra dots. Add to the `redirects.txt`, but of course, it would only redirect to the latest version and has to be removed in later versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #32306: Moves `AzureBlobStorageToGCSOperator` from Azure to Google provider
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #32306:
URL: https://github.com/apache/airflow/pull/32306#discussion_r1249309906
##########
docs/apache-airflow-providers-google/operators/transfer/azure_blob_to_gcs.rst:
##########
Review Comment:
Can you check with `redirects.txt` if it's possible to add entry to redirect the doc between providers?
I'm not sure if this will work but worth trying.
The goal is that when users try to view this doc in newer provider version they will be forward to the google doc rather than getting 404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Adaverse commented on a diff in pull request #32306: Moves `AzureBlobStorageToGCSOperator` from Azure to Google provider
Posted by "Adaverse (via GitHub)" <gi...@apache.org>.
Adaverse commented on code in PR #32306:
URL: https://github.com/apache/airflow/pull/32306#discussion_r1249470566
##########
docs/apache-airflow-providers-google/operators/transfer/azure_blob_to_gcs.rst:
##########
Review Comment:
Yes! It works with a few extra dots. Added to the `redirects.txt`, but of course, it would only redirect to the latest version and has to be removed in later versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #32306: Moves `AzureBlobStorageToGCSOperator` from Azure to Google provider
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #32306:
URL: https://github.com/apache/airflow/pull/32306#discussion_r1249341607
##########
docs/apache-airflow-providers-google/operators/transfer/azure_blob_to_gcs.rst:
##########
Review Comment:
Theorethically it should work if you use ../apache-airlfow-providers-google/operators/...... in the amazon redirects.txt I think , but It might have some side effects especially when one provider package docs is being generated.
If it won't work - we can always manually generate the redirects by modifying the "post-docs.py" script:
https://github.com/apache/airflow-site/blob/main/post-docs/add-back-references.py
This could be our escape hatch for that one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal merged pull request #32306: Moves `AzureBlobStorageToGCSOperator` from Azure to Google provider
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #32306:
URL: https://github.com/apache/airflow/pull/32306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org