You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "nailo2c (via GitHub)" <gi...@apache.org> on 2026/01/25 21:58:32 UTC
[PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]
nailo2c opened a new pull request, #61048:
URL: https://github.com/apache/airflow/pull/61048
Related: #11323
# How
Refactor [AzureBlobStorageToGCSOperator](https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/transfers/azure_blob_to_gcs.py#L38) and [AzureFileShareToGCSOperator](https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/transfers/azure_fileshare_to_gcs.py#L41) to return lists of GCS URIs instead of strings.
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR, please
change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
-->
- [ ] No (please specify the tool below)
<!--
Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]
Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on PR #61048:
URL: https://github.com/apache/airflow/pull/61048#issuecomment-3802093922
I was torn between `return_list` and `as_list`, and I went with `as_list` because I think it aligns better with the issue's intent and is more generic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #61048:
URL: https://github.com/apache/airflow/pull/61048#issuecomment-3801287450
Before approving and merging (as technically it looks good!) - I need some time to consider how we're going to handle it across the operators in terms of naming the "as_list" flag, as it should be consistent - and even maybe what should be the default behavior for single entities.
I'm certain that we had use cases like that in the past, probably in operators of different providers, so it needs some exploration (you're welcome to do it on your own as well).
It does make sense to me that it will return the list by default, and upon deprecation we'll use a flag to return the single entity.
If you have some insights about it, feel free to share.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]
Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on PR #61048:
URL: https://github.com/apache/airflow/pull/61048#issuecomment-3831876564
Thanks for the hint, the code looks much better now! I kept `return_gcs_uris` in `azure_fileshare_to_gcs.py` because the original return is not a single unwrapped string - it returns a list of filenames, so `unwrap_single` doesn't quite fit here.
I tweaked the code to avoid a breaking change while staying more aligned with the other operator. I think it'll also be easier to remove in the future if we want to :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]
Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on PR #61048:
URL: https://github.com/apache/airflow/pull/61048#issuecomment-3829531826
> If the parameter is not provided at all, raise the deprecation warning that this parameter will be set to default False in future release which will be a breaking change.
Hi @shahar1, great proposal, thanks for digging up the precedent. One implementation detail: if we define it as `unwrap_single=True`, we can't distinguish "omitted" from "explicitly passed True", so we can't emit the deprecation warning only when the parameter is not provided.
How about putting the deprecation warning under the `unwrap_single=True` block?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #61048:
URL: https://github.com/apache/airflow/pull/61048#issuecomment-3858690697
@nailo2c are you able to run the system tests for the modified operators and attach screenshots that they pass?
Following recent convesrations with Google provider team, it is necessary to ensure that we don't have issues there.
If you're not to do it on your own, we'll have to wait until Google team or someone else will do it (I might do it on my own eventually if no one else does, but it will probably be faster if someone who's familiar with Azure and GCS altogether does it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #61048:
URL: https://github.com/apache/airflow/pull/61048#issuecomment-3804061151
> I was torn between `return_list` and `as_list`, and I went with `as_list` because I think it aligns better with the issue's intent and is more generic.
I think that I'm good with this -
I want to check some past similar work and ensure that the rest of the transfer operators could be aligned with this naming.
Give me a couple of days to check, you could work in the meanwhile on other issues (or even other operators as part of the original issue).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #61048:
URL: https://github.com/apache/airflow/pull/61048#issuecomment-3827887805
> Sounds good! Let me know once you've finished the survey :)
OK, found it - we have [unwrap_single](https://github.com/apache/airflow/blob/c7a8e43ad919e19e263cb52bd1d2576cd64c2aab/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/job.py#L102) for `KubernetesJobOperator`, which is even better than `as_list` as the latter is meaningless for multiple results.
Let's do it as follows:
1. Rename the parameter to `unwrap_single` - default for now should be `True` to retain backward compatibility.
2. If the parameter is not provided at all, raise the deprecation warning that this parameter will be set to default `False` in future release. If it's explicitly provided (either `True` or `False`), it means that the user is acknowledged of this parameter and fixated its value, so no need to warn them.
I'll update the instructions in the original issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #61048:
URL: https://github.com/apache/airflow/pull/61048#discussion_r2725971181
##########
providers/google/src/airflow/providers/google/cloud/transfers/azure_blob_to_gcs.py:
##########
@@ -122,7 +122,7 @@ def execute(self, context: Context) -> str:
self.blob_name,
self.bucket_name,
)
- return f"gs://{self.bucket_name}/{self.object_name}"
+ return [f"gs://{self.bucket_name}/{self.object_name}"]
Review Comment:
See my comment in the original issue regarding breaking changes (relevant to all related PRs):
https://github.com/apache/airflow/issues/11323#issuecomment-3797323872
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]
Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on PR #61048:
URL: https://github.com/apache/airflow/pull/61048#issuecomment-3863076559
Hi @shahar1, sure :) but let me confirm the system tests are [example_azure_blob_to_gcs.py](providers/google/tests/system/google/cloud/azure/example_azure_blob_to_gcs.py) and [example_azure_fileshare_to_gcs.py](https://github.com/apache/airflow/blob/main/providers/google/tests/system/google/cloud/azure/example_azure_fileshare_to_gcs.py), right?
If so, here are the results:
# example_azure_blob_to_gcs.py
+ It works well on Airflow (the first two failures were due to my environment misconfigurations, not code issues).
<img width="2555" height="608" alt="example_azure_blob_to_gcs_1_airflow" src="https://github.com/user-attachments/assets/77c49902-6f0d-4fee-8b5c-2627434baba6" />
+ It reads the file from Azure Blob Storage and transfers it to GCS successfully.
<img width="2545" height="1144" alt="example_azure_blob_to_gcs_2_azure" src="https://github.com/user-attachments/assets/01b9d962-a11b-45db-b1b5-285baa1a1b57" />
<img width="2549" height="538" alt="example_azure_blob_to_gcs_3_gcp" src="https://github.com/user-attachments/assets/5f0e9593-079b-4942-a939-59303924b759" />
+ It's not a breaking change-- when `unwrap_single` is not set, it returns a string as before.
<img width="2013" height="399" alt="example_azure_blob_to_gcs_4_old_return_str" src="https://github.com/user-attachments/assets/e0da9b55-9400-4142-a2ca-bdd86346b8d8" />
+ After setting `unwrap_single=False`, it returns a list of URIs.
<img width="2017" height="426" alt="example_azure_blob_to_gcs_5_new_return_list" src="https://github.com/user-attachments/assets/dee6c181-2726-43c5-bb14-3471f4b5a100" />
# example_azure_fileshare_to_gcs.py
+ Same for `example_azure_fileshare_to_gcs.py`
<img width="2550" height="1351" alt="sync_azure_files_with_gcs" src="https://github.com/user-attachments/assets/64a1dbe2-c8ae-4a25-b5c8-490c5014ddcc" />
---
Please let me know if there's anything else I can help with :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 merged PR #61048:
URL: https://github.com/apache/airflow/pull/61048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]
Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on code in PR #61048:
URL: https://github.com/apache/airflow/pull/61048#discussion_r2726028982
##########
providers/google/src/airflow/providers/google/cloud/transfers/azure_blob_to_gcs.py:
##########
@@ -122,7 +122,7 @@ def execute(self, context: Context) -> str:
self.blob_name,
self.bucket_name,
)
- return f"gs://{self.bucket_name}/{self.object_name}"
+ return [f"gs://{self.bucket_name}/{self.object_name}"]
Review Comment:
Hi, thanks for the review! I've updated the PR to use the flag approach and added a deprecation warning. This should prevent any breaking changes :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]
Posted by "nailo2c (via GitHub)" <gi...@apache.org>.
nailo2c commented on PR #61048:
URL: https://github.com/apache/airflow/pull/61048#issuecomment-3806288337
Sounds good! Let me know once you've finished the survey :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #61048:
URL: https://github.com/apache/airflow/pull/61048#issuecomment-3830410709
> > If the parameter is not provided at all, raise the deprecation warning that this parameter will be set to default False in future release which will be a breaking change.
>
> Hi @shahar1, great proposal, thanks for digging up the precedent. One implementation detail: if we define it as `unwrap_single=True`, we can't distinguish "omitted" from "explicitly passed True", so we can't emit the deprecation warning only when the parameter is not provided.
>
> How about putting the deprecation warning under the `unwrap_single=True` block?
You could let it to be optional (`None`), and if it's `None` (i.e., user has no set it explicitly) - then set it to True + raise a FutureWarning (there's another PR that implemented it, feel free to take inspiration)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Return list of GCS URIs from Azure*ToGCS operators [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #61048:
URL: https://github.com/apache/airflow/pull/61048#issuecomment-3863738227
> Hi @shahar1, sure :) but let me confirm the system tests are [example_azure_blob_to_gcs.py](providers/google/tests/system/google/cloud/azure/example_azure_blob_to_gcs.py) and [example_azure_fileshare_to_gcs.py](https://github.com/apache/airflow/blob/main/providers/google/tests/system/google/cloud/azure/example_azure_fileshare_to_gcs.py), right?
>
> If so, here are the results:
>
> # example_azure_blob_to_gcs.py
> * It works well on Airflow (the first two failures were due to my environment misconfigurations, not code issues).
>
> <img alt="example_azure_blob_to_gcs_1_airflow" width="2000" height="608" src="https://private-user-images.githubusercontent.com/11390254/546464284-77c49902-6f0d-4fee-8b5c-2627434baba6.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzA0NDY1NDgsIm5iZiI6MTc3MDQ0NjI0OCwicGF0aCI6Ii8xMTM5MDI1NC81NDY0NjQyODQtNzdjNDk5MDItNmYwZC00ZmVlLThiNWMtMjYyNzQzNGJhYmE2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjA3VDA2MzcyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTIxOTI5MTdjYzg3N2E4Yzk3YWNhYzE5NzliY2ZhOWIzZjZjMTA3NDgwNTIwZDc2MDRkNzg1NDNiOWQ2OTA4MDkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.ZG_P3Xo1UFyC1-xX-vx7mVHcJ_CVlB_fVFSGH61PfcI">
> * It reads the file from Azure Blob Storage and transfers it to GCS successfully.
>
> <img alt="example_azure_blob_to_gcs_2_azure" width="2000" height="1144" src="https://private-user-images.githubusercontent.com/11390254/546464388-01b9d962-a11b-45db-b1b5-285baa1a1b57.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzA0NDY1NDgsIm5iZiI6MTc3MDQ0NjI0OCwicGF0aCI6Ii8xMTM5MDI1NC81NDY0NjQzODgtMDFiOWQ5NjItYTExYi00NWRiLWIxYjUtMjg1YmFhMWExYjU3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjA3VDA2MzcyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWM3MjVkY2EzNGQ3YmUyOTBiODA0OWI4ZTNlYjczNjkyY2YzNjg5ZDA1MDc0YTUzNDQ2Y2M0NDQzY2RiYzFkMjQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.J_KBm9GSEU2irTIckKAC4E0Fj-fr8N28T0wQ2ImGMyM"> <img alt="example_azure_blob_to_gcs_3_gcp" width="2000" height="538" src="https://private-user-images.githubusercontent.com/11390254/54
6464405-5f0e9593-079b-4942-a939-59303924b759.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzA0NDY1NDgsIm5iZiI6MTc3MDQ0NjI0OCwicGF0aCI6Ii8xMTM5MDI1NC81NDY0NjQ0MDUtNWYwZTk1OTMtMDc5Yi00OTQyLWE5MzktNTkzMDM5MjRiNzU5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjA3VDA2MzcyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWJkYTBmMjllYTk3Y2FhMzdhZWU2ZTlhZjIwNTBjNTc5NWViODk0MjY5NzQ4MTIwZDZkNTY3NGNjNTc1Yjk0OWQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.wYKcht0ifG1Df9Z9e_zxKLRfM-e5xliS6niURiO7Kmc">
> * It's not a breaking change-- when `unwrap_single` is not set, it returns a string as before.
>
> <img alt="example_azure_blob_to_gcs_4_old_return_str" width="2000" height="399" src="https://private-user-images.githubusercontent.com/11390254/546464592-e0da9b55-9400-4142-a2ca-bdd86346b8d8.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzA0NDY1NDgsIm5iZiI6MTc3MDQ0NjI0OCwicGF0aCI6Ii8xMTM5MDI1NC81NDY0NjQ1OTItZTBkYTliNTUtOTQwMC00MTQyLWEyY2EtYmRkODYzNDZiOGQ4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjA3VDA2MzcyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWY5YjQ5MDA5ZTEzNzM1ZTk0ZDEyZjg0ZjRhMjEzNWFkNzJiOGE2YmFlODc0M2M3MDJjNWFhYTQwODk0YWIyZTgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.yjoFV-VZWtxdXKBzeJ7BDsDdAnDkpQDNvkmYfkNqMPU">
> * After setting `unwrap_single=False`, it returns a list of URIs.
>
> <img alt="example_azure_blob_to_gcs_5_new_return_list" width="2000" height="426" src="https://private-user-images.githubusercontent.com/11390254/546464672-dee6c181-2726-43c5-bb14-3471f4b5a100.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzA0NDY1NDgsIm5iZiI6MTc3MDQ0NjI0OCwicGF0aCI6Ii8xMTM5MDI1NC81NDY0NjQ2NzItZGVlNmMxODEtMjcyNi00M2M1LWJiMTQtMzQ3MWY0YjVhMTAwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjA3VDA2MzcyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBlYmFjODA3OGU3NjgwMTVhOWY1ZjViMjdmNDQ4ZTZlMGM2MDRkNzY4NzVjMGYzOWI2ODMyNjJiZDc0ZDIwMGEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.lqBSisJXQtAAIWmRUUcszKScCc-NC-sRF0RMTS07kcg">
> # example_azure_fileshare_to_gcs.py
> * Same for `example_azure_fileshare_to_gcs.py`
>
> <img alt="sync_azure_files_with_gcs" width="2000" height="1351" src="https://private-user-images.githubusercontent.com/11390254/546464721-64a1dbe2-c8ae-4a25-b5c8-490c5014ddcc.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzA0NDY1NDgsIm5iZiI6MTc3MDQ0NjI0OCwicGF0aCI6Ii8xMTM5MDI1NC81NDY0NjQ3MjEtNjRhMWRiZTItYzhhZS00YTI1LWI1YzgtNDkwYzUwMTRkZGNjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAyMDclMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMjA3VDA2MzcyOFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWE2MDNlZTQ3ZWRiMTg4Mzg0ZDA3YzM3ODBkZDk5NzU4MGIxN2QwMjdmMGFlMzcwYjI0ZjAxYjEzM2RjMDY0MjQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.DR1jWCarCjBEHkcIfsReQmddZi9QqF2nwCnKm0ywJZ0">
> * Also, while running the system test, I noticed `dest_gcs` in `example_azure_fileshare_to_gcs.py` was missing the `gs://` prefix, which causes `_parse_gcs_url` to fail. I fixed it incidentally.
>
> Please let me know if there's anything else I can help with :)
Awesome! It seems that the system tests are indeed useful for detecting such "last minute" bugs.
Now that everything passes I think that we're all set for this PR to be merged.
If no objections are made, it will be merged by the upcoming release on Tuesday.
Feel free to work on other issues in the meanwhile. Also, if you're able to run system tests for other approved PRs related to the original issue, where we still don't have any - it would be very helpful.
CC: @VladaZakharova @MaksYermak
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org