You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/09/15 12:54:16 UTC
[GitHub] [airflow] ephraimbuddy opened a new pull request #10951: Add example dag and system test for S3ToGCSOperator
ephraimbuddy opened a new pull request #10951:
URL: https://github.com/apache/airflow/pull/10951
This PR closes one of the issues in #8280
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10951: Add example dag and system test for S3ToGCSOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #10951:
URL: https://github.com/apache/airflow/pull/10951#issuecomment-692862828
Some static checks :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #10951: Add example dag and system test for S3ToGCSOperator
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #10951:
URL: https://github.com/apache/airflow/pull/10951#discussion_r488935277
##########
File path: docs/operators-and-hooks-ref.rst
##########
@@ -934,11 +934,15 @@ These integrations allow you to copy data from/to Google Cloud.
- :doc:`How to use <howto/operator/amazon/aws/google_api_to_s3_transfer>`
- :mod:`airflow.providers.amazon.aws.transfers.google_api_to_s3`
- * - `Amazon Simple Storage Service (S3) <https://aws.amazon.com/s3/>`__
+ * - Any Data Storage service
Review comment:
https://github.com/apache/airflow/pull/10951/files#diff-471d3adab9e55d525a899f46345f3b2fL880-R884
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10951: Add example dag and system test for S3ToGCSOperator
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #10951:
URL: https://github.com/apache/airflow/pull/10951#issuecomment-693554784
Cool :) 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #10951: Add example dag and system test for S3ToGCSOperator
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #10951:
URL: https://github.com/apache/airflow/pull/10951#issuecomment-692888108
> Some static checks :)
Yeah. Addressing it now :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #10951: Add example dag and system test for S3ToGCSOperator
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #10951:
URL: https://github.com/apache/airflow/pull/10951#discussion_r488933951
##########
File path: docs/operators-and-hooks-ref.rst
##########
@@ -934,11 +934,15 @@ These integrations allow you to copy data from/to Google Cloud.
- :doc:`How to use <howto/operator/amazon/aws/google_api_to_s3_transfer>`
- :mod:`airflow.providers.amazon.aws.transfers.google_api_to_s3`
- * - `Amazon Simple Storage Service (S3) <https://aws.amazon.com/s3/>`__
+ * - Any Data Storage service
Review comment:
I have removed this entirely. It's a duplicate. It's covered in google cloud operators as Storage Transfer Service and in Cloud Transfer Operators as s3 to gcs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #10951: Add example dag and system test for S3ToGCSOperator
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #10951:
URL: https://github.com/apache/airflow/pull/10951#discussion_r488910813
##########
File path: docs/operators-and-hooks-ref.rst
##########
@@ -934,11 +934,15 @@ These integrations allow you to copy data from/to Google Cloud.
- :doc:`How to use <howto/operator/amazon/aws/google_api_to_s3_transfer>`
- :mod:`airflow.providers.amazon.aws.transfers.google_api_to_s3`
- * - `Amazon Simple Storage Service (S3) <https://aws.amazon.com/s3/>`__
+ * - Any Data Storage service
Review comment:
I'm not sure the best way to word this but I see that Google Transfer service is not only for the S3. https://cloud.google.com/storage-transfer-service.
Also, having it as S3 to GCS transfer will overshadow the S3ToGCSOperator. That's the reason I changed it and added the S3 to GCS below for S3ToGCSOperator
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #10951: Add example dag and system test for S3ToGCSOperator
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #10951:
URL: https://github.com/apache/airflow/pull/10951#issuecomment-693171348
Hi @potiuk , I can't seem to figure out this pylint error. I will appreciate your assistance 🙂. Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #10951: Add example dag and system test for S3ToGCSOperator
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #10951:
URL: https://github.com/apache/airflow/pull/10951
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #10951: Add example dag and system test for S3ToGCSOperator
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #10951:
URL: https://github.com/apache/airflow/pull/10951#discussion_r488910813
##########
File path: docs/operators-and-hooks-ref.rst
##########
@@ -934,11 +934,15 @@ These integrations allow you to copy data from/to Google Cloud.
- :doc:`How to use <howto/operator/amazon/aws/google_api_to_s3_transfer>`
- :mod:`airflow.providers.amazon.aws.transfers.google_api_to_s3`
- * - `Amazon Simple Storage Service (S3) <https://aws.amazon.com/s3/>`__
+ * - Any Data Storage service
Review comment:
I'm not sure the best way to word this but I see that Google Transfer service is not only for the S3. https://cloud.google.com/storage-transfer-service.
Also, having it as S3 to GCS transfer will overshadow the S3ToGCSOperator. That's the reason I changed it and added the S3 to GCS below for S3ToGCSOperator
##########
File path: docs/operators-and-hooks-ref.rst
##########
@@ -934,11 +934,15 @@ These integrations allow you to copy data from/to Google Cloud.
- :doc:`How to use <howto/operator/amazon/aws/google_api_to_s3_transfer>`
- :mod:`airflow.providers.amazon.aws.transfers.google_api_to_s3`
- * - `Amazon Simple Storage Service (S3) <https://aws.amazon.com/s3/>`__
+ * - Any Data Storage service
Review comment:
I have removed this entirely. It's a duplicate. It's covered in google cloud operators as Storage Transfer Service and in Cloud Transfer Operators as s3 to gcs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on pull request #10951: Add example dag and system test for S3ToGCSOperator
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on pull request #10951:
URL: https://github.com/apache/airflow/pull/10951#issuecomment-693507584
All tests have passed🚀🚀🚀🚀
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org