You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/24 11:35:38 UTC
[GitHub] [airflow] bhirsz opened a new pull request, #25934: Fix gcs to sftp system test
bhirsz opened a new pull request, #25934:
URL: https://github.com/apache/airflow/pull/25934
``gcs_to_sftp`` example was missing setup and teardown stages after
migration to the new design. This PR adds them - now we upload file to GCS bucket before running transfer operators and remove GCS bucket at the end of the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bhirsz commented on a diff in pull request #25934: Fix gcs to sftp system test
Posted by GitBox <gi...@apache.org>.
bhirsz commented on code in PR #25934:
URL: https://github.com/apache/airflow/pull/25934#discussion_r954649131
##########
tests/system/providers/google/cloud/transfers/example_gcs_to_sftp.py:
##########
@@ -46,12 +53,35 @@
catchup=False,
tags=['example', 'gcs'],
) as dag:
+ create_bucket = GCSCreateBucketOperator(
+ task_id="create_bucket", bucket_name=BUCKET_NAME, project_id=PROJECT_ID
+ )
+
+ upload_file_1 = LocalFilesystemToGCSOperator(
Review Comment:
I realized one of the operators has a "move_object" parameter which will delete files from the source directory. That's why we need all three separate uploads to cover all test scenarios:
Copy single file - 1st upload
Move file from the directory (removing source) - 2nd upload
Copy file from the directory when preserving dir structure - 3rd upload
Copy file from the directory without preserving dir structure - 3rd upload
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bhirsz commented on a diff in pull request #25934: Fix gcs to sftp system test
Posted by GitBox <gi...@apache.org>.
bhirsz commented on code in PR #25934:
URL: https://github.com/apache/airflow/pull/25934#discussion_r953697936
##########
tests/system/providers/google/cloud/transfers/example_gcs_to_sftp.py:
##########
@@ -46,12 +53,35 @@
catchup=False,
tags=['example', 'gcs'],
) as dag:
+ create_bucket = GCSCreateBucketOperator(
+ task_id="create_bucket", bucket_name=BUCKET_NAME, project_id=PROJECT_ID
+ )
+
+ upload_file_1 = LocalFilesystemToGCSOperator(
Review Comment:
To ensure I have file available on different paths - directly in bucket root, under directory etc. It's later used by transfer operator and we are testing if its possible to copy file, copy file from dir (and recreate dir structures), copy with wildcard. But thanks to your question I have noticed we dont need third upload - since 2nd upload already puts file under dir and we came reuse it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajastro commented on a diff in pull request #25934: Fix gcs to sftp system test
Posted by GitBox <gi...@apache.org>.
pankajastro commented on code in PR #25934:
URL: https://github.com/apache/airflow/pull/25934#discussion_r953688904
##########
tests/system/providers/google/cloud/transfers/example_gcs_to_sftp.py:
##########
@@ -46,12 +53,35 @@
catchup=False,
tags=['example', 'gcs'],
) as dag:
+ create_bucket = GCSCreateBucketOperator(
+ task_id="create_bucket", bucket_name=BUCKET_NAME, project_id=PROJECT_ID
+ )
+
+ upload_file_1 = LocalFilesystemToGCSOperator(
Review Comment:
wondering why you are uploading multiple times
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #25934: Fix gcs to sftp system test
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #25934:
URL: https://github.com/apache/airflow/pull/25934
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bhirsz commented on a diff in pull request #25934: Fix gcs to sftp system test
Posted by GitBox <gi...@apache.org>.
bhirsz commented on code in PR #25934:
URL: https://github.com/apache/airflow/pull/25934#discussion_r953697936
##########
tests/system/providers/google/cloud/transfers/example_gcs_to_sftp.py:
##########
@@ -46,12 +53,35 @@
catchup=False,
tags=['example', 'gcs'],
) as dag:
+ create_bucket = GCSCreateBucketOperator(
+ task_id="create_bucket", bucket_name=BUCKET_NAME, project_id=PROJECT_ID
+ )
+
+ upload_file_1 = LocalFilesystemToGCSOperator(
Review Comment:
To ensure I have file available on different paths - directly in bucket root, under directory etc. It's łatwe used by transfer operator and we are testing if its possible to copy file, copy file from dir (and recreate dir structures), copy with wildcard. But thanks to your question I have noticed we dont need third upload - since 2nd upload already puts file under dir and we came reuse it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org