You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/27 10:17:20 UTC

[GitHub] [airflow] feluelle opened a new pull request #8581: Add google_api_to_s3_transfer example dags and system tests

feluelle opened a new pull request #8581:
URL: https://github.com/apache/airflow/pull/8581


   This PR / change adds google_api_to_s3_transfer systems tests by running the newly added example dags. It also adds a aws system helper which can be used for providing aws context like connection or a s3 bucket for example to the system tests so you do not need to care about how to create or delete resources.
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   
   cc @xinbinhuang @mustafagok - PTAL :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #8581: Add google_api_to_s3_transfer example dags and system tests

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #8581:
URL: https://github.com/apache/airflow/pull/8581#issuecomment-619894329


   There are some problems with test collection though :( see the GA errors..


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] feluelle commented on pull request #8581: Add google_api_to_s3_transfer example dags and system tests

Posted by GitBox <gi...@apache.org>.
feluelle commented on pull request #8581:
URL: https://github.com/apache/airflow/pull/8581#issuecomment-625123558


   Thank you Kamil 👍 :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mustafagok commented on pull request #8581: Add google_api_to_s3_transfer example dags and system tests

Posted by GitBox <gi...@apache.org>.
mustafagok commented on pull request #8581:
URL: https://github.com/apache/airflow/pull/8581#issuecomment-623156223


   I see that you have much bigger problems, I would like to suggest a small change, about naming "aws_system_helpers.py" as "amazon_system_helpers.py" since it contains `AmazonSystemTest` class and `@pytest.mark.system("amazon")`


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] feluelle commented on pull request #8581: Add google_api_to_s3_transfer example dags and system tests

Posted by GitBox <gi...@apache.org>.
feluelle commented on pull request #8581:
URL: https://github.com/apache/airflow/pull/8581#issuecomment-623628892


   > I see that you have much bigger problems, I would like to suggest a small change, about naming "aws_system_helpers.py" as "amazon_system_helpers.py" since it contains AmazonSystemTest class and @pytest.mark.system("amazon")
   
   Fixed. 👍 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] feluelle edited a comment on pull request #8581: Add google_api_to_s3_transfer example dags and system tests

Posted by GitBox <gi...@apache.org>.
feluelle edited a comment on pull request #8581:
URL: https://github.com/apache/airflow/pull/8581#issuecomment-625104158


   The docs build failed on master due to missing :doc:`How to use <howto/operator/amazon/aws/google_api_to_s3_transfer>` - follow-up PR is coming. #8761
   
   I am wondering why the build docs task succeeded on this branch but fails on master.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on a change in pull request #8581: Add google_api_to_s3_transfer example dags and system tests

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8581:
URL: https://github.com/apache/airflow/pull/8581#discussion_r415700240



##########
File path: tests/test_utils/aws_system_helpers.py
##########
@@ -0,0 +1,84 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+from contextlib import contextmanager
+from typing import List
+
+import pytest
+
+from tests.test_utils import AIRFLOW_MAIN_FOLDER
+from tests.test_utils.system_tests_class import SystemTest
+from tests.utils.logging_command_executor import get_executor
+
+AWS_DAG_FOLDER = os.path.join(
+    AIRFLOW_MAIN_FOLDER, "airflow", "providers", "amazon", "aws", "example_dags"
+)
+
+
+@contextmanager
+def provide_aws_context():

Review comment:
       Nice! I like the way you used context managers here :)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on a change in pull request #8581: Add google_api_to_s3_transfer example dags and system tests

Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8581:
URL: https://github.com/apache/airflow/pull/8581#discussion_r420121822



##########
File path: backport_packages/setup_backport_packages.py
##########
@@ -251,6 +251,13 @@ def _remover(node: LN, capture: Capture, filename: Filename) -> None:
         .modify(add_provide_context_to_python_operator)
     )
 
+    (
+        qry.select_function("BranchPythonOperator")
+        .is_call()
+        .is_filename(include=r"example_google_api_to_s3_transfer_advanced.py$")
+        .modify(add_provide_context_to_python_operator)
+    )

Review comment:
       I think it's worth limiting changes and not changing all files automatically. If we need to change another example, it will be easy to add here., ..so LGTM




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] feluelle commented on pull request #8581: Add google_api_to_s3_transfer example dags and system tests

Posted by GitBox <gi...@apache.org>.
feluelle commented on pull request #8581:
URL: https://github.com/apache/airflow/pull/8581#issuecomment-623999244


   @mik-laj or @turbaszek want to have a last look? I would like to merge it (Jarek approved it already) but I also would like to have an opinion on the change in `setup_backport_packages.py`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] feluelle commented on pull request #8581: Add google_api_to_s3_transfer example dags and system tests

Posted by GitBox <gi...@apache.org>.
feluelle commented on pull request #8581:
URL: https://github.com/apache/airflow/pull/8581#issuecomment-621643901


   Thanks @potiuk for the information. So would I, for now, just add the file to the `is_filename` expression or should we really run it on all `*PythonOperator`s. PTAL @turbaszek :)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mik-laj commented on pull request #8581: Add google_api_to_s3_transfer example dags and system tests

Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #8581:
URL: https://github.com/apache/airflow/pull/8581#issuecomment-625120453


   You didn't do rebase before merging it. This check has been added 3 days ago. https://github.com/apache/airflow/commit/923f4231126b84de90d1e78ce0470c153a67f0ea


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] feluelle commented on a change in pull request #8581: Add google_api_to_s3_transfer example dags and system tests

Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #8581:
URL: https://github.com/apache/airflow/pull/8581#discussion_r415691133



##########
File path: tests/test_utils/aws_system_helpers.py
##########
@@ -0,0 +1,84 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+from contextlib import contextmanager
+from typing import List
+
+import pytest
+
+from tests.test_utils import AIRFLOW_MAIN_FOLDER
+from tests.test_utils.system_tests_class import SystemTest
+from tests.utils.logging_command_executor import get_executor
+
+AWS_DAG_FOLDER = os.path.join(
+    AIRFLOW_MAIN_FOLDER, "airflow", "providers", "amazon", "aws", "example_dags"
+)
+
+
+@contextmanager
+def provide_aws_context():
+    """
+    Authenticates the context to be able use aws resources.
+
+    Falls back to awscli default authentication methods via `.aws`` folder.
+    """
+    # TODO: Implement more authentication methods
+    yield
+
+
+@contextmanager
+def provide_aws_s3_bucket(name):
+    AmazonSystemTest.create_aws_s3_bucket(name)
+    yield
+    AmazonSystemTest.delete_aws_s3_bucket(name)
+
+
+@pytest.mark.system("amazon")
+class AmazonSystemTest(SystemTest):
+
+    @classmethod
+    def execute_with_ctx(cls, cmd: List[str]):
+        """
+        Executes command with context created by provide_gcp_context and activated
+        service key.

Review comment:
       ```suggestion
           Executes command with context created by provide_aws_context.
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] feluelle commented on a change in pull request #8581: Add google_api_to_s3_transfer example dags and system tests

Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #8581:
URL: https://github.com/apache/airflow/pull/8581#discussion_r417822601



##########
File path: backport_packages/setup_backport_packages.py
##########
@@ -251,6 +251,13 @@ def _remover(node: LN, capture: Capture, filename: Filename) -> None:
         .modify(add_provide_context_to_python_operator)
     )
 
+    (
+        qry.select_function("BranchPythonOperator")
+        .is_call()
+        .is_filename(include=r"example_google_api_to_s3_transfer_advanced.py$")
+        .modify(add_provide_context_to_python_operator)
+    )

Review comment:
       For now I added this specific case here. If we want to remove the `.is_filename` filter this should be a separate PR. WDYT @potiuk ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #8581: Add google_api_to_s3_transfer example dags and system tests

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #8581:
URL: https://github.com/apache/airflow/pull/8581#issuecomment-619893247


   :heart: 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] xinbinhuang commented on pull request #8581: Add google_api_to_s3_transfer example dags and system tests

Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on pull request #8581:
URL: https://github.com/apache/airflow/pull/8581#issuecomment-620953560


   Awesome! Thanks for creating the `aws_system_helper.py`!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org