You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2023/07/05 10:51:52 UTC

[GitHub] [airflow] amoghrajesh opened a new pull request, #32368: Migrating Google AutoML example_dags to sys tests

amoghrajesh opened a new pull request, #32368:
URL: https://github.com/apache/airflow/pull/32368

   <!--
    Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
   
      http://www.apache.org/licenses/LICENSE-2.0
   
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.
    -->
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   related: https://github.com/apache/airflow/issues/32295
   
   Moving the `example_dags` for google cloud sensors to sys tests.
   This is a step by step effort to migrate all the google cloud tests to sys tests.
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] amoghrajesh commented on a diff in pull request #32368: Migrating Google AutoML example_dags to sys tests

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #32368:
URL: https://github.com/apache/airflow/pull/32368#discussion_r1252944777


##########
tests/system/providers/google/cloud/automl/example_automl_nl_text_classification.py:
##########
@@ -35,17 +35,14 @@
     AutoMLTrainModelOperator,
 )
 
-GCP_PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "your-project-id")
-GCP_AUTOML_LOCATION = os.environ.get("GCP_AUTOML_LOCATION", "us-central1")
-GCP_AUTOML_TEXT_CLS_BUCKET = os.environ.get("GCP_AUTOML_TEXT_CLS_BUCKET", "gs://INVALID BUCKET NAME")
-
-# Example values
-DATASET_ID = ""
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = "example_automl_classification"
+GCP_PROJECT_ID = os.environ.get("SYSTEM_TESTS_GCP_PROJECT", "default")
+GCP_AUTOML_LOCATION = "us-central1"

Review Comment:
   I was following the convention of the other system tests written, for example in https://github.com/apache/airflow/pull/30003 and also in the automl folder. Does it make sense to auto hard code them?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Adaverse commented on a diff in pull request #32368: Migrating Google AutoML example_dags to sys tests

Posted by "Adaverse (via GitHub)" <gi...@apache.org>.
Adaverse commented on code in PR #32368:
URL: https://github.com/apache/airflow/pull/32368#discussion_r1252938609


##########
tests/system/providers/google/cloud/automl/example_automl_nl_text_classification.py:
##########
@@ -35,17 +35,14 @@
     AutoMLTrainModelOperator,
 )
 
-GCP_PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "your-project-id")
-GCP_AUTOML_LOCATION = os.environ.get("GCP_AUTOML_LOCATION", "us-central1")
-GCP_AUTOML_TEXT_CLS_BUCKET = os.environ.get("GCP_AUTOML_TEXT_CLS_BUCKET", "gs://INVALID BUCKET NAME")
-
-# Example values
-DATASET_ID = ""
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = "example_automl_classification"
+GCP_PROJECT_ID = os.environ.get("SYSTEM_TESTS_GCP_PROJECT", "default")
+GCP_AUTOML_LOCATION = "us-central1"

Review Comment:
   Should we hardcode these values? Shouldn't it be fetched from env var?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] amoghrajesh commented on a diff in pull request #32368: Migrating Google AutoML example_dags to sys tests

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #32368:
URL: https://github.com/apache/airflow/pull/32368#discussion_r1253090499


##########
tests/system/providers/google/cloud/automl/example_automl_nl_text_classification.py:
##########
@@ -35,17 +35,14 @@
     AutoMLTrainModelOperator,
 )
 
-GCP_PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "your-project-id")
-GCP_AUTOML_LOCATION = os.environ.get("GCP_AUTOML_LOCATION", "us-central1")
-GCP_AUTOML_TEXT_CLS_BUCKET = os.environ.get("GCP_AUTOML_TEXT_CLS_BUCKET", "gs://INVALID BUCKET NAME")
-
-# Example values
-DATASET_ID = ""
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = "example_automl_classification"
+GCP_PROJECT_ID = os.environ.get("SYSTEM_TESTS_GCP_PROJECT", "default")
+GCP_AUTOML_LOCATION = "us-central1"

Review Comment:
   Thanks. Do you mean to say that we should have it as: 
   ```
   GCP_PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "your-project-id")
   GCP_AUTOML_LOCATION = os.environ.get("GCP_AUTOML_LOCATION", "us-central1")
   GCP_AUTOML_TEXT_CLS_BUCKET = os.environ.get("GCP_AUTOML_TEXT_CLS_BUCKET", "gs://INVALID BUCKET NAME")
   ```
   And so on?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] amoghrajesh commented on pull request #32368: Migrating Google AutoML example_dags to sys tests

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #32368:
URL: https://github.com/apache/airflow/pull/32368#issuecomment-1623050950

   @Adaverse I have handled your review comments in the newest commit.
   cc @potiuk @VladaZakharova 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #32368: Migrating Google AutoML example_dags to sys tests

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #32368:
URL: https://github.com/apache/airflow/pull/32368#discussion_r1252955435


##########
tests/system/providers/google/cloud/automl/example_automl_nl_text_classification.py:
##########
@@ -35,17 +35,14 @@
     AutoMLTrainModelOperator,
 )
 
-GCP_PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "your-project-id")
-GCP_AUTOML_LOCATION = os.environ.get("GCP_AUTOML_LOCATION", "us-central1")
-GCP_AUTOML_TEXT_CLS_BUCKET = os.environ.get("GCP_AUTOML_TEXT_CLS_BUCKET", "gs://INVALID BUCKET NAME")
-
-# Example values
-DATASET_ID = ""
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = "example_automl_classification"
+GCP_PROJECT_ID = os.environ.get("SYSTEM_TESTS_GCP_PROJECT", "default")
+GCP_AUTOML_LOCATION = "us-central1"

Review Comment:
   cc: @VladaZakharova might help here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Adaverse commented on a diff in pull request #32368: Migrating Google AutoML example_dags to sys tests

Posted by "Adaverse (via GitHub)" <gi...@apache.org>.
Adaverse commented on code in PR #32368:
URL: https://github.com/apache/airflow/pull/32368#discussion_r1254691133


##########
tests/system/providers/google/cloud/automl/example_automl_nl_text_classification.py:
##########
@@ -99,10 +95,17 @@
         project_id=GCP_PROJECT_ID,
     )
 
+    # TEST BODY
     import_dataset_task >> create_model
+    # TEST TEARDOWN
     delete_model_task >> delete_datasets_task
 
     # Task dependencies created via `XComArgs`:
     #   create_dataset_task >> import_dataset_task
     #   create_dataset_task >> create_model
     #   create_dataset_task >> delete_datasets_task
+

Review Comment:
   Seems like we need a watcher wdyt? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] amoghrajesh commented on a diff in pull request #32368: Migrating Google AutoML example_dags to sys tests

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #32368:
URL: https://github.com/apache/airflow/pull/32368#discussion_r1252977868


##########
tests/system/providers/google/cloud/automl/example_automl_nl_text_classification.py:
##########
@@ -35,17 +35,14 @@
     AutoMLTrainModelOperator,
 )
 
-GCP_PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "your-project-id")
-GCP_AUTOML_LOCATION = os.environ.get("GCP_AUTOML_LOCATION", "us-central1")
-GCP_AUTOML_TEXT_CLS_BUCKET = os.environ.get("GCP_AUTOML_TEXT_CLS_BUCKET", "gs://INVALID BUCKET NAME")
-
-# Example values
-DATASET_ID = ""
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = "example_automl_classification"
+GCP_PROJECT_ID = os.environ.get("SYSTEM_TESTS_GCP_PROJECT", "default")
+GCP_AUTOML_LOCATION = "us-central1"

Review Comment:
   Nice point. Any suggestions that i can follow? It will help me tweak the DAG according to your use cases



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Adaverse commented on a diff in pull request #32368: Migrating Google AutoML example_dags to sys tests

Posted by "Adaverse (via GitHub)" <gi...@apache.org>.
Adaverse commented on code in PR #32368:
URL: https://github.com/apache/airflow/pull/32368#discussion_r1253077528


##########
tests/system/providers/google/cloud/automl/example_automl_nl_text_classification.py:
##########
@@ -35,17 +35,14 @@
     AutoMLTrainModelOperator,
 )
 
-GCP_PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "your-project-id")
-GCP_AUTOML_LOCATION = os.environ.get("GCP_AUTOML_LOCATION", "us-central1")
-GCP_AUTOML_TEXT_CLS_BUCKET = os.environ.get("GCP_AUTOML_TEXT_CLS_BUCKET", "gs://INVALID BUCKET NAME")
-
-# Example values
-DATASET_ID = ""
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = "example_automl_classification"
+GCP_PROJECT_ID = os.environ.get("SYSTEM_TESTS_GCP_PROJECT", "default")
+GCP_AUTOML_LOCATION = "us-central1"

Review Comment:
   Generally, all the tests look good, except for the first point.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] amoghrajesh commented on a diff in pull request #32368: Migrating Google AutoML example_dags to sys tests

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #32368:
URL: https://github.com/apache/airflow/pull/32368#discussion_r1255590616


##########
tests/system/providers/google/cloud/automl/example_automl_nl_text_classification.py:
##########
@@ -99,10 +95,17 @@
         project_id=GCP_PROJECT_ID,
     )
 
+    # TEST BODY
     import_dataset_task >> create_model
+    # TEST TEARDOWN
     delete_model_task >> delete_datasets_task
 
     # Task dependencies created via `XComArgs`:
     #   create_dataset_task >> import_dataset_task
     #   create_dataset_task >> create_model
     #   create_dataset_task >> delete_datasets_task
+

Review Comment:
   Sure, we can have a watcher too for each of these tests



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Adaverse commented on a diff in pull request #32368: Migrating Google AutoML example_dags to sys tests

Posted by "Adaverse (via GitHub)" <gi...@apache.org>.
Adaverse commented on code in PR #32368:
URL: https://github.com/apache/airflow/pull/32368#discussion_r1252974978


##########
tests/system/providers/google/cloud/automl/example_automl_nl_text_classification.py:
##########
@@ -35,17 +35,14 @@
     AutoMLTrainModelOperator,
 )
 
-GCP_PROJECT_ID = os.environ.get("GCP_PROJECT_ID", "your-project-id")
-GCP_AUTOML_LOCATION = os.environ.get("GCP_AUTOML_LOCATION", "us-central1")
-GCP_AUTOML_TEXT_CLS_BUCKET = os.environ.get("GCP_AUTOML_TEXT_CLS_BUCKET", "gs://INVALID BUCKET NAME")
-
-# Example values
-DATASET_ID = ""
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = "example_automl_classification"
+GCP_PROJECT_ID = os.environ.get("SYSTEM_TESTS_GCP_PROJECT", "default")
+GCP_AUTOML_LOCATION = "us-central1"

Review Comment:
   The main principle is to keep these tests self-sufficient. But they need an external resource and infra on which those resources run. We can control those resources, like creating a bucket on the fly, but if a user has a certain infra, IMO it would be best to give flexibility to the user to set up the test given the infra rather than looking at the test and setting up the infra. That would end up making these tests less useful wdyt?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk merged pull request #32368: Migrating Google AutoML example_dags to sys tests

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #32368:
URL: https://github.com/apache/airflow/pull/32368


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Adaverse commented on a diff in pull request #32368: Migrating Google AutoML example_dags to sys tests

Posted by "Adaverse (via GitHub)" <gi...@apache.org>.
Adaverse commented on code in PR #32368:
URL: https://github.com/apache/airflow/pull/32368#discussion_r1254691133


##########
tests/system/providers/google/cloud/automl/example_automl_nl_text_classification.py:
##########
@@ -99,10 +95,17 @@
         project_id=GCP_PROJECT_ID,
     )
 
+    # TEST BODY
     import_dataset_task >> create_model
+    # TEST TEARDOWN
     delete_model_task >> delete_datasets_task
 
     # Task dependencies created via `XComArgs`:
     #   create_dataset_task >> import_dataset_task
     #   create_dataset_task >> create_model
     #   create_dataset_task >> delete_datasets_task
+

Review Comment:
   Seems like we need a watcher?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org