[GitHub] potiuk commented on a change in pull request #4493: [AIRFLOW-3680] Consistency update in tests for All GCP-related operators

[GitBox <gi...@apache.org>]

potiuk commented on a change in pull request #4493: [AIRFLOW-3680] Consistency update in tests for All GCP-related operators
URL: https://github.com/apache/airflow/pull/4493#discussion_r247299947
 
 

 ##########
 File path: tests/contrib/operators/postgres_local_executor.cfg
 ##########
 @@ -0,0 +1,28 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+# This configuration file is used to override
+# the executor to be LocalExecutor and switch to using Postgres as the database
+# Some system tests require LocalExecutor to be used because of parallelism required
+# One example is test_gcp_bigtable_operator_system.py in which sensor
+# Has to be run in parallell with a task to create BigTable table
+[core]
+executor = LocalExecutor
+sql_alchemy_conn = postgresql:///airflow/airflow.db
 
 Review comment:
   There are some system tests that actually require Local Executor - It's actually explained in the comment above :) . We have one test (test_gcp_bigtable_operator_system.py) which is based on example_gcp_bigtable_operators.py DAG. In this test we start an operator (create bigtable database) and sensor (wait for database replication) in parallel. It makes sense to start them in parallel. only when both are complete we attempt to delete the database. This way we make sure that the sensor is actually waiting for something and not simply firing off immediately after create. And that it does some polling even before the database is created and in the process of being created and works fine for that statuses as well. 
   
   Note that this is merely a configuration file that you can set in AIRFLOW_CONFIG - it is disabled by default in local environment and IDE system tests (there we use sqlite + sequential executor by default). Only when you set AIRFLOW_CONFIG variable to this file before starting the tests, local executor and postgres are used. But the tests in the container environment (and in Cloud Build) are executed using Postgres + LocalExecutor by default. This way we also check that our examples are not working simply because they run sequentially. We caught couple of errors (races/parallel execution problems) which we would not have found if we run them with Sequential Executor.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services