You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/03/25 18:31:51 UTC

[GitHub] [airflow] roitvt opened a new pull request #7875: add spark_kubernetes system test

roitvt opened a new pull request #7875: add spark_kubernetes system test
URL: https://github.com/apache/airflow/pull/7875
 
 
   add spark_kubernetes system test
   ---
   Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] roitvt commented on issue #7875: add spark_kubernetes system test

Posted by GitBox <gi...@apache.org>.

roitvt commented on issue #7875: add spark_kubernetes system test
URL: https://github.com/apache/airflow/pull/7875#issuecomment-606567765
 
 
   @potiuk fixed the ci error :) 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7875: add spark_kubernetes system test

Posted by GitBox <gi...@apache.org>.

mik-laj commented on a change in pull request #7875: add spark_kubernetes system test
URL: https://github.com/apache/airflow/pull/7875#discussion_r402750518
 
 

 ##########
 File path: tests/providers/cncf/kubernetes/operators/test_spark_kubernetes_system.py
 ##########
 @@ -0,0 +1,72 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import os
+import subprocess
+
+import pytest
+
+from airflow.models import Connection
+from airflow.utils import db
+from tests.test_utils import AIRFLOW_MAIN_FOLDER
+from tests.test_utils.system_tests_class import SystemTest
+
+KUBERNETES_DAG_FOLDER = os.path.join(
+    AIRFLOW_MAIN_FOLDER, "airflow", "providers", "cncf", "kubernetes", "example_dags"
+)
+
+
+def _spark_operator_util(action):
 
 Review comment:
   @potiuk We must separate Kubernetes from the core. For now it won't be possible to use it on 1.10.X.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7875: add spark_kubernetes system test

Posted by GitBox <gi...@apache.org>.

potiuk commented on a change in pull request #7875: add spark_kubernetes system test
URL: https://github.com/apache/airflow/pull/7875#discussion_r398122902
 
 

 ##########
 File path: tests/providers/cncf/kubernetes/operators/test_spark_kubernetes_system.py
 ##########
 @@ -0,0 +1,72 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import os
+import subprocess
+
+import pytest
+
+from airflow.models import Connection
+from airflow.utils import db
+from tests.test_utils import AIRFLOW_MAIN_FOLDER
+from tests.test_utils.system_tests_class import SystemTest
+
+KUBERNETES_DAG_FOLDER = os.path.join(
+    AIRFLOW_MAIN_FOLDER, "airflow", "providers", "cncf", "kubernetes", "example_dags"
+)
+
+
+def _spark_operator_util(action):
 
 Review comment:
   @mik-laj - -will it work in 1.10 *? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] roitvt commented on issue #7875: add spark_kubernetes system test

Posted by GitBox <gi...@apache.org>.

roitvt commented on issue #7875: add spark_kubernetes system test
URL: https://github.com/apache/airflow/pull/7875#issuecomment-607093814
 
 
   ping @mik-laj 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #7875: add spark_kubernetes system test

Posted by GitBox <gi...@apache.org>.

mik-laj commented on issue #7875: add spark_kubernetes system test
URL: https://github.com/apache/airflow/pull/7875#issuecomment-604063097
 
 
   @potiuk I am a bit busy working on client stuff. I will try to look at it and run on Monday.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on issue #7875: add spark_kubernetes system test

Posted by GitBox <gi...@apache.org>.

mik-laj commented on issue #7875: add spark_kubernetes system test
URL: https://github.com/apache/airflow/pull/7875#issuecomment-608243074
 
 
   ```
   root@01a7f91cd980:/opt/airflow# pytest tests/providers/cncf/kubernetes/operators/test_spark_kubernetes_system.py  -s --systems cncf.kubernetes
   =========================================================================================================================================================================== test session starts ============================================================================================================================================================================
   platform linux -- Python 3.6.10, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /usr/local/bin/python
   cachedir: .pytest_cache
   rootdir: /opt/airflow, inifile: pytest.ini
   plugins: flaky-3.6.1, instafail-0.4.1.post0, requests-mock-1.7.0, celery-4.4.2, cov-2.8.1
   collected 1 item
   
   tests/providers/cncf/kubernetes/operators/test_spark_kubernetes_system.py::SparkKubernetesExampleDagsSystemTest::test_run_example_dag_spark_pi ========================= AIRFLOW ==========================
   Home of the user: /root
   Airflow home /root/airflow
   Initializing the DB - first time after entering the container.
   You can force re-initialization the database by adding --with-db-init switch to run-tests.
   [2020-04-03 05:41:31,708] {db.py:552} INFO - Dropping tables that exist
   [2020-04-03 05:41:31,951] {migration.py:155} INFO - Context impl PostgresqlImpl.
   [2020-04-03 05:41:31,952] {migration.py:162} INFO - Will assume transactional DDL.
   [2020-04-03 05:41:32,163] {db.py:535} INFO - Creating tables
   INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
   INFO  [alembic.runtime.migration] Will assume transactional DDL.
   INFO  [alembic.runtime.migration] Running upgrade  -> e3a246e0dc1, current schema
   INFO  [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
   INFO  [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 13eb55f81627, maintain history for compatibility with earlier migrations
   INFO  [alembic.runtime.migration] Running upgrade 13eb55f81627 -> 338e90f54d61, More logging into task_instance
   INFO  [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, job_id indices
   INFO  [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84, Adding extra to Log
   INFO  [alembic.runtime.migration] Running upgrade 502898887f84 -> 1b38cef5b76e, add dagrun
   INFO  [alembic.runtime.migration] Running upgrade 1b38cef5b76e -> 2e541a1dcfed, task_duration
   INFO  [alembic.runtime.migration] Running upgrade 2e541a1dcfed -> 40e67319e3a9, dagrun_config
   INFO  [alembic.runtime.migration] Running upgrade 40e67319e3a9 -> 561833c1c74b, add password column to user
   INFO  [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588, dagrun start end
   INFO  [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e, Add notification_sent column to sla_miss
   INFO  [alembic.runtime.migration] Running upgrade bbc73705a13e -> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field in connection
   INFO  [alembic.runtime.migration] Running upgrade bba5a7cfc896 -> 1968acfc09e3, add is_encrypted column to variable table
   INFO  [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> 2e82aab8ef20, rename user table
   INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 -> 211e584da130, add TI state index
   INFO  [alembic.runtime.migration] Running upgrade 211e584da130 -> 64de9cddf6c9, add task fails journal table
   INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 -> f2ca10b85618, add dag_stats table
   INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 -> 4addfa1236f1, Add fractional seconds to mysql tables
   INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 -> 8504051e801b, xcom dag task indices
   INFO  [alembic.runtime.migration] Running upgrade 8504051e801b -> 5e7d17757c7a, add pid field to TaskInstance
   INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a -> 127d2bf2dfa7, Add dag_id/state index on dag_run table
   INFO  [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> cc1e65623dc7, add max tries column to task instance
   INFO  [alembic.runtime.migration] Running upgrade cc1e65623dc7 -> bdaa763e6c56, Make xcom value column a large binary
   INFO  [alembic.runtime.migration] Running upgrade bdaa763e6c56 -> 947454bf1dff, add ti job_id index
   INFO  [alembic.runtime.migration] Running upgrade 947454bf1dff -> d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text types)
   INFO  [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 0e2a74e0fc9f, Add time zone awareness
   INFO  [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 33ae817a1ff4, kubernetes_resource_checkpointing
   INFO  [alembic.runtime.migration] Running upgrade 33ae817a1ff4 -> 27c6a30d7c24, kubernetes_resource_checkpointing
   INFO  [alembic.runtime.migration] Running upgrade 27c6a30d7c24 -> 86770d1215c0, add kubernetes scheduler uniqueness
   INFO  [alembic.runtime.migration] Running upgrade 86770d1215c0, 0e2a74e0fc9f -> 05f30312d566, merge heads
   INFO  [alembic.runtime.migration] Running upgrade 05f30312d566 -> f23433877c24, fix mysql not null constraint
   INFO  [alembic.runtime.migration] Running upgrade f23433877c24 -> 856955da8476, fix sqlite foreign key
   INFO  [alembic.runtime.migration] Running upgrade 856955da8476 -> 9635ae0956e7, index-faskfail
   INFO  [alembic.runtime.migration] Running upgrade 9635ae0956e7 -> dd25f486b8ea, add idx_log_dag
   INFO  [alembic.runtime.migration] Running upgrade dd25f486b8ea -> bf00311e1990, add index to taskinstance
   INFO  [alembic.runtime.migration] Running upgrade 9635ae0956e7 -> 0a2a5b66e19d, add task_reschedule table
   INFO  [alembic.runtime.migration] Running upgrade 0a2a5b66e19d, bf00311e1990 -> 03bc53e68815, merge_heads_2
   INFO  [alembic.runtime.migration] Running upgrade 03bc53e68815 -> 41f5f12752f8, add superuser field
   INFO  [alembic.runtime.migration] Running upgrade 41f5f12752f8 -> c8ffec048a3b, add fields to dag
   INFO  [alembic.runtime.migration] Running upgrade 41f5f12752f8 -> cf5dc11e79ad, drop_user_and_chart
   INFO  [alembic.runtime.migration] Running upgrade c8ffec048a3b -> dd4ecb8fbee3, Add schedule interval to dag
   INFO  [alembic.runtime.migration] Running upgrade dd4ecb8fbee3, cf5dc11e79ad -> 4ebbffe0a39a, Merge heads
   INFO  [alembic.runtime.migration] Running upgrade 4ebbffe0a39a -> 939bb1e647c8, task reschedule fk on cascade delete
   INFO  [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 6e96a59344a4, Make TaskInstance.pool not nullable
   INFO  [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 004c1210f153, increase queue name size limit
   INFO  [alembic.runtime.migration] Running upgrade 004c1210f153 -> bbf4a7ad0465, Remove id column from xcom
   INFO  [alembic.runtime.migration] Running upgrade 6e96a59344a4 -> d38e04c12aa2, add serialized_dag table
   INFO  [alembic.runtime.migration] Running upgrade d38e04c12aa2 -> b3b105409875, add root_dag_id to DAG
   Revision ID: b3b105409875
   Revises: d38e04c12aa2
   Create Date: 2019-09-28 23:20:01.744775
   INFO  [alembic.runtime.migration] Running upgrade 6e96a59344a4 -> 74effc47d867, change datetime to datetime2(6) on MSSQL tables
   INFO  [alembic.runtime.migration] Running upgrade c8ffec048a3b -> a56c9515abdc, Remove dag_stat table
   INFO  [alembic.runtime.migration] Running upgrade a56c9515abdc, 74effc47d867, b3b105409875, bbf4a7ad0465 -> 08364691d074, merge multiple heads
   INFO  [alembic.runtime.migration] Running upgrade 08364691d074 -> fe461863935f, increase_length_for_connection_password
   INFO  [alembic.runtime.migration] Running upgrade fe461863935f -> 7939bcff74ba, Add DagTags table
   INFO  [alembic.runtime.migration] Running upgrade 7939bcff74ba -> a4c2fd67d16b, add pool_slots field to task_instance
   INFO  [alembic.runtime.migration] Running upgrade a4c2fd67d16b -> b25a55525161, Increase length of pool name
   INFO  [alembic.runtime.migration] Running upgrade b25a55525161 -> 852ae6c715af, Add RenderedTaskInstanceFields table
   INFO  [alembic.runtime.migration] Running upgrade 852ae6c715af -> 952da73b5eff, add dag_code table
   
   Removing all log files except previous_runs
   
   customresourcedefinition.apiextensions.k8s.io/sparkapplications.sparkoperator.k8s.io created
   customresourcedefinition.apiextensions.k8s.io/scheduledsparkapplications.sparkoperator.k8s.io created
   namespace/spark-operator created
   serviceaccount/sparkoperator created
   clusterrole.rbac.authorization.k8s.io/sparkoperator created
   clusterrolebinding.rbac.authorization.k8s.io/sparkoperator created
   clusterrole.rbac.authorization.k8s.io/sparkoperator-aggregate-to-admin created
   deployment.apps/sparkoperator created
   serviceaccount/spark created
   role.rbac.authorization.k8s.io/spark-role created
   rolebinding.rbac.authorization.k8s.io/spark-role-binding created
   INFO  [tests.providers.cncf.kubernetes.operators.test_spark_kubernetes_system.SparkKubernetesExampleDagsSystemTest] Looking for DAG: spark_pi in /opt/airflow/airflow/providers/cncf/kubernetes/example_dags
   INFO  [airflow.models.dagbag.DagBag] Filling up the DagBag from /opt/airflow/airflow/providers/cncf/kubernetes/example_dags
   INFO  [tests.providers.cncf.kubernetes.operators.test_spark_kubernetes_system.SparkKubernetesExampleDagsSystemTest] Attempting to run DAG: spark_pi
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies all met for <TaskInstance: spark_pi.spark_pi_submit 2020-04-02 00:00:00+00:00 [scheduled]>
   INFO  [airflow.executors.local_executor.LocalExecutor] Adding to queue: ['airflow', 'tasks', 'run', 'spark_pi', 'spark_pi_submit', '2020-04-02T00:00:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/opt/airflow/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_operator.py', '--cfg-path', '/tmp/tmpwc7d9syu']
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies all met for <TaskInstance: spark_pi.spark_pi_submit 2020-04-03 00:00:00+00:00 [scheduled]>
   INFO  [airflow.executors.local_executor.LocalExecutor] Adding to queue: ['airflow', 'tasks', 'run', 'spark_pi', 'spark_pi_submit', '2020-04-03T00:00:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/opt/airflow/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_operator.py', '--cfg-path', '/tmp/tmp4edqex3_']
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-02 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-03 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.executors.local_executor.QueuedLocalWorker] QueuedLocalWorker running ['airflow', 'tasks', 'run', 'spark_pi', 'spark_pi_submit', '2020-04-02T00:00:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/opt/airflow/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_operator.py', '--cfg-path', '/tmp/tmpwc7d9syu']
   INFO  [airflow.executors.local_executor.QueuedLocalWorker] QueuedLocalWorker running ['airflow', 'tasks', 'run', 'spark_pi', 'spark_pi_submit', '2020-04-03T00:00:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/opt/airflow/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_operator.py', '--cfg-path', '/tmp/tmp4edqex3_']
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 2 | succeeded: 0 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-02 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-03 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 2 | succeeded: 0 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-02 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-03 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 2 | succeeded: 0 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-02 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-03 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 2 | succeeded: 0 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-02 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-03 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 2 | succeeded: 0 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-02 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-03 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 2 | succeeded: 0 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-02 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-03 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   [2020-04-03 05:41:50,875] {dagbag.py:368} INFO - Filling up the DagBag from /opt/airflow/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_operator.py
   [2020-04-03 05:41:50,875] {dagbag.py:368} INFO - Filling up the DagBag from /opt/airflow/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_operator.py
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 2 | succeeded: 0 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-02 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-03 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   Running <TaskInstance: spark_pi.spark_pi_submit 2020-04-03T00:00:00+00:00 [None]> on host 01a7f91cd980
   Running <TaskInstance: spark_pi.spark_pi_submit 2020-04-02T00:00:00+00:00 [None]> on host 01a7f91cd980
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 2 | succeeded: 0 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-02 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies not met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-03 00:00:00+00:00 [scheduled]>, dependency 'Trigger Rule' FAILED: Task's trigger rule 'all_success' requires all upstream tasks to have succeeded, but found 1 non-success(es). upstream_tasks_state={'total': 1, 'successes': 0, 'skipped': 0, 'failed': 0, 'upstream_failed': 0, 'done': 0}, upstream_task_ids={'spark_pi_submit'}
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 2 | succeeded: 2 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies all met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-02 00:00:00+00:00 [scheduled]>
   INFO  [airflow.executors.local_executor.LocalExecutor] Adding to queue: ['airflow', 'tasks', 'run', 'spark_pi', 'spark_pi_monitor', '2020-04-02T00:00:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/opt/airflow/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_operator.py', '--cfg-path', '/tmp/tmp6wk1u4rg']
   INFO  [airflow.models.taskinstance.TaskInstance] Dependencies all met for <TaskInstance: spark_pi.spark_pi_monitor 2020-04-03 00:00:00+00:00 [scheduled]>
   INFO  [airflow.executors.local_executor.LocalExecutor] Adding to queue: ['airflow', 'tasks', 'run', 'spark_pi', 'spark_pi_monitor', '2020-04-03T00:00:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/opt/airflow/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_operator.py', '--cfg-path', '/tmp/tmp4_2_oitk']
   INFO  [airflow.executors.local_executor.QueuedLocalWorker] QueuedLocalWorker running ['airflow', 'tasks', 'run', 'spark_pi', 'spark_pi_monitor', '2020-04-02T00:00:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/opt/airflow/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_operator.py', '--cfg-path', '/tmp/tmp6wk1u4rg']
   INFO  [airflow.executors.local_executor.QueuedLocalWorker] QueuedLocalWorker running ['airflow', 'tasks', 'run', 'spark_pi', 'spark_pi_monitor', '2020-04-03T00:00:00+00:00', '--local', '--pool', 'default_pool', '--subdir', '/opt/airflow/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_operator.py', '--cfg-path', '/tmp/tmp4_2_oitk']
   WARNI [airflow.jobs.backfill_job.BackfillJob] ('spark_pi', 'spark_pi_submit', datetime.datetime(2020, 4, 2, 0, 0, tzinfo=<TimezoneInfo [UTC, GMT, +00:00:00, STD]>), 1) state success not in running=dict_values([<TaskInstance: spark_pi.spark_pi_monitor 2020-04-02 00:00:00+00:00 [queued]>, <TaskInstance: spark_pi.spark_pi_monitor 2020-04-03 00:00:00+00:00 [queued]>])
   WARNI [airflow.jobs.backfill_job.BackfillJob] ('spark_pi', 'spark_pi_submit', datetime.datetime(2020, 4, 3, 0, 0, tzinfo=<TimezoneInfo [UTC, GMT, +00:00:00, STD]>), 1) state success not in running=dict_values([<TaskInstance: spark_pi.spark_pi_monitor 2020-04-02 00:00:00+00:00 [queued]>, <TaskInstance: spark_pi.spark_pi_monitor 2020-04-03 00:00:00+00:00 [queued]>])
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   [2020-04-03 05:41:58,807] {dagbag.py:368} INFO - Filling up the DagBag from /opt/airflow/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_operator.py
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   [2020-04-03 05:41:59,181] {dagbag.py:368} INFO - Filling up the DagBag from /opt/airflow/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes_operator.py
   Running <TaskInstance: spark_pi.spark_pi_monitor 2020-04-02T00:00:00+00:00 [None]> on host 01a7f91cd980
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   Running <TaskInstance: spark_pi.spark_pi_monitor 2020-04-03T00:00:00+00:00 [None]> on host 01a7f91cd980
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 0 of 2 | tasks waiting: 0 | succeeded: 2 | running: 2 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.models.dagrun.DagRun] Marking run <DagRun spark_pi @ 2020-04-02 00:00:00+00:00: backfill__2020-04-02T00:00:00+00:00, externally triggered: False> successful
   INFO  [airflow.models.dagrun.DagRun] Marking run <DagRun spark_pi @ 2020-04-03 00:00:00+00:00: backfill__2020-04-03T00:00:00+00:00, externally triggered: False> successful
   INFO  [airflow.jobs.backfill_job.BackfillJob] [backfill progress] | finished run 2 of 2 | tasks waiting: 0 | succeeded: 4 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 0
   INFO  [airflow.jobs.backfill_job.BackfillJob] Backfill done. Exiting.
   
   Saving all log files to /root/airflow/logs/previous_runs/2020-04-03_05_47_02
   
   customresourcedefinition.apiextensions.k8s.io "sparkapplications.sparkoperator.k8s.io" deleted
   customresourcedefinition.apiextensions.k8s.io "scheduledsparkapplications.sparkoperator.k8s.io" deleted
   namespace "spark-operator" deleted
   serviceaccount "sparkoperator" deleted
   clusterrole.rbac.authorization.k8s.io "sparkoperator" deleted
   clusterrolebinding.rbac.authorization.k8s.io "sparkoperator" deleted
   clusterrole.rbac.authorization.k8s.io "sparkoperator-aggregate-to-admin" deleted
   serviceaccount "spark" deleted
   role.rbac.authorization.k8s.io "spark-role" deleted
   rolebinding.rbac.authorization.k8s.io "spark-role-binding" deleted
   PASSED
   
   ====================================================================================================================================================================== 1 passed in 361.89s (0:06:01) =======================================================================================================================================================================
   ```
   Everything works as expected.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] mik-laj commented on a change in pull request #7875: add spark_kubernetes system test

Posted by GitBox <gi...@apache.org>.

mik-laj commented on a change in pull request #7875: add spark_kubernetes system test
URL: https://github.com/apache/airflow/pull/7875#discussion_r398092115
 
 

 ##########
 File path: tests/providers/cncf/kubernetes/operators/test_spark_kubernetes_system.py
 ##########
 @@ -0,0 +1,72 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import os
+import subprocess
+
+import pytest
+
+from airflow.models import Connection
+from airflow.utils import db
+from tests.test_utils import AIRFLOW_MAIN_FOLDER
+from tests.test_utils.system_tests_class import SystemTest
+
+KUBERNETES_DAG_FOLDER = os.path.join(
+    AIRFLOW_MAIN_FOLDER, "airflow", "providers", "cncf", "kubernetes", "example_dags"
+)
+
+
+def _spark_operator_util(action):
 
 Review comment:
   I think this code can be greatly simplified. Thanks to my change, methods ``kubectl_apply_list`` and ``kubectl_delete_list`` can be used in other places and integrations in the future.
   ```python
   SPARK_VERSION = "v1beta2-1.1.1-2.4.5"
   
   MANIFEST_BASE_URL = \
       f'https://raw.githubusercontent.com/GoogleCloudPlatform/spark-on-k8s-operator/{SPARK_VERSION}/manifest/'
   
   SPARK_OPERATOR_MANIFESTS = [
       f"{MANIFEST_BASE_URL}crds/sparkoperator.k8s.io_sparkapplications.yaml",
       f"{MANIFEST_BASE_URL}crds/sparkoperator.k8s.io_scheduledsparkapplications.yaml",
       f"{MANIFEST_BASE_URL}spark-operator-rbac.yaml",
       f"{MANIFEST_BASE_URL}spark-operator.yaml",
       f"{MANIFEST_BASE_URL}spark-rbac.yaml"
   ]
   
   
   def kubectl_apply_list(manifests):
       for manifest in manifests:
           command = ['kubectl', 'apply', '-f', manifest]
           subprocess.run(command, check=True)
   
   
   def kubectl_delete_list(manifests):
       for manifest in manifests:
           command = ['kubectl', 'delete', '--ignore-not-found', '-f', manifest]
           subprocess.run(command, check=True)
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] roitvt commented on issue #7875: add spark_kubernetes system test

Posted by GitBox <gi...@apache.org>.

roitvt commented on issue #7875: add spark_kubernetes system test
URL: https://github.com/apache/airflow/pull/7875#issuecomment-604103872
 
 
   @potiuk it's working on 1.10 I checked it with breeze against 1.10.9

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7875: add spark_kubernetes system test

Posted by GitBox <gi...@apache.org>.

codecov-io edited a comment on issue #7875: add spark_kubernetes system test
URL: https://github.com/apache/airflow/pull/7875#issuecomment-606100069
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7875?src=pr&el=h1) Report
   > Merging [#7875](https://codecov.io/gh/apache/airflow/pull/7875?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6018532d21796e04fdf47a77f36ae8308cc928ba&el=desc) will **decrease** coverage by `0.64%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7875/graphs/tree.svg?width=650&height=150&src=pr&token=WdLKlKHOAU)](https://codecov.io/gh/apache/airflow/pull/7875?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7875      +/-   ##
   ==========================================
   - Coverage   87.17%   86.52%   -0.65%     
   ==========================================
     Files         933      933              
     Lines       45290    45343      +53     
   ==========================================
   - Hits        39482    39235     -247     
   - Misses       5808     6108     +300     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7875?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/utils/db.py](https://codecov.io/gh/apache/airflow/pull/7875/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYi5weQ==) | `98.33% <100.00%> (+0.01%)` | :arrow_up: |
   | [...flow/providers/apache/cassandra/hooks/cassandra.py](https://codecov.io/gh/apache/airflow/pull/7875/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2Nhc3NhbmRyYS9ob29rcy9jYXNzYW5kcmEucHk=) | `21.25% <0.00%> (-72.50%)` | :arrow_down: |
   | [...w/providers/apache/hive/operators/mysql\_to\_hive.py](https://codecov.io/gh/apache/airflow/pull/7875/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvYXBhY2hlL2hpdmUvb3BlcmF0b3JzL215c3FsX3RvX2hpdmUucHk=) | `35.84% <0.00%> (-64.16%)` | :arrow_down: |
   | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/7875/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
   | [airflow/providers/postgres/operators/postgres.py](https://codecov.io/gh/apache/airflow/pull/7875/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcG9zdGdyZXMvb3BlcmF0b3JzL3Bvc3RncmVzLnB5) | `47.82% <0.00%> (-52.18%)` | :arrow_down: |
   | [airflow/providers/redis/operators/redis\_publish.py](https://codecov.io/gh/apache/airflow/pull/7875/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvcmVkaXMvb3BlcmF0b3JzL3JlZGlzX3B1Ymxpc2gucHk=) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
   | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/7875/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
   | [airflow/providers/mongo/sensors/mongo.py](https://codecov.io/gh/apache/airflow/pull/7875/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbW9uZ28vc2Vuc29ycy9tb25nby5weQ==) | `53.33% <0.00%> (-46.67%)` | :arrow_down: |
   | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/7875/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
   | [airflow/providers/mysql/operators/mysql.py](https://codecov.io/gh/apache/airflow/pull/7875/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvbXlzcWwvb3BlcmF0b3JzL215c3FsLnB5) | `55.00% <0.00%> (-45.00%)` | :arrow_down: |
   | ... and [16 more](https://codecov.io/gh/apache/airflow/pull/7875/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7875?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7875?src=pr&el=footer). Last update [6018532...e62710a](https://codecov.io/gh/apache/airflow/pull/7875?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services