[jira] [Created] (AIRFLOW-6003) Cyclic imports detected by pylint randomly

["Jarek Potiuk (Jira)" <ji...@apache.org>]

Jarek Potiuk created AIRFLOW-6003:
-------------------------------------

             Summary: Cyclic imports detected by pylint randomly
                 Key: AIRFLOW-6003
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6003
             Project: Apache Airflow
          Issue Type: Bug
          Components: ci
    Affects Versions: 2.0.0
            Reporter: Jarek Potiuk


There are cyclic imports detected seemingly randomly by pylint checks when some of the PRs are run in CI:

An example of such pylint problem: [https://travis-ci.org/apache/airflow/jobs/612096100?utm_medium=notification&utm_source=github_status]

 
************* Module airflow.utils.log.json_formatter 
1492airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.pod_generator) (cyclic-import)
1493airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow -> airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.pod_launcher) (cyclic-import)
1494airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.worker_configuration -> airflow.kubernetes.pod_generator) (cyclic-import)
 
The problem is that airflow's __init__ contains a few convenience imports (AirflowException, Executors etc.) but it also imports a number of packages (for example kubernetes_executor) that in turn import the airflow package objects - for example airflow.Executor. This leads to cyclic imports if you import first the executors before airflow. Similar problem happens with executor.__init__.py containing class "Executors" imported by all executors but at the same time some of the executors (for example KubernetesExecutor) import the very same Executor class. 
 
This might happen in pylint checks in pre-commit because they split a number of files they process between the multiple threads you have at your machine and sometimes it might happen that the files are imported in different order.
 
As a solution, the executors "list" should be moved to a separate module and a number of classes (such as AirflowException) should be moved out from __init__ package and imported from where they belong (so for example AirflowException should be universally imported from airflow.exceptions). 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)