[jira] [Assigned] (AIRFLOW-6003) Cyclic imports detected by pylint randomly

["Jarek Potiuk (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jarek Potiuk reassigned AIRFLOW-6003:
-------------------------------------

    Assignee: Jarek Potiuk

> Cyclic imports detected by pylint randomly
> ------------------------------------------
>
>                 Key: AIRFLOW-6003
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6003
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: ci
>    Affects Versions: 2.0.0
>            Reporter: Jarek Potiuk
>            Assignee: Jarek Potiuk
>            Priority: Major
>
> There are cyclic imports detected seemingly randomly by pylint checks when some of the PRs are run in CI:
> An example of such pylint problem: [https://travis-ci.org/apache/airflow/jobs/612096100?utm_medium=notification&utm_source=github_status]
>  
>  
> {code:java}
> ************* Module airflow.utils.log.json_formatter
> airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.pod_generator) (cyclic-import)
> airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow -> airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.pod_launcher) (cyclic-import)
> airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.worker_configuration -> airflow.kubernetes.pod_generator) (cyclic-import)
>  
> {code}
>  
> The problem is that airflow's __init__ contains a few convenience imports (AirflowException, Executors etc.) but it also imports a number of packages (for example kubernetes_executor) that in turn import the airflow package objects - for example airflow.Executor. This leads to cyclic imports if you import first the executors before airflow. Similar problem happens with executor.__init__.py containing class "Executors" imported by all executors but at the same time some of the executors (for example KubernetesExecutor) import the very same Executor class. 
>   
>  This might happen in pylint checks in pre-commit because they split a number of files they process between the multiple threads you have at your machine and sometimes it might happen that the files are imported in different order.
>   
>  As a solution, the executors "list" should be moved to a separate module and a number of classes (such as AirflowException) should be moved out from __init__ package and imported from where they belong (so for example AirflowException should be universally imported from airflow.exceptions).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)