You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Stijn Verhaegen (JIRA)" <ji...@apache.org> on 2017/03/01 13:26:45 UTC
[jira] [Comment Edited] (AIRFLOW-860) Circular module dependency
prevents loading of custom executor plugin
[ https://issues.apache.org/jira/browse/AIRFLOW-860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890154#comment-15890154 ]
Stijn Verhaegen edited comment on AIRFLOW-860 at 3/1/17 1:25 PM:
-----------------------------------------------------------------
I have the exact same issue. made a plugin with a custom operator and executor. All goes well until you actually choose the custom executor in airflow.cfg, then the custom operator fails because "import airflow.models" fails.
I spit the operator and executor over two separate plugins: the executor always works, but when it is selected, the Operator fails.
EDIT: I use airflow 1.7.1.3 so the affected version should be updated.
was (Author: stverhae):
I have the exact same issue. made a plugin with a custom operator and executor. All goes well until you actually choose the custom executor in airflow.cfg, then the custom operator fails because "import airflow.models" fails.
I spit the operator and executor over two separate plugins: the executor always works, but when it is selected, the Operator fails.
> Circular module dependency prevents loading of custom executor plugin
> ---------------------------------------------------------------------
>
> Key: AIRFLOW-860
> URL: https://issues.apache.org/jira/browse/AIRFLOW-860
> Project: Apache Airflow
> Issue Type: Bug
> Components: executor, plugins
> Affects Versions: Airflow 1.8
> Environment: Linux RHEL 7.6, Python 2.7.13
> Reporter: Nick Ivanov
>
> Custom plugins cannot be loaded, which prevents airflow from running, due to apparent cyclic dependency in plugins_manager called in {{executors.\_\_init\_\_}} -- the top-level {{\_\_init\_\_}} attempts to load the default executor, which then goes back to plugins_manager etc.
> {noformat}
> File "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow", line 17, in <module>
> from airflow import configuration
> File "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/__init__.py", line 31, in <module>
> from airflow.models import DAG
> File "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/models.py", line 62, in <module>
> from airflow.executors import DEFAULT_EXECUTOR, LocalExecutor
> File "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/executors/__init__.py", line 55, in <module>
> raise AirflowException("Executor {0} not supported.".format(_EXECUTOR))
> airflow.exceptions.AirflowException: Executor TestExecutor not supported.
> {noformat}
> While attempting {{from airflow.plugins_manager import executors_modules}} cycles right back where it came from:
> {noformat}
> >>> from airflow.plugins_manager import executor_modules
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/__init__.py", line 31, in <module>
> from airflow.models import DAG
> File "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/models.py", line 62, in <module>
> from airflow.executors import DEFAULT_EXECUTOR, LocalExecutor
> File "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/executors/__init__.py", line 55, in <module>
> raise AirflowException("Executor {0} not supported.".format(_EXECUTOR))
> airflow.exceptions.AirflowException: Executor LSFExecutor not supported.
> {noformat}
> To reproduce:
> # Create a custom executor, e.g. by copying {{local_executor.py}} into {{$AIRFLOW_HOME/plugins/test_executor.py}} and replacing occurrences of "Local" with "Test" in the code, which will produce {{TestExecutor}}
> # Update {{$AIRFLOW_HOME/airflow.cfg}} by setting in the {{\[core\]}} section: {{executor = TestExecutor}}
> # Run any airflow command, e.g. "airflow version"
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)