You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/13 12:48:24 UTC
[GitHub] [airflow] mpolatcan opened a new issue #12341: KubernetesExecutor single task run error: Only works with the Celery or Kubernetes executors, sorry
mpolatcan opened a new issue #12341:
URL: https://github.com/apache/airflow/issues/12341
Hi Airflow community ✋ I have detected bug on Airflow webserver. I have deployed Airflow cluster that uses Kubernetes executor, runs on production AWS EKS cluster. All configurations are correct and runs successfully tasks on DAGs. But when I want to run specific task on DAG, that throws error like that:
But, I have checked my KubernetesExecutor configurations all of them are correct. So, I have digged error on source code of Airflow webserver at tag 1.10.12 which I have deployed version of Airflow. And found that some import errors are ignored for KubernetesExecutor on file https://github.com/apache/airflow/blob/1.10.12/airflow/www/views.py at line 1152
```python
executor = ExecutorLoader.get_default_executor()
valid_celery_config = False
valid_kubernetes_config = False
try:
from airflow.executors.celery_executor import CeleryExecutor # noqa
valid_celery_config = isinstance(executor, CeleryExecutor)
except ImportError:
pass
try:
from airflow.contrib.executors.kubernetes_executor import KubernetesExecutor # noqa
valid_kubernetes_config = isinstance(executor, KubernetesExecutor)
except ImportError:
pass
if not valid_celery_config and not valid_kubernetes_config:
flash("Only works with the Celery or Kubernetes executors, sorry", "error")
return redirect(origin)
```
In there, bug in this code snippet
```python
try:
from airflow.contrib.executors.kubernetes_executor import KubernetesExecutor # BUG IS HERE !!!
valid_kubernetes_config = isinstance(executor, KubernetesExecutor)
except ImportError:
pass
```
Import statement raises error KubernetesExecutor not found in this module because when you look that module at tag 1.10.12 https://github.com/apache/airflow/blob/1.10.12/airflow/contrib/executors/kubernetes_executor.py, there is no KubernetesExecutor class, only includes kubernetes_executor module which it is KubernetesExecutor's location. But when you use that you need to change the code like that:
```python
try:
from airflow.contrib.executors.kubernetes_executor import kubernetes_executor
valid_kubernetes_config = isinstance(executor, kubernetes_executor.KubernetesExecutor)
except ImportError:
pass
```
or:
```python
try:
from airflow.executors.kubernetes_executor import KubernetesExecutor
valid_kubernetes_config = isinstance(executor, KubernetesExecutor)
except ImportError:
pass
```
This error block user to run single task with KubernetesExecutor without run whole DAG.
Thanks, have a nice day 👍 😄
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #12341: KubernetesExecutor single task run error: Only works with the Celery or Kubernetes executors, sorry
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #12341:
URL: https://github.com/apache/airflow/issues/12341#issuecomment-731573939
This is already fixed in 1.10.13rc1 : https://github.com/apache/airflow/commit/25c16e87536b4c12acf1ae19bfa3ce1e3f5d6171
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #12341: KubernetesExecutor single task run error: Only works with the Celery or Kubernetes executors, sorry
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #12341:
URL: https://github.com/apache/airflow/issues/12341#issuecomment-726746060
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] gfeldman edited a comment on issue #12341: KubernetesExecutor single task run error: Only works with the Celery or Kubernetes executors, sorry
Posted by GitBox <gi...@apache.org>.
gfeldman edited a comment on issue #12341:
URL: https://github.com/apache/airflow/issues/12341#issuecomment-731509781
Confirming @mpolatcan diagnosis and that changing the import from `airflow.contrib.executors.kubernetes_executor` to `airflow.executors.kubernetes_executor` in `airflow/www/views.py` and `airflow/www_rbac/views.py` resolves the issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #12341: KubernetesExecutor single task run error: Only works with the Celery or Kubernetes executors, sorry
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #12341:
URL: https://github.com/apache/airflow/issues/12341#issuecomment-731568268
CC @dimberman @kaxil , should we release this in 1.10.13?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #12341: KubernetesExecutor single task run error: Only works with the Celery or Kubernetes executors, sorry
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #12341:
URL: https://github.com/apache/airflow/issues/12341
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] gfeldman commented on issue #12341: KubernetesExecutor single task run error: Only works with the Celery or Kubernetes executors, sorry
Posted by GitBox <gi...@apache.org>.
gfeldman commented on issue #12341:
URL: https://github.com/apache/airflow/issues/12341#issuecomment-731509781
Confirming @mpolatcan diagnosis and that changing the import under `airflow/www/views.py` and `airflow/www_rbac/views.py` resolves the issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mpolatcan commented on issue #12341: KubernetesExecutor single task run error: Only works with the Celery or Kubernetes executors, sorry
Posted by GitBox <gi...@apache.org>.
mpolatcan commented on issue #12341:
URL: https://github.com/apache/airflow/issues/12341#issuecomment-731612982
Thanks @kaxil, I am awaiting Airflow version 1.10.13 😄 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org