You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/20 05:29:39 UTC
[GitHub] [airflow] MatrixManAtYrService opened a new issue, #23107: Mapped KubernetesPodOperator "fails" but UI shows it is as still running
MatrixManAtYrService opened a new issue, #23107:
URL: https://github.com/apache/airflow/issues/23107
### Apache Airflow version
2.3.0b1 (pre-release)
### What happened
This dag has a problem. The `name` kwarg is missing from one of the mapped instances.
```python3
from datetime import datetime
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
KubernetesPodOperator,
)
from airflow.configuration import conf
namespace = conf.get("kubernetes", "NAMESPACE")
with DAG(
dag_id="kpo_mapped",
start_date=datetime(1970, 1, 1),
schedule_interval=None,
) as dag:
KubernetesPodOperator(
task_id="cowsay_static_named",
name="cowsay_statc",
namespace=namespace,
image="docker.io/rancher/cowsay",
cmds=["cowsay"],
arguments=["moo"],
)
KubernetesPodOperator.partial(
task_id="cowsay_mapped",
# name="cowsay_mapped", # required field missing
image="docker.io/rancher/cowsay",
namespace=namespace,
cmds=["cowsay"],
).expand(arguments=[["mooooove"], ["cow"], ["get out the way"]])
KubernetesPodOperator.partial(
task_id="cowsay_mapped_named",
name="cowsay_mapped",
namespace=namespace,
image="docker.io/rancher/cowsay",
cmds=["cowsay"],
).expand(arguments=[["mooooove"], ["cow"], ["get out the way"]])
```
If you omit that field in an unmapped task, you get a dag parse error, which is appropriate. But omitting it from the mapped task gives you this runtime error in the task logs:
```
[2022-04-20, 05:11:02 UTC] {standard_task_runner.py:52} INFO - Started process 60 to run task
[2022-04-20, 05:11:02 UTC] {standard_task_runner.py:79} INFO - Running: ['airflow', 'tasks', 'run', 'kpo_mapped', 'cowsay_mapped', 'manual__2022-04-20T05:11:01+00:00', '--job-id', '12', '--raw', '--subdir', 'DAGS_FOLDER/dags/taskmap/kpo_mapped.py', '--cfg-path', '/tmp/tmp_g3sj496', '--map-index', '0', '--error-file', '/tmp/tmp2_313wxj']
[2022-04-20, 05:11:02 UTC] {standard_task_runner.py:80} INFO - Job 12: Subtask cowsay_mapped
[2022-04-20, 05:11:02 UTC] {task_command.py:369} INFO - Running <TaskInstance: kpo_mapped.cowsay_mapped manual__2022-04-20T05:11:01+00:00 map_index=0 [running]> on host airflow-worker-65f9fd9d5b-vpgnk
[2022-04-20, 05:11:02 UTC] {taskinstance.py:1863} WARNING - We expected to get frame set in local storage but it was not. Please report this as an issue with full logs at https://github.com/apache/airflow/issues/new
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1440, in _run_raw_task
self._execute_task_with_callbacks(context, test_mode)
File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1544, in _execute_task_with_callbacks
task_orig = self.render_templates(context=context)
File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 2210, in render_templates
rendered_task = self.task.render_template_fields(context)
File "/usr/local/lib/python3.9/site-packages/airflow/models/mappedoperator.py", line 722, in render_template_fields
unmapped_task = self.unmap(unmap_kwargs=kwargs)
File "/usr/local/lib/python3.9/site-packages/airflow/models/mappedoperator.py", line 508, in unmap
op = self.operator_class(**unmap_kwargs, _airflow_from_mapped=True)
File "/usr/local/lib/python3.9/site-packages/airflow/models/baseoperator.py", line 390, in apply_defaults
result = func(self, **kwargs, default_args=default_args)
File "/usr/local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 259, in __init__
self.name = self._set
_name(name)
File "/usr/local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 442, in _set_name
raise AirflowException("`name` is required unless `pod_template_file` or `full_pod_spec` is set")
airflow.exceptions.AirflowException: `name` is required unless `pod_template_file` or `full_pod_spec` is set
```
But rather than failing the task, Airflow just thinks that the task is still running:
<img width="833" alt="Screen Shot 2022-04-19 at 11 13 47 PM" src="https://user-images.githubusercontent.com/5834582/164156155-41986d3a-d171-4943-8443-a0fc3c542988.png">
### What you think should happen instead
Ideally this error would be surfaced when the dag is first parsed. If that's not possible, then it should fail the task completely (i.e. a red square should show up in the grid view).
### How to reproduce
Run the dag above
### Operating System
ubuntu (microk8s)
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes | 4.0.0
### Deployment
Astronomer
### Deployment details
Deployed via the astronomer airflow helm chart, values:
```
airflow:
airflowHome: /usr/local/airflow
defaultAirflowRepository: 172.28.11.191:30500/airflow
defaultAirflowTag: tb11c-inner-operator-expansion
env:
- name: AIRFLOW__CORE__DAGBAG_IMPORT_ERROR_TRACEBACK_DEPTH
value: '99'
executor: CeleryExecutor
gid: 50000
images:
airflow:
pullPolicy: Always
repository: 172.28.11.191:30500/airflow
flower:
pullPolicy: Always
pod_template:
pullPolicy: Always
logs:
persistence:
enabled: true
size: 2Gi
scheduler:
livenessProbe:
timeoutSeconds: 45
triggerer:
livenessProbe:
timeoutSeconds: 45
```
Image base: `quay.io/astronomer/ap-airflow-dev:main`
Airflow version: `2.3.0.dev20220414`
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed issue #23107: Mapped KubernetesPodOperator "fails" but UI shows it is as still running
Posted by GitBox <gi...@apache.org>.
ashb closed issue #23107: Mapped KubernetesPodOperator "fails" but UI shows it is as still running
URL: https://github.com/apache/airflow/issues/23107
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #23107: Mapped KubernetesPodOperator "fails" but UI shows it is as still running
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #23107:
URL: https://github.com/apache/airflow/issues/23107#issuecomment-1103575199
```
[2022-04-20, 05:11:02 UTC] {taskinstance.py:1863} WARNING - We expected to get frame set in local storage but it was not. Please report this as an issue with full logs at https://github.com/apache/airflow/issues/new
```
This line looks suspicious
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #23107: Mapped KubernetesPodOperator "fails" but UI shows it is as still running
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #23107:
URL: https://github.com/apache/airflow/issues/23107#issuecomment-1103574418
Curious, something is up with the zombie detection too:
```[2022-04-20 08:42:36,843] 331186 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1360}} WARNING - Failing (3) jobs without heartbeat after 2022-04-20 07:37:36.841761+00:00
[2022-04-20 08:42:36,844] 331186 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1368}} ERROR - Detected zombie job: {'full_filepath': '/home/ash/code/airflow/airflow/kpodag.py', 'msg': 'Detected <TaskInstance: kpo_mapped.cowsay_mapped manual__2022-04-20T07:42:30.995741+00:00 map_index=2 [running]> as zombie', 'simple_task_instance': <airflow.models.taskinstance.SimpleTaskInstance object at 0x7f6088b5ebc0>, 'is_failure_callback': True}
[2022-04-20 08:42:36,844] 331186 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1368}} ERROR - Detected zombie job: {'full_filepath': '/home/ash/code/airflow/airflow/kpodag.py', 'msg': 'Detected <TaskInstance: kpo_mapped.cowsay_mapped manual__2022-04-20T07:42:30.995741+00:00 map_index=1 [running]> as zombie', 'simple_task_instance': <airflow.models.taskinstance.SimpleTaskInstance object at 0x7f6088c1cd60>, 'is_failure_callback': True}
[2022-04-20 08:42:36,844] 331186 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1368}} ERROR - Detected zombie job: {'full_filepath': '/home/ash/code/airflow/airflow/kpodag.py', 'msg': 'Detected <TaskInstance: kpo_mapped.cowsay_mapped manual__2022-04-20T07:42:30.995741+00:00 map_index=0 [running]> as zombie', 'simple_task_instance': <airflow.models.taskinstance.SimpleTaskInstance object at 0x7f6088b7e320>, 'is_failure_callback': True}
[2022-04-20 08:42:46,880] 331186 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1360}} WARNING - Failing (3) jobs without heartbeat after 2022-04-20 07:37:46.878339+00:00
[2022-04-20 08:42:46,881] 331186 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1368}} ERROR - Detected zombie job: {'full_filepath': '/home/ash/code/airflow/airflow/kpodag.py', 'msg': 'Detected <TaskInstance: kpo_mapped.cowsay_mapped manual__2022-04-20T07:42:30.995741+00:00 map_index=2 [running]> as zombie', 'simple_task_instance': <airflow.models.taskinstance.SimpleTaskInstance object at 0x7f6088b8bd30>, 'is_failure_callback': True}
[2022-04-20 08:42:46,881] 331186 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1368}} ERROR - Detected zombie job: {'full_filepath': '/home/ash/code/airflow/airflow/kpodag.py', 'msg': 'Detected <TaskInstance: kpo_mapped.cowsay_mapped manual__2022-04-20T07:42:30.995741+00:00 map_index=1 [running]> as zombie', 'simple_task_instance': <airflow.models.taskinstance.SimpleTaskInstance object at 0x7f6088b7b8e0>, 'is_failure_callback': True}
[2022-04-20 08:42:46,881] 331186 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1368}} ERROR - Detected zombie job: {'full_filepath': '/home/ash/code/airflow/airflow/kpodag.py', 'msg': 'Detected <TaskInstance: kpo_mapped.cowsay_mapped manual__2022-04-20T07:42:30.995741+00:00 map_index=0 [running]> as zombie', 'simple_task_instance': <airflow.models.taskinstance.SimpleTaskInstance object at 0x7f6088b5ebc0>, 'is_failure_callback': True}
[2022-04-20 08:42:49 +0100] [331200] [INFO] Handling signal: winch
[2022-04-20 08:42:56,915] 331186 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1360}} WARNING - Failing (3) jobs without heartbeat after 2022-04-20 07:37:56.912725+00:00
[2022-04-20 08:42:56,915] 331186 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1368}} ERROR - Detected zombie job: {'full_filepath': '/home/ash/code/airflow/airflow/kpodag.py', 'msg': 'Detected <TaskInstance: kpo_mapped.cowsay_mapped manual__2022-04-20T07:42:30.995741+00:00 map_index=2 [running]> as zombie', 'simple_task_instance': <airflow.models.taskinstance.SimpleTaskInstance object at 0x7f6088b5d6f0>, 'is_failure_callback': True}
[2022-04-20 08:42:56,915] 331186 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1368}} ERROR - Detected zombie job: {'full_filepath': '/home/ash/code/airflow/airflow/kpodag.py', 'msg': 'Detected <TaskInstance: kpo_mapped.cowsay_mapped manual__2022-04-20T07:42:30.995741+00:00 map_index=1 [running]> as zombie', 'simple_task_instance': <airflow.models.taskinstance.SimpleTaskInstance object at 0x7f6088b7e320>, 'is_failure_callback': True}
[2022-04-20 08:42:56,915] 331186 MainProcess {{airflow.jobs.scheduler_job.SchedulerJob scheduler_job.py:1368}} ERROR - Detected zombie job: {'full_filepath': '/home/ash/code/airflow/airflow/kpodag.py', 'msg': 'Detected <TaskInstance: kpo_mapped.cowsay_mapped manual__2022-04-20T07:42:30.995741+00:00 map_index=0 [running]> as zombie', 'simple_task_instance': <airflow.models.taskinstance.SimpleTaskInstance object at 0x7f6088b7f460>, 'is_failure_callback': True}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #23107: Mapped KubernetesPodOperator "fails" but UI shows it is as still running
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #23107:
URL: https://github.com/apache/airflow/issues/23107#issuecomment-1103542448
Yeah, two things we can do here:
1. Improve the validation for KPO when mapped (framework is in place already)
2. (This one is most important) Find out why the task isn't being detected as failed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #23107: Mapped KubernetesPodOperator "fails" but UI shows it is as still running
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #23107:
URL: https://github.com/apache/airflow/issues/23107#issuecomment-1103598797
@uranusjr That error happens if the excpetion occurs "outside" of user task code (before or after). Not a problem here, but we should fix it, yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org