You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/10 16:06:20 UTC
[GitHub] [airflow] akizminet opened a new issue, #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
akizminet opened a new issue, #27593:
URL: https://github.com/apache/airflow/issues/27593
### Apache Airflow version
2.4.2
### What happened
If some dag has task cannot failed. Airflow Scheduler run and crash with exception:
```
[2022-11-10T15:29:59.886+0000] {scheduler_job.py:1526} ERROR - Detected zombie job: {'full_filepath': '/opt/airflow/dags/path.py', 'processor_subdir': '/opt/airflow/dags', 'msg': "{'DAG Id': 'dag_id', 'Task Id': 'taskid', 'Run Id': 'manual__2022-11-10T10:21:25.330307+00:00', 'Hostname': 'hostname'}", 'simple_task_instance': <airflow.models.taskinstance.SimpleTaskInstance object at 0x7fde9c91dcd0>, 'is_failure_callback': True}
[2022-11-10T15:29:59.887+0000] {scheduler_job.py:763} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 746, in _execute
self._run_scheduler_loop()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 878, in _run_scheduler_loop
next_event = timers.run(blocking=False)
File "/usr/local/lib/python3.7/sched.py", line 151, in run
action(*argument, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/event_scheduler.py", line 37, in repeat
action(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 75, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1527, in _find_zombies
self.executor.send_callback(request)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/base_executor.py", line 400, in send_callback
self.callback_sink.send(request)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 75, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/callbacks/database_callback_sink.py", line 34, in send
db_callback = DbCallbackRequest(callback=callback, priority_weight=10)
File "<string>", line 4, in __init__
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 480, in _initialize_instance
manager.dispatch.init_failure(self, args, kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
with_traceback=exc_tb,
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 477, in _initialize_instance
return manager.original_init(*mixed[1:], **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/db_callback_request.py", line 46, in __init__
self.callback_data = callback.to_json()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/callbacks/callback_requests.py", line 89, in to_json
return json.dumps(dict_obj)
File "/usr/local/lib/python3.7/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/local/lib/python3.7/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type V1Pod is not JSON serializable
```
### What you think should happen instead
DbCallbackRequest should do to_json successfully
### How to reproduce
Start airflow with KubernetesExecutor
Make zombie task.
### Operating System
docker.io/apache/airflow:2.4.2
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310902280
This is likely related to #24356; with this patch, the scheduler now fully loads V1Pod instances in the process. cc @dstandish
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310972983
fix forthcoming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] akizminet commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
akizminet commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310568505
Thank you for your quick reply @potiuk . I confirm `airflow dags reserialize` doesn't fix the problem. This is a clean setup. I migrated dag from 2.2.4 to 2.4.2 for testing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310921789
can you post dag code? minimal reproducible dag please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] KulykDmytro commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
KulykDmytro commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1343250768
linking #28227
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310527465
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eTopchik commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
eTopchik commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310562087
Having same problem ` airflow dags reserialize` ain't helping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] KulykDmytro commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
KulykDmytro commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1329212780
@potiuk pls reonen due to 2.5.0rc2 caused
```
AttributeError: 'V1PodSpec' object has no attribute '_ephemeral_containers'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1329558713
actually, @KulykDmytro it might be best to just create a new issue with your bug report.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1329796733
Yes. Please provide the details. Asking to "reopen" without providing any more details is not really helpful in diagnosing and fixing an issue.
Already closed issue has no chance to be solved if there is not enough details for a new appearance (likely different root cause) to be able to diagnose and reproduce it - so just asking anyone to reopen an issue without details that would help to diagnose it is counterproductive - makes no difference of whether it will be solved or not.
On the other hand - providing useful information, stack-traces, circumstances and results of your investigation, actually enormously help in any attempt to try to investigate and fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eTopchik commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
eTopchik commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1320132449
Will it come with new release or i just need rebuild my image?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1329476846
@KulykDmytro hi please provide more information. attach traceback and ideally minimal dag for repro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310546924
Did you try `airflow dags reserialize` ? https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#reserialize
Can you check if it fixes your problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310961111
ok here's repro code:
```python
from __future__ import annotations
from kubernetes.client import models as k8s
from airflow.callbacks.callback_requests import TaskCallbackRequest
from airflow.models import TaskInstance
from airflow.models.taskinstance import SimpleTaskInstance
from airflow.operators.bash import BashOperator
op = BashOperator(task_id="hi", executor_config={"pod_override": k8s.V1Pod()}, bash_command="hi")
ti = TaskInstance(task=op)
s = SimpleTaskInstance.from_ti(ti)
TaskCallbackRequest("hi", s).to_json()
```
i don't think it's caused by that PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1320432981
It will be in next release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
URL: https://github.com/apache/airflow/issues/27593
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org