You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/11/10 16:06:20 UTC

[GitHub] [airflow] akizminet opened a new issue, #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

akizminet opened a new issue, #27593:
URL: https://github.com/apache/airflow/issues/27593

   ### Apache Airflow version
   
   2.4.2
   
   ### What happened
   
   If some dag has task cannot failed. Airflow Scheduler run and crash with exception:
   
   ```
   [2022-11-10T15:29:59.886+0000] {scheduler_job.py:1526} ERROR - Detected zombie job: {'full_filepath': '/opt/airflow/dags/path.py', 'processor_subdir': '/opt/airflow/dags', 'msg': "{'DAG Id': 'dag_id', 'Task Id': 'taskid', 'Run Id': 'manual__2022-11-10T10:21:25.330307+00:00', 'Hostname': 'hostname'}", 'simple_task_instance': <airflow.models.taskinstance.SimpleTaskInstance object at 0x7fde9c91dcd0>, 'is_failure_callback': True}
   [2022-11-10T15:29:59.887+0000] {scheduler_job.py:763} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 746, in _execute
       self._run_scheduler_loop()
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 878, in _run_scheduler_loop
       next_event = timers.run(blocking=False)
     File "/usr/local/lib/python3.7/sched.py", line 151, in run
       action(*argument, **kwargs)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/event_scheduler.py", line 37, in repeat
       action(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 75, in wrapper
       return func(*args, session=session, **kwargs)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1527, in _find_zombies
       self.executor.send_callback(request)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/base_executor.py", line 400, in send_callback
       self.callback_sink.send(request)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 75, in wrapper
       return func(*args, session=session, **kwargs)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/callbacks/database_callback_sink.py", line 34, in send
       db_callback = DbCallbackRequest(callback=callback, priority_weight=10)
     File "<string>", line 4, in __init__
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 480, in _initialize_instance
       manager.dispatch.init_failure(self, args, kwargs)
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
       with_traceback=exc_tb,
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
       raise exception
     File "/home/airflow/.local/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 477, in _initialize_instance
       return manager.original_init(*mixed[1:], **kwargs)
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/db_callback_request.py", line 46, in __init__
       self.callback_data = callback.to_json()
     File "/home/airflow/.local/lib/python3.7/site-packages/airflow/callbacks/callback_requests.py", line 89, in to_json
       return json.dumps(dict_obj)
     File "/usr/local/lib/python3.7/json/__init__.py", line 231, in dumps
       return _default_encoder.encode(obj)
     File "/usr/local/lib/python3.7/json/encoder.py", line 199, in encode
       chunks = self.iterencode(o, _one_shot=True)
     File "/usr/local/lib/python3.7/json/encoder.py", line 257, in iterencode
       return _iterencode(o, 0)
     File "/usr/local/lib/python3.7/json/encoder.py", line 179, in default
       raise TypeError(f'Object of type {o.__class__.__name__} '
   TypeError: Object of type V1Pod is not JSON serializable
   ``` 
   
   ### What you think should happen instead
   
   DbCallbackRequest should do to_json successfully
   
   ### How to reproduce
   
   Start airflow with KubernetesExecutor
   Make zombie task.
   
   ### Operating System
   
   docker.io/apache/airflow:2.4.2
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310902280

   This is likely related to #24356; with this patch, the scheduler now fully loads V1Pod instances in the process. cc @dstandish 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310972983

   fix forthcoming


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] akizminet commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
akizminet commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310568505

   Thank you for your quick reply @potiuk . I confirm `airflow dags reserialize` doesn't fix the problem. This is a clean setup. I migrated dag from 2.2.4 to 2.4.2 for testing
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310921789

   can you post dag code?  minimal reproducible dag please.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] KulykDmytro commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
KulykDmytro commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1343250768

   linking #28227


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] boring-cyborg[bot] commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310527465

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eTopchik commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
eTopchik commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310562087

   Having same problem ` airflow dags reserialize` ain't helping 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] KulykDmytro commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
KulykDmytro commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1329212780

   @potiuk pls reonen due to 2.5.0rc2 caused
   ```
   AttributeError: 'V1PodSpec' object has no attribute '_ephemeral_containers'
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1329558713

   actually, @KulykDmytro it might be best to just create a new issue with your bug report.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1329796733

   Yes. Please provide the details. Asking to "reopen" without providing any more details is not really helpful in diagnosing and fixing an issue. 
   
   Already closed issue has no chance to be solved if there is not enough details for a new appearance (likely different root cause) to be able to diagnose and reproduce it - so just asking anyone to reopen an issue without details that would help to diagnose it is counterproductive -  makes no difference of whether it will be solved or not.
   
   On the other hand - providing useful information, stack-traces, circumstances and results of your investigation, actually enormously help in any attempt to try to investigate and fix it. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eTopchik commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
eTopchik commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1320132449

   Will it come with new release or i just need rebuild my image?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1329476846

   @KulykDmytro hi please provide more information.  attach traceback and ideally minimal dag for repro.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310546924

   Did you try `airflow dags reserialize` ?  https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#reserialize
   
   Can you check if it fixes your problem? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1310961111

   ok here's repro code: 
   ```python
   from __future__ import annotations
   
   from kubernetes.client import models as k8s
   
   from airflow.callbacks.callback_requests import TaskCallbackRequest
   from airflow.models import TaskInstance
   from airflow.models.taskinstance import SimpleTaskInstance
   from airflow.operators.bash import BashOperator
   
   op = BashOperator(task_id="hi", executor_config={"pod_override": k8s.V1Pod()}, bash_command="hi")
   ti = TaskInstance(task=op)
   
   s = SimpleTaskInstance.from_ti(ti)
   
   TaskCallbackRequest("hi", s).to_json()
   ```
   
   i don't think it's caused by that PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] dstandish commented on issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #27593:
URL: https://github.com/apache/airflow/issues/27593#issuecomment-1320432981

   It will be in next release


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #27593: Object of type V1Pod is not JSON serializable after detecting zombie jobs cause Scheduler CrashLoopBack
URL: https://github.com/apache/airflow/issues/27593


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org