You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/12/08 15:44:32 UTC
[GitHub] [airflow] KulykDmytro opened a new issue, #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'
KulykDmytro opened a new issue, #28227:
URL: https://github.com/apache/airflow/issues/28227
### Apache Airflow version
2.5.0
### What happened
After upgrade 2.2.5 -> 2.5.0 scheduler failing with error:
```
AttributeError: 'V1PodSpec' object has no attribute '_ephemeral_containers'
```
tried with no luck:
```
airflow dags reserialize
```
### What you think should happen instead
_No response_
### How to reproduce
_No response_
### Operating System
Debian 11 (bullseye)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
AWS EKS
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #28227:
URL: https://github.com/apache/airflow/issues/28227#issuecomment-1357099778
The problem is that when you take a kubernetes object created from one lib version, and pickle it, then when you unpickle within a python env with a diff kubernetes lib version, you may get an object that will not work properly, and may fail when repr is called.
This first appeared with executor_config pod_override pod objects. We also saw it in db callback request objects.
The solution was to run through airflow's serde before / after pickling (executor config is stored as pickle object)
You were are on 2.2.5. So your executor_config objects are raw pickle objects. You also had an older version of k8s. Now you have a new k8s version and the objects are getting unpickled and blowing up.
Our solution makes things stable on a go forward basis but doesn't go back and "fix" the executor configs pickled with the old k8s version (we can't practically do so because we can't unpickle them properly in the new k8s version). I mean, probably we could do some immense hackery to fix the issue but it's really a k8s lib problem, just an unfortunate consequence of the way that the kubernetes objects behave with pickle, and in practice this should only appear for a few TI's shortly after upgrade.
You may try clearing the offending TI / nulling out the executor config.
But... i will take another look at the report / stacktrace to see if the scheduler was actually blowing up because, it shouldn't be doing that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on issue #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'
Posted by GitBox <gi...@apache.org>.
dstandish commented on issue #28227:
URL: https://github.com/apache/airflow/issues/28227#issuecomment-1357235927
Ok so... maybe there is a solution that is not _quite_ immense hackery:
https://github.com/apache/airflow/pull/28454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] 1984man commented on issue #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'
Posted by GitBox <gi...@apache.org>.
1984man commented on issue #28227:
URL: https://github.com/apache/airflow/issues/28227#issuecomment-1362675135
I have same issue from 2.1.2 to 2.4.1 (I use airflow helm chart version 1.7.0)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28227:
URL: https://github.com/apache/airflow/issues/28227#issuecomment-1356887060
Also @KulykDmytro - could you please explain the exact deplouyment you have - which versionsm, what kind of chart or any other deployment you have, how exactly did you upgrade (did you use constraints?). For me this might also be caused by mis-configuration of the k8s version you have and k8s client libraries you have. It almost look like you have an old client library while you are using newer k8S.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28227:
URL: https://github.com/apache/airflow/issues/28227#issuecomment-1356885469
Did you try `airflow dags reserialize` command @KulykDmytro after the upgrade?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #28227:
URL: https://github.com/apache/airflow/issues/28227#issuecomment-1356946147
This looks like an issue in the Kubernetes package to me, from what I can tell there’s nothing out of the ordinary from Airflow’s perspective. I suspect the upgrade is not done correctly and the Kubernetes installation is broken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28227:
URL: https://github.com/apache/airflow/issues/28227#issuecomment-1356886524
Ah I see you seeem to do run it. @dstandish @jedcunningham - does it ring a bell ?
Did you have anything specfic in your upgrade also around the k8s version change during migration @KulykDmytro ? Ephemeral containers seems to be a new feature of K8S so maybe this somehow impacts the problem.
Did you also run "reserialize" after or before the migration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish closed issue #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'
Posted by GitBox <gi...@apache.org>.
dstandish closed issue #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'
URL: https://github.com/apache/airflow/issues/28227
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org