[GitHub] [airflow] dstandish commented on issue #28227: Scheduler error: 'V1PodSpec' object has no attribute '_ephemeral_containers'

[GitBox <gi...@apache.org>]

dstandish commented on issue #28227:
URL: https://github.com/apache/airflow/issues/28227#issuecomment-1357099778

   The problem is that when you take a kubernetes object created from one lib version, and pickle it, then when you unpickle within a python env with a diff kubernetes lib version, you may get an object that will not work properly, and may fail when repr is called.
   
   This first appeared with executor_config pod_override pod objects.  We also saw it in db callback request objects.
   
   The solution was to run through airflow's serde before  / after pickling (executor config is stored as pickle object)
   
   You were are on 2.2.5.  So your executor_config objects are raw pickle objects.  You also had an older version of k8s.  Now you have a new k8s version and the objects are getting unpickled and blowing up.  
   
   Our solution makes things stable on a go forward basis but doesn't go back and "fix" the executor configs pickled with the old k8s version (we can't practically do so because we can't unpickle them properly in the new k8s version).  I mean, probably we could do some immense hackery to fix the issue but it's really a k8s lib problem, just an unfortunate consequence of the way that the kubernetes objects behave with pickle, and in practice this should only appear for a few TI's shortly after upgrade.
   
   You may try clearing the offending TI / nulling out the executor config.  
   
   But... i will take another look at the report / stacktrace to see if the scheduler was actually blowing up because, it shouldn't be doing that.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org