[GitHub] [airflow] m1racoli commented on issue #22404: tempfile.TemporaryDirectory does not get deleted after task failure

[GitBox <gi...@apache.org>]

m1racoli commented on issue #22404:
URL: https://github.com/apache/airflow/issues/22404#issuecomment-1075163309


   Yeah, I know that when using a context manager the problem doesn't occur, as the context manager explicitly cleans up the directory upon completion.
   
   And yes, running
   
   ```python
   import os
   import tempfile
   
   tmpdir = tempfile.TemporaryDirectory()
   print(f"directory {tmpdir.name} created")
   assert os.path.exists(tmpdir.name)
   
   raise Exception("error!")
   ```
   
   directly in Python cleans up the directory (done by the finalizer as @potiuk said).
   
   So now I wonder what's different in Airflow. My judgement is based on the assumption that each task is executed in it's own process and therefore also it's own Python process. Thus a failing task's Python process should clean up the directory too (unless Airflow messes with the finalizers). Maybe the child Python process knows it's a child of a parent process and doesn't invoke the finalizer (pure speculation). 🤔 
   
   I'm not an expert of the Python runtime and I did not verify (yet) if this also occurs when running in a sub-process. I initially was hoping that someone on Airflow's side knows what's going on. :)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org