You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/18 08:43:45 UTC

[GitHub] [airflow] m1racoli opened a new issue #22357: tempfile.TemporaryDirectory does not get deleted after task failure

m1racoli opened a new issue #22357:
URL: https://github.com/apache/airflow/issues/22357


   ### Apache Airflow version
   
   2.2.4 (latest released)
   
   ### What happened
   
   When creating a temporary directory with `tempfile.TemporaryDirectory()` and then failing a task, the corresponding directory does not get deleted.
   
   This happens in Airflow on Astronomer as well as locally in for `astro dev` setups for LocalExecutor and CeleryExecutor.
   
   ### What you think should happen instead
   
   As in normal Python environments, the directory should get cleaned up, even in the case of a raised exception.
   
   ### How to reproduce
   
   Running this DAG will leave a temporary directory in the corresponding location:
   
   ```python
   import os
   import tempfile
   
   from airflow.decorators import dag, task
   from airflow.utils.dates import days_ago
   
   
   class MyException(Exception):
       pass
   
   
   @task
   def run():
       tmpdir = tempfile.TemporaryDirectory()
       print(f"directory {tmpdir.name} created")
       assert os.path.exists(tmpdir.name)
   
       raise MyException("error!")
   
   
   @dag(start_date=days_ago(1))
   def tempfile_test():
       run()
   
   
   _ = tempfile_test()
   ```
   
   
   ### Operating System
   
   Debian (Astronomer Airflow Docker image)
   
   ### Versions of Apache Airflow Providers
   
   ```
   apache-airflow-providers-amazon==1!3.0.0
   apache-airflow-providers-cncf-kubernetes==1!3.0.2
   apache-airflow-providers-elasticsearch==1!2.2.0
   apache-airflow-providers-ftp==1!2.0.1
   apache-airflow-providers-google==1!6.4.0
   apache-airflow-providers-http==1!2.0.3
   apache-airflow-providers-imap==1!2.2.0
   apache-airflow-providers-microsoft-azure==1!3.6.0
   apache-airflow-providers-mysql==1!2.2.0
   apache-airflow-providers-postgres==1!3.0.0
   apache-airflow-providers-redis==1!2.0.1
   apache-airflow-providers-slack==1!4.2.0
   apache-airflow-providers-sqlite==1!2.1.0
   apache-airflow-providers-ssh==1!2.4.0
   ```
   
   ### Deployment
   
   Astronomer
   
   ### Deployment details
   
   GKE, vanilla `astro dev`, LocalExecutor and CeleryExecutor.
   
   ### Anything else
   
   Always
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #22357: tempfile.TemporaryDirectory does not get deleted after task failure

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #22357:
URL: https://github.com/apache/airflow/issues/22357#issuecomment-1072421463


   Could you please try the code below and see if the problem persists?
   ```python
       with tempfile.TemporaryDirectory() as tmpdir
             print(f"directory {tmpdir.name} created")
             assert os.path.exists(tmpdir.name)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #22357: tempfile.TemporaryDirectory does not get deleted after task failure

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #22357:
URL: https://github.com/apache/airflow/issues/22357#issuecomment-1072421463


   Could you please try the code below and see if the problem persists?
   ```python
       with tempfile.TemporaryDirectory() as tmpdir
             print(f"directory {tmpdir.name} created")
             assert os.path.exists(tmpdir.name)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org