You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "GergelyKalmar (via GitHub)" <gi...@apache.org> on 2024/07/31 09:31:09 UTC

[I] Cannot delete DAG with many runs [airflow]

GergelyKalmar opened a new issue, #41148:
URL: https://github.com/apache/airflow/issues/41148

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   2.8.1 (AWS MWAA)
   
   ### What happened?
   
   I tried to delete a DAG with many runs (14000+). After about 4 minutes I receive "xxx.yyy.airflow.amazonaws.com didn’t send any data. ERR_EMPTY_RESPONSE", and the DAG is not deleted.
   
   ### What you think should happen instead?
   
   The DAG should have been deleted.
   
   ### How to reproduce
   
   Create a DAG with 14000 runs and try to delete it.
   
   ### Operating System
   
   AWS MWAA
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Amazon (AWS) MWAA
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Cannot delete DAG with many runs [airflow]

Posted by "GergelyKalmar (via GitHub)" <gi...@apache.org>.
GergelyKalmar commented on issue #41148:
URL: https://github.com/apache/airflow/issues/41148#issuecomment-2262808513

   > yes this is a bug but just to clarify the delete partially works. It deletes some of the records before raising the error, so every time you click on the button it will reduce the amount of records. I assume 2-3 times will delete all the records
   
   Are you sure? The DAG run instance numbers seems to be the same after the crash, could it be that the deletion is rolled back and so it does not execute partially? I've tried to re-run the deletion many times, and I saw no change.
   
   > DAG can not be deleted from the UI. the delete only removes the records associated with it (DagRun, TaskInstance...) if you want to remove dag completely then you need also to remove the .py file of the DAG and then delete the records (with the UI or API if the dag is not present)
   
   Yes, I'm aware of this, I meant the removal of the related records.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Cannot delete DAG with many runs [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #41148:
URL: https://github.com/apache/airflow/issues/41148#issuecomment-2260726141

   yes this is a bug but just to clarify the delete partially works.
   It deletes some of the records before raising the error, so every time you click on the button it will reduce the amount of records. I assume 2-3 times will delete all the records
   
   > and the DAG is not deleted.
   
   DAG can not be deleted from the UI. the delete only removes the records associated with it (DagRun, TaskInstance...)
   if you want to remove dag completely then you need also to remove the .py file of the DAG and then delete the records (with the UI or API if the dag is not present)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Cannot delete DAG with many runs [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #41148:
URL: https://github.com/apache/airflow/issues/41148#issuecomment-2260729083

   tagged with DB label as this is probably related to inefficient query to the DB.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Cannot delete DAG with many runs [airflow]

Posted by "omkar-foss (via GitHub)" <gi...@apache.org>.
omkar-foss commented on issue #41148:
URL: https://github.com/apache/airflow/issues/41148#issuecomment-2270491305

   >I tried to delete a DAG with many runs (14000+). After about 4 minutes I receive "xxx.yyy.airflow.amazonaws.com didn’t send any data. ERR_EMPTY_RESPONSE", and the DAG is not deleted.
   
   @GergelyKalmar the `ERR_EMPTY_RESPONSE` suggests that the connection is closed before the DAG deletion completed, particularly because the  of 14k+ runs deletion operation [in this function](https://github.com/apache/airflow/blob/main/airflow/api/common/delete_dag.py#L42-L108) is taking too long. This is a bug as @eladkal pointed out, and when this bug gets fixed here in upstream Airflow, it may be a while before it reflects in downstream MWAA which you're using.
   
   Meanwhile, I'd suggest you try to delete the DAG via the Airflow REST API with an increased request timeout. I've written a small script for this ([Gist here](https://gist.github.com/omkar-foss/51fe2e79a8da628833412652e69f32d1)) by referring to the MWAA docs, please check it out. You can tweak the script as per your need and increase the timeout beyond 10 mins if the deletion takes longer and fails. Hope this helps.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Cannot delete DAG with many runs [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #41148:
URL: https://github.com/apache/airflow/issues/41148#issuecomment-3261434677

   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Cannot delete DAG with many runs [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #41148: Cannot delete DAG with many runs
URL: https://github.com/apache/airflow/issues/41148


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Cannot delete DAG with many runs [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #41148:
URL: https://github.com/apache/airflow/issues/41148#issuecomment-3157656802

   This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org