You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dominikkedziak (via GitHub)" <gi...@apache.org> on 2025/05/28 13:06:03 UTC

[I] airflow db export-archived doesn't page results [airflow]

dominikkedziak opened a new issue, #51156:
URL: https://github.com/apache/airflow/issues/51156

   ### Apache Airflow version
   
   3.0.1
   
   ### If "Other Airflow 2 version" selected, which one?
   
   All 2.x
   
   ### What happened?
   
   While using [airflow db export-archived](https://airflow.apache.org/docs/apache-airflow/stable/howto/usage-cli.html#export-the-purged-records-from-the-archive-tables) command it fetches all the results at once instead of paging them. 
   
   In case of big instances and archiving a lot of data a lot of memory is needed to use this command, othervise it ends up with OOMErrors
   
   Exact line:
   https://github.com/apache/airflow/blob/3.0.1/airflow-core/src/airflow/utils/db_cleanup.py#L163
   
   ### What you think should happen instead?
   
   use server-side cursor for paging if available. 
   
   ### How to reproduce
   
   Create a database that has tables with the size bigger than your memory and then try to archive data with use of 
   airflow db export-archived command. 
   
   
   
   ### Operating System
   
   K8s - pods with Ubuntu
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] airflow db export-archived doesn't page results [airflow]

Posted by "dominikhei (via GitHub)" <gi...@apache.org>.
dominikhei commented on issue #51156:
URL: https://github.com/apache/airflow/issues/51156#issuecomment-2925128756

   @potiuk What is your take on this. I was thinking of using a buffer including n rows and using `.fetchmany(n)` to append the rows batch by batch to the `csv_writer`. With this approach, we would only hold one batch at a time in memory. 
   
   I can come up with a PR for this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] airflow db export-archived doesn't page results [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #51156:
URL: https://github.com/apache/airflow/issues/51156#issuecomment-2925154800

   Yep. sounds like that's the right approach.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] airflow db export-archived doesn't page results [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #51156: airflow db export-archived doesn't page results
URL: https://github.com/apache/airflow/issues/51156


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] airflow db export-archived doesn't page results [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #51156:
URL: https://github.com/apache/airflow/issues/51156#issuecomment-2916256091

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org