You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "2010YOUY01 (via GitHub)" <gi...@apache.org> on 2025/05/21 09:10:41 UTC
[I] Support compression in spill files [datafusion]
2010YOUY01 opened a new issue, #16130:
URL: https://github.com/apache/datafusion/issues/16130
### Is your feature request related to a problem or challenge?
Part of https://github.com/apache/datafusion/issues/16065
In memory-limited queries, temporary results will be written to disk to reduce memory pressure. Now those spills are using plain arrow encoding, they can be applied some general-purpose compression like lz4 and zstd for more efficient disk space usage.
### Describe the solution you'd like
1. Add a configuration for spill compression type
2. Pass an option to Arrow-IPC writer/reader for compression
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support compression in spill files [datafusion]
Posted by "ding-young (via GitHub)" <gi...@apache.org>.
ding-young commented on issue #16130:
URL: https://github.com/apache/datafusion/issues/16130#issuecomment-2897255498
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support compression in spill files [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on issue #16130:
URL: https://github.com/apache/datafusion/issues/16130#issuecomment-2897217243
I think this issue can be added to your current TODO list @ding-young.
In the future, if we introduce a dedicated spill format, we might revise the implementation to apply compression to the entire spill file.
But I think it's good to be implemented now since the code footprint is small (should be just passing an option into Arrow IPC reader/writer)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support compression in spill files [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 closed issue #16130: Support compression in spill files
URL: https://github.com/apache/datafusion/issues/16130
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support compression in spill files [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #16130:
URL: https://github.com/apache/datafusion/issues/16130#issuecomment-2913950663
related/prior art: https://github.com/apache/datafusion/issues/14078#issuecomment-2585363202
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support compression in spill files [datafusion]
Posted by "ding-young (via GitHub)" <gi...@apache.org>.
ding-young commented on issue #16130:
URL: https://github.com/apache/datafusion/issues/16130#issuecomment-2897255287
Sure. I also think that introducing compression option first is good to start though we may try more complicated techniques.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org