[I] limit max disk usage for spilling queries [datafusion]

["alamb (via GitHub)" <gi...@apache.org>]

alamb opened a new issue, #15358:
URL: https://github.com/apache/datafusion/issues/15358

   ### Is your feature request related to a problem or challenge?
   
   Breaking rationale from https://github.com/apache/datafusion/pull/14975#issue-2890626662 into its own ticket:
   
   
   For memory-limit queries, executors might write temporary results into the disk to reduce memory load. It's important to have a configuration option to limit the max disk usage, in case some query would bloat the disk and cause other issues.
   
   DuckDB provides a similar configuration:
   ```
   max_temp_directory_size | The maximum amount of data stored inside the 'temp_directory' (when set) (e.g., 1GB)
   ```
   
   ### Describe the solution you'd like
   
   Provide a way to limit the total disk usage of queries that spill to disk
   
   ### Describe alternatives you've considered
   
   Add a configuration option max_temp_directory_size to disk manager (default to 100GB), if the limit is reached for all spilled files, an error will be returned. 
   
   
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] limit max disk usage for spilling queries [datafusion]

["alamb (via GitHub)" <gi...@apache.org>]

alamb closed issue #15358: limit max disk usage for spilling queries
URL: https://github.com/apache/datafusion/issues/15358


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org