You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "milenkovicm (via GitHub)" <gi...@apache.org> on 2025/11/02 05:41:08 UTC

Re: [I] External sort failing with non-spillable operators as input (RepartitionExec) [datafusion]

milenkovicm commented on issue #17334:
URL: https://github.com/apache/datafusion/issues/17334#issuecomment-3477475301

   Long story short, in spark operators could ask others to spill when they get under pressure, and I think it makes sense as in the most cases, I saw, there is a spillable operator down the pipeline which could spill.
   
   I did not see easy way to implement cooperative memory management in DF. It would add additional set of complexity, threading issues ...
   
   KISS solution, ignore unspillable operator allocation error, chances are there is a spillable operator down the road which will get under memory pressure 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org