You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/10/24 16:53:29 UTC

[GitHub] [arrow-datafusion] milenkovicm commented on issue #3941: Generate runtime errors if the memory budget is exceeded [EPIC]

milenkovicm commented on issue #3941:
URL: https://github.com/apache/arrow-datafusion/issues/3941#issuecomment-1289320935

   Should memory limit be optimistic? What I mean in case of aggregation we could first process record batch, compare memory before and after batch is process and request delta value from memory manager. Otherwise we'd need to do two passes over records batch, or request memory for every record, which may lead to contention on memory manager and trigger spill in middle of batch processing. End of batch processing would be a "safe point" which should have correct memory usage, or trigger spill. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org