You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/01/20 06:51:09 UTC

[GitHub] [arrow-datafusion] yjshen commented on issue #1573: SQL level tests for when sorting exceeded available memory and had to spill to disk

yjshen commented on issue #1573:
URL: https://github.com/apache/arrow-datafusion/issues/1573#issuecomment-1017170185


   Currently, #1596 has one SQL test on spill called `test_sort_spill` that spills during the test by controlling the total memory of the pool. 
   
   https://github.com/apache/arrow-datafusion/blob/6a458c67b5d11cf9dfa09281289c4d2da9484a2d/datafusion/src/physical_plan/sorts/sort.rs#L739-L765
   
   However, since spill metrics are not exposed by sort yet, the current test cannot assert spill count > 0. Do you think it's appropriate if I add spill count as well as spill bytes into `BaselineMetrics`? I think these might be common metrics as we proceed with more memory consumers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org