[GitHub] drill issue #1075: DRILL-6030: Managed sort should minimize number of batche...

[paul-rogers <gi...@git.apache.org>]

Github user paul-rogers commented on the issue:

    https://github.com/apache/drill/pull/1075
  
    One additional thought. This bug was found when sorting 18 GB of data in 8 GB of memory. That is, a case in which the sort must spill.
    
    What happens in the case in which the 18 GB of data is sorted in, say, 20 GB of memory (an in-memory sort)? We don't want the merge limit to force a spill in this case; kind of defeats the purpose of an in-memory sort.
    
    So:
    
    1. Does the limit affect in memory sort? If so, we need to revise the solution.
    2. Does the in-memory sort suffer from a similar performance issue? If so, we need to revise the in memory sort.
    
    One possible solution is to:
    
    1. Defer sorting of individual batches until necessary.
    2. Sort batches just before spilling.
    3. If all batches fit in memory, do a single, combined sort (using an SV4).


---