You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@drill.apache.org by "Venki Korukanti (JIRA)" <ji...@apache.org> on 2015/04/14 03:28:12 UTC

[jira] [Commented] (DRILL-2728) Merge spill files in ExternalSort

    [ https://issues.apache.org/jira/browse/DRILL-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14493422#comment-14493422 ] 

Venki Korukanti commented on DRILL-2728:
----------------------------------------

LGTM, +1.

> Merge spill files in ExternalSort
> ---------------------------------
>
>                 Key: DRILL-2728
>                 URL: https://issues.apache.org/jira/browse/DRILL-2728
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Execution - Relational Operators
>            Reporter: Steven Phillips
>            Assignee: Venki Korukanti
>             Fix For: 0.9.0
>
>         Attachments: DRILL-2728.patch
>
>
> When external sort spills, it first merges all of the in-memory batches, and then keeps one batch in memory, while spilling the rest to disk. It's possible that we will still run out of memory, due to the single batch held in memory for each spill. When the memory due to these batches becomes too large, we should combine the spill files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)