You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/06/23 03:59:00 UTC
[jira] [Commented] (DRILL-5601) Rollup of External Sort memory
management fixes
[ https://issues.apache.org/jira/browse/DRILL-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060385#comment-16060385 ]
ASF GitHub Bot commented on DRILL-5601:
---------------------------------------
GitHub user paul-rogers opened a pull request:
https://github.com/apache/drill/pull/860
DRILL-5601: Rollup of external sort fixes an improvements
- DRILL-5513: Managed External Sort : OOM error during the merge phase
- DRILL-5519: Sort fails to spill and results in an OOM
- DRILL-5522: OOM during the merge and spill process of the managed external sort
- DRILL-5594: Excessive buffer reallocations during merge phase of external sort
- DRILL-5597: Incorrect "bits" vector allocation in nullable vectors allocateNew()
- DRILL-5602: Repeated List Vector fails to initialize the offset vector
All of the bugs have to do with handling low-memory conditions, and with
correctly estimating the sizes of vectors, even when those vectors come
from the spill file or from an exchange. Hence, the changes for all of
the above issues are interrelated.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/paul-rogers/drill DRILL-5601
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/860.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #860
----
commit 63faa99ba0ef2e51bb18ac5a91cd95101a54ae88
Author: Paul Rogers <pr...@maprtech.com>
Date: 2017-04-06T20:57:19Z
DRILL-5601: Rollup of external sort fixes an improvements
- DRILL-5513: Managed External Sort : OOM error during the merge phase
- DRILL-5519: Sort fails to spill and results in an OOM
- DRILL-5522: OOM during the merge and spill process of the managed external sort
- DRILL-5594: Excessive buffer reallocations during merge phase of external sort
- DRILL-5597: Incorrect "bits" vector allocation in nullable vectors allocateNew()
- DRILL-5602: Repeated List Vector fails to initialize the offset vector
All of the bugs have to do with handling low-memory conditions, and with
correctly estimating the sizes of vectors, even when those vectors come
from the spill file or from an exchange. Hence, the changes for all of
the above issues are interrelated.
----
> Rollup of External Sort memory management fixes
> -----------------------------------------------
>
> Key: DRILL-5601
> URL: https://issues.apache.org/jira/browse/DRILL-5601
> Project: Apache Drill
> Issue Type: Task
> Affects Versions: 1.11.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
> Fix For: 1.11.0
>
>
> Rollup of a set of specific JIRA entries that all relate to the very difficult problem of managing memory within Drill in order for the external sort to stay within a memory budget. In general, the fixes relate to better estimating memory used by the three ways that Drill allocates vector memory (see DRILL-5522) and to predicting the size of vectors that the sort will create, to avoid repeated realloc-copy cycles (see DRILL-5594).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)