You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by "Venki Korukanti (JIRA)" <ji...@apache.org> on 2015/01/13 02:59:34 UTC

[jira] [Created] (DRILL-1993) Fix allocation issues in HashTable and HashAgg to reduce memory waste

Venki Korukanti created DRILL-1993:
--------------------------------------

             Summary: Fix allocation issues in HashTable and HashAgg to reduce memory waste
                 Key: DRILL-1993
                 URL: https://issues.apache.org/jira/browse/DRILL-1993
             Project: Apache Drill
          Issue Type: Bug
          Components: Execution - Operators
            Reporter: Venki Korukanti
            Assignee: Venki Korukanti
             Fix For: 0.8.0


Issues found:

+ Key container allocation issue in HashTable
Currently we allocate 2^16 records capacity memory for "hashValues" and "links" vectors in each BatchHolder, but for "key" holders we use the allocateNew which by default allocates low capacity memory compared to "hashValues" and "links" vector capacities (incase of Integer key, capacity is 4096 records). This causes "key" holders to fill up much sooner even though "hashValues" and "links" vectors still have lot of free entries. As a result we create more BatchHolders than required causing wasted space in "links" and "hashValues" vectors in each BatchHolder. And for each new BatchHolder we create a SV4 vector in HashJoinHelper which is another overhead.

+ Allocation issues in HashAggTemplate
HashAggTemplate has its own BatchHolders which has vectors allocated using allocateNew (i.e small capacity). Whenever a BatchHolder in HashAggTemplate reaches its capacity, we add a new BatchHolder in HashTable. This causes the HashTable BatchHolders to be not space efficient.

+ Update the HashAggTemplate.outputCurrentBatch to consider cases where all entries in a single BatchHolder are can't be copied over to output vectors in a single pass (output vectors capacity is lower than the number of records in BatchHolder)

+ Lazy BatchHolder creation for both HashAgg and HashTable
Don't allocate the BatchHolder until first put request is received. This way we don't waste space in fragments which don't receive any input records. This is possible when the group by key has very few distinct values (such as shipmode in TPCH) only few fragments receive the data. Our current parallelization code is not considering the distinct values when parallelizing hash exchanges.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)