[jira] [Commented] (DRILL-6126) Allocate memory for value vectors upfront in flatten operator

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-6126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16371766#comment-16371766 ] 

ASF GitHub Bot commented on DRILL-6126:
---------------------------------------

GitHub user ppadma opened a pull request:

    https://github.com/apache/drill/pull/1125

     DRILL-6126: Allocate memory for value vectors upfront in flatten operator

    Made changes to allocate memory upfront for flatten operator based on sizing calculations.
    Need to do allocation of single column (can be nested) for a particular record count
    and allocation hints. Refactored the code a bit for that.
    Instead of assuming worst case fragmentation factor of 2, changed the logic to round down the number of rows calculated to nearest power of two. This will allow us to pack value vectors more densely and will help with memory utilization.
    RepeatedMapvector and RepeatedListVector are extended from RepeatedFixedWidthVectorLike. This is wrong and causing problems in Allocation logic (allocatePrecomputedChildCount in AllocationHelper more specifically). Fixed that.
    This PR has 2 commits. One for all of the above and second one for
    DRILL-6162: Enhance record batch sizer to retain nesting information.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ppadma/drill DRILL-6126

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/drill/pull/1125.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1125
    
----
commit 58c6b9ad584e56c71d982feaaa43ad32b5011eef
Author: Padma Penumarthy <pp...@...>
Date:   2018-02-21T17:33:12Z

    DRILL-6162: Enhance record batch sizer to retain nesting information for map columns.

commit f7c09131179b75d10ffe195785c9aef3b9c7ed97
Author: Padma Penumarthy <pp...@...>
Date:   2018-02-21T17:35:47Z

    DRILL-6126: Allocate memory for value vectors upfront in flatten operator

----


> Allocate memory for value vectors upfront in flatten operator
> -------------------------------------------------------------
>
>                 Key: DRILL-6126
>                 URL: https://issues.apache.org/jira/browse/DRILL-6126
>             Project: Apache Drill
>          Issue Type: Improvement
>            Reporter: Padma Penumarthy
>            Assignee: Padma Penumarthy
>            Priority: Critical
>             Fix For: 1.12.0
>
>
> With recent changes to control batch size for flatten operator, we figure out row count in the output batch based on memory. Since we know how many rows we are going to include in the batch, we can also allocate the memory needed upfront instead of starting with initial value (4096) and doubling, copying every time we need more. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)