[jira] [Commented] (DRILL-6032) Use RecordBatchSizer to estimate size of columns in HashAgg

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341885#comment-16341885 ] 

ASF GitHub Bot commented on DRILL-6032:
---------------------------------------

GitHub user ilooner opened a pull request:

    https://github.com/apache/drill/pull/1101

    DRILL-6032: Made the batch sizing for HashAgg more accurate.

    ## RecordBatchSizer changes
    
     - The RecordBatchSizer previously computed fixed width column sizes by measuring the total size of a vector and dividing by the number of elements. Because of this the RecordBatchSizer would return a zero size for FixedWidth vectors that had no data. So I added a method to FixedWidth vectors to get the size of a record and use that method to compute the column width in the RecordBatchSizer.
 - In some cases it was possible for the RecordBatchSizer to return a column width of 0, when it is not possible to have vectors with a width of 1 in practice. So I made the minimum column width returned by the RecordBatchSizer 1.
    

    ## HashAgg changes
    
     - Removed commented out code and unused variables.
     - Removed if statements for printing debug statements and instead used logger.debug
     - Removed the extraNonNullColumns and extraRowBytes tweak parameters for computing the sizes of batches
     - The RecordBatchSizer is used to compute the width of each column instead of adhoc custom logic. 
     - Using the real width of each column to estimate column sizes instead of taking the max width of all columns and assuming each column has the max width
     - Removed the assumption that varchars will not exceed 50 characters in length
     - Removed unnecessary condition checks in delayedSetup and updateEstMaxBatchSize

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ilooner/drill DRILL-6032

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/drill/pull/1101.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1101
    
----
commit b8682f6e09889e5ba334f36006fb9ed754f571f6
Author: Timothy Farkas <ti...@...>
Date:   2017-12-13T23:44:28Z

    DRILL-6032: Made the batch sizing for HashAgg more accurate.

----


> Use RecordBatchSizer to estimate size of columns in HashAgg
> -----------------------------------------------------------
>
>                 Key: DRILL-6032
>                 URL: https://issues.apache.org/jira/browse/DRILL-6032
>             Project: Apache Drill
>          Issue Type: Improvement
>            Reporter: Timothy Farkas
>            Assignee: Timothy Farkas
>            Priority: Major
>             Fix For: 1.13.0
>
>
> We need to use the RecordBatchSize to estimate the size of columns in the Partition batches created by HashAgg.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)