[jira] [Commented] (DRILL-2996) ValueVectors shouldn't call reAlloc() in a while() loop

["Jason Altekruse (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14535167#comment-14535167 ] 

Jason Altekruse commented on DRILL-2996:
----------------------------------------

Along with this functional refactoring, we need to come up with a hard specification of how much repetition we support. It is a combination of large varlength values and large lists that are bringing out out these scenarios with excessive allocation.

We have a hard limit that a batch of records can only have 65K elements (so that we know that we can use a two-byte unsigned int to index into them). Currently we impose no specific limit on the number of child values we can have in a list, but these will hit limits as they inner values are stored in vectors themselves (so they can hit this 65k limit, if we have 65k lists, this could happen with a single element list in every index and a few extra elements in just one of the lists). Most cases are handled in regular execution by the field read/writer abstractions, as well as our default behavior to only fill vectors with ~4000 values. At the Value Vector level we do not have enforcement of a hard limit for these inner values and I think that is part of the problem.

> ValueVectors shouldn't call reAlloc() in a while() loop
> -------------------------------------------------------
>
>                 Key: DRILL-2996
>                 URL: https://issues.apache.org/jira/browse/DRILL-2996
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Data Types
>            Reporter: Chris Westin
>            Assignee: Hanifi Gunes
>
> Instead, reAlloc() should be change to take a "new minimum size" as an argument. This value is just the value used to determine the while loops' termination. Then reAlloc() can figure out how much more to allocate once and for all, instead of possibly reallocating and copying more than once, and it can make sure that the size doesn't overflow (we've seen some instances of the allocator being called with negative sizes).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)