[jira] [Comment Edited] (DRILL-2198) Vector re-allocation fails for large records

["Hanifi Gunes (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335542#comment-14335542 ] 

Hanifi Gunes edited comment on DRILL-2198 at 2/24/15 10:05 PM:
---------------------------------------------------------------

Unfortunately I don't have a test case. Creating one is trivial though. In many places (almost all) where re-allocation is triggered, we just do reallocate only once without making sure that we have enough space in the buffer after reallocation. For instance, all of the setSafe methods in fixed value vectors suffers from this when the argument index  is sufficiently larger than the value capacity after reallocation. Please refer to FixedValueVectors.java in code templates for a more concrete look. Here follows an example:

{code:title=This is unsafe - excerpt from FixedValueVectors.java#setSafe}
if (index >= getValueCapacity()) {
  reAlloc();
}
{code}

{code:title=This is safe}
while (index >= getValueCapacity()) {
  reAlloc();
}
{code}


was (Author: hgunes):
Unfortunately I don't have a test case. Creating one is trivial though. In many places (almost all) where re-allocation is triggered, we just do reallocate only once without making sure that we have enough space in the buffer after reallocation. For instance, all of the setSafe methods in fixed value vectors suffers from this when the argument index  is sufficiently larger than the value capacity after reallocation. Please refer to FixedValueVectors.java in code templates for a more concrete look. Here follows an example:

{code:title=This is unsafe - excerpt from NullableValueVectors.java#setSafe}
if (index >= getValueCapacity()) {
  reAlloc();
}
{code}

{code:title=This is safe}
while (index >= getValueCapacity()) {
  reAlloc();
}
{code}

> Vector re-allocation fails for large records
> --------------------------------------------
>
>                 Key: DRILL-2198
>                 URL: https://issues.apache.org/jira/browse/DRILL-2198
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Execution - Data Types
>            Reporter: Hanifi Gunes
>            Priority: Minor
>             Fix For: Future
>
>
> In many places we reallocate vectors without making sure that there is enough space after reallocation. This causes problem for large records that won't fit into buffer even after reallocating once. We should keep re-allocating until there is enough space for writing the record. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)