[jira] [Created] (DRILL-5759) Performance hit: JSON reader causes repeated vector reallocations

["Paul Rogers (JIRA)" <ji...@apache.org>]

Paul Rogers created DRILL-5759:
----------------------------------

             Summary: Performance hit: JSON reader causes repeated vector reallocations
                 Key: DRILL-5759
                 URL: https://issues.apache.org/jira/browse/DRILL-5759
             Project: Apache Drill
          Issue Type: Improvement
    Affects Versions: 1.10.0
            Reporter: Paul Rogers
            Priority: Minor


Run the query in DRILL-5753 with DEBUG level logging. This query uses the JSON record reader. You will see many messages about vector reallocation. These messages show that the JSON reader incorrectly allocated vector sizes.

The specific problem is with repeated types. In the messages, note that vectors start off with 4096 values, which is the target row count. But, the data contains repeated columns, and so the vectors must grow multiple times to hold the repeated items. On the first batch, heuristics might be used to guess the cardinality. On the second and subsequent batches, empirical data gathered on the first batch could be used to better allocate the subsequent batches.

The result is memory thrashing and a performance hit as memory is allocated, copied and released multiple times.

{code}
BigIntVector - Reallocating vector [$data$(BIGINT:REQUIRED)]. # of bytes: [32768] -> [65536]
UInt4Vector - Reallocating vector [$offsets$(UINT4:REQUIRED)]. # of bytes: [16384] -> [32768]
UInt4Vector - Reallocating vector [$offsets$(UINT4:REQUIRED)]. # of bytes: [16384] -> [32768]
UInt1Vector - Reallocating vector [$bits$(UINT1:REQUIRED)]. # of bytes: [4096] -> [8192]
UInt1Vector - Reallocating vector [$bits$(UINT1:REQUIRED)]. # of bytes: [4096] -> [8192]
UInt1Vector - Reallocating vector [$bits$(UINT1:REQUIRED)]. # of bytes: [4096] -> [8192]
BigIntVector - Reallocating vector [c(BIGINT:OPTIONAL)]. # of bytes: [32768] -> [65536]
UInt1Vector - Reallocating vector [$bits$(UINT1:REQUIRED)]. # of bytes: [4096] -> [8192]
Float8Vector - Reallocating vector [d(FLOAT8:OPTIONAL)]. # of bytes: [32768] -> [65536]
BigIntVector - Reallocating vector [$data$(BIGINT:REQUIRED)]. # of bytes: [65536] -> [131072]
UInt4Vector - Reallocating vector [$offsets$(UINT4:REQUIRED)]. # of bytes: [32768] -> [65536]
UInt4Vector - Reallocating vector [$offsets$(UINT4:REQUIRED)]. # of bytes: [32768] -> [65536]
UInt1Vector - Reallocating vector [$bits$(UINT1:REQUIRED)]. # of bytes: [8192] -> [16384]
UInt1Vector - Reallocating vector [$bits$(UINT1:REQUIRED)]. # of bytes: [8192] -> [16384]
BigIntVector - Reallocating vector [c(BIGINT:OPTIONAL)]. # of bytes: [65536] -> [131072]
UInt1Vector - Reallocating vector [$bits$(UINT1:REQUIRED)]. # of bytes: [8192] -> [16384]
Float8Vector - Reallocating vector [d(FLOAT8:OPTIONAL)]. # of bytes: [65536] -> [131072]
UInt1Vector - Reallocating vector [$bits$(UINT1:REQUIRED)]. # of bytes: [8192] -> [16384]
UInt4Vector - Reallocating vector [$offsets$(UINT4:REQUIRED)]. # of bytes: [16384] -> [32768]
UInt4Vector - Reallocating vector [$offsets$(UINT4:REQUIRED)]. # of bytes: [16384] -> [32768]
UInt4Vector - Reallocating vector [$offsets$(UINT4:REQUIRED)]. # of bytes: [16384] -> [32768]
UInt1Vector - Reallocating vector [$bits$(UINT1:REQUIRED)]. # of bytes: [4096] -> [8192]
UInt1Vector - Reallocating vector [$bits$(UINT1:REQUIRED)]. # of bytes: [4096] -> [8192]
BigIntVector - Reallocating vector [col1(BIGINT:OPTIONAL)]. # of bytes: [32768] -> [65536]
BigIntVector - Reallocating vector [$data$(BIGINT:REQUIRED)]. # of bytes: [131072] -> [262144]
UInt4Vector - Reallocating vector [$offsets$(UINT4:REQUIRED)]. # of bytes: [65536] -> [131072]
UInt1Vector - Reallocating vector [$bits$(UINT1:REQUIRED)]. # of bytes: [16384] -> [32768]
Float8Vector - Reallocating vector [d(FLOAT8:OPTIONAL)]. # of bytes: [131072] -> [262144]
UInt1Vector - Reallocating vector [$bits$(UINT1:REQUIRED)]. # of bytes: [16384] -> [32768]
UInt4Vector - Reallocating vector [$offsets$(UINT4:REQUIRED)]. # of bytes: [65536] -> [131072]
UInt1Vector - Reallocating vector [$bits$(UINT1:REQUIRED)]. # of bytes: [16384] -> [32768]
UInt1Vector - Reallocating vector [$bits$(UINT1:REQUIRED)]. # of bytes: [16384] -> [32768]
BigIntVector - Reallocating vector [c(BIGINT:OPTIONAL)]. # of bytes: [131072] -> [262144]
UInt4Vector - Reallocating vector [$offsets$(UINT4:REQUIRED)]. # of bytes: [16384] -> [32768]
UInt4Vector - Reallocating vector [$offsets$(UINT4:REQUIRED)]. # of bytes: [16384] -> [32768]
UInt4Vector - Reallocating vector [$offsets$(UINT4:REQUIRED)]. # of bytes: [32768] -> [65536]
UInt4Vector - Reallocating vector [$offsets$(UINT4:REQUIRED)]. # of bytes: [32768] -> [65536]
UInt4Vector - Reallocating vector [$offsets$(UINT4:REQUIRED)]. # of bytes: [32768] -> [65536]
UInt4Vector - Reallocating vector [$offsets$(UINT4:REQUIRED)]. # of bytes: [16384] -> [32768]
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)