[jira] [Created] (DRILL-5588) Hash Aggregate: Avoid copy on output of aggregate columns

["Boaz Ben-Zvi (JIRA)" <ji...@apache.org>]

Boaz Ben-Zvi created DRILL-5588:
-----------------------------------

             Summary: Hash Aggregate: Avoid copy on output of aggregate columns
                 Key: DRILL-5588
                 URL: https://issues.apache.org/jira/browse/DRILL-5588
             Project: Apache Drill
          Issue Type: Improvement
          Components: Execution - Relational Operators
    Affects Versions: 1.10.0
            Reporter: Boaz Ben-Zvi


 When the Hash Aggregate operator outputs its result batches downstream, the key columns (value vectors) are returned as is, but for the aggregate columns new value vectors are allocated and the values are copied. This has an impact on performance. (see the method allocateOutgoing() ). A second effect is on memory management (as this allocation is not planned for by the code that controls spilling, etc).
   For some simple aggregate functions (e.g. SUM), the stored value vectors for the aggregate values can be returned as is. For functions like AVG, there is a need to divide the SUM values by the COUNT values. Still this can be done in-place (of the SUM values) and avoid new allocation and copy. 
   For VarChar type aggregate values (only used by MAX or MIN), there is another issue -- currently any such value vector is allocated as an ObjectVector (see BatchHolder()) (and on the JVM heap, not in direct memory). This is to manage the sizes of the values, which could change as the aggregation progresses (e.g., for MAX(name) -- first record has 'abe', but the next record has 'benjamin' which is both bigger ('b' > 'a') and longer). For the final output, this requires a new allocation and a copy in order to have a compact value vector in direct memory. Maybe the ObjectVector could be replaced with some direct memory implementation that is optimized for "good" values (e.g., all are of similar size), but penalized "bad" values (e.g., reallocates or moves values, when needed) ?






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)