[jira] [Resolved] (IMPALA-5911) Grouping aggregations with having conjuncts and Serialize()/Finalize() functions uses excessive expr memory

["Tim Armstrong (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-5911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Armstrong resolved IMPALA-5911.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.11.0

Fixed by this commit - the memory is no longer allocated from the partition's pool

IMPALA-5844: use a MemPool for expr result allocations

This is also a step towards IMPALA-2399 (remove QueryMaintenance()).

"local" allocations containing expression results (either intermediate
or final results) have the following properties:

    They are usually small allocations
    They can be made frequently (e.g. every function call)
    They are owned and managed by the Impala runtime
    They are freed in bulk at various points in query execution.

A MemPool (i.e. bump allocator) is the right mechanism to manage
allocations with the above properties. Before this patch
FunctionContext's used a FreePool + vector of allocations to emulate the
above behaviour. This patch switches to using a MemPool to bring these
allocations in line with the rest of the codebase.

The steps required to do this conversion.

    Use a MemPool for FunctionContext local allocations.
    Identify appropriate MemPools for all of the local allocations from
    function contexts so that the memory lifetime is correct.
    Various cleanup and documentation of existing MemPools.
    Replaces calls to FreeLocalAllocations() with calls to
    MemPool::Clear()

More involved surgery was required in a few places:

    Made the Sorter own its comparator, exprs and MemPool.
    Remove FunctionContextImpl::ReallocateLocal() and just have
    StringFunctions::Replace() do the doubling itself to avoid
    the need for a special interface. Worst-case this doubles
    the memory requirements for Replace() since n / 2 + n / 4
    + n / 8 + .... bytes of memory could be wasted instead of recycled
    for an n-byte output string.
    Provide a way redirect agg fn Serialize()/Finalize() allocations
    to come directly from the output RowBatch's MemPool. This is
    also potentially applicable to other places where we currently
    copy out strings from local allocations, e.g.
    AnalyticEvalNode::AddResultTuple() and Tuple::MaterializeExprs().
    --stress_free_pool_alloc was changed to instead intercept at the
    FunctionContext layer so that it retains the old behaviour even
    though allocations do not all come from FreePools.

The "local" allocation concept was not exposed directly in udf.h so this
patch also renames them to better reflect that they're used for expr
results.

Testing:

    ran exhaustive and ASAN

Change-Id: I4ba5a7542ed90a49a4b5586c040b5985a7d45b61
Reviewed-on: http://gerrit.cloudera.org:8080/8025
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins


> Grouping aggregations with having conjuncts and Serialize()/Finalize() functions uses excessive expr memory
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: IMPALA-5911
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5911
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 2.10.0
>            Reporter: Tim Armstrong
>            Assignee: Tim Armstrong
>            Priority: Minor
>             Fix For: Impala 2.11.0
>
>
> The loop in GetRowsFromPartition() invokes Finalize()/Serialize() on all of the rows in output_partition_, which may allocate local allocations in output_partition_->agg_fn_evals. These are not cleared out until the partition is destroyed.
> If the having conjuncts are very select this can result in a lot of excess memory. E.g. the following query results in a lot of non-buffer-pool memory overhead:
> {code}
> select l_orderkey, min(l_comment) from tpch_parquet.lineitem group by l_orderkey having min(l_comment) like '%slow dolphins%' order by l_orderkey
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)