[jira] [Created] (IMPALA-5844) Fix management of FunctionContext "local" allocations.

["Tim Armstrong (JIRA)" <ji...@apache.org>]

Tim Armstrong created IMPALA-5844:
-------------------------------------

             Summary: Fix management of FunctionContext "local" allocations.
                 Key: IMPALA-5844
                 URL: https://issues.apache.org/jira/browse/IMPALA-5844
             Project: IMPALA
          Issue Type: Improvement
          Components: Backend
            Reporter: Tim Armstrong
            Assignee: Tim Armstrong


FunctionContexts associated with expressions make two kind of allocations with very different lifecycles. One type of allocation is owned and managed by the expression, while the other "local" allocation is implicitly transferred to the Impala daemon after control flow returns from the expression. Both are currently allocated from the same pool.

RowBatches returned from plan nodes may reference variable-length data stored in local allocations so this memory should be attached to the RowBatches.

One approach here is:
* Separate local and other allocations to allocate from different MemPools. 
* Manage local allocations in bulk by clearing, freeing, or transferring data from that MemPool, similar to other memory that would be allocated from a MemPool.

I think there are some potential wrinkles here and work to set up correct MemPools for all places that use expressions but I think the high-level approach should work.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)