[jira] [Created] (IMPALA-5707) Parquet table sink should operator within a memory constraint

["Tim Armstrong (JIRA)" <ji...@apache.org>]

Tim Armstrong created IMPALA-5707:
-------------------------------------

             Summary: Parquet table sink should operator within a memory constraint
                 Key: IMPALA-5707
                 URL: https://issues.apache.org/jira/browse/IMPALA-5707
             Project: IMPALA
          Issue Type: Sub-task
          Components: Backend
    Affects Versions: Impala 2.10.0
            Reporter: Tim Armstrong


We should modify the Parquet table sink so that it reserves memory upfront and allocates the bulk of its memory (column buffers, etc) from the buffer pool.

The reservation calculation should be fairly straightforward for clustered inserts or single-partition inserts, but currently the memory consumption of dynamically partitioned non-cluster inserts is not bounded. In that case we would need to flush partitions to disk to free memory.

One possibly tricky edge case is inserting large string values, since the values could be larger than the regular 2MB buffer size.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)