[jira] [Created] (IMPALA-6054) Parquet dictionary pages should be freed on dictionary construction

["Joe McDonnell (JIRA)" <ji...@apache.org>]

Joe McDonnell created IMPALA-6054:
-------------------------------------

             Summary: Parquet dictionary pages should be freed on dictionary construction
                 Key: IMPALA-6054
                 URL: https://issues.apache.org/jira/browse/IMPALA-6054
             Project: IMPALA
          Issue Type: Improvement
          Components: Backend
    Affects Versions: Impala 2.10.0
            Reporter: Joe McDonnell


The Parquet scanner uses the dictionary_pool_ to allocate memory for the dictionary page (see BaseScalarColumnReader::InitDictionary()). This dictionary page is used to initialize the dictionary in CreateDictionaryDecoder(). The resulting dictionary is a vector of values. For some datatypes, such as strings, the resulting dictionary has an array of StringValue's that contain pointers into the dictionary page (see the StringValue specialization in ParquetPlainEncoder::Decode()). In this case, the dictionary page must be kept and attached to the last row batch that references it. However, for other datatypes, the values are copied into the dictionary and the dictionary page is no longer needed after the dictionary is constructed.

Currently, these dictionary pages remain in the dictionary_pool_ and are attached to the last row batch to be passed to other ExecNodes (see FlushRowGroupResources()). This should only pass StringValue dictionary pages (or other types that point to data in the page) on the row batch. The other types should be freed immediately once the dictionary has been constructed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)