You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@parquet.apache.org by "Matt Martin (JIRA)" <ji...@apache.org> on 2014/06/20 05:47:24 UTC

[jira] [Created] (PARQUET-4) Use LRU caching for footers in ParquetInputFormat.

Matt Martin created PARQUET-4:
---------------------------------

             Summary: Use LRU caching for footers in ParquetInputFormat.
                 Key: PARQUET-4
                 URL: https://issues.apache.org/jira/browse/PARQUET-4
             Project: Parquet
          Issue Type: Bug
          Components: parquet-mr
            Reporter: Matt Martin


The caching approach needs to change because of issues that occur when the same ParquetInputFormat instance is reused to generate splits for different input directories. For example, it causes problems in Hive's FetchOperator when the FetchOperator is attempting to operate over more than one partition (sidenote: as far as I could tell, Hive has been reusing inputformat instances in this way for quite some time). The details of how this issue manifests itself with respect to Hive are described in more detail here: https://groups.google.com/d/msg/parquet-dev/0aXql-3z7vE/Gn5m094V7PMJ

The proposed patch can be found here: https://github.com/apache/incubator-parquet-mr/pull/2



--
This message was sent by Atlassian JIRA
(v6.2#6252)