[jira] [Created] (PARQUET-372) Parquet stats can have awkwardly large values

["Ryan Blue (JIRA)" <ji...@apache.org>]

Ryan Blue created PARQUET-372:
---------------------------------

             Summary: Parquet stats can have awkwardly large values
                 Key: PARQUET-372
                 URL: https://issues.apache.org/jira/browse/PARQUET-372
             Project: Parquet
          Issue Type: Bug
          Components: parquet-format, parquet-mr
            Reporter: Ryan Blue


If a column is storing very large values, say 2-4 MB, then the page header's min and max values can also be this large. It is wasteful to keep that much data in a page header, so we should look at options for decreasing the size required in these cases.

One idea is to truncate the size of binary data and change the last byte to 0xFF (max) or 0x00 (min) to get a roughly equivalent min and max that isn't huge. This probably has some problems when the data stores multi-byte characters in UTF8 so we have to be careful and look into byte-wise sorting and UTF8.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)