[jira] [Created] (PARQUET-505) Column reader: automatically handle large data pages

["Wes McKinney (JIRA)" <ji...@apache.org>]

Wes McKinney created PARQUET-505:
------------------------------------

             Summary: Column reader: automatically handle large data pages
                 Key: PARQUET-505
                 URL: https://issues.apache.org/jira/browse/PARQUET-505
             Project: Parquet
          Issue Type: Bug
          Components: parquet-cpp
            Reporter: Wes McKinney


Currently, we are only supporting data pages whose headers are 64K or less (see {{parquet/column/serialized-page.cc}}. Since page headers can essentially be arbitrarily large (in pathological cases) because of the page statistics, if deserializing the page header fails, we should attempt to read a progressively larger amount of file data in effort to find the end of the page header. 

As part of this (and to make testing easier!), the maximum data page header size should be configurable. We can write test cases by defining appropriate Statistics structs to yield serialized page headers of whatever desired size.

On malformed files, we may run past the end of the file, in such cases we should raise a reasonable exception. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)