[jira] [Created] (PARQUET-1401) RowGroup offset and total compressed size fields

["Gidon Gershinsky (JIRA)" <ji...@apache.org>]

Gidon Gershinsky created PARQUET-1401:
-----------------------------------------

             Summary: RowGroup offset and total compressed size fields
                 Key: PARQUET-1401
                 URL: https://issues.apache.org/jira/browse/PARQUET-1401
             Project: Parquet
          Issue Type: Sub-task
          Components: parquet-cpp, parquet-format
            Reporter: Gidon Gershinsky
            Assignee: Gidon Gershinsky


Spark uses filterFileMetaData* methods in ParquetMetadataConverter class, that  calculate the offset and total compressed size of a RowGroup data.
The offset calculation is done by extracting the ColumnMetaData of the first column, and using its offset fields.
The total compressed size calculation is done by running a loop over all column chunks in the RowGroup, and summing up the size values from each chunk's ColumnMetaData .
If one or more columns are hidden (encrypted with a key unavailable to the reader), these calculations can't be performed, because the column metadata is protected. 
 
But: these calculations don't really need the individual column values. The results pertain to the whole RowGroup, not specific columns. 
Therefore, we will define two new optional fields in the RowGroup Thrift structure:
 
_optional i64 file_offset_
_optional i64 total_compressed_size_
 
and calculate/set them upon file writing. Then, Spark will be able to query a file with hidden columns (of course, only if the query itself doesn't need the hidden columns - works with a masked version of them, or reads columns with available keys).
 
These values can be set only for encrypted files (or for all files, to skip the loop upon reading). I've tested this, works fine in Spark writers and readers.
 
I've also checked other references to ColumnMetaData fields in parquet-mr. There are none - therefore, its the only change we need in parquet.thrift to handle hidden columns.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)