You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "Gang Wu (Jira)" <ji...@apache.org> on 2024/06/03 02:45:00 UTC
[jira] [Commented] (PARQUET-2139) Bogus file offset for ColumnMetaData written to ColumnChunk metadata of single parquet files
[ https://issues.apache.org/jira/browse/PARQUET-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17851484#comment-17851484 ]
Gang Wu commented on PARQUET-2139:
----------------------------------
[~etseidl] I couldn't find it either.
IMO, the best fix is to write ColumnMetaData at the end of column chunk as per the spec. However, this has been incorrectly implemented since day 1 and it takes non-trivial effort to implement this in parquet-java. I think we can instead clarify this is deprecated and should not be used due to the known issue?
Perhaps it is also a good time to add a note to the spec: writing 0 or -1 to the file_offset when the ColumnMetaData is unavailable at the end of column chunk.
> Bogus file offset for ColumnMetaData written to ColumnChunk metadata of single parquet files
> --------------------------------------------------------------------------------------------
>
> Key: PARQUET-2139
> URL: https://issues.apache.org/jira/browse/PARQUET-2139
> Project: Parquet
> Issue Type: Bug
> Components: parquet-mr
> Affects Versions: 1.12.2
> Reporter: Timothy Miller
> Priority: Major
>
> In an effort to understand the parquet format better, I've so far written my own Thrift parser, and upon examining the output, I noticed something peculiar.
> To begin with, check out the definition for ColumnChunk here: [https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift]
> You'll notice that if there's an element 2 in the struct, this is supposed to be a file offset to where a redundant copy of the ColumnMetaData.
> Next, have a look at the file called "modified.parquet" attached to https://issues.apache.org/jira/browse/PARQUET-2069. When I dump the metadata at the end of the file, I get this:
> {{Struct(FileMetaData):}}
> {{ 1: i32(version) = I32(1)}}
> {{ 2: List(SchemaElement schema):}}
> {{ ...
> 3: i64(num_rows) = I64(1)
> 4: List(RowGroup row_groups):
> 1: Struct(RowGroup row_groups):
> 1: List(ColumnChunk columns):
> 1: Struct(ColumnChunk columns):
> 2: i64(file_offset) = I64(4)
> 3: Struct(ColumnMetaData meta_data):
> 1: Type(type) = I32(6) = BYTE_ARRAY
> 2: List(Encoding encodings):
> 1: Encoding(encodings) = I32(0) = PLAIN
> 2: Encoding(encodings) = I32(3) = RLE
> 3: List(string path_in_schema):
> 1: string(path_in_schema) = Binary("destination_addresses")
> 2: string(path_in_schema) = Binary("array")
> 3: string(path_in_schema) = Binary("element")
> 4: CompressionCodec(codec) = I32(0) = UNCOMPRESSED
> 5: i64(num_values) = I64(6)
> 6: i64(total_uncompressed_size) = I64(197)
> 7: i64(total_compressed_size) = I64(197)
> 9: i64(data_page_offset) = I64(4)
> }}
> As you can see, element 2 of the ColumnChunk indicates that there is another copy of the ColumnMetaData at offset 4 of the file. But then we see that element 9 of the ColumnMetaData shown above indicates that the data page offset is ALSO 4, where we should find a Thrift encoding of a PageHeader structure. Obviously, both structures can't be in the same place, and in fact a PageHeader is what is located there.
> Based on what I'm seeing here, I believe that element 2 of ColumnChunk should be omitted entirely in this scenario, so as to not falsely indicate that there would be another copy of the ColumnMetadata in this location in the file where indeed something else is present.
> It may take me a while to locate the offending code, but I thought I'd go ahead and point this out before I set off to investigate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org