You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "abbit (via GitHub)" <gi...@apache.org> on 2024/09/24 16:13:10 UTC

[I] [Go][Parquet] Inconsistency in RowGroup 'total byte size' field values between implementations [arrow-go]

abbit opened a new issue, #142:
URL: https://github.com/apache/arrow-go/issues/142

   ### Describe the usage question you have. Please include as many useful details as  possible.
   
   
   `parquet.thrift` in parquet-format  repo describes `RowGroup` `total_byte_size` field meaning [as](https://github.com/apache/parquet-format/blob/737ea12e56357e83b14fd3e27ef274145beed399/src/main/thrift/parquet.thrift#L920C7-L920C76)
   > Total byte size of all the uncompressed column data in this row group
   
   This is also the case for C++ implementation of parquet in arrow repo.
   
   But in case of Go implementation `total_byte_size` is described [as](https://github.com/apache/arrow-go/blob/14844aea32054a0b7cc086df58a4a74610b0b306/parquet/metadata/row_group.go#L62)
   > TotalByteSize is the total size of this rowgroup on disk
   
   The difference between these values can be large, when compression is applied to column chunks.
   
   My question is: Is that intentional inconsistency with format definition and other implementations? And if so, why does this distinction has been made?
   
   ### Component(s)
   
   Go, Parquet
   
   P.S.:
   This issue is duplicate of https://github.com/apache/arrow/issues/44205, but as I see this repo is now main location of Arrow Go


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Go][Parquet] Inconsistency in RowGroup 'total byte size' field values between implementations [arrow-go]

Posted by "abbit (via GitHub)" <gi...@apache.org>.
abbit commented on issue #142:
URL: https://github.com/apache/arrow-go/issues/142#issuecomment-2380613881

   This is strange, because from my experiments with different parquet implementations (including Go and C++ through Python from arrow repo) I've seen that C++ one writes correct `total_byte_size`.
   
   Also thank you for such quick fix!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Go][Parquet] Inconsistency in RowGroup 'total byte size' field values between implementations [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #142:
URL: https://github.com/apache/arrow-go/issues/142#issuecomment-2374269796

   Honestly, I think it was just a misunderstanding on my part when I wrote this and no one else noticed or caught it. 
   
   It seems like an easy fix honestly, aside from the comment, it should be fairly easy to fix the `RowGroupMetaDataBuilder` so that it uses the correct value to write for it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Go][Parquet] Inconsistency in RowGroup 'total byte size' field values between implementations [arrow-go]

Posted by "abbit (via GitHub)" <gi...@apache.org>.
abbit commented on issue #142:
URL: https://github.com/apache/arrow-go/issues/142#issuecomment-2374880741

   Thanks for reply! I'll try to take a look, but no promises :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Go][Parquet] Inconsistency in RowGroup 'total byte size' field values between implementations [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #142:
URL: https://github.com/apache/arrow-go/issues/142#issuecomment-2379428934

   So looking at the C++ implementation again, it looks like despite the comment *saying* that `total_byte_size` is the uncompressed size of all the columns, it is actually populating it with the *total_bytes_written*, i.e. the total size on disk. Given that I based much of the Go implementation on the C++ implementation, it's not surprising that this bug exists in both. 
   
   We should file an equivalent issue on the Arrow repo for the C++ parquet implementation. In the meantime i'll fix the Go impl


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org