You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/10/19 07:36:00 UTC
[jira] [Commented] (PARQUET-1928) Interpret Parquet INT96 type as
FIXED[12] AVRO Schema
[ https://issues.apache.org/jira/browse/PARQUET-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17216502#comment-17216502 ]
ASF GitHub Bot commented on PARQUET-1928:
-----------------------------------------
anantdamle opened a new pull request #831:
URL: https://github.com/apache/parquet-mr/pull/831
Make sure you have checked _all_ steps below.
Reading Parquet files in Apache Beam using ParquetIO uses `AvroParquetReader` causing it to throw `IllegalArgumentException("INT96 not implemented and is deprecated")`
Customers have large datasets which can't be reprocessed again to convert into a supported type. An easier approach would be to convert into a byte array of 12 bytes, that can then be interpreted by the developer in any way they want to interpret it.
This patch interprets the INT96 parquet type as a byte array of 12-bytes. the developer/user can then handle it appropriate to interpret into a timestamp or simple some bytes.
- [x ] My PR adds the following unit tests `testParquetInt96AsFixed12AvroType`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Interpret Parquet INT96 type as FIXED[12] AVRO Schema
> -----------------------------------------------------
>
> Key: PARQUET-1928
> URL: https://issues.apache.org/jira/browse/PARQUET-1928
> Project: Parquet
> Issue Type: Bug
> Components: parquet-avro
> Reporter: Anant Damle
> Priority: Minor
> Labels: patch
>
> Reading Parquet files in Apache Beam using ParquetIO uses `AvroParquetReader` causing it to throw `IllegalArgumentException("INT96 not implemented and is deprecated")`
> Customers have large datasets which can't be reprocessed again to convert into a supported type. An easier approach would be to convert into a byte array of 12 bytes, that can then be interpreted by the developer in any way they want to interpret it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)