You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Ismaël Mejía (Jira)" <ji...@apache.org> on 2021/05/07 13:52:00 UTC

[jira] [Created] (BEAM-12306) Add a native ParquetIO.ReadRows transform

Ismaël Mejía created BEAM-12306:
-----------------------------------

             Summary: Add a native ParquetIO.ReadRows transform
                 Key: BEAM-12306
                 URL: https://issues.apache.org/jira/browse/BEAM-12306
             Project: Beam
          Issue Type: Improvement
          Components: dsl-sql, io-java-parquet
            Reporter: Ismaël Mejía


ParquetIO today uses Avro's GenericRecord to materialize reads from Parquet records, this implies an extra conversion from GenericRecord to Beam's Row. We can 'push' this conversion into the source read process by implementing a RecordMaterializer that builds Rows at read time xtending Parquet's [ReadSupport|https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/api/ReadSupport.java].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)