You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Reuven Lax (JIRA)" <ji...@apache.org> on 2018/12/02 06:29:00 UTC

[jira] [Commented] (BEAM-5807) Add Schema support for AVRO

    [ https://issues.apache.org/jira/browse/BEAM-5807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16706115#comment-16706115 ] 

Reuven Lax commented on BEAM-5807:
----------------------------------

AvroIO.readRows isn't necessary. As long as AvroIO can set a schema on the downstream PCollection, the existing AvroIO signatures are fine.

> Add Schema support for AVRO
> ---------------------------
>
>                 Key: BEAM-5807
>                 URL: https://issues.apache.org/jira/browse/BEAM-5807
>             Project: Beam
>          Issue Type: Improvement
>          Components: dsl-sql
>            Reporter: Gleb Kanterov
>            Assignee: Gleb Kanterov
>            Priority: Major
>          Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> # Motivation
> At the moment the only way to read AVRO is through code generation with avro-compiler and JavaBeanSchema. It makes it not possible to write transforms that can work with dynamic schemas. AVRO has generic data type called GenericRecord, reading is implemented in AvroIO.
> readGenericRecords. There is a code to convert GenericRecord to Row shipped as a part of BigQueryIO. However, it doesn't support all types and nested records.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)