You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Kenneth Knowles (JIRA)" <ji...@apache.org> on 2019/07/25 04:21:00 UTC

[jira] [Commented] (BEAM-7755) BigQuery Repeated Records do not seem to work

    [ https://issues.apache.org/jira/browse/BEAM-7755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892397#comment-16892397 ] 

Kenneth Knowles commented on BEAM-7755:
---------------------------------------

I think I have seen comments on this area of the code from [~kedin] [~reuvenlax] [~rarokni@gmail.com] too.

> BigQuery Repeated Records do not seem to work
> ---------------------------------------------
>
>                 Key: BEAM-7755
>                 URL: https://issues.apache.org/jira/browse/BEAM-7755
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-avro, io-java-gcp
>    Affects Versions: 2.12.0, 2.13.0
>            Reporter: Sahith Nallapareddy
>            Assignee: Sahith Nallapareddy
>            Priority: Major
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> When translating BigQuery rows to beam rows, specifically using theĀ  BigQueryUtils.toBeamRow(record, beamSchema) method, REPEATEDĀ RECORDS causes an error. This seems to be caused that avro arrays are thought to only have primitive types but these are arrays with a ROW type:
> {noformat}
> Caused by: java.lang.RuntimeException: ROW is not primitive type.
> 	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.convertAvroPrimitiveTypes(BigQueryUtils.java:467)
> 	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.convertAvroArray(BigQueryUtils.java:427)
> 	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.convertAvroFormat(BigQueryUtils.java:373)
> 	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.lambda$toBeamRow$2(BigQueryUtils.java:222)
> 	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> 	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1376)
> 	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> 	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> 	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> 	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> 	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> 	at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRow(BigQueryUtils.java:223)
> 	at com.spotify.data.sql.RowSource.lambda$bigquery$120a5f9f$1(RowSource.java:55)
> 	at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:242)
> 	at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase$1.apply(BigQuerySourceBase.java:235)
> 	at org.apache.beam.sdk.io.AvroSource$AvroBlock.readNextRecord(AvroSource.java:597)
> 	at org.apache.beam.sdk.io.BlockBasedSource$BlockBasedReader.readNextRecord(BlockBasedSource.java:209)
> 	at org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:484)
> 	at org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:479)
> 	at org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:249)
> 	at org.apache.beam.runners.dataflow.worker.WorkerCustomSources$BoundedReaderIterator.start(WorkerCustomSources.java:601)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)