You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Matthew Ouyang (Jira)" <ji...@apache.org> on 2021/03/22 15:29:00 UTC
[jira] [Comment Edited] (BEAM-9613) BigQuery IO not support convert
double type for beam row
[ https://issues.apache.org/jira/browse/BEAM-9613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292299#comment-17292299 ]
Matthew Ouyang edited comment on BEAM-9613 at 3/22/21, 3:28 PM:
----------------------------------------------------------------
This phenomenon also happens with the boolean type in BigQuery. When a result set references a BQ BOOLEAN (or FLOAT64 in Maksim's case), the TableRow will have a value of type Boolean (Double) which will not work with BigQueryUtils. BigQueryUtils works with BQ INT64.
-My guess is that the BigQueryIO.Read calls the [BQ jobs.query API|https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query]. Follow the links for {{rows[]}} and the protobuf documentation suggests that boolean and float64 are represented by the JSON Boolean and Number types respectively. All other types relevant to BQ appear to be treated as strings.- (Edit Mar 22 guess appears to be wrong after some more investigation.)
was (Author: mouyang):
This phenomenon also happens with the boolean type in BigQuery. When a result set references a BQ BOOLEAN (or FLOAT64 in Maksim's case), the TableRow will have a value of type Boolean (Double) which will not work with BigQueryUtils. BigQueryUtils works with BQ INT64.
My guess is that the BigQueryIO.Read calls the [BQ jobs.query API|https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query]. Follow the links for {{rows[]}} and the protobuf documentation suggests that boolean and float64 are represented by the JSON Boolean and Number types respectively. All other types relevant to BQ appear to be treated as strings.
> BigQuery IO not support convert double type for beam row
> --------------------------------------------------------
>
> Key: BEAM-9613
> URL: https://issues.apache.org/jira/browse/BEAM-9613
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Reporter: MAKSIM TSYGAN
> Priority: P3
>
> If execute query with double column via BigQueryIO.readFrom(), I get exception:
> Caused by: java.lang.UnsupportedOperationException: Converting BigQuery type 'class java.lang.Double' to 'FieldType\{typeName=DOUBLE, nullable=true, logicalType=null, collectionElementType=null, mapKeyType=null, mapValueType=null, rowSchema=null, metadata={}}' is not supported
> at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamValue(BigQueryUtils.java:532)
> at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRowFieldValue(BigQueryUtils.java:483)
> at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.lambda$toBeamRow$6(BigQueryUtils.java:469)
> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
> at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRow(BigQueryUtils.java:470)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)