You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/08/12 18:00:09 UTC
[jira] [Work logged] (BEAM-12479) UnsupportedOperationException
when reading from BigQuery tables and converting TableRows to Beam Rows
[ https://issues.apache.org/jira/browse/BEAM-12479?focusedWorklogId=637462&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-637462 ]
ASF GitHub Bot logged work on BEAM-12479:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Aug/21 17:59
Start Date: 12/Aug/21 17:59
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #14997:
URL: https://github.com/apache/beam/pull/14997#issuecomment-897852984
LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 637462)
Time Spent: 20m (was: 10m)
> UnsupportedOperationException when reading from BigQuery tables and converting TableRows to Beam Rows
> -----------------------------------------------------------------------------------------------------
>
> Key: BEAM-12479
> URL: https://issues.apache.org/jira/browse/BEAM-12479
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.29.0
> Environment: Beam Java SDK
> Reporter: Pascal GILLET
> Priority: P2
> Labels: pull-request-available
> Fix For: 2.33.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{UnsupportedOperationExceptions}} are thrown in
> {{org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils#toBeamValue(FieldType, Object)}}
> when reading from BigQuery tables with
> {{org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO#readTableRowsWithSchema()}} and converting the returned {{TableRows}} to Beam {{Rows}}
> Example:
> {code:java}
> PCollection<Row> rows =
> pipeline
> .apply(
> "Read from BigQuery table",
> BigQueryIO.readTableRowsWithSchema().from(String.format("%s:%s.%s", project, dataset, table)))
> .apply(Convert.toRows());{code}
>
> {{UnsupportedOperationException}} messages that I have encountered are of the type:
> {{Converting BigQuery type "java.lang.Boolean" to "BOOLEAN" is not supported}}
> {{Converting BigQuery type "java.lang.Double" to "DOUBLE" is not supported}}
> ...While the conversion of these Java types should be straightforward.
> Indeed, the method {{BigQueryUtils#toBeamValue(FieldType, Object)}} expects only {{String}} objects or {{Collections}} of {{Strings}}.
> I had to upgrade {{com.google.cloud:google-cloud-bigquery}} from 1.108.0 to 1.132.0 in my project. So my guess is this latest version is now able to map BiqQuery (SQL) types to Java types instead of raw Strings, in particular BOOL to Boolean, INT64 to Long and FLOAT64 to Double.
> In my understanding though, from BigQuery to Beam, there would be no need to manage Java Byte, Short, Integer and Float since BigQuery types are "limited" to standard SQL types INT64, and FLOAT64 that encompass them all (on the other hand, BQ NUMERIC type is mapped to Java BigDecimal).
> I propose a pull request to also support {{Number}} and {{Boolean}} objects in method {{BigQueryUtils#toBeamValue(FieldType, Object)}}. It is only added behavior and the updated method is still compatible with the current functioning.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)