You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Kenneth Knowles (Jira)" <ji...@apache.org> on 2021/05/05 16:50:00 UTC
[jira] [Commented] (BEAM-10021) BigQueryUtils DATETIME conversion
requires unix millis
[ https://issues.apache.org/jira/browse/BEAM-10021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339771#comment-17339771 ]
Kenneth Knowles commented on BEAM-10021:
----------------------------------------
If I understand correctly from chatting with you outside the bug, this was in the context of SQL, so SQL owns how to configure the BigQueryIO. Can you share which version? [~robinyqiu] may have fixed some issues here. There may also be a flag you need to pass. It seems likely that this has to do with the fact that Beam/Joda/Calcite all use milliseconds while BigQuery uses microseconds, so we give you the choice of truncating the value versus failing the job (so as not to write incorrect results, if the precision matters).
> BigQueryUtils DATETIME conversion requires unix millis
> ------------------------------------------------------
>
> Key: BEAM-10021
> URL: https://issues.apache.org/jira/browse/BEAM-10021
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Reporter: Ben Birt
> Assignee: Chamikara Madhusanka Jayalath
> Priority: P2
>
> I'm trying to run a very simple GCP Dataflow SQL job (copy all columns from table A to B), and it's failing with the following exception:
> {code:java}
> Caused by: java.lang.ClassCastException: org.apache.avro.util.Utf8 cannot be cast to java.lang.Longat org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.truncateToMillis (BigQueryUtils.java:643)at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.convertAvroFormat (BigQueryUtils.java:597)at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.lambda$toBeamRow$5 (BigQueryUtils.java:389)
> {code}
> It looks to me like this is because BigQueryUtils.convertAvroFormat expects that any value whose (beam) type is "DATETIME" must be expressed in unix millis. However, these values come out of BigQuery as a string, (AFAICT) in ISO 8601 format.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)