You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/10/01 08:14:00 UTC
[jira] [Work logged] (BEAM-10655) Cannot write
"google.protobuf.Timestamp" (NanosInstant) to BigQuery through schemas
[ https://issues.apache.org/jira/browse/BEAM-10655?focusedWorklogId=658808&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-658808 ]
ASF GitHub Bot logged work on BEAM-10655:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 01/Oct/21 08:13
Start Date: 01/Oct/21 08:13
Worklog Time Spent: 10m
Work Description: blackhogz commented on a change in pull request #15485:
URL: https://github.com/apache/beam/pull/15485#discussion_r719102764
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java
##########
@@ -551,9 +548,9 @@ public static TableRow toTableRow(Row row) {
return toTableRow((Row) fieldValue);
case DATETIME:
- return ((Instant) fieldValue)
- .toDateTime(DateTimeZone.UTC)
- .toString(BIGQUERY_TIMESTAMP_PRINTER);
+ org.joda.time.Instant jodaInstant = (org.joda.time.Instant) fieldValue;
+ java.time.Instant javaInstant = java.time.Instant.ofEpochMilli(jodaInstant.getMillis());
+ return BIGQUERY_TIMESTAMP_PRINTER.format(javaInstant);
Review comment:
Got it. IIUC, this means we should not error out in `BigQueryUtils.toTableSchema()` but rather check the flag and error out case by case in `BigQueryUtils.toTableRow()` similar to the snippet you linked.
I think @amuletxheart and I will have to defer the re-architecting work to the maintainers who definitely have a much better understanding of the overall architecture than us, even though I agree passing an argument to `toTableRow()` is a bit messy.
That said, @TheNeuralBit are you okay with the proposed `BigQueryIO.Write#allowTruncatedTimestamps()` change, as I think this is kinda part of the "public API" of the BigQueryIO, while `toTableRow()` is an private implementation that is easier to change when we later undergo the re-architecturing work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 658808)
Time Spent: 3h 40m (was: 3.5h)
> Cannot write "google.protobuf.Timestamp" (NanosInstant) to BigQuery through schemas
> -----------------------------------------------------------------------------------
>
> Key: BEAM-10655
> URL: https://issues.apache.org/jira/browse/BEAM-10655
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Reporter: Chamikara Madhusanka Jayalath
> Assignee: Xinshen Yong
> Priority: P2
> Labels: starter
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> See here for details and instructions for reproducing.
> [https://lists.apache.org/thread.html/r1657ccf41dda3f2d9d082c5ebb006dd2da92863983971ad23485f16e%40%3Cdev.beam.apache.org%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)