You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2023/05/20 01:29:22 UTC

[GitHub] [beam] Abacn commented on issue #26796: [Bug]: Records get silently dropped in BigQueryIO storage write when timestamp (possibly) invalid

Abacn commented on issue #26796:
URL: https://github.com/apache/beam/issues/26796#issuecomment-1555413523

   Ok, this seems WAI as all messages are written into DLQ actually. The following exception is suppressed:
   
   ```
   org.apache.beam.sdk.Pipeline$PipelineExecutionException: org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto$SchemaDoesntMatchException: Problem converting field root.time_stamp_value expected type: TIMESTAMP. Exception: java.lang.NumberFormatException: For input string: "2009-01-03 12:34:56+00"
   	at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:374)
   Caused by: org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto$SchemaDoesntMatchException: Problem converting field root.time_stamp_value expected type: TIMESTAMP. Exception: java.lang.NumberFormatException: For input string: "2009-01-03 12:34:56+00"
   	at org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto.messageFromMap(TableRowToStorageApiProto.java:487)
   	at org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto.messageFromTableRow(TableRowToStorageApiProto.java:589)
   	at org.apache.beam.sdk.io.gcp.bigquery.StorageApiDynamicDestinationsTableRow$TableRowConverter.toMessage(StorageApiDynamicDestinationsTableRow.java:152)
   	at org.apache.beam.sdk.io.gcp.bigquery.StorageApiDynamicDestinationsTableRow$TableRowConverter.toMessage(StorageApiDynamicDestinationsTableRow.java:140)
   	at org.apache.beam.sdk.io.gcp.bigquery.StorageApiConvertMessages$ConvertMessagesDoFn.processElement(StorageApiConvertMessages.java:142)
   Caused by: java.lang.NumberFormatException: For input string: "2009-01-03 12:34:56+00"
   	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
   	at java.lang.Long.parseLong(Long.java:589)
   	at java.lang.Long.parseLong(Long.java:631)
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org