You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/10/14 07:37:36 UTC

[GitHub] [beam] algirdas-k opened a new issue, #23636: [Bug]: BQ Storage Write API converter fails for small timestamps with ArithmeticException: long overflow

algirdas-k opened a new issue, #23636:
URL: https://github.com/apache/beam/issues/23636

   ### What happened?
   
   While trying to write to `TIMESTAMP` column in BQ a small value (for example `0001-01-01T00:00:00Z`), Storage Write API converter utility `TableRowToStorageApiProto` fails to convert from `String` to proto `INT64` with an error:
   
   ```
   "org.apache.beam.sdk.io.gcp.bigquery.TableRowToStorageApiProto$SchemaDoesntMatchException: 
   Problem converting field __root__.event_timestamp expected type: TIMESTAMP. 
   Exception: java.lang.ArithmeticException: long overflow"
   ```
   Reproduced on Apache Beam 2.41.0 SDK on GCP Dataflow.
   
   This is converter issue, as BQ perfectly supports such dates.
   
   ---
   
   After some investigation it seems problem was introduced with [[BEAM-13990] support date and timestamp fields](https://github.com/apache/beam/pull/17404/files#top), where [time conversion is done via](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java#L441-L457):
   ```
   return ChronoUnit.MICROS.between(Instant.EPOCH, Instant.parse((String) value));
   ```
   Internally, this hits [this](https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/time/Instant.java#L1154) and [this](https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/time/Instant.java#L1175) with values being `-62135596800 * 1000_000_000L`, which results `ArithmeticException: long overflow`.
   
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: io-java-gcp


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] algirdas-k commented on issue #23636: [Bug]: BQ Storage Write API converter fails for small timestamps with ArithmeticException: long overflow

Posted by "algirdas-k (via GitHub)" <gi...@apache.org>.
algirdas-k commented on issue #23636:
URL: https://github.com/apache/beam/issues/23636#issuecomment-1439533271

   This issue was solved in later JDK versions, example https://github.com/openjdk/jdk/commit/81d2acee57188a4507c798b46b0bd129dc302fec, closing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] kennknowles commented on issue #23636: [Bug]: BQ Storage Write API converter fails for small timestamps with ArithmeticException: long overflow

Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #23636:
URL: https://github.com/apache/beam/issues/23636#issuecomment-1335394344

   Nice investigation! Since you understand the issue very well, would you be interested in trying to fix it? :-) :-) :-)
   
   Also pinging @reuvenlax since he knows the original logic.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] algirdas-k closed issue #23636: [Bug]: BQ Storage Write API converter fails for small timestamps with ArithmeticException: long overflow

Posted by "algirdas-k (via GitHub)" <gi...@apache.org>.
algirdas-k closed issue #23636: [Bug]: BQ Storage Write API converter fails for small timestamps with ArithmeticException: long overflow
URL: https://github.com/apache/beam/issues/23636


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] algirdas-k commented on issue #23636: [Bug]: BQ Storage Write API converter fails for small timestamps with ArithmeticException: long overflow

Posted by GitBox <gi...@apache.org>.
algirdas-k commented on issue #23636:
URL: https://github.com/apache/beam/issues/23636#issuecomment-1336204536

   @kennknowles :D 
   
   Here we go then: https://github.com/apache/beam/pull/24514 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] kennknowles commented on issue #23636: [Bug]: BQ Storage Write API converter fails for small timestamps with ArithmeticException: long overflow

Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #23636:
URL: https://github.com/apache/beam/issues/23636#issuecomment-1338087329

   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org