You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 16:15:51 UTC
[GitHub] [beam] damccorm opened a new issue, #20205: AvroUtils is converting incorrectly LogicalType Timestamps from long into Joda DateTimes
damccorm opened a new issue, #20205:
URL: https://github.com/apache/beam/issues/20205
Copied from the mailing list report:
I think the method AvroUtils.toBeamSchema has a not expected side effect.
I found out that, if you invoke it and then you run a pipeline of GenericRecords containing a timestamp (l tried with logical-type timestamp-millis), Beam converts such timestamp from long to org.joda.time.DateTime. Even if you don't apply any transformation to the pipeline.
Do you think it's a bug?
More details on how to reproduce here:
https://lists.apache.org/thread.html/r43fb2896e496b7493a962207eb3b95360abc30b9d091b26f110264d0%40%3Cuser.beam.apache.org%3E
Imported from Jira [BEAM-9863](https://issues.apache.org/jira/browse/BEAM-9863). Original Jira may contain additional context.
Reported by: iemejia.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] RustedBones commented on issue #20205: AvroUtils is converting incorrectly LogicalType Timestamps from long into Joda DateTimes
Posted by GitBox <gi...@apache.org>.
RustedBones commented on issue #20205:
URL: https://github.com/apache/beam/issues/20205#issuecomment-1212844873
As stated in #16271
> [BEAM-9144](https://issues.apache.org/jira/browse/BEAM-9144) is a 'snowflake'. All other joda logical time conversions are still failing. (date, time-millis, time-micros, timestampt-micros). This should not be in the framework but in the user code
This should definitely be out the framework since this create side effect by simply having the class in scope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] slouc commented on issue #20205: AvroUtils is converting incorrectly LogicalType Timestamps from long into Joda DateTimes
Posted by GitBox <gi...@apache.org>.
slouc commented on issue #20205:
URL: https://github.com/apache/beam/issues/20205#issuecomment-1211963386
I have the same problem, although I haven't noticed `AvroUtils.toBeamSchema` to be the culprit, but rather [this conversion in the AvroUtils static block](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java#L109).
Seems like the same thing though. I have also shown how it can cause runtime exceptions when building the avro record.
Here's a small repo that contains more info and demonstrates the problem: https://github.com/slouc/avro-beam-test.
BTW apache beam is using avro 1.8.2, which is the problematic one. If we instead explicitly bring avro 1.11.1 (latest version right now) into our dependency tree, the problem goes away.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org