You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 19:58:06 UTC
[GitHub] [beam] damccorm opened a new issue, #20865: BeamCalcRel implementation of date/time types aren't always Integer
damccorm opened a new issue, #20865:
URL: https://github.com/apache/beam/issues/20865
For some reason date and time offsets aren't always represented in generated code as Integer, sometimes they get unconverted to Long. This inconsistent behavior may lead to bugs. We should figure out the root cause.
Imported from Jira [BEAM-12175](https://issues.apache.org/jira/browse/BEAM-12175). Original Jira may contain additional context.
Reported by: apilloud.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] BeamCalcRel implementation of date/time types aren't always Integer [beam]
Posted by "vampire-hunter-lincoln (via GitHub)" <gi...@apache.org>.
vampire-hunter-lincoln commented on issue #20865:
URL: https://github.com/apache/beam/issues/20865#issuecomment-3499252489
Hi @damccorm,
**Context**
My team is using [logical DATE](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/logicaltypes/SqlTypes.java#L33)(`LogicalType<LocalDate, Long>`) to represent `DATE` in Beam SQL(backed by Apache Calcite), but I find [this comment](https://github.com/Abacn/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java#L422).
**Analysis**
After reading some related source code, here is what happened for logical DATE(`LogicalType<LocalDate, Long>`) processing in Beam SQL according to my understanding, please correct me if I am wrong:
1. Since Beam configures Calcite to use java.lang.Long as the physical type for Calcite DATE, it means the compiled code will expect a Long object to hold value of Calcite DATE. Janino, as a Java compiler, will compile the expression according to Java's type rules.
2. During `Beam value` -> `Calcite Value`, an Integer value is extracted from the Java LocalDate([code](https://github.com/Abacn/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java#L668)). We provide this integer as a Calcite DATE input in [Janino evaluation step](https://github.com/Abacn/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java#L317).
3. During Janino evaluation step, if any conversion(expression) happens to this Integer, the result will be saved into a Long object and return a Long as output. Otherwise, it will still be the same Integer as output. That can explain the [original comment](https://github.com/Abacn/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java#L422).
**Solution**
So I believe, we can modify the step 2 above, extract a Long value from the Java LocalDate during `Beam value` -> `Calcite Value`(those two lines: [line1](https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java#L671) and [line2](https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel.java#L681)). Then everything will be consistent as Long since both the value and definition will be Long.
**Notes**
This issue is same as https://github.com/apache/beam/issues/20864
Please let me know if you have any thoughts! Thank you!
CC: @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org