You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2023/09/07 18:18:35 UTC
[GitHub] [beam] Abacn opened a new issue, #28359: [Task]: Switch to portable Data and Time type for JdbcIO
Abacn opened a new issue, #28359:
URL: https://github.com/apache/beam/issues/28359
### What needs to happen?
Currently there are still a couple of non-portable logical types defined in https://github.com/apache/beam/blob/926774dd02be5eacbe899ee5eceab23afb30abca/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/LogicalTypes.java
This prevents cross-lang JdbcIO to read / write rows with these types. Most commonly used are Date and Time types. We should migrate them to portable logical types and also support them in Python side.
### Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Switch to portable Data and Time type for JdbcIO [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #28359:
URL: https://github.com/apache/beam/issues/28359#issuecomment-2427223697
Thanks, just update the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Switch to portable Data and Time type for JdbcIO [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28359:
URL: https://github.com/apache/beam/issues/28359#issuecomment-5509745663
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Switch to portable Data and Time type for JdbcIO [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #28359: [Task]: Switch to portable Data and Time type for JdbcIO
URL: https://github.com/apache/beam/issues/28359
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Switch to portable Data and Time type for JdbcIO [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #28359:
URL: https://github.com/apache/beam/issues/28359#issuecomment-3482083379
We should still do this preferrably in Beam 3. Currently JdbcIO Date and Time types are not supported by Beam SQL as a result of this issue:
We should eliminate most non-portable logical types in https://github.com/apache/beam/blob/master/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/LogicalTypes.java
including Date and Time. Add this to Beam 3 milestone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #28359: [Task]: Switch to portable Data and Time type for JdbcIO
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #28359:
URL: https://github.com/apache/beam/issues/28359#issuecomment-1711874721
Note: the following workaround can be used for columns involving Date (or Time) type, that is register your own logical type:
```python
@LogicalType.register_logical_type
class DateType(LogicalType[datetime.date, MillisInstant, str]):
def __init__(self, unused=""):
pass
@classmethod
def representation_type(cls):
# type: () -> type
return Timestamp
@classmethod
def urn(cls):
return "beam:logical_type:javasdk:v1"
@classmethod
def language_type(cls):
return datetime.date
def to_representation_type(self, value):
# type: (datetime.date) -> Timestamp
return Timestamp.from_utc_datetime(datetime.datetime.combine(value, datetime.datetime.min.time(), tzinfo=datetime.timezone.utc))
def to_language_type(self, value):
# type: (Timestamp) -> datetime.date
return value.to_utc_datetime().date()
@classmethod
def argument_type(cls):
return str
def argument(self):
return ""
@classmethod
def _from_typing(cls, typ):
return cls()
```
-----
I recall why the Date type support was incomplete. The Java JdbcIO implemented Date and Time with non-portable logical type backed by joda Instant, while the Beam portable Date and Time logical type are backed by more mortan java.time.localDate or localTime. We cannot simply change the Java JdbcIO due to concern of breaking change. This involves two difficult compatibility issues
- migrating from joda time to java.time - [go/beampr/19215](https://goto.google.com/beampr/19215)
- migrate from non-portable to portable logical types
One fix could be done for transition is that adding a flag to IOs that guide them to produce java.time results and use portable logical types. This would enable Python implementation (and also eliminate the need of call `LogicalType.register_logical_type(MillisInstant)` if need to overwrite logical type mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Switch to portable Data and Time type for JdbcIO [beam]
Posted by "lazarillo (via GitHub)" <gi...@apache.org>.
lazarillo commented on issue #28359:
URL: https://github.com/apache/beam/issues/28359#issuecomment-2426442281
The link provided in migrating from joda time to java.time points to a PR that is behind Google's corp firewall.
It would help if the issues were exposed to the OS community, since it's an OS issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Switch to portable Data and Time type for JdbcIO [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28359:
URL: https://github.com/apache/beam/issues/28359#issuecomment-5143195600
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org