You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 12:53:47 UTC
[GitHub] [beam] damccorm opened a new issue, #19715: Support logical types when translating the portable row coder representation
damccorm opened a new issue, #19715:
URL: https://github.com/apache/beam/issues/19715
Originally SchemaCoder and RowCoder relied on including serialized java classes and functions to support logical types, but the [portable schema representation](https://s.apache.org/beam-schema) only includes a URN for a logical type. We will need to be able to re-construct logical types given just a URN, presumably by creating some sort of logical type registry indexed by URN.
Imported from Jira [BEAM-7855](https://issues.apache.org/jira/browse/BEAM-7855). Original Jira may contain additional context.
Reported by: bhulette.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn closed issue #19715: Support logical types when translating the portable row coder representation
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #19715: Support logical types when translating the portable row coder representation
URL: https://github.com/apache/beam/issues/19715
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #19715: Support logical types when translating the portable row coder representation
Posted by GitBox <gi...@apache.org>.
Abacn commented on issue #19715:
URL: https://github.com/apache/beam/issues/19715#issuecomment-1232193783
From the TODO mark in source:
https://github.com/apache/beam/blob/3ede5b76e48b41e89bc67541ea5044ebe704e905/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaTranslation.java#L183
Any known but yet standardized logical types (those in STANDARD_LOGICAL_TYPES) will assign the same urn "beam:logical_type:javasdk:v1". These include all jdbc logical types and also many defined in `org.apache.beam.sdk.schemas.logicaltypes` package. While user can register this URN and implement with there own class in a cross-language pipeline, (e.g. https://stackoverflow.com/questions/68758361/error-beamlogical-typejavasdkv1-while-using-apache-beam-io-jdbc-readfromjdbc), they can do at most one logical type because they share the same URN. Would be preferrable to encode the identifier in URN as well, for example, in the example stackoverflow case, The involved logical type from Java side is `org.apache.beam.sdk.io.jdbc.LogicalTypes.VariableLengthString` with identifier 'VARCHAR', It would then have URN "beam:logical_type:javasdk_varchar::v1" or "beam:logical_type:javasdk_variablelengthstring::v1" (or "beam:logical_type:javasdk:varchar::v1" or "beam:logical_type:javasdk:variablel
engthstring::v1")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org