[jira] [Commented] (BEAM-4081) Review of schema metadata vs schema types

["Kenneth Knowles (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-4081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16491833#comment-16491833 ] 

Kenneth Knowles commented on BEAM-4081:
---------------------------------------

Digging into the code, I see there are a number of issues and probably some standard ways to resolve them.

1. We need to be able to convert Beam schemas to something Calcite understands. This should live in Beam SQL. Right now it is partially addressed by Beam type with metadata. When converting back, the SQL statement should retain enough metadata that Beam does not need to propagate it (as when you compile a typed language to assembly, etc, the higher-level language can handle managing the metadata).
2. We need to convert Calcite types to Beam schemas. This is fine to do by just an embedding.
3. A lot of things are done with type names instead of types. I'm fixing that up.

I'm working through it. There are a lot of coupled decisions that make basic changes pretty sweeping.

> Review of schema metadata vs schema types
> -----------------------------------------
>
>                 Key: BEAM-4081
>                 URL: https://issues.apache.org/jira/browse/BEAM-4081
>             Project: Beam
>          Issue Type: Sub-task
>          Components: sdk-java-core
>            Reporter: Kenneth Knowles
>            Priority: Major
>
> The Schema basic types have a place for metadata that can say "this int is really millis since epoch". This deserves some careful design review and perhaps some of these need to be promoted to basic types.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)