You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/08/13 23:04:06 UTC
[GitHub] [beam] TheNeuralBit opened a new pull request #12573: [BEAM-10009] Add support for logical types and beam:logical_type:millis_instant:v1 in Python schemas
TheNeuralBit opened a new pull request #12573:
URL: https://github.com/apache/beam/pull/12573
This PR adds support for logical types in Python schemas, and adds the first logical type implementation there: for `beam:logical_type:millis_instant:v1`.
I chose `apache_beam.utils.timestamp.Timestamp` as the Python type mapping for millis_instant . Note that `Timestamp` is represented as microseconds since the epoch, so this is a lossy mapping, but there is precedent for this in Python's `WindowedValueCoder`: https://github.com/apache/beam/blob/699f872ea1ef3bdb1588a029fc6b1e3185e986a6/sdks/python/apache_beam/coders/coder_impl.py#L1236-L1240
Also in this PR:
- some changes to the Java SDK implementation of Beam schemas to make its `FieldType.DATETIME` type behave exactly like `beam:logical_type:millis_instant:v1` when used in portabillity (afa939e). `DATETIME` will be deprecated and removed in favor of a Java implementation of millis_instant in the future (BEAM-7554).
- Unit testing in schemas_test and row_coder_test.
- Tests using Timestamp in the Python SqlTransform.
- Test cases in standard_coders_test.yaml (the first logical type tested there).
Note the Python implementation of LogicalTypes in this PR is limited to those that do not use an argument.
R: @robertwb
CC: @lostluck
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit closed pull request #12573: [BEAM-10009] Add support for logical types and beam:logical_type:millis_instant:v1 in Python schemas
Posted by GitBox <gi...@apache.org>.
TheNeuralBit closed pull request #12573:
URL: https://github.com/apache/beam/pull/12573
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12573: [BEAM-10009] Add support for logical types and beam:logical_type:millis_instant:v1 in Python schemas
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12573:
URL: https://github.com/apache/beam/pull/12573#issuecomment-673775240
Run XVR_Direct PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12573: [BEAM-10009] Add support for logical types and beam:logical_type:millis_instant:v1 in Python schemas
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12573:
URL: https://github.com/apache/beam/pull/12573#issuecomment-686085823
Closing this in favor of #12764 which adds a micros_instant logical type to more faithfully represent `Timestamp`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12573: [BEAM-10009] Add support for logical types and beam:logical_type:millis_instant:v1 in Python schemas
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12573:
URL: https://github.com/apache/beam/pull/12573#issuecomment-677773121
@robertwb, @robinyqiu - In this PR I've mapped python's Timestamp (microsecond precision) to the millis_instant logical type lossily so it could be mapped to DATETIME on the Java side and used in SQL.
Based on discussion in https://github.com/apache/beam/pull/12493, now I'm thinking the better answer would be to add a micros_instant logical type so Python's data can go across the xlang boundary faithfully, and it's up to SQL to throw an error when the timestamp precision is too high, or silently drop precision if the user chooses.
There's an open question about what to do if the output of the SqlTransform includes a millis_instant logical type. Would that still get mapped to a python Timestamp?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org