You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/08/06 16:31:00 UTC
[jira] [Work logged] (BEAM-10571) Use schemas for external protocol
spec
[ https://issues.apache.org/jira/browse/BEAM-10571?focusedWorklogId=467386&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-467386 ]
ASF GitHub Bot logged work on BEAM-10571:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 06/Aug/20 16:30
Start Date: 06/Aug/20 16:30
Worklog Time Spent: 10m
Work Description: TheNeuralBit opened a new pull request #12481:
URL: https://github.com/apache/beam/pull/12481
- 16bc9bb: The bulk of the change. Changes external_transform.proto to encode a Schema and a Row, and modifies Java's ExpansionService, and Python's SchemaBasedPayloadBuilders, to use the new representation.
- 5ff29d8: Changes related to KafkaIO. Since KV is not supported in schemas, KafkaIO configuration types are changed to use `Map<String, String>` instead of `List<KV<String, String>>`. Also updates references to external_transform.proto in KafkaIOExternalTest.
- 0ef3701: Updates references to external_transform.proto in PubsubIOExternalTest.
- b8936db: Updates references to external_transform.proto in XVR test fixtures (Java and Python).
Some notes about the implementation:
- Since KV is not supported by Rows, I've updated the tests that used them in this context to use a Map instead. In many unit tests this is not a true replacement, but I think it's valid since the only application that currently requires KV is encoding a Map as a List<KV>.
- For users of Python's SchemaBasedPayloadBuilder and implementors of Java's ExternalTransformBuilder the change should be transparent (with the exception that they can no longer use KV).
- For ExternalTransformBuilder implementations it is now possible to register a Schema for ConfigT and we will use that to populate an instance rather than the existing setter approach.
- Due to BEAM-10632 I've temporarily added a testCompile dependency on checker framework to :sdks:java:expansion-service so that I can still test the schema inference capability.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 467386)
Remaining Estimate: 0h
Time Spent: 10m
> Use schemas for external protocol spec
> --------------------------------------
>
> Key: BEAM-10571
> URL: https://issues.apache.org/jira/browse/BEAM-10571
> Project: Beam
> Issue Type: Improvement
> Components: cross-language
> Reporter: Brian Hulette
> Assignee: Brian Hulette
> Priority: P2
> Fix For: 2.24.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Targeting 2.24.0 for this as we'd like to have xlang protos stabilized for it. It looks like it will be the last Python 2 release, and we want to have xlang as an escape hatch for using python 2.7.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)