You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/01/03 23:23:00 UTC
[jira] [Work logged] (BEAM-3684) Update well-known coder URNs in Go
SDK
[ https://issues.apache.org/jira/browse/BEAM-3684?focusedWorklogId=366089&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-366089 ]
ASF GitHub Bot logged work on BEAM-3684:
----------------------------------------
Author: ASF GitHub Bot
Created on: 03/Jan/20 23:22
Start Date: 03/Jan/20 23:22
Worklog Time Spent: 10m
Work Description: lukecwik commented on pull request #10501: [BEAM-3684] Add support for channel splitting to the gRPC read "source" and propagate "split" calls to the downstream receiver
URL: https://github.com/apache/beam/pull/10501
This code mirrors the logic/implementation within [Python](https://github.com/apache/beam/blob/16757ef9a6da4d0ac218c6c4d6b19e2a49ccca45/sdks/python/apache_beam/runners/worker/bundle_processor.py#L206).
To be able to propagate the split call to the downstream receiver, I collapsed all the harness FnDataReceiver types into two existing implementations and one new implementations.
The previous hierarchy was:
element counting receiver -> time counting receiver -> multiplexing receiver (possibly the original receiver)
The current implementation combined the element counting, time counting and multiplexing into the MultiplexingMetricTrackingFnDataReceiver while for the singleton case into the MetricTrackingFnDataReceiver.
To propagate splits, a SplittingMetricTrackingFnDataReceiver was created that extends the MetricTrackingFnDataReceiver. Note, like in [Python](https://github.com/apache/beam/blob/c167d8ef99b21148bcab7c37538a6ef2f64864c7/sdks/python/apache_beam/runners/worker/operations.py#L133), there is currently no support for splitting a multiplexed consumer.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 366089)
Time Spent: 1h 10m (was: 1h)
> Update well-known coder URNs in Go SDK
> --------------------------------------
>
> Key: BEAM-3684
> URL: https://issues.apache.org/jira/browse/BEAM-3684
> Project: Beam
> Issue Type: Bug
> Components: sdk-go
> Reporter: Eugene Kirpichov
> Assignee: Henning Rohde
> Priority: Major
> Labels: portability
> Fix For: Not applicable
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoderRegistrar.java] has recently changed the URNs of well-known coders. The Go SDK needs to update accordingly https://github.com/apache/beam/blob/go-sdk/sdks/go/pkg/beam/core/runtime/graphx/coder.go#L34
--
This message was sent by Atlassian Jira
(v8.3.4#803005)