You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/08/14 21:16:00 UTC
[jira] [Work logged] (BEAM-7009) Add Go SDK test for Standard
Coders using the yaml data.
[ https://issues.apache.org/jira/browse/BEAM-7009?focusedWorklogId=470889&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-470889 ]
ASF GitHub Bot logged work on BEAM-7009:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Aug/20 21:15
Start Date: 14/Aug/20 21:15
Worklog Time Spent: 10m
Work Description: lostluck opened a new pull request #12588:
URL: https://github.com/apache/beam/pull/12588
This PR has the Go SDK join Java and Python in having its coders tested against the standard_coder.yaml suite.
* It adds representation markers for standard types that previously didn't have representation (such as iterables, rows, length prefixes, etc) in the graph/coder package.
* It adds execution wrappers for encoding, decoding those values in the exec package.
* It adds serialization support for these representations in the graphx package.
* Adds hashing support for row coded values.
Testing of the code here is largely through the newly introduced fromyaml test harness which reads standard_coder.yaml and helps it along a touch, in particular in how to interpret the yaml versions of expected structures, which don't map to go types exactly.
While this PR plumbs in Timers and ParamWindowCoders, these coders are not implemented in the SDK at this time, and those tests are skipped. At best, the implementation is vestigial, but leaving them wholely unimplemented caused certain difficulties with the standard yaml test.
Similarly, while Windows are now fully plumbed through the stack, this is largely for convenience when testing those codings vs the standard_coders.yaml file.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 470889)
Remaining Estimate: 0h
Time Spent: 10m
> Add Go SDK test for Standard Coders using the yaml data.
> --------------------------------------------------------
>
> Key: BEAM-7009
> URL: https://issues.apache.org/jira/browse/BEAM-7009
> Project: Beam
> Issue Type: Bug
> Components: beam-model, sdk-go
> Reporter: Robert Burke
> Assignee: Robert Burke
> Priority: P1
> Labels: stale-P2, stale-assigned
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The Go SDK doesn't currently do this validation, against the standard yaml file. [1]
> The Java and Python equivalents of the test can be found from here [2].
>
> Care would need to be taken so that Beam Go SDK users (such as they are) aren't forced to run them, and not have the yaml file to read. I'd suggest putting it with the integration tests [3].
> The other thing of note is that the Go SDK has no notion of "nested" vs "unnested" coders. All coders are "nested" in the Go SDK, and should have their lengths prefixed to them as appropriate.
> 1: [https://github.com/apache/beam/blob/master/model/fn-execution/src/main/resources/org/apache/beam/model/fnexecution/v1/standard_coders.yaml]
> 2: [https://github.com/apache/beam/search?q=standard_coders.yaml&unscoped_q=standard_coders.yaml]
> 3: [https://github.com/apache/beam/tree/master/sdks/go/test |https://github.com/apache/beam/tree/master/sdks/go/test]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)