You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/06/11 16:47:57 UTC
[GitHub] [beam] lostluck opened a new pull request #11989: [BEAM-9615] Add String UTF8 coder.
lostluck opened a new pull request #11989:
URL: https://github.com/apache/beam/pull/11989
Adds support for the StringUTF8 beam coder to the Go SDK.
This is functionally how strings were encoded before, but now we use the standard URN rather than a custom coder to wrap it.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Samza | Spark
--- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #11989: [BEAM-9615] Add String UTF8 coder.
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #11989:
URL: https://github.com/apache/beam/pull/11989#discussion_r439546660
##########
File path: sdks/go/pkg/beam/core/runtime/coderx/string.go
##########
@@ -23,6 +23,8 @@ import (
// NewString returns a coder for the string type. It uses the native
// []byte to string conversion.
+//
+// Only for custom coder test use.
func NewString() (*coder.CustomCoder, error) {
Review comment:
Note that most things that aren't in the main `beam` package are really intended for user use. I'll send a separate PR moving/removing this code, I just didn't want this PR to become too messy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #11989: [BEAM-9615] Add String UTF8 coder.
Posted by GitBox <gi...@apache.org>.
lostluck commented on pull request #11989:
URL: https://github.com/apache/beam/pull/11989#issuecomment-643546739
That's been filed as https://issues.apache.org/jira/browse/BEAM-7009
And I have the rest of schemas to finish up which will definely require not putting that off any further.. In the precursor CLto this, we do verify against "canonical" encoded bytes that exist in the Java encoding test for the same. https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/graph/coder/stringutf8_test.go
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #11989: [BEAM-9615] Add String UTF8 coder.
Posted by GitBox <gi...@apache.org>.
lostluck commented on pull request #11989:
URL: https://github.com/apache/beam/pull/11989#issuecomment-642842184
R: @tysonjh @youngoli
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #11989: [BEAM-9615] Add String UTF8 coder.
Posted by GitBox <gi...@apache.org>.
lostluck commented on pull request #11989:
URL: https://github.com/apache/beam/pull/11989#issuecomment-642815708
Run Go PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #11989: [BEAM-9615] Add String UTF8 coder.
Posted by GitBox <gi...@apache.org>.
lostluck commented on pull request #11989:
URL: https://github.com/apache/beam/pull/11989#issuecomment-642803751
Run Go PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #11989: [BEAM-9615] Add String UTF8 coder.
Posted by GitBox <gi...@apache.org>.
lostluck commented on pull request #11989:
URL: https://github.com/apache/beam/pull/11989#issuecomment-642831227
Run Go PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #11989: [BEAM-9615] Add String UTF8 coder.
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #11989:
URL: https://github.com/apache/beam/pull/11989#issuecomment-643512623
Can go add a test which imports standard_coders.yaml and ensures the encoding matches the bytes and vice versa?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck edited a comment on pull request #11989: [BEAM-9615] Add String UTF8 coder.
Posted by GitBox <gi...@apache.org>.
lostluck edited a comment on pull request #11989:
URL: https://github.com/apache/beam/pull/11989#issuecomment-643546739
@lukecwik That's been filed as https://issues.apache.org/jira/browse/BEAM-7009
And I have the rest of schemas to finish up which will definely require not putting that off any further.. In the precursor CLto this, we do verify against "canonical" encoded bytes that exist in the Java encoding test for the same. https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/core/graph/coder/stringutf8_test.go
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] youngoli commented on a change in pull request #11989: [BEAM-9615] Add String UTF8 coder.
Posted by GitBox <gi...@apache.org>.
youngoli commented on a change in pull request #11989:
URL: https://github.com/apache/beam/pull/11989#discussion_r439145448
##########
File path: sdks/go/pkg/beam/core/runtime/coderx/string.go
##########
@@ -23,6 +23,8 @@ import (
// NewString returns a coder for the string type. It uses the native
// []byte to string conversion.
+//
+// Only for custom coder test use.
func NewString() (*coder.CustomCoder, error) {
Review comment:
If this is only meant to be used for custom coder tests, should it just be renamed instead? It seems unlikely to be used by user code right now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck merged pull request #11989: [BEAM-9615] Add String UTF8 coder.
Posted by GitBox <gi...@apache.org>.
lostluck merged pull request #11989:
URL: https://github.com/apache/beam/pull/11989
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org