You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/10/22 09:51:59 UTC
[GitHub] [beam] JozoVilcek opened a new pull request #13166: [BEAM-11091] Allow to specify coder for HadoopFormatIO.Read
JozoVilcek opened a new pull request #13166:
URL: https://github.com/apache/beam/pull/13166
[BEAM-11091] Allow to specify coder for HadoopFormatIO.Read
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] JozoVilcek commented on pull request #13166: [BEAM-11091] Allow to specify coder for HadoopFormatIO.Read
Posted by GitBox <gi...@apache.org>.
JozoVilcek commented on pull request #13166:
URL: https://github.com/apache/beam/pull/13166#issuecomment-719429707
R: @aromanenko-dev
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev merged pull request #13166: [BEAM-11091] Allow to specify coder for HadoopFormatIO.Read
Posted by GitBox <gi...@apache.org>.
aromanenko-dev merged pull request #13166:
URL: https://github.com/apache/beam/pull/13166
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a change in pull request #13166: [BEAM-11091] Allow to specify coder for HadoopFormatIO.Read
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on a change in pull request #13166:
URL: https://github.com/apache/beam/pull/13166#discussion_r516840231
##########
File path: sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOReadTest.java
##########
@@ -229,6 +284,14 @@ public void testReadObjectCreationFailsIfValueTranslationFunctionIsNull() {
HadoopFormatIO.<Text, String>read().withConfiguration(serConf.get()).withValueTranslation(null);
}
+ @Test
+ public void testReadObjectCreationFailsIfValueCoderFunctionIsNull() {
Review comment:
nit: I guess the word `Function` is redundant here (like you called a similar key test `testReadObjectCreationFailsIfKeyCoderIsNull `)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on pull request #13166: [BEAM-11091] Allow to specify coder for HadoopFormatIO.Read
Posted by GitBox <gi...@apache.org>.
lukecwik commented on pull request #13166:
URL: https://github.com/apache/beam/pull/13166#issuecomment-714584591
I won't be able to do another round of review though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #13166: [BEAM-11091] Allow to specify coder for HadoopFormatIO.Read
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #13166:
URL: https://github.com/apache/beam/pull/13166#issuecomment-721684594
Run Java HadoopFormatIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] JozoVilcek commented on pull request #13166: [BEAM-11091] Allow to specify coder for HadoopFormatIO.Read
Posted by GitBox <gi...@apache.org>.
JozoVilcek commented on pull request #13166:
URL: https://github.com/apache/beam/pull/13166#issuecomment-719371222
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #13166: [BEAM-11091] Allow to specify coder for HadoopFormatIO.Read
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #13166:
URL: https://github.com/apache/beam/pull/13166#discussion_r510266391
##########
File path: sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java
##########
@@ -437,6 +445,12 @@
.build();
}
+ /** Transforms the keys read from the source using the given key translation function. */
+ public Read<K, V> withKeyTranslation(SimpleFunction<?, K> function, Coder<K> coder) {
Review comment:
Since we don't need the type information anymore, we should use SerializableFunction instead of SimpleFunction. SimpleFunction implements SerializableFunction so this change enables using simpler lambdas.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] JozoVilcek commented on a change in pull request #13166: [BEAM-11091] Allow to specify coder for HadoopFormatIO.Read
Posted by GitBox <gi...@apache.org>.
JozoVilcek commented on a change in pull request #13166:
URL: https://github.com/apache/beam/pull/13166#discussion_r514255701
##########
File path: sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java
##########
@@ -437,6 +445,12 @@
.build();
}
+ /** Transforms the keys read from the source using the given key translation function. */
+ public Read<K, V> withKeyTranslation(SimpleFunction<?, K> function, Coder<K> coder) {
Review comment:
Good catch. Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] JozoVilcek commented on a change in pull request #13166: [BEAM-11091] Allow to specify coder for HadoopFormatIO.Read
Posted by GitBox <gi...@apache.org>.
JozoVilcek commented on a change in pull request #13166:
URL: https://github.com/apache/beam/pull/13166#discussion_r514257586
##########
File path: sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java
##########
@@ -437,6 +445,12 @@
.build();
}
+ /** Transforms the keys read from the source using the given key translation function. */
+ public Read<K, V> withKeyTranslation(SimpleFunction<?, K> function, Coder<K> coder) {
Review comment:
I did try to pass in `SerializableFunction` but inner code of HadoopFormat does require input and output types for the function for validation, e.g. that input format value class does match translation function input class. I choose to stick with `SimpleFunction` but just allow to pass in the specific coder.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #13166: [BEAM-11091] Allow to specify coder for HadoopFormatIO.Read
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #13166:
URL: https://github.com/apache/beam/pull/13166#discussion_r510266391
##########
File path: sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java
##########
@@ -437,6 +445,12 @@
.build();
}
+ /** Transforms the keys read from the source using the given key translation function. */
+ public Read<K, V> withKeyTranslation(SimpleFunction<?, K> function, Coder<K> coder) {
Review comment:
Since we don't need the type information anymore, we should use SerializableFunction instead of SimpleFunction.
##########
File path: sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java
##########
@@ -437,6 +445,12 @@
.build();
}
+ /** Transforms the keys read from the source using the given key translation function. */
+ public Read<K, V> withKeyTranslation(SimpleFunction<?, K> function, Coder<K> coder) {
Review comment:
You need to clear the keyCoder/valueCoder in the non coder based variants otherwise we won't honor the typedescriptor when the user changes the translation function
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #13166: [BEAM-11091] Allow to specify coder for HadoopFormatIO.Read
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on pull request #13166:
URL: https://github.com/apache/beam/pull/13166#issuecomment-721277553
Run Java HadoopFormatIO Performance Test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] JozoVilcek commented on a change in pull request #13166: [BEAM-11091] Allow to specify coder for HadoopFormatIO.Read
Posted by GitBox <gi...@apache.org>.
JozoVilcek commented on a change in pull request #13166:
URL: https://github.com/apache/beam/pull/13166#discussion_r516925444
##########
File path: sdks/java/io/hadoop-format/src/test/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIOReadTest.java
##########
@@ -229,6 +284,14 @@ public void testReadObjectCreationFailsIfValueTranslationFunctionIsNull() {
HadoopFormatIO.<Text, String>read().withConfiguration(serConf.get()).withValueTranslation(null);
}
+ @Test
+ public void testReadObjectCreationFailsIfValueCoderFunctionIsNull() {
Review comment:
I agree. Improved by next commit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org