You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/08/06 16:30:54 UTC

[GitHub] [beam] TheNeuralBit opened a new pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

TheNeuralBit opened a new pull request #12481:
URL: https://github.com/apache/beam/pull/12481


   - 16bc9bb: The bulk of the change. Changes external_transform.proto to encode a Schema and a Row, and modifies Java's ExpansionService, and Python's SchemaBasedPayloadBuilders, to use the new representation.
   - 5ff29d8: Changes related to KafkaIO. Since KV is not supported in schemas, KafkaIO configuration types are changed to use `Map<String, String>` instead of `List<KV<String, String>>`. Also updates references to external_transform.proto in KafkaIOExternalTest.
   - 0ef3701: Updates references to external_transform.proto in PubsubIOExternalTest.
   - b8936db: Updates references to external_transform.proto in XVR test fixtures (Java and Python).
   
   Some notes about the implementation:
   - Since KV is not supported by Rows, I've updated the tests that used them in this context to use a Map instead. In many unit tests this is not a true replacement, but I think it's valid since the only application that currently requires KV is encoding a Map as a List<KV>.
   - For users of Python's SchemaBasedPayloadBuilder and implementors of Java's ExternalTransformBuilder the change should be transparent (with the exception that they can no longer use KV).
   - For ExternalTransformBuilder implementations it is now possible to register a Schema for ConfigT and we will use that to populate an instance rather than the existing setter approach.
   - Due to BEAM-10632 I've temporarily added a testCompile dependency on checker framework to :sdks:java:expansion-service so that I can still test the schema inference capability.
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
   Java | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](htt
 ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_
 Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_P
 ostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
   XLang | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   --- |Java | Python | Go | Website
   --- | --- | --- | --- | ---
   Non-portable | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/b
 eam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
   Portable | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   ![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-672256723


   Run Java PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-670046044


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-670076791


   retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on a change in pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12481:
URL: https://github.com/apache/beam/pull/12481#discussion_r466592633



##########
File path: runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java
##########
@@ -222,14 +228,21 @@ public void partitionTest() {
   }
 
   private byte[] toStringPayloadBytes(String data) throws IOException {
+    Row configRow =
+        Row.withSchema(Schema.of(Field.of("data", FieldType.STRING)))
+            .withFieldValue("data", data)
+            .build();
+
+    ByteString.Output outputStream = ByteString.newOutput();
+    try {
+      RowCoder.of(configRow.getSchema()).encode(configRow, outputStream);
+    } catch (IOException e) {
+      throw new RuntimeException(e);
+    }
     ExternalTransforms.ExternalConfigurationPayload payload =
         ExternalTransforms.ExternalConfigurationPayload.newBuilder()
-            .putConfiguration(
-                "data",
-                ExternalTransforms.ConfigValue.newBuilder()
-                    .addCoderUrn("beam:coder:string_utf8:v1")
-                    .setPayload(ByteString.copyFrom(encodeString(data)))
-                    .build())
+            .setSchema(SchemaTranslation.schemaToProto(configRow.getSchema(), false))
+            .setPayload(outputStream.toByteString())

Review comment:
       This is just mirroring what was there before, which used the old ExternalConfigurationPayload to encode a single field "data" with the string_utf8 coder. The equivalent with the new version  of ExternalConfigurationPayload is to encode a single field "data" with type STRING.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit merged pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit merged pull request #12481:
URL: https://github.com/apache/beam/pull/12481


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-672308417


   PreCommit seems to have passed, just no update here: https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/2603/
   I've also validated this with some internal test suites. I'll go ahead and merge. Thank you @amaliujia and @mxm for the reviews!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-670710921


   Run Java PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-670787524


   Run Python PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] amaliujia commented on a change in pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #12481:
URL: https://github.com/apache/beam/pull/12481#discussion_r466614630



##########
File path: runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java
##########
@@ -222,14 +228,21 @@ public void partitionTest() {
   }
 
   private byte[] toStringPayloadBytes(String data) throws IOException {
+    Row configRow =
+        Row.withSchema(Schema.of(Field.of("data", FieldType.STRING)))
+            .withFieldValue("data", data)
+            .build();
+
+    ByteString.Output outputStream = ByteString.newOutput();
+    try {
+      RowCoder.of(configRow.getSchema()).encode(configRow, outputStream);
+    } catch (IOException e) {
+      throw new RuntimeException(e);
+    }
     ExternalTransforms.ExternalConfigurationPayload payload =
         ExternalTransforms.ExternalConfigurationPayload.newBuilder()
-            .putConfiguration(
-                "data",
-                ExternalTransforms.ConfigValue.newBuilder()
-                    .addCoderUrn("beam:coder:string_utf8:v1")
-                    .setPayload(ByteString.copyFrom(encodeString(data)))
-                    .build())
+            .setSchema(SchemaTranslation.schemaToProto(configRow.getSchema(), false))
+            .setPayload(outputStream.toByteString())

Review comment:
       Got it. Thanks for clarification.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit edited a comment on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit edited a comment on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-670704670


   @mxm or @chadrik any chance you'd be able to review the Python changes in this PR? I'd like to get it merged before the 2.24.0 release cut on Aug 12


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] mxm commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
mxm commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-672083483






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on a change in pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #12481:
URL: https://github.com/apache/beam/pull/12481#discussion_r466611999



##########
File path: sdks/java/expansion-service/build.gradle
##########
@@ -44,6 +44,8 @@ dependencies {
   compile library.java.slf4j_api
   runtimeOnly library.java.slf4j_jdk14
   testCompile library.java.junit
+  // TODO(BEAM-10632): Remove this. Currently Schema inference (used in ExpansionServiceTest) hits an NPE when checker is enabled, and checkerframework is not in the classpath.

Review comment:
       Done, thanks




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-670788340


   Run XVR_Direct PostCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-670704406


   Run PythonDocker PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-670037799


   Run XVR_Direct PostCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-670704670


   @mxm or @chadrik any chance you'd be able to review the Python changes in this PR? I'd like to get it merge before the 2.24.0 release cut on Aug 12


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] mxm commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
mxm commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-672083732


   Run XVR_Direct PostCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] amaliujia commented on a change in pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
amaliujia commented on a change in pull request #12481:
URL: https://github.com/apache/beam/pull/12481#discussion_r466586877



##########
File path: sdks/java/expansion-service/build.gradle
##########
@@ -44,6 +44,8 @@ dependencies {
   compile library.java.slf4j_api
   runtimeOnly library.java.slf4j_jdk14
   testCompile library.java.junit
+  // TODO(BEAM-10632): Remove this. Currently Schema inference (used in ExpansionServiceTest) hits an NPE when checker is enabled, and checkerframework is not in the classpath.

Review comment:
       Split this comment to multiple lines?

##########
File path: runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ValidateRunnerXlangTest.java
##########
@@ -222,14 +228,21 @@ public void partitionTest() {
   }
 
   private byte[] toStringPayloadBytes(String data) throws IOException {
+    Row configRow =
+        Row.withSchema(Schema.of(Field.of("data", FieldType.STRING)))
+            .withFieldValue("data", data)
+            .build();
+
+    ByteString.Output outputStream = ByteString.newOutput();
+    try {
+      RowCoder.of(configRow.getSchema()).encode(configRow, outputStream);
+    } catch (IOException e) {
+      throw new RuntimeException(e);
+    }
     ExternalTransforms.ExternalConfigurationPayload payload =
         ExternalTransforms.ExternalConfigurationPayload.newBuilder()
-            .putConfiguration(
-                "data",
-                ExternalTransforms.ConfigValue.newBuilder()
-                    .addCoderUrn("beam:coder:string_utf8:v1")
-                    .setPayload(ByteString.copyFrom(encodeString(data)))
-                    .build())
+            .setSchema(SchemaTranslation.schemaToProto(configRow.getSchema(), false))
+            .setPayload(outputStream.toByteString())

Review comment:
       So this is a design decision?
   
   this configRow is a Row with a String field and the payload is already a string/byte array. At least in this case it seems that without a scheme that payload can still be constructed back to a Row. 
   
   
   The schema looks right in general when payload is not a string/byte array




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-672106456






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] mxm commented on a change in pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #12481:
URL: https://github.com/apache/beam/pull/12481#discussion_r467556781



##########
File path: sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java
##########
@@ -1452,12 +1446,12 @@ public void populateDisplayData(DisplayData.Builder builder) {
       public static class Configuration {
 
         // All byte arrays are UTF-8 encoded strings
-        private Iterable<KV<String, String>> producerConfig;
+        private Map<String, String> producerConfig;
         private String topic;
         private String keySerializer;
         private String valueSerializer;

Review comment:
       Comment should be updated (no byte arrays anymore).
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] mxm commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
mxm commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-671026344


   Awesome @TheNeuralBit! I'll check this out in more detail beginning of the week.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-670735024


   Run XVR_Direct PostCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on pull request #12481: [BEAM-10571] Use schemas in ExternalConfigurationPayload

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12481:
URL: https://github.com/apache/beam/pull/12481#issuecomment-672186752


   Run Java PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org