You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/11/07 02:29:29 UTC

[GitHub] [beam] ihji opened a new pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

ihji opened a new pull request #13283:
URL: https://github.com/apache/beam/pull/13283


   …ner v2
   
   **Please** add a meaningful description for your change here
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
    - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
   
   Post-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
   --- | --- | --- | --- | --- | --- | ---
   Go | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
   Java | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/badge/i
 con)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](htt
 ps://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
   Python | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[![Build Status](https://ci-beam
 .apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
   XLang | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
   
   Pre-Commit Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   
   --- |Java | Python | Go | Website | Whitespace | Typescript
   --- | --- | --- | --- | --- | --- | ---
   Non-portable | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/be
 am_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
   Portable | --- | [![Build Status](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
   
   See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
   
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji removed a comment on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji removed a comment on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-723412944






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji removed a comment on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji removed a comment on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-723377029


   run XVR_flink postcommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-723412944


   run xvr_direct postcommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz commented on a change in pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
boyuanzz commented on a change in pull request #13283:
URL: https://github.com/apache/beam/pull/13283#discussion_r524523450



##########
File path: runners/google-cloud-dataflow-java/build.gradle
##########
@@ -312,6 +313,36 @@ task validatesRunnerStreaming {
   ))
 }
 
+createCrossLanguageValidatesRunnerTask(

Review comment:
       This task makes all Dataflow runner v2 tests fail because the sdk container is cleaned up as soon as it is built. The cause is that in the common configuration, you have
   config.startJobServer.finalizedBy config.cleanupJobServer
   This will change the task dependency incorrectly.
   
   Filed jira here: https://issues.apache.org/jira/browse/BEAM-11270
   
   cc: @tysonjh




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on a change in pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #13283:
URL: https://github.com/apache/beam/pull/13283#discussion_r521581974



##########
File path: sdks/python/apache_beam/transforms/ptransform.py
##########
@@ -695,9 +695,9 @@ def from_runner_api(cls,
     # type: (...) -> Optional[PTransform]
     if proto is None or proto.spec is None or not proto.spec.urn:
       return None
-    parameter_type, constructor = cls._known_urns[proto.spec.urn]
 
     try:
+      parameter_type, constructor = cls._known_urns[proto.spec.urn]

Review comment:
       It was for fixing Java xlang pipelines but the fix is still incomplete anyway. Will revert this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-726432960


   Run Python PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on a change in pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #13283:
URL: https://github.com/apache/beam/pull/13283#discussion_r521580860



##########
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
##########
@@ -542,19 +593,32 @@ def run_pipeline(self, pipeline, options):
       # TODO(chamikara): remove following pipeline and pipeline proto recreation
       # after portable job submission path is fully in place.
       from apache_beam import Pipeline
-      pipeline = Pipeline.from_runner_api(
+      pipeline, src_context = Pipeline.from_runner_api(
           self.proto_pipeline,
           pipeline.runner,
           options,
+          return_context=True,
           allow_proto_holders=True)
 
       # Pipelines generated from proto do not have output set to PDone set for
       # leaf elements.
       pipeline.visit(self._set_pdone_visitor(pipeline))
 
+      from apache_beam.runners import pipeline_context
+      dst_context = pipeline_context.PipelineContext(
+          component_id_map=pipeline.component_id_map,
+          default_environment=self._default_environment)
+
+      # Copy external environments to prevent dangling environment ids
+      pipeline.visit(

Review comment:
       Python pipeline using python external transform doesn't work without handling dangling environment ids.
   
   It's not because of duplicated environments. It's because of missing environments. Dangling environment IDs are generated when 1) PTransform URN from external environment is known (therefore `RunnerAPIPTransformHolder` won't work as expected) 2) PipelineContext from rehydration process is not used in the second proto conversion. Python external transforms in Python pipelines are rehydrated to `AppliedPTransform` since external PTransform URN is known to the SDK however AppliedPTransform only keeps an environment id not environment itself. Environments are saved separately in PipelineContext but we dropped the PipelineContext from rehydration in the second proto conversion.
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-725699557


   run seed job


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-723376932


   run seed job


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on a change in pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #13283:
URL: https://github.com/apache/beam/pull/13283#discussion_r522410940



##########
File path: .test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Dataflow.groovy
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the suite of ValidatesRunner tests against the Dataflow runner.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_XVR_Dataflow',
+    'Run XVR_Dataflow PostCommit', 'Dataflow CrossLanguageValidatesRunner Tests', this) {
+      description('Runs the CrossLanguageValidatesRunner suite on the Dataflow runner.')
+
+      // Set common parameters.
+      commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+      // Publish all test results to Jenkins
+      publishers {
+        archiveJunit('**/nosetests*.xml')
+      }
+
+      // Gradle goals for this job.
+      // TODO: We only support Python pipeline using Java external transforms at the moment.
+      // Enable testing for Java pipeline using Python external transforms when BEAM-11203
+      // is implemented.
+      steps {
+        shell('echo "*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA EXTERNAL TRANSFORMS USING PYTHON 3.6 ***"')
+        gradle {
+          rootBuildScriptDir(commonJobProperties.checkoutDir)
+          tasks(':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava')
+          commonJobProperties.setGradleSwitches(delegate)
+          switches('-PpythonVersion=3.6')
+        }
+        shell('echo "*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA EXTERNAL TRANSFORMS USING PYTHON 3.8 ***"')
+        gradle {
+          rootBuildScriptDir(commonJobProperties.checkoutDir)
+          tasks(':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava')
+          commonJobProperties.setGradleSwitches(delegate)
+          switches('-PpythonVersion=3.8')
+        }
+        shell('echo "*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA SQL TRANSFORMS USING PYTHON 3.8 ***"')

Review comment:
       I think it's not necessary. Kafka xlang test is already a part of python postcommit tests: https://github.com/apache/beam/blob/master/sdks/python/test-suites/portable/common.gradle#L177




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-726327513


   run seed job


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-726351889


   Run Java_Examples_Dataflow PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-725629510


   This should run the SQL tests as well since they are added as part of createCrossLanguageValidatesRunnerTask right?
   
   https://github.com/apache/beam/blob/88d1696b6a4bb0b98eb289e9118ac4064127378e/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1980
   
   If so this will also address BEAM-10011


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-723492962


   R: @chamikaramj @robertwb 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-723377029


   run XVR_flink postcommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji removed a comment on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji removed a comment on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-725699557






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] chamikaramj commented on a change in pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #13283:
URL: https://github.com/apache/beam/pull/13283#discussion_r521618026



##########
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
##########
@@ -542,19 +593,32 @@ def run_pipeline(self, pipeline, options):
       # TODO(chamikara): remove following pipeline and pipeline proto recreation
       # after portable job submission path is fully in place.
       from apache_beam import Pipeline
-      pipeline = Pipeline.from_runner_api(
+      pipeline, src_context = Pipeline.from_runner_api(
           self.proto_pipeline,
           pipeline.runner,
           options,
+          return_context=True,
           allow_proto_holders=True)
 
       # Pipelines generated from proto do not have output set to PDone set for
       # leaf elements.
       pipeline.visit(self._set_pdone_visitor(pipeline))
 
+      from apache_beam.runners import pipeline_context
+      dst_context = pipeline_context.PipelineContext(
+          component_id_map=pipeline.component_id_map,
+          default_environment=self._default_environment)
+
+      # Copy external environments to prevent dangling environment ids
+      pipeline.visit(

Review comment:
       "RunnerAPIPTransformHolder" is expected to go away soon (when we enable portable job submission). Do you think we can skip this test (that use Python external transforms in a Python pipeline) for now and try without these updates when we have portable job submission ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-723413408


   run xvr_dataflow postcommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on a change in pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #13283:
URL: https://github.com/apache/beam/pull/13283#discussion_r522453982



##########
File path: .test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Dataflow.groovy
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the suite of ValidatesRunner tests against the Dataflow runner.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_XVR_Dataflow',
+    'Run XVR_Dataflow PostCommit', 'Dataflow CrossLanguageValidatesRunner Tests', this) {
+      description('Runs the CrossLanguageValidatesRunner suite on the Dataflow runner.')
+
+      // Set common parameters.
+      commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+      // Publish all test results to Jenkins
+      publishers {
+        archiveJunit('**/nosetests*.xml')
+      }
+
+      // Gradle goals for this job.
+      // TODO: We only support Python pipeline using Java external transforms at the moment.
+      // Enable testing for Java pipeline using Python external transforms when BEAM-11203
+      // is implemented.
+      steps {
+        shell('echo "*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA EXTERNAL TRANSFORMS USING PYTHON 3.6 ***"')
+        gradle {
+          rootBuildScriptDir(commonJobProperties.checkoutDir)
+          tasks(':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava')
+          commonJobProperties.setGradleSwitches(delegate)
+          switches('-PpythonVersion=3.6')
+        }
+        shell('echo "*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA EXTERNAL TRANSFORMS USING PYTHON 3.8 ***"')
+        gradle {
+          rootBuildScriptDir(commonJobProperties.checkoutDir)
+          tasks(':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava')
+          commonJobProperties.setGradleSwitches(delegate)
+          switches('-PpythonVersion=3.8')
+        }
+        shell('echo "*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA SQL TRANSFORMS USING PYTHON 3.8 ***"')

Review comment:
       Oh, that's right. Good catch 😄
   Will take a look if we can easily add Kafka xlang test too.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] boyuanzz commented on a change in pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
boyuanzz commented on a change in pull request #13283:
URL: https://github.com/apache/beam/pull/13283#discussion_r524523450



##########
File path: runners/google-cloud-dataflow-java/build.gradle
##########
@@ -312,6 +313,36 @@ task validatesRunnerStreaming {
   ))
 }
 
+createCrossLanguageValidatesRunnerTask(

Review comment:
       This task makes all Dataflow runner v2 tests fail because the sdk container is cleaned up as soon as it is built. The cause is that in the common configuration, you have
   `config.startJobServer.finalizedBy config.cleanupJobServer`
   This will change the task dependency incorrectly.
   
   Filed jira here: https://issues.apache.org/jira/browse/BEAM-11270
   
   cc: @tysonjh




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji removed a comment on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji removed a comment on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-723377652


   run XVR_dataflow postcommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji removed a comment on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji removed a comment on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-723413408


   run xvr_dataflow postcommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] chamikaramj commented on a change in pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #13283:
URL: https://github.com/apache/beam/pull/13283#discussion_r521541029



##########
File path: sdks/python/setup.py
##########
@@ -256,70 +256,71 @@ def run(self):
       'Python %s.%s. You may encounter bugs or missing features.' % (
           sys.version_info.major, sys.version_info.minor))
 
-setuptools.setup(
-    name=PACKAGE_NAME,
-    version=PACKAGE_VERSION,
-    description=PACKAGE_DESCRIPTION,
-    long_description=PACKAGE_LONG_DESCRIPTION,
-    url=PACKAGE_URL,
-    download_url=PACKAGE_DOWNLOAD_URL,
-    author=PACKAGE_AUTHOR,
-    author_email=PACKAGE_EMAIL,
-    packages=setuptools.find_packages(),
-    package_data={'apache_beam': [
-        '*/*.pyx', '*/*/*.pyx', '*/*.pxd', '*/*/*.pxd', '*/*.h', '*/*/*.h',
-        'testing/data/*.yaml', 'portability/api/*.yaml']},
-    ext_modules=cythonize([
-        'apache_beam/**/*.pyx',
-        'apache_beam/coders/coder_impl.py',
-        'apache_beam/metrics/cells.py',
-        'apache_beam/metrics/execution.py',
-        'apache_beam/runners/common.py',
-        'apache_beam/runners/worker/logger.py',
-        'apache_beam/runners/worker/opcounters.py',
-        'apache_beam/runners/worker/operations.py',
-        'apache_beam/transforms/cy_combiners.py',
-        'apache_beam/utils/counters.py',
-        'apache_beam/utils/windowed_value.py',
-    ]),
-    install_requires=REQUIRED_PACKAGES,
-    python_requires=python_requires,
-    test_suite='nose.collector',
-    # BEAM-8840: Do NOT use tests_require or setup_requires.
-    extras_require={
-        'docs': ['Sphinx>=1.5.2,<2.0'],
-        'test': REQUIRED_TEST_PACKAGES,
-        'gcp': GCP_REQUIREMENTS,
-        'interactive': INTERACTIVE_BEAM,
-        'interactive_test': INTERACTIVE_BEAM_TEST,
-        'aws': AWS_REQUIREMENTS,
-        'azure': AZURE_REQUIREMENTS
-    },
-    zip_safe=False,
-    # PyPI package information.
-    classifiers=[
-        'Intended Audience :: End Users/Desktop',
-        'License :: OSI Approved :: Apache Software License',
-        'Operating System :: POSIX :: Linux',
-        'Programming Language :: Python :: 3.6',
-        'Programming Language :: Python :: 3.7',
-        'Programming Language :: Python :: 3.8',
-        # When updating vesion classifiers, also update version warnings
-        # above and in apache_beam/__init__.py.
-        'Topic :: Software Development :: Libraries',
-        'Topic :: Software Development :: Libraries :: Python Modules',
-    ],
-    license='Apache License, Version 2.0',
-    keywords=PACKAGE_KEYWORDS,
-    entry_points={
-        'nose.plugins.0.10': [
-            'beam_test_plugin = test_config:BeamTestPlugin',
-        ]},
-    cmdclass={
-        'build_py': generate_protos_first(build_py),
-        'develop': generate_protos_first(develop),
-        'egg_info': generate_protos_first(egg_info),
-        'test': generate_protos_first(test),
-        'mypy': generate_protos_first(mypy),
-    },
-)
+if __name__ == '__main__':

Review comment:
       This probably should be a separate PR.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] chamikaramj commented on a change in pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #13283:
URL: https://github.com/apache/beam/pull/13283#discussion_r521540557



##########
File path: sdks/python/apache_beam/transforms/ptransform.py
##########
@@ -695,9 +695,9 @@ def from_runner_api(cls,
     # type: (...) -> Optional[PTransform]
     if proto is None or proto.spec is None or not proto.spec.urn:
       return None
-    parameter_type, constructor = cls._known_urns[proto.spec.urn]
 
     try:
+      parameter_type, constructor = cls._known_urns[proto.spec.urn]

Review comment:
       Ditto. Not sure why we need this update given that tests should already pass for Dataflow.

##########
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
##########
@@ -542,19 +593,32 @@ def run_pipeline(self, pipeline, options):
       # TODO(chamikara): remove following pipeline and pipeline proto recreation
       # after portable job submission path is fully in place.
       from apache_beam import Pipeline
-      pipeline = Pipeline.from_runner_api(
+      pipeline, src_context = Pipeline.from_runner_api(
           self.proto_pipeline,
           pipeline.runner,
           options,
+          return_context=True,
           allow_proto_holders=True)
 
       # Pipelines generated from proto do not have output set to PDone set for
       # leaf elements.
       pipeline.visit(self._set_pdone_visitor(pipeline))
 
+      from apache_beam.runners import pipeline_context
+      dst_context = pipeline_context.PipelineContext(
+          component_id_map=pipeline.component_id_map,
+          default_environment=self._default_environment)
+
+      # Copy external environments to prevent dangling environment ids
+      pipeline.visit(

Review comment:
       Why do we need to update dataflow_runner for the test ?
   Existing tests should already work for dataflow_runner without these updates.
   
   Duplicates environments are handled here when submitting the Dataflow job request: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py#L303




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-725701986


   run xvr_dataflow postcommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-723377652


   run XVR_dataflow postcommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on a change in pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #13283:
URL: https://github.com/apache/beam/pull/13283#discussion_r521603410



##########
File path: sdks/python/setup.py
##########
@@ -256,70 +256,71 @@ def run(self):
       'Python %s.%s. You may encounter bugs or missing features.' % (
           sys.version_info.major, sys.version_info.minor))
 
-setuptools.setup(
-    name=PACKAGE_NAME,
-    version=PACKAGE_VERSION,
-    description=PACKAGE_DESCRIPTION,
-    long_description=PACKAGE_LONG_DESCRIPTION,
-    url=PACKAGE_URL,
-    download_url=PACKAGE_DOWNLOAD_URL,
-    author=PACKAGE_AUTHOR,
-    author_email=PACKAGE_EMAIL,
-    packages=setuptools.find_packages(),
-    package_data={'apache_beam': [
-        '*/*.pyx', '*/*/*.pyx', '*/*.pxd', '*/*/*.pxd', '*/*.h', '*/*/*.h',
-        'testing/data/*.yaml', 'portability/api/*.yaml']},
-    ext_modules=cythonize([
-        'apache_beam/**/*.pyx',
-        'apache_beam/coders/coder_impl.py',
-        'apache_beam/metrics/cells.py',
-        'apache_beam/metrics/execution.py',
-        'apache_beam/runners/common.py',
-        'apache_beam/runners/worker/logger.py',
-        'apache_beam/runners/worker/opcounters.py',
-        'apache_beam/runners/worker/operations.py',
-        'apache_beam/transforms/cy_combiners.py',
-        'apache_beam/utils/counters.py',
-        'apache_beam/utils/windowed_value.py',
-    ]),
-    install_requires=REQUIRED_PACKAGES,
-    python_requires=python_requires,
-    test_suite='nose.collector',
-    # BEAM-8840: Do NOT use tests_require or setup_requires.
-    extras_require={
-        'docs': ['Sphinx>=1.5.2,<2.0'],
-        'test': REQUIRED_TEST_PACKAGES,
-        'gcp': GCP_REQUIREMENTS,
-        'interactive': INTERACTIVE_BEAM,
-        'interactive_test': INTERACTIVE_BEAM_TEST,
-        'aws': AWS_REQUIREMENTS,
-        'azure': AZURE_REQUIREMENTS
-    },
-    zip_safe=False,
-    # PyPI package information.
-    classifiers=[
-        'Intended Audience :: End Users/Desktop',
-        'License :: OSI Approved :: Apache Software License',
-        'Operating System :: POSIX :: Linux',
-        'Programming Language :: Python :: 3.6',
-        'Programming Language :: Python :: 3.7',
-        'Programming Language :: Python :: 3.8',
-        # When updating vesion classifiers, also update version warnings
-        # above and in apache_beam/__init__.py.
-        'Topic :: Software Development :: Libraries',
-        'Topic :: Software Development :: Libraries :: Python Modules',
-    ],
-    license='Apache License, Version 2.0',
-    keywords=PACKAGE_KEYWORDS,
-    entry_points={
-        'nose.plugins.0.10': [
-            'beam_test_plugin = test_config:BeamTestPlugin',
-        ]},
-    cmdclass={
-        'build_py': generate_protos_first(build_py),
-        'develop': generate_protos_first(develop),
-        'egg_info': generate_protos_first(egg_info),
-        'test': generate_protos_first(test),
-        'mypy': generate_protos_first(mypy),
-    },
-)
+if __name__ == '__main__':

Review comment:
       Done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-726351816


   Run Java PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on a change in pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #13283:
URL: https://github.com/apache/beam/pull/13283#discussion_r521688625



##########
File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
##########
@@ -542,19 +593,32 @@ def run_pipeline(self, pipeline, options):
       # TODO(chamikara): remove following pipeline and pipeline proto recreation
       # after portable job submission path is fully in place.
       from apache_beam import Pipeline
-      pipeline = Pipeline.from_runner_api(
+      pipeline, src_context = Pipeline.from_runner_api(
           self.proto_pipeline,
           pipeline.runner,
           options,
+          return_context=True,
           allow_proto_holders=True)
 
       # Pipelines generated from proto do not have output set to PDone set for
       # leaf elements.
       pipeline.visit(self._set_pdone_visitor(pipeline))
 
+      from apache_beam.runners import pipeline_context
+      dst_context = pipeline_context.PipelineContext(
+          component_id_map=pipeline.component_id_map,
+          default_environment=self._default_environment)
+
+      # Copy external environments to prevent dangling environment ids
+      pipeline.visit(

Review comment:
       Make sense. Updated to enable PythonUsingJava and PythonUsingSql only.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on a change in pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #13283:
URL: https://github.com/apache/beam/pull/13283#discussion_r522409840



##########
File path: runners/google-cloud-dataflow-java/build.gradle
##########
@@ -312,6 +313,37 @@ task validatesRunnerStreaming {
   ))
 }
 
+createCrossLanguageValidatesRunnerTask(
+  startJobServer: buildAndPushDockerContainer,
+  cleanupJobServer: cleanUpDockerImages,
+  classpath: configurations.validatesRunner,
+  numParallelTests: Integer.MAX_VALUE,
+  needsSdkLocation: true,
+  pythonPipelineOptions: [
+    "--runner=TestDataflowRunner",
+    "--project=${dataflowProject}",
+    "--region=${dataflowRegion}",
+    "--sdk_harness_container_image_overrides=.*java.*,${dockerImageContainer}:${dockerTag}",
+    "--experiments=beam_fn_api",
+    "--experiments=use_unified_worker",

Review comment:
       done.

##########
File path: runners/google-cloud-dataflow-java/build.gradle
##########
@@ -312,6 +313,37 @@ task validatesRunnerStreaming {
   ))
 }
 
+createCrossLanguageValidatesRunnerTask(
+  startJobServer: buildAndPushDockerContainer,
+  cleanupJobServer: cleanUpDockerImages,
+  classpath: configurations.validatesRunner,
+  numParallelTests: Integer.MAX_VALUE,
+  needsSdkLocation: true,
+  pythonPipelineOptions: [
+    "--runner=TestDataflowRunner",
+    "--project=${dataflowProject}",
+    "--region=${dataflowRegion}",
+    "--sdk_harness_container_image_overrides=.*java.*,${dockerImageContainer}:${dockerTag}",
+    "--experiments=beam_fn_api",
+    "--experiments=use_unified_worker",
+  ],
+  javaPipelineOptions: [
+    "--runner=TestDataflowRunner",
+    "--project=${dataflowProject}",
+    "--region=${dataflowRegion}",
+    "--tempRoot=${dataflowValidatesTempRoot}",
+    //"--sdkHarnessContainerImageOverrides=.*java.*,${dockerImageContainer}:${dockerTag}",
+    "--experiments=beam_fn_api,use_unified_worker",

Review comment:
       done.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji removed a comment on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji removed a comment on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-723376932


   run seed job


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji merged pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji merged pull request #13283:
URL: https://github.com/apache/beam/pull/13283


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji removed a comment on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji removed a comment on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-726432960


   Run Python PreCommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji removed a comment on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji removed a comment on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-726351889






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] chamikaramj commented on a change in pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #13283:
URL: https://github.com/apache/beam/pull/13283#discussion_r521702306



##########
File path: runners/google-cloud-dataflow-java/build.gradle
##########
@@ -312,6 +313,37 @@ task validatesRunnerStreaming {
   ))
 }
 
+createCrossLanguageValidatesRunnerTask(
+  startJobServer: buildAndPushDockerContainer,
+  cleanupJobServer: cleanUpDockerImages,
+  classpath: configurations.validatesRunner,
+  numParallelTests: Integer.MAX_VALUE,
+  needsSdkLocation: true,
+  pythonPipelineOptions: [
+    "--runner=TestDataflowRunner",
+    "--project=${dataflowProject}",
+    "--region=${dataflowRegion}",
+    "--sdk_harness_container_image_overrides=.*java.*,${dockerImageContainer}:${dockerTag}",
+    "--experiments=beam_fn_api",
+    "--experiments=use_unified_worker",

Review comment:
       Please remove experiments "beam_fn_api" and "unified_worker" and add "use_runner_v2"

##########
File path: runners/google-cloud-dataflow-java/build.gradle
##########
@@ -312,6 +313,37 @@ task validatesRunnerStreaming {
   ))
 }
 
+createCrossLanguageValidatesRunnerTask(
+  startJobServer: buildAndPushDockerContainer,
+  cleanupJobServer: cleanUpDockerImages,
+  classpath: configurations.validatesRunner,
+  numParallelTests: Integer.MAX_VALUE,
+  needsSdkLocation: true,
+  pythonPipelineOptions: [
+    "--runner=TestDataflowRunner",
+    "--project=${dataflowProject}",
+    "--region=${dataflowRegion}",
+    "--sdk_harness_container_image_overrides=.*java.*,${dockerImageContainer}:${dockerTag}",
+    "--experiments=beam_fn_api",
+    "--experiments=use_unified_worker",
+  ],
+  javaPipelineOptions: [
+    "--runner=TestDataflowRunner",
+    "--project=${dataflowProject}",
+    "--region=${dataflowRegion}",
+    "--tempRoot=${dataflowValidatesTempRoot}",
+    //"--sdkHarnessContainerImageOverrides=.*java.*,${dockerImageContainer}:${dockerTag}",
+    "--experiments=beam_fn_api,use_unified_worker",

Review comment:
       Ditto.

##########
File path: .test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Dataflow.groovy
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the suite of ValidatesRunner tests against the Dataflow runner.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_XVR_Dataflow',
+    'Run XVR_Dataflow PostCommit', 'Dataflow CrossLanguageValidatesRunner Tests', this) {
+      description('Runs the CrossLanguageValidatesRunner suite on the Dataflow runner.')
+
+      // Set common parameters.
+      commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+      // Publish all test results to Jenkins
+      publishers {
+        archiveJunit('**/nosetests*.xml')
+      }
+
+      // Gradle goals for this job.
+      // TODO: We only support Python pipeline using Java external transforms at the moment.
+      // Enable testing for Java pipeline using Python external transforms when BEAM-11203
+      // is implemented.
+      steps {
+        shell('echo "*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA EXTERNAL TRANSFORMS USING PYTHON 3.6 ***"')
+        gradle {
+          rootBuildScriptDir(commonJobProperties.checkoutDir)
+          tasks(':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava')
+          commonJobProperties.setGradleSwitches(delegate)
+          switches('-PpythonVersion=3.6')
+        }
+        shell('echo "*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA EXTERNAL TRANSFORMS USING PYTHON 3.8 ***"')
+        gradle {
+          rootBuildScriptDir(commonJobProperties.checkoutDir)
+          tasks(':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava')
+          commonJobProperties.setGradleSwitches(delegate)
+          switches('-PpythonVersion=3.8')
+        }
+        shell('echo "*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA SQL TRANSFORMS USING PYTHON 3.8 ***"')

Review comment:
       Can we add Kafka as well ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-723412895






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] TheNeuralBit commented on a change in pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #13283:
URL: https://github.com/apache/beam/pull/13283#discussion_r522450944



##########
File path: .test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Dataflow.groovy
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the suite of ValidatesRunner tests against the Dataflow runner.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_XVR_Dataflow',
+    'Run XVR_Dataflow PostCommit', 'Dataflow CrossLanguageValidatesRunner Tests', this) {
+      description('Runs the CrossLanguageValidatesRunner suite on the Dataflow runner.')
+
+      // Set common parameters.
+      commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+      // Publish all test results to Jenkins
+      publishers {
+        archiveJunit('**/nosetests*.xml')
+      }
+
+      // Gradle goals for this job.
+      // TODO: We only support Python pipeline using Java external transforms at the moment.
+      // Enable testing for Java pipeline using Python external transforms when BEAM-11203
+      // is implemented.
+      steps {
+        shell('echo "*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA EXTERNAL TRANSFORMS USING PYTHON 3.6 ***"')
+        gradle {
+          rootBuildScriptDir(commonJobProperties.checkoutDir)
+          tasks(':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava')
+          commonJobProperties.setGradleSwitches(delegate)
+          switches('-PpythonVersion=3.6')
+        }
+        shell('echo "*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA EXTERNAL TRANSFORMS USING PYTHON 3.8 ***"')
+        gradle {
+          rootBuildScriptDir(commonJobProperties.checkoutDir)
+          tasks(':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava')
+          commonJobProperties.setGradleSwitches(delegate)
+          switches('-PpythonVersion=3.8')
+        }
+        shell('echo "*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA SQL TRANSFORMS USING PYTHON 3.8 ***"')

Review comment:
       That only runs Kafka on Flink though right?
   
   and thank you for adding the SQL task :) it looks like it's working:
   ![image](https://user-images.githubusercontent.com/675055/99000877-cde3b200-24ee-11eb-98e2-17b0322f03bb.png)
   
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on pull request #13283:
URL: https://github.com/apache/beam/pull/13283#issuecomment-726335708


   run xvr_dataflow postcommit


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [beam] ihji commented on a change in pull request #13283: [BEAM-11142] Enable CrossLanguageValidateRunner test for dataflow run…

Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #13283:
URL: https://github.com/apache/beam/pull/13283#discussion_r522534265



##########
File path: .test-infra/jenkins/job_PostCommit_CrossLanguageValidatesRunner_Dataflow.groovy
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the suite of ValidatesRunner tests against the Dataflow runner.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_XVR_Dataflow',
+    'Run XVR_Dataflow PostCommit', 'Dataflow CrossLanguageValidatesRunner Tests', this) {
+      description('Runs the CrossLanguageValidatesRunner suite on the Dataflow runner.')
+
+      // Set common parameters.
+      commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+      // Publish all test results to Jenkins
+      publishers {
+        archiveJunit('**/nosetests*.xml')
+      }
+
+      // Gradle goals for this job.
+      // TODO: We only support Python pipeline using Java external transforms at the moment.
+      // Enable testing for Java pipeline using Python external transforms when BEAM-11203
+      // is implemented.
+      steps {
+        shell('echo "*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA EXTERNAL TRANSFORMS USING PYTHON 3.6 ***"')
+        gradle {
+          rootBuildScriptDir(commonJobProperties.checkoutDir)
+          tasks(':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava')
+          commonJobProperties.setGradleSwitches(delegate)
+          switches('-PpythonVersion=3.6')
+        }
+        shell('echo "*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA EXTERNAL TRANSFORMS USING PYTHON 3.8 ***"')
+        gradle {
+          rootBuildScriptDir(commonJobProperties.checkoutDir)
+          tasks(':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava')
+          commonJobProperties.setGradleSwitches(delegate)
+          switches('-PpythonVersion=3.8')
+        }
+        shell('echo "*** RUN CROSS-LANGUAGE DATAFLOW PYTHON WITH JAVA SQL TRANSFORMS USING PYTHON 3.8 ***"')

Review comment:
       We can't simply adding xlang Kafkaio test here since it needs a separate Kafka cluster setup on Dataflow (on Flink it uses locally launched Kafka cluster). Will merge without xlang Kafkaio test.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org