You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ie...@apache.org on 2019/06/04 22:11:27 UTC
[beam] branch master updated: [BEAM-7175] Add Spark Portable
ValidatesRunner Batch postcommit test
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c3ea964 [BEAM-7175] Add Spark Portable ValidatesRunner Batch postcommit test
new 3710eea Merge pull request #8432: [BEAM-7175] Add Spark Portable ValidatesRunner Batch postcommit test
c3ea964 is described below
commit c3ea964054c273801ac11d4e67c78b5acdfe4963
Author: Kyle Weaver <kc...@google.com>
AuthorDate: Mon Apr 29 15:02:06 2019 -0700
[BEAM-7175] Add Spark Portable ValidatesRunner Batch postcommit test
---
.github/PULL_REQUEST_TEMPLATE.md | 4 +-
.test-infra/jenkins/README.md | 2 +
...Java_PortableValidatesRunner_Spark_Batch.groovy | 43 ++++++++++++++++++++++
..._PostCommit_Python_ValidatesRunner_Spark.groovy | 38 +++++++++++++++++++
README.md | 4 +-
.../runners/portability/spark_runner_test.py | 4 ++
sdks/python/build.gradle | 6 +--
7 files changed, 94 insertions(+), 7 deletions(-)
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index b88e127..d511f4e 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -14,8 +14,8 @@ Post-Commit Tests Status (on master branch)
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | ---
-Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [
------------------------------------------------------------------------------------------------
diff --git a/.test-infra/jenkins/README.md b/.test-infra/jenkins/README.md
index 1183b08..09ccfa1 100644
--- a/.test-infra/jenkins/README.md
+++ b/.test-infra/jenkins/README.md
@@ -55,6 +55,7 @@ Beam Jenkins overview page: [link](https://builds.apache.org/view/A-D/view/Beam/
| beam_PostCommit_Java_Nexmark_Spark | [cron](https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Spark/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Spark_PR/) | `Spark Runner Nexmark Tests` | [](https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Spark) |
| beam_PostCommit_Java_PVR_Flink_Batch | [cron](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch_PR/) | `Run Java Flink PortableValidatesRunner Batch` | [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch) |
| beam_PostCommit_Java_PVR_Flink_Streaming | [cron](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming_PR/) | `Run Java Flink PortableValidatesRunner Streaming` | [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming) |
+| beam_PostCommit_Java_PVR_Spark_Batch | [cron](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch_PR/) | `Run Java Spark PortableValidatesRunner Batch` | [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch) |
| beam_PostCommit_Java_PortabilityApi | [cron](https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi_PR/) | `Run Java PortabilityApi PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi) |
| beam_PostCommit_Java_ValidatesRunner_Apex | [cron](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_PR/) | `Run Apex ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex) |
| beam_PostCommit_Java_ValidatesRunner_Dataflow | [cron](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_PR/) | `Run Dataflow ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow) |
@@ -68,6 +69,7 @@ Beam Jenkins overview page: [link](https://builds.apache.org/view/A-D/view/Beam/
| beam_PostCommit_Java_ValidatesRunner_Spark | [cron](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/), [phrase](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_PR/) | `Run Spark ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark) |
| beam_PostCommit_Py_VR_Dataflow | [cron](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/), [phrase](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_PR/) | `Run Python Dataflow ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow) |
| beam_PostCommit_Py_ValCont | [cron](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/), [phrase](https://builds.apache.org/job/beam_PostCommit_Py_ValCont_PR/) | `Run Python Dataflow ValidatesContainer` | [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont) |
+| beam_PostCommit_Python_VR_Spark | [cron](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/), [phrase](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/) | `Run Python Spark ValidatesRunner` | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark) |
| beam_PostCommit_Python_Verify | [cron](https://builds.apache.org/job/beam_PostCommit_Python_Verify/), [phrase](https://builds.apache.org/job/beam_PostCommit_Python_Verify_PR/) | `Run Python PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify) |
| beam_PostCommit_Python3_Verify | [cron](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/), [phrase](https://builds.apache.org/job/beam_PostCommit_Python3_Verify_PR/) | `Run Python PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_Python3_Verify) |
| beam_PostCommit_SQL | [cron](https://builds.apache.org/job/beam_PostCommit_SQL/), [phrase](https://builds.apache.org/job/beam_PostCommit_SQL_PR/) | `Run SQL PostCommit` | [](https://builds.apache.org/job/beam_PostCommit_SQL) |
diff --git a/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Spark_Batch.groovy b/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Spark_Batch.groovy
new file mode 100644
index 0000000..1e9728a
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Spark_Batch.groovy
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the suite of Java ValidatesRunner tests against the Spark runner in batch mode.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_PVR_Spark_Batch',
+ 'Run Java Spark PortableValidatesRunner Batch', 'Java Spark PortableValidatesRunner Batch Tests', this) {
+ description('Runs the Java PortableValidatesRunner suite on the Spark runner in batch mode.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Publish all test results to Jenkins
+ publishers {
+ archiveJunit('**/build/test-results/**/*.xml')
+ }
+
+ // Gradle goals for this job.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':runners:spark:job-server:validatesPortableRunnerBatch')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Spark.groovy b/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Spark.groovy
new file mode 100644
index 0000000..cd69560
--- /dev/null
+++ b/.test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Spark.groovy
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the suite of Python ValidatesRunner tests against the Spark runner.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_VR_Spark',
+ 'Run Python Spark ValidatesRunner', 'Python Spark ValidatesRunner Tests', this) {
+ description('Runs the Python ValidatesRunner suite on the Spark runner.')
+
+ // Set common parameters.
+ commonJobProperties.setTopLevelMainJobProperties(delegate)
+
+ // Gradle goals for this job.
+ steps {
+ gradle {
+ rootBuildScriptDir(commonJobProperties.checkoutDir)
+ tasks(':sdks:python:sparkValidatesRunner')
+ commonJobProperties.setGradleSwitches(delegate)
+ }
+ }
+}
diff --git a/README.md b/README.md
index c26c22b..614075e 100644
--- a/README.md
+++ b/README.md
@@ -33,8 +33,8 @@
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | ---
-Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastComplete [...]
## Overview
diff --git a/sdks/python/apache_beam/runners/portability/spark_runner_test.py b/sdks/python/apache_beam/runners/portability/spark_runner_test.py
index 478e609..bf1d8d7 100644
--- a/sdks/python/apache_beam/runners/portability/spark_runner_test.py
+++ b/sdks/python/apache_beam/runners/portability/spark_runner_test.py
@@ -121,6 +121,10 @@ if __name__ == '__main__':
# Skip until Spark runner supports SDF.
raise unittest.SkipTest("BEAM-7222")
+ def test_sdf_synthetic_source(self):
+ # Skip until Spark runner supports SDF.
+ raise unittest.SkipTest("BEAM-7222")
+
def test_external_transforms(self):
# Skip until Spark runner supports external transforms.
raise unittest.SkipTest("BEAM-7232")
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index 2afddca..165456a 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -263,12 +263,12 @@ task hdfsIntegrationTest(dependsOn: 'installGcpTest') {
task sparkValidatesRunner() {
dependsOn 'createProcessWorker'
dependsOn 'setupVirtualenv'
- dependsOn ':beam-runners-spark-job-server:shadowJar'
+ dependsOn ':runners:spark:job-server:shadowJar'
doLast {
- def environment_config = "'{\"command\": \"${project(":beam-sdks-python:").buildDir.absolutePath}/sdk_worker.sh\"}'"
+ def environment_config = "'{\"command\": \"${project(":sdks:python:").buildDir.absolutePath}/sdk_worker.sh\"}'"
def argMap = [
"environment_type" : "PROCESS",
- "spark_job_server_jar": project(":beam-runners-spark-job-server:").shadowJar.archivePath,
+ "spark_job_server_jar": project(":runners:spark:job-server:").shadowJar.archivePath,
"environment_config": environment_config,
]
def argString = project.mapToArgString(argMap)