You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by mx...@apache.org on 2018/11/30 11:15:50 UTC
[beam] branch master updated (fc373df -> b06b8e5)
This is an automated email from the ASF dual-hosted git repository.
mxm pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from fc373df Merge pull request #7166 from apache/revert-7130-fast-coders
new 5b9641d [BEAM-5978] Use dynamic port when starting auto starting jobserver
new 38e6be9 [BEAM-5978] Adding libltdl7 to flink job server docker
new 65136cf [BEAM-5978] Correctly pick the docker executable
new 63a4c18 [BEAM-5978] Increase portable wordcount threads to avoid dead lock
new 49c8386 [BEAM-6146] Add precommit for portable python
new 5506335 [BEAM-6146] Portable Python Precommit test
new b06b8e5 Merge pull request #6954: [BEAM-6146] Add portable WordCount to Python PreCommit
The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
...groovy => job_PreCommit_Portable_Python.groovy} | 8 +++----
build.gradle | 4 ++++
runners/flink/job-server-container/Dockerfile | 2 ++
.../apache_beam/runners/portability/job_server.py | 27 ++++++++++++++++++----
sdks/python/build.gradle | 10 ++++++++
5 files changed, 43 insertions(+), 8 deletions(-)
copy .test-infra/jenkins/{job_PreCommit_Go.groovy => job_PreCommit_Portable_Python.groovy} (89%)
[beam] 05/07: [BEAM-6146] Add precommit for portable python
Posted by mx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mxm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 49c8386b48b2e57b38041ace7824132b8806bc08
Author: Ankur Goenka <an...@gmail.com>
AuthorDate: Thu Nov 29 19:06:08 2018 -0800
[BEAM-6146] Add precommit for portable python
---
sdks/python/build.gradle | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index 7c54239..aba3904 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -161,6 +161,12 @@ task preCommit() {
dependsOn "lint"
}
+task portablePreCommit() {
+ dependsOn ':beam-runners-flink_2.11-job-server-container:docker'
+ dependsOn ':beam-sdks-python-container:docker'
+ dependsOn 'portableWordCount'
+}
+
/*************************************************************************************************/
// E2E integration testing and validates runner testing
@@ -283,6 +289,9 @@ task portableWordCount(dependsOn: 'installGcpTest') {
}
}
}
+// Make sure that the job server is built before the portableWordCount is executed.
+portableWordCount.mustRunAfter ':beam-runners-flink_2.11-job-server-container:docker'
+portableWordCount.mustRunAfter ':beam-sdks-python-container:docker'
// Run single or a set of integration tests with provided test options and pipeline options.
task integrationTest(dependsOn: ['installGcpTest', 'sdist']) {
[beam] 06/07: [BEAM-6146] Portable Python Precommit test
Posted by mx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mxm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 5506335bf2b11a5e38c1bb99d9204573548705a2
Author: Ankur Goenka <an...@gmail.com>
AuthorDate: Wed Nov 28 19:31:56 2018 -0800
[BEAM-6146] Portable Python Precommit test
---
.../jenkins/job_PreCommit_Portable_Python.groovy | 32 ++++++++++++++++++++++
build.gradle | 4 +++
2 files changed, 36 insertions(+)
diff --git a/.test-infra/jenkins/job_PreCommit_Portable_Python.groovy b/.test-infra/jenkins/job_PreCommit_Portable_Python.groovy
new file mode 100644
index 0000000..1f26898
--- /dev/null
+++ b/.test-infra/jenkins/job_PreCommit_Portable_Python.groovy
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import PrecommitJobBuilder
+
+PrecommitJobBuilder builder = new PrecommitJobBuilder(
+ scope: this,
+ nameBase: 'Portable_Python',
+ gradleTask: ':portablePythonPreCommit',
+ triggerPathPatterns: [
+ '^model/.*$',
+ '^runners/.*$',
+ '^sdks/python/.*$',
+ '^release/.*$',
+ ]
+)
+builder.build {}
diff --git a/build.gradle b/build.gradle
index 269da86..a74817f 100644
--- a/build.gradle
+++ b/build.gradle
@@ -246,6 +246,10 @@ task pythonPostCommit() {
dependsOn ":beam-sdks-python:postCommit"
}
+task portablePythonPreCommit() {
+ dependsOn ":beam-sdks-python:portablePreCommit"
+}
+
task websitePreCommit() {
dependsOn ":beam-website:preCommit"
}
[beam] 04/07: [BEAM-5978] Increase portable wordcount threads to
avoid dead lock
Posted by mx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mxm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 63a4c182e501b4b7a4d3c2b5d5012cbd12e4260d
Author: Ankur Goenka <an...@gmail.com>
AuthorDate: Thu Nov 29 19:05:13 2018 -0800
[BEAM-5978] Increase portable wordcount threads to avoid dead lock
---
sdks/python/build.gradle | 1 +
1 file changed, 1 insertion(+)
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index d26e292..7c54239 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -267,6 +267,7 @@ task portableWordCount(dependsOn: 'installGcpTest') {
"--input=/etc/profile",
"--output=/tmp/py-wordcount-direct",
"--runner=PortableRunner",
+ "--experiments=worker_threads=100",
]
if (project.hasProperty("streaming"))
options += ["--streaming"]
[beam] 03/07: [BEAM-5978] Correctly pick the docker executable
Posted by mx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mxm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 65136cff720ce6949a9b41fb94908cbaa2dbe13d
Author: Ankur Goenka <an...@gmail.com>
AuthorDate: Thu Nov 29 19:13:32 2018 -0800
[BEAM-5978] Correctly pick the docker executable
---
sdks/python/apache_beam/runners/portability/job_server.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/sdks/python/apache_beam/runners/portability/job_server.py b/sdks/python/apache_beam/runners/portability/job_server.py
index b5e7f76..8498deb 100644
--- a/sdks/python/apache_beam/runners/portability/job_server.py
+++ b/sdks/python/apache_beam/runners/portability/job_server.py
@@ -25,6 +25,7 @@ import socket
import sys
import time
from subprocess import Popen
+from subprocess import check_output
from threading import Lock
@@ -50,10 +51,11 @@ class DockerizedJobServer(object):
# TODO This is hardcoded to Flink at the moment but should be changed
job_server_image_name = os.environ['USER'] + \
"-docker-apache.bintray.io/beam/flink-job-server:latest"
+ docker_path = check_output(['which', 'docker']).strip()
cmd = ["docker", "run",
# We mount the docker binary and socket to be able to spin up
# "sibling" containers for the SDK harness.
- "-v", "/usr/local/bin/docker:/bin/docker",
+ "-v", ':'.join([docker_path, "/bin/docker"]),
"-v", "/var/run/docker.sock:/var/run/docker.sock"]
self.job_port = DockerizedJobServer._pick_port(self.job_port)
# artifact_port 0 suggest to pick a dynamic port.
[beam] 01/07: [BEAM-5978] Use dynamic port when starting auto
starting jobserver
Posted by mx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mxm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 5b9641d81dd0e26e83c6b1a7c808f22eba4d750c
Author: Ankur Goenka <an...@gmail.com>
AuthorDate: Thu Nov 29 18:21:15 2018 -0800
[BEAM-5978] Use dynamic port when starting auto starting jobserver
---
.../apache_beam/runners/portability/job_server.py | 23 +++++++++++++++++++---
1 file changed, 20 insertions(+), 3 deletions(-)
diff --git a/sdks/python/apache_beam/runners/portability/job_server.py b/sdks/python/apache_beam/runners/portability/job_server.py
index 25b8666..b5e7f76 100644
--- a/sdks/python/apache_beam/runners/portability/job_server.py
+++ b/sdks/python/apache_beam/runners/portability/job_server.py
@@ -21,6 +21,7 @@ import atexit
import logging
import os
import signal
+import socket
import sys
import time
from subprocess import Popen
@@ -33,8 +34,8 @@ class DockerizedJobServer(object):
"""
def __init__(self, job_host="localhost",
- job_port=8099,
- artifact_port=8098,
+ job_port=None,
+ artifact_port=None,
harness_port_range=(8100, 8200),
max_connection_retries=5):
self.job_host = job_host
@@ -54,7 +55,12 @@ class DockerizedJobServer(object):
# "sibling" containers for the SDK harness.
"-v", "/usr/local/bin/docker:/bin/docker",
"-v", "/var/run/docker.sock:/var/run/docker.sock"]
- args = ["--job-host", self.job_host, "--job-port", str(self.job_port)]
+ self.job_port = DockerizedJobServer._pick_port(self.job_port)
+ # artifact_port 0 suggest to pick a dynamic port.
+ self.artifact_port = self.artifact_port if self.artifact_port else 0
+ args = ['--job-host', self.job_host,
+ '--job-port', str(self.job_port),
+ '--artifact-port', str(self.artifact_port)]
if sys.platform == "darwin":
# Docker-for-Mac doesn't support host networking, so we need to explictly
@@ -99,3 +105,14 @@ class DockerizedJobServer(object):
time.sleep(1)
if self.docker_process.poll is None:
self.docker_process.kill()
+
+ @staticmethod
+ def _pick_port(port):
+ if port:
+ return port
+ # Not perfect, but we have to provide a port to the subprocess.
+ s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ s.bind(('localhost', 0))
+ _, port = s.getsockname()
+ s.close()
+ return port
[beam] 02/07: [BEAM-5978] Adding libltdl7 to flink job server docker
Posted by mx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mxm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 38e6be99e2e1054e1c44ba0812900a79934b6aed
Author: Ankur Goenka <an...@gmail.com>
AuthorDate: Thu Nov 29 19:02:26 2018 -0800
[BEAM-5978] Adding libltdl7 to flink job server docker
---
runners/flink/job-server-container/Dockerfile | 2 ++
1 file changed, 2 insertions(+)
diff --git a/runners/flink/job-server-container/Dockerfile b/runners/flink/job-server-container/Dockerfile
index 569c2ab..51f0e0f 100644
--- a/runners/flink/job-server-container/Dockerfile
+++ b/runners/flink/job-server-container/Dockerfile
@@ -19,6 +19,8 @@
FROM openjdk:8
MAINTAINER "Apache Beam <de...@beam.apache.org>"
+RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y libltdl7
+
ADD target/beam-runners-flink-job-server.jar /opt/apache/beam/jars/
ADD target/flink-job-server.sh /opt/apache/beam/
[beam] 07/07: Merge pull request #6954: [BEAM-6146] Add portable
WordCount to Python PreCommit
Posted by mx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mxm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit b06b8e5df5738e7dc3620f67134da67a4a806758
Merge: fc373df 5506335
Author: Maximilian Michels <mx...@apache.org>
AuthorDate: Thu Nov 29 12:03:51 2018 +0100
Merge pull request #6954: [BEAM-6146] Add portable WordCount to Python PreCommit
This closes #6146 since its changes have been incorporated into this PR.
.../jenkins/job_PreCommit_Portable_Python.groovy | 32 ++++++++++++++++++++++
build.gradle | 4 +++
runners/flink/job-server-container/Dockerfile | 2 ++
.../apache_beam/runners/portability/job_server.py | 27 +++++++++++++++---
sdks/python/build.gradle | 10 +++++++
5 files changed, 71 insertions(+), 4 deletions(-)