You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dongjoon-hyun <gi...@git.apache.org> on 2018/01/11 07:24:09 UTC
[GitHub] spark pull request #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK...
GitHub user dongjoon-hyun opened a pull request:
https://github.com/apache/spark/pull/20230
[SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
## What changes were proposed in this pull request?
This PR aims to update the followings in `docker/spark-test`.
- JDK7 -> JDK8
Spark 2.2+ supports JDK8 only.
- Ubuntu 12.04.5 LTS(precise) -> Ubuntu 16.04.3 LTS(xeniel)
The end of life of `precise` was April 28, 2017.
## How was this patch tested?
Manual.
* Master
```
$ cd external/docker
$ ./build
$ export SPARK_HOME=...
$ docker run -v $SPARK_HOME:/opt/spark spark-test-master
CONTAINER_IP=172.17.0.3
...
18/01/11 06:50:25 INFO MasterWebUI: Bound MasterWebUI to 172.17.0.3, and started at http://172.17.0.3:8080
18/01/11 06:50:25 INFO Utils: Successfully started service on port 6066.
18/01/11 06:50:25 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066
18/01/11 06:50:25 INFO Master: I have been elected leader! New state: ALIVE
```
* Slave
```
$ docker run -v $SPARK_HOME:/opt/spark spark-test-worker spark://172.17.0.3:7077
CONTAINER_IP=172.17.0.4
...
18/01/11 06:51:54 INFO Worker: Successfully registered with master spark://172.17.0.3:7077
```
After slave starts, master will show
```
18/01/11 06:51:54 INFO Master: Registering worker 172.17.0.4:8888 with 4 cores, 1024.0 MB RAM
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjoon-hyun/spark SPARK-23038
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20230.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20230
----
commit cc3321c20fd0dc2ef75a8740b5c0292beef98beb
Author: Dongjoon Hyun <do...@...>
Date: 2018-01-11T07:18:48Z
[SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20230
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20230
**[Test build #85953 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85953/testReport)** for PR 20230 at commit [`cc3321c`](https://github.com/apache/spark/commit/cc3321c20fd0dc2ef75a8740b5c0292beef98beb).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20230
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85977/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20230#discussion_r160880463
--- Diff: external/docker/spark-test/base/Dockerfile ---
@@ -15,14 +15,14 @@
# limitations under the License.
#
-FROM ubuntu:precise
+FROM ubuntu:xenial
# Upgrade package index
-# install a few other useful packages plus Open Jdk 7
+# install a few other useful packages plus Open Jdk 8
# Remove unneeded /var/lib/apt/lists/* after install to reduce the
# docker image size (by ~30MB)
RUN apt-get update && \
- apt-get install -y less openjdk-7-jre-headless net-tools vim-tiny sudo openssh-server && \
+ apt-get install -y less openjdk-8-jre-headless iproute2 vim-tiny sudo openssh-server && \
--- End diff --
This is required to use [ip](https://github.com/apache/spark/blob/master/external/docker/spark-test/master/default_cmd#L20) command in `xenial`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/20230
merged to master/2.3/2.2
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/20230#discussion_r160926186
--- Diff: external/docker/spark-test/base/Dockerfile ---
@@ -15,14 +15,14 @@
# limitations under the License.
#
-FROM ubuntu:precise
+FROM ubuntu:xenial
--- End diff --
This is probably a good change, but is it required in order to get Java 8?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/20230#discussion_r161016953
--- Diff: external/docker/spark-test/base/Dockerfile ---
@@ -15,14 +15,14 @@
# limitations under the License.
#
-FROM ubuntu:precise
+FROM ubuntu:xenial
--- End diff --
Yes. It's required. When we change only jdk7 to jdk8, it fails.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20230
Retest this please.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20230
**[Test build #85977 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85977/testReport)** for PR 20230 at commit [`cc3321c`](https://github.com/apache/spark/commit/cc3321c20fd0dc2ef75a8740b5c0292beef98beb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on a diff in the pull request:
https://github.com/apache/spark/pull/20230#discussion_r161367060
--- Diff: external/docker/spark-test/base/Dockerfile ---
@@ -15,14 +15,14 @@
# limitations under the License.
#
-FROM ubuntu:precise
+FROM ubuntu:xenial
--- End diff --
yea no 8 on precise
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20230
**[Test build #85953 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85953/testReport)** for PR 20230 at commit [`cc3321c`](https://github.com/apache/spark/commit/cc3321c20fd0dc2ef75a8740b5c0292beef98beb).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20230
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20230
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85953/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20230
Thank you, @felixcheung and @srowen .
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20230
Thank you for review and approval, @felixcheung .
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20230
**[Test build #85977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85977/testReport)** for PR 20230 at commit [`cc3321c`](https://github.com/apache/spark/commit/cc3321c20fd0dc2ef75a8740b5c0292beef98beb).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20230
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20230: [SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/20230
Hi, @aarondav .
Could you review this too?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org