You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2024/08/27 09:31:04 UTC
(spark) tag v3.4.3 created (now a9e3ddf43d51)
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a change to tag v3.4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
at a9e3ddf43d51 (commit)
This tag includes the following new commits:
new 69acd265a8bf [SPARK-49402][PYTHON] Fix Binder integration in PySpark documentation
new c2a436cc7c8c [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
new d05c30f6c831 [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
new d278e81103cb [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
new 1fd3265b37a9 [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
new 1d82b0e9aff9 [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
new dfdc667db9a6 [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
new a9e3ddf43d51 [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
The 8 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
(spark) 06/08: [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to tag v3.4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 1d82b0e9aff9704457654e7e95f7760138af8aa1
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Tue Aug 27 17:55:00 2024 +0900
[SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Followup PR to change JRE version from 8 to 11
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 9fc1e05)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
binder/Dockerfile | 18 +++---------------
1 file changed, 3 insertions(+), 15 deletions(-)
diff --git a/binder/Dockerfile b/binder/Dockerfile
index 87a69fef69ce..daa9d2d76c39 100644
--- a/binder/Dockerfile
+++ b/binder/Dockerfile
@@ -38,21 +38,9 @@ USER root
RUN chown -R ${NB_UID} ${HOME}
# To solve add-apt-repository : command not found
-RUN apt-get update && apt-get -y install software-properties-common
-
-# Install Java
-RUN \
- echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | debconf-set-selections && \
- add-apt-repository -y ppa:webupd8team/java && \
- apt-get update && \
- apt-get install -y oracle-java8-installer --allow-unauthenticated && \
- rm -rf /var/lib/apt/lists/* && \
- rm -rf /var/cache/oracle-jdk8-installer
-
-# Define commonly used JAVA_HOME variable
-ENV JAVA_HOME /usr/lib/jvm/java-8-oracle
-
-RUN apt-get update && apt-get install -y git coreutils
+RUN apt-get update && apt-get install -y software-properties-common
+RUN add-apt-repository ppa:openjdk-r/ppa
+RUN apt-get update && apt-get install -y openjdk-8-jre git coreutils
USER ${NB_USER}
RUN binder/postBuild
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
(spark) 08/08: [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to tag v3.4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
commit a9e3ddf43d515e3de3d8205f61c6fa7525811bbe
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Tue Aug 27 18:29:37 2024 +0900
[SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Followup PR to change JRE version from 8 to 17
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 9fc1e05)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
binder/Dockerfile | 6 +-----
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git a/binder/Dockerfile b/binder/Dockerfile
index 794eb7f0d2e9..6e3dd9155fb7 100644
--- a/binder/Dockerfile
+++ b/binder/Dockerfile
@@ -36,11 +36,7 @@ USER ${USER}
COPY . ${HOME}
USER root
RUN chown -R ${NB_UID} ${HOME}
-
-# To solve add-apt-repository : command not found
-RUN apt-get update && apt-get install -y software-properties-common python3-launchpadlib
-RUN add-apt-repository ppa:openjdk-r/ppa
-RUN apt-get update && apt-get install -y openjdk-8-jre git coreutils
+RUN apt-get update && apt-get install -y openjdk-17-jre git coreutils
USER ${NB_USER}
RUN binder/postBuild
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
(spark) 01/08: [SPARK-49402][PYTHON] Fix Binder integration in PySpark documentation
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to tag v3.4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 69acd265a8bff260559c506f855c218bd971914a
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Tue Aug 27 14:51:02 2024 +0900
[SPARK-49402][PYTHON] Fix Binder integration in PySpark documentation
This PR proposes to fix Binder integration by using `Dockerfile` directly.
Binder integration is broken now (https://mybinder.org/v2/gh/apache/spark/bb7846dd487?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart_df.ipynb):
This seems to be related to the size of the repository (https://github.com/jupyterhub/mybinder.org-deploy/issues/3074).
I tried all the ways out but could not find the way except using `Dockerfile`.
Yes. This should recover the Binder integration.
Manually tested within my fork:
https://mybinder.org/v2/gh/HyukjinKwon/spark/binder-test1?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart_df.ipynb
No.
Closes #47883 from HyukjinKwon/binder-test1.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 9fc1e05224d2b4207885a1d4f822fed6db0cb1a1)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
binder/Dockerfile | 43 +++++++++++++++++++++++++++++++++++++++++++
binder/apt.txt | 2 --
binder/postBuild | 2 +-
3 files changed, 44 insertions(+), 3 deletions(-)
diff --git a/binder/Dockerfile b/binder/Dockerfile
new file mode 100644
index 000000000000..a0af7312ef4f
--- /dev/null
+++ b/binder/Dockerfile
@@ -0,0 +1,43 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM python:3.10-slim
+# install the notebook package
+RUN pip install --no-cache notebook jupyterlab
+
+# create user with a home directory
+ARG NB_USER
+ARG NB_UID
+ENV USER ${NB_USER}
+ENV HOME /home/${NB_USER}
+
+RUN adduser --disabled-password \
+ --gecos "Default user" \
+ --uid ${NB_UID} \
+ ${NB_USER}
+WORKDIR ${HOME}
+USER ${USER}
+
+# Make sure the contents of our repo are in ${HOME}
+COPY . ${HOME}
+USER root
+RUN chown -R ${NB_UID} ${HOME}
+RUN apt-get update && apt-get install -y openjdk-8-jre git coreutils
+USER ${NB_USER}
+
+RUN binder/postBuild
+
diff --git a/binder/apt.txt b/binder/apt.txt
deleted file mode 100644
index 3d86667d4b91..000000000000
--- a/binder/apt.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-openjdk-8-jre
-git
diff --git a/binder/postBuild b/binder/postBuild
old mode 100644
new mode 100755
index 70ae23b39370..9cf90a1038a6
--- a/binder/postBuild
+++ b/binder/postBuild
@@ -21,7 +21,7 @@
# Jupyter notebook.
VERSION=$(python -c "exec(open('python/pyspark/version.py').read()); print(__version__)")
-TAG=$(git describe --tags --exact-match 2>/dev/null)
+TAG=$(git describe --tags --exact-match 2> /dev/null || true)
# If a commit is tagged, exactly specified version of pyspark should be installed to avoid
# a kind of accident that an old version of pyspark is installed in the live notebook environment.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
(spark) 04/08: [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to tag v3.4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
commit d278e81103cb9e5391e76295af7beeb200901088
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Tue Aug 27 17:36:47 2024 +0900
[SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Followup PR to change JRE version from 8 to 11
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 9fc1e05)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
binder/Dockerfile | 19 +++++++++++++++++--
1 file changed, 17 insertions(+), 2 deletions(-)
diff --git a/binder/Dockerfile b/binder/Dockerfile
index b91ca1dce0d9..ac6a77b8cda0 100644
--- a/binder/Dockerfile
+++ b/binder/Dockerfile
@@ -36,8 +36,23 @@ USER ${USER}
COPY . ${HOME}
USER root
RUN chown -R ${NB_UID} ${HOME}
-RUN add-apt-repository ppa:openjdk-r/ppa
-RUN apt-get update && apt-get install -y openjdk-11-jre git coreutils
+
+# To solve add-apt-repository : command not found
+RUN apt-get -y install software-properties-common
+
+# Install Java
+RUN \
+ echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | debconf-set-selections && \
+ add-apt-repository -y ppa:webupd8team/java && \
+ apt-get update && \
+ apt-get install -y oracle-java8-installer --allow-unauthenticated && \
+ rm -rf /var/lib/apt/lists/* && \
+ rm -rf /var/cache/oracle-jdk8-installer
+
+# Define commonly used JAVA_HOME variable
+ENV JAVA_HOME /usr/lib/jvm/java-8-oracle
+
+RUN apt-get update && apt-get install -y git coreutils
USER ${NB_USER}
RUN binder/postBuild
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
(spark) 03/08: [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to tag v3.4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
commit d05c30f6c831111c11c401a09c514b4bb9ba577a
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Tue Aug 27 17:22:53 2024 +0900
[SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Followup PR to add a repository for JDK
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 9fc1e05)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
binder/Dockerfile | 1 +
1 file changed, 1 insertion(+)
diff --git a/binder/Dockerfile b/binder/Dockerfile
index 6eb8c7c04882..b91ca1dce0d9 100644
--- a/binder/Dockerfile
+++ b/binder/Dockerfile
@@ -36,6 +36,7 @@ USER ${USER}
COPY . ${HOME}
USER root
RUN chown -R ${NB_UID} ${HOME}
+RUN add-apt-repository ppa:openjdk-r/ppa
RUN apt-get update && apt-get install -y openjdk-11-jre git coreutils
USER ${NB_USER}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
(spark) 05/08: [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to tag v3.4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 1fd3265b37a9e9f2905c419e3d668523195cc0c0
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Tue Aug 27 17:43:14 2024 +0900
[SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Followup PR to change JRE version from 8 to 11
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 9fc1e05)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
binder/Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/binder/Dockerfile b/binder/Dockerfile
index ac6a77b8cda0..87a69fef69ce 100644
--- a/binder/Dockerfile
+++ b/binder/Dockerfile
@@ -38,7 +38,7 @@ USER root
RUN chown -R ${NB_UID} ${HOME}
# To solve add-apt-repository : command not found
-RUN apt-get -y install software-properties-common
+RUN apt-get update && apt-get -y install software-properties-common
# Install Java
RUN \
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
(spark) 07/08: [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to tag v3.4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
commit dfdc667db9a6cf4435336efe687bba0b7d201234
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Tue Aug 27 18:07:05 2024 +0900
[SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Followup PR to change JRE version from 8 to 11
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 9fc1e05)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
binder/Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/binder/Dockerfile b/binder/Dockerfile
index daa9d2d76c39..794eb7f0d2e9 100644
--- a/binder/Dockerfile
+++ b/binder/Dockerfile
@@ -38,7 +38,7 @@ USER root
RUN chown -R ${NB_UID} ${HOME}
# To solve add-apt-repository : command not found
-RUN apt-get update && apt-get install -y software-properties-common
+RUN apt-get update && apt-get install -y software-properties-common python3-launchpadlib
RUN add-apt-repository ppa:openjdk-r/ppa
RUN apt-get update && apt-get install -y openjdk-8-jre git coreutils
USER ${NB_USER}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
(spark) 02/08: [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to tag v3.4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
commit c2a436cc7c8cdccd0df5108d6eed31e9b8839b2e
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Tue Aug 27 17:12:17 2024 +0900
[SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Followup PR to change JRE version from 8 to 11
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 9fc1e05)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
binder/Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/binder/Dockerfile b/binder/Dockerfile
index a0af7312ef4f..6eb8c7c04882 100644
--- a/binder/Dockerfile
+++ b/binder/Dockerfile
@@ -36,7 +36,7 @@ USER ${USER}
COPY . ${HOME}
USER root
RUN chown -R ${NB_UID} ${HOME}
-RUN apt-get update && apt-get install -y openjdk-8-jre git coreutils
+RUN apt-get update && apt-get install -y openjdk-11-jre git coreutils
USER ${NB_USER}
RUN binder/postBuild
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org