You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2024/08/27 08:24:04 UTC
(spark) tag v3.5.2 created (now 8e45c9cb510f)
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a change to tag v3.5.2
in repository https://gitbox.apache.org/repos/asf/spark.git
at 8e45c9cb510f (commit)
This tag includes the following new commits:
new 02ca2f007f3f [SPARK-49402][PYTHON] Fix Binder integration in PySpark documentation
new 52275c0a9844 [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
new 8e45c9cb510f [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
(spark) 01/03: [SPARK-49402][PYTHON] Fix Binder integration in PySpark documentation
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to tag v3.5.2
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 02ca2f007f3f8230ef257980a5770e3fec74bcb1
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Tue Aug 27 14:51:02 2024 +0900
[SPARK-49402][PYTHON] Fix Binder integration in PySpark documentation
This PR proposes to fix Binder integration by using `Dockerfile` directly.
Binder integration is broken now (https://mybinder.org/v2/gh/apache/spark/bb7846dd487?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart_df.ipynb):
This seems to be related to the size of the repository (https://github.com/jupyterhub/mybinder.org-deploy/issues/3074).
I tried all the ways out but could not find the way except using `Dockerfile`.
Yes. This should recover the Binder integration.
Manually tested within my fork:
https://mybinder.org/v2/gh/HyukjinKwon/spark/binder-test1?filepath=python%2Fdocs%2Fsource%2Fgetting_started%2Fquickstart_df.ipynb
No.
Closes #47883 from HyukjinKwon/binder-test1.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 9fc1e05224d2b4207885a1d4f822fed6db0cb1a1)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
binder/Dockerfile | 43 +++++++++++++++++++++++++++++++++++++++++++
binder/apt.txt | 2 --
binder/postBuild | 2 +-
3 files changed, 44 insertions(+), 3 deletions(-)
diff --git a/binder/Dockerfile b/binder/Dockerfile
new file mode 100644
index 000000000000..a0af7312ef4f
--- /dev/null
+++ b/binder/Dockerfile
@@ -0,0 +1,43 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM python:3.10-slim
+# install the notebook package
+RUN pip install --no-cache notebook jupyterlab
+
+# create user with a home directory
+ARG NB_USER
+ARG NB_UID
+ENV USER ${NB_USER}
+ENV HOME /home/${NB_USER}
+
+RUN adduser --disabled-password \
+ --gecos "Default user" \
+ --uid ${NB_UID} \
+ ${NB_USER}
+WORKDIR ${HOME}
+USER ${USER}
+
+# Make sure the contents of our repo are in ${HOME}
+COPY . ${HOME}
+USER root
+RUN chown -R ${NB_UID} ${HOME}
+RUN apt-get update && apt-get install -y openjdk-8-jre git coreutils
+USER ${NB_USER}
+
+RUN binder/postBuild
+
diff --git a/binder/apt.txt b/binder/apt.txt
deleted file mode 100644
index 3d86667d4b91..000000000000
--- a/binder/apt.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-openjdk-8-jre
-git
diff --git a/binder/postBuild b/binder/postBuild
old mode 100644
new mode 100755
index fec233f8c8ce..801f0a4c2763
--- a/binder/postBuild
+++ b/binder/postBuild
@@ -26,7 +26,7 @@ set -o pipefail
set -e
VERSION=$(python -c "exec(open('python/pyspark/version.py').read()); print(__version__)")
-TAG=$(git describe --tags --exact-match 2>/dev/null)
+TAG=$(git describe --tags --exact-match 2> /dev/null || true)
# If a commit is tagged, exactly specified version of pyspark should be installed to avoid
# a kind of accident that an old version of pyspark is installed in the live notebook environment.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
(spark) 03/03: [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to tag v3.5.2
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 8e45c9cb510f7fcaeb3f41d5fc1a51e55c05c7ab
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Tue Aug 27 17:22:53 2024 +0900
[SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Followup PR to add a repository for JDK
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 9fc1e05)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
binder/Dockerfile | 1 +
1 file changed, 1 insertion(+)
diff --git a/binder/Dockerfile b/binder/Dockerfile
index 6eb8c7c04882..b91ca1dce0d9 100644
--- a/binder/Dockerfile
+++ b/binder/Dockerfile
@@ -36,6 +36,7 @@ USER ${USER}
COPY . ${HOME}
USER root
RUN chown -R ${NB_UID} ${HOME}
+RUN add-apt-repository ppa:openjdk-r/ppa
RUN apt-get update && apt-get install -y openjdk-11-jre git coreutils
USER ${NB_USER}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
(spark) 02/03: [SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Posted by gu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to tag v3.5.2
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 52275c0a984438a9318e87eb44a81483f82eb9ba
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Tue Aug 27 17:12:17 2024 +0900
[SPARK-49402][PYTHON][FOLLOW-UP] Fix Binder integration in PySpark documentation
Followup PR to change JRE version from 8 to 11
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 9fc1e05)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
binder/Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/binder/Dockerfile b/binder/Dockerfile
index a0af7312ef4f..6eb8c7c04882 100644
--- a/binder/Dockerfile
+++ b/binder/Dockerfile
@@ -36,7 +36,7 @@ USER ${USER}
COPY . ${HOME}
USER root
RUN chown -R ${NB_UID} ${HOME}
-RUN apt-get update && apt-get install -y openjdk-8-jre git coreutils
+RUN apt-get update && apt-get install -y openjdk-11-jre git coreutils
USER ${NB_USER}
RUN binder/postBuild
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org