You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/03/19 21:13:40 UTC
(spark-connect-swift) branch main updated: [SPARK-56083] Add `pyspark-connect` example
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 27c62fd [SPARK-56083] Add `pyspark-connect` example
27c62fd is described below
commit 27c62fd48ded40fb942f4c314d2379cfea940198
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Thu Mar 19 14:13:33 2026 -0700
[SPARK-56083] Add `pyspark-connect` example
### What changes were proposed in this pull request?
This PR aims to add `pyspark-connect` example as a reference in order to provide a fair comparison among
- `Apache PySpark Image`
- `Apache PySpark Connect Image`
- `Apache Spark Swift Client Image`
### Why are the changes needed?
`Apache Spark Swift Client` image is **9.4 times smaller** (801MB / 85.1MB) than `PySpark Image` and **7.1 times smaller** (609MB / 85.1MB) than `PySpark Connect Image`.
```
IMAGE ID DISK USAGE CONTENT SIZE EXTRA
apache/spark:4.1.1-python3 0d4eebff1893 2.16GB 801MB
apache/spark-connect-swift:pyspark-connect d2abec2df6f5 1.63GB 609MB
apache/spark-connect-swift:pi 12f449bb32ca 373MB 85.1MB
```
### Does this PR introduce _any_ user-facing change?
No because this is an example.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: `Gemini 3.1 Pro (High)` on `Antigravity`.
Closes #324 from dongjoon-hyun/SPARK-56083.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
Examples/pyspark-connect/Dockerfile | 39 ++++++++++++++++++++++++++++++
Examples/pyspark-connect/README.md | 27 +++++++++++++++++++++
Examples/pyspark-connect/main.py | 47 +++++++++++++++++++++++++++++++++++++
3 files changed, 113 insertions(+)
diff --git a/Examples/pyspark-connect/Dockerfile b/Examples/pyspark-connect/Dockerfile
new file mode 100644
index 0000000..f83ecd0
--- /dev/null
+++ b/Examples/pyspark-connect/Dockerfile
@@ -0,0 +1,39 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+FROM python:3.14-slim
+
+ARG SPARK_UID=185
+
+LABEL org.opencontainers.image.authors="Apache Spark project <de...@spark.apache.org>"
+LABEL org.opencontainers.image.licenses="Apache-2.0"
+LABEL org.opencontainers.image.ref.name="Apache Spark Connect for PySpark"
+
+ENV APP_HOME=/app
+
+WORKDIR $APP_HOME
+
+COPY main.py .
+
+RUN pip install --no-cache-dir pyspark-connect
+
+RUN groupadd --system --gid=$SPARK_UID spark && \
+ useradd --system --home-dir $APP_HOME --uid=$SPARK_UID --gid=spark spark && \
+ chown -R spark:spark $APP_HOME
+
+USER spark
+
+ENTRYPOINT ["python", "main.py"]
diff --git a/Examples/pyspark-connect/README.md b/Examples/pyspark-connect/README.md
new file mode 100644
index 0000000..5e54ce6
--- /dev/null
+++ b/Examples/pyspark-connect/README.md
@@ -0,0 +1,27 @@
+# A Python application computing an approximation to pi with Apache Spark Connect Python Client
+
+This is a reference example Python application to show how to use Apache Spark Connect Python Client library and compare it with Swift Client.
+
+## How to run
+
+Prepare `Spark Connect Server` via running Docker image.
+
+```bash
+docker run --rm -p 15002:15002 apache/spark:4.1.1 bash -c "/opt/spark/sbin/start-connect-server.sh --wait"
+```
+
+Build an application Docker image.
+
+```bash
+$ docker build -t apache/spark-connect-swift:pyspark-connect .
+$ docker images apache/spark-connect-swift:pyspark-connect
+IMAGE ID DISK USAGE CONTENT SIZE EXTRA
+apache/spark-connect-swift:pyspark-connect d2abec2df6f5 1.63GB 609MB
+```
+
+Run `pyspark-connect` docker image.
+
+```bash
+$ docker run --rm -e SPARK_REMOTE=sc://host.docker.internal:15002 apache/spark-connect-swift:pyspark-connect
+Pi is roughly 3.142020
+```
diff --git a/Examples/pyspark-connect/main.py b/Examples/pyspark-connect/main.py
new file mode 100644
index 0000000..24f8424
--- /dev/null
+++ b/Examples/pyspark-connect/main.py
@@ -0,0 +1,47 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import sys
+from random import random
+from operator import add
+
+from pyspark.sql import SparkSession
+
+
+if __name__ == "__main__":
+ """
+ Usage: pi [partitions]
+ """
+ spark = SparkSession.builder.appName("PythonPi").getOrCreate()
+
+ partitions = int(sys.argv[1]) if len(sys.argv) > 1 else 2
+ n = 100000 * partitions
+
+ def f(_: int) -> float:
+ x = random() * 2 - 1
+ y = random() * 2 - 1
+ return 1 if x ** 2 + y ** 2 <= 1 else 0
+
+ count = spark\
+ .range(n)\
+ .selectExpr("(pow(rand() * 2 - 1, 2) + pow(rand() * 2 - 1, 2)) as v")\
+ .where("v <= 1")\
+ .count()
+
+ print("Pi is roughly %f" % (4.0 * count / n))
+
+ spark.stop()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org