You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2024/08/11 18:57:57 UTC
(spark) branch master updated: [SPARK-49171][CONNECT][DOCS] Update Spark Shell with Spark Connect
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d1af1f5951b6 [SPARK-49171][CONNECT][DOCS] Update Spark Shell with Spark Connect
d1af1f5951b6 is described below
commit d1af1f5951b6cee3d4242c7f9fa1b000f4c9461f
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Sun Aug 11 11:57:47 2024 -0700
[SPARK-49171][CONNECT][DOCS] Update Spark Shell with Spark Connect
### What changes were proposed in this pull request?
This PR updates Spark Connect documentation per the change in https://github.com/apache/spark/pull/47402. Now both `bin/pyspark` and `bin/spark-shell` are pretty similar.
### Why are the changes needed?
So the end users can use Spark Connect.
### Does this PR introduce _any_ user-facing change?
Yes, it updates the user-facing documentation.
### How was this patch tested?
Manual documentation build.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #47676 from HyukjinKwon/SPARK-49171.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
docs/spark-connect-overview.md | 57 ++++++++++++++++--------------------------
1 file changed, 21 insertions(+), 36 deletions(-)
diff --git a/docs/spark-connect-overview.md b/docs/spark-connect-overview.md
index 970d460ad441..b8e378b5e7ed 100644
--- a/docs/spark-connect-overview.md
+++ b/docs/spark-connect-overview.md
@@ -97,7 +97,7 @@ be monitored using the application's framework native metrics and logging librar
# How to use Spark Connect
-Starting with Spark 3.4, Spark Connect is available and supports PySpark and Scala
+Spark Connect is available and supports PySpark and Scala
applications. We will walk through how to run an Apache Spark server with Spark
Connect and connect to it from a client application using the Spark Connect client
library.
@@ -105,9 +105,8 @@ library.
## Download and start Spark server with Spark Connect
First, download Spark from the
-[Download Apache Spark](https://spark.apache.org/downloads.html) page. Spark Connect
-was introduced in Apache Spark version 3.4 so make sure you choose 3.4.0 or newer in
-the release drop down at the top of the page. Then choose your package type, typically
+[Download Apache Spark](https://spark.apache.org/downloads.html) page. Choose the
+latest release in the release drop down at the top of the page. Then choose your package type, typically
“Pre-built for Apache Hadoop 3.3 and later”, and click the link to download.
Now extract the Spark package you just downloaded on your computer, for example:
@@ -124,10 +123,8 @@ Spark Connect, like in this example:
./sbin/start-connect-server.sh
{% endhighlight %}
-Note that we include a Spark Connect package (`spark-connect_2.13:{{site.SPARK_VERSION_SHORT}}`), when starting
-Spark server. This is required to use Spark Connect. Make sure to use the same version
-of the package as the Spark version you downloaded previously. In this example,
-Spark {{site.SPARK_VERSION_SHORT}} with Scala 2.13.
+Make sure to use the same version of the package as the Spark version you
+downloaded previously. In this example, Spark {{site.SPARK_VERSION_SHORT}} with Scala 2.13.
Now Spark server is running and ready to accept Spark Connect sessions from client
applications. In the next section we will walk through how to use Spark Connect
@@ -204,8 +201,8 @@ SparkSession available as 'spark'.
Now you can run PySpark code in the shell to see Spark Connect in action:
{% highlight python %}
->>> columns = ["id","name"]
->>> data = [(1,"Sarah"),(2,"Maria")]
+>>> columns = ["id", "name"]
+>>> data = [(1,"Sarah"), (2,"Maria")]
>>> df = spark.createDataFrame(data).toDF(*columns)
>>> df.show()
+---+-----+
@@ -219,29 +216,23 @@ Now you can run PySpark code in the shell to see Spark Connect in action:
</div>
<div data-lang="scala" markdown="1">
-For the Scala shell, we use an Ammonite-based REPL that is currently not included in the Apache Spark package.
+For the Scala shell, we use an Ammonite-based REPL. Otherwise, very similar with PySpark shell.
-To set up the new Scala shell, first download and install [Coursier CLI](https://get-coursier.io/docs/cli-installation).
-Then, install the REPL using the following command in a terminal window:
{% highlight bash %}
-cs install --contrib spark-connect-repl
-{% endhighlight %}
-
-And now you can start the Ammonite-based Scala REPL/shell to connect to your Spark server like this:
-
-{% highlight bash %}
-spark-connect-repl
+./bin/spark-shell --remote "sc://localhost"
{% endhighlight %}
A greeting message will appear when the REPL successfully initializes:
{% highlight bash %}
+Welcome to
+ ____ __
+ / __/__ ___ _____/ /__
+ _\ \/ _ \/ _ `/ __/ '_/
+ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT
+ /_/
+
+Type in expressions to have them evaluated.
Spark session available as 'spark'.
- _____ __ ______ __
- / ___/____ ____ ______/ /__ / ____/___ ____ ____ ___ _____/ /_
- \__ \/ __ \/ __ `/ ___/ //_/ / / / __ \/ __ \/ __ \/ _ \/ ___/ __/
- ___/ / /_/ / /_/ / / / ,< / /___/ /_/ / / / / / / / __/ /__/ /_
-/____/ .___/\__,_/_/ /_/|_| \____/\____/_/ /_/_/ /_/\___/\___/\__/
- /_/
{% endhighlight %}
By default, the REPL will attempt to connect to a local Spark Server.
@@ -265,25 +256,19 @@ connection that is initialized at REPL startup.
{% highlight bash %}
export SPARK_REMOTE="sc://myhost.com:443/;token=ABCDEFG"
-spark-connect-repl
-{% endhighlight %}
-or
-{% highlight bash %}
-SPARK_REMOTE="sc://myhost.com:443/;token=ABCDEFG" spark-connect-repl
+./bin/spark-shell
{% endhighlight %}
-#### Use CLI arguments
+or
-The customizations may also be passed in through CLI arguments as shown below:
{% highlight bash %}
-spark-connect-repl --host myhost.com --port 443 --token ABCDEFG
+SPARK_REMOTE="sc://myhost.com:443/;token=ABCDEFG" spark-connect-repl
{% endhighlight %}
-The supported list of CLI arguments may be found [here](https://github.com/apache/spark/blob/master/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClientParser.scala).
-
#### Configure programmatically with a connection string
The connection may also be programmatically created using _SparkSession#builder_ as in this example:
+
{% highlight scala %}
@ import org.apache.spark.sql.SparkSession
@ val spark = SparkSession.builder.remote("sc://localhost:443/;token=ABCDEFG").build()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org