Re: [PR] [SPARK-50605][CONNECT] Support SQL API mode for easier migration to Spark Connect [spark]

["HyukjinKwon (via GitHub)" <gi...@apache.org>]

HyukjinKwon commented on code in PR #49107:
URL: https://github.com/apache/spark/pull/49107#discussion_r1890010605


##########
docs/app-dev-spark-connect.md:
##########
@@ -58,6 +58,59 @@ to the client via the blue box as part of the Spark Connect API. The client uses
 alongside PySpark or the Spark Scala client, making it easy for Spark client applications to work
 with the custom logic/library. 
 
+## Spark API Mode: Spark Client and Spark Classic
+
+Spark provides API mode (`spark.api.mode`) configuration that can seamlessly start Spark Classic
+applications to use Spark Connect. Based on `spark.api.mode` configuration, the application either
+be executed in Spark Classic or Spark Connect mode. See the examples below:
+
+{% highlight python %}
+from pyspark.sql import SparkSession
+
+SparkSession.builder.config("spark.api.mode", "connect").master("...").getOrCreate()
+{% endhighlight %}
+
+
+This configuration can be also effectively set for both Scala Spark application and PySpark
+applications via Spark submission:
+
+{% highlight bash %}
+spark-submit --master "..." --conf spark.api.mode=connect
+{% endhighlight %}
+
+In addition, Spark Connect provides convient special cases for you to locally test. `spark.remote`
+can be set as `local[...]` and `local-cluster[...]`. In this case, it creates a locally running
+Spark Connect server, and provide users the Spark Connect session. This is similar when
+`--conf spark.api.mode=connect` and `--master ...` are set together. However `spark.remote`
+and `--remote` only supports `local*` whereas `--conf spark.api.mode=connect` and `--master ...`
+allow other Spark cluster URLs such as `spark://` for better compatibility.

Review Comment:
   @hvanhovell this part is slightly different from what we discussed offline. My logic here is to restrict it first instead of allowing it: `--remote` and `spark.remote` will support only `local*` as it was.



##########
docs/app-dev-spark-connect.md:
##########
@@ -58,6 +58,34 @@ to the client via the blue box as part of the Spark Connect API. The client uses
 alongside PySpark or the Spark Scala client, making it easy for Spark client applications to work
 with the custom logic/library. 
 
+## Spark API Mode: Spark Client and Spark Classic
+
+Spark provides the API mode, `spark.api.mode` configuration, enabling Spark Classic applications
+to seamlessly switch to Spark Connect. Depending on the value of `spark.api.mode`, the application
+can run in either Spark Classic or Spark Connect mode. Here is an example:
+
+{% highlight python %}
+from pyspark.sql import SparkSession
+
+SparkSession.builder.config("spark.api.mode", "connect").master("...").getOrCreate()
+{% endhighlight %}
+
+You can also apply this configuration to both Scala and PySpark applications when submitting yours:
+
+{% highlight bash %}
+spark-submit --master "..." --conf spark.api.mode=connect
+{% endhighlight %}
+
+Additionally, Spark Connect offers convenient options for local testing. By setting `spark.remote`
+to `local[...]` or `local-cluster[...]`, you can start a local Spark Connect server and access a Spark
+Connect session.
+
+This is similar to using `--conf spark.api.mode=connect` with `--master ...`. However, note that
+`spark.remote` and `--remote` are limited to `local*` values, while `--conf spark.api.mode=connect`
+with `--master ...` supports additional cluster URLs, such as spark://, for broader compatibility with
+Spark Classic.

Review Comment:
   @hvanhovell this part is slightly different from what we discussed offline. My logic here is to restrict it first instead of allowing it: `--remote` and `spark.remote` will support only `local*` as it was.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org