You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by "dmatrix (via GitHub)" <gi...@apache.org> on 2025/02/22 01:15:35 UTC
[PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
dmatrix opened a new pull request, #591:
URL: https://github.com/apache/spark-website/pull/591
- Add new download page with Spark Connect description
- Added decision matrix
- instructions for multiple distributions
- instructions for pyspark install
-
<img width="1715" alt="Screenshot 2025-02-21 at 1 17 35 PM" src="https://github.com/user-attachments/assets/3517392a-5ab8-4f24-994d-fb79714e66b7" />
<img width="1360" alt="Screenshot 2025-02-21 at 1 17 53 PM" src="https://github.com/user-attachments/assets/291761e4-1463-4b71-8ba0-b0c48c729dbf" />
<img width="1353" alt="Screenshot 2025-02-21 at 1 18 06 PM" src="https://github.com/user-attachments/assets/d3f4ee26-4239-4b98-8a9d-543e2a8ac6bc" />
<!-- *Make sure that you generate site HTML with `bundle exec jekyll build`, and include the changes to the HTML in your pull request. See README.md for more information.* -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on PR #591:
URL: https://github.com/apache/spark-website/pull/591#issuecomment-2676222383
Yeah the HTML is there but maybe your jekyll version is different ? CI/CD is saying it doesn't match. Double check your gems vs what's in the repo's config file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen closed pull request #591: [DOC] Add new download page with Spark Connect description and instructions for multiple distributions
URL: https://github.com/apache/spark-website/pull/591
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #591:
URL: https://github.com/apache/spark-website/pull/591#discussion_r1966947842
##########
downloads.md:
##########
@@ -16,6 +16,32 @@ window.onload = function () {
}
</script>
+## Introduction
+
+Unlike previous Apache Spark™ releases, Spark 4.0 has two distinct distributions: _classic_ and _connect_. As the names suggest, the _classic_ Spark version is the usual distribution you would expect for any new Spark release. The _connect_ distribution, in contrast, is the version with [Spark Connect](https://spark.apache.org/docs/4.0.0-preview2/spark-connect-overview.html) enabled by default. Which one should you download?
+
+Select the _connect_ version if your workloads only use standard [DataFrame](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/dataframe.html) and [Spark SQL](https://spark.apache.org/docs/latest/api/sql/) APIs. Choose the _classic_ version for traditional workloads requiring access to [RDD APIs](https://spark.apache.org/docs/latest/api/python/reference/pyspark.html#rdd-apis), [SparkContext APIs](https://spark.apache.org/docs/latest/api/python/reference/pyspark.html#spark-context-apis), JVM properties, and custom catalyst rules/plans.
+
+If you are not familiar with Spark Connect, the primary benefit is that it provides a stable client API, decoupling the client from the Spark Driver. This makes Spark projects much easier to maintain over time, allowing you to update the Spark Driver and server-side dependencies without having to update the client. To learn more about Spark Connect, and explore its architecture details and benefits, visit here: [Spark Connect architecture](https://spark.apache.org/spark-connect/).
+
+## Selection Matrix for Spark Distributions
+
+This table guides you to which of the two distributions to select based on the type of Spark workloads.
+
+| Workloads Types | Spark Distribution and PySpark Package Mode | Spark Config Change |
+|-----------------------------------------------------------------------------------------------------|--------------------------------------------|---------------------------------------------|
+| - Only use standard DataFrame and Spark SQL APIs | _connect_ | None |
Review Comment:
nit but let's either remove spaces between the contents, e.g., :
```
| - Only use standard DataFrame and Spark SQL APIs | _connect_ | None |
```
or make it consistent between other lines, e.g.,
```
| - Ability to access and debug Spark from IDE or interact in notebooks | | |
| - Use of thin client to access Spark cluster from non-JVM languages | | |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "dmatrix (via GitHub)" <gi...@apache.org>.
dmatrix commented on PR #591:
URL: https://github.com/apache/spark-website/pull/591#issuecomment-2675982277
@srowen What did I miss? I did build an entire site following the README instructions:
- `bundle exec jekyll build --watch.`
- `bundle exec jekyll serve`
That generated all the html files under the site subdirectory, and I perused them via the local host, http://127.0.0.1:4000/.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on code in PR #591:
URL: https://github.com/apache/spark-website/pull/591#discussion_r1968129724
##########
downloads.md:
##########
@@ -27,19 +53,39 @@ window.onload = function () {
3. Download Spark: <span id="spanDownloadLink"></span>
4. Verify this release using the <span id="sparkDownloadVerify"></span> and [project release KEYS](https://downloads.apache.org/spark/KEYS) by following these [procedures](https://www.apache.org/info/verification.html).
+classic
-Note that Spark 3 is pre-built with Scala 2.12 in general and Spark 3.2+ provides additional pre-built distribution with Scala 2.13.
+Note that Spark 4 is pre-built with Scala 2.13 in general, and Spark 3.5+ provides additional pre-built distribution with Scala 2.13.
-### Link with Spark
+### Link with Spark ###
Spark artifacts are [hosted in Maven Central](https://search.maven.org/search?q=g:org.apache.spark). You can add a Maven dependency with the following coordinates:
groupId: org.apache.spark
- artifactId: spark-core_2.12
- version: 3.5.4
+ artifactId: spark-core_2.13
+ version: 4.0.0
+
+### Installing with PyPi ###
+Like the two distributions mentioned above, PyPI will also have two PySpark package versions. The default is the _classic_ __pyspark__, while the _connect_ version is __pyspark-connect__ and is dependent on __pyspark__.
+
+Use the decision matrix above to select which PyPI PySpark package to use for your Spark workloads. Both <a href="https://pypi.org/project/pyspark/">PySpark</a> package versions are available on PyPI.
+
+### Installing PySpark Connect ###
+
+Since __pyspark-connect__ package is dependent on __pyspark__, __pyspark-connect__ will automatically install __pyspark__ for you. The __pyspark-connect__ package is mostly empty and merely enables Spark config `spark.api.mode` to _connect_ mode in the underlying pyspark package.
+
+`pip install pyspark-connect==4.0.0`
+
+Thereafter, follow the Spark Connect [quickstart guide](https://spark.apache.org/docs/4.0.0-preview2/api/python/getting_started/quickstart_connect.html) on how to use SparkSession.
+
+### Installing PySpark Classic ###
+
+Simply use `pip install pyspark==4.0.0`
Review Comment:
I would suggest reordering it, if required - we can flip the decision matrix as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "dmatrix (via GitHub)" <gi...@apache.org>.
dmatrix commented on code in PR #591:
URL: https://github.com/apache/spark-website/pull/591#discussion_r1968123460
##########
downloads.md:
##########
@@ -27,19 +53,39 @@ window.onload = function () {
3. Download Spark: <span id="spanDownloadLink"></span>
4. Verify this release using the <span id="sparkDownloadVerify"></span> and [project release KEYS](https://downloads.apache.org/spark/KEYS) by following these [procedures](https://www.apache.org/info/verification.html).
+classic
-Note that Spark 3 is pre-built with Scala 2.12 in general and Spark 3.2+ provides additional pre-built distribution with Scala 2.13.
+Note that Spark 4 is pre-built with Scala 2.13 in general, and Spark 3.5+ provides additional pre-built distribution with Scala 2.13.
-### Link with Spark
+### Link with Spark ###
Spark artifacts are [hosted in Maven Central](https://search.maven.org/search?q=g:org.apache.spark). You can add a Maven dependency with the following coordinates:
groupId: org.apache.spark
- artifactId: spark-core_2.12
- version: 3.5.4
+ artifactId: spark-core_2.13
+ version: 4.0.0
+
+### Installing with PyPi ###
+Like the two distributions mentioned above, PyPI will also have two PySpark package versions. The default is the _classic_ __pyspark__, while the _connect_ version is __pyspark-connect__ and is dependent on __pyspark__.
+
+Use the decision matrix above to select which PyPI PySpark package to use for your Spark workloads. Both <a href="https://pypi.org/project/pyspark/">PySpark</a> package versions are available on PyPI.
+
+### Installing PySpark Connect ###
+
+Since __pyspark-connect__ package is dependent on __pyspark__, __pyspark-connect__ will automatically install __pyspark__ for you. The __pyspark-connect__ package is mostly empty and merely enables Spark config `spark.api.mode` to _connect_ mode in the underlying pyspark package.
+
+`pip install pyspark-connect==4.0.0`
+
+Thereafter, follow the Spark Connect [quickstart guide](https://spark.apache.org/docs/4.0.0-preview2/api/python/getting_started/quickstart_connect.html) on how to use SparkSession.
+
+### Installing PySpark Classic ###
+
+Simply use `pip install pyspark==4.0.0`
Review Comment:
@mridulm Thanks for the feedback. The order here is not intentionally to lead or steer one to use _connect_ instead of _classic_. Instead, it is from easier and simpler more expansive and comprehensive, switching between modes as the last: simple to complex.
I believe it's much easier to read and follow the decision matrix than the reverse: complex to simple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "dmatrix (via GitHub)" <gi...@apache.org>.
dmatrix commented on code in PR #591:
URL: https://github.com/apache/spark-website/pull/591#discussion_r1968428097
##########
downloads.md:
##########
@@ -27,19 +53,39 @@ window.onload = function () {
3. Download Spark: <span id="spanDownloadLink"></span>
4. Verify this release using the <span id="sparkDownloadVerify"></span> and [project release KEYS](https://downloads.apache.org/spark/KEYS) by following these [procedures](https://www.apache.org/info/verification.html).
+classic
-Note that Spark 3 is pre-built with Scala 2.12 in general and Spark 3.2+ provides additional pre-built distribution with Scala 2.13.
+Note that Spark 4 is pre-built with Scala 2.13 in general, and Spark 3.5+ provides additional pre-built distribution with Scala 2.13.
-### Link with Spark
+### Link with Spark ###
Spark artifacts are [hosted in Maven Central](https://search.maven.org/search?q=g:org.apache.spark). You can add a Maven dependency with the following coordinates:
groupId: org.apache.spark
- artifactId: spark-core_2.12
- version: 3.5.4
+ artifactId: spark-core_2.13
+ version: 4.0.0
+
+### Installing with PyPi ###
+Like the two distributions mentioned above, PyPI will also have two PySpark package versions. The default is the _classic_ __pyspark__, while the _connect_ version is __pyspark-connect__ and is dependent on __pyspark__.
+
+Use the decision matrix above to select which PyPI PySpark package to use for your Spark workloads. Both <a href="https://pypi.org/project/pyspark/">PySpark</a> package versions are available on PyPI.
+
+### Installing PySpark Connect ###
+
+Since __pyspark-connect__ package is dependent on __pyspark__, __pyspark-connect__ will automatically install __pyspark__ for you. The __pyspark-connect__ package is mostly empty and merely enables Spark config `spark.api.mode` to _connect_ mode in the underlying pyspark package.
+
+`pip install pyspark-connect==4.0.0`
+
+Thereafter, follow the Spark Connect [quickstart guide](https://spark.apache.org/docs/4.0.0-preview2/api/python/getting_started/quickstart_connect.html) on how to use SparkSession.
+
+### Installing PySpark Classic ###
+
+Simply use `pip install pyspark==4.0.0`
Review Comment:
@srowen These are the steps I took to address your non-matching configs:
1. `bundle update`, updating the Gemfile.lock
2. Check the configs in GemFile vs. Gemfile.lock, they seem to have the same versions for Jekyll, rouge, and webrick
3. Commit the Gemfile.lock
4. `bundle exec jekyll build`
5. `bundle exec jekyll serve`
6. Check the download page.
Push the commits. Hope I didn't miss anything. Do I have to change th Gemfile if the configs are the same in Gemfile.lock? Where else should I check for config compatibility that the CI/CDI checks againt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "dmatrix (via GitHub)" <gi...@apache.org>.
dmatrix commented on code in PR #591:
URL: https://github.com/apache/spark-website/pull/591#discussion_r1968269571
##########
downloads.md:
##########
@@ -16,6 +16,32 @@ window.onload = function () {
}
</script>
+## Introduction
+
+Unlike previous Apache Spark™ releases, Spark 4.0 has two distinct distributions: _classic_ and _connect_. As the names suggest, the _classic_ Spark version is the usual distribution you would expect for any new Spark release. The _connect_ distribution, in contrast, is the version with [Spark Connect](https://spark.apache.org/docs/4.0.0-preview2/spark-connect-overview.html) enabled by default. Which one should you download?
+
+Select the _connect_ version if your workloads only use standard [DataFrame](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/dataframe.html) and [Spark SQL](https://spark.apache.org/docs/latest/api/sql/) APIs. Choose the _classic_ version for traditional workloads requiring access to [RDD APIs](https://spark.apache.org/docs/latest/api/python/reference/pyspark.html#rdd-apis), [SparkContext APIs](https://spark.apache.org/docs/latest/api/python/reference/pyspark.html#spark-context-apis), JVM properties, and custom catalyst rules/plans.
+
+If you are not familiar with Spark Connect, the primary benefit is that it provides a stable client API, decoupling the client from the Spark Driver. This makes Spark projects much easier to maintain over time, allowing you to update the Spark Driver and server-side dependencies without having to update the client. To learn more about Spark Connect, and explore its architecture details and benefits, visit here: [Spark Connect architecture](https://spark.apache.org/spark-connect/).
+
+## Selection Matrix for Spark Distributions
+
+This table guides you to which of the two distributions to select based on the type of Spark workloads.
+
+| Workloads Types | Spark Distribution and PySpark Package Mode | Spark Config Change |
+|-----------------------------------------------------------------------------------------------------|--------------------------------------------|---------------------------------------------|
+| - Only use standard DataFrame and Spark SQL APIs | _connect_ | None |
+| - Ability to access and debug Spark from IDE or interact in notebooks | | |
+| - Use of thin client to access Spark cluster from non-JVM languages | | |
+||||
+| - Access to RDD APIs | _classic_ | None |
+| - Access to SparkContext API and properties | | |
+| - Access to JVM properties | | |
+| - Access to private catalyst APIs: custom analyzer/optimizer rules, custom query plans | | |
Review Comment:
Added as per your request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "dmatrix (via GitHub)" <gi...@apache.org>.
dmatrix commented on code in PR #591:
URL: https://github.com/apache/spark-website/pull/591#discussion_r1968428097
##########
downloads.md:
##########
@@ -27,19 +53,39 @@ window.onload = function () {
3. Download Spark: <span id="spanDownloadLink"></span>
4. Verify this release using the <span id="sparkDownloadVerify"></span> and [project release KEYS](https://downloads.apache.org/spark/KEYS) by following these [procedures](https://www.apache.org/info/verification.html).
+classic
-Note that Spark 3 is pre-built with Scala 2.12 in general and Spark 3.2+ provides additional pre-built distribution with Scala 2.13.
+Note that Spark 4 is pre-built with Scala 2.13 in general, and Spark 3.5+ provides additional pre-built distribution with Scala 2.13.
-### Link with Spark
+### Link with Spark ###
Spark artifacts are [hosted in Maven Central](https://search.maven.org/search?q=g:org.apache.spark). You can add a Maven dependency with the following coordinates:
groupId: org.apache.spark
- artifactId: spark-core_2.12
- version: 3.5.4
+ artifactId: spark-core_2.13
+ version: 4.0.0
+
+### Installing with PyPi ###
+Like the two distributions mentioned above, PyPI will also have two PySpark package versions. The default is the _classic_ __pyspark__, while the _connect_ version is __pyspark-connect__ and is dependent on __pyspark__.
+
+Use the decision matrix above to select which PyPI PySpark package to use for your Spark workloads. Both <a href="https://pypi.org/project/pyspark/">PySpark</a> package versions are available on PyPI.
+
+### Installing PySpark Connect ###
+
+Since __pyspark-connect__ package is dependent on __pyspark__, __pyspark-connect__ will automatically install __pyspark__ for you. The __pyspark-connect__ package is mostly empty and merely enables Spark config `spark.api.mode` to _connect_ mode in the underlying pyspark package.
+
+`pip install pyspark-connect==4.0.0`
+
+Thereafter, follow the Spark Connect [quickstart guide](https://spark.apache.org/docs/4.0.0-preview2/api/python/getting_started/quickstart_connect.html) on how to use SparkSession.
+
+### Installing PySpark Classic ###
+
+Simply use `pip install pyspark==4.0.0`
Review Comment:
@srowen These are the steps I took to address your non-matching configs:
1. `bundle update`, updating the Gemfile.lock
2. Check the configs in GemFile vs. Gemfile.lock, they seem to have the same versions for Jekyll, rouge, and webrick
3. Commit the Gemfile.lock
4. `bundle exec jekyll build`
5. `bundle exec jekyll serve`
6. Check the download page.
Push the commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on code in PR #591:
URL: https://github.com/apache/spark-website/pull/591#discussion_r1968129724
##########
downloads.md:
##########
@@ -27,19 +53,39 @@ window.onload = function () {
3. Download Spark: <span id="spanDownloadLink"></span>
4. Verify this release using the <span id="sparkDownloadVerify"></span> and [project release KEYS](https://downloads.apache.org/spark/KEYS) by following these [procedures](https://www.apache.org/info/verification.html).
+classic
-Note that Spark 3 is pre-built with Scala 2.12 in general and Spark 3.2+ provides additional pre-built distribution with Scala 2.13.
+Note that Spark 4 is pre-built with Scala 2.13 in general, and Spark 3.5+ provides additional pre-built distribution with Scala 2.13.
-### Link with Spark
+### Link with Spark ###
Spark artifacts are [hosted in Maven Central](https://search.maven.org/search?q=g:org.apache.spark). You can add a Maven dependency with the following coordinates:
groupId: org.apache.spark
- artifactId: spark-core_2.12
- version: 3.5.4
+ artifactId: spark-core_2.13
+ version: 4.0.0
+
+### Installing with PyPi ###
+Like the two distributions mentioned above, PyPI will also have two PySpark package versions. The default is the _classic_ __pyspark__, while the _connect_ version is __pyspark-connect__ and is dependent on __pyspark__.
+
+Use the decision matrix above to select which PyPI PySpark package to use for your Spark workloads. Both <a href="https://pypi.org/project/pyspark/">PySpark</a> package versions are available on PyPI.
+
+### Installing PySpark Connect ###
+
+Since __pyspark-connect__ package is dependent on __pyspark__, __pyspark-connect__ will automatically install __pyspark__ for you. The __pyspark-connect__ package is mostly empty and merely enables Spark config `spark.api.mode` to _connect_ mode in the underlying pyspark package.
+
+`pip install pyspark-connect==4.0.0`
+
+Thereafter, follow the Spark Connect [quickstart guide](https://spark.apache.org/docs/4.0.0-preview2/api/python/getting_started/quickstart_connect.html) on how to use SparkSession.
+
+### Installing PySpark Classic ###
+
+Simply use `pip install pyspark==4.0.0`
Review Comment:
I would suggest reordering it, if required - flip the decision matrix as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on PR #591:
URL: https://github.com/apache/spark-website/pull/591#issuecomment-2685867931
https://github.com/apache/spark-website/pull/592
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "dmatrix (via GitHub)" <gi...@apache.org>.
dmatrix commented on code in PR #591:
URL: https://github.com/apache/spark-website/pull/591#discussion_r1968123460
##########
downloads.md:
##########
@@ -27,19 +53,39 @@ window.onload = function () {
3. Download Spark: <span id="spanDownloadLink"></span>
4. Verify this release using the <span id="sparkDownloadVerify"></span> and [project release KEYS](https://downloads.apache.org/spark/KEYS) by following these [procedures](https://www.apache.org/info/verification.html).
+classic
-Note that Spark 3 is pre-built with Scala 2.12 in general and Spark 3.2+ provides additional pre-built distribution with Scala 2.13.
+Note that Spark 4 is pre-built with Scala 2.13 in general, and Spark 3.5+ provides additional pre-built distribution with Scala 2.13.
-### Link with Spark
+### Link with Spark ###
Spark artifacts are [hosted in Maven Central](https://search.maven.org/search?q=g:org.apache.spark). You can add a Maven dependency with the following coordinates:
groupId: org.apache.spark
- artifactId: spark-core_2.12
- version: 3.5.4
+ artifactId: spark-core_2.13
+ version: 4.0.0
+
+### Installing with PyPi ###
+Like the two distributions mentioned above, PyPI will also have two PySpark package versions. The default is the _classic_ __pyspark__, while the _connect_ version is __pyspark-connect__ and is dependent on __pyspark__.
+
+Use the decision matrix above to select which PyPI PySpark package to use for your Spark workloads. Both <a href="https://pypi.org/project/pyspark/">PySpark</a> package versions are available on PyPI.
+
+### Installing PySpark Connect ###
+
+Since __pyspark-connect__ package is dependent on __pyspark__, __pyspark-connect__ will automatically install __pyspark__ for you. The __pyspark-connect__ package is mostly empty and merely enables Spark config `spark.api.mode` to _connect_ mode in the underlying pyspark package.
+
+`pip install pyspark-connect==4.0.0`
+
+Thereafter, follow the Spark Connect [quickstart guide](https://spark.apache.org/docs/4.0.0-preview2/api/python/getting_started/quickstart_connect.html) on how to use SparkSession.
+
+### Installing PySpark Classic ###
+
+Simply use `pip install pyspark==4.0.0`
Review Comment:
@mridulm Thanks for the feedback. The order here is not intentionally to lead or steer one to use _connect_ instead of _classic_. Instead, it is easier and simpler to be more expansive and comprehensive, switching between modes as the last.
It's much easier to read and follow the decision matrix than the reverse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on code in PR #591:
URL: https://github.com/apache/spark-website/pull/591#discussion_r1968129724
##########
downloads.md:
##########
@@ -27,19 +53,39 @@ window.onload = function () {
3. Download Spark: <span id="spanDownloadLink"></span>
4. Verify this release using the <span id="sparkDownloadVerify"></span> and [project release KEYS](https://downloads.apache.org/spark/KEYS) by following these [procedures](https://www.apache.org/info/verification.html).
+classic
-Note that Spark 3 is pre-built with Scala 2.12 in general and Spark 3.2+ provides additional pre-built distribution with Scala 2.13.
+Note that Spark 4 is pre-built with Scala 2.13 in general, and Spark 3.5+ provides additional pre-built distribution with Scala 2.13.
-### Link with Spark
+### Link with Spark ###
Spark artifacts are [hosted in Maven Central](https://search.maven.org/search?q=g:org.apache.spark). You can add a Maven dependency with the following coordinates:
groupId: org.apache.spark
- artifactId: spark-core_2.12
- version: 3.5.4
+ artifactId: spark-core_2.13
+ version: 4.0.0
+
+### Installing with PyPi ###
+Like the two distributions mentioned above, PyPI will also have two PySpark package versions. The default is the _classic_ __pyspark__, while the _connect_ version is __pyspark-connect__ and is dependent on __pyspark__.
+
+Use the decision matrix above to select which PyPI PySpark package to use for your Spark workloads. Both <a href="https://pypi.org/project/pyspark/">PySpark</a> package versions are available on PyPI.
+
+### Installing PySpark Connect ###
+
+Since __pyspark-connect__ package is dependent on __pyspark__, __pyspark-connect__ will automatically install __pyspark__ for you. The __pyspark-connect__ package is mostly empty and merely enables Spark config `spark.api.mode` to _connect_ mode in the underlying pyspark package.
+
+`pip install pyspark-connect==4.0.0`
+
+Thereafter, follow the Spark Connect [quickstart guide](https://spark.apache.org/docs/4.0.0-preview2/api/python/getting_started/quickstart_connect.html) on how to use SparkSession.
+
+### Installing PySpark Classic ###
+
+Simply use `pip install pyspark==4.0.0`
Review Comment:
I would suggest reordering it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "dmatrix (via GitHub)" <gi...@apache.org>.
dmatrix commented on code in PR #591:
URL: https://github.com/apache/spark-website/pull/591#discussion_r1968123460
##########
downloads.md:
##########
@@ -27,19 +53,39 @@ window.onload = function () {
3. Download Spark: <span id="spanDownloadLink"></span>
4. Verify this release using the <span id="sparkDownloadVerify"></span> and [project release KEYS](https://downloads.apache.org/spark/KEYS) by following these [procedures](https://www.apache.org/info/verification.html).
+classic
-Note that Spark 3 is pre-built with Scala 2.12 in general and Spark 3.2+ provides additional pre-built distribution with Scala 2.13.
+Note that Spark 4 is pre-built with Scala 2.13 in general, and Spark 3.5+ provides additional pre-built distribution with Scala 2.13.
-### Link with Spark
+### Link with Spark ###
Spark artifacts are [hosted in Maven Central](https://search.maven.org/search?q=g:org.apache.spark). You can add a Maven dependency with the following coordinates:
groupId: org.apache.spark
- artifactId: spark-core_2.12
- version: 3.5.4
+ artifactId: spark-core_2.13
+ version: 4.0.0
+
+### Installing with PyPi ###
+Like the two distributions mentioned above, PyPI will also have two PySpark package versions. The default is the _classic_ __pyspark__, while the _connect_ version is __pyspark-connect__ and is dependent on __pyspark__.
+
+Use the decision matrix above to select which PyPI PySpark package to use for your Spark workloads. Both <a href="https://pypi.org/project/pyspark/">PySpark</a> package versions are available on PyPI.
+
+### Installing PySpark Connect ###
+
+Since __pyspark-connect__ package is dependent on __pyspark__, __pyspark-connect__ will automatically install __pyspark__ for you. The __pyspark-connect__ package is mostly empty and merely enables Spark config `spark.api.mode` to _connect_ mode in the underlying pyspark package.
+
+`pip install pyspark-connect==4.0.0`
+
+Thereafter, follow the Spark Connect [quickstart guide](https://spark.apache.org/docs/4.0.0-preview2/api/python/getting_started/quickstart_connect.html) on how to use SparkSession.
+
+### Installing PySpark Classic ###
+
+Simply use `pip install pyspark==4.0.0`
Review Comment:
@mridulm Thanks for the feedback. The order here is not intentionally to lead or steer one to use _connect_ instead of _classic_. Instead, it is easier and simpler to be more expansive and comprehensive, switching between modes as the last: simple to complex.
I believe it's much easier to read and follow the decision matrix than the reverse: complex to simple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "dmatrix (via GitHub)" <gi...@apache.org>.
dmatrix commented on code in PR #591:
URL: https://github.com/apache/spark-website/pull/591#discussion_r1967004181
##########
downloads.md:
##########
@@ -16,6 +16,32 @@ window.onload = function () {
}
</script>
+## Introduction
+
+Unlike previous Apache Spark™ releases, Spark 4.0 has two distinct distributions: _classic_ and _connect_. As the names suggest, the _classic_ Spark version is the usual distribution you would expect for any new Spark release. The _connect_ distribution, in contrast, is the version with [Spark Connect](https://spark.apache.org/docs/4.0.0-preview2/spark-connect-overview.html) enabled by default. Which one should you download?
+
+Select the _connect_ version if your workloads only use standard [DataFrame](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/dataframe.html) and [Spark SQL](https://spark.apache.org/docs/latest/api/sql/) APIs. Choose the _classic_ version for traditional workloads requiring access to [RDD APIs](https://spark.apache.org/docs/latest/api/python/reference/pyspark.html#rdd-apis), [SparkContext APIs](https://spark.apache.org/docs/latest/api/python/reference/pyspark.html#spark-context-apis), JVM properties, and custom catalyst rules/plans.
+
+If you are not familiar with Spark Connect, the primary benefit is that it provides a stable client API, decoupling the client from the Spark Driver. This makes Spark projects much easier to maintain over time, allowing you to update the Spark Driver and server-side dependencies without having to update the client. To learn more about Spark Connect, and explore its architecture details and benefits, visit here: [Spark Connect architecture](https://spark.apache.org/spark-connect/).
+
+## Selection Matrix for Spark Distributions
+
+This table guides you to which of the two distributions to select based on the type of Spark workloads.
+
+| Workloads Types | Spark Distribution and PySpark Package Mode | Spark Config Change |
+|-----------------------------------------------------------------------------------------------------|--------------------------------------------|---------------------------------------------|
+| - Only use standard DataFrame and Spark SQL APIs | _connect_ | None |
Review Comment:
The spaces or empty lines are there as a demarker, if that's what you are referring to. Tables in MD are a bit clunky; I tried putting in "---" to create a line mark, but that didn't work. Not quite getting the same look as the table in the approved google doc. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on code in PR #591:
URL: https://github.com/apache/spark-website/pull/591#discussion_r1967330100
##########
downloads.md:
##########
@@ -16,6 +16,32 @@ window.onload = function () {
}
</script>
+## Introduction
+
+Unlike previous Apache Spark™ releases, Spark 4.0 has two distinct distributions: _classic_ and _connect_. As the names suggest, the _classic_ Spark version is the usual distribution you would expect for any new Spark release. The _connect_ distribution, in contrast, is the version with [Spark Connect](https://spark.apache.org/docs/4.0.0-preview2/spark-connect-overview.html) enabled by default. Which one should you download?
+
+Select the _connect_ version if your workloads only use standard [DataFrame](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/dataframe.html) and [Spark SQL](https://spark.apache.org/docs/latest/api/sql/) APIs. Choose the _classic_ version for traditional workloads requiring access to [RDD APIs](https://spark.apache.org/docs/latest/api/python/reference/pyspark.html#rdd-apis), [SparkContext APIs](https://spark.apache.org/docs/latest/api/python/reference/pyspark.html#spark-context-apis), JVM properties, and custom catalyst rules/plans.
+
+If you are not familiar with Spark Connect, the primary benefit is that it provides a stable client API, decoupling the client from the Spark Driver. This makes Spark projects much easier to maintain over time, allowing you to update the Spark Driver and server-side dependencies without having to update the client. To learn more about Spark Connect, and explore its architecture details and benefits, visit here: [Spark Connect architecture](https://spark.apache.org/spark-connect/).
+
+## Selection Matrix for Spark Distributions
+
+This table guides you to which of the two distributions to select based on the type of Spark workloads.
+
+| Workloads Types | Spark Distribution and PySpark Package Mode | Spark Config Change |
+|-----------------------------------------------------------------------------------------------------|--------------------------------------------|---------------------------------------------|
+| - Only use standard DataFrame and Spark SQL APIs | _connect_ | None |
+| - Ability to access and debug Spark from IDE or interact in notebooks | | |
+| - Use of thin client to access Spark cluster from non-JVM languages | | |
+||||
+| - Access to RDD APIs | _classic_ | None |
+| - Access to SparkContext API and properties | | |
+| - Access to JVM properties | | |
+| - Access to private catalyst APIs: custom analyzer/optimizer rules, custom query plans | | |
Review Comment:
Also add:
```
- Access to DataFrame and Spark SQL API's.
- Ability to access and debug Spark from IDE or interact in notebooks
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on code in PR #591:
URL: https://github.com/apache/spark-website/pull/591#discussion_r1967330100
##########
downloads.md:
##########
@@ -16,6 +16,32 @@ window.onload = function () {
}
</script>
+## Introduction
+
+Unlike previous Apache Spark™ releases, Spark 4.0 has two distinct distributions: _classic_ and _connect_. As the names suggest, the _classic_ Spark version is the usual distribution you would expect for any new Spark release. The _connect_ distribution, in contrast, is the version with [Spark Connect](https://spark.apache.org/docs/4.0.0-preview2/spark-connect-overview.html) enabled by default. Which one should you download?
+
+Select the _connect_ version if your workloads only use standard [DataFrame](https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/dataframe.html) and [Spark SQL](https://spark.apache.org/docs/latest/api/sql/) APIs. Choose the _classic_ version for traditional workloads requiring access to [RDD APIs](https://spark.apache.org/docs/latest/api/python/reference/pyspark.html#rdd-apis), [SparkContext APIs](https://spark.apache.org/docs/latest/api/python/reference/pyspark.html#spark-context-apis), JVM properties, and custom catalyst rules/plans.
+
+If you are not familiar with Spark Connect, the primary benefit is that it provides a stable client API, decoupling the client from the Spark Driver. This makes Spark projects much easier to maintain over time, allowing you to update the Spark Driver and server-side dependencies without having to update the client. To learn more about Spark Connect, and explore its architecture details and benefits, visit here: [Spark Connect architecture](https://spark.apache.org/spark-connect/).
+
+## Selection Matrix for Spark Distributions
+
+This table guides you to which of the two distributions to select based on the type of Spark workloads.
+
+| Workloads Types | Spark Distribution and PySpark Package Mode | Spark Config Change |
+|-----------------------------------------------------------------------------------------------------|--------------------------------------------|---------------------------------------------|
+| - Only use standard DataFrame and Spark SQL APIs | _connect_ | None |
+| - Ability to access and debug Spark from IDE or interact in notebooks | | |
+| - Use of thin client to access Spark cluster from non-JVM languages | | |
+||||
+| - Access to RDD APIs | _classic_ | None |
+| - Access to SparkContext API and properties | | |
+| - Access to JVM properties | | |
+| - Access to private catalyst APIs: custom analyzer/optimizer rules, custom query plans | | |
Review Comment:
Also add: - Access to DataFrame and Spark SQL API's.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on code in PR #591:
URL: https://github.com/apache/spark-website/pull/591#discussion_r1967331721
##########
downloads.md:
##########
@@ -27,19 +53,39 @@ window.onload = function () {
3. Download Spark: <span id="spanDownloadLink"></span>
4. Verify this release using the <span id="sparkDownloadVerify"></span> and [project release KEYS](https://downloads.apache.org/spark/KEYS) by following these [procedures](https://www.apache.org/info/verification.html).
+classic
-Note that Spark 3 is pre-built with Scala 2.12 in general and Spark 3.2+ provides additional pre-built distribution with Scala 2.13.
+Note that Spark 4 is pre-built with Scala 2.13 in general, and Spark 3.5+ provides additional pre-built distribution with Scala 2.13.
-### Link with Spark
+### Link with Spark ###
Spark artifacts are [hosted in Maven Central](https://search.maven.org/search?q=g:org.apache.spark). You can add a Maven dependency with the following coordinates:
groupId: org.apache.spark
- artifactId: spark-core_2.12
- version: 3.5.4
+ artifactId: spark-core_2.13
+ version: 4.0.0
+
+### Installing with PyPi ###
+Like the two distributions mentioned above, PyPI will also have two PySpark package versions. The default is the _classic_ __pyspark__, while the _connect_ version is __pyspark-connect__ and is dependent on __pyspark__.
+
+Use the decision matrix above to select which PyPI PySpark package to use for your Spark workloads. Both <a href="https://pypi.org/project/pyspark/">PySpark</a> package versions are available on PyPI.
+
+### Installing PySpark Connect ###
+
+Since __pyspark-connect__ package is dependent on __pyspark__, __pyspark-connect__ will automatically install __pyspark__ for you. The __pyspark-connect__ package is mostly empty and merely enables Spark config `spark.api.mode` to _connect_ mode in the underlying pyspark package.
+
+`pip install pyspark-connect==4.0.0`
+
+Thereafter, follow the Spark Connect [quickstart guide](https://spark.apache.org/docs/4.0.0-preview2/api/python/getting_started/quickstart_connect.html) on how to use SparkSession.
+
+### Installing PySpark Classic ###
+
+Simply use `pip install pyspark==4.0.0`
Review Comment:
I would suggest reordering these - until adoption increases.
Keep Classic above Connect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
Re: [PR] [DOC] Add new download page with Spark Connect description and instructions for multiple distributions [spark-website]
Posted by "srowen (via GitHub)" <gi...@apache.org>.
srowen commented on PR #591:
URL: https://github.com/apache/spark-website/pull/591#issuecomment-2685860420
Ah I missed this was 4.0-specific. Yeah I'd just revert it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org