You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2019/03/05 14:28:37 UTC
[spark] branch branch-2.3 updated: [MINOR][DOCS] Clarify that Spark
apps should mark Spark as a 'provided' dependency, not package it
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-2.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.3 by this push:
new c326628 [MINOR][DOCS] Clarify that Spark apps should mark Spark as a 'provided' dependency, not package it
c326628 is described below
commit c32662877d90b379df87cb356c5d32b0bd0f4943
Author: Sean Owen <se...@databricks.com>
AuthorDate: Tue Mar 5 08:26:30 2019 -0600
[MINOR][DOCS] Clarify that Spark apps should mark Spark as a 'provided' dependency, not package it
## What changes were proposed in this pull request?
Spark apps do not need to package Spark. In fact it can cause problems in some cases. Our examples should show depending on Spark as a 'provided' dependency.
Packaging Spark makes the app much bigger by tens of megabytes. It can also bring in conflicting dependencies that wouldn't otherwise be a problem. https://issues.apache.org/jira/browse/SPARK-26146 was what reminded me of this.
## How was this patch tested?
Doc build
Closes #23938 from srowen/Provided.
Authored-by: Sean Owen <se...@databricks.com>
Signed-off-by: Sean Owen <se...@databricks.com>
(cherry picked from commit 39092236819da097e9c8a3b2fa975105f08ae5b9)
Signed-off-by: Sean Owen <se...@databricks.com>
---
docs/cloud-integration.md | 1 +
docs/quick-start.md | 1 +
docs/streaming-programming-guide.md | 3 ++-
3 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/docs/cloud-integration.md b/docs/cloud-integration.md
index c150d9ef..dfb4691 100644
--- a/docs/cloud-integration.md
+++ b/docs/cloud-integration.md
@@ -87,6 +87,7 @@ is set to the chosen version of Spark:
<groupId>org.apache.spark</groupId>
<artifactId>hadoop-cloud_2.11</artifactId>
<version>${spark.version}</version>
+ <scope>provided</scope>
</dependency>
...
</dependencyManagement>
diff --git a/docs/quick-start.md b/docs/quick-start.md
index 07c520c..31f06c9 100644
--- a/docs/quick-start.md
+++ b/docs/quick-start.md
@@ -336,6 +336,7 @@ Note that Spark artifacts are tagged with a Scala version.
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_{{site.SCALA_BINARY_VERSION}}</artifactId>
<version>{{site.SPARK_VERSION}}</version>
+ <scope>provided</scope>
</dependency>
</dependencies>
</project>
diff --git a/docs/streaming-programming-guide.md b/docs/streaming-programming-guide.md
index 9efa9fc..a664d96 100644
--- a/docs/streaming-programming-guide.md
+++ b/docs/streaming-programming-guide.md
@@ -385,11 +385,12 @@ Similar to Spark, Spark Streaming is available through Maven Central. To write y
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_{{site.SCALA_BINARY_VERSION}}</artifactId>
<version>{{site.SPARK_VERSION}}</version>
+ <scope>provided</scope>
</dependency>
</div>
<div data-lang="SBT" markdown="1">
- libraryDependencies += "org.apache.spark" % "spark-streaming_{{site.SCALA_BINARY_VERSION}}" % "{{site.SPARK_VERSION}}"
+ libraryDependencies += "org.apache.spark" % "spark-streaming_{{site.SCALA_BINARY_VERSION}}" % "{{site.SPARK_VERSION}}" % "provided"
</div>
</div>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org