You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2019/03/05 14:27:50 UTC
[spark] branch branch-2.4 updated: [MINOR][DOCS] Clarify that Spark
apps should mark Spark as a 'provided' dependency, not package it
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 498fb70 [MINOR][DOCS] Clarify that Spark apps should mark Spark as a 'provided' dependency, not package it
498fb70 is described below
commit 498fb70522f9320632eb2dffd8e62ce767278387
Author: Sean Owen <se...@databricks.com>
AuthorDate: Tue Mar 5 08:26:30 2019 -0600
[MINOR][DOCS] Clarify that Spark apps should mark Spark as a 'provided' dependency, not package it
## What changes were proposed in this pull request?
Spark apps do not need to package Spark. In fact it can cause problems in some cases. Our examples should show depending on Spark as a 'provided' dependency.
Packaging Spark makes the app much bigger by tens of megabytes. It can also bring in conflicting dependencies that wouldn't otherwise be a problem. https://issues.apache.org/jira/browse/SPARK-26146 was what reminded me of this.
## How was this patch tested?
Doc build
Closes #23938 from srowen/Provided.
Authored-by: Sean Owen <se...@databricks.com>
Signed-off-by: Sean Owen <se...@databricks.com>
(cherry picked from commit 39092236819da097e9c8a3b2fa975105f08ae5b9)
Signed-off-by: Sean Owen <se...@databricks.com>
---
docs/cloud-integration.md | 1 +
docs/quick-start.md | 1 +
docs/streaming-programming-guide.md | 3 ++-
3 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/docs/cloud-integration.md b/docs/cloud-integration.md
index 36753f6..b455cd4 100644
--- a/docs/cloud-integration.md
+++ b/docs/cloud-integration.md
@@ -87,6 +87,7 @@ is set to the chosen version of Spark:
<groupId>org.apache.spark</groupId>
<artifactId>hadoop-cloud_2.11</artifactId>
<version>${spark.version}</version>
+ <scope>provided</scope>
</dependency>
...
</dependencyManagement>
diff --git a/docs/quick-start.md b/docs/quick-start.md
index 28186c1..8bf0d6e 100644
--- a/docs/quick-start.md
+++ b/docs/quick-start.md
@@ -341,6 +341,7 @@ Note that Spark artifacts are tagged with a Scala version.
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_{{site.SCALA_BINARY_VERSION}}</artifactId>
<version>{{site.SPARK_VERSION}}</version>
+ <scope>provided</scope>
</dependency>
</dependencies>
</project>
diff --git a/docs/streaming-programming-guide.md b/docs/streaming-programming-guide.md
index 0ca0f2a..87ff4be 100644
--- a/docs/streaming-programming-guide.md
+++ b/docs/streaming-programming-guide.md
@@ -385,11 +385,12 @@ Similar to Spark, Spark Streaming is available through Maven Central. To write y
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_{{site.SCALA_BINARY_VERSION}}</artifactId>
<version>{{site.SPARK_VERSION}}</version>
+ <scope>provided</scope>
</dependency>
</div>
<div data-lang="SBT" markdown="1">
- libraryDependencies += "org.apache.spark" % "spark-streaming_{{site.SCALA_BINARY_VERSION}}" % "{{site.SPARK_VERSION}}"
+ libraryDependencies += "org.apache.spark" % "spark-streaming_{{site.SCALA_BINARY_VERSION}}" % "{{site.SPARK_VERSION}}" % "provided"
</div>
</div>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org