You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by vanzin <gi...@git.apache.org> on 2015/04/07 02:05:50 UTC
[GitHub] spark pull request: [minor] [examples] Avoid re-packaging unneeded...
GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/5379
[minor] [examples] Avoid re-packaging unneeded classes.
Add exclusions and explicit dependencies so that the examples
assembly does not duplicate classes already packaged in the main
assembly.
Also avoid relocating the commons-math3 package since it's already
a dependency of spark-core, and thus is already available in the
main assembly.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark examples-deps
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5379.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5379
----
commit 12c258ea561456bc6fa1ba04afc6c60827c3fad0
Author: Marcelo Vanzin <va...@cloudera.com>
Date: 2015-04-07T00:04:48Z
[minor] [examples] Avoid re-packaging unneeded classes.
Add exclusions and explicit dependencies so that the examples
assembly does not duplicate classes already packaged in the main
assembly.
Also avoid relocating the commons-math3 package since it's already
a dependency of spark-core, and thus is already available in the
main assembly.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [minor] [examples] Avoid packaging duplicate c...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5379#issuecomment-90312646
Yes, I checked all the exclusions I'm adding; they're either direct dependencies of Spark (and thus can be provided), or are transitive (e.g. Hadoop or some other Spark dependency pulls them in), so the assembly (or the cluster where the assembly is being run) is expected to provide them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [minor] [examples] Avoid packaging duplicate c...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5379#issuecomment-90311406
[Test build #29763 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29763/consoleFull) for PR 5379 at commit [`12c258e`](https://github.com/apache/spark/commit/12c258ea561456bc6fa1ba04afc6c60827c3fad0).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [minor] [examples] Avoid re-packaging unneeded...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5379#issuecomment-90293922
[Test build #29763 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29763/consoleFull) for PR 5379 at commit [`12c258e`](https://github.com/apache/spark/commit/12c258ea561456bc6fa1ba04afc6c60827c3fad0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [minor] [examples] Avoid packaging duplicate c...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5379#issuecomment-90301038
Individually those changes seem believable, like, marking Scala as provided and not including the Spark-shaded classes. Most affect the Cassandra dependency and the logic there is that they are definitely provided by Spark?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [minor] [examples] Avoid packaging duplicate c...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/5379
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [minor] [examples] Avoid packaging duplicate c...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5379#issuecomment-90859380
This LGTM as that is a huge decrease in the size of the examples JAR. I'll leave it open for comments one more day.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [minor] [examples] Avoid re-packaging unneeded...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5379#issuecomment-90293716
Locally, with a few `*-provided` profiles, the examples assembly shrunk from > 80MB to about 16MB.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [minor] [examples] Avoid packaging duplicate c...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5379#issuecomment-90311443
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29763/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org