You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by fangshil <gi...@git.apache.org> on 2018/05/15 21:30:06 UTC
[GitHub] spark pull request #21339: [SPARK-24287][Core] Spark -packages option should...
GitHub user fangshil opened a pull request:
https://github.com/apache/spark/pull/21339
[SPARK-24287][Core] Spark -packages option should support classifier, no-transitive, and custom conf
## What changes were proposed in this pull request?
We should extend Spark's -package option to support:
1. Turn-off transitive dependency on a given artifact(like spark-avro)
2. Resolving a given artifact with classifier (like avro-mapred-1.7.4-h2.jar
3. Resolving a given artifact with custom ivy conf
4. Excluding particular transitive dependencies from a given artifact. We currently only have top-level exclusion rule applies for all artifacts.
We have tested this patch internally and it greatly increases the flexibility when user uses -packages option
## How was this patch tested?
added unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fangshil/spark SPARK-24287
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21339.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21339
----
commit 30f378f1ddba7a23b7a72ee96bb1648fd80dbbd3
Author: Fangshi Li <fl...@...>
Date: 2017-04-11T08:27:05Z
[SPARK-24287][Core] Spark -packages option should support classifier, no-transitive, and custom conf
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21339: [SPARK-24287][Core] Spark -packages option should suppor...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/21339
Hoo boy, I remember trying to make this work a long time ago. See https://github.com/apache/spark/pull/17416 . I wasn't able to. Let me run tests to see how this goes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21339: [SPARK-24287][Core] Spark -packages option should suppor...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21339
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21339: [SPARK-24287][Core] Spark -packages option should suppor...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21339
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21339: [SPARK-24287][Core] Spark -packages option should suppor...
Posted by wangyum <gi...@git.apache.org>.
Github user wangyum commented on the issue:
https://github.com/apache/spark/pull/21339
ping @fangshil
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21339: [SPARK-24287][Core] Spark -packages option should suppor...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21339
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21339: [SPARK-24287][Core] Spark -packages option should suppor...
Posted by taklwu <gi...@git.apache.org>.
Github user taklwu commented on the issue:
https://github.com/apache/spark/pull/21339
hi there, we also found the same problem, do we have any progress on this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21339: [SPARK-24287][Core] Spark -packages option should suppor...
Posted by fangshil <gi...@git.apache.org>.
Github user fangshil commented on the issue:
https://github.com/apache/spark/pull/21339
thanks for the follow up. i will rebase this patch to latest master
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21339: [SPARK-24287][Core] Spark -packages option should suppor...
Posted by fangshil <gi...@git.apache.org>.
Github user fangshil commented on the issue:
https://github.com/apache/spark/pull/21339
Sorry. I will start the progress now
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21339: [SPARK-24287][Core] Spark -packages option should suppor...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21339
**[Test build #4223 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4223/testReport)** for PR 21339 at commit [`30f378f`](https://github.com/apache/spark/commit/30f378f1ddba7a23b7a72ee96bb1648fd80dbbd3).
* This patch **fails Scala style tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21339: [SPARK-24287][Core] Spark -packages option should suppor...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21339
**[Test build #4223 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4223/testReport)** for PR 21339 at commit [`30f378f`](https://github.com/apache/spark/commit/30f378f1ddba7a23b7a72ee96bb1648fd80dbbd3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org