You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by srowen <gi...@git.apache.org> on 2017/03/24 21:19:02 UTC
[GitHub] spark pull request #17416: [SPARK-20075][CORE][WIP
GitHub user srowen opened a pull request:
https://github.com/apache/spark/pull/17416
[SPARK-20075][CORE][WIP
Initial attempt to add classifier, packaging support. Ivy integration\u2026 still not quite right.
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/srowen/spark SPARK-20075
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17416.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17416
----
commit 1524390d3140b7746c92e04439ba2ba8946ee9cd
Author: Sean Owen <so...@cloudera.com>
Date: 2017-03-24T21:17:54Z
Initial attempt to add classifier, packaging support. Ivy integration still not quite right.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
The essence of the problem is that the `ResolveReport` from `ivy.resolve` no longer contains any information about the classifier. I can't see why or what needs to be configured in Ivy to make that happen.
It is possible to write some code to stitch that extra info back onto the resolved artifacts, I guess. I think that would still leave some corner cases that would not work, like, depending on two artifacts that are identical except for a classifier.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/17416
So is the problem that it downloads `stanford-corenlp-3.4.1-models.jar` but thinks it is `stanford-corenlp-3.4.1.jar`?
It looks like it might be possible to add the classifier to `ModuleRevisionId.newInstance`, have you tried just doing that instead of `dd.addDependencyArtifact`?
If I have some time, I'll give it a shot to run it and see what's going on..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17416: [SPARK-20075][CORE][WIP] Support classifier, pack...
Posted by brkyvz <gi...@git.apache.org>.
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/17416#discussion_r108507118
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -829,33 +830,56 @@ private[spark] object SparkSubmitUtils {
var printStream = SparkSubmit.printStream
/**
- * Represents a Maven Coordinate
+ * Represents a Maven Coordinate. Refer to https://maven.apache.org/pom.html#Maven_Coordinates
+ * for more information. Standard ordering for a full coordinate is
+ * `groupId:artifactId:packaging:classifier:version` although packaging and classifier
+ * are optional.
+ *
* @param groupId the groupId of the coordinate
* @param artifactId the artifactId of the coordinate
+ * @param packaging Maven packaging type (e.g. jar), if any
+ * @param classifier Maven classifier, if any
* @param version the version of the coordinate
*/
- private[deploy] case class MavenCoordinate(groupId: String, artifactId: String, version: String) {
- override def toString: String = s"$groupId:$artifactId:$version"
+ private[deploy] case class MavenCoordinate(
+ groupId: String,
+ artifactId: String,
+ packaging: Option[String],
+ classifier: Option[String],
+ version: String) {
+
+ def this(groupId: String, artifactId: String, version: String) =
+ this(groupId, artifactId, None, None, version)
+
+ override def toString: String = {
+ (Seq(groupId, artifactId) ++ packaging ++ classifier ++ Seq(version)).mkString(":")
+ }
}
/**
* Extracts maven coordinates from a comma-delimited string. Coordinates should be provided
- * in the format `groupId:artifactId:version` or `groupId/artifactId:version`.
- * @param coordinates Comma-delimited string of maven coordinates
+ * in the format `groupId:artifactId:version`, `groupId:artifactId:packaging:version`, or
+ * `groupId:artifactId:packaging:classifier:version`. '/' can be used as a separator instead
+ * of ':'.
+ *
+ * @param coordinates Comma-delimited string of Maven coordinates
* @return Sequence of Maven coordinates
*/
def extractMavenCoordinates(coordinates: String): Seq[MavenCoordinate] = {
- coordinates.split(",").map { p =>
- val splits = p.replace("/", ":").split(":")
- require(splits.length == 3, s"Provided Maven Coordinates must be in the form " +
- s"'groupId:artifactId:version'. The coordinate provided is: $p")
- require(splits(0) != null && splits(0).trim.nonEmpty, s"The groupId cannot be null or " +
- s"be whitespace. The groupId provided is: ${splits(0)}")
- require(splits(1) != null && splits(1).trim.nonEmpty, s"The artifactId cannot be null or " +
- s"be whitespace. The artifactId provided is: ${splits(1)}")
- require(splits(2) != null && splits(2).trim.nonEmpty, s"The version cannot be null or " +
- s"be whitespace. The version provided is: ${splits(2)}")
- new MavenCoordinate(splits(0), splits(1), splits(2))
+ coordinates.split(",").map { coordinate =>
+ val splits = coordinate.split("[:/]")
+ require(splits.forall(split => split != null && split.trim.nonEmpty),
+ s"All elements of coordinate must be non-null and not whitespace; got $coordinate")
+ splits match {
+ case Array(groupId, artifactId, version) =>
+ new MavenCoordinate(groupId, artifactId, version)
+ case Array(groupId, artifactId, packaging, version) =>
+ new MavenCoordinate(groupId, artifactId, Some(packaging), None, version)
--- End diff --
can a classifier be given without the packaging?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/17416
Sure, I can PR against your branch. I didn't do anything new, just used the prev version of your `addDependenciesToIvy` method and it seemed to work for me. I also tried out what would happen if you tried to get 2 artifacts with the same mvn coords, just different classifiers, and it just picks the last one. Not sure if it was spark or ivy that ignored the other though... for example:
```
--packages edu.stanford.nlp:stanford-corenlp:3.4.1,edu.stanford.nlp:stanford-corenlp:jar:models:3.4.1
```
both get downloaded but only the models get on the classpath
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75179 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75179/testReport)** for PR 17416 at commit [`1524390`](https://github.com/apache/spark/commit/1524390d3140b7746c92e04439ba2ba8946ee9cd).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75179 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75179/testReport)** for PR 17416 at commit [`1524390`](https://github.com/apache/spark/commit/1524390d3140b7746c92e04439ba2ba8946ee9cd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17416: [SPARK-20075][CORE][WIP] Support classifier, pack...
Posted by brkyvz <gi...@git.apache.org>.
Github user brkyvz commented on a diff in the pull request:
https://github.com/apache/spark/pull/17416#discussion_r108508229
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -941,6 +965,12 @@ private[spark] object SparkSubmitUtils {
val ri = ModuleRevisionId.newInstance(mvn.groupId, mvn.artifactId, mvn.version)
val dd = new DefaultDependencyDescriptor(ri, false, false)
dd.addDependencyConfiguration(ivyConfName, ivyConfName + "(runtime)")
+ if (mvn.classifier.isDefined) {
--- End diff --
I have a feeling we might be missing it when we copy the files from the `local repo` to the ivy cache. The way the resolution works is as follows:
1. Ivy searches local cache
2. Ivy searches remote repos
3. Ivy downloads from remote repo to local repo
4. Ivy copies the file from local repo to cache
Then the files are served from the local cache. Step 4 is called `retrieval` and happens on line 1159. I think that's what you're missing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17416: [SPARK-20075][CORE][WIP] Support classifier, pack...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17416#discussion_r108002222
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -941,6 +965,12 @@ private[spark] object SparkSubmitUtils {
val ri = ModuleRevisionId.newInstance(mvn.groupId, mvn.artifactId, mvn.version)
val dd = new DefaultDependencyDescriptor(ri, false, false)
dd.addDependencyConfiguration(ivyConfName, ivyConfName + "(runtime)")
+ if (mvn.classifier.isDefined) {
--- End diff --
CC @brkyvz and @BryanCutler (and maybe @vanzin) for a look. This does _not_ quite work yet, but it's close. I believe it's this stanza that's not quite right, where I try to add classifier info to Ivy's description of an artifact.
For example `./bin/spark-shell --packages edu.stanford.nlp:stanford-corenlp:jar:models:3.4.1` works and seems to resolve the right artifact with classifier locally, no errors, but it doesn't seem to be on the classpath.
I don't need you to debug it for me but mostly wondering if you know Ivy much better than I (not seen it until today) and can see an obvious error in this approach. I can't quite figure Ivy's concept that maps to "classifier" in Maven.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75936/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75914/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75389/testReport)** for PR 17416 at commit [`dec7bfb`](https://github.com/apache/spark/commit/dec7bfb8911fc03c8813fe809c3b09014a659791).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/17416
@srowen , I opened https://github.com/srowen/spark/pull/1 in case you didn't see it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75179/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Thanks @BryanCutler ! so, eh, do you mean you have a working version of this change? feel free to open it as a PR to master or to this PR or whatever, whatever's simplest. I'd be pleased to have a look and get it in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75936/testReport)** for PR 17416 at commit [`4126702`](https://github.com/apache/spark/commit/41267020701be4877be352e1678113bd9870ec12).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/17416
@srowen , I finally had some time to look into this and I was able to get the correct jar on the classpath. The fix was to use the code you had in the previous commit for `SparkSubmit.addDependenciesToIvy` so that the extraAttributes is set with `dd.addDependencyArtifact` and doesn't need to be in the `ModuleRevisionId` - so it was my bad advice that probably screwed this up :<
The reason is that when the `DefaultDependencyDescriptor` gets resolved in DefaultModuleDescriptor.java, if there are no artifacts defined, it adds 1 but does not copy over the `extraAttributes`, that's why the resolve report doesn't know about it. But if there are artifacts (which come from `addDependencyArtifact`) then the `extraAttributes` are carried over. wow, this is really confusing - hopefully this makes sense, see the code below `BasicResolver. getDependency(DependencyDescriptor dd, ResolveData data)` calls `DefaultModuleDescriptor.newDefaultInstance`
```java
public static DefaultModuleDescriptor newDefaultInstance(ModuleRevisionId mrid,
DependencyArtifactDescriptor[] artifacts) {
DefaultModuleDescriptor moduleDescriptor = new DefaultModuleDescriptor(mrid, "release",
null, true);
moduleDescriptor.addConfiguration(new Configuration(DEFAULT_CONFIGURATION));
if (artifacts != null && artifacts.length > 0) {
for (int i = 0; i < artifacts.length; i++) {
moduleDescriptor.addArtifact(DEFAULT_CONFIGURATION,
new MDArtifact(moduleDescriptor, artifacts[i].getName(),
artifacts[i].getType(), artifacts[i].getExt(), artifacts[i].getUrl(),
artifacts[i].getExtraAttributes()));
}
} else {
moduleDescriptor.addArtifact(DEFAULT_CONFIGURATION, new MDArtifact(moduleDescriptor,
mrid.getName(), "jar", "jar"));
}
moduleDescriptor.setLastModified(System.currentTimeMillis());
return moduleDescriptor;
}
```
I think that some other code you added in the second commit was also required, which is maybe why it didn't work for you in the first place, but give it another try. Here is the output from my test, looks like it should work now:
```
bin/spark-submit --packages edu.stanford.nlp:stanford-corenlp:jar:models:3.4.1 -v examples/src/main/python/pi.py
Using properties file: /home/bryan/git/spark/conf/spark-defaults.conf
Adding default property: spark.history.fs.logDirectory=/home/bryan/git/spark/logs/history
Adding default property: spark.eventLog.dir=/home/bryan/git/spark/logs/history
Adding default property: drill.enable_unsafe_memory_access=false
Warning: Ignoring non-spark config property: drill.enable_unsafe_memory_access=false
Parsed arguments:
master local[*]
deployMode null
executorMemory null
executorCores null
totalExecutorCores null
propertiesFile /home/bryan/git/spark/conf/spark-defaults.conf
driverMemory null
driverCores null
driverExtraClassPath null
driverExtraLibraryPath null
driverExtraJavaOptions null
supervise false
queue null
numExecutors null
files null
pyFiles null
archives null
mainClass null
primaryResource file:/home/bryan/git/spark/examples/src/main/python/pi.py
name pi.py
childArgs []
jars null
packages edu.stanford.nlp:stanford-corenlp:jar:models:3.4.1
packagesExclusions null
repositories null
verbose true
Spark properties used, including those specified through
--conf and those from the properties file /home/bryan/git/spark/conf/spark-defaults.conf:
(spark.history.fs.logDirectory,/home/bryan/git/spark/logs/history)
(spark.eventLog.dir,/home/bryan/git/spark/logs/history)
Ivy Default Cache set to: /home/bryan/.ivy2/cache
The jars for the packages stored in: /home/bryan/.ivy2/jars
:: loading settings :: url = jar:file:/home/bryan/git/spark/assembly/target/scala-2.11/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
edu.stanford.nlp#stanford-corenlp added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found edu.stanford.nlp#stanford-corenlp;3.4.1 in central
downloading https://repo1.maven.org/maven2/edu/stanford/nlp/stanford-corenlp/3.4.1/stanford-corenlp-3.4.1-models.jar ...
[SUCCESSFUL ] edu.stanford.nlp#stanford-corenlp;3.4.1!stanford-corenlp.jar (118730ms)
:: resolution report :: resolve 1164ms :: artifacts dl 118732ms
:: modules in use:
edu.stanford.nlp#stanford-corenlp;3.4.1 from central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 1 | 1 | 0 | 0 || 1 | 1 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
0 artifacts copied, 1 already retrieved (0kB/5ms)
Main class:
org.apache.spark.deploy.PythonRunner
Arguments:
file:/home/bryan/git/spark/examples/src/main/python/pi.py
/home/bryan/.ivy2/jars/edu.stanford.nlp_stanford-corenlp-models-3.4.1.jar
System properties:
(SPARK_SUBMIT,true)
(spark.submit.pyFiles,/home/bryan/.ivy2/jars/edu.stanford.nlp_stanford-corenlp-models-3.4.1.jar)
(spark.history.fs.logDirectory,/home/bryan/git/spark/logs/history)
(spark.files,file:/home/bryan/git/spark/examples/src/main/python/pi.py,file:/home/bryan/.ivy2/jars/edu.stanford.nlp_stanford-corenlp-models-3.4.1.jar)
(spark.app.name,pi.py)
(spark.jars,file:/home/bryan/.ivy2/jars/edu.stanford.nlp_stanford-corenlp-models-3.4.1.jar)
(spark.submit.deployMode,client)
(spark.eventLog.dir,/home/bryan/git/spark/logs/history)
(spark.master,local[*])
Classpath elements:
/home/bryan/.ivy2/jars/edu.stanford.nlp_stanford-corenlp-models-3.4.1.jar
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17416: [SPARK-20075][CORE][WIP] Support classifier, pack...
Posted by srowen <gi...@git.apache.org>.
Github user srowen closed the pull request at:
https://github.com/apache/spark/pull/17416
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Yeah @BryanCutler basically I was just modifying the existing test to include an artifact with a classifier. If you just add one to the first line in this test in `SparkSubmitUtilsSuite` I think you'd see it:
```
test("search for artifact at local repositories") {
val main = new MavenCoordinate("my.great.lib", "mylib", "0.1")
val dep = "my.great.dep:mydep:0.5"
```
It won't find the dependency then.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75389/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Yeah, I think it's a little bigger than that -- the test suggests that it won't load dependencies of an artifact with a classifier, which is better than nothing maybe, but still broken enough that I'm worried it is confusing to commit this.
I may noodle on this a bit more but I couldn't make a breakthrough yesterday. Just checking if you had any bright ideas from your investigation but if not I may shelve this one for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
OK @BryanCutler whatever's easiest for you. Whatever you have working, I can test and get merged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Well it's a little different, now that I've modified the tests to cover the classifier case. In this case, when the main artifact has a classifier, it doesn't find the dependency that the test sets up. It's definitely declared correctly in the POM that's created. I'm guessing somehow adding the DependencyArtifactDescriptor prevents it from attempting resolution or something. Still looking into it...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17416: [SPARK-20075][CORE][WIP] Support classifier, pack...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/17416#discussion_r108502945
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -941,6 +965,12 @@ private[spark] object SparkSubmitUtils {
val ri = ModuleRevisionId.newInstance(mvn.groupId, mvn.artifactId, mvn.version)
val dd = new DefaultDependencyDescriptor(ri, false, false)
dd.addDependencyConfiguration(ivyConfName, ivyConfName + "(runtime)")
+ if (mvn.classifier.isDefined) {
--- End diff --
As far as I can tell, this looks ok to me. This seems like the right way to add an artifact with classifier, but I'm not certain. Does the artifact show up in the `ResolveReport` at the end of `SparkSubmit.resolveMavenCoordinates`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
@sethah yeah it downloads the right models jar and copies it into the ivy cache. It just doesn't know that it's what belongs on the classpath. spark.jars doesn't get the memo and I can see why, because the `ResolveReport` doesn't return the fact that these artifacts have classifiers attached.
I can't figure it out; I'd be surprised if it can't work but don't know enough Ivy to fix it. I was thinking of manually mapping the input artifacts' classifiers to the resolved ones' classifiers but that won't work in the general case.
This is mostly seeing if you see something I missed. It's a corner case though causes a problem for a few key artifacts that need classifiers. If it's not obvious how to continue I think I'd shelve this for now and admit defeat.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75389 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75389/testReport)** for PR 17416 at commit [`dec7bfb`](https://github.com/apache/spark/commit/dec7bfb8911fc03c8813fe809c3b09014a659791).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75936 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75936/testReport)** for PR 17416 at commit [`4126702`](https://github.com/apache/spark/commit/41267020701be4877be352e1678113bd9870ec12).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17416: [SPARK-20075][CORE][WIP] Support classifier, pack...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17416#discussion_r108526688
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -829,33 +830,56 @@ private[spark] object SparkSubmitUtils {
var printStream = SparkSubmit.printStream
/**
- * Represents a Maven Coordinate
+ * Represents a Maven Coordinate. Refer to https://maven.apache.org/pom.html#Maven_Coordinates
+ * for more information. Standard ordering for a full coordinate is
+ * `groupId:artifactId:packaging:classifier:version` although packaging and classifier
+ * are optional.
+ *
* @param groupId the groupId of the coordinate
* @param artifactId the artifactId of the coordinate
+ * @param packaging Maven packaging type (e.g. jar), if any
+ * @param classifier Maven classifier, if any
* @param version the version of the coordinate
*/
- private[deploy] case class MavenCoordinate(groupId: String, artifactId: String, version: String) {
- override def toString: String = s"$groupId:$artifactId:$version"
+ private[deploy] case class MavenCoordinate(
+ groupId: String,
+ artifactId: String,
+ packaging: Option[String],
+ classifier: Option[String],
+ version: String) {
+
+ def this(groupId: String, artifactId: String, version: String) =
+ this(groupId, artifactId, None, None, version)
+
+ override def toString: String = {
+ (Seq(groupId, artifactId) ++ packaging ++ classifier ++ Seq(version)).mkString(":")
+ }
}
/**
* Extracts maven coordinates from a comma-delimited string. Coordinates should be provided
- * in the format `groupId:artifactId:version` or `groupId/artifactId:version`.
- * @param coordinates Comma-delimited string of maven coordinates
+ * in the format `groupId:artifactId:version`, `groupId:artifactId:packaging:version`, or
+ * `groupId:artifactId:packaging:classifier:version`. '/' can be used as a separator instead
+ * of ':'.
+ *
+ * @param coordinates Comma-delimited string of Maven coordinates
* @return Sequence of Maven coordinates
*/
def extractMavenCoordinates(coordinates: String): Seq[MavenCoordinate] = {
- coordinates.split(",").map { p =>
- val splits = p.replace("/", ":").split(":")
- require(splits.length == 3, s"Provided Maven Coordinates must be in the form " +
- s"'groupId:artifactId:version'. The coordinate provided is: $p")
- require(splits(0) != null && splits(0).trim.nonEmpty, s"The groupId cannot be null or " +
- s"be whitespace. The groupId provided is: ${splits(0)}")
- require(splits(1) != null && splits(1).trim.nonEmpty, s"The artifactId cannot be null or " +
- s"be whitespace. The artifactId provided is: ${splits(1)}")
- require(splits(2) != null && splits(2).trim.nonEmpty, s"The version cannot be null or " +
- s"be whitespace. The version provided is: ${splits(2)}")
- new MavenCoordinate(splits(0), splits(1), splits(2))
+ coordinates.split(",").map { coordinate =>
+ val splits = coordinate.split("[:/]")
+ require(splits.forall(split => split != null && split.trim.nonEmpty),
+ s"All elements of coordinate must be non-null and not whitespace; got $coordinate")
+ splits match {
+ case Array(groupId, artifactId, version) =>
+ new MavenCoordinate(groupId, artifactId, version)
+ case Array(groupId, artifactId, packaging, version) =>
+ new MavenCoordinate(groupId, artifactId, Some(packaging), None, version)
--- End diff --
It doesn't seem like it, given https://maven.apache.org/pom.html#Maven_Coordinates . If there are four dimensions, the third is packaging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75914 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75914/testReport)** for PR 17416 at commit [`23a3d0d`](https://github.com/apache/spark/commit/23a3d0dcaf0c69c29f68aa7d72f3ded21f04731c).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/17416
>There's a smaller corner case that doesn't quite seem to work, and that is adding both a classified and unclassified version of the same artifact,
Yeah, I'm not sure why exactly that doesn't work. I know I saw it downloads them both so I think it should work, but I didn't have time to look where it goes wrong. I agree that could be addressed later and this would be fine to be merged, pending the test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Thanks @BryanCutler , here's the current full output after trying to fix per https://github.com/apache/spark/pull/17416#discussion_r108508229 . Looks like progress I think but still missed something here. Working on it ...
```
Parsed arguments:
master local[*]
deployMode null
executorMemory null
executorCores null
totalExecutorCores null
propertiesFile null
driverMemory null
driverCores null
driverExtraClassPath null
driverExtraLibraryPath null
driverExtraJavaOptions null
supervise false
queue null
numExecutors null
files null
pyFiles null
archives null
mainClass org.apache.spark.repl.Main
primaryResource spark-shell
name Spark shell
childArgs []
jars null
packages edu.stanford.nlp:stanford-corenlp:jar:models:3.4.1
packagesExclusions null
repositories null
verbose true
Spark properties used, including those specified through
--conf and those from the properties file null:
Ivy Default Cache set to: /Users/srowen/.ivy2/cache
The jars for the packages stored in: /Users/srowen/.ivy2/jars
:: loading settings :: url = jar:file:/Users/srowen/Documents/Cloudera/spark/assembly/target/scala-2.11/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
edu.stanford.nlp#stanford-corenlp added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
confs: [default]
found edu.stanford.nlp#stanford-corenlp;3.4.1 in local-m2-cache
found com.io7m.xom#xom;1.2.10 in local-m2-cache
found xml-apis#xml-apis;1.3.03 in local-m2-cache
found xerces#xercesImpl;2.8.0 in local-m2-cache
found xalan#xalan;2.7.0 in local-m2-cache
found joda-time#joda-time;2.1 in local-m2-cache
found de.jollyday#jollyday;0.4.7 in local-m2-cache
found javax.xml.bind#jaxb-api;2.2.7 in local-m2-cache
found com.googlecode.efficient-java-matrix-library#ejml;0.23 in local-m2-cache
found javax.json#javax.json-api;1.0 in local-m2-cache
downloading file:/Users/srowen/.m2/repository/edu/stanford/nlp/stanford-corenlp/3.4.1/stanford-corenlp-3.4.1-models.jar ...
[SUCCESSFUL ] edu.stanford.nlp#stanford-corenlp;3.4.1!stanford-corenlp.jar (264ms)
downloading file:/Users/srowen/.m2/repository/com/io7m/xom/xom/1.2.10/xom-1.2.10.jar ...
[SUCCESSFUL ] com.io7m.xom#xom;1.2.10!xom.jar (2ms)
downloading file:/Users/srowen/.m2/repository/joda-time/joda-time/2.1/joda-time-2.1.jar ...
[SUCCESSFUL ] joda-time#joda-time;2.1!joda-time.jar (3ms)
downloading file:/Users/srowen/.m2/repository/de/jollyday/jollyday/0.4.7/jollyday-0.4.7.jar ...
[SUCCESSFUL ] de.jollyday#jollyday;0.4.7!jollyday.jar (2ms)
downloading file:/Users/srowen/.m2/repository/com/googlecode/efficient-java-matrix-library/ejml/0.23/ejml-0.23.jar ...
[SUCCESSFUL ] com.googlecode.efficient-java-matrix-library#ejml;0.23!ejml.jar (2ms)
downloading file:/Users/srowen/.m2/repository/javax/json/javax.json-api/1.0/javax.json-api-1.0.jar ...
[SUCCESSFUL ] javax.json#javax.json-api;1.0!javax.json-api.jar(bundle) (1ms)
downloading file:/Users/srowen/.m2/repository/xml-apis/xml-apis/1.3.03/xml-apis-1.3.03.jar ...
[SUCCESSFUL ] xml-apis#xml-apis;1.3.03!xml-apis.jar (2ms)
downloading file:/Users/srowen/.m2/repository/xerces/xercesImpl/2.8.0/xercesImpl-2.8.0.jar ...
[SUCCESSFUL ] xerces#xercesImpl;2.8.0!xercesImpl.jar (4ms)
downloading file:/Users/srowen/.m2/repository/xalan/xalan/2.7.0/xalan-2.7.0.jar ...
[SUCCESSFUL ] xalan#xalan;2.7.0!xalan.jar (6ms)
downloading file:/Users/srowen/.m2/repository/javax/xml/bind/jaxb-api/2.2.7/jaxb-api-2.2.7.jar ...
[SUCCESSFUL ] javax.xml.bind#jaxb-api;2.2.7!jaxb-api.jar (2ms)
:: resolution report :: resolve 7773ms :: artifacts dl 295ms
:: modules in use:
com.googlecode.efficient-java-matrix-library#ejml;0.23 from local-m2-cache in [default]
com.io7m.xom#xom;1.2.10 from local-m2-cache in [default]
de.jollyday#jollyday;0.4.7 from local-m2-cache in [default]
edu.stanford.nlp#stanford-corenlp;3.4.1 from local-m2-cache in [default]
javax.json#javax.json-api;1.0 from local-m2-cache in [default]
javax.xml.bind#jaxb-api;2.2.7 from local-m2-cache in [default]
joda-time#joda-time;2.1 from local-m2-cache in [default]
xalan#xalan;2.7.0 from local-m2-cache in [default]
xerces#xercesImpl;2.8.0 from local-m2-cache in [default]
xml-apis#xml-apis;1.3.03 from local-m2-cache in [default]
:: evicted modules:
xml-apis#xml-apis;2.0.2 by [xml-apis#xml-apis;1.3.03] in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 11 | 10 | 10 | 1 || 10 | 10 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
confs: [default]
10 artifacts copied, 0 already retrieved (212755kB/187ms)
Main class:
org.apache.spark.repl.Main
Arguments:
System properties:
(SPARK_SUBMIT,true)
(spark.app.name,Spark shell)
(spark.jars,file:/Users/srowen/.ivy2/jars/edu.stanford.nlp_stanford-corenlp-3.4.1.jar,file:/Users/srowen/.ivy2/jars/com.io7m.xom_xom-1.2.10.jar,file:/Users/srowen/.ivy2/jars/joda-time_joda-time-2.1.jar,file:/Users/srowen/.ivy2/jars/de.jollyday_jollyday-0.4.7.jar,file:/Users/srowen/.ivy2/jars/com.googlecode.efficient-java-matrix-library_ejml-0.23.jar,file:/Users/srowen/.ivy2/jars/javax.json_javax.json-api-1.0.jar,file:/Users/srowen/.ivy2/jars/xml-apis_xml-apis-1.3.03.jar,file:/Users/srowen/.ivy2/jars/xerces_xercesImpl-2.8.0.jar,file:/Users/srowen/.ivy2/jars/xalan_xalan-2.7.0.jar,file:/Users/srowen/.ivy2/jars/javax.xml.bind_jaxb-api-2.2.7.jar)
(spark.submit.deployMode,client)
(spark.master,local[*])
Classpath elements:
/Users/srowen/.ivy2/jars/edu.stanford.nlp_stanford-corenlp-3.4.1.jar
/Users/srowen/.ivy2/jars/com.io7m.xom_xom-1.2.10.jar
/Users/srowen/.ivy2/jars/joda-time_joda-time-2.1.jar
/Users/srowen/.ivy2/jars/de.jollyday_jollyday-0.4.7.jar
/Users/srowen/.ivy2/jars/com.googlecode.efficient-java-matrix-library_ejml-0.23.jar
/Users/srowen/.ivy2/jars/javax.json_javax.json-api-1.0.jar
/Users/srowen/.ivy2/jars/xml-apis_xml-apis-1.3.03.jar
/Users/srowen/.ivy2/jars/xerces_xercesImpl-2.8.0.jar
/Users/srowen/.ivy2/jars/xalan_xalan-2.7.0.jar
/Users/srowen/.ivy2/jars/javax.xml.bind_jaxb-api-2.2.7.jar
Warning: Local jar /Users/srowen/.ivy2/jars/edu.stanford.nlp_stanford-corenlp-3.4.1.jar does not exist, skipping.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/03/28 22:31:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/03/28 22:31:39 ERROR SparkContext: Failed to add file:/Users/srowen/.ivy2/jars/edu.stanford.nlp_stanford-corenlp-3.4.1.jar to Spark environment
java.io.FileNotFoundException: Jar /Users/srowen/.ivy2/jars/edu.stanford.nlp_stanford-corenlp-3.4.1.jar not found
at org.apache.spark.SparkContext.liftedTree1$1(SparkContext.scala:1820)
at org.apache.spark.SparkContext.addJar(SparkContext.scala:1817)
at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:466)
at org.apache.spark.SparkContext$$anonfun$12.apply(SparkContext.scala:466)
at scala.collection.immutable.List.foreach(List.scala:381)
...
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17416
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17416
**[Test build #75914 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75914/testReport)** for PR 17416 at commit [`23a3d0d`](https://github.com/apache/spark/commit/23a3d0dcaf0c69c29f68aa7d72f3ded21f04731c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Closing this for now -- happy to revive it if anyone has other ideas or wants to run with it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Thanks @BryanCutler ! yes that's working better. I can run, for example, `./bin/spark-shell --packages edu.stanford.nlp:stanford-corenlp:jar:models:3.4.1` and I can see that it's now on the classpath, because I can load its files with `getResourceAsStream`.
There's a smaller corner case that doesn't quite seem to work, and that is adding both a classified and unclassified version of the same artifact, as in `./bin/spark-shell --packages edu.stanford.nlp:stanford-corenlp:jar:models:3.4.1,edu.stanford.nlp:stanford-corenlp:3.4.1` The classifier version isn't loaded in this case.
Unless you see a clean fix for that too (I don't yet, still looking), I'm still inclined to merge this as it makes a significant set of artifacts now addressable, and loading via `spark-shell` is a bit best-effort anyway (this problem doesn't exist for a packaged app).
I need to add a test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17416
Yeah, it clearly downloads the right models .jar file, puts it in the .ivy cache correctly (I can find it there as expected) but then `spark.jars` isn't set correctly:
```
...
(spark.jars,file:/Users/srowen/.ivy2/jars/edu.stanford.nlp_stanford-corenlp-3.4.1.jar)
...
```
Let me push some more updates including the one you suggested, which didn't seem to change this, but I wouldn't expect it to. Still tracing down exactly what sets this value of spark.jars.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/17416
Oh, I see. I would have expected it to resolve those dependencies too, but it's hard to tell from the Ivy code what it will do. Was this part of a unit test you added here, or just some manual testing you were doing?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17416: [SPARK-20075][CORE][WIP] Support classifier, packaging i...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on the issue:
https://github.com/apache/spark/pull/17416
@srowen Can you confirm what happens when the jars are not found in your local m2 cache? Do you still find the `-models` jar in the ivy2 cache?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org