You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by nishkamravi2 <gi...@git.apache.org> on 2015/03/19 03:24:46 UTC
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
GitHub user nishkamravi2 opened a pull request:
https://github.com/apache/spark/pull/5085
[SPARK-6406] Launcher backward compatibility issue-- "hadoop" should not be mandatory in spark assembly name
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nishkamravi2/spark master_nravi
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5085.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5085
----
commit 681b36f5fb63e14dc89e17813894227be9e2324f
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-05-08T07:05:33Z
Fix for SPARK-1758: failing test org.apache.spark.JavaAPISuite.wholeTextFiles
The prefix "file:" is missing in the string inserted as key in HashMap
commit 5108700230fd70b995e76598f49bdf328c971e77
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-06-03T22:25:22Z
Fix in Spark for the Concurrent thread modification issue (SPARK-1097, HADOOP-10456)
commit 6b840f017870207d23e75de224710971ada0b3d0
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-06-03T22:34:02Z
Undo the fix for SPARK-1758 (the problem is fixed)
commit df2aeb179fca4fc893803c72a657317f5b5539d7
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-06-09T19:02:59Z
Improved fix for ConcurrentModificationIssue (Spark-1097, Hadoop-10456)
commit eb663ca20c73f9c467192c95fc528c6f55f202be
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-06-09T19:04:39Z
Merge branch 'master' of https://github.com/apache/spark
commit 5423a03ddf4d747db7261d08a64e32f44e8be95e
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-06-10T20:06:07Z
Merge branch 'master' of https://github.com/apache/spark
commit 3bf8fad85813037504189cf1323d381fefb6dfbe
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-06-16T05:47:00Z
Merge branch 'master' of https://github.com/apache/spark
commit 2b630f94079b82df3ebae2b26a3743112afcd526
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-06-16T06:00:31Z
Accept memory input as "30g", "512M" instead of an int value, to be consistent with rest of Spark
commit efd688a4e15b79e92d162073035b03362fcf66f0
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-07-13T00:04:17Z
Merge branch 'master' of https://github.com/apache/spark
commit 2e69f112d1be59951cd32da4127d8b51bfa03338
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-21T23:17:15Z
Merge branch 'master' of https://github.com/apache/spark into master_nravi
commit ebcde10252e6c45169ea086e8426ec9997d46490
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-22T06:44:40Z
Modify default YARN memory_overhead-- from an additive constant to a multiplier (redone to resolve merge conflicts)
commit 1cf2d1ef57ed6d783df06dad36b9505bc74329fb
Author: nishkamravi2 <ni...@gmail.com>
Date: 2014-09-22T08:54:33Z
Update YarnAllocator.scala
commit f00fa311945c1eafa8957eae5c84719521761dcd
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-22T23:06:07Z
Improving logging for AM memoryOverhead
commit c726bd9f707ce182ec8d56ffecf9da87dcdb3091
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-25T01:19:32Z
Merge branch 'master' of https://github.com/apache/spark into master_nravi
Conflicts:
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
commit 362da5edfd04bd8bad990fb210a9e11b8494fa62
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-25T19:56:13Z
Additional changes for yarn memory overhead
commit 42c2c3d18862d3632c20931ecfe2c64883c5febf
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-25T20:02:49Z
Additional changes for yarn memory overhead issue
commit dac1047995c99f5a2670f934eb8d3a4ad9b532c8
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-25T21:20:38Z
Additional documentation for yarn memory overhead issue
commit 5ac2ec11629e19030ad5577da1eee2d135cc3d1c
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-25T21:25:44Z
Remove out
commit 35daa6498048cabb736316e2f19e565c99243b7e
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-25T21:59:22Z
Slight change in the doc for yarn memory overhead
commit 8f76c8b46379736aeb7dbe1a4d88729424a041f7
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-25T22:03:00Z
Doc change for yarn memory overhead
commit 636a9ffeb4a4ae0b941edd849dcbabf38821db53
Author: nishkamravi2 <ni...@gmail.com>
Date: 2014-09-30T18:33:28Z
Update YarnAllocator.scala
commit 5f8f9ede0fda5c7a4f6a411c746a3d893f550524
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-11-19T01:46:58Z
Merge branch 'master' of https://github.com/apache/spark into master_nravi
Conflicts:
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
commit 9f6583eb92b59e88b0994ef45dd0e1a418d349a6
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-11-19T03:42:25Z
Changes to maxResultSize code (improve error message and add condition to check if maxResultSize > 0)
commit 3e1b616a56cedec0b963b7f813a8e0195c3417e8
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-11-19T06:56:55Z
Modify test for maxResultSize
commit 535295a371b259a4fb1eb2ef01ea9f396aed40dd
Author: nishkamravi2 <ni...@gmail.com>
Date: 2014-11-19T21:56:18Z
Update TaskSetManager.scala
commit 5c9a4cb4eb12eb6cefbf9e9dda42b722abe1d3ed
Author: nishkamravi2 <ni...@gmail.com>
Date: 2014-11-19T21:57:01Z
Update TaskSetManagerSuite.scala
commit b446edc5ce672139564bcbc98217862a2921783b
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-11-19T23:30:17Z
Merge branch 'master' of https://github.com/apache/spark into master_nravi
commit 79ea8b418f709232ec6e106d7f58bb11ba86fa4a
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2015-02-19T06:03:18Z
Merge branch 'master' of https://github.com/apache/spark into master_nravi
commit f0d12de3a2179d2ee32bf7ae049223b5ee246795
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2015-02-19T07:45:31Z
Workaround for IllegalStateException caused by recent changes to BlockManager.stop
commit 3c5ddba51441b676391e43f3b09b9c59624ffef0
Author: nishkamravi2 <ni...@gmail.com>
Date: 2015-02-19T09:31:30Z
Update DiskBlockManager.scala
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5085#discussion_r26985197
--- Diff: launcher/src/main/java/org/apache/spark/launcher/Main.java ---
@@ -47,10 +47,14 @@
* character. On Windows, the output is a command line suitable for direct execution from the
* script.
*/
+
+ static String uberJarPath;
--- End diff --
This looks really ugly. I'd really prefer plumbing this to the command builders through the constructor. It's a little bit more code but much cleaner.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84216181
@nishkamravi2 - It's about the startup time of the interactive shell. That's the usability trade off. If the shell takes 300 extra ms to start up, it is not acceptable from a usability perspective. This is not about performance for a long running batch processing job.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84140696
re: 1, you gain 50ms (out of 150ms) on every invocation of the library. Might not be much, but there *is* a difference.
re: 2, why is it better? You haven't yet described a single case where it is better. It shouldn't matter for anyone, except someone who's trying to create a different layout for the Spark distribution, and I don't see any compelling argument for supporting that use case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84063585
> I'm a savvy customer. I name my jar file "spark-assembly.jar" because I see that as the outermost jar file in CDH (I get rid of unnecessary symlinks).
Well, in my view the customer is on his own as soon as he starts to modify the distribution. If he makes that change in 1.3, he'll have to figure out he needs to change other scripts too. Yes, it's easier to change a shell script than it is to change a compiled library, but I still don't see why Spark needs to support a random user who wants to deploy his own view of what the Spark distribution should look like. If that user is really savvy, he's savvy enough to fix what needs to be fixed on his own.
The problem with looseing the regex is theoretical, as Sean says. But consider this output, after running `sbt package assembly`:
$ ls -l assembly/target/scala-2.10/
total 156928
-rw-rw-r-- 1 vanzin vanzin 160683349 Mar 18 06:50 spark-assembly-1.3.0-SNAPSHOT-hadoop2.5.0.jar
-rw-rw-r-- 1 vanzin vanzin 290 Mar 20 09:35 spark-assembly_2.10-1.4.0-SNAPSHOT.jar
Now you have a non-theoretical breakage because of your change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-83891733
Marcelo, consider the following use case and see if it makes sense: I'm a savvy customer. I name my jar file "spark-assembly.jar" because I see that as the outermost jar file in CDH (I get rid of unnecessary symlinks). I modify (or rather bypass) most of compute-classpath.sh including the check for jar file name. Now I find that the check for the jar file name (which was a bad idea to begin with) is part of Spark code that gets compiled and therefore cannot be easily modified.
Btw, is having strict checks on assembly jar names a good idea? Or should I be able to name my jar "assembly.jar" as long as it is the only one and in the right folder?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84443666
Okay - if there is no big performance hit in launching from 1.3->1.4, that's fine with me.
I guess then the remaining question is whether to make a smaller change (simply make the wildcard match more liberal) vs. doing a larger change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-86850717
Addressed comments, fixed a couple of bugs. Can this be retested? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84140905
On top of that, placing the logic to find the assembly in a shell script means that now `CommandUtils.scala` would need to either replicate that logic in Scala, or execute the shell script (which is what it did before).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84165646
I talked to @nishkamravi2 offline, and we agreed on a different approach: getting rid of the launcher library (the perf difference not being in any way worrisome), and passing the location of the assembly file to the launcher's `Main` class.
That way the logic to find the assembly (still with the `*hadoop*` restriction) replaces the logic to find the launcher in the scripts, and the launcher library doesn't need code to look for the assembly, which should make use cases similar to his easier to handle.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-87183130
Given @vanzin and @nishkamravi2 have thoroughly discussed and approved this, I'm inclined to merge it. The changes look sane and do simplify the launcher somewhat.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-83825622
Sorry, clicked on the close button in error.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5085#discussion_r26746794
--- Diff: launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java ---
@@ -341,7 +341,7 @@ private String findAssembly(String scalaVersion) {
libdir = new File(sparkHome, String.format("assembly/target/scala-%s", scalaVersion));
}
- final Pattern re = Pattern.compile("spark-assembly.*hadoop.*\\.jar");
+ final Pattern re = Pattern.compile("spark-assembly.*[hadoop]*.*\\.jar");
--- End diff --
Shouldn't this bit be `.*(hadoop)?.*` but then can't it just collapse to matching anything with `.*`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-92546252
@vanzin @nishkamravi2 When I try to run `bin/spark-shell --master local-cluster[2,1,512]` my executors keep failing complaining that the scala classes are not found. I dug into this a little and it turns out that the assembly jar path from [this line](https://github.com/apache/spark/blob/d7f2c198678d1f5ffc1c6da3a6fb189b0a4ef070/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java#L189) is not what I expected: `$SPARK_HOME/launcher/target/scala-2.10/classes/`. Before this commit, it used to be the path of the assembly jar itself.
Is this a known issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-87011742
LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-86868475
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-86885995
[Test build #29295 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29295/consoleFull) for PR 5085 at commit [`bad4349`](https://github.com/apache/spark/commit/bad43499ed2518e0d5153ac56ce03beca3980e2c).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84414031
BTW just a pre-emptive comment that if you really feel like going down this path, you'll have to move the "invalid CEN header" (SPARK-1703) checks from AbstractCommandBuilder.java back to the shell scripts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-86154896
I can't comment on random code that you didn't change, but you'll need to change the code around this code in AbstractCommandBuilder:
needsDataNucleus = assemblyJar.getEntry("org/apache/hadoop/hive/ql/exec/") != null;
You can just skip the check for Hive and add all datanucleus jars if they're present. That's the same thing Sean did in the PR I mentioned before. (Bonus points for getting rid of the annoying message printed to stderr.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84509211
Sorry, I'm out for the weekend, not able to respond to the comments on this PR as promptly as I'd like to. I can make the changes and update the PR on Monday (hopefully) after some testing. If at that point we feel that the PR is not helping, we can certainly choose to not commit it. I think that the changes are small and in the right direction.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5085#discussion_r27091016
--- Diff: bin/spark-class ---
@@ -40,35 +40,44 @@ else
fi
fi
-# Look for the launcher. In non-release mode, add the compiled classes directly to the classpath
-# instead of looking for a jar file.
-SPARK_LAUNCHER_CP=
+# Find assembly jar
+SPARK_ASSEMBLY_JAR=
if [ -f $SPARK_HOME/RELEASE ]; then
--- End diff --
You're not changing this, but could you add quotes here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-86886035
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29295/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84214306
What is the intuitive explanation for why it is faster than before? It seems like launching the assembly twice would be slower.
Separately, is it worth doing a big refactoring for this issue? If we want to support this maybe we should just widen the regex. Or do you now feel the refactoring is an important design improvement, and it was simply instigated by this issue being reported?
For instance, if a change makes starting spark-shell > 100ms slower for people, IMO that's bad usability. Responsiveness for things like this can matter when people getting an impression of the overall performance of a system, and we have components specifically designed for quick interactive experience.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84137316
That's not a 30% gain. That's a constant gain of 0.05 seconds.
> What backwards compatibility concerns do you have? The library didn't exist before, so it's pretty hard to think of any.
The library didn't exist before and now it's required. That's a backward compatibility issue.
> As for performance, I didn't measure explicitly, but even if the gain is not that big, I really don't see why you see this as complicating anything.
Not sure what it brings to the table then?
> The issue with the breakage is not that it can be easily fixed. Is that it would exist at all. You'd be forcing people to fix their build when they did absolutely nothing wrong. What I haven't seen here is any argument that supports relaxing the regex without using some use case where the user is intentionally breaking the Spark distribution.
Your arguments may be considered somewhat contradictory to idea of this launcher library which is to provide additional flexibility in deploying Spark. For every customer who wants to launch Spark programmatically (which itself is a corner case), there is one who has a custom Spark deployment setup.
Besides, as I said, if you are convinced that strict checking of assembly file name is a good idea (which is very subjective), it is better to do that in a shell script than hardcode it in Spark code that gets compiled.
I think it may be best to weigh in opinion from others at this point @srowen @pwendell @andrewor14
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5085#discussion_r26985093
--- Diff: bin/spark-class ---
@@ -40,36 +40,24 @@ else
fi
fi
-# Look for the launcher. In non-release mode, add the compiled classes directly to the classpath
-# instead of looking for a jar file.
-SPARK_LAUNCHER_CP=
-if [ -f $SPARK_HOME/RELEASE ]; then
- LAUNCHER_DIR="$SPARK_HOME/lib"
- num_jars="$(ls -1 "$LAUNCHER_DIR" | grep "^spark-launcher.*\.jar$" | wc -l)"
- if [ "$num_jars" -eq "0" -a -z "$SPARK_LAUNCHER_CP" ]; then
- echo "Failed to find Spark launcher in $LAUNCHER_DIR." 1>&2
- echo "You need to build Spark before running this program." 1>&2
- exit 1
- fi
-
- LAUNCHER_JARS="$(ls -1 "$LAUNCHER_DIR" | grep "^spark-launcher.*\.jar$" || true)"
- if [ "$num_jars" -gt "1" ]; then
- echo "Found multiple Spark launcher jars in $LAUNCHER_DIR:" 1>&2
- echo "$LAUNCHER_JARS" 1>&2
- echo "Please remove all but one jar." 1>&2
- exit 1
- fi
-
- SPARK_LAUNCHER_CP="${LAUNCHER_DIR}/${LAUNCHER_JARS}"
-else
- LAUNCHER_DIR="$SPARK_HOME/launcher/target/scala-$SPARK_SCALA_VERSION"
- if [ ! -d "$LAUNCHER_DIR/classes" ]; then
- echo "Failed to find Spark launcher classes in $LAUNCHER_DIR." 1>&2
- echo "You need to build Spark before running this program." 1>&2
- exit 1
- fi
- SPARK_LAUNCHER_CP="$LAUNCHER_DIR/classes"
+# Find assembly jar
+SPARK_ASSEMBLY_JAR=
+ASSEMBLY_DIR="$SPARK_HOME/lib"
--- End diff --
Where are you looking for the assembly under `assembly/target/scala-$SPARK_SCALA_VERSION`?
That's needed to not break dev builds.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-83760527
Wait, I'm confused. What's the bug? Even your own comment says that the "hadoop" part is mandatory elsewhere. Here's the original shell script code:
https://github.com/apache/spark/blob/branch-1.3/bin/compute-classpath.sh#L79
"hadoop" is definitely mandatory there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-85743891
PR updated (instead of passing assembly jar on cmdline, we can find it dynamically using getLocation).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-85817443
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29144/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84139977
1. What do you gain by creating a separate launcher library? If you deviate from what we had before you need a compelling argument, not the other way round.
2. If at all needed, is it better to do a spark assembly file name check in spark-submit (or some such) than hardcode it in Spark?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-92548540
Looks like @davies filed a JIRA for this already: https://issues.apache.org/jira/browse/SPARK-6890
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-83638140
Thanks @srowen, that looks better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-92695146
Hey @andrewor14, is this a Mac/Windows issue? I can't seem to be able to reproduce it on Linux. Also, do you happen to have the path that was returned by getLocation().getPath() ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-85817437
[Test build #29144 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29144/consoleFull) for PR 5085 at commit [`4a45d6a`](https://github.com/apache/spark/commit/4a45d6adc12ac24df284aa2dc65fe915ad11243f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` implicit class DslSymbol(sym: Symbol) extends ImplicitAttribute `
* `case class Alias(child: Expression, name: String)(`
* `class Column(protected[sql] val expr: Expression) extends Logging `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84141192
There are two questions here.
"Should assembly file names be required to contain `hadoop`?"
Yes argument: it's the current behavior and might avoid picking up other random assembly-like JARs
No argument: might help people pick up a customized assembly JAR more naturally.
I personally think it doesn't matter much, and all else equal would leave it, considering the discussion here. Is there an end-user problem that hasn't been voiced yet?
"Should the launcher .jar be separate from the assembly?"
Yes argument: slightly faster startup. Maybe a desirable decoupling? it's how it works now.
No argument: deployment now means managing two files
I personally don't think the startup time difference matters. Right? This is a deployment question. It's not worse than when one had to ship around both scripts and an assembly, so I find this the same as before. That doesn't mean there isn't an argument for one uber jar.
Is that it then? question is: should the launcher be a part of the uber jar to simplify deployment over the current and past state of things, at the cost of a little startup time hit? I could see that one go either way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84003610
Going "off road" is up to the adventurous consumer. I think I'd produce a build with standard `make-distribution.sh` tools or the like, which should dump out a binary with `hadoop` in the name anyway, and link to that. I don't think I would modify scripts anyway.
That said, yeah, why not just look for `spark-assembly.*\.jar`? I don't see why `hadoop` is so important there. That's a plausible way to identify _the_ intended assembly JAR. Is there an outside risk that somebody is running with two `spark-assembly` files in the same dir, one of which has `hadoop` in the name, and this breaks that?
My hunch is there's a reason for the separate JAR, like decoupling the launcher from the launchee as was done previously with the scripts.
Anyway, net-net at the moment I'd support loosening the regex but it is a sort of theoretical problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-83824416
Please ignore the comment above (I misread the regex). However, we do need to relax the check on "hadoop". CDH itself names the outermost jar spark-assembly.jar. As to why we did not catch this issue with compute-classpath.sh, short answer: because we had our own custom version of it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84109293
Are we never loading the uber jar in this implementation? Can you quantify the gains you think you get by loading a separate launcher jar as compared to the uber jar? Are we convinced that those are significant enough to take away the simplicity of a single uber jar and potentially breaking backward compatibility? An uber jar allows a customer to copy it anywhere and point their classpath to it.
With regards to the assembly jar name issue: I'm not sure the above example is a breakage if it can be fixed by simply deleting one of those two jar files once the Spark launcher prints an explicit message stating so. In general, I'm not convinced that a strict assembly name check should be a solution for anything. But, if you are, a good compromise would be to do that check in spark-submit (or an outer shell script) than hard-coding it in the Spark code.
Sean, I think from a customer's point of view, the launcher is the application and the launchee is Spark library. That simplicity is desired, as compared to separating out stub code within Spark into a different library (for very little in return) and increasing complexity?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84207738
> can you explain a bit why the performance impact isn't worrisome?
Because it doesn't really save a significant amount of time. Invoking `org.apache.spark.launcher.Main` against the small library and the uber jar results in ~50ms extra for the uber jar in my system (probably a little more if you have spinning disks). That would still make the new launch path much faster than the old one (see the measurements I did in the original bug, SPARK-4924).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-85793537
Can an admin trigger jenkins for this PR please?
@nishkamravi2 , I think you can also remove the line in make-distribution.sh that packages the launcher jar. Otherwise, some minor things, but looks good.
Can you also change the PR title since it's not accurate? Something like "Launch Spark using the assembly instead of a separate launcher jar".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-85281486
Addressed two of the three comments @vanzin. The third one is tricky, let's discuss tomorrow when you're around.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-85802224
[Test build #29144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29144/consoleFull) for PR 5085 at commit [`4a45d6a`](https://github.com/apache/spark/commit/4a45d6adc12ac24df284aa2dc65fe915ad11243f).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 closed the pull request at:
https://github.com/apache/spark/pull/5085
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84476359
Just to clarify, personally, I still thing there's nothing wrong and that anybody who uses Spark the way it expects to be used (i.e. make-distribution.sh + calling bin/spark-submit) is unaffected by any recent changes. Nishkam feels strongly that I've broken his use case, even though I believe his use case is technically unsupported since he's modifying the launch scripts. But if he can change the library in a way that achieves many of the same original goals and still supports his use case, then I'm fine with it.
Pulling more logic into the shell scripts is something that goes against those goals, but it feels like a small enough change that I'm willing to overlook it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-83269405
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-83825207
And btw, we need to check this in
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84137997
> The library didn't exist before and now it's required. That's a backward compatibility issue.
That is not a backwards compatibility issue. If you try to launch your app against Spark 1.2 or 1.3 it will still work. If it didn't, *that* would be a backwards compatibility issue. The library being there is completely internal to Spark, and no user should need to care about it being there.
> there is one who has a custom Spark deployment setup.
As I've said before, if someone is customizing the layout of the Spark installation, it's up to them to fix what needs to be fixed. The shell scripts before made the same assumptions the launcher library currently does. So nothing has been broken.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84222414
When I said "before" I meant the code currently in branch-1.3. It's faster now, even with using the assembly for both invocations, because that's still loading the assembly less times than it did before (which could be up to 3 IIRC), it avoids invoking `java` just to see what version it is, and Java code is generally faster than bash code.
The added overhead of using the assembly for the first invocation (instead of the smaller jar) is not enough to make performance be as slow as before; it's still considerably faster (400ms before vs 150ms now in my system). So that's not really a big source of worries here, and the code changes shouldn't really be too bad.
Also, I'd argue that most of the startup time of spark-shell is actually loading all the Scala classes (which the launcher invocation doesn't do since it's pure java) and contacting the cluster manager. That should make the overhead of the startup script be mostly insignificant.
The remainder change then just builds on top of this change; instead of having code both in the scripts and in the library to find the assembly, it's just easier to tell the library where the assembly is (since the scripts already know).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5085#discussion_r27090991
--- Diff: bin/spark-class ---
@@ -40,35 +40,44 @@ else
fi
fi
-# Look for the launcher. In non-release mode, add the compiled classes directly to the classpath
-# instead of looking for a jar file.
-SPARK_LAUNCHER_CP=
+# Find assembly jar
+SPARK_ASSEMBLY_JAR=
if [ -f $SPARK_HOME/RELEASE ]; then
- LAUNCHER_DIR="$SPARK_HOME/lib"
- num_jars="$(ls -1 "$LAUNCHER_DIR" | grep "^spark-launcher.*\.jar$" | wc -l)"
- if [ "$num_jars" -eq "0" -a -z "$SPARK_LAUNCHER_CP" ]; then
- echo "Failed to find Spark launcher in $LAUNCHER_DIR." 1>&2
- echo "You need to build Spark before running this program." 1>&2
- exit 1
- fi
+ ASSEMBLY_DIR="$SPARK_HOME/lib"
+else
+ ASSEMBLY_DIR="$SPARK_HOME/assembly/target/scala-$SPARK_SCALA_VERSION"
+fi
- LAUNCHER_JARS="$(ls -1 "$LAUNCHER_DIR" | grep "^spark-launcher.*\.jar$" || true)"
- if [ "$num_jars" -gt "1" ]; then
- echo "Found multiple Spark launcher jars in $LAUNCHER_DIR:" 1>&2
- echo "$LAUNCHER_JARS" 1>&2
- echo "Please remove all but one jar." 1>&2
- exit 1
- fi
+num_jars="$(ls -1 "$ASSEMBLY_DIR" | grep "^spark-assembly.*hadoop.*\.jar$" | wc -l)"
+if [ "$num_jars" -eq "0" -a -z "$SPARK_ASSEMBLY_JAR" ]; then
+ echo "Failed to find Spark assembly in $ASSEMBLY_DIR." 1>&2
+ echo "You need to build Spark before running this program." 1>&2
+ exit 1
+fi
+ASSEMBLY_JARS="$(ls -1 "$ASSEMBLY_DIR" | grep "^spark-assembly.*hadoop.*\.jar$" || true)"
+if [ "$num_jars" -gt "1" ]; then
+ echo "Found multiple Spark assembly jars in $ASSEMBLY_DIR:" 1>&2
+ echo "$ASSEMBLY_JARS" 1>&2
+ echo "Please remove all but one jar." 1>&2
+ exit 1
+fi
- SPARK_LAUNCHER_CP="${LAUNCHER_DIR}/${LAUNCHER_JARS}"
+SPARK_ASSEMBLY_JAR="${ASSEMBLY_DIR}/${ASSEMBLY_JARS}"
+
+# Verify that versions of java used to build the jars and run Spark are compatible
+if [ -n "$JAVA_HOME" ]; then
+ JAR_CMD="$JAVA_HOME/bin/jar"
else
- LAUNCHER_DIR="$SPARK_HOME/launcher/target/scala-$SPARK_SCALA_VERSION"
- if [ ! -d "$LAUNCHER_DIR/classes" ]; then
- echo "Failed to find Spark launcher classes in $LAUNCHER_DIR." 1>&2
- echo "You need to build Spark before running this program." 1>&2
- exit 1
- fi
- SPARK_LAUNCHER_CP="$LAUNCHER_DIR/classes"
+ JAR_CMD="jar"
+fi
+
+jar_error_check=$("$JAR_CMD" -tf "$SPARK_ASSEMBLY_JAR" nonexistent/class/path 2>&1)
--- End diff --
You may need to add something like #4873 here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84475832
Loosening the wildcard match by itself can break sbt builds, so that doesn't sound like a good plan unless the wildcard still avoids matching other sbt-generated artifacts.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5085#discussion_r27091094
--- Diff: launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java ---
@@ -182,7 +186,7 @@ void addOptionString(List<String> cmd, String options) {
addToClassPath(cp, String.format("%s/core/target/jars/*", sparkHome));
}
- String assembly = findAssembly();
+ final String assembly = AbstractCommandBuilder.class.getProtectionDomain().getCodeSource().getLocation().getPath();
--- End diff --
Can you do just `getClass()` here instead of the more verbose version? Also, line looks pretty long.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84133444
And since you asked, here are times for the following command, over 100 runs:
SPARK_HOME=$PWD java -cp $JAR org.apache.spark.launcher.Main org.apache.spark.deploy.SparkSubmit
Against the assembly:
avg: 0.156157615185, min: 0.148868083954, max: 0.186276912689
Against the launcher jar:
avg: 0.101513769627, min: 0.0895018577576, max: 0.130180120468
So it's about 30% faster on average (with an SSD).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84205836
Hey @vanzin - can you explain a bit why the performance impact isn't worrisome? I think a first order goal was to avoid invoking multiple JVM's that have the assembly jar in the classpath, because it is slow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84165356
Marcelo and I discussed this offline and we have agreed on how to resolve this issue:
1. We will remove the check for launcher library and look for the uber jar instead
2. For the regex expression problem, we will move that check out into spark-class (with hadoop string included in the check) and pass the assembly file location as an argument to Launcher. That way we can completely eliminate the check for spark assembly jar file name from Spark code.
Will modify the PR accordingly after some testing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5085#discussion_r27091105
--- Diff: launcher/src/main/java/org/apache/spark/launcher/Main.java ---
@@ -31,7 +31,7 @@
class Main {
/**
- * Usage: Main [class] [class args]
+ * Usage: Main [uberJarPath] [class] [class args]
--- End diff --
Changes to this file are not needed anymore, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-85800826
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-83861301
Nishkam, can you explain why we need to remove "hadoop" from the regex now when the previous code had it? Can you explain one use case where the current code fails and the previous code worked?
CDH ships the correctly named assembly with a symlink named "spark-assembly.jar". So it works fine with the existing code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84127213
What backwards compatibility concerns do you have? The library didn't exist before, so it's pretty hard to think of any.
As for performance, I didn't measure explicitly, but even if the gain is not that big, I really don't see why you see this as complicating anything.
The issue with the breakage is not that it can be easily fixed. Is that it would exist at all. You'd be forcing people to fix their build when they did absolutely nothing wrong. What I haven't see here is any argument that supports relaxing the regex without using some use case where the user is intentionally breaking the Spark distribution.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84142521
> Is there an end-user problem that hasn't been voiced yet?
No, but there is an end-user issue that has been voice already: it can break launching jobs after you build spark with sbt (e.g. `sbt package assembly`). That's why the old scripts also had the same naming restriction.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-85244894
Thanks for the comments @vanzin. Will address them soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
GitHub user nishkamravi2 reopened a pull request:
https://github.com/apache/spark/pull/5085
[SPARK-6406] Launcher backward compatibility issue-- "hadoop" should not be mandatory in spark assembly name
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nishkamravi2/spark master_nravi
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5085.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5085
----
commit 681b36f5fb63e14dc89e17813894227be9e2324f
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-05-08T07:05:33Z
Fix for SPARK-1758: failing test org.apache.spark.JavaAPISuite.wholeTextFiles
The prefix "file:" is missing in the string inserted as key in HashMap
commit 5108700230fd70b995e76598f49bdf328c971e77
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-06-03T22:25:22Z
Fix in Spark for the Concurrent thread modification issue (SPARK-1097, HADOOP-10456)
commit 6b840f017870207d23e75de224710971ada0b3d0
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-06-03T22:34:02Z
Undo the fix for SPARK-1758 (the problem is fixed)
commit df2aeb179fca4fc893803c72a657317f5b5539d7
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-06-09T19:02:59Z
Improved fix for ConcurrentModificationIssue (Spark-1097, Hadoop-10456)
commit eb663ca20c73f9c467192c95fc528c6f55f202be
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-06-09T19:04:39Z
Merge branch 'master' of https://github.com/apache/spark
commit 5423a03ddf4d747db7261d08a64e32f44e8be95e
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-06-10T20:06:07Z
Merge branch 'master' of https://github.com/apache/spark
commit 3bf8fad85813037504189cf1323d381fefb6dfbe
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-06-16T05:47:00Z
Merge branch 'master' of https://github.com/apache/spark
commit 2b630f94079b82df3ebae2b26a3743112afcd526
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date: 2014-06-16T06:00:31Z
Accept memory input as "30g", "512M" instead of an int value, to be consistent with rest of Spark
commit efd688a4e15b79e92d162073035b03362fcf66f0
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-07-13T00:04:17Z
Merge branch 'master' of https://github.com/apache/spark
commit 2e69f112d1be59951cd32da4127d8b51bfa03338
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-21T23:17:15Z
Merge branch 'master' of https://github.com/apache/spark into master_nravi
commit ebcde10252e6c45169ea086e8426ec9997d46490
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-22T06:44:40Z
Modify default YARN memory_overhead-- from an additive constant to a multiplier (redone to resolve merge conflicts)
commit 1cf2d1ef57ed6d783df06dad36b9505bc74329fb
Author: nishkamravi2 <ni...@gmail.com>
Date: 2014-09-22T08:54:33Z
Update YarnAllocator.scala
commit f00fa311945c1eafa8957eae5c84719521761dcd
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-22T23:06:07Z
Improving logging for AM memoryOverhead
commit c726bd9f707ce182ec8d56ffecf9da87dcdb3091
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-25T01:19:32Z
Merge branch 'master' of https://github.com/apache/spark into master_nravi
Conflicts:
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
commit 362da5edfd04bd8bad990fb210a9e11b8494fa62
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-25T19:56:13Z
Additional changes for yarn memory overhead
commit 42c2c3d18862d3632c20931ecfe2c64883c5febf
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-25T20:02:49Z
Additional changes for yarn memory overhead issue
commit dac1047995c99f5a2670f934eb8d3a4ad9b532c8
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-25T21:20:38Z
Additional documentation for yarn memory overhead issue
commit 5ac2ec11629e19030ad5577da1eee2d135cc3d1c
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-25T21:25:44Z
Remove out
commit 35daa6498048cabb736316e2f19e565c99243b7e
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-25T21:59:22Z
Slight change in the doc for yarn memory overhead
commit 8f76c8b46379736aeb7dbe1a4d88729424a041f7
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-09-25T22:03:00Z
Doc change for yarn memory overhead
commit 636a9ffeb4a4ae0b941edd849dcbabf38821db53
Author: nishkamravi2 <ni...@gmail.com>
Date: 2014-09-30T18:33:28Z
Update YarnAllocator.scala
commit 5f8f9ede0fda5c7a4f6a411c746a3d893f550524
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-11-19T01:46:58Z
Merge branch 'master' of https://github.com/apache/spark into master_nravi
Conflicts:
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
yarn/common/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
commit 9f6583eb92b59e88b0994ef45dd0e1a418d349a6
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-11-19T03:42:25Z
Changes to maxResultSize code (improve error message and add condition to check if maxResultSize > 0)
commit 3e1b616a56cedec0b963b7f813a8e0195c3417e8
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-11-19T06:56:55Z
Modify test for maxResultSize
commit 535295a371b259a4fb1eb2ef01ea9f396aed40dd
Author: nishkamravi2 <ni...@gmail.com>
Date: 2014-11-19T21:56:18Z
Update TaskSetManager.scala
commit 5c9a4cb4eb12eb6cefbf9e9dda42b722abe1d3ed
Author: nishkamravi2 <ni...@gmail.com>
Date: 2014-11-19T21:57:01Z
Update TaskSetManagerSuite.scala
commit b446edc5ce672139564bcbc98217862a2921783b
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2014-11-19T23:30:17Z
Merge branch 'master' of https://github.com/apache/spark into master_nravi
commit 79ea8b418f709232ec6e106d7f58bb11ba86fa4a
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2015-02-19T06:03:18Z
Merge branch 'master' of https://github.com/apache/spark into master_nravi
commit f0d12de3a2179d2ee32bf7ae049223b5ee246795
Author: Nishkam Ravi <nr...@cloudera.com>
Date: 2015-02-19T07:45:31Z
Workaround for IllegalStateException caused by recent changes to BlockManager.stop
commit 3c5ddba51441b676391e43f3b09b9c59624ffef0
Author: nishkamravi2 <ni...@gmail.com>
Date: 2015-02-19T09:31:30Z
Update DiskBlockManager.scala
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-85223186
As I mentioned before, you need to take the "invalid CEN header" check from AbstractCommandBuilder and do it in the shell scripts now, otherwise that check is useless.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-92993834
Yes, the path was `$SPARK_HOME/launcher/target/scala-2.10/classes/`. I'm in a mac.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84215052
Hey @pwendell, a constant sub-second increase in execution time during the lifetime of an app should be acceptable. We tend to regress significantly more than with every Spark version as we pile up more code (we generally try to avoid reporting regressions of < 5%). Having to worry about two jar files as opposed to one on the other hand seems like a much more fundamental usability concern IMO.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-83922993
Also, we should not look for a separate launcher library jar. Like any other spark component it should be part of the uber jar. Modified PR accordingly (please see modified spark-class).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-84060892
Please *DO NOT* make a change to not have a separate launcher jar. That choice was explicit, because it's much, MUCH faster to load a jar file that is a few kB large than loading the uber jar.
Seriously, I still don't see any bug here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/5085
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5085#issuecomment-86868649
[Test build #29295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29295/consoleFull) for PR 5085 at commit [`bad4349`](https://github.com/apache/spark/commit/bad43499ed2518e0d5153ac56ce03beca3980e2c).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org