You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by nishkamravi2 <gi...@git.apache.org> on 2015/03/19 03:24:46 UTC

[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

GitHub user nishkamravi2 opened a pull request:

    https://github.com/apache/spark/pull/5085

    [SPARK-6406] Launcher backward compatibility issue-- "hadoop" should not be mandatory in spark assembly name

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nishkamravi2/spark master_nravi

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/5085.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5085
    
----
commit 681b36f5fb63e14dc89e17813894227be9e2324f
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-05-08T07:05:33Z

    Fix for SPARK-1758: failing test org.apache.spark.JavaAPISuite.wholeTextFiles
    
    The prefix "file:" is missing in the string inserted as key in HashMap

commit 5108700230fd70b995e76598f49bdf328c971e77
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-06-03T22:25:22Z

    Fix in Spark for the Concurrent thread modification issue (SPARK-1097, HADOOP-10456)

commit 6b840f017870207d23e75de224710971ada0b3d0
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-06-03T22:34:02Z

    Undo the fix for SPARK-1758 (the problem is fixed)

commit df2aeb179fca4fc893803c72a657317f5b5539d7
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-06-09T19:02:59Z

    Improved fix for ConcurrentModificationIssue (Spark-1097, Hadoop-10456)

commit eb663ca20c73f9c467192c95fc528c6f55f202be
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-06-09T19:04:39Z

    Merge branch 'master' of https://github.com/apache/spark

commit 5423a03ddf4d747db7261d08a64e32f44e8be95e
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-06-10T20:06:07Z

    Merge branch 'master' of https://github.com/apache/spark

commit 3bf8fad85813037504189cf1323d381fefb6dfbe
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-06-16T05:47:00Z

    Merge branch 'master' of https://github.com/apache/spark

commit 2b630f94079b82df3ebae2b26a3743112afcd526
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-06-16T06:00:31Z

    Accept memory input as "30g", "512M" instead of an int value, to be consistent with rest of Spark

commit efd688a4e15b79e92d162073035b03362fcf66f0
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-07-13T00:04:17Z

    Merge branch 'master' of https://github.com/apache/spark

commit 2e69f112d1be59951cd32da4127d8b51bfa03338
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-21T23:17:15Z

    Merge branch 'master' of https://github.com/apache/spark into master_nravi

commit ebcde10252e6c45169ea086e8426ec9997d46490
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-22T06:44:40Z

    Modify default YARN memory_overhead-- from an additive constant to a multiplier (redone to resolve merge conflicts)

commit 1cf2d1ef57ed6d783df06dad36b9505bc74329fb
Author: nishkamravi2 <ni...@gmail.com>
Date:   2014-09-22T08:54:33Z

    Update YarnAllocator.scala

commit f00fa311945c1eafa8957eae5c84719521761dcd
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-22T23:06:07Z

    Improving logging for AM memoryOverhead

commit c726bd9f707ce182ec8d56ffecf9da87dcdb3091
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-25T01:19:32Z

    Merge branch 'master' of https://github.com/apache/spark into master_nravi
    
    Conflicts:
    	yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala

commit 362da5edfd04bd8bad990fb210a9e11b8494fa62
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-25T19:56:13Z

    Additional changes for yarn memory overhead

commit 42c2c3d18862d3632c20931ecfe2c64883c5febf
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-25T20:02:49Z

    Additional changes for yarn memory overhead issue

commit dac1047995c99f5a2670f934eb8d3a4ad9b532c8
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-25T21:20:38Z

    Additional documentation for yarn memory overhead issue

commit 5ac2ec11629e19030ad5577da1eee2d135cc3d1c
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-25T21:25:44Z

    Remove out

commit 35daa6498048cabb736316e2f19e565c99243b7e
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-25T21:59:22Z

    Slight change in the doc for yarn memory overhead

commit 8f76c8b46379736aeb7dbe1a4d88729424a041f7
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-25T22:03:00Z

    Doc change for yarn memory overhead

commit 636a9ffeb4a4ae0b941edd849dcbabf38821db53
Author: nishkamravi2 <ni...@gmail.com>
Date:   2014-09-30T18:33:28Z

    Update YarnAllocator.scala

commit 5f8f9ede0fda5c7a4f6a411c746a3d893f550524
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-11-19T01:46:58Z

    Merge branch 'master' of https://github.com/apache/spark into master_nravi
    
    Conflicts:
    	yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
    	yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
    	yarn/common/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala

commit 9f6583eb92b59e88b0994ef45dd0e1a418d349a6
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-11-19T03:42:25Z

    Changes to maxResultSize code (improve error message and add condition to check if maxResultSize > 0)

commit 3e1b616a56cedec0b963b7f813a8e0195c3417e8
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-11-19T06:56:55Z

    Modify test for maxResultSize

commit 535295a371b259a4fb1eb2ef01ea9f396aed40dd
Author: nishkamravi2 <ni...@gmail.com>
Date:   2014-11-19T21:56:18Z

    Update TaskSetManager.scala

commit 5c9a4cb4eb12eb6cefbf9e9dda42b722abe1d3ed
Author: nishkamravi2 <ni...@gmail.com>
Date:   2014-11-19T21:57:01Z

    Update TaskSetManagerSuite.scala

commit b446edc5ce672139564bcbc98217862a2921783b
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-11-19T23:30:17Z

    Merge branch 'master' of https://github.com/apache/spark into master_nravi

commit 79ea8b418f709232ec6e106d7f58bb11ba86fa4a
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2015-02-19T06:03:18Z

    Merge branch 'master' of https://github.com/apache/spark into master_nravi

commit f0d12de3a2179d2ee32bf7ae049223b5ee246795
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2015-02-19T07:45:31Z

    Workaround for IllegalStateException caused by recent changes to BlockManager.stop

commit 3c5ddba51441b676391e43f3b09b9c59624ffef0
Author: nishkamravi2 <ni...@gmail.com>
Date:   2015-02-19T09:31:30Z

    Update DiskBlockManager.scala

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5085#discussion_r26985197
  
    --- Diff: launcher/src/main/java/org/apache/spark/launcher/Main.java ---
    @@ -47,10 +47,14 @@
        * character. On Windows, the output is a command line suitable for direct execution from the
        * script.
        */
    +
    +  static String uberJarPath;
    --- End diff --
    
    This looks really ugly. I'd really prefer plumbing this to the command builders through the constructor. It's a little bit more code but much cleaner.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84216181
  
    @nishkamravi2 - It's about the startup time of the interactive shell. That's the usability trade off. If the shell takes 300 extra ms to start up, it is not acceptable from a usability perspective. This is not about performance for a long running batch processing job.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84140696
  
    re: 1, you gain 50ms (out of 150ms) on every invocation of the library. Might not be much, but there *is* a difference.
    
    re: 2, why is it better? You haven't yet described a single case where it is better. It shouldn't matter for anyone, except someone who's trying to create a different layout for the Spark distribution, and I don't see any compelling argument for supporting that use case.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84063585
  
    >  I'm a savvy customer. I name my jar file "spark-assembly.jar" because I see that as the outermost jar file in CDH (I get rid of unnecessary symlinks).
    
    Well, in my view the customer is on his own as soon as he starts to modify the distribution. If he makes that change in 1.3, he'll have to figure out he needs to change other scripts too. Yes, it's easier to change a shell script than it is to change a compiled library, but I still don't see why Spark needs to support a random user who wants to deploy his own view of what the Spark distribution should look like. If that user is really savvy, he's savvy enough to fix what needs to be fixed on his own.
    
    The problem with looseing the regex is theoretical, as Sean says. But consider this output, after running `sbt package assembly`:
    
        $ ls -l assembly/target/scala-2.10/
        total 156928
        -rw-rw-r-- 1 vanzin vanzin 160683349 Mar 18 06:50 spark-assembly-1.3.0-SNAPSHOT-hadoop2.5.0.jar
        -rw-rw-r-- 1 vanzin vanzin       290 Mar 20 09:35 spark-assembly_2.10-1.4.0-SNAPSHOT.jar
    
    Now you have a non-theoretical breakage because of your change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-83891733
  
    Marcelo, consider the following use case and see if it makes sense: I'm a savvy customer. I name my jar file "spark-assembly.jar" because I see that as the outermost jar file in CDH (I get rid of unnecessary symlinks). I modify (or rather bypass) most of compute-classpath.sh including the check for jar file name. Now I find that the check for the jar file name (which was a bad idea to begin with) is part of Spark code that gets compiled and therefore cannot be easily modified. 
    
    Btw, is having strict checks on assembly jar names a good idea? Or should I be able to name my jar "assembly.jar" as long as it is the only one and in the right folder? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84443666
  
    Okay - if there is no big performance hit in launching from 1.3->1.4, that's fine with me.
    
    I guess then the remaining question is whether to make a smaller change (simply make the wildcard match more liberal) vs. doing a larger change.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-86850717
  
    Addressed comments, fixed a couple of bugs. Can this be retested? Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84140905
  
    On top of that, placing the logic to find the assembly in a shell script means that now `CommandUtils.scala` would need to either replicate that logic in Scala, or execute the shell script (which is what it did before).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84165646
  
    I talked to @nishkamravi2 offline, and we agreed on a different approach: getting rid of the launcher library (the perf difference not being in any way worrisome), and passing the location of the assembly file to the launcher's `Main` class.
    
    That way the logic to find the assembly (still with the `*hadoop*` restriction) replaces the logic to find the launcher in the scripts, and the launcher library doesn't need code to look for the assembly, which should make use cases similar to his easier to handle.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-87183130
  
    Given @vanzin and @nishkamravi2 have thoroughly discussed and approved this, I'm inclined to merge it. The changes look sane and do simplify the launcher somewhat.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-83825622
  
    Sorry, clicked on the close button in error.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5085#discussion_r26746794
  
    --- Diff: launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java ---
    @@ -341,7 +341,7 @@ private String findAssembly(String scalaVersion) {
           libdir = new File(sparkHome, String.format("assembly/target/scala-%s", scalaVersion));
         }
     
    -    final Pattern re = Pattern.compile("spark-assembly.*hadoop.*\\.jar");
    +    final Pattern re = Pattern.compile("spark-assembly.*[hadoop]*.*\\.jar");
    --- End diff --
    
    Shouldn't this bit be `.*(hadoop)?.*` but then can't it just collapse to matching anything with `.*`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-92546252
  
    @vanzin @nishkamravi2 When I try to run `bin/spark-shell --master local-cluster[2,1,512]` my executors keep failing complaining that the scala classes are not found. I dug into this a little and it turns out that the assembly jar path from [this line](https://github.com/apache/spark/blob/d7f2c198678d1f5ffc1c6da3a6fb189b0a4ef070/launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java#L189) is not what I expected: `$SPARK_HOME/launcher/target/scala-2.10/classes/`. Before this commit, it used to be the path of the assembly jar itself.
    
    Is this a known issue?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-87011742
  
    LGTM.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-86868475
  
    Jenkins, retest this please. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-86885995
  
      [Test build #29295 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29295/consoleFull) for   PR 5085 at commit [`bad4349`](https://github.com/apache/spark/commit/bad43499ed2518e0d5153ac56ce03beca3980e2c).
     * This patch **passes all tests**.
    
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84414031
  
    BTW just a pre-emptive comment that if you really feel like going down this path, you'll have to move the "invalid CEN header" (SPARK-1703) checks from AbstractCommandBuilder.java back to the shell scripts.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-86154896
  
    I can't comment on random code that you didn't change, but you'll need to change the code around this code in AbstractCommandBuilder:
    
         needsDataNucleus = assemblyJar.getEntry("org/apache/hadoop/hive/ql/exec/") != null;
    
    You can just skip the check for Hive and add all datanucleus jars if they're present. That's the same thing Sean did in the PR I mentioned before. (Bonus points for getting rid of the annoying message printed to stderr.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84509211
  
    Sorry, I'm out for the weekend, not able to respond to the comments on this PR as promptly as I'd like to. I can make the changes and update the PR on Monday (hopefully) after some testing. If at that point we feel that the PR is not helping, we can certainly choose to not commit it. I think that the changes are small and in the right direction. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5085#discussion_r27091016
  
    --- Diff: bin/spark-class ---
    @@ -40,35 +40,44 @@ else
       fi
     fi
     
    -# Look for the launcher. In non-release mode, add the compiled classes directly to the classpath
    -# instead of looking for a jar file.
    -SPARK_LAUNCHER_CP=
    +# Find assembly jar
    +SPARK_ASSEMBLY_JAR=
     if [ -f $SPARK_HOME/RELEASE ]; then
    --- End diff --
    
    You're not changing this, but could you add quotes here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-86886035
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29295/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84214306
  
    What is the intuitive explanation for why it is faster than before? It seems like launching the assembly twice would be slower.
    
    Separately, is it worth doing a big refactoring for this issue? If we want to support this maybe we should just widen the regex. Or do you now feel the refactoring is an important design improvement, and it was simply instigated by this issue being reported?
    
    For instance, if a change makes starting spark-shell > 100ms slower for people, IMO that's bad usability. Responsiveness for things like this can matter when people getting an impression of the overall performance of a system, and we have components specifically designed for quick interactive experience.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84137316
  
    That's not a 30% gain. That's a constant gain of 0.05 seconds.
    
    > What backwards compatibility concerns do you have? The library didn't exist before, so it's pretty hard to think of any.
    
    The library didn't exist before and now it's required. That's a backward compatibility issue.
    
    > As for performance, I didn't measure explicitly, but even if the gain is not that big, I really don't see why you see this as complicating anything. 
    
    Not sure what it brings to the table then?
    
    > The issue with the breakage is not that it can be easily fixed. Is that it would exist at all. You'd be forcing people to fix their build when they did absolutely nothing wrong. What I haven't seen here is any argument that supports relaxing the regex without using some use case where the user is intentionally breaking the Spark distribution.
    
    Your arguments may be considered somewhat contradictory to idea of this launcher library which is to provide additional flexibility in deploying Spark. For every customer who wants to launch Spark programmatically (which itself is a corner case), there is one who has a custom Spark deployment setup.  
    
    Besides, as I said, if you are convinced that strict checking of assembly file name is a good idea (which is very subjective), it is better to do that in a shell script than hardcode it in Spark code that gets compiled. 
    
    I think it may be best to weigh in opinion from others at this point @srowen @pwendell @andrewor14


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5085#discussion_r26985093
  
    --- Diff: bin/spark-class ---
    @@ -40,36 +40,24 @@ else
       fi
     fi
     
    -# Look for the launcher. In non-release mode, add the compiled classes directly to the classpath
    -# instead of looking for a jar file.
    -SPARK_LAUNCHER_CP=
    -if [ -f $SPARK_HOME/RELEASE ]; then
    -  LAUNCHER_DIR="$SPARK_HOME/lib"
    -  num_jars="$(ls -1 "$LAUNCHER_DIR" | grep "^spark-launcher.*\.jar$" | wc -l)"
    -  if [ "$num_jars" -eq "0" -a -z "$SPARK_LAUNCHER_CP" ]; then
    -    echo "Failed to find Spark launcher in $LAUNCHER_DIR." 1>&2
    -    echo "You need to build Spark before running this program." 1>&2
    -    exit 1
    -  fi
    -
    -  LAUNCHER_JARS="$(ls -1 "$LAUNCHER_DIR" | grep "^spark-launcher.*\.jar$" || true)"
    -  if [ "$num_jars" -gt "1" ]; then
    -    echo "Found multiple Spark launcher jars in $LAUNCHER_DIR:" 1>&2
    -    echo "$LAUNCHER_JARS" 1>&2
    -    echo "Please remove all but one jar." 1>&2
    -    exit 1
    -  fi
    -
    -  SPARK_LAUNCHER_CP="${LAUNCHER_DIR}/${LAUNCHER_JARS}"
    -else
    -  LAUNCHER_DIR="$SPARK_HOME/launcher/target/scala-$SPARK_SCALA_VERSION"
    -  if [ ! -d "$LAUNCHER_DIR/classes" ]; then
    -    echo "Failed to find Spark launcher classes in $LAUNCHER_DIR." 1>&2
    -    echo "You need to build Spark before running this program." 1>&2
    -    exit 1
    -  fi
    -  SPARK_LAUNCHER_CP="$LAUNCHER_DIR/classes"
    +# Find assembly jar
    +SPARK_ASSEMBLY_JAR=
    +ASSEMBLY_DIR="$SPARK_HOME/lib"
    --- End diff --
    
    Where are you looking for the assembly under `assembly/target/scala-$SPARK_SCALA_VERSION`?
    
    That's needed to not break dev builds.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-83760527
  
    Wait, I'm confused. What's the bug? Even your own comment says that the "hadoop" part is mandatory elsewhere. Here's the original shell script code:
    
    https://github.com/apache/spark/blob/branch-1.3/bin/compute-classpath.sh#L79
    
    "hadoop" is definitely mandatory there.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-85743891
  
    PR updated (instead of passing assembly jar on cmdline, we can find it dynamically using getLocation).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-85817443
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29144/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84139977
  
    1. What do you gain by creating a separate launcher library? If you deviate from what we had before you need a compelling argument, not the other way round.
    2. If at all needed, is it better to do a spark assembly file name check in spark-submit (or some such) than hardcode it in Spark?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-92548540
  
    Looks like @davies filed a JIRA for this already: https://issues.apache.org/jira/browse/SPARK-6890


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-83638140
  
    Thanks @srowen, that looks better.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-92695146
  
    Hey @andrewor14, is this a Mac/Windows issue? I can't seem to be able to reproduce it on Linux. Also, do you happen to have the path that was returned by getLocation().getPath() ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-85817437
  
      [Test build #29144 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29144/consoleFull) for   PR 5085 at commit [`4a45d6a`](https://github.com/apache/spark/commit/4a45d6adc12ac24df284aa2dc65fe915ad11243f).
     * This patch **fails Spark unit tests**.
    
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `    implicit class DslSymbol(sym: Symbol) extends ImplicitAttribute `
      * `case class Alias(child: Expression, name: String)(`
      * `class Column(protected[sql] val expr: Expression) extends Logging `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84141192
  
    There are two questions here.
    
    "Should assembly file names be required to contain `hadoop`?"
    
    Yes argument: it's the current behavior and might avoid picking up other random assembly-like JARs
    No argument: might help people pick up a customized assembly JAR more naturally.
    
    I personally think it doesn't matter much, and all else equal would leave it, considering the discussion here. Is there an end-user problem that hasn't been voiced yet?
    
    "Should the launcher .jar be separate from the assembly?"
    
    Yes argument: slightly faster startup. Maybe a desirable decoupling? it's how it works now.
    No argument: deployment now means managing two files
    
    I personally don't think the startup time difference matters. Right? This is a deployment question. It's not worse than when one had to ship around both scripts and an assembly, so I find this the same as before. That doesn't mean there isn't an argument for one uber jar.
    
    Is that it then? question is: should the launcher be a part of the uber jar to simplify deployment over the current and past state of things, at the cost of a little startup time hit? I could see that one go either way. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84003610
  
    Going "off road" is up to the adventurous consumer. I think I'd produce a build with standard `make-distribution.sh` tools or the like, which should dump out a binary with `hadoop` in the name anyway, and link to that. I don't think I would modify scripts anyway.
    
    That said, yeah, why not just look for `spark-assembly.*\.jar`? I don't see why `hadoop` is so important there. That's a plausible way to identify _the_ intended assembly JAR. Is there an outside risk that somebody is running with two `spark-assembly` files in the same dir, one of which has `hadoop` in the name, and this breaks that?
    
    My hunch is there's a reason for the separate JAR, like decoupling the launcher from the launchee as was done previously with the scripts.
    
    Anyway, net-net at the moment I'd support loosening the regex but it is a sort of theoretical problem.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-83824416
  
    Please ignore the comment above (I misread the regex). However, we do need to relax the check on "hadoop". CDH itself names the outermost jar spark-assembly.jar. As to why we did not catch this issue with compute-classpath.sh, short answer: because we had our own custom version of it. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84109293
  
    Are we never loading the uber jar in this implementation? Can you quantify the gains you think you get by loading a separate launcher jar as compared to the uber jar? Are we convinced that those are significant enough to take away the simplicity of a single uber jar and potentially breaking backward compatibility? An uber jar allows a customer to copy it anywhere and point their classpath to it. 
    
    With regards to the assembly jar name issue: I'm not sure the above example is a breakage if it can be fixed by simply deleting one of those two jar files once the Spark launcher prints an explicit message stating so. In general, I'm not convinced that a strict assembly name check should be a solution for anything. But, if you are, a good compromise would be to do that check in spark-submit (or an outer shell script) than hard-coding it in the Spark code. 
    
    Sean, I think from a customer's point of view, the launcher is the application and the launchee is Spark library. That simplicity is desired, as compared to separating out stub code within Spark into a different library (for very little in return) and increasing complexity?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84207738
  
    > can you explain a bit why the performance impact isn't worrisome?
    
    Because it doesn't really save a significant amount of time. Invoking `org.apache.spark.launcher.Main` against the small library and the uber jar results in ~50ms extra for the uber jar in my system (probably a little more if you have spinning disks). That would still make the new launch path much faster than the old one (see the measurements I did in the original bug, SPARK-4924).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-85793537
  
    Can an admin trigger jenkins for this PR please?
    
    @nishkamravi2 , I think you can also remove the line in make-distribution.sh that packages the launcher jar. Otherwise, some minor things, but looks good.
    
    Can you also change the PR title since it's not accurate? Something like "Launch Spark using the assembly instead of a separate launcher jar".


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-85281486
  
    Addressed two of the three comments @vanzin. The third one is tricky, let's discuss tomorrow when you're around.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-85802224
  
      [Test build #29144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29144/consoleFull) for   PR 5085 at commit [`4a45d6a`](https://github.com/apache/spark/commit/4a45d6adc12ac24df284aa2dc65fe915ad11243f).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 closed the pull request at:

    https://github.com/apache/spark/pull/5085


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84476359
  
    Just to clarify, personally, I still thing there's nothing wrong and that anybody who uses Spark the way it expects to be used (i.e. make-distribution.sh + calling bin/spark-submit) is unaffected by any recent changes. Nishkam feels strongly that I've broken his use case, even though I believe his use case is technically unsupported since he's modifying the launch scripts. But if he can change the library in a way that achieves many of the same original goals and still supports his use case, then I'm fine with it.
    
    Pulling more logic into the shell scripts is something that goes against those goals, but it feels like a small enough change that I'm willing to overlook it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-83269405
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-83825207
  
    And btw, we need to check this in 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84137997
  
    > The library didn't exist before and now it's required. That's a backward compatibility issue.
    
    That is not a backwards compatibility issue. If you try to launch your app against Spark 1.2 or 1.3 it will still work. If it didn't, *that* would be a backwards compatibility issue. The library being there is completely internal to Spark, and no user should need to care about it being there.
    
    >  there is one who has a custom Spark deployment setup. 
    
    As I've said before, if someone is customizing the layout of the Spark installation, it's up to them to fix what needs to be fixed. The shell scripts before made the same assumptions the launcher library currently does. So nothing has been broken.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84222414
  
    When I said "before" I meant the code currently in branch-1.3. It's faster now, even with using the assembly for both invocations, because that's still loading the assembly less times than it did before (which could be up to 3 IIRC), it avoids invoking `java` just to see what version it is, and Java code is generally faster than bash code.
    
    The added overhead of using the assembly for the first invocation (instead of the smaller jar) is not enough to make performance be as slow as before; it's still considerably faster (400ms before vs 150ms now in my system). So that's not really a big source of worries here, and the code changes shouldn't really be too bad.
    
    Also, I'd argue that most of the startup time of spark-shell is actually loading all the Scala classes (which the launcher invocation doesn't do since it's pure java) and contacting the cluster manager. That should make the overhead of the startup script be mostly insignificant.
    
    The remainder change then just builds on top of this change; instead of having code both in the scripts and in the library to find the assembly, it's just easier to tell the library where the assembly is (since the scripts already know).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5085#discussion_r27090991
  
    --- Diff: bin/spark-class ---
    @@ -40,35 +40,44 @@ else
       fi
     fi
     
    -# Look for the launcher. In non-release mode, add the compiled classes directly to the classpath
    -# instead of looking for a jar file.
    -SPARK_LAUNCHER_CP=
    +# Find assembly jar
    +SPARK_ASSEMBLY_JAR=
     if [ -f $SPARK_HOME/RELEASE ]; then
    -  LAUNCHER_DIR="$SPARK_HOME/lib"
    -  num_jars="$(ls -1 "$LAUNCHER_DIR" | grep "^spark-launcher.*\.jar$" | wc -l)"
    -  if [ "$num_jars" -eq "0" -a -z "$SPARK_LAUNCHER_CP" ]; then
    -    echo "Failed to find Spark launcher in $LAUNCHER_DIR." 1>&2
    -    echo "You need to build Spark before running this program." 1>&2
    -    exit 1
    -  fi
    +  ASSEMBLY_DIR="$SPARK_HOME/lib"
    +else
    +  ASSEMBLY_DIR="$SPARK_HOME/assembly/target/scala-$SPARK_SCALA_VERSION"
    +fi
     
    -  LAUNCHER_JARS="$(ls -1 "$LAUNCHER_DIR" | grep "^spark-launcher.*\.jar$" || true)"
    -  if [ "$num_jars" -gt "1" ]; then
    -    echo "Found multiple Spark launcher jars in $LAUNCHER_DIR:" 1>&2
    -    echo "$LAUNCHER_JARS" 1>&2
    -    echo "Please remove all but one jar." 1>&2
    -    exit 1
    -  fi
    +num_jars="$(ls -1 "$ASSEMBLY_DIR" | grep "^spark-assembly.*hadoop.*\.jar$" | wc -l)"
    +if [ "$num_jars" -eq "0" -a -z "$SPARK_ASSEMBLY_JAR" ]; then
    +  echo "Failed to find Spark assembly in $ASSEMBLY_DIR." 1>&2
    +  echo "You need to build Spark before running this program." 1>&2
    +  exit 1
    +fi
    +ASSEMBLY_JARS="$(ls -1 "$ASSEMBLY_DIR" | grep "^spark-assembly.*hadoop.*\.jar$" || true)"
    +if [ "$num_jars" -gt "1" ]; then
    +  echo "Found multiple Spark assembly jars in $ASSEMBLY_DIR:" 1>&2
    +  echo "$ASSEMBLY_JARS" 1>&2
    +  echo "Please remove all but one jar." 1>&2
    +  exit 1
    +fi
     
    -  SPARK_LAUNCHER_CP="${LAUNCHER_DIR}/${LAUNCHER_JARS}"
    +SPARK_ASSEMBLY_JAR="${ASSEMBLY_DIR}/${ASSEMBLY_JARS}"
    +
    +# Verify that versions of java used to build the jars and run Spark are compatible
    +if [ -n "$JAVA_HOME" ]; then
    +  JAR_CMD="$JAVA_HOME/bin/jar"
     else
    -  LAUNCHER_DIR="$SPARK_HOME/launcher/target/scala-$SPARK_SCALA_VERSION"
    -  if [ ! -d "$LAUNCHER_DIR/classes" ]; then
    -    echo "Failed to find Spark launcher classes in $LAUNCHER_DIR." 1>&2
    -    echo "You need to build Spark before running this program." 1>&2
    -    exit 1
    -  fi
    -  SPARK_LAUNCHER_CP="$LAUNCHER_DIR/classes"
    +  JAR_CMD="jar"
    +fi
    +
    +jar_error_check=$("$JAR_CMD" -tf "$SPARK_ASSEMBLY_JAR" nonexistent/class/path 2>&1)
    --- End diff --
    
    You may need to add something like #4873 here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84475832
  
    Loosening the wildcard match by itself can break sbt builds, so that doesn't sound like a good plan unless the wildcard still avoids matching other sbt-generated artifacts.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5085#discussion_r27091094
  
    --- Diff: launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java ---
    @@ -182,7 +186,7 @@ void addOptionString(List<String> cmd, String options) {
           addToClassPath(cp, String.format("%s/core/target/jars/*", sparkHome));
         }
     
    -    String assembly = findAssembly();
    +    final String assembly = AbstractCommandBuilder.class.getProtectionDomain().getCodeSource().getLocation().getPath();
    --- End diff --
    
    Can you do just `getClass()` here instead of the more verbose version? Also, line looks pretty long.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84133444
  
    And since you asked, here are times for the following command, over 100 runs:
    
        SPARK_HOME=$PWD java -cp $JAR org.apache.spark.launcher.Main org.apache.spark.deploy.SparkSubmit
    
    Against the assembly:
    
        avg: 0.156157615185, min: 0.148868083954, max: 0.186276912689
    
    Against the launcher jar:
    
        avg: 0.101513769627, min: 0.0895018577576, max: 0.130180120468
    
    So it's about 30% faster on average (with an SSD).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84205836
  
    Hey @vanzin - can you explain a bit why the performance impact isn't worrisome? I think a first order goal was to avoid invoking multiple JVM's that have the assembly jar in the classpath, because it is slow.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84165356
  
    Marcelo and I discussed this offline and we have agreed on how to resolve this issue:
    
    1. We will remove the check for launcher library and look for the uber jar instead
    2. For the regex expression problem, we will move that check out into spark-class (with hadoop string included in the check) and pass the assembly file location as an argument to Launcher. That way we can completely eliminate the check for spark assembly jar file name from Spark code.
    
    Will modify the PR accordingly after some testing


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5085#discussion_r27091105
  
    --- Diff: launcher/src/main/java/org/apache/spark/launcher/Main.java ---
    @@ -31,7 +31,7 @@
     class Main {
     
       /**
    -   * Usage: Main [class] [class args]
    +   * Usage: Main [uberJarPath] [class] [class args]
    --- End diff --
    
    Changes to this file are not needed anymore, right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-85800826
  
    retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-83861301
  
    Nishkam, can you explain why we need to remove "hadoop" from the regex now when the previous code had it? Can you explain one use case where the current code fails and the previous code worked?
    
    CDH ships the correctly named assembly with a symlink named "spark-assembly.jar". So it works fine with the existing code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84127213
  
    What backwards compatibility concerns do you have? The library didn't exist before, so it's pretty hard to think of any.
    
    As for performance, I didn't measure explicitly, but even if the gain is not that big, I really don't see why you see this as complicating anything.
    
    The issue with the breakage is not that it can be easily fixed. Is that it would exist at all. You'd be forcing people to fix their build when they did absolutely nothing wrong. What I haven't see here is any argument that supports relaxing the regex without using some use case where the user is intentionally breaking the Spark distribution.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84142521
  
    >  Is there an end-user problem that hasn't been voiced yet?
    
    No, but there is an end-user issue that has been voice already: it can break launching jobs after you build spark with sbt (e.g. `sbt package assembly`). That's why the old scripts also had the same naming restriction.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-85244894
  
    Thanks for the comments @vanzin. Will address them soon.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
GitHub user nishkamravi2 reopened a pull request:

    https://github.com/apache/spark/pull/5085

    [SPARK-6406] Launcher backward compatibility issue-- "hadoop" should not be mandatory in spark assembly name

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/nishkamravi2/spark master_nravi

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/5085.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5085
    
----
commit 681b36f5fb63e14dc89e17813894227be9e2324f
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-05-08T07:05:33Z

    Fix for SPARK-1758: failing test org.apache.spark.JavaAPISuite.wholeTextFiles
    
    The prefix "file:" is missing in the string inserted as key in HashMap

commit 5108700230fd70b995e76598f49bdf328c971e77
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-06-03T22:25:22Z

    Fix in Spark for the Concurrent thread modification issue (SPARK-1097, HADOOP-10456)

commit 6b840f017870207d23e75de224710971ada0b3d0
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-06-03T22:34:02Z

    Undo the fix for SPARK-1758 (the problem is fixed)

commit df2aeb179fca4fc893803c72a657317f5b5539d7
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-06-09T19:02:59Z

    Improved fix for ConcurrentModificationIssue (Spark-1097, Hadoop-10456)

commit eb663ca20c73f9c467192c95fc528c6f55f202be
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-06-09T19:04:39Z

    Merge branch 'master' of https://github.com/apache/spark

commit 5423a03ddf4d747db7261d08a64e32f44e8be95e
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-06-10T20:06:07Z

    Merge branch 'master' of https://github.com/apache/spark

commit 3bf8fad85813037504189cf1323d381fefb6dfbe
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-06-16T05:47:00Z

    Merge branch 'master' of https://github.com/apache/spark

commit 2b630f94079b82df3ebae2b26a3743112afcd526
Author: nravi <nr...@c1704.halxg.cloudera.com>
Date:   2014-06-16T06:00:31Z

    Accept memory input as "30g", "512M" instead of an int value, to be consistent with rest of Spark

commit efd688a4e15b79e92d162073035b03362fcf66f0
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-07-13T00:04:17Z

    Merge branch 'master' of https://github.com/apache/spark

commit 2e69f112d1be59951cd32da4127d8b51bfa03338
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-21T23:17:15Z

    Merge branch 'master' of https://github.com/apache/spark into master_nravi

commit ebcde10252e6c45169ea086e8426ec9997d46490
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-22T06:44:40Z

    Modify default YARN memory_overhead-- from an additive constant to a multiplier (redone to resolve merge conflicts)

commit 1cf2d1ef57ed6d783df06dad36b9505bc74329fb
Author: nishkamravi2 <ni...@gmail.com>
Date:   2014-09-22T08:54:33Z

    Update YarnAllocator.scala

commit f00fa311945c1eafa8957eae5c84719521761dcd
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-22T23:06:07Z

    Improving logging for AM memoryOverhead

commit c726bd9f707ce182ec8d56ffecf9da87dcdb3091
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-25T01:19:32Z

    Merge branch 'master' of https://github.com/apache/spark into master_nravi
    
    Conflicts:
    	yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala

commit 362da5edfd04bd8bad990fb210a9e11b8494fa62
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-25T19:56:13Z

    Additional changes for yarn memory overhead

commit 42c2c3d18862d3632c20931ecfe2c64883c5febf
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-25T20:02:49Z

    Additional changes for yarn memory overhead issue

commit dac1047995c99f5a2670f934eb8d3a4ad9b532c8
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-25T21:20:38Z

    Additional documentation for yarn memory overhead issue

commit 5ac2ec11629e19030ad5577da1eee2d135cc3d1c
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-25T21:25:44Z

    Remove out

commit 35daa6498048cabb736316e2f19e565c99243b7e
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-25T21:59:22Z

    Slight change in the doc for yarn memory overhead

commit 8f76c8b46379736aeb7dbe1a4d88729424a041f7
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-09-25T22:03:00Z

    Doc change for yarn memory overhead

commit 636a9ffeb4a4ae0b941edd849dcbabf38821db53
Author: nishkamravi2 <ni...@gmail.com>
Date:   2014-09-30T18:33:28Z

    Update YarnAllocator.scala

commit 5f8f9ede0fda5c7a4f6a411c746a3d893f550524
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-11-19T01:46:58Z

    Merge branch 'master' of https://github.com/apache/spark into master_nravi
    
    Conflicts:
    	yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
    	yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
    	yarn/common/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala

commit 9f6583eb92b59e88b0994ef45dd0e1a418d349a6
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-11-19T03:42:25Z

    Changes to maxResultSize code (improve error message and add condition to check if maxResultSize > 0)

commit 3e1b616a56cedec0b963b7f813a8e0195c3417e8
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-11-19T06:56:55Z

    Modify test for maxResultSize

commit 535295a371b259a4fb1eb2ef01ea9f396aed40dd
Author: nishkamravi2 <ni...@gmail.com>
Date:   2014-11-19T21:56:18Z

    Update TaskSetManager.scala

commit 5c9a4cb4eb12eb6cefbf9e9dda42b722abe1d3ed
Author: nishkamravi2 <ni...@gmail.com>
Date:   2014-11-19T21:57:01Z

    Update TaskSetManagerSuite.scala

commit b446edc5ce672139564bcbc98217862a2921783b
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2014-11-19T23:30:17Z

    Merge branch 'master' of https://github.com/apache/spark into master_nravi

commit 79ea8b418f709232ec6e106d7f58bb11ba86fa4a
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2015-02-19T06:03:18Z

    Merge branch 'master' of https://github.com/apache/spark into master_nravi

commit f0d12de3a2179d2ee32bf7ae049223b5ee246795
Author: Nishkam Ravi <nr...@cloudera.com>
Date:   2015-02-19T07:45:31Z

    Workaround for IllegalStateException caused by recent changes to BlockManager.stop

commit 3c5ddba51441b676391e43f3b09b9c59624ffef0
Author: nishkamravi2 <ni...@gmail.com>
Date:   2015-02-19T09:31:30Z

    Update DiskBlockManager.scala

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-85223186
  
    As I mentioned before, you need to take the "invalid CEN header" check from AbstractCommandBuilder and do it in the shell scripts now, otherwise that check is useless.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...

Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-92993834
  
    Yes, the path was `$SPARK_HOME/launcher/target/scala-2.10/classes/`. I'm in a mac.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84215052
  
    Hey @pwendell, a constant sub-second increase in execution time during the lifetime of an app should be acceptable. We tend to regress significantly more than with every Spark version as we pile up more code (we generally try to avoid reporting regressions of < 5%). Having to worry about two jar files as opposed to one on the other hand seems like a much more fundamental usability concern IMO. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by nishkamravi2 <gi...@git.apache.org>.
Github user nishkamravi2 commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-83922993
  
    Also, we should not look for a separate launcher library jar. Like any other spark component it should be part of the uber jar. Modified PR accordingly (please see modified spark-class).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launcher backward compatibility i...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-84060892
  
    Please *DO NOT* make a change to not have a separate launcher jar. That choice was explicit, because it's much, MUCH faster to load a jar file that is a few kB large than loading the uber jar.
    
    Seriously, I still don't see any bug here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/5085


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6406] Launch Spark using assembly jar i...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5085#issuecomment-86868649
  
      [Test build #29295 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29295/consoleFull) for   PR 5085 at commit [`bad4349`](https://github.com/apache/spark/commit/bad43499ed2518e0d5153ac56ce03beca3980e2c).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org