You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by aarondav <gi...@git.apache.org> on 2014/03/07 19:33:38 UTC

[GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM

GitHub user aarondav opened a pull request:

    https://github.com/apache/spark/pull/99

    SPARK-929: Fully deprecate usage of SPARK_MEM

    (Continued from old repo, prior discussion at https://github.com/apache/incubator-spark/pull/615)
    
    This patch cements our deprecation of the SPARK_MEM environment variable by replacing it with three more specialized variables:
    SPARK_DAEMON_MEMORY, SPARK_EXECUTOR_MEMORY, and SPARK_DRIVER_MEMORY
    
    The creation of the latter two variables means that we can safely set driver/job memory without accidentally setting the executor memory. Neither is public.
    
    SPARK_EXECUTOR_MEMORY is only used by the Mesos scheduler (and set within SparkContext). The proper way of configuring executor memory is through the "spark.executor.memory" property.
    
    SPARK_DRIVER_MEMORY is the new way of specifying the amount of memory run by jobs launched by spark-class, without possibly affecting executor memory.
    
    Other memory considerations:
    - The repl's memory can be set through the "--drivermem" command-line option, which really just sets SPARK_DRIVER_MEMORY.
    - run-example doesn't use spark-class, so the only way to modify examples' memory is actually an unusual use of SPARK_JAVA_OPTS (which is normally overriden in all cases by spark-class).
    
    This patch also fixes a lurking bug where spark-shell misused spark-class (the first argument is supposed to be the main class name, not java options), as well as a bug in the Windows spark-class2.cmd. I have not yet tested this patch on either Windows or Mesos, however.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/aarondav/spark sparkmem

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/99.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #99
    
----
commit 9df4c68262dac0edde1ec5bdd1fd065d2bf34e00
Author: Aaron Davidson <aa...@databricks.com>
Date:   2014-02-17T23:09:51Z

    SPARK-929: Fully deprecate usage of SPARK_MEM
    
    This patch cements our deprecation of the SPARK_MEM environment variable
    by replacing it with case-specific variables:
    SPARK_DAEMON_MEMORY, SPARK_EXECUTOR_MEMORY, and SPARK_DRIVER_MEMORY
    
    The creation of the latter two variables means that we can safely
    set driver/job memory without accidentally setting the executor memory.
    Neither is public.
    
    SPARK_EXECUTOR_MEMORY is only used by the Mesos scheduler (and set
    within SparkContext). The proper way of configuring executor memory
    is through the "spark.executor.memory" property.
    
    SPARK_DRIVER_MEMORY is the new way of specifying the amount of memory
    run by jobs launched by spark-class, without possibly affecting executor
    memory.
    
    Other memory considerations:
    - The repl's memory can be set through the "--drivermem" command-line option,
      which really just sets SPARK_DRIVER_MEMORY.
    - run-example doesn't use spark-class, so the only way to modify examples'
      memory is actually an unusual use of SPARK_JAVA_OPTS (which is normally
      overriden in all cases by spark-class).
    
    This patch also fixes a lurking bug where spark-shell misused spark-class
    (the first argument is supposed to be the main class name, not java
    options).

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/99


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/99#issuecomment-37059804
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/99#issuecomment-37059803
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/99#issuecomment-37064301
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13054/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM

Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the pull request:

    https://github.com/apache/spark/pull/99#issuecomment-37891403
  
    Digging into this a little more, it looks like SPARK_WORKER_MEMORY is still used in several places, and SPARK_MASTER_MEMORY is there too.  Some of those are related to YARN, which I've never used.  Mesos seems to have its own thing too.
    
    Can you help me figure out how to configure each thing?  I've opened up https://spark-project.atlassian.net/browse/SPARK-1264 to track what the truth is and then I'll write that into the documentation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM

Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:

    https://github.com/apache/spark/pull/99#issuecomment-37091450
  
    Hey @aarondav this LGTM. Anything else before you can merge this? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/99#issuecomment-37058577
  
    One or more automated tests failed
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13051/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/99#issuecomment-37058576
  
    Merged build finished.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/99#issuecomment-37053201
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/99#issuecomment-37064300
  
    Merged build finished.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM

Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on the pull request:

    https://github.com/apache/spark/pull/99#issuecomment-37213831
  
    You might also look at my comment summarizing how one can configure memory for each of the components here: https://github.com/apache/incubator-spark/pull/615#issuecomment-35818768. There is also sometimes a Client JVM which simply submits a driver to be run inside a cluster, but right now we can't configure this memory separately (it's transient and low-memory anyway). Also sometimes there may be more than 1 master for fault tolerance purposes.
    
    SPARK_DAEMON_MEMORY does control both the master and worker JVM heap sizes -- I think they were mainly introduced so that the daemons would not use SPARK_MEM rather than to really make the daemon memory configurable, since they use little memory.
    
    More documentation is always useful! Sometimes PRs and code aren't the best place for informing users...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM

Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on the pull request:

    https://github.com/apache/spark/pull/99#issuecomment-37059772
  
    Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM

Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the pull request:

    https://github.com/apache/spark/pull/99#issuecomment-37155320
  
    I'm used to thinking of Spark standalone clusters as having 4 different "types" of JVMs -- 1 driver, 1 master, N workers, and M executors.  Does SPARK_DAEMON_MEMORY control both the master and worker JVM heap size?
    
    I'd like to add some clarifying documentation as to how these are used as well.  Do you mind if I take the description in this PR as a base for the documentation?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: SPARK-929: Fully deprecate usage of SPARK_MEM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/99#issuecomment-37053200
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---