[jira] [Commented] (SPARK-11340) Support setting driver properties when starting Spark from R programmatically or from RStudio

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-11340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14975742#comment-14975742 ] 

Apache Spark commented on SPARK-11340:
--------------------------------------

User 'felixcheung' has created a pull request for this issue:
https://github.com/apache/spark/pull/9290

> Support setting driver properties when starting Spark from R programmatically or from RStudio
> ---------------------------------------------------------------------------------------------
>
>                 Key: SPARK-11340
>                 URL: https://issues.apache.org/jira/browse/SPARK-11340
>             Project: Spark
>          Issue Type: Bug
>          Components: SparkR
>    Affects Versions: 1.5.1
>            Reporter: Felix Cheung
>            Priority: Minor
>
> Currently when sparkR.init() is called in 'client' mode, it launches the JVM backend but driver properties (like driver-memory) are not passed or settable by the user calling sparkR.init().
> [~sunrui][~shivaram] and I discussed this offline and think we should support this.
> This is the original thread:
> >> From: rui.sun@intel.com
> >> To: dirceu.semighini@gmail.com
> >> CC: user@spark.apache.org
> >> Subject: RE: How to set memory for SparkR with master="local[*]"
> >> Date: Mon, 26 Oct 2015 02:24:00 +0000
> >>
> >> As documented in
> >> http://spark.apache.org/docs/latest/configuration.html#available-prop
> >> e
> >> rties,
> >>
> >> Note for “spark.driver.memory”:
> >>
> >> Note: In client mode, this config must not be set through the 
> >> SparkConf directly in your application, because the driver JVM has 
> >> already started at that point. Instead, please set this through the 
> >> --driver-memory command line option or in your default properties file.
> >>
> >>
> >>
> >> If you are to start a SparkR shell using bin/sparkR, then you can use 
> >> bin/sparkR –driver-memory. You have no chance to set the driver 
> >> memory size after the R shell has been launched via bin/sparkR.
> >>
> >>
> >>
> >> Buf if you are to start a SparkR shell manually without using 
> >> bin/sparkR (for example, in Rstudio), you can:
> >>
> >> library(SparkR)
> >>
> >> Sys.setenv("SPARKR_SUBMIT_ARGS" = "--conf spark.driver.memory=2g
> >> sparkr-shell")
> >>
> >> sc <- sparkR.init()
> >>
> >>
> >>
> >> From: Dirceu Semighini Filho [mailto:dirceu.semighini@gmail.com]
> >> Sent: Friday, October 23, 2015 7:53 PM
> >> Cc: user
> >> Subject: Re: How to set memory for SparkR with master="local[*]"
> >>
> >>
> >>
> >> Hi Matej,
> >>
> >> I'm also using this and I'm having the same behavior here, my driver 
> >> has only 530mb which is the default value.
> >>
> >>
> >>
> >> Maybe this is a bug.
> >>
> >>
> >>
> >> 2015-10-23 9:43 GMT-02:00 Matej Holec <ho...@gmail.com>:
> >>
> >> Hello!
> >>
> >> How to adjust the memory settings properly for SparkR with master="local[*]"
> >> in R?
> >>
> >>
> >> *When running from  R -- SparkR doesn't accept memory settings :(*
> >>
> >> I use the following commands:
> >>
> >> R>  library(SparkR)
> >> R>  sc <- sparkR.init(master = "local[*]", sparkEnvir =
> >> list(spark.driver.memory = "5g"))
> >>
> >> Despite the variable spark.driver.memory is correctly set (checked in 
> >> http://node:4040/environment/), the driver has only the default 
> >> amount of memory allocated (Storage Memory 530.3 MB).
> >>
> >> *But when running from  spark-1.5.1-bin-hadoop2.6/bin/sparkR -- OK*
> >>
> >> The following command:
> >>
> >> ]$ spark-1.5.1-bin-hadoop2.6/bin/sparkR --driver-memory 5g
> >>
> >> creates SparkR session with properly adjustest driver memory (Storage 
> >> Memory
> >> 2.6 GB).
> >>
> >>
> >> Any suggestion?
> >>
> >> Thanks
> >> Matej
> >>
> >>



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org