You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by witgo <gi...@git.apache.org> on 2014/06/26 17:53:00 UTC

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

GitHub user witgo opened a pull request:

    https://github.com/apache/spark/pull/1233

    [WIP] Loading spark-defaults.conf when creating SparkConf instances

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/witgo/spark defaults-conf

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/1233.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1233
    
----
commit 135343cf45890c9868bf428a13399a8cad0dd604
Author: witgo <wi...@qq.com>
Date:   2014-06-26T15:50:47Z

    Loading spark-defaults.conf when creating SparkConf instances

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1233#issuecomment-47244019
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1233#issuecomment-47245240
  
    Merged build started. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1233#issuecomment-47244487
  
    Merged build finished. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1233#issuecomment-47250100
  
    Merged build finished. All automated tests passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:

    https://github.com/apache/spark/pull/1233#issuecomment-47428611
  
    @vanzin  I submitted a new PR #1256 . I close this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/1233#issuecomment-47252173
  
    Hi @witgo, can you elaborate, in the change summary, why this is needed? Maybe file a bug?
    
    I think you guys don't use spark-submit, which is why you might need this; but spark-submit translates the config file into system properties, so the config is picked up by SparkConf. And it seems like this change is breaking the semantics of how spark-submit works.
    
    (I think it would be nice to stop using system properties like that - it gets really confusing when parts of the code use system properties and others use SparkConf - but that's a separate discussion.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by witgo <gi...@git.apache.org>.
Github user witgo closed the pull request at:

    https://github.com/apache/spark/pull/1233


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1233#issuecomment-47244003
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1233#discussion_r14252544
  
    --- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
    @@ -291,4 +294,25 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {
       def toDebugString: String = {
         settings.toArray.sorted.map{case (k, v) => k + "=" + v}.mkString("\n")
       }
    +
    +  def loadDefaultProperties() {
    +    var seq = Seq[(String, String)]()
    +
    +    val defaultConfigFile = "spark-defaults.conf"
    --- End diff --
    
    I think this will end up causing disagreement with spark-submit options.
    
    spark-submit will only load spark-defaults.conf if `--properties-file` is not defined in the command line. It seems like this code will always load spark-defaults.conf if one exists.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1233#issuecomment-47244488
  
    
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16159/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1233#issuecomment-47250102
  
    All automated tests passed.
    Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16160/


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:

    https://github.com/apache/spark/pull/1233#issuecomment-47372715
  
    @vanzin The situation is  `sbin/start-*.sh` are not support `spark-defaults.conf`. 
    
    eg:  `sbin/start-history-server.sh`  cannot load the`spark.history.fs.logDirectory` configuration  from `spark-defaults.conf`. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/1233#issuecomment-47245226
  
     Merged build triggered. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:

    https://github.com/apache/spark/pull/1233#issuecomment-47373161
  
    Ah, so it's SPARK-2098.
    
    I think it's a nice feature to have (I filed the bug after all), but we can't break the existing semantics. For daemons, the command line parsers could do that (by having a "--properties-file" argument similar to spark-submit).
    
    But if you want to support arbitrary SparkConf instances to read these conf files, it will become trickier, since now you need to propagate that command line information somehow.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:

    https://github.com/apache/spark/pull/1233#issuecomment-47373587
  
    You're right, the corresponding code should be submitted at the weekend.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] spark pull request: [WIP] Loading spark-defaults.conf when creatin...

Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:

    https://github.com/apache/spark/pull/1233#discussion_r14252580
  
    --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
    @@ -383,20 +382,5 @@ private[spark] class SparkSubmitArguments(args: Seq[String]) {
     
     object SparkSubmitArguments {
       /** Load properties present in the given file. */
    -  def getPropertiesFromFile(file: File): Seq[(String, String)] = {
    -    require(file.exists(), s"Properties file $file does not exist")
    -    require(file.isFile(), s"Properties file $file is not a normal file")
    -    val inputStream = new FileInputStream(file)
    -    try {
    -      val properties = new Properties()
    -      properties.load(inputStream)
    -      properties.stringPropertyNames().toSeq.map(k => (k, properties(k).trim))
    -    } catch {
    -      case e: IOException =>
    -        val message = s"Failed when loading Spark properties file $file"
    -        throw new SparkException(message, e)
    -    } finally {
    -      inputStream.close()
    -    }
    -  }
    +  def getPropertiesFromFile(file: File) = Utils.getPropertiesFromFile(file)
    --- End diff --
    
    At this point, might as well remove the object.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---