[jira] [Resolved] (SPARK-8135) Don't load defaults when reconstituting Hadoop Configurations

["Josh Rosen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-8135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Josh Rosen resolved SPARK-8135.
-------------------------------
       Resolution: Fixed
    Fix Version/s: 1.5.0

Issue resolved by pull request 6679
[https://github.com/apache/spark/pull/6679]

> Don't load defaults when reconstituting Hadoop Configurations
> -------------------------------------------------------------
>
>                 Key: SPARK-8135
>                 URL: https://issues.apache.org/jira/browse/SPARK-8135
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>             Fix For: 1.5.0
>
>
> Calling "new Configuration()" is an expensive operation because it loads any Hadoop configuration XMLs from disk.
> In SerializableWritable, we call new Configuration needlessly when instantiating an ObjectWritable.  The ObjectWritable only needs the Configuration for its class cache, not for any Hadoop properties that might be in XML files, so it should be ok to call new Configuration with loadDefaults = false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org