You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Andrew Or (JIRA)" <ji...@apache.org> on 2015/02/27 01:56:04 UTC

[jira] [Created] (SPARK-6048) SparkConf.translateConfKey should translate on get, not set

Andrew Or created SPARK-6048:
--------------------------------

             Summary: SparkConf.translateConfKey should translate on get, not set
                 Key: SPARK-6048
                 URL: https://issues.apache.org/jira/browse/SPARK-6048
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.3.0
            Reporter: Andrew Or
            Assignee: Andrew Or
            Priority: Critical


There are several issues with translating on set.

(1) The most serious one is that if the user has both the deprecated and the latest version of the same config set, then the value picked up by SparkConf will be arbitrary. Why? Because during initialization of the conf we call `conf.set` on each property in `sys.props` in an order arbitrarily defined by Java. Instead, we should always use the value of the latest version of the config if that is provided.

(2) If we translate on set, then we must keep translating everywhere else. In fact, the current code does not translate on remove, which means the following won't work if X is deprecated:
{code}
conf.set(X, Y)
conf.remove(X) // X is not in the conf
{code}

(3) Since we call `conf.set` on all configs when initializing the conf, we print all deprecation warnings in the beginning. Elsewhere in Spark, however, we warn the user when the deprecated config / option / env var is actually being used.
We should keep this consistent so the user won't expect to find all deprecation messages in the beginning of his logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org