[jira] [Commented] (HADOOP-9854) Configuration.set() may be called before all the deprecated keys are registered, causing inconsistent state

["Sangjin Lee (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-9854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13734257#comment-13734257 ] 

Sangjin Lee commented on HADOOP-9854:
-------------------------------------

One easy way to recreate this problem is as follows. Do

{code}
hadoop jar some_jar -D mapred.reduce.tasks=100 ...
{code}

where my class uses ToolRunner to do its task:

{code}
public static void main(String[] args) {
  ToolRunner.run(new Configuration(), new MyClass(), args);
}
{code}

ToolRunner will start using GenericOptionsParser to parse the -D overrides, and it will call Configuration.set() to set the overrides. However, in this case the JobConf class is still not initialized. Therefore, the "mapred.reduce.tasks" key is not registered as deprecated. As a result, the Configuration will have "mapred.reduce.tasks" set to 100, but "mapreduce.job.reduces" (the new key) still set to 1 (the default).

The configuration will be left in this inconsistent state until the JobConf class is initialized and a get() call for either of these keys is made.

In our particular scenario, a map was created out of this Configuration using Configuration.iterator(), and eventually caused it to lose the overridden value.

I think this inconsistent state has many ways to cause havoc, and I believe this is a major bug.
                
> Configuration.set() may be called before all the deprecated keys are registered, causing inconsistent state
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-9854
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9854
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: conf
>    Affects Versions: 2.0.5-alpha
>            Reporter: Sangjin Lee
>
> Currently deprecated keys are registered at various times. Some are registered  when the Configuration class itself is initialized, but the vast majority are registered when the JobConf class is initialized.
> Therefore, it is entirely possible (and does happen) that Configuration.set() is called for a key before its deprecation mapping is registered, thus leaving the internal state of Configuration in an inconsistent state.
> We actually had this problem occur in real life, causing the set value not to be recognized.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira