[jira] Commented: (HADOOP-167) reducing the number of Configuration & JobConf objects created

["Michel Tourn (JIRA)" <ji...@apache.org>]

    [ http://issues.apache.org/jira/browse/HADOOP-167?page=comments#action_12376516 ] 

Michel Tourn commented on HADOOP-167:
-------------------------------------

Avoiding the multiple config-loading messages is a good thing.
This could also be controlled with a verbosity  / logging level setting.

Please don't remove the JobConf(Configuration) constructor.

This is the only mechanism available to programatically change your Configuration.
We rely on this in two ways: 
1. to select from multiple XML config files that correspond to multiple Hadoop systems.
2. to make some properties (paths) user-dependant.

Ex for 1.
    config_ = new Configuration();
    config_.addFinalResource(getHadoopAliasConfFile());
    jobConf_ = new JobConf(config_);

In fact, it is important for Hadoop to maintain this property:
 ALL uses of a JobConf must be configurable at the outset by the caller by passing in a Configuration object.
 Common examples of such top-level Hadoop entry points: 
     Job submission, MapRed in local mode, DFS client calls.

In general we should make sure that we don't FORCE 
a long lifetime for a 'cached' JobConf object:
There are applications that need to use new JobConf-s along the way:
1. bec. they must first discover properties of the Hadoop cluster (list files, then submit job)
2. bec. they talk to multiple Hadoop systems (import / export files)


> reducing the number of Configuration & JobConf objects created
> --------------------------------------------------------------
>
>          Key: HADOOP-167
>          URL: http://issues.apache.org/jira/browse/HADOOP-167
>      Project: Hadoop
>         Type: Improvement

>   Components: conf
>     Versions: 0.1.1
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley
>      Fix For: 0.2
>  Attachments: remove-confs.patch
>
> Currently, Configuration and JobConf objects are created many times during executing a job. In particular, the Task Tracker creates a lot of them. They both clutter up the logs and parse the xml config files over and over again.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira