ConfigKeys wrappers for MapReduce source code base

[Jay Vyas <ja...@gmail.com>]

Hi guys:

A breif check wiht "find ./ -name *ConfigKeys*" doesn't seem to indicate
that there is a "MapRedConfigKeys" class... Should there be one to help get
rid of magic numbers and unify the namespace?  This seems to be the goal of
the DFSConfigKeys class in the HDFS source tree.

... Or are there differences in the way configuration values are handled in
mapred versus hdfs code bases ?

For example :

"job.getInt(JobContext.IO_SORT_FACTOR, 100)" in the ReduceTask class would
more typically be implemented (if in hdfs) using the DFSConfigKeys static
class which stores defaults and configuration parameter names.

Just curious wether there is any goal to take the Mapred configuration
values and unify their namespace in mapred/common in the same way that
seems to have been done in hdfs using the DFSConfigKeys class.

-- 
Jay Vyas
http://jayunit100.blogspot.com

Re: ConfigKeys wrappers for MapReduce source code base

[Sandy Ryza <sa...@cloudera.com>]

Hi Jay,

In trunk, these keys are in MRJobConfig (extended by JobContext) and
MRConfig.  I'm not sure what the state of things are in branch-1.  Agreed
that it would be nice for the naming convention to be consistent with HDFS,
but personally I like *Config more than *ConfigKeys better because the
class holds default values as well.

-Sandy


On Sun, Apr 21, 2013 at 12:29 PM, Jay Vyas <ja...@gmail.com> wrote:

> Hi guys:
>
> A breif check wiht "find ./ -name *ConfigKeys*" doesn't seem to indicate
> that there is a "MapRedConfigKeys" class... Should there be one to help get
> rid of magic numbers and unify the namespace?  This seems to be the goal of
> the DFSConfigKeys class in the HDFS source tree.
>
> ... Or are there differences in the way configuration values are handled in
> mapred versus hdfs code bases ?
>
> For example :
>
> "job.getInt(JobContext.IO_SORT_FACTOR, 100)" in the ReduceTask class would
> more typically be implemented (if in hdfs) using the DFSConfigKeys static
> class which stores defaults and configuration parameter names.
>
> Just curious wether there is any goal to take the Mapred configuration
> values and unify their namespace in mapred/common in the same way that
> seems to have been done in hdfs using the DFSConfigKeys class.
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>