[jira] [Commented] (HADOOP-12021) Augmenting Configuration to accomodate

["Joep Rottinghuis (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-12021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14557135#comment-14557135 ] 

Joep Rottinghuis commented on HADOOP-12021:
-------------------------------------------

If you are providing a patch, you'll probably will want to make it optional to turn on.
Configs get pushed out to the cluster for each job. Already configs can get pretty large. Adding text will only make them larger. Each node that launches containers for a job needs to receive these configs. Memory and network pressure will certainly mount.

When doing work for hRaven and YARN-2928 we do find that configs can be large and costly at times.
I suspect that changing a simple key->value map into something more complex will be interesting.

Btw. during debugging issues on our clusters we often see unzip taking considerable time. That usually indicated that a Configuration object is getting created (which often reads jars from the classpath). Making that even more expensive will probably not be great.
Perhaps you can provide a special constructor to create an extensively loaded Configuration so that you're not burdening everybody else, but I don't know if you need the full description to be passed around on the cluster, or if this a client-side only use case.

> Augmenting Configuration to accomodate <description>
> ----------------------------------------------------
>
>                 Key: HADOOP-12021
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12021
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: conf
>            Reporter: Lewis John McGibbney
>            Priority: Minor
>             Fix For: 1.3.0, 2.8.0
>
>
> Over on the [common-dev|http://www.mail-archive.com/common-dev%40hadoop.apache.org/msg16099.html] ML I explained a use case which requires me to obtain the value of the Configuration <description> tags.
> [~cnauroth] advised me to raise the issue to Jira for discussion.
> I am happy to provide a patch so that the <description> values are parsed out of the various XML files and stored, and also that the Configuration class is augmented to provide accessors to accommodate the use case.
> I wanted to find out what people think about this one and whether I should check out Hadoop source and submit a patch. If you guys could provide some advice it would be appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)